Monitoring - Storing Prometheus metrics in the INFLUXDB engine

dHENRY
16/01/2020
(Reading time : 9 mn)

(**) Translated with www.DeepL.com/Translator

As stated in the Prometheus documentation :

[..]
Note that a limitation of the local storage is that it is not clustered or replicated. Thus, it is not arbitrarily scalable or durable in the face of disk or node outages and should be treated as you would any other kind of single node database. Using RAID for disk availiablity, snapshots for backups, capacity planning, etc, is recommended for improved durability. With proper storage durability and planning storing years of data in the local storage is possible.

Alternatively, external storage may be used via the remote read/write APIs. Careful evaluation is required for these systems as they vary greatly in durability, performance, and efficiency.
[..]

(Source)

Understand that the storage system is not scalable as a database system can be. Prometheus supports multiple database engines. I tried Postgres, inconclusive on Raspberry Pi or Rock64 type units (obvious lack of performance). While the InfluxDB engine does the job perfectly.

This article explains how to install the InfluxDB system on Rock64, how to couple Prometheus with Influxdb and how to check that it works. This procedure should also work on a Raspberry PI4 and easily be adapted to other platforms supported by InfluxDB. InfluxDB is not intended to work on Raspberry PI3.

Please note: Grafana requires no modifications

Installing the InfluxDB service

On the future InfluxDB server (Rock64 or Raspberry PI4), copy the following content into the file /tmp/installinflux.sh. This shell installs the InfluxDB engine version 1.7.9 :

#!/bin/bash
######################################################################
## Built with mytinydc-automation process - https://www.mytinydc.com
######################################################################
export APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE="DontWarn"
export DEBIAN_FRONTEND="noninteractive"

apt-get update
apt-get -y install curl net-tools
########################################################
UserExec=influxdb
LINK="https://dl.influxdata.com/influxdb/releases/influxdb-1.7.9_linux_armhf.tar.gz"
SERVICEFILE="/etc/systemd/system/influxdb.service"
INFLUXDBDIR="/var/lib/influxdb"
INFLUXDBFILESLISTINSTALLED="/tmp/influxdbfileslistinstalled"


SERVER="`uname -n`"
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Code Erreur] $ERR" >&2
    echo "[Commande] SERVER=`uname -n`" >&2
    exit $ERR
fi

ARCH="`uname -m`"
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] ARCH=`uname -m`" >&2
    exit $ERR
fi

if [ -f "/usr/bin/influxd" ];then
    echo "[ERR] An installation already exists "
    exit 1
fi

cd /tmp
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] cd /tmp" >&2
    exit $ERR
fi

curl "$LINK" | tar xvfz -
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] curl $LINK | tar xvfz -" >&2
    exit $ERR
fi

cd influxdb-1.7.9*
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] cd influxdb-1.7.9*" >&2
    exit $ERR
fi
# Map list files
find  . > $INFLUXDBFILESLISTINSTALLED
cat $INFLUXDBFILESLISTINSTALLED
echo "-----"
echo ""
echo "[INFO] Above is a list of files installed on your system."
echo "       Keep a copy of this list"
echo "       to allow a removal of this service."
echo "       This list is also available in the file: "
echo "       $INFLUXDBFILESLISTINSTALLED"
echo "-----"

cp -rp * /.
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] cp -rp service /." >&2
    exit $ERR
fi

ISGROUPEXISTS=`getent group $UserExec` #DONTCATCH
if [ "$ISGROUPEXISTS" = "" ];then
    addgroup $UserExec
    ERR=$?
    if [ "$ERR" != "0" ];then
        echo "[Code Erreur] $ERR" >&2
        echo "[Serveur] $SERVER" >&2
        echo "[Commande] addgroup $UserExec" >&2
        exit $ERR
    fi
fi

ISUSEREXISTS=`getent passwd $UserExec` #DONTCATCH
if [ "$ISUSEREXISTS" = "" ];then
    adduser --system --home /var/lib/infludb --no-create-home --disabled-password  --shell /bin/nologin --ingroup $UserExec $UserExec
    ERR=$?
    if [ "$ERR" != "0" ];then
        echo "[Code Erreur] $ERR" >&2
        echo "[Serveur] $SERVER" >&2
        echo "[Commande] adduser --system --home /var/lib/infludb --no-create-home --disabled-password --shell /bin/nologin --ingroup $UserExec $UserExec" >&2
        exit $ERR
    fi
fi

# InfluxDB data directory Debian compliance
mkdir -p $INFLUXDBDIR
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] mkdir -p $INFLUXDBDIR" >&2
    exit $ERR
fi
# permissions
chown -R $UserExec:$UserExec $INFLUXDBDIR
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] chown -R $UserExec:$UserExec /var/lib/influxdb" >&2
    exit $ERR
fi
chmod 750 /var/lib/influxdb
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] chmod 750 /var/lib/influxdb" >&2
    exit $ERR
fi

# Nettoyage
rm -rf /tmp/influxdb-1.7.9*
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] rm -f /tmp/influxdb-1.7.9*" >&2
    exit $ERR
fi

## Service File
cat << 'EOT' > $SERVICEFILE
[Unit]
Description=InfluxDB service - #ARCH#
After=network.target syslog.target

[Service]
User=influxdb
Group=influxdb
Type=simple
ExecStart=/usr/bin/influxd  -config /etc/influxdb/influxdb.conf
Restart=on-failure
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=influxdb

[Install]
WantedBy=multi-user.target
EOT
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] service file creation $SERVICEFILE" >&2
    exit $ERR
fi


# permission service file
chmod 644 $SERVICEFILE
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] chmod 644 $SERVICEFILE" >&2
    exit $ERR
fi

# Replace #ARCH# fichier service
sed -i "s/#ARCH#/$ARCH/" $SERVICEFILE
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] sed -i "s/#ARCH#/$ARCH/" $SERVICEFILE" >&2
    exit $ERR
fi


systemctl daemon-reload
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] systemctl daemon-reload" >&2
    exit $ERR
fi

echo "[OK] Installation complete"

Make this shell executable :

chmod 755 installinflux.sh

Execute the installation :

./installinflux.sh

Modification of the influxDB configuration

**My server’s IP address is 172.21.0.11 **

vi /etc/influxdb/influxdb.conf

# Do not send telemetry add this line
reporting-disabled = true

# Data path
[meta]
  # Where the metadata/raft database is stored
  dir = "/var/lib/influxdb/meta"
[data]
  # The directory where the TSM storage engine stores TSM files.
  dir = "/var/lib/influxdb/data"

  # The directory where the TSM storage engine stores WAL files.
  wal-dir = "/var/lib/influxdb/wal"

[http]
  # Listening address indicate the IP address of the server on the network
  bind-address = "172.21.0.11:8086"
  
  # Disable http logs
  log-enabled = false
  write-tracing = false

[logging]
  # Log only errors 
  level = "error"

[continuous_queries]
  # Deactivate logs
  log-enabled = false

Starting the InfluxDB service

systemctl start influxdb

Checking the operation of the service

# service started ?
systemctl status influxdb

# Typical answer : 
● influxdb.service - InfluxDB service - armhf
   Loaded: loaded (/etc/systemd/system/influxdb.service; disabled; vendor preset: enabled)
   Active: active (running) since Sat 2020-01-18 10:32:29 CET; 4s ago
 Main PID: 8394 (influxd)
    Tasks: 6 (limit: 1150)
   Memory: 22.6M
   CGroup: /system.slice/influxdb.service
           └─8394 /usr/bin/influxd -config /etc/influxdb/influxdb.conf

Check listening on ports 8086 and 8088.

The IP address of my server is 172.21.0.11, port 8086 is listening on the interface connected to the network, to be reachable by the Prometheus server, port “8088” is listening on localhost only.

netstat -ant|grep -E "8088|8086"

# Typical answer
tcp        0      0 172.21.0.11:8086         0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:8088          0.0.0.0:*               LISTEN

Creation of the Prometheus base

On the InfluxDB server

$ influx -precision rfc3339
Connected to http://localhost:8086 version 1.7.x
InfluxDB shell 1.7.x
>

type

CREATE DATABASE "prometheus"
exit

Firewall

On the InfluxDB server :

Port	direction	State	Source
8086	Input	Open	Prometheus server

On the Prometheus server :

Port	direction	State	Destination
8086	Output	Open	InfluxDB server

Prometheus server settings

modify the configuration file /opt/prometheus/prometheus.yaml by adding after the “global” property

global:
[..]
remote_write:
  - url: "http://[Adresse IP or FQDN InfluxDB server]:8086/api/v1/prom/write?db=prometheus"

remote_read:
  - url: "http://[Adresse IP or FQDN InfluxDB server ]:8086/api/v1/prom/read?db=prometheus"
[..]

Example for the InfluxDB server with the FQDN “infludb.mtdc”.

global:
[..]
remote_write:
  - url: "http://infludb.mtdc:8086/api/v1/prom/write?db=prometheus"

remote_read:
  - url: "http://infludb.mtdc:8086/api/v1/prom/read?db=prometheus"
[..]

Example for the InfluxDB server with IP address “172.21.0.11”.

global:
[..]
    remote_write:
    - url: "http://172.21.0.11:8086/api/v1/prom/write?db=prometheus"

    remote_read:
    - url: "http://172.21.0.11:8086/api/v1/prom/read?db=prometheus"
[..]

Starting the Prometheus startup logs display in the background :

tail -f /var/log/syslog &

Restart of the Prometheus service

systemctl restart prometheus

Prometheus injects the data into the InfluxDB engine :

Nov 10 00:05:00 raspi4 influxdb[10898]: [httpd] 127.0.0.1 - username [10/Nov/2019:00:05:00 +0100] "POST /api/v1/prom/write?db=prometheus&p=%5BREDACTED%5D&u=username HTTP/1.1" 204 0 "-" "Prometheus/2.6.0" 5b14d2df-0345-11ea-818f-b827eb69fd03 29699
Nov 10 00:05:00 raspi4 influxdb[10898]: [httpd] 127.0.0.1 - username [10/Nov/2019:00:05:00 +0100] "POST /api/v1/prom/write?db=prometheus&p=%5BREDACTED%5D&u=username HTTP/1.1" 204 0 "-" "Prometheus/2.6.0" 5b1a8954-0345-11ea-8190-b827eb69fd03 26346
Nov 10 00:05:00 raspi4 influxdb[10898]: [httpd] 127.0.0.1 - username [10/Nov/2019:00:05:00 +0100] "POST /api/v1/prom/write?db=prometheus&p=%5BREDACTED%5D&u=username HTTP/1.1" 204 0 "-" "Prometheus/2.6.0" 5b1fbeb1-0345-11ea-8191-b827eb69fd03 25807
Nov 10 00:05:00 raspi4 influxdb[10898]: [httpd] 127.0.0.1 - username [10/Nov/2019:00:05:00 +0100] "POST /api/v1/prom/write?db=prometheus&p=%5BREDACTED%5D&u=username HTTP/1.1" 204 0 "-" "Prometheus/2.6.0" 5b24cd98-0345-11ea-8192-b827eb69fd03 29254

Stop viewing logs :

fg
CTRL+C

Let it run for 3 hours while checking your Grafana dashboard..

Verification of correct operation

To check, open a Grafana Dashboard, with the data of the last 3 hours..

# End of Prometheus Service
systemctl stop prometheus
cd /opt/prometheus/
# Copy the data directory to data.old
cp -rp data data.old
# Deleting data located in the directory **data**
rm -rf data/*
systemctl start prometheus

Check that your Grafana dashboard displays the data for the last 3 hours.
The Prometheus data are no longer accessible, so it is the InfluxDB data that feeds the Grafana dashboards. You can delete the /opt/prometheus/data.old directory.

InfluxDB data backup

Adding a scheduling file : /etc/cron.d/backup-influxDB

#INFLUXDB at 23:00 every day in the directory /var/lib/backup-influxdb/
0 23 * * * root /usr/bin/influxd backup -portable -database prometheus /var/lib/backup-influxdb/
# Cleaning of influxDB backups at 1h every day
0 1 * * * root /opt/mytinydc/sbin/backup-influxdb-nettoyage.sh

Create the directory /var/lib/backup-influxdb/

mkdir /var/lib/backup-influxdb/

Create the file /opt/mytinydc/sbin/backup-influxdb-nettoyage.sh

mkdir -p /opt/mytinydc/sbin/

# Create the file /opt/mytinydc/sbin/backup-influxdb-nettoyage.sh use this content : 
#!/bin/bash
DIRBACKUP=/var/lib/backup-influxdb/
NBRJOURRETENTION=5

## Functions

function end_success {
echo ""
echo "*****************************************************************"
echo " Successfully completed treatment"
echo "*****************************************************************"
exit 0
}

function end_unsuccess {
echo ""
echo "*****************************************************************"
echo " An error has occurred : $ERR"
echo "*****************************************************************"
}

# Deletion of backups of more than $NBREJOURRETENTION
echo "*****************************************************************"
echo "   Night processing - Influxdb - Backup cleaning"
echo "*****************************************************************"
echo " Parameters :"
echo "         - Directory : $DIRBACKUP"
echo "         - Number of days (retention) : $NBRJOURRETENTION"
echo ""
echo " List of files to be deleted : "
FILES=`find $DIRBACKUP -type f -mtime +$NBRJOURRETENTION`
if [ "$FILES" = "" ];then
echo "      None"
end_success
else
for F in $FILES
do
echo "    * $F"
done
fi
echo ""
find $DIRBACKUP -type f -mtime +$NBRJOURRETENTION -exec /bin/rm -f {} \;
ERR=$?
if [ "$ERR" = "0" ];then
end_success
else
end_unsuccess
exit $ERR
fi

Make this file executable :

chmod 755 /opt/mytinydc/sbin/backup-influxdb-nettoyage.sh

The “root” user will receive two emails every day, mentioning the progress of these operations.

Interesting links

Document licence : Creative Commons (CC BY-NC-ND 4.0)

THIS DOCUMENTATION IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND AND DISTRIBUTED FOR EDUCATIONAL PURPOSES ONLY. THE AUTHOR, CONTRIBUTORS TO THIS DOCUMENTATION OR ©MYTINYDC.COM SHALL IN NO EVENT BE LIABLE FOR ANY DIRECT OR INDIRECT DAMAGE THAT MAY RESULT FROM THE APPLICATION OF THE PROCEDURES IMPLEMENTED IN THIS DOCUMENTATION, OR FROM THE INCORRECT INTERPRETATION OF THIS DOCUMENT.

(**) Translated with www.DeepL.com/Translator