Update the MATRIX service

dHENRY
06/05/2020
(Reading time : 9 mn)

(**) Translated with www.DeepL.com/Translator

The update will take place in 3 steps:

saving the production source code,
database backup,
Matrix code update .

It is important that each of the steps starts when the previous step went well.

Database and authentication

The shell that will perform the backup operation will not include the Matrix database access account or password. This information will be “retrieved” using a “YAML parser”, which will extract this information from the MATRIX configuration file (homeserver.yaml). The nugget, in this process, is assigned to Stefan Farestam which offers this functionality through the “sed” command, and can be used directly with bash :

#!/bin/bash
# This source code is the property of its author Stefan Farestam. ( https://stackoverflow.com/a/21189044)
# Parser yaml source :
# https://stackoverflow.com/questions/5014632/how-can-i-parse-a-yaml-file-from-a-linux-shell-script
#
#
if [ "$1" = "" ];then
    echo "You must provide the complete location of the yaml file to read"
    exit 1
fi
function parse_yaml {
   local prefix=$2
   local s='[[:space:]]*' w='[a-zA-Z0-9_]*' fs=$(echo @|tr @ '\034')
   sed -ne "s|^\($s\):|\1|" -e "s|^\($s\)\($w\)$s:$s[\"']\(.*\)[\"']$s\$|\1$fs\2$fs\3|p" -e "s|^\($s\)\($w\)$s:$s\(.*\)$s\$|\1$fs\2$fs\3|p"  $1 |
   awk -F$fs '{
               indent = length($1)/2;
               vname[indent] = $2;
               for (i in vname) {if (i > indent) {delete vname[i]}}
                    if (length($3) > 0) {
                     vn=""; for (i=0; i<indent; i++) {vn=(vn)(vname[i])("_")}
                      printf("%s%s%s=\"%s\"\n", "'$prefix'",vn, $2, $3);
                    }
               }'
    }

parse_yaml "$1"

Robot

To perform this operation you need the package “mytinydc-automation” (https://www.mytinydc.com/automates/)

Creating the robot

Type the command: mytinydc-automation.sh -c matrix-update. Paste this code and adjust the value of the variables according to your environment:


# Il s'agit d'un automate d'upgrade
# Je considère que les paquets DEBIAN nécessaire ont déjà été
# installés lors de la mise en place de ce service
#RESS tools-parser-yaml.sh

apt -y install postgresql-client

DATEJOUR=`date +%Y-%m-%d-%H-%M-%S`
DIRMATRIX="/opt/synapse"
DIRBACKUP="/opt/synapse/ressources/backupcode"
OPTTAR="--exclude=ressources --exclude=homeserver.log*"
SERVICEMATRIX="matrix-synapse"

# Pré-check
if [ ! -d "$DIRMATRIX" ];then
    echo "[ERR] Matrix n'est pas installé sur ce système"
    exit 1
fi
if [ ! -d "$DIRBACKUP" ];then
    mkdir -p "$DIRBACKUP"
fi
#Parse le fichier de conf yaml de matrix
ENVMATRIXYAML="/tmp/envmatrix"
if [ ! -x "$RESS_1" ];then
    echo "[ERR] Impossible de continuer $RESS_1 n'est pas exécutable"
    exit 1
fi
$RESS_1 $DIRMATRIX/homeserver.yaml > $ENVMATRIXYAML
# Sécurité
chmod 600 $ENVMATRIXYAML
source $ENVMATRIXYAML
rm $ENVMATRIXYAML

# Upgrade du service avec backup base et code
    # Stop matrix
    systemctl stop $SERVICEMATRIX
    # Sauvegarde du code
    TARFILE="$DIRBACKUP/backup-matrix-synapse-$DATEJOUR.tgz"
    tar $OPTTAR -cvzf "$TARFILE" "$DIRMATRIX" 2>&1
    # Backup Database
    BACKUPDONE="0"
    if [ "$database_name" = "psycopg2" ];then
        echo "[INFO] Database is postgresl - try to backup"
        if [ "$database_args_user" != "" ] && [ "$database_args_password" != "" ] && [ "$database_args_database" != "" ] && [ "$database_args_host" != "" ] && [ "database_args_port" != "" ];then

            echo "[INFO] backup in progress vers $DIRBACKUP/backup-database-$DATEJOUR.sql"
            export PGPASSWORD="$database_args_password"
            pg_dump -d "$database_args_database" -h "$database_args_host" -p "$database_args_port" -U "$database_args_user"  > "$DIRBACKUP/backup-database-$DATEJOUR.sql"
            unset PGPASSWORD
            echo "[INFO] backup done"
            if [ -f "$DIRBACKUP/backup-database-$DATEJOUR.sql" ];then
	        BACKUPDONE="1"
            fi
        fi
    fi
    if [ "$BACKUPDONE" = "0" ];then
	echo "[ERREUR] La mise à jour ne peut être effectuée"
        echo "         la base de données n'a pas été sauvegardée"
	exit 1
    fi
    # update du service
    cd "$DIRMATRIX"
    source $DIRMATRIX/env/bin/activate
    pip install --upgrade pip
    pip install --upgrade setuptools
    pip install --upgrade matrix-synapse[all]
    deactivate
    # Démarrage du service
    systemctl start $SERVICEMATRIX
    echo "[INFO] Code sauvegardé dans $TARFILE"
    echo "[INFO] Database sauvegardée dans $DIRBACKUP/backup-database-$DATEJOUR.sql"

Save and exit

Then type :

mytinydc-automation.sh matrix-update -cd
#then create the resource file tools-parser-yaml.sh
vi resources/tools-parser-yaml.sh
# and paste the bash code for the YAML parser located above
# Save
# Make this file executable
chmod 700 resources/tools-parser-yaml.sh
# then
exit

Run the automaton by passing the address of the server that hosts Matrix, (localhost if the automaton system is installed on the Matrix server): mytinydc-automation.sh matrix-update [FQDN or Matrix server ip address] -exec -dsp.

The service restarts automatically after the update.

Without mytinydc-automation

On the Matrix server, create the file “/tmp/tools-parser-yaml.sh” and paste the bash code for the YAML parser located above. Save, then make it executable: chmod 700 /tmp/tools-parser-yaml.sh.

Create the file /tmp/matrix-update.sh and paste these lines (you will have to adapt the values of the variables used according to your configuration):

#!/bin/bash
######################################################################
## Built with mytinydc-automation process - https://www.mytinydc.com
######################################################################
export APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE="DontWarn"
export DEBIAN_FRONTEND="noninteractive"
export DATEBUILT="Wed May  6 23:42:31 CEST 2020"
export SERVER="`uname -n`"
export PARAMSSH="-q -o UserKnownHostsFile=/dev/null -o PasswordAuthentication=no -o ConnectTimeout=5 -o StrictHostKeyChecking=no"
# Il s'agit d'un automate d'upgrade
# Je considère que les paquets DEBIAN nécessaire ont déjà été
# installé lors de la mise en place de ce service
#Ressource accessible sur le serveur géré
RESS_1="/tmp/tools-parser-yaml.sh"

DIRMATRIX="/opt/synapse"
DIRBACKUP="/opt/synapse/ressources/backupcode"
OPTTAR="--exclude=ressources --exclude=homeserver.log*"
SERVICEMATRIX="matrix-synapse"

# Install postgresql client si non installé
apt -y install postgresql-client
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Automate] /var/lib/mytinydc/automates/matrix-service-update/script.mtya" >&2
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] apt -y install postgresql-client" >&2

    exit $ERR
fi

DATEJOUR=`date +%Y-%m-%d-%H-%M-%S`
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Automate] /var/lib/mytinydc/automates/matrix-service-update/script.mtya" >&2
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] DATEJOUR=date +%Y-%m-%d-%H-%M-%S" >&2

    exit $ERR
fi

# Pré-check
if [ ! -d "$DIRMATRIX" ];then
    echo "[ERR] Matrix n'est pas installé sur ce système"
    exit 1
fi
if [ ! -d "$DIRBACKUP" ];then
    mkdir -p "$DIRBACKUP"
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Automate] /var/lib/mytinydc/automates/matrix-service-update/script.mtya" >&2
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] mkdir -p '$DIRBACKUP'" >&2

    exit $ERR
fi

fi
#Parse le fichier de conf yaml de matrix
ENVMATRIXYAML="/tmp/envmatrix"
if [ ! -x "$RESS_1" ];then
    echo "[ERR] Impossible de continuer $RESS_1 n'est pas exécutable"
    exit 1
fi
$RESS_1 $DIRMATRIX/homeserver.yaml > $ENVMATRIXYAML
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Automate] /var/lib/mytinydc/automates/matrix-service-update/script.mtya" >&2
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] $RESS_1 $DIRMATRIX/homeserver.yaml > $ENVMATRIXYAML" >&2

    exit $ERR
fi

# Sécurité
chmod 600 $ENVMATRIXYAML
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Automate] /var/lib/mytinydc/automates/matrix-service-update/script.mtya" >&2
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] chmod 600 $ENVMATRIXYAML" >&2

    exit $ERR
fi

source $ENVMATRIXYAML
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Automate] /var/lib/mytinydc/automates/matrix-service-update/script.mtya" >&2
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] source $ENVMATRIXYAML" >&2

    exit $ERR
fi

rm $ENVMATRIXYAML
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Automate] /var/lib/mytinydc/automates/matrix-service-update/script.mtya" >&2
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] rm $ENVMATRIXYAML" >&2

    exit $ERR
fi

# Upgrade du service avec backup base et code
# Stop matrix
systemctl stop $SERVICEMATRIX
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Automate] /var/lib/mytinydc/automates/matrix-service-update/script.mtya" >&2
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] systemctl stop $SERVICEMATRIX" >&2

    exit $ERR
fi

# Sauvegarde du code
TARFILE="$DIRBACKUP/backup-matrix-synapse-$DATEJOUR.tgz"
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Automate] /var/lib/mytinydc/automates/matrix-service-update/script.mtya" >&2
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] TARFILE='$DIRBACKUP/backup-matrix-synapse-$DATEJOUR.tgz'" >&2

    exit $ERR
fi

tar $OPTTAR -cvzf "$TARFILE" "$DIRMATRIX" 2>&1
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Automate] /var/lib/mytinydc/automates/matrix-service-update/script.mtya" >&2
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] tar $OPTTAR -cvzf '$TARFILE' '$DIRMATRIX' 2>&1" >&2

    exit $ERR
fi

# Backup Database
BACKUPDONE="0"
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Automate] /var/lib/mytinydc/automates/matrix-service-update/script.mtya" >&2
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] BACKUPDONE='0'" >&2

    exit $ERR
fi

if [ "$database_name" = "psycopg2" ];then
    echo "[INFO] Database is postgresl - try to backup"
    if [ "$database_args_user" != "" ] && [ "$database_args_password" != "" ] && [ "$database_args_database" != "" ] && [ "$database_args_host" != "" ] && [ "database_args_port" != "" ];then
        echo "[INFO] backup in progress vers $DIRBACKUP/backup-database-$DATEJOUR.sql"
        export PGPASSWORD="$database_args_password"
        pg_dump -d "$database_args_database" -h "$database_args_host" -p "$database_args_port" -U "$database_args_user"  > "$DIRBACKUP/backup-database-$DATEJOUR.sql"
        ERR=$?
        if [ "$ERR" != "0" ];then
            echo "[Automate] /var/lib/mytinydc/automates/matrix-service-update/script.mtya" >&2
            echo "[Code Erreur] $ERR" >&2
            echo "[Serveur] $SERVER" >&2
            echo "[Commande] pg_dump -d '$database_args_database' -h '$database_args_host' -p '$database_args_port' -U '$database_args_user' > '$DIRBACKUP/backup-database-$DATEJOUR.sql'" >&2

            exit $ERR
        fi
        unset PGPASSWORD
        echo "[INFO] backup done"
        if [ -f "$DIRBACKUP/backup-database-$DATEJOUR.sql" ];then
            BACKUPDONE="1"
        fi
    fi
fi
if [ "$BACKUPDONE" = "0" ];then
    echo "[ERREUR] La mise à jour ne peut être effectuée"
    echo "         la base de données n'a pas été sauvegardée"
    exit 1
fi
# update du service
cd "$DIRMATRIX"
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Automate] /var/lib/mytinydc/automates/matrix-service-update/script.mtya" >&2
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] cd '$DIRMATRIX'" >&2

    exit $ERR
fi

source $DIRMATRIX/env/bin/activate
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Automate] /var/lib/mytinydc/automates/matrix-service-update/script.mtya" >&2
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] source $DIRMATRIX/env/bin/activate" >&2

    exit $ERR
fi

pip install --upgrade pip
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Automate] /var/lib/mytinydc/automates/matrix-service-update/script.mtya" >&2
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] pip install --upgrade pip" >&2

    exit $ERR
fi

pip install --upgrade setuptools
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Automate] /var/lib/mytinydc/automates/matrix-service-update/script.mtya" >&2
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] pip install --upgrade setuptools" >&2

    exit $ERR
fi

pip install --upgrade matrix-synapse[all]
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Automate] /var/lib/mytinydc/automates/matrix-service-update/script.mtya" >&2
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] pip install --upgrade matrix-synapse[all]" >&2

    exit $ERR
fi

deactivate
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Automate] /var/lib/mytinydc/automates/matrix-service-update/script.mtya" >&2
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] deactivate" >&2

    exit $ERR
fi

# Démarrage du service
systemctl start $SERVICEMATRIX
ERR=$?
if [ "$ERR" != "0" ];then
    echo "[Automate] /var/lib/mytinydc/automates/matrix-service-update/script.mtya" >&2
    echo "[Code Erreur] $ERR" >&2
    echo "[Serveur] $SERVER" >&2
    echo "[Commande] systemctl start $SERVICEMATRIX" >&2

    exit $ERR
fi

echo "[INFO] Code sauvegardé dans $TARFILE"
echo "[INFO] Database sauvegardée dans $DIRBACKUP/backup-database-$DATEJOUR.sql"


#--- ATTENTION CE SHELL DEPEND DE RESSOURCES COMPLEMENTAIRES ---
#    Sans les ressources ce shell ne fonctionnera pas
#    Les ressources à fournir sont situées dans le répertoire :
#    /var/lib/mytinydc/automates/matrix-service-update/ressources/
#
#    Vous devrez copier ces ressources sur le système qui exécutera ce shell.
#    Puis modifier ce shell en modifiant l'emplacement de ces ressources
#    pour les variable qui suivent :
#         - RESS_1="/tmp/tools-parser-yaml.sh"

Run this shell: bash /tmp/matrix-update.sh.

POST execution

If the update of MATRIX is not conclusive at the time of the reboot, you have all the backups at your disposal to put back Matrix in its state before the update. These elements (source code and database) are located in the backup directory indicated in the shell.

Having read all the posts concerning Matrix, you have all the elements to set up a complete automation of the Matrix update (detection, backup, update). In my case I have no manual intervention, everything happens automatically, as soon as a new version is detected. Don’t be afraid to update your Matrix system, I never had to restore the whole system after an update.

Enjoy+++

Document licence : Creative Commons (CC BY-NC-ND 4.0)

THIS DOCUMENTATION IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND AND DISTRIBUTED FOR EDUCATIONAL PURPOSES ONLY. THE AUTHOR, CONTRIBUTORS TO THIS DOCUMENTATION OR ©MYTINYDC.COM SHALL IN NO EVENT BE LIABLE FOR ANY DIRECT OR INDIRECT DAMAGE THAT MAY RESULT FROM THE APPLICATION OF THE PROCEDURES IMPLEMENTED IN THIS DOCUMENTATION, OR FROM THE INCORRECT INTERPRETATION OF THIS DOCUMENT.

(**) Translated with www.DeepL.com/Translator