Recent disruptions to two undersea internet cables in the Baltic Sea have yet again...
2.13.1 Initial failover configuration
Prerequisites #
Before proceeding with failover configuration the following prerequisites must be met:
- One IRP node is configured and fully functional. We will refer to this node as $IRPMASTER.
- Second IRP node is installed with the same version of IRP as on $IRPMASTER. We will refer to this node as $IRPSLAVE.
Second IRP node MUST run the same operating system as $IRPMASTER.
When troubleshooting problems, besides checking matching IRP versions ensure the same versions of irp MySQL databases are installed on both failover nodes.- IRP services, MySQL and HTTP daemons are stopped on $IRPSLAVE node.
- Network operator can SSH to both $IRPMASTER and $IRPSLAVE and subsequent commands are assumed to be run from a $IRPMASTER console.
$IRPMASTER and $IRPSLAVE must have different hostnames.
Configure communication channel from $IRPMASTER to $IRPSLAVE #
This channel is used during failover configuration and subsequently $IRPMASTER uses it to synchronize IRP configuration changes when these occur. It uses key-based authentication without a passphrase to allow automated logins on $IRPSLAVE by $IRPMASTER processes.
Adjust firewalls if any so that $IRPMASTER node can access $IRPSLAVE via SSH.Generate SSH keys pair WITHOUT passphrase:
Listing 2.56: Generate keys on $IRPMASTER
root@IRPMASTER ~ # ssh-keygen -t rsa -b 2048 -f ~/.ssh/id_rsa -C "failover@noction"
Default keys files are used. In case your system needs additional keys for other purposes we advise that those keys are assigned a different name. If this is not possible then keys file name designated for failover use should be also specified in IRP configuration parameter global.failover_identity_file.Copy public SSH key from master to slave instance:
Listing 2.57: Install public key on $IRPSLAVE
root@IRPMASTER ~ # cat ~/.ssh/id_rsa.pub | while read key; do ssh $IRPSLAVE "echo $key >> ~/.ssh/authorized_keys"; done
Check if $IRPMASTER can login to $IRPSLAVE without using a password:
Listing 2.58: Check SSH certificate-based authentication works
root@IRPMASTER ~ # ssh $IRPSLAVE
Install certificate and keys for MySQL Multi-Master replication between $IRPMASTER and $IRPSLAVE #
MySQL Multi-Master replication uses separate communication channels that also require authentication. IRP failover uses key-based authentication for these channels too.
Adjust firewalls if any so that $IRPMASTER and $IRPSLAVE can communicate with each other bidirectionally.Create Certificate Authority and server/client certificates and keys. Commands must be run on both $IRPMASTER and $IRPSLAVE nodes:
Listing 2.59: Generate CA and certificates
# cd && rm -rvf irp-certs && mkdir -p irp-certs && cd irp-certs # openssl genrsa 2048 > $(hostname -s)-ca-key.pem # openssl req -new -x509 -nodes -days 3600 -subj "/C=US/ST=CA/L=Palo Alto/O=Noction/OU=Intelligent Routing Platform/CN=$(/bin/hostname) CA/emailAddress=support@noction.com" -key $(hostname -s)-ca-key.pem -out $(hostname -s)-ca-cert.pem # openssl req -newkey rsa:2048 -days 3600 -subj "/C=US/ST=CA/L=Palo Alto/O=Noction/OU=Intelligent Routing Platform/CN=$(/bin/hostname) server/emailAddress=support@noction.com" -nodes -keyout $(hostname -s)-server-key.pem -out $(hostname -s)-server-req.pem # openssl rsa -in $(hostname -s)-server-key.pem -out $(hostname -s)-server-key.pem # openssl x509 -req -in $(hostname -s)-server-req.pem -days 3600 -CA $(hostname -s)-ca-cert.pem -CAkey $(hostname -s)-ca-key.pem -set_serial 01 -out $(hostname -s)-server-cert.pem # openssl req -newkey rsa:2048 -days 3600 -subj "/C=US/ST=CA/L=Palo Alto/O=Noction/OU=Intelligent Routing Platform/CN=$(/bin/hostname) client/emailAddress=support@noction.com" -nodes -keyout $(hostname -s)-client-key.pem -out $(hostname -s)-client-req.pem # openssl rsa -in $(hostname -s)-client-key.pem -out $(hostname -s)-client-key.pem # openssl x509 -req -in $(hostname -s)-client-req.pem -days 3600 -CA $(hostname -s)-ca-cert.pem -CAkey $(hostname -s)-ca-key.pem -set_serial 01 -out $(hostname -s)-client-cert.pem
Verify certificates. Commands must be run on both $IRPMASTER and $IRPSLAVE nodes:
Listing 2.60: Verify certificates
# openssl verify -CAfile $(hostname -s)-ca-cert.pem $(hostname -s)-server-cert.pem $(hostname -s)-client-cert.pem server-cert.pem: OK client-cert.pem: OK
Install certificates in designated directories. Commands must be run on both $IRPMASTER and $IRPSLAVE nodes:Cross copy client key and certificates:
Listing 2.61: Install certificates in designated directories
# mkdir -p /etc/pki/tls/certs/mysql/server/ /etc/pki/tls/certs/mysql/client/ /etc/pki/tls/private/mysql/server/ /etc/pki/tls/private/mysql/client/ # cp $(hostname -s)-ca-cert.pem $(hostname -s)-server-cert.pem /etc/pki/tls/certs/mysql/server/ # cp $(hostname -s)-ca-key.pem $(hostname -s)-server-key.pem /etc/pki/tls/private/mysql/server/ # cp $(hostname -s)-client-cert.pem /etc/pki/tls/certs/mysql/client/ # cp $(hostname -s)-client-key.pem /etc/pki/tls/private/mysql/client/ # cd && rm -rvf irp-certs
Listing 2.62: Cross copy client key and certificates
root@IRPMASTER ~# scp "/etc/pki/tls/certs/mysql/server/$IRPMASTER-ca-cert.pem" "$IRPSLAVE:/etc/pki/tls/certs/mysql/client/" root@IRPMASTER ~# scp "/etc/pki/tls/certs/mysql/client/$IRPMASTER-client-cert.pem" "$IRPSLAVE:/etc/pki/tls/certs/mysql/client/" root@IRPMASTER ~# scp "/etc/pki/tls/private/mysql/client/$IRPMASTER-client-key.pem" "$IRPSLAVE:/etc/pki/tls/private/mysql/client/" root@IRPMASTER ~# scp "$IRPSLAVE:/etc/pki/tls/certs/mysql/server/$IRPSLAVE-ca-cert.pem" "/etc/pki/tls/certs/mysql/client/" root@IRPMASTER ~# scp "$IRPSLAVE:/etc/pki/tls/certs/mysql/client/$IRPSLAVE-client-cert.pem" "/etc/pki/tls/certs/mysql/client/" root@IRPMASTER ~# scp "$IRPSLAVE:/etc/pki/tls/private/mysql/client/$IRPSLAVE-client-key.pem" "/etc/pki/tls/private/mysql/client/"
Adjust file permissions. Commands must be run on both $IRPMASTER and $IRPSLAVE nodes:
Listing 2.63: Set file permissions for keys and certificates
# chown -R mysql:mysql /etc/pki/tls/certs/mysql/ /etc/pki/tls/private/mysql/ # chmod 0600 /etc/pki/tls/private/mysql/server/* /etc/pki/tls/private/mysql/client/*
Configure MySQL replication on $IRPSLAVE #
Each node of MySQL Multi-Master replication is assigned its own identifier and references previously configured certificates. There are also configuration parameters such as binary/relay log file names and auto-increment and auto-increment-offset values.
IRP includes a template config file /usr/share/doc/irp/irp.my_repl_slave.cnf.template. The template designates $IRPSLAVE as second server of the Multi-Master replication and includes references to $(hostname -s) that need to be replaced with the actual hostname of $IRPSLAVE before installing. Apply the changes and review the configuration file. Alternatively a command like in the below example can be used to create $IRPSLAVE config file from template. Ensure using actual short host name instead of the provided variable:
Listing 2.64: Example $IRPSLAVE configuration from template
# Ubuntu
root@IRPSLAVE ~# sed 's|$(hostname -s)|$IRPSLAVE|' < /usr/share/doc/irp/irp.my_repl_slave.cnf.template > /etc/mysql/conf.d/irp.my_repl_slave.cnf
# RedHat
root@IRPSLAVE ~# sed 's|$(hostname -s)|$IRPSLAVE|' < /usr/share/doc/irp/irp.my_repl_slave.cnf.template > /etc/my.cnf.d/irp.my_repl_slave.cnfThe config file created above must be included into $IRPSLAVE node’s MySQL config my.cnf. It is recommended to store these files inside OS-specific directories for MariaDB configuration files (Ubuntu: /etc/mysql/conf.d, RedHat: /etc/my.cnf.d/, otherwise it should be included via !include /path/to/file from main MariaDB config.
Check Multi-Master configuration on $IRPSLAVE:
Listing 2.65: Check MySQL on $IRPSLAVE works correctly
root@IRPSLAVE ~# systemctl start mariadb root@IRPSLAVE ~# tail -f /var/log/mysqld.log root@IRPSLAVE ~# mysql irp -e "show master status \G" root@IRPSLAVE ~# systemctl stop mariadb
Configure MySQL replication on $IRPMASTER #
Configuring $IRPMASTER is done with running services.
Configuring MySQL Multi-Master replication on $IRPMASTER should only be done after confirming it works on $IRPSLAVE.Similarly to $IRPSLAVE above IRP comes with a template configuration file for $IRPMASTER – /usr/share/doc/irp/irp.my_repl_master.cnf.template. The template designates $IRPMASTER as first server of the Multi-Master replication and includes references to $(hostname -s) that need to be replaced with the actual hostname of $IRPMASTER before installing. Apply the changes and review the resulting configuration file.
Alternatively a command like the example below can be used to create $IRPMASTER config file from template. Ensure using actual short host name instead of the provided variable:
Listing 2.66: Set $IRPMASTER as a first node for Multi-Master replication
#Ubuntu root@IRPMASTER ~# sed 's|$(hostname -s)|$IRPMASTER|' < /usr/share/doc/irp/irp.my_repl_master.cnf.template > /etc/mysql/conf.d/irp.my_repl_master.cnf #Redhat root@IRPMASTER ~# sed ’s|$(hostname -s)|$IRPMASTER|’ < /usr/share/doc/irp/irp.my_repl_master.cnf.template > /etc/my.cnf.d/irp.my_repl_master.cnf
Again, the config file created above must be included into $IRPMASTER node’s MySQL config my.cnf. It is recommended to store these files inside OS-specific directories for MariaDB configuration files (Ubuntu: /etc/mysql/conf.d, RedHat: /etc/my.cnf.d/, otherwise it should be included via !include /path/to/file from main MariaDB config.
Check MySQL Multi-Master configuration on $IRPMASTER:
Listing 2.67: Check MySQL on $IRPMASTER works correctly
root@IRPMASTER ~# systemctl restart mariadb root@IRPMASTER ~# tail -f /var/log/mysqld.log root@IRPMASTER ~# mysql irp -e "show master status \G"
If Multi-Master configuration on $IRPMASTER fails or causes unrecoverable errors, a first troubleshooting step is to comment back the included line in /etc/my.cnf on master and slave and restart mysqld service to revert to previous good configuration.
Create replication grants on $IRPMASTER #
Replication requires a MySQL user with corresponding access rights to replicated databases. The user must be assigned a password and is designated as connecting correspondingly from $IRPMASTER or $IRPSLAVE. Unfortunately hostnames cannot be used for this and the exact IP addresses of the corresponding nodes must be specified.
The user is created only once in our procedure since after being created the database on $IRPMASTER is manually transferred to $IRPSLAVE and the user will be copied as part of this process.Grant replication access to replication user:
Listing 2.68: Replication user and grants
mysql> CREATE USER 'irprepl'@'<mysql_slave1_ip_address>' IDENTIFIED BY '<replication_user_password>'; mysql> GRANT REPLICATION SLAVE ON *.* TO 'irprepl'@'<mysql_masterslave1_ip_address>' REQUIRE CIPHER 'DHE-RSA-AES256-SHA'; mysql> CREATE USER 'irprepl'@'<mysql_master2_ip_address>' IDENTIFIED BY '<replication_user_password>'; mysql> GRANT REPLICATION SLAVE ON *.* TO 'irprepl'@'<mysql_slave2_ip_address>' REQUIRE CIPHER 'DHE-RSA-AES256-SHA';
Copy IRP database configuration and database to $IRPSLAVE #
IRP failover copies the irp database and corresponding configuration file from $IRPMASTER to $IRPSLAVE before activating second server. This avoids the need to manually verify and synchronize identifiers.
Copy root’s .my.cnf config file if exists:
Listing 2.69: Copy database root user configuration file
root@IRPMASTER ~# scp /root/.my.cnf $IRPSLAVE:/root/
Copy config files:
Listing 2.70: Copy database configuration files
root@IRPMASTER ~# scp /etc/noction/db.global.conf $IRPSLAVE:/etc/noction/ root@IRPMASTER ~# scp /etc/noction/clickhouse/users.xml $IRPSLAVE:/etc/noction/clickhouse/
Preliminary copy database files:
Listing 2.71: Copy database data files
root@IRPMASTER ~# rsync -av --progress --delete --delete-after --exclude="master.info" --exclude="relay-log.info" --exclude="*-bin.*" --exclude="*-relay.*" /var/lib/mysql/ $IRPSLAVE:/var/lib/mysql/
Preliminary copy ensures that large files that take a long time to copy are synced to $IRPSLAVE without stopping MySQL daemon on $IRPMASTER and only a reduced number of differences will need to by synced while MySQL is stopped. This operation can be rerun one more time to reduce the duration of the downtime on $IRPMASTER even more.
Finish copy of database files (OS with Systemd):
Listing 2.72: Copy differences of database files (OS with Systemd)
root@IRPMASTER ~# systemctl stop mariadb clickhouse-server # RedHat Enterprise Linux root@IRPMASTER ~# systemctl stop mysql clickhouse-server # Ubuntu root@IRPMASTER ~# systemctl start irp-stop-nobgpd.target systemctl start irp-shutdown-except-bgpd.target systemctl start irp-shutdown.target root@IRPMASTER ~# cd /var/lib/mysql && rm -vf ./master.info ./relay-log.info ./*-bin.* ./*-relay.* root@IRPMASTER ~# rsync -av --progress --delete --delete-after /var/lib/mysql/ $IRPSLAVE:/var/lib/mysql/
The procedure above tries to reduce the downtime of MySQL daemon on $IRPMASTER. During this time Bgpd preserves IRP Improvements. Make sure this action takes less than bgpd.db.timeout.withdraw.Start MySQL on $IRPSLAVE and check if there are no errors at /var/log/mysqld.log on RedHat Enterprise Linux or /var/log/mysql/error.log on Ubuntu.
First $IRPSLAVE must be checked.Start MySQL on IRP master and check if there are no errors in MySQL logs as above.
Start replication (Slaves) on both $IRPMASTER and $IRPSLAVE #
MySQL Multi-Master replication uses a replication scheme where a replicating master acts as both replication master and replication slave. The steps above configured IRP nodes to be capable of acting as replication masters. Here we ensure that both nodes are capable of acting as replication slaves.
IRP provides a template command that needs to be adjusted for each $IRPMASTER and $IRPSLAVE and will instruct MySQL daemon to take the replication slave role. The template is /usr/share/doc/irp/changemasterto.template.
The template generates a different command for each $IRPMASTER and $IRPSLAVE nodes and requires multiple values to be reused from configuration settings described above. The command that is run on one node points to the other node as its master.
Make $IRPMASTER a replication slave for $IRPSLAVE:
Listing 2.73: Set $IRPMASTER as replication slave
$IRPMASTER-mysql> CHANGE MASTER TO MASTER_HOST='$IRPSLAVE-ip-address', MASTER_USER='irprepl', MASTER_PASSWORD='$IRPSLAVE-password>', MASTER_PORT=3306, MASTER_LOG_FILE= '$IRPSLAVE--bin.000001', MASTER_LOG_POS= <$IRPSLAVE-bin-log-position>, MASTER_CONNECT_RETRY=10, MASTER_SSL=1, MASTER_SSL_CAPATH='/etc/pki/tls/certs/mysql/client/', MASTER_SSL_CA='/etc/pki/tls/certs/mysql/client/$IRPSLAVE-ca-cert.pem', MASTER_SSL_CERT='/etc/pki/tls/certs/mysql/client/$IRPSLAVE-client-cert.pem', MASTER_SSL_KEY='/etc/pki/tls/private/mysql/client/$IRPSLAVE-client-key.pem', MASTER_SSL_CIPHER='DHE-RSA-AES256-SHA';
You must manually check what values to assign to
$IRPSLAVE-bin.000001 and <$IRPSLAVE-bin-log-position>
by running the following MySQL command on $IRPSLAVE
mysql> show master status
For the initial configuration the values for $IRPSLAVE-bin.000001 and <$IRPSLAVE-bin-log-position> must be as follows:
Binlog file: $IRPSLAVE–bin.000001
Binlog position: 106
Run the commands below to run the replication and check the slave status:
Listing 2.74: Starting replication slave on $IRPMASTER
mysql> START SLAVE \G mysql> show slave status \G
Check the Slave_IO_State, Last_IO_Errno, Last_IO_Error, Last_SQL_Errno, Last_SQL_Error values for errors. Make sure there are no errors.
Make $IRPSLAVE a replication slave for $IRPMASTER:
Listing 2.75: Set $IRPSLAVE as replication slave
$IRPSLAVE-mysql> CHANGE MASTER TO MASTER_HOST='$IRPMASTER-ip-address', MASTER_USER='irprepl', MASTER_PASSWORD='$IRPMASTER-password>', MASTER_PORT=3306, MASTER_LOG_FILE= '$IRPMASTER-bin.000001', MASTER_LOG_POS= <$IRPMASTER-bin-log-position>, MASTER_CONNECT_RETRY=10, MASTER_SSL=1, MASTER_SSL_CAPATH='/etc/pki/tls/certs/mysql/client/', MASTER_SSL_CA='/etc/pki/tls/certs/mysql/client/$IRPMASTER-ca-cert.pem', MASTER_SSL_CERT='/etc/pki/tls/certs/mysql/client/$IRPMASTER-client-cert.pem', MASTER_SSL_KEY='/etc/pki/tls/private/mysql/client/$IRPMASTER-client-key.pem', MASTER_SSL_CIPHER='DHE-RSA-AES256-SHA';
You must manually check what values to assign to
$IRPMASTER-bin.000001and <$IRPMASTER-bin-log-position>
by running the following MySQL command on $IRPMASTER
mysql> show master status
For the initial configuration the values for $IRPMASTER-bin.000001 and <$IRPMASTER-bin-log-position> must be as follows:
Binlog file: $IRPMASTER–bin.000001
Binlog position: 106
Run the commands below to run the replication and check the slave status:
Listing 2.76: Starting replication slave
mysql> START SLAVE \G mysql> show slave status \G
Run IRP services on $IRPMASTER and $IRPSLAVE:
Listing 2.77: Starting IRP services and Frontend (OS with Systemd)
# systemctl start irp.target
Start services on $IRPMASTER first if the actions above took very long in order to shorten MySQL downtime.
Configure Failover using Wizard on $IRPMASTER #
These steps should only be performed once SSH communication channel and MySQL Multi-Master irp database replication have been setup and are verified to work. Refer subsections of Failover Configuration above.
Run failover wizard: #
Login into IRP’s Frontend on $IRPMASTER node and run Configuration -> Setup wizard -> Setup Failover.
A valid failover license should be acquired before failover configuration settings become available.
Configure IRP failover: #
Follow Setup Failover wizard steps and provide the required details. Refer Setup Failover wizard for details. Once your configuration changes are submitted IRP will validate configuration and if it is valid the configuration will be synced to $IRPSLAVE.
Only after this synchronization step takes place will $IRPSLAVE node know what is its role in this setup.
Apply configuration changes to edge routers: #
Ensure designated probing IPs, BGP session(s) are also setup on edge routers. Refer to corresponding sections of this document for details.
Enable failover: #
Use IRP’s Frontend Configuration -> Global -> Failover to configure IRP failover on both nodes. Monitor both IRP nodes and ensure everything is within expectations.
It is recommended that after finishing the preparatory steps above both IRP master and slave nodes run with disabled failover for a short period of time (less than 1 hour) in order to verify that all IRP components and MySQL Multi-Master replication work as expected. Keep this time interval short to avoid a split-brain situation when the two nodes make contradictory decisions.
Synchronize RRD statistics to $IRPSLAVE #
RRD statistics is collected by both IRP failover nodes and needs not be synchronized during normal operation. Still, when IRP failover nodes are setup at a large time interval between them it is recommended that RRD files are synchronized too. This will ensure that past statistics is available on both IRP nodes. During short downtimes of each of the nodes synchronization is less usefull but can be performed in order to cover the short downtime gaps in RRD based graphs.
Sample command to synchronize IRP’s RRD statistics:
Listing 2.78: Synchronize RRD
root@IRPMASTER ~ # rsync -av /var/spool/irp/ $IRPSLAVE:/var/spool/irp