TODOs
On Thursday…
Brian will make backup of /etc on gg , es1, es2 , db1 , db2 and /work on gg.
Shufei will shutdown site and start DB backup: ggkbase_production , ggkbase_help_wp , iwgm_wp
On Friday…
Brian will update the OS for gg , es1, es2 , db1 , db2
Get db1 and db2 up and running with mirroring and start the refill the database
Move all the files back in place for gg and install necessary apps: nginx, php, redis, rbenv, pyenv, system libraries
Once the database are in place, then we can start the ES index for gg-prod
Introduction
To make system management easier and more mainstream, all the servers support ggKbase web application will be moved to Ubuntu. Here are the server migration steps.
Schedule and Notifications
We need to first schedule the shutdown for the entire ggKbase website. The estimated time of migration would be 36 hours:
- Backup and shutdown (6 hours)
- Database Servers Setup (10 hours)
We need to notify the lab a week in advance of, and the subsequent days until, the day of migration.
Backup and Shutdown
Shutdown ggKbase (1 min)
RAILS_ENV=production bin/rails puma:stop RAILS_ENV=production bin/rails sidekiq:stop ./redis_init.sh stop
Backup Database (3 hours)
mysqldump -h db1 ggkbase_production > ggkbase_production_20180419.sql mysqldump -h db1 ggkbase_help_wp > ggkbase_help_wp_20180419.sql mysqldump -h x1 ggkbase_development > ggkbase_development_20180419.sql mysqldump -h x1 ggkbase_demo > ggkbase_demo_20180419.sql
Backup Web App Files (30 minutes – done in advance)
tar -czvf ggkbase_production_app.tar.gz /work/railsapps/ggkbase_production tar -czvf ggkbase_dev_app.tar.gz /work/railsapps/ggkbase_dev tar -czvf ggkbase_demo.tar.gz /work/railsapps/ggkbase_demo tar -czvf ggkbase_api.tar.gz /work/railsapps/ [WHERE IS GG_API???] tar -czvf ggkbase_help_wp.tar.gz /work/phpapps/ggkbase_help_wp tar -czvf banfieldlab.tar.gz /var/www/banfieldlab
Backup Configuration Files (1 minutes – done in advance)
nginx: /etc/nginx mariadb: BRIAN WILL TAKE CARE OF THIS
Database Servers Setup (db1 and db2)
Reload databases (10 hours)
After db servers are properly set up, reload the data into the databases.
mysql -h db1 cdtdb_production < cdtdb_production_20180419.sql mysql -h db1 cdtdb_help_wp < cdtdb_help_wp_20180419.sql
Web and Application Server Setup (prod, dev, demo, api, help)
Restore files from above (15 minutes)
Install and configure server applications (15 minutes)
- nginx
- php-fpm
- redis
- tmux, tmux plugin manager, .tmux.conf
- rbenv, rbenv-vars, ruby, bundler
- pyenv, python
Set up nginx (5 minutes)
Set up ggKbase, dev, demo, api, help (10 minutes)
Refer to this doc more details: https://ggkbase-help.berkeley.edu/codes/setup-development-server/
Elasticsearch Servers Setup (es1)
Back up configuration files
~/elasticsearch-6.0.0/config/elasticsearch.yml
The jvm.options file does not need to be backed up. We just need to change the following parameters in the new instance:
# Xms represents the initial size of total heap space # Xmx represents the maximum size of total heap space -Xms31g -Xmx31g
In the new setup, we will have set up es2 for greater scalability. Here is the servers architecture from ElastiCon:
* 2 nodes (2 data/manager) * 4 shards ~ 50GB (Recommended limit) * 1 replica
Upgrade ES and rebuild index (3 hours)
RAILS_ENV=production bin/thor es:create_index RAILS_ENV=production bin/thor es:index