Migrating Servers to Ubuntu


On Thursday…

Brian will make backup of /etc on gg , es1, es2 , db1 , db2 and /work on gg.

Shufei will shutdown site and start DB backup: ggkbase_production , ggkbase_help_wp , iwgm_wp

On Friday…

Brian will update the OS for gg , es1, es2 , db1 , db2

Get db1 and db2 up and running with mirroring and start the refill the database

Move all the files back in place for gg and install necessary apps: nginx, php, redis, rbenv, pyenv, system libraries

Once the database are in place, then we can start the ES index for gg-prod



To make system management easier and more mainstream, all the servers support ggKbase web application will be moved to Ubuntu. Here are the server migration steps.

Schedule and Notifications

We need to first schedule the shutdown for the entire ggKbase website. The estimated time of migration would be 36 hours:

  • Backup and shutdown (6 hours)
  • Database Servers Setup (10 hours)

We need to notify the lab a week in advance of, and the subsequent days until, the day of migration.

Backup and Shutdown

Shutdown ggKbase (1 min)

RAILS_ENV=production bin/rails puma:stop
RAILS_ENV=production bin/rails sidekiq:stop
./redis_init.sh stop

Backup Database (3 hours)

mysqldump -h db1 ggkbase_production > ggkbase_production_20180419.sql
mysqldump -h db1 ggkbase_help_wp > ggkbase_help_wp_20180419.sql
mysqldump -h x1 ggkbase_development > ggkbase_development_20180419.sql
mysqldump -h x1 ggkbase_demo > ggkbase_demo_20180419.sql

Backup Web App Files (30 minutes – done in advance)

tar -czvf ggkbase_production_app.tar.gz /work/railsapps/ggkbase_production
tar -czvf ggkbase_dev_app.tar.gz /work/railsapps/ggkbase_dev
tar -czvf ggkbase_demo.tar.gz /work/railsapps/ggkbase_demo
tar -czvf ggkbase_api.tar.gz /work/railsapps/ [WHERE IS GG_API???]
tar -czvf ggkbase_help_wp.tar.gz /work/phpapps/ggkbase_help_wp
tar -czvf banfieldlab.tar.gz /var/www/banfieldlab

Backup Configuration Files (1 minutes – done in advance)

nginx: /etc/nginx

Database Servers Setup (db1 and db2)

Reload databases (10 hours)

After db servers are properly set up, reload the data into the databases.

mysql -h db1 cdtdb_production < cdtdb_production_20180419.sql
mysql -h db1 cdtdb_help_wp < cdtdb_help_wp_20180419.sql

Web and Application Server Setup (prod, dev, demo, api, help)

Restore files from above (15 minutes)

Install and configure server applications (15 minutes)

  • nginx
  • php-fpm
  • redis
  • tmux, tmux plugin manager, .tmux.conf
  • rbenv, rbenv-vars, ruby, bundler
  • pyenv, python

Set up nginx (5 minutes)

Set up ggKbase, dev, demo, api, help (10 minutes)

Refer to this doc more details: https://ggkbase-help.berkeley.edu/codes/setup-development-server/

Elasticsearch Servers Setup (es1)

Back up configuration files


The jvm.options file does not need to be backed up. We just need to change the following parameters in the new instance:

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

In the new setup, we will have set up es2 for greater scalability. Here is the servers architecture from ElastiCon:

* 2 nodes (2 data/manager)
* 4 shards ~ 50GB (Recommended limit)
* 1 replica

Upgrade ES and rebuild index (3 hours)

RAILS_ENV=production bin/thor es:create_index
RAILS_ENV=production bin/thor es:index