STL to ORD Migration Best Practices

Customers with slices in our St. Louis datacenters (STL-A and STL-B) will be migrated to our Chicago datacenter (ORD) as part of the transition from Slicehost to Rackspace Cloud. Customers will be notified in advance of the slice migration, giving them time to optimize their slices for the move.

The migration

At its most basic the migration is a copy operation in two phases.

Phase 1

The files on the original slice are copied to the new server. During this phase the slice is running and available.

Phase 2

The slice is brought down. The system then runs a sync operation so any files that changed during the first pass will be re-copied to the new server instance.

After these two phases complete the new server instance will boot and become available.

Mitigation

Review the contents of your slice to look for ways to reduce the length of each phase of the migration.

You can shorten phase 1 by eliminating unneeded files from the original slice, like archived logs and application cache files. You can also reduce the size of files to be copied by weeding old entries out of database tables.

You can reduce the length of phase 2, and thus your downtime, by keeping the number and size of files that will change during phase 1 to a minimum. With less to copy the new server will be ready that much sooner.

Advance preparation

Some mitigation tasks can be performed well in advance of the migration.

Establish a log rotation policy

Large log files can potentially increase the time for the initial copy and increase downtime during the final sync. Application and virtual host logs aren't generally rotated unless you specifically set up rotation for them.

You can use a utility like logrotate to make sure your logs stay at a reasonable size.

Prune and archive old data

Look for archived log files, application cache files, and old tables or entries in databases. The more you can remove the less there will be to copy to the new server.

Ruby on Rails applications in particular can create a large number of session files and never delete them. Before the migration make sure you look into where your apps might store those session and cache files.

For more advice on locating and pruning old session and cache files, see this article on preparing for slice resizes (the process behind the scenes is very similar to the migration process).

Check for large files

Removing other large files from the slice beforehand can also cut down on the overall copy time.

You can look for the largest files on your system using the find utility:

sudo find / -mount -type f -ls|sort -rnk7 |head -30|awk '{printf "%10d MB\t%s\n",($7/1024)/1024,$NF}'

The output will show the largest files on your slice so you can determine if any can be deleted or archived elsewhere.

Right before the migration

You can reduce downtime during the final sync operation by taking steps to reduce the number of files that will change during the first copy.

Use public IP addresses

Telling services that talk to other slices to use the public IP addresses or domain names of the target servers can help minimize the effects of a multiple-slice migration.

The migration will change the private IP addresses of your slices and switch them to a new private (backend) network. While the migration progresses, using the public network interfaces for slice communication will let those slices talk to each other even as they're brought up in the new datacenter.

We know customers often use private IPs for communications between slices to minimize bandwidth charges. With that in mind we will not charge for bandwidth overages for slices scheduled for migration.

Once the migration completes you can switch services to the new private IP addresses.

Force log rotations

Applications that are still running during the migration may generate new log entries. When the slice is brought down for the final data sync those changed log files will have to be re-synced. That can prolong your downtime.

If you force a log rotation beforehand you can ensure that any changed log files will be relatively small. Most log rotation utilities provide a means of forcing rotations manually.

For example, if you're using logrotate to manage your application logs you can force a rotation with the command:

sudo logrotate -f /etc/logrotate.conf  

Lock databases

Databases that change during the first copy will have to be recopied in phase 2. Since database files tend to be on the large side this can noticeably extend the length of the final sync.

It's a lot safer to bring the database down entirely for the migration. If that isn't practical, however, the second-best approach is to make your tables read-only so they won't be written to during the copy.

To lock your tables in mysql run the command:

mysql -u root -p --execute="FLUSH TABLES WITH READ LOCK"

Flush application caches

Clearing out old, unneeded session and cache files can prevent them from dragging a migration to a crawl.

For each file on a server the sync process has to perform an operation to look at each file, to copy it, and to confirm its copy at the end. That means syncing a lot of very small cache or data files can take more time than syncing one file of the same size as all of them put together.

After the migration

When the migration completes and your new server boots you should test your web sites and applications. Make sure applications are responsive and that they can write information to their databases.

If you have any services that need to communicate to other servers, at this point you can switch them to use the private (backend) IP addresses for those servers instead of their public addresses.

Further reading

The migration process is very similar to a slice resize operation. As a result, the more detailed advice in this article about optimizing for a resize applies to the STL to ORD migration as well.

And of course, if you have any questions about the migration, don't hesitate to contact our support team or post in our forum.

--Jered

Want to comment?


(not made public)

(optional)

(use plain text or Markdown syntax)