Bitbucket will be unavailable up to one hour starting December 5th at 02:00 GMT. During this maintenance window we will complete the database migration from our previous maintenance window and move our database to newer, more powerful servers.
Unfortunately, last weekend we were unable to complete the migration within the maintenance window, so we reverted to our original configuration.
The database migration we attempted last week had several steps. 1) We needed to perform a restart to enable write-ahead-log shipping to the new server. 2) We needed to perform an rsync which we expected to take no more than 20 minutes. 3) We planned to promote the new replica to be our master database. Last weekend, the rsync between the old server and the new one exceeded the 20 minutes allotted. It became was clear something was wrong and we rolled back the deployment changes.
After investigation, we determined the cause of the problem was that the new server’s primary network interface was bridged for use with KVM and libvirt. This bridging allowed us to practice the database migration on virtual machines. However, it also resulted in a degradation of network performance which caused the rsync to take longer than expected. To resolve the performance problem immediately we unbridged the NIC on the new database server.
On December 5th, we’ll attempt the migration again, following the same procedure outlined above. Total downtime should be no more than one hour. We apologize for the inconvenience of downtime, but we’ll be back with an even faster Bitbucket.
