Scheduled Maintenance December 5th at 0200 GMT

By on December 3, 2011

Bitbucket will be unavailable up to one hour starting December 5th at 02:00 GMT. During this maintenance window we will complete the database migration from our previous maintenance window and move our database to newer, more powerful servers.

Unfortunately, last weekend we were unable to complete the migration within the maintenance window, so we reverted to our original configuration.

The database migration we attempted last week had several steps. 1) We needed to perform a restart to enable write-ahead-log shipping to the new server. 2) We needed to perform an rsync which we expected to take no more than 20 minutes. 3) We planned to promote the new replica to be our master database. Last weekend, the rsync between the old server and the new one exceeded the 20 minutes allotted. It became was clear something was wrong and we rolled back the deployment changes.

After investigation, we determined the cause of the problem was that the new server’s primary network interface was bridged for use with KVM and libvirt. This bridging allowed us to practice the database migration on virtual machines. However, it also resulted in a degradation of network performance which caused the rsync to take longer than expected. To resolve the performance problem immediately we unbridged the NIC on the new database server.

On December 5th, we’ll attempt the migration again, following the same procedure outlined above. Total downtime should be no more than one hour. We apologize for the inconvenience of downtime, but we’ll be back with an even faster Bitbucket.

  • Anonymous

    So how did the upgrade go? Are we now on faster servers, or will there be another maintenance window needed.

    • Charles McLaughlin

      The last maintenance window went well.  There was only five minutes of downtime and we were able replicate the database to the new hardware.  We will schedule another maintenance window in the next few weeks to switch the new replica database to be our master database.  In order to keep downtime to a minimum we’re waiting to group some other maintenance along with it it, such as a new switch installation.

  • Anonymous

    So how did the upgrade go? Are we now on faster servers, or will there be another maintenance window needed.

    • Anonymous

      The last maintenance window went well.  There was only five minutes of downtime and we were able replicate the database to the new hardware.  We will schedule another maintenance window in the next few weeks to switch the new replica database to be our master database.  In order to keep downtime to a minimum we’re waiting to group some other maintenance along with it it, such as a new switch installation.

  • http://www.shineglassrenewal.com/ Remove Hard Water Stains

     your thinking is right about this matter.It helps me a lot to understand the whole thing.We should follow the information which you tell above.

  • Server Computer

    Just blowing some free time on Digg and I found your post . Not
    typically what I prefer to learn about, but it was definitely worth
    my time.

    Server
    Computer

  • http://www.smtrafficguru.com/ adri roy

    I fell very much greatfull for your kind information related to this subject.It makes me aware about this matter.your articale related to this matter is very much informative and up to date
    http://www.smtrafficguru.com