Bitbucket migration post-mortem

By on August 31, 2010

Over night we ran into a hiccup with SSH authentications. For this we apologize.

When anyone uploaded a new SSH key after the roll out of our new setup, the new key would overwrite the existing SSH key store. If you attempted to authenticate to your repositories using SSH during this time, your authentication would have failed. A stray process in charge of handling the store kept us in check and it took us a while to track it down.

A couple of hours ago we rolled out a fix for the problem and all of your SSH authentications should be working as expected.

Onto better news…

After switching the main application servers over during our 1 hour maintenance period, we began migrating everyone’s repositories from our EC2 storage system to the Contegix data center asynchronously. When the last repository was switched over, we turned off the EC2 load balancer and began redirecting all traffic directly to Contegix.

The performance numbers are in and hopefully you all are seeing the same level of increase that we are!

Old

New

Many thanks to everyone for their patience during this migration period.

We’ll be posting a follow-up post with more technical juicy details on the migration and how we did it.

21 Comments

  • openschnitzel
    Posted August 30, 2010 at 10:13 pm | Permalink

    The SSH issue was the last straw on a busy day for me, so yesterday I felt a bit annoyed. But in retrospect the migration went smoothly (I've seen more problems with other hosting services). I could feel some performance increase when I did my first push this morning. Thanks! Overall, bitbucket.org is a service which I enjoy paying money for, because it is such a valuable tool to have.

    I also like the fact that contegix says a bit more about security and privacy than the old servers did…

  • ddevine
    Posted August 30, 2010 at 11:17 pm | Permalink

    Hey – one small problem but hardly the epic fail I was worried about!

  • openschnitzel
    Posted August 31, 2010 at 5:13 am | Permalink

    The SSH issue was the last straw on a busy day for me, so yesterday I felt a bit annoyed. But in retrospect the migration went smoothly (I’ve seen more problems with other hosting services). I could feel some performance increase when I did my first push this morning. Thanks! Overall, bitbucket.org is a service which I enjoy paying money for, because it is such a valuable tool to have.

    I also like the fact that contegix says a bit more about security and privacy than the old servers did…

  • openschnitzel
    Posted August 31, 2010 at 5:13 am | Permalink

    The SSH issue was the last straw on a busy day for me, so yesterday I felt a bit annoyed. But in retrospect the migration went smoothly (I’ve seen more problems with other hosting services). I could feel some performance increase when I did my first push this morning. Thanks! Overall, bitbucket.org is a service which I enjoy paying money for, because it is such a valuable tool to have.

    I also like the fact that contegix says a bit more about security and privacy than the old servers did…

  • Anonymous
    Posted August 31, 2010 at 6:17 am | Permalink

    Hey – one small problem but hardly the epic fail I was worried about!

  • Anonymous
    Posted August 31, 2010 at 6:17 am | Permalink

    Hey – one small problem but hardly the epic fail I was worried about!

    • Guest
      Posted August 31, 2010 at 8:14 am | Permalink

      They did have an epic fail… For some moments, the home page of the website was replaced with turbogears: http://omploader.org/vNWUzNw
      Also the performance of the site was near unusable for many hours (each HTTP request would take half a minute)
      So it was hardly the great success that they try to show it as.

  • T27duck
    Posted August 31, 2010 at 5:28 am | Permalink

    Still unable to either see my repo or do a hg pull.
    It's been well past the “expected downtime” and I am still unable to get any work done.
    I will be recommending to my IT department that we find more reliable alternatives to BitBucket.

  • T27duck
    Posted August 31, 2010 at 12:28 pm | Permalink

    Still unable to either see my repo or do a hg pull.
    It’s been well past the “expected downtime” and I am still unable to get any work done.
    I will be recommending to my IT department that we find more reliable alternatives to BitBucket.

  • T27duck
    Posted August 31, 2010 at 12:28 pm | Permalink

    Still unable to either see my repo or do a hg pull.
    It’s been well past the “expected downtime” and I am still unable to get any work done.
    I will be recommending to my IT department that we find more reliable alternatives to BitBucket.

  • Paul
    Posted September 1, 2010 at 5:18 am | Permalink

    Thanks Jesper.

  • Posted September 1, 2010 at 6:24 am | Permalink

    Everything works. Thank you!

  • Paul
    Posted September 1, 2010 at 12:18 pm | Permalink

    Thanks Jesper.

  • Paul
    Posted September 1, 2010 at 12:18 pm | Permalink

    Thanks Jesper.

  • Posted September 1, 2010 at 1:24 pm | Permalink

    Everything works. Thank you!

  • Posted September 1, 2010 at 1:24 pm | Permalink

    Everything works. Thank you!

  • Posted September 7, 2010 at 5:49 am | Permalink

    Everything works. Thank ..!

  • Posted September 7, 2010 at 12:49 pm | Permalink

    Everything works. Thank ..!

  • Posted September 7, 2010 at 12:49 pm | Permalink

    Everything works. Thank ..!

  • Shrawan Patel
    Posted June 8, 2011 at 11:14 am | Permalink

    This post excellently highlights what the author is trying to communicate. Nonetheless, the article has been framed excellently well and all credits to the author. For more information on how to load balance your web servers, please visit ..nhttp://serverloadbalancing.biz/wordpressbiz/, nhttp://serverloadbalancing.info/wordpressinfo/

  • u5e74 u51af
    Posted August 16, 2011 at 1:08 am | Permalink

    A wide range of Australian made UGG Boots and sheepskin leather products. Supra Shoes on hot sale now,we carry a large inventory from Supra Footwear, Including Supra Shoes,Supra Skytop And Supra Avenger,Supra Bandit,Supra Bullet.Huge Range, all with Fast, Free Delivery.nBryant, the gap in the purchase price of shoes in kobeshoes.cc! Zoom discounts and cheap to buy the lowest price every day Kobe Shoes.Huge Puma Trainers Range at LQshoes Massive Sale Now On. Free Delivery to UK USA Canada… Buy Cheap Puma Shoes at Puma Shoes Online Store Now!