Bitbucket migration post-mortem

By on August 31, 2010

Over night we ran into a hiccup with SSH authentications. For this we apologize.

When anyone uploaded a new SSH key after the roll out of our new setup, the new key would overwrite the existing SSH key store. If you attempted to authenticate to your repositories using SSH during this time, your authentication would have failed. A stray process in charge of handling the store kept us in check and it took us a while to track it down.

A couple of hours ago we rolled out a fix for the problem and all of your SSH authentications should be working as expected.

Onto better news…

After switching the main application servers over during our 1 hour maintenance period, we began migrating everyone’s repositories from our EC2 storage system to the Contegix data center asynchronously. When the last repository was switched over, we turned off the EC2 load balancer and began redirecting all traffic directly to Contegix.

The performance numbers are in and hopefully you all are seeing the same level of increase that we are!

Old

New

Many thanks to everyone for their patience during this migration period.

We’ll be posting a follow-up post with more technical juicy details on the migration and how we did it.

  • openschnitzel

    The SSH issue was the last straw on a busy day for me, so yesterday I felt a bit annoyed. But in retrospect the migration went smoothly (I've seen more problems with other hosting services). I could feel some performance increase when I did my first push this morning. Thanks! Overall, bitbucket.org is a service which I enjoy paying money for, because it is such a valuable tool to have.

    I also like the fact that contegix says a bit more about security and privacy than the old servers did…

  • ddevine

    Hey – one small problem but hardly the epic fail I was worried about!

  • openschnitzel

    The SSH issue was the last straw on a busy day for me, so yesterday I felt a bit annoyed. But in retrospect the migration went smoothly (I’ve seen more problems with other hosting services). I could feel some performance increase when I did my first push this morning. Thanks! Overall, bitbucket.org is a service which I enjoy paying money for, because it is such a valuable tool to have.

    I also like the fact that contegix says a bit more about security and privacy than the old servers did…

  • openschnitzel

    The SSH issue was the last straw on a busy day for me, so yesterday I felt a bit annoyed. But in retrospect the migration went smoothly (I’ve seen more problems with other hosting services). I could feel some performance increase when I did my first push this morning. Thanks! Overall, bitbucket.org is a service which I enjoy paying money for, because it is such a valuable tool to have.

    I also like the fact that contegix says a bit more about security and privacy than the old servers did…

  • Anonymous

    Hey – one small problem but hardly the epic fail I was worried about!

    • Guest

      They did have an epic fail… For some moments, the home page of the website was replaced with turbogears: http://omploader.org/vNWUzNw
      Also the performance of the site was near unusable for many hours (each HTTP request would take half a minute)
      So it was hardly the great success that they try to show it as.

  • Anonymous

    Hey – one small problem but hardly the epic fail I was worried about!

  • T27duck

    Still unable to either see my repo or do a hg pull.
    It's been well past the “expected downtime” and I am still unable to get any work done.
    I will be recommending to my IT department that we find more reliable alternatives to BitBucket.

  • T27duck

    Still unable to either see my repo or do a hg pull.
    It’s been well past the “expected downtime” and I am still unable to get any work done.
    I will be recommending to my IT department that we find more reliable alternatives to BitBucket.

  • T27duck

    Still unable to either see my repo or do a hg pull.
    It’s been well past the “expected downtime” and I am still unable to get any work done.
    I will be recommending to my IT department that we find more reliable alternatives to BitBucket.

  • Paul

    Thanks Jesper.

  • http://twitter.com/vedranmiletic Vedran Miletić

    Everything works. Thank you!

  • Paul

    Thanks Jesper.

  • Paul

    Thanks Jesper.

  • http://twitter.com/vedranmiletic Vedran Miletić

    Everything works. Thank you!

  • http://twitter.com/vedranmiletic Vedran Miletić

    Everything works. Thank you!

  • http://www.bestpriceforsales.com/ iba01

    Everything works. Thank ..!

  • http://www.bestpriceforsales.com/ iba01

    Everything works. Thank ..!

  • http://www.bestpriceforsales.com/ iba01

    Everything works. Thank ..!

  • Shrawan Patel

    This post excellently highlights what the author is trying to communicate. Nonetheless, the article has been framed excellently well and all credits to the author. For more information on how to load balance your web servers, please visit ..nhttp://serverloadbalancing.biz/wordpressbiz/, nhttp://serverloadbalancing.info/wordpressinfo/

  • u5e74 u51af

    A wide range of Australian made UGG Boots and sheepskin leather products. Supra Shoes on hot sale now,we carry a large inventory from Supra Footwear, Including Supra Shoes,Supra Skytop And Supra Avenger,Supra Bandit,Supra Bullet.Huge Range, all with Fast, Free Delivery.nBryant, the gap in the purchase price of shoes in kobeshoes.cc! Zoom discounts and cheap to buy the lowest price every day Kobe Shoes.Huge Puma Trainers Range at LQshoes Massive Sale Now On. Free Delivery to UK USA Canada… Buy Cheap Puma Shoes at Puma Shoes Online Store Now!

  • Guest

    They did have an epic fail… For some moments, the home page of the website was replaced with turbogears: http://omploader.org/vNWUzNw
    Also the performance of the site was near unusable for many hours (each HTTP request would take half a minute)
    So it was hardly the great success that they try to show it as.

  • Guest

    They did have an epic fail… For some moments, the home page of the website was replaced with turbogears: http://omploader.org/vNWUzNw
    Also the performance of the site was near unusable for many hours (each HTTP request would take half a minute)
    So it was hardly the great success that they try to show it as.