Migrating to new harddrives

By on February 9, 2010

In case you haven’t noticed, Bitbucket’s been suffering from slowdowns. This is mainly due to poor I/O performance we get from our Amazon EBS mounts.

After much planning and benchmarking, we’ve decided to move our data over to an 8-disk RAID 0 setup instead. The past few days, we’ve been synchronizing data to the new mountpoint, and earlier this evening, we made the switch. This resulted in about 2 minutes downtime.

Now, seeing as causing more I/O load on the live drives would result in Bitbucket being completely unreachable, we did our best to get the new RAID disks into action with as-fresh-as-possible data before we made the switch. This allows us to have minimal downtime, but it also means that the live data will now be a few hours old.

If you’re seeing old data in your repositories (or in the worst case, missing repositories that were created within the last few hours before the switch), that’s to be expected.

We’re currently synchronizing everything back from the now-legacy drives to the live drives. Synchronizing data can be a somewhat lengthy operation, but it’s going pretty fast.

In the end, this should result in a much faster Bitbucket. We’ve been under I/O strains for some time now, and this is the first (large) step in resolving these issues for good.

  • Martin

    Jesper –

    you’re a brave man; I hope your new Raid setup screams.
    Do you know why EBS doesn’t perform as you would wish?

  • http://bitbucket.org/jespern/ Jesper

    @Martin:

    Well, it’s a known issue. EBS drives don’t perform very well, and even Amazon recommends RAID0’ing them.

    You can read more here: http://orion.heroku.com/past/2009/7/29/io_performance_on_ebs/

  • Mike Wynn

    Well that’s just great – I noticed missing parts of a repository earlier and re-pushed a whole bunch of change sets. I wonder how much things will be fucked up when you system settles down.

    A heads up would’ve been nice.

  • Martin v. Löwis

    Did this migration affect the mirrors? I notice that the Django mirrors are not getting updated.

  • http://twitter.com/ltackmann Lars Tackmann

    Interesting approach. How about using a parallel file system instead with EBS, this would allow you to stripe the reads among multiple machines. Checkout http://www.pvfs.org and http://www.lustre.org

  • http://twitter.com/ltackmann Lars Tackmann

    Interesting approach. How about using a parallel file system instead with EBS, this would allow you to stripe the reads among multiple machines. Checkout http://www.pvfs.org and http://www.lustre.org

  • http://twitter.com/ltackmann Lars Tackmann

    Interesting approach. How about using a parallel file system instead with EBS, this would allow you to stripe the reads among multiple machines. Checkout http://www.pvfs.org and http://www.lustre.org

  • http://www.telephonefaxswitch.net Telephone Fax Switch

    Oh!…that’s great helpful, it’s so right to me! Million thanks for the article,

  • http://www.telephonefaxswitch.net Telephone Fax Switch

    Oh!…that’s great helpful, it’s so right to me! Million thanks for the article,

  • http://www.prada-outlet-store.com prada

    Here elaborates the matter not only extensively but also detailly .I support the
    write's unique point.It is useful and benefit to your daily life.You can go those
    sits to know more relate things.They are strongly recommended by friends.Personally
    I feel quite well.. http://www.prada-outlet-store.com

  • http://www.saleboots.co.uk ugg sale

    Mark S. is definitely on the right track. If you want to get a professional looking email address, Id recommend buying your name domain name, like or
    discount ugg boots
    If its common it might be difficult to get, however, be creative and you can usually find something.

  • Pingback: FREDERICK

  • Pingback: SERGIO

  • Pingback: BRETT

  • Pingback: ALFONSO

  • Pingback: BYRON

  • Shrawan Patel

    This post excellently highlights what the author is trying to communicate. Nonetheless, the article has been framed excellently well and all credits to the author. For more information on how to load balance your web servers, please visit ..nhttp://serverloadbalancing.biz/wordpressbiz/, nhttp://serverloadbalancing.info/wordpressinfo/

  • http://bitbucket.org/jespern/ Jesper

    Mike,

    Our migration job is set to not override anything newer than it’s looking at. If you pushed your changesets while the migration was going on, that’s fine, we won’t overwrite anything.

    During the sync, it was also impossible to push things that would collide with the sync, due to the graph implementation in Mercurial.

  • http://bitbucket.org/jespern/ Jesper

    Yep, we temporarily disabled the mirroring crons. They’ll be back today.