Scheduled downtime: Bitbucket moves data centers on July 7

By on July 2, 2013

Bitbucket data center migration

In a few days we’ll flip a switch and migrate Bitbucket to a new hosting provider. During this migration Bitbucket will be unavailable up to six hours starting Sunday, July 07, 2013 at 19:00:00 UTC.

UPDATE: the migration is now complete — thanks for your patience everyone!

Scheduled downtime

Over the last several months we’ve put together a detailed plan that limits downtime to no more than six hours. Why so long? Bitbucket has maintained a synchronization of our filesystems and databases between the two data centers for several weeks. When we bring down Bitbucket at the current data center, it will take a few hours to finalize the replication between the two data centers before we can bring up the new location.

Once the synchronization is complete, we will bring the new location up and perform a series of integrity and performance tests to assure the new hardware is ready to go before making it publicly available.

Why we’re moving

Bitbucket has seen amazing growth since introducing free unlimited Git repositories for individuals and small teams. With over a million users now utilizing Bitbucket to build better software, moving our data center allows us to:

How this affects you

With the move to a new data center comes the addition of IP addresses.

Incoming IP addresses will be:

Outgoing IP addresses will be:

When pushing and pulling over SSH you’ll see a warning similar to this:

Warning: the RSA host key for ‘bitbucket.org’ differs from the key for the IP address ‘131.103.20.167’

The warning message will also tell you which lines in your ~/.ssh/known_hosts need to change. Open that file in your favorite editor, make the necessary changes, then retry your push or pull.

Stay updated

During the migration, we’ll post updates on Twitter and our status site providing the current status so you can follow our progress.

Thanks for your patience as we work to increase Bitbucket’s performance and reliability. Please contact us at support@bitbucket.org if you have any questions about or during the migration.

76 Comments

  • noname
    Posted July 2, 2013 at 8:58 pm | Permalink

    Many thanks for the great work

  • Anonymous
    Posted July 2, 2013 at 9:25 pm | Permalink

    Good luck, Thanks for all the Fish!

    • Katharina Sabel
      Posted July 5, 2013 at 2:38 pm | Permalink

      Well I’d hope they not just take all our software and leave permanently XD

  • SSH User
    Posted July 2, 2013 at 9:33 pm | Permalink

    Can you post here what the expected SSH fingerprints will be for the new IP addresses?

    • Charles McLaughlin
      Posted July 2, 2013 at 10:54 pm | Permalink

      The host keys and fingerprints will remain the same.

      • PT
        Posted July 5, 2013 at 7:38 am | Permalink

        Thats not what the article implies 😉

        Warning: the RSA host key for ‘bitbucket.org’ differs from the key for the IP address ’131.103.20.167′

        The warning message will also tell you which lines in your ~/.ssh/known_hosts need to change. Open that file in your favorite editor, make the necessary changes, then retry your push or pull.

  • DeusEx
    Posted July 2, 2013 at 10:50 pm | Permalink

    Well i hope a new Datacenter will make it possible that we could use a bigger internal Wiki with more links,- i wait since weeks.

  • Posted July 2, 2013 at 11:19 pm | Permalink

    Good luck guys, You’re all awesome

  • faber75
    Posted July 3, 2013 at 12:36 am | Permalink

    Wonderful, thanks for the great service you provide

  • Robin Speekenbrink
    Posted July 3, 2013 at 1:05 am | Permalink

    Will the new location with its direct link to AWS speed up GIT transfers etc? It isn’t clear what the actual effects will be of this new move in real use cases?

  • Ruslan Madatov
    Posted July 3, 2013 at 1:39 am | Permalink

    good luck))

  • Posted July 3, 2013 at 2:04 am | Permalink

    Good luck!!

  • Shrek
    Posted July 3, 2013 at 2:41 am | Permalink

    TNX for this service!

  • Jérémie Gomez
    Posted July 3, 2013 at 3:35 am | Permalink

    Thanks a lot !

  • Steven
    Posted July 3, 2013 at 3:40 am | Permalink

    Thanks <3 Luv you guys.

  • Ayesh
    Posted July 3, 2013 at 3:52 am | Permalink

    Good luck guys. You are doing a really good job!

  • Tomas Cayuelas Ruiz
    Posted July 3, 2013 at 4:46 am | Permalink

    Yeah!! Good luck!

  • Adrian Pirvulescu
    Posted July 3, 2013 at 5:36 am | Permalink

    Great job so far. I am loving BitBucket! Good luck with the update, and wish you a smoth one 😀

  • D4niel
    Posted July 3, 2013 at 6:27 am | Permalink

    The banner you have at the top of the pages on BitBucket says this is from 10:00 PM to 2:00 AM on July 7 (4 hours). Is that starting or ending on July 7th? Also, it doesn’t say what time zone, though. The link there goes to the URL below, but that shows a 6 hour maintenance window. I don’t know whether the start time or end time is wrong.

    http://www.timeanddate.com/worldclock/fixedtime.html?msg=Bitbucket+data+center+migration&iso=20130707T12&p1=224&ah=6

  • Guillermo Gutiérrez Vásquez
    Posted July 3, 2013 at 6:59 am | Permalink

    Good Lock and Success !

  • Jose
    Posted July 3, 2013 at 7:07 am | Permalink

    Well done team! You provide an amazing service.

  • Hg aficionado
    Posted July 3, 2013 at 7:40 am | Permalink

    I don’t like that you’re progressively treating Hg like a second-class citizen.

    Git, git, git, git, that’s all you’ve been talking about lately. If I wanted Git I’d switch over to Github, but Hg is just a better decision overall, way less painful, real, hardcore branching workflow and little to no friction.

    It bothers me that you should be the Mercurial ambassadors, embracing it and promoting it like a platform, and instead you’re acting more like VCS traitors.

    Joining the hip train. And leaving Hg as alternative.
    That’s just not right.

    • Anonymous
      Posted July 3, 2013 at 7:55 am | Permalink

      There is only one (completely irrelevant) mention of Git in the whole article…

      • znogger
        Posted July 3, 2013 at 4:44 pm | Permalink

        He does have a point. This is BitBucket’s latest banner logo. Git, Git, Git.

        • Anonymous
          Posted July 5, 2013 at 2:31 pm | Permalink

          I think this is just ment to make clear that they’re competing against github… 😉 with THE feature why I’ve got a bitbucket account too…

        • Anonymous
          Posted July 8, 2013 at 7:08 am | Permalink

          True, but to be fair, git is by far the bigger market & is much more well known due to Github. That’s what they’ve got to pander to with regards to PR.

          As long as the support they provide for Hg is up to standard, does it really matter which they advertise?

          • Posted August 3, 2013 at 3:09 am | Permalink

            I does, because perception matters. Also, Mercurial could do with more marketing, given it’s a favorite of mine 🙂

          • Anonymous
            Posted October 31, 2013 at 2:43 pm | Permalink

            Why does perception matter if they support it?

            The simple fact is that very, very few people use Mercurial. BitBucket is essentially the only major Hg hosting service. It’s just not an important market.

            & since BitBucket’s biggest competitor is git-only, that’s the focus – as it should be.

          • Posted November 1, 2013 at 11:40 pm | Permalink

            It matters because people would think it’s treated as second class. I also doubt the *very few* people thing… many people use Mercurial, notably in the Python community. Git may see 10x more use, but that’s still BIG.

    • Benjamin Kruger
      Posted July 3, 2013 at 6:07 pm | Permalink

      Maybe it’s time for you to git hip, too! ;-]

  • Posted July 3, 2013 at 8:14 am | Permalink

    Good luck
    you guys are awesome 🙂

  • Posted July 3, 2013 at 9:56 am | Permalink

    Boa sorte no update!

  • Ovanes
    Posted July 3, 2013 at 11:26 am | Permalink

    Good luck!

  • Chaz Hinkle
    Posted July 3, 2013 at 12:57 pm | Permalink

    Very cool. Thanks for the heads up. Love “The Bucket” and all you guys are doing!

  • Lance Cleveland
    Posted July 3, 2013 at 6:55 pm | Permalink

    Good luck & thanks for the great service!

  • Jules
    Posted July 3, 2013 at 10:17 pm | Permalink

    Dear Gurus,
    Could someone clarify the old addresses which are being replaced? Will it affect bitbucket.org -> 207.223.240.181 and .182?
    Regards,

  • baila
    Posted July 3, 2013 at 11:51 pm | Permalink

    @bitbucket do not crash anything 😀

  • Amit
    Posted July 4, 2013 at 3:03 am | Permalink

    Thanks for your great work. All the best!

  • Emi
    Posted July 4, 2013 at 3:25 am | Permalink

    Good luck with this, thanks for everything, you are all awesome!

  • vasi
    Posted July 4, 2013 at 6:15 am | Permalink

    I love the graphic!

  • Manthan
    Posted July 4, 2013 at 6:35 am | Permalink

    Great job guys! Cheers for that!

  • Spider-Smart-System
    Posted July 4, 2013 at 9:14 am | Permalink

    Wish u luck with the update. Please don’t brake our repositories =)

  • Gabriele Santomaggio
    Posted July 4, 2013 at 2:06 pm | Permalink

    Thanks guys!

  • O8P8
    Posted July 4, 2013 at 6:11 pm | Permalink

    Nice. Power full. 🙂

  • Rajesh Vassey
    Posted July 4, 2013 at 9:34 pm | Permalink

    Awesome, thanks for the upgrade. All the Best !

  • Lex
    Posted July 5, 2013 at 5:21 am | Permalink

    It,s very very good! Good luck!

  • Jan Wąsak
    Posted July 5, 2013 at 7:17 am | Permalink

    In that case I will be wearing my 1mil bitbucket tee and enjoying a beer in you honor on the 7th. Good luck! 🙂

  • Obj
    Posted July 5, 2013 at 11:46 am | Permalink

    Do I have to backup my data by myself before 7 July 19:00 ?

  • Posted July 5, 2013 at 10:14 pm | Permalink

    Could you offer technical details of your provider, some stats and a high overview of the infrastructure you’re using? we’re programmers, we love those kind of details.

  • Posted July 6, 2013 at 6:06 pm | Permalink

    We are waiting! 🙂 Thx!

  • Posted July 6, 2013 at 8:38 pm | Permalink

    Good luck with the migration process

  • Posted July 6, 2013 at 10:48 pm | Permalink

    Thanks,

    For the last two years, your service has become better and better, and now “The Best”.

    I am huge fan!

  • Dominik Köllges
    Posted July 7, 2013 at 5:20 am | Permalink

    Wish you a peaceful migration to the new data center. Keep it kicking 🙂 And don’t you dare mess up my new baby… I hold bitbucket precisely for the private repositories quite dear to me already! So don’t mess it up :>

  • Luis Lewin
    Posted July 7, 2013 at 9:29 am | Permalink

    good luck, keep doing great, and keep our repositories safe!

  • Oliver Montes
    Posted July 7, 2013 at 12:02 pm | Permalink

    Good luck, amazing service 🙂 note: plz update to 100s of millions of users HAHA

  • Maggy
    Posted July 7, 2013 at 12:07 pm | Permalink

    good luck

  • Arkadius
    Posted July 7, 2013 at 1:07 pm | Permalink

    Good luck guys and thank’s for outstanding job !

  • Noon
    Posted July 7, 2013 at 6:12 pm | Permalink

    Hooray it’s back!

  • Taker
    Posted July 7, 2013 at 6:18 pm | Permalink

    I almost finished the BitQuest, but you deleted it 🙁

  • George Griffin
    Posted July 7, 2013 at 6:27 pm | Permalink

    Can you put the link to bit’s quest back up? I was still playing! :/

    • Posted July 7, 2013 at 6:29 pm | Permalink

      We’ll launch a version of the game soon… need to wrap up all our migration tasks still. Thanks for your patience.

      • George Griffin
        Posted July 7, 2013 at 6:32 pm | Permalink

        No worries. It was just surprisingly fun. Any chance that you folks will publish a repo? I can think of some fun places to take it.

  • Posted July 7, 2013 at 6:32 pm | Permalink

    Cool – where did you move to?

    • Jean-Michel Lemieux
      Posted July 7, 2013 at 10:29 pm | Permalink

      In case some bad guys are listening 😉 sorry for the vagueness. We are hosting with NTT and have moved somewhere faster which can support our growth.

      • Posted July 7, 2013 at 10:34 pm | Permalink

        Totally understandable! I just like nerding out about datacenters. 🙂

        Keep up the great work!

  • Posted July 7, 2013 at 6:36 pm | Permalink

    I completed the level “#14 – Exterminate” of the “Bit’s Quest” but the “Next level” button failed!

  • Leandro Kersting de Freitas
    Posted July 7, 2013 at 6:50 pm | Permalink

    Great job.

  • Anonymous
    Posted July 8, 2013 at 1:13 am | Permalink

    Any chance of creating a page in your confluence like this one: https://confluence.atlassian.com/pages/viewpage.action?pageId=321847940

    This way we can watch it and get updated when anything changes. 🙂

  • Anonymous
    Posted July 8, 2013 at 7:22 am | Permalink

    I am getting a “ssh_exchange_identification : read: Connection reset by peer” what should i do?

  • Michael J.
    Posted July 8, 2013 at 8:04 am | Permalink

    Hi, should we expect the DNS to have propagated? I can’t get to the website, nor can I push code. Below is ping and traceroute — note the “multiple addresses” warning in the traceroute…

    BigIron:~ mjy$ ping bitbucket.org
    PING bitbucket.org (131.103.20.167): 56 data bytes
    64 bytes from 131.103.20.167: icmp_seq=0 ttl=48 time=51.331 ms
    64 bytes from 131.103.20.167: icmp_seq=1 ttl=48 time=50.499 m
    ^C
    — bitbucket.org ping statistics —
    8 packets transmitted, 8 packets received, 0.0% packet loss
    round-trip min/avg/max/stddev = 50.499/51.899/53.490/1.056 ms

    BigIron:~ mjy$ traceroute bitbucket.org
    traceroute: Warning: bitbucket.org has multiple addresses; using 131.103.20.167
    traceroute to bitbucket.org (131.103.20.167), 64 hops max, 52 byte packets
    1 10.0.1.1 (10.0.1.1) 0.492 ms 0.542 ms 0.353 ms
    2 * * * 7.879 ms 7.559 ms 8.195 ms
    3 te-9-2-ur01.lebanon.nh.boston.comcast.net (68.87.156.5) 8.073 ms 7.640 ms 8.468 ms
    4 po-50-ur01.berlin.vt.boston.comcast.net (68.87.146.214) 10.492 ms 17.590 ms 11.983 ms
    5 be-70-ar01.needham.ma.boston.comcast.net (68.85.69.177) 21.446 ms 19.258 ms 20.144 ms
    6 he-2-8-0-0-cr01.newyork.ny.ibone.comcast.net (68.86.93.185) 46.618 ms 47.518 ms 47.903 ms
    7 pos-0-1-0-0-pe01.111eighthave.ny.ibone.comcast.net (68.86.87.170) 46.949 ms 47.546 ms 48.060 ms
    8 ix-0-1-1-0.tcore1.nto-newyork.as6453.net (209.58.26.117) 44.722 ms 52.413 ms 45.302 ms
    9 ae-9.r05.nycmny01.us.bb.gin.ntt.net (129.250.9.213) 46.470 ms 47.469 ms 46.400 ms
    10 ae-1.r23.nycmny01.us.bb.gin.ntt.net (129.250.4.68) 47.498 ms 47.739 ms 46.097 ms
    11 ae-1.r20.asbnva02.us.bb.gin.ntt.net (129.250.2.9) 53.196 ms 53.033 ms 51.068 ms
    12 * * *
    13 po-1.a00.asbnva03.us.da.verio.net (129.250.27.130) 64.512 ms 51.411 ms 54.187 ms
    14 * * *
    15 * * *
    16 * * *
    17 * * *
    ^C
    BigIron:~ mjy$

  • Dt
    Posted July 8, 2013 at 8:25 am | Permalink

    We cannot access to the bitbucket.org anymore 🙁 At least from Hungary

    However I can ping the 131.103.20.167 IP

  • Anonymous
    Posted July 8, 2013 at 1:53 pm | Permalink

    If you’re having any kind of connectivity issue, please add a traceroute to this issue: https://bitbucket.org/site/master/issue/7718/cant-access-bitbucketorg-from-some-of-pc

    A very small set of users seem to be having issues and we’re tracking the issue there.