Scheduled downtime: Bitbucket moves data centers on July 7

By on July 2, 2013

Bitbucket data center migration

In a few days we’ll flip a switch and migrate Bitbucket to a new hosting provider. During this migration Bitbucket will be unavailable up to six hours starting Sunday, July 07, 2013 at 19:00:00 UTC.

UPDATE: the migration is now complete — thanks for your patience everyone!

Scheduled downtime

Over the last several months we’ve put together a detailed plan that limits downtime to no more than six hours. Why so long? Bitbucket has maintained a synchronization of our filesystems and databases between the two data centers for several weeks. When we bring down Bitbucket at the current data center, it will take a few hours to finalize the replication between the two data centers before we can bring up the new location.

Once the synchronization is complete, we will bring the new location up and perform a series of integrity and performance tests to assure the new hardware is ready to go before making it publicly available.

Why we’re moving

Bitbucket has seen amazing growth since introducing free unlimited Git repositories for individuals and small teams. With over a million users now utilizing Bitbucket to build better software, moving our data center allows us to:

How this affects you

With the move to a new data center comes the addition of IP addresses.

Incoming IP addresses will be:

Outgoing IP addresses will be:

When pushing and pulling over SSH you’ll see a warning similar to this:

Warning: the RSA host key for ‘bitbucket.org’ differs from the key for the IP address ’131.103.20.167′

The warning message will also tell you which lines in your ~/.ssh/known_hosts need to change. Open that file in your favorite editor, make the necessary changes, then retry your push or pull.

Stay updated

During the migration, we’ll post updates on Twitter and our status site providing the current status so you can follow our progress.

Thanks for your patience as we work to increase Bitbucket’s performance and reliability. Please contact us at support@bitbucket.org if you have any questions about or during the migration.

  • noname

    Many thanks for the great work

  • Anonymous

    Good luck, Thanks for all the Fish!

    • Katharina Sabel

      Well I’d hope they not just take all our software and leave permanently XD

  • SSH User

    Can you post here what the expected SSH fingerprints will be for the new IP addresses?

    • Charles McLaughlin

      The host keys and fingerprints will remain the same.

      • PT

        Thats not what the article implies ;)

        Warning: the RSA host key for ‘bitbucket.org’ differs from the key for the IP address ’131.103.20.167′

        The warning message will also tell you which lines in your ~/.ssh/known_hosts need to change. Open that file in your favorite editor, make the necessary changes, then retry your push or pull.

  • DeusEx

    Well i hope a new Datacenter will make it possible that we could use a bigger internal Wiki with more links,- i wait since weeks.

  • http://junian.net/ Junian

    Good luck guys, You’re all awesome

  • faber75

    Wonderful, thanks for the great service you provide

  • Robin Speekenbrink

    Will the new location with its direct link to AWS speed up GIT transfers etc? It isn’t clear what the actual effects will be of this new move in real use cases?

  • Ruslan Madatov

    good luck))

  • http://abacmedia.com/ abacmedia

    Good luck!!

  • Shrek

    TNX for this service!

  • Jérémie Gomez

    Thanks a lot !

  • Steven

    Thanks <3 Luv you guys.

  • Ayesh

    Good luck guys. You are doing a really good job!

  • Tomas Cayuelas Ruiz

    Yeah!! Good luck!

  • Adrian Pirvulescu

    Great job so far. I am loving BitBucket! Good luck with the update, and wish you a smoth one :D

  • D4niel

    The banner you have at the top of the pages on BitBucket says this is from 10:00 PM to 2:00 AM on July 7 (4 hours). Is that starting or ending on July 7th? Also, it doesn’t say what time zone, though. The link there goes to the URL below, but that shows a 6 hour maintenance window. I don’t know whether the start time or end time is wrong.

    http://www.timeanddate.com/worldclock/fixedtime.html?msg=Bitbucket+data+center+migration&iso=20130707T12&p1=224&ah=6

  • Guillermo Gutiérrez Vásquez

    Good Lock and Success !

  • Jose

    Well done team! You provide an amazing service.

  • Hg aficionado

    I don’t like that you’re progressively treating Hg like a second-class citizen.

    Git, git, git, git, that’s all you’ve been talking about lately. If I wanted Git I’d switch over to Github, but Hg is just a better decision overall, way less painful, real, hardcore branching workflow and little to no friction.

    It bothers me that you should be the Mercurial ambassadors, embracing it and promoting it like a platform, and instead you’re acting more like VCS traitors.

    Joining the hip train. And leaving Hg as alternative.
    That’s just not right.

    • Anonymous

      There is only one (completely irrelevant) mention of Git in the whole article…

      • znogger

        He does have a point. This is BitBucket’s latest banner logo. Git, Git, Git.

        • Anonymous

          I think this is just ment to make clear that they’re competing against github… ;) with THE feature why I’ve got a bitbucket account too…

        • Anonymous

          True, but to be fair, git is by far the bigger market & is much more well known due to Github. That’s what they’ve got to pander to with regards to PR.

          As long as the support they provide for Hg is up to standard, does it really matter which they advertise?

          • http://tshepang.net/ Tshepang Lekhonkhobe

            I does, because perception matters. Also, Mercurial could do with more marketing, given it’s a favorite of mine :)

          • Anonymous

            Why does perception matter if they support it?

            The simple fact is that very, very few people use Mercurial. BitBucket is essentially the only major Hg hosting service. It’s just not an important market.

            & since BitBucket’s biggest competitor is git-only, that’s the focus – as it should be.

          • http://tshepang.net/ Tshepang Lekhonkhobe

            It matters because people would think it’s treated as second class. I also doubt the *very few* people thing… many people use Mercurial, notably in the Python community. Git may see 10x more use, but that’s still BIG.

    • Benjamin Kruger

      Maybe it’s time for you to git hip, too! ;-]

  • http://www.computersnyou.com/ Alok Yadav

    Good luck
    you guys are awesome :)

  • http://www.thefirm.com.br/ Ze Luiz Rodrigues

    Boa sorte no update!

  • Ovanes

    Good luck!

  • Chaz Hinkle

    Very cool. Thanks for the heads up. Love “The Bucket” and all you guys are doing!

  • Lance Cleveland

    Good luck & thanks for the great service!

  • Jules

    Dear Gurus,
    Could someone clarify the old addresses which are being replaced? Will it affect bitbucket.org -> 207.223.240.181 and .182?
    Regards,

  • baila

    @bitbucket do not crash anything :D

  • Amit

    Thanks for your great work. All the best!

  • Emi

    Good luck with this, thanks for everything, you are all awesome!

  • vasi

    I love the graphic!

  • Manthan

    Great job guys! Cheers for that!

  • Spider-Smart-System

    Wish u luck with the update. Please don’t brake our repositories =)

  • Gabriele Santomaggio

    Thanks guys!

  • O8P8

    Nice. Power full. :)

  • Rajesh Vassey

    Awesome, thanks for the upgrade. All the Best !

  • Lex

    It,s very very good! Good luck!

  • Jan Wąsak

    In that case I will be wearing my 1mil bitbucket tee and enjoying a beer in you honor on the 7th. Good luck! :)

  • Obj

    Do I have to backup my data by myself before 7 July 19:00 ?

  • http://about.me/tudorconstantin Tudor Constantin

    Could you offer technical details of your provider, some stats and a high overview of the infrastructure you’re using? we’re programmers, we love those kind of details.

  • http://jeliucalexandr.blogspot.com/ Alexandr Jeliuc

    We are waiting! :) Thx!

  • http://www.wezaz.com/ Hamza Aid

    Good luck with the migration process

  • http://adnmaster.apphb.com/ Mohamed Amine SEBBANE

    Thanks,

    For the last two years, your service has become better and better, and now “The Best”.

    I am huge fan!

  • Dominik Köllges

    Wish you a peaceful migration to the new data center. Keep it kicking :) And don’t you dare mess up my new baby… I hold bitbucket precisely for the private repositories quite dear to me already! So don’t mess it up :>

  • Luis Lewin

    good luck, keep doing great, and keep our repositories safe!

  • Oliver Montes

    Good luck, amazing service :) note: plz update to 100s of millions of users HAHA

  • Maggy

    good luck

  • Arkadius

    Good luck guys and thank’s for outstanding job !

  • Noon

    Hooray it’s back!

  • Taker

    I almost finished the BitQuest, but you deleted it :(

  • George Griffin

    Can you put the link to bit’s quest back up? I was still playing! :/

    • http://www.bitbucket.org Justen Stepka

      We’ll launch a version of the game soon… need to wrap up all our migration tasks still. Thanks for your patience.

      • George Griffin

        No worries. It was just surprisingly fun. Any chance that you folks will publish a repo? I can think of some fun places to take it.

  • http://www.vidmakr.com Dale Emmons

    Cool – where did you move to?

    • Jean-Michel Lemieux

      In case some bad guys are listening ;) sorry for the vagueness. We are hosting with NTT and have moved somewhere faster which can support our growth.

      • http://www.vidmaker.com/ Dale Emmons

        Totally understandable! I just like nerding out about datacenters. :)

        Keep up the great work!

  • http://www.opimedia.be/ Olivier Pirson

    I completed the level “#14 – Exterminate” of the “Bit’s Quest” but the “Next level” button failed!

  • Leandro Kersting de Freitas

    Great job.

  • Anonymous

    Any chance of creating a page in your confluence like this one: https://confluence.atlassian.com/pages/viewpage.action?pageId=321847940

    This way we can watch it and get updated when anything changes. :-)

  • Anonymous

    I am getting a “ssh_exchange_identification : read: Connection reset by peer” what should i do?

  • Michael J.

    Hi, should we expect the DNS to have propagated? I can’t get to the website, nor can I push code. Below is ping and traceroute — note the “multiple addresses” warning in the traceroute…

    BigIron:~ mjy$ ping bitbucket.org
    PING bitbucket.org (131.103.20.167): 56 data bytes
    64 bytes from 131.103.20.167: icmp_seq=0 ttl=48 time=51.331 ms
    64 bytes from 131.103.20.167: icmp_seq=1 ttl=48 time=50.499 m
    ^C
    — bitbucket.org ping statistics —
    8 packets transmitted, 8 packets received, 0.0% packet loss
    round-trip min/avg/max/stddev = 50.499/51.899/53.490/1.056 ms

    BigIron:~ mjy$ traceroute bitbucket.org
    traceroute: Warning: bitbucket.org has multiple addresses; using 131.103.20.167
    traceroute to bitbucket.org (131.103.20.167), 64 hops max, 52 byte packets
    1 10.0.1.1 (10.0.1.1) 0.492 ms 0.542 ms 0.353 ms
    2 * * * 7.879 ms 7.559 ms 8.195 ms
    3 te-9-2-ur01.lebanon.nh.boston.comcast.net (68.87.156.5) 8.073 ms 7.640 ms 8.468 ms
    4 po-50-ur01.berlin.vt.boston.comcast.net (68.87.146.214) 10.492 ms 17.590 ms 11.983 ms
    5 be-70-ar01.needham.ma.boston.comcast.net (68.85.69.177) 21.446 ms 19.258 ms 20.144 ms
    6 he-2-8-0-0-cr01.newyork.ny.ibone.comcast.net (68.86.93.185) 46.618 ms 47.518 ms 47.903 ms
    7 pos-0-1-0-0-pe01.111eighthave.ny.ibone.comcast.net (68.86.87.170) 46.949 ms 47.546 ms 48.060 ms
    8 ix-0-1-1-0.tcore1.nto-newyork.as6453.net (209.58.26.117) 44.722 ms 52.413 ms 45.302 ms
    9 ae-9.r05.nycmny01.us.bb.gin.ntt.net (129.250.9.213) 46.470 ms 47.469 ms 46.400 ms
    10 ae-1.r23.nycmny01.us.bb.gin.ntt.net (129.250.4.68) 47.498 ms 47.739 ms 46.097 ms
    11 ae-1.r20.asbnva02.us.bb.gin.ntt.net (129.250.2.9) 53.196 ms 53.033 ms 51.068 ms
    12 * * *
    13 po-1.a00.asbnva03.us.da.verio.net (129.250.27.130) 64.512 ms 51.411 ms 54.187 ms
    14 * * *
    15 * * *
    16 * * *
    17 * * *
    ^C
    BigIron:~ mjy$

  • Dt

    We cannot access to the bitbucket.org anymore :( At least from Hungary

    However I can ping the 131.103.20.167 IP

  • Anonymous

    If you’re having any kind of connectivity issue, please add a traceroute to this issue: https://bitbucket.org/site/master/issue/7718/cant-access-bitbucketorg-from-some-of-pc

    A very small set of users seem to be having issues and we’re tracking the issue there.