Please join us at the new location: John Colins Cocktails at 138 Minna Street in San Francisco.

If you’re already taking advantage of our post-receive hooks and have ingress firewall rules to allow us to communicate with your servers, then you’ll need to update your rules.

The old source IP address was:
63.246.22.222

The new source IP addresses are:
207.223.240.187
207.223.240.188

For more information on our post-receive hooks, please see our docs.

This will be Kieran’s first time at WWDC, and he’d love to meet you! Shoot the breeze, get your questions answered, or just share some SourceTree love. Plus, he’s got a SourceTree t-shirt with your name all over it.

How can you resist this face?

Didn’t get a WWDC ticket? Can’t find him at the event? No worries, Kieran will be around for some WWDC parties during the week. You can catch him at:

• WWDC Bash Mini - 1355 Market Street on Monday June 10th at 7pm
• Stir - The Grand, 520 4th Street June 11th at 8pm
• Go-Karts and Booze at Scribd HQ - 539 Bryant Street on Wednesday June 12th at 6pm

Today the Bitbucket team is thrilled to announce we have added our millionth user, and we want to express our appreciation.

Help spread the message on Twitter

Meet and code with the Bitbucket team

To thank all of the 1,000,000 users and 80,000 teams who have signed up for Bitbucket, we’re offering the opportunity for you and a colleague to join us for the next ShipIt 24-hour hackathon day at Atlassian in San Francisco.

The Bitbucket team will be hosting our next ShipIt in our beautiful San Francisco office during the month of September, when San Francisco weather is at its finest.

One winner will get:

• Round-trip flights to SF for you and a member of your team
• One-week hotel stay in downtown SF or Berkeley
• All the caffeine, beer, and pizza you can handle

Enter now!

Here’s how: For each friend you invite who joins Bitbucket, you’ll be entered into the raffle. Want to know more? Check out our contest details to get all the legal bits.

Calling all locals: Come party with us in SF

Come and have a drink (or two) with the Bitbucket team to help us celebrate 1M users. We’ll be rocking it out with an open bar and great food at The House of Shields located in downtown San Francisco on June 20 at 6pm.

What should you bring? Yourself and anyone from your team. Register now for the party >>

UPDATE: So many Bitbucket fans were excited to take part in our 1M user celebration that we’ve moved it to John Colins Cocktails at 138 Minna Street in San Francisco. Please see the latest blog post for more information.

Over 75,000 teams are using Bitbucket to manage, review and share code in the cloud, and we’re looking for a support engineer to help those teams code better together.

• Are you a technical swiss army knife?
• Do you love learning and teaching everyone around you about technology?
• Are you the early adopter?
• Have you ever scripted your way out of a job?

If so, the Bitbucket team in San Francisco wants YOU to join our support engineering team!

What will you be doing?

You will have the responsibility of working with a wide range of customers, from start-ups to large corporations, to help identify and resolve product issues.

• Verify bugs and work with development on providing fixes
• Influence product direction based on customer feedback
• Communicate with customers to resolve issues
• Publish knowledge base articles to improve our self-help offering
• Collaborate with our global support team to serve customers worldwide

You’ll love it here…

Bitbucket is a part of Atlassian, one of the “Best Small & Medium Companies to Work for in America.” Our products reach over 25,000 enterprise customers globally, including Pixar, Twitter, Netflix, NASA and Facebook.

With offices in Sydney, San Francisco and Amsterdam, we are growing fast, and we are building a different kind of software company: one that listens to customers, values innovation and solves customer problems with brilliant simplicity. You’ll have a direct impact on millions of users as soon as you start! Find out about Life at Atlassian.

Does this sound like you?

Show us your portfolio, blog, Bitbucket or GitHub profile, open-source contributions, or things you’ve built.

Not for you? What about a mate? If you know of a great support engineer who fits the build, we’ll give you a referral bonus.

This morning’s brief outage was due to what has been discovered to be a faulty component of Bitbucket’s datacenter networking infrastructure. This networking equipment must be replaced as soon as possible to avoid further unexpected outages.

We apologize for the sudden nature of this downtime. Please contact us at support@bitbucket.org if you have any questions.

Edit any file, anywhere

From Bitbucket, you can edit any file, anywhere, all you need is your browser. Once you’re happy with your edits, you can commit straight away or create a pull request to contribute your changes.

Once you select Edit from the source browser, Bitbucket activates the editor and away you go. Syntax highlighting and diff view are all an integrated part of the code editing experience.

The editor is intelligent enough to determine the indentation style from the file’s original contents. If it’s off the mark, you can manually set your preferred indent mode and tab size.

The workflow is flexible for every team’s process – commit directly to the branch you’re editing on, or create a pull request and select your reviewers. If you don’t have write access, Bitbucket will automatically create a fork for you and commit your changes before submitting a pull request.

Quickly react to pull request feedback

If your team is like ours, you’re using pull requests for code review. Often times, you have changes you need to make based on feedback from your teammates before your code is merged. Editing online shouldn’t get in your way. The Bitbucket online editor allows you to edit your open pull requests directly. We’ll even update your pull request for you once you’ve committed your changes.

Built with CodeMirror

At the core of the Bitbucket online editor is CodeMirror – an excellent open-source JavaScript code editor component. The project’s terrific extension API as well as its top-notch documentation made implementation nearly painless. In addition, the friendly MIT license, the editor’s blazing-fast performance, and the project’s support for a wide number of languages made the choice of CodeMirror an easy one.

Windows client adds Git Flow support

Our Git Flow support in SourceTree for Mac turned out to be really popular, and it’s been one of the top feature requests from the community since we released SourceTree for Windows; so we’re very happy to announce Git Flow support for our Windows client!

If you’d like some more detail on what Git Flow is, check out Atlassian’s Git Flow tutorial or our blog Smart branching with SourceTree and Git Flow

SourceTree for Mac version 1.6 adds interactive rebase

Having to bring up a terminal when you want to do an interactive rebase is painful. If you aren’t familiar with rebasing or the command-line, the pain level ratchets up to excruciating. We asked ourselves: “How can we make this simpler and keep everything in SourceTree?”. Our solution is a visual, drag-and-drop interactive rebase:

The UI gives you visual feedback on each step of this previously advanced function. Want to re-order your commits? Want two commits to be one commit? No problem, just drag and drop the rows around.

There is so much more…

• Desktop notifications for incoming changesets
• New icons to match our Windows version of SourceTree
• “Log Selected…” on multiple files at once
• The push sheet in Mercurial shows which branch you’re pushing to
• New preference menu shows the pull count for the currently checked out branch
• Git repositories now remember the previously selected options in the pull/merge sheets
• Support for the latest Araxis diff/merge tools
• The “commit merged changes immediately” setting is now remembered between pulls
• Support for longer passwords for Bitbucket, Stash, GitHub and Kiln

Get SourceTree for Free!

If you’re new to Git, or just want a handy tool to make you even faster, download SourceTree – it’s free!

Every email has been rethought and redesigned for clarity and to keep the most important information front and center. Also, if you want to quickly react to changes you can quickly jump to the context of the notifications to Bitbucket with one-click. If your email does not accept HTML content, Bitbucket will fallback to text.

It’s not all looks, we added a few more improvements:

• Reply to comments via email: by simply replying to an email you can reply to code comments and contribute to the discussion
• Un-clutter you inbox: all emails now include a single click “Unwatch” link to help you un-watch activity that is not relevant to you.

Control the flow

Even beautiful emails can be annoying if you get too many or the wrong ones. To this end Bitbucket has changed some of the default rules around when you are notified of specific events. We think we have gotten the balance just right, however we do realize that everyones workflow is different so we have a included a number of new ways to customize the notifications Bitbucket sends.

Repository subscriptions

On the repository header you’ll notice a new control that allows you to customize the messages you receive when collaborating on a repository.

When watching a repository you’ll receive updates via your newsfeed. At times, you may want to receive emails on specific types of messages you care most about including:

• Pull requests – all pull request changes (new, updated commits, comments, transitions)
• Commits – when a new commit is added or a comment is left
• Forks – when a new contributor folks your repository
• Issue  – all issues changes (new, updates, transitions, or comments)
• Wiki – all wiki changes (new page creation or updates)

If you’re already watching a repository and have write or admin access, we’ve configured the watch setting so that you’ll receive notifications for pull requests and issues by default. Other times you might expect to receive an email:

• For any pull request, commit, or issue that you’ve created, commented, or updated in the past
• If you leave a comment or someone @mentions you – you’ll automatically begin watching the thread so you can stay on top of updated conversations

Command and control center

You can use the new “Notifications” account management preference center to fine-tune your existing watches.

• Manage which individual repositories, pull requests and issues you’re watching
• Unsubscribe to all notifications on Bitbucket
• Configure one-off emails such as when someone mentions you or if you want to receive product updates about Bitbucket

Get Started Now

Join the growing number of teams that host their code on Bitbucket and stay better connected with unlimited private repositories FREE for 5 users.

Timeline

At roughly 3am Pacific time on Sunday morning April 7th we (Bitbucket’s SF-based developers) were alerted about reduced availability of the site. Initially responded to by our support engineers, the problem required the help of the Bitbucket developers, which at this particular time of day complicated the investigation a bit.

When we went in we noticed extremely high load on all of our webservers, making them incapable of keeping up with incoming traffic. As load on our fileservers had also gone up significantly, we initially focused our attention on some of the recent changes we had made to our storage infrastructure and configuration.

When this did not reveal any regressions, we saw that our Dogslow reports were reporting an excessive number of page timeouts on a very specific, popular public repository. We saw that this repo, as well as its forks, were being flooded with requests, many hitting pages that are relatively expensive for us to render. At its peak as much as 10% of all traffic went to these repositories and since its access pattern differed dramatically from normal patterns (mostly hammering expensive pages), it overwhelmed us, filling up our worker pools and causing pages to time out.

As the traffic appeared to target just this one repo and its forks, we proceeded to temporarily make these repositories unavailable. This resulted in requests for them serving a 503 Service Unavailable. This immediately brought the site back, confirming that the load was indeed caused by this targeted traffic.

Next we looked for patterns in the now blocked traffic and noticed that while it seemed to come from unique IP addresses from all over the world, they shared a distinct User-Agent string identifying it as coming from a webcrawler.

We contacted the people at this company about their excessive traffic and preemptively went on to block them. We added them to our robots.txt, but since we couldn’t afford to wait for their crawlers to re-fetch that file, we also blacklisted their User-Agent string on our HAProxy load balancers.

By now the crawler people had gotten back to us saying they had reduced the aggressiveness of their crawling. However, at no point did their traffic show any sign of reduction and it was clear that we needed to keep the blacklist in place.

Immediately after deploying this, the site appeared healthy again and we were very keen to go back to sleep when we noticed a different problem. After a little while the site became unresponsive once again. Requests started to time out and a very ugly 504 page was being served by our Nginx-based SSL terminators. All the while, at about 60%, the actual load on our servers was a lot lower than normal. Something was preventing traffic from reaching the backend.

It turned out that when we blocked the crawler in HAProxy using its reqtarpit directive, we failed to realize that reqtarpit keeps the connection open for several seconds before closing it and with the crawler still opening a ton of concurrent connections per second, this starved our HAProxy connection pools, triggering 503s from Nginx. Instead of reqtarpit, we should have used reqideny to immediately reject the connection. We quickly corrected our mistake and brought the site back to life.

Conclusion

In hindsight we’re not too impressed by our performance in addressing this issue and would have liked a more speedy resolution. It’s worth noting that in addition to automated monitoring, we have staff in different time zones covering all 24 hours. In this instance staff in Asia were first to respond, but sometimes investigation of site-wide calamities requires the help of core developers, who in this case had to be woken up, delaying our response a bit.

It was about 6am by the time we flipped our status site back to green, ending a significant period of limited availability, for which we apologize.

We’d love to say that this will never happen again and of course we’ll be much better prepared to handle similar incidents in the future. Calamities are often unique, making it hard to predict and anticipate the unknown, but we are committed to making our infrastructure more flexible to make dealing with issues like this one less complicated and we’ll work as hard as we can to ensure uptime remains what you have come to expect from Bitbucket.