Last week we experienced unexpected downtime. Anytime something like this happens we perform a full post-incident investigation to ensure it never happens again. Here are our findings.
The Crash
We determined the problem started with our syslog server crashing. We found log rotation occurred at 04:05 as the cron.daily logrotate script ran:
-rw------- 1 root root 101714 Jan 5 04:05 messages-20120105.gz
-rw------- 1 root root 5012515 Jan 5 15:11 messages
Instead of responding to HUP with a graceful reload, rsyslogd completely stopped:
Jan 5 04:05:04 bitbucket04 rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="1905" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'restart'.
Jan 5 04:05:04 bitbucket04 kernel: Kernel logging (proc) stopped.
Based on previous log entries this is normal, but it typically restarts properly. In this case, rsyslogd didn’t start back up at all.
We’re currently running rsyslog 4.6.2 that ships with RHEL 6.2. We suspect we hit a known bug – the fix isn’t yet packaged for RHEL.
Blocking on syslog()
So we determined rsyslog crashed. We wouldn’t have expected that to be much of a problem. However, closer inspection shows we were syslogging over TCP. Our syslog server was configured for both TCP and UDP:
# Provides UDP syslog reception
$ModLoad imudp.so
$UDPServerRun 514
# Provides TCP syslog reception
$ModLoad imtcp.so
$InputTCPServerRun 514
But our clients were specifically configured to log over TCP – two @ signs prior to the hostname means use TCP and one would mean use UDP:
*.* @@syslog.private.bitbucket.org
When our central rsyslog crashed all applications logging to it over TCP were blocked and our systems became unresponsive. This included our custom applications and almost every other piece of our technology stack. Unknown to us, we were vulnerable to this problem since switching data centers over a year ago. Moreover, recently we began syslogging much more information, such as Nginx logs. In this instance, the crash together with the increased logging combined to make the situation much worse.
We’ve made these changes to ensure this doesn’t happen again:
- Disabled rsyslog’s HUPIsRestart option
- Added a Monit check to ensure rsyslog is restarted even if it crashes
- Switched from syslog over TCP to UDP
We’re sorry for the inconvenience this caused. We’re always working to make Bitbucket more reliable and we won’t see this particular issue occur again.
Posted in Uncategorized.
By Charles
– January 12, 2012
Make your Bitbucket experience faster by dropping the mouse and taking advantage of the new keyboard shortcuts.
Keyboard shortcuts provide a quick and easy way of navigating through Bitbucket and performing common (repetitive) actions without having to take your fingers off the keyboard. Pages you’ll find keyboard shortcuts useful for include:
- source code browser
- commit history
- pull requests
- searching from any page
Row highlighting
Press “j” and “k” to quickly move up and down between table rows and the source browser.
A yellow row highlighter is used when navigating tables and source to show you which row is selected, so you can quickly move between pages when pressing enter.
Shortcuts are just a key away
Navigate over to your Bitbucket repo and try out a few easy shortcuts to get going:
- Press “g” then “d” and you will quickly jump to your dashboard
- Press “/” to search your repo from any page
To see a list of all keyboard shortcuts, simply press the ‘?’ key from any page or read our keyboard shortcuts documentation and start using keyboard shortcuts today!
Posted in bitbucket, new stuff.
Tagged with changelog, changesets, code, commits, improvements, keyboard, pull request, shortcuts, ui.
By jstepka
– January 11, 2012
This morning around 6:00 am UTC Bitbucket began to fail with intermittent 500 errors, which continued for several hours.
Our investigation shows that the root cause was our syslog server crashing. The syslog queues on all of our other servers filled up and they became unresponsive.
We’re currently re-examining our syslog configuration, particularly looking at on-disk queuing, to ensure this single point of failure is avoided. We’re also adding new monitoring to detect if a similar situation were to reoccur.
We’re very sorry for the inconvenience this downtime caused. This and other service related updates can be found on our status site status.bitbucket.org.
Posted in bitbucket, status.
Tagged with availability, network, servers.
By jstepka
– January 5, 2012
Ever had the need to access Bitbucket when away from your desktop? At least a few Bitbucket users have and gone one step further!
Bitbucket has gone mobile with the help of several community developed mobile apps. Whether you are using an iPhone or Android device there are several apps to help keep tabs on Bitbucket source.
Bitbeaker
Bitbeaker is a free Android client that allows you to browse your repo source, issues, wiki pages, commit history and search public repos.
RepoMan
RepoMan is a free Android client that allows you to view your Bitbucket and GitHub repo issues, commit history, source diffs, branch history and search public repos.
Issue Bucket
Issue Bucket is a free iPhone application that offers a simple way to keep track of the issues and changesets for your Mercurial repositories on Bitbucket
iOpenSource
iOpenSource is a unique paid mobile phone client that allows you to search and download source codes in zip archive from Google Code, Github, SourceForge, and Bitbucket.
Go Mobile
These are just a few of the apps out there to go mobile with Bitbucket. Have you given any of them a try?
More Ideas?
Heard of any other good apps or other cool uses of the Bitbucket REST API? Let us know in the comments.
If you are interested in writing your own custom integration, check out our Bitbucket Developer Documentation.
Posted in bitbucket.
Tagged with android, api, changesets, commits, ios, iphone, issues, mobile, rest, search.
By jstepka
– December 21, 2011
Bitbucket will be unavailable for up to an hour on December 19, 2011 beginning at 02:00 UTC. During this time we will complete the database migration we began during our last maintenance window.
When we’re done, Bitbucket’s database server will be running on new hardware that will increase the performance and reliability of the site. We think you’ll like the results and thank you for your patience.
Posted in Uncategorized.
By Charles
– December 17, 2011
Bitbucket now has the option to view file changes—IDE-style—with side-by-side diffs:
With both versions of the file shown in full with synchronized scrolling, you can more quickly grok what has changed, making code review easier in:
- pull requests
- the compare view
- individual commits
Show me more: expandable context
Unified diffs are great for reviewing small changes, but lines hidden beyond the 3 lines of context can leave out important information. We’ve updated unified diffs with clickable skipped sections to expand the context in the diff, instantly.
Using side-by-side diff
Want to switch from unified mode to side-by-side mode? Simply click on any side-by-side button.
Posted in bitbucket.
Tagged with changesets, code, commits, feature, improvements, pull request, review.
By Nicolas Venegas
– December 8, 2011
Bitbucket will be unavailable up to one hour starting December 5th at 02:00 GMT. During this maintenance window we will complete the database migration from our previous maintenance window and move our database to newer, more powerful servers.
Unfortunately, last weekend we were unable to complete the migration within the maintenance window, so we reverted to our original configuration.
The database migration we attempted last week had several steps. 1) We needed to perform a restart to enable write-ahead-log shipping to the new server. 2) We needed to perform an rsync which we expected to take no more than 20 minutes. 3) We planned to promote the new replica to be our master database. Last weekend, the rsync between the old server and the new one exceeded the 20 minutes allotted. It became was clear something was wrong and we rolled back the deployment changes.
After investigation, we determined the cause of the problem was that the new server’s primary network interface was bridged for use with KVM and libvirt. This bridging allowed us to practice the database migration on virtual machines. However, it also resulted in a degradation of network performance which caused the rsync to take longer than expected. To resolve the performance problem immediately we unbridged the NIC on the new database server.
On December 5th, we’ll attempt the migration again, following the same procedure outlined above. Total downtime should be no more than one hour. We apologize for the inconvenience of downtime, but we’ll be back with an even faster Bitbucket.
Posted in bitbucket.
By Charles
– December 3, 2011
Bitbucket will be unavailable up to one hour starting November 28 at 02:00 GMT. During the maintenance window we will move our database to newer, more powerful servers.
Thanks for your patience as we work to increase Bitbucket’s performance and reliability.
Posted in bitbucket.
By Charles
– November 26, 2011
