Scheduled maintenance Sunday, August 21, 1800 GMT

Bitbucket will be unavailable for up to one hour starting Sunday, August 21st, 18:00 GMT. During this maintenance window we will:

  • Replace our core switches and upgrade our uplink Internet connections from dual 100 Mbps to dual 1 Gbps.
  • Upgrade kernels and apply performance improvements to our database servers.

The kernel upgrades and reboots will address two issues:

  • One of our servers randomly rebooted recently. After working with Red Hat to analyze a core dump we concluded it was a result of a divide by zero bug in the kernel.
  • While performing routine security audits and stress tests of our infrastructure we found network performance degradation on servers with Broadcom network drivers and Intel c-states enabled.

For more information on the c-state problem we recommend posts from Citrix and Dell’s Linux mailing list.

We found that disabling c-states in the BIOS wasn’t enough – as discussed on the Dell mailing list, we used intel_idle.max_cstate=2 as a kernel boot option, however this did not resolve our performance issues. After a bit more investigation we concluded passing intel_idle.max_cstate=0 to the kernel fixes the problem.

We’ve already applied that change and upgraded kernels on all of our boxes except our database servers. A relatively quick reboot this weekend will put these issues behind us once and for all.