Data study: When do errors happen?

Every developer has shipped memorable errors in their code, it’s part of the job description and the learning curve. But are there any trends for when these errors occur? Is there a link between the number of releases made and the number of errors generated? And how does it fit with the working hours of the development team?

Pushing the release button

We analysed a sample of over 10 million errors and 40 thousand releases from our database, this is what we found out:

Spoilers

TL;DR? Here’s a summary of the key stats:

  • 73% of all errors occur outside of office hours. But most code is shipped during office hours.
  • The most popular time for shipping is after lunch, and the most popular day is Wednesday.
  • Developers push out 25% less releases on Fridays than on other weekdays.
  • Europeans are more likely than Americans to ship code on a Friday, but less likely to work over the weekend.

The data

Most organizations have a certain amount of background errors, on average 57.2 errors per organization in any 24 hour period. To reduce noise, we filtered out instances of old and recurrent errors via our error grouping function. This meant that only new species of bugs were reported as they occur, reducing the number of errors reported in any 24 hour period to an average 1.5 new error groups per organization.

An analysis of this new error rate showed a slight increase in the daytime when new code is released and traffic volumes are higher, but as websites rarely sleep new error groups continued on through the night.

A whopping 73% of all errors occur outside of office hours. Which has some pretty big implications for workflow in web development teams.

Error groups by hour of day

Chart 1. The percentage of errors occurring by hour (Monday to Friday). New error groups per hour are shown as a percentage of the total number of error groups occurring in a 24-hour period.

Breaking down the error rate by day of the week showed a drop in new error groups over the weekend, but when viewed together nearly 30% of errors happen on the weekend.

Error group by day of week

Chart 2. The percentage of errors occurring by day. New error groups by day are shown as a percentage of the total number of error groups occurring over a 7-day period.

Most code is shipped during office hours. The peak time for shipping is immediately after lunch and falls sharply after 4:00 p.m. The American day appears to start a little earlier, but Europeans are more likely to ship code in the evening.

Releases by hour of the day (weekdays)

Chart 3. The percentage of releases made by hour on a weekday. Releases are shown as a percentage of the total number of releases made in a standardized 24-hour period (Monday to Fridays).

Do people ship on a Friday?

It is considered good practice to avoid shipping code on a Friday. Generally accepted wisdom states that new code will introduce new errors, and these could in turn create big problems, or worse still, cancelled weekends.

Releases by day of week

Chart 4. The percentage of code releases made by day. Releases are shown as a percentage of releases made over a standardized week. Public holidays were not excluded.

Developers make 25% less releases on Fridays than on other weekdays: 14% of a week’s releases are made on a Friday, compared to an average of 18% for Monday through Thursday.

Americans are slightly more observant of the ‘don’t ship on Fridays rule’ than Europeans who are perhaps slightly more laissez-faire: Americans release 12% of their code on a Friday, compared to 14% for Europeans. In spite of this, Europeans are less likely to work on weekends than their American counterparts, shipping 12% of a week’s releases over the weekend, compared to 18% in the U.S.

Is there a correlation between releases and errors?

Error groups and releases by day of week

Chart 5. New error groups and releases by day of the week. Both errors and releases are shown as a percentage of the total occurring over a standardized week. Note the separate axes.

There appears to be a good correlation between the number of releases made and new error group instances (note the separate axes). However, this does not necessarily imply causation. It could be argued that new releases introduce new error groups, and the lower levels of errors over the weekend are a result of fewer releases. Alternatively it could also be argued that web traffic is lower over weekends, and so errors are less likely to occur.

For the moment, it is difficult to isolate these two variables, but the recent launch of Opbeats performance metrics should make this possible in the future. 

Footnote

  • The data analysis is based on a sample of over 10 million web application errors logged, and approximately 40,000 releases made in Europe and the U.S. using the Opbeat web operations platform.
  • For the purpose of this study we have calculated office hours as between 9:00 a.m. and 5:00 p.m., Monday to Friday (including public holidays).
  • Opbeat is now available on both iOS and Android, allowing developers to manage releases, errors and workflow when they’re not in the office, or outside of office hours.
About the author
Tom Stenson is Opbeat's primary communicator, and legend has it that he has actually ghostwritten everything on this blog. He can be reached at tom@opbeat.com for grammatical errors.
You can follow him on Twitter or connect on LinkedIn.