Opbeat is joining forces with Elastic – Learn more about Elastic APM

Staying sane with small incremental releases

Annoyed by huge code reviews? Blocked by something your co-worker is refactoring? Nervous about shipping? Having trouble giving feedback on a design draft because everything has changed? These are all symptoms that your releases are too big.

Huge releases are bad

Software used to be shipped as huge releases on diskettes or CD-roms, but in the browser-era that is no longer necessary.

At Opbeat, we aim to ship something significant at least every week (and small things several times a day). We are big proponents of small, incremental and frequent releases. We have simply found it to be a much better way to work.

However, from time to time, we slip up. In this post we will share our first-hand experience with you.

What happened?

We were sprinting towards the next level of features on Opbeat. We expanded some of our features so you can manage your web app better. Concurrently, we were tidying up some pain points we found in our information architecture, and we were giving the interface a little fresh paint here and there.

The list of changes included UI changes, IA clean-up and expanded features. On top of that we were reworking our Settings a bit, so they would be ready for the next version of our mobile app, amongst other things.

This was shaping up to be a huge release, and we experienced quite a bit of pain as a result. We had to remind ourselves that small releases are the way to go.

The advantages of small releases

When changes are small and incremental, they move more quickly through your release pipeline. The advantages:

  • less code to write
  • less code to review
  • testing the changes is faster, as the scope is smaller
  • more parallelism on the team as a result of less blocking
  • easier analysis of impact

The advantages of small and incremental changes are usually ascribed to the technical side of software development, but the same is true for design and UI changes. Rather than releasing one big redesign of everything, incremental changes in the UI are easier to make, review, test and analyze for impact.

Moreover, when you release, it sometimes breaks, and should you end up in a situation where you need to roll back a release, you are much better off with just a small change, rather than a full feature you need to take out.

Finally, and just as importantly, developers who ship frequently are much happier. Shipping is wonderful.

What we should have done

In our situation, we found ourselves with a branch with the IA changes as well as updates to the various Settings templates. They were all related (changes in the IA reflected how we linked to the Settings templates), but instead of making small, incremental changes, we had a branch with 100+ commits and a time-to-release of more than six weeks.

When working on products for the web, we are constantly tasked with scope decisions. Limiting scope of individual releases is hard, and there are a few reasons for that:

  • We, as developers (and designers), have a tendency to exaggerate the connection between changes
  • We will try to follow good DRY practice (or be lazy) and reuse shiny new things in other branches, and end up with massive change-sets.

An example:

“This is a tiny change that can be built nicely if I rely on that thing you’re building. I’ll will just go ahead and do it in your branch.”

Often, it turns out the “tiny” change itself grows in scope, and suddenly you have two, mostly unrelated, change-sets in a branch.

The mother-refactor is also a common pattern, in which a refactor just seems to grow and grow and grow in scope. The train of thought is:

“I’m in the process of refactoring A, and A uses B, and B could be much better. I’ll just refactor B while I’m at it.”

When you hear yourself thinking “I’ll just refactor this other thing while I’m at it”, resist. Tell yourself that even though that other thing is far from perfect, you will refactor it in the next cycle.

The solution to these scenarios is to continuously ask yourself if changes are so tightly connected that they can not, in any way, be shipped separately. Often you will find that things that initially seemed related, are in fact not.

If the problem relates to DRY, e.g. there is something new in another branch that is still in development that you want to use, the solution is often to branch off of the branch that has the new thing you need, but still ship them successively. This can be troublesome in itself, because it means you cannot ship your branch before the base branch is shipped, and if the scope of the base branch is too wide, this can drag out, blocking your release.

This is what happened to us, even when we tried avoiding it.

Practice makes perfect

With smaller, more frequent releases, it is important to invest in the deployment process in order to minimize deployment overhead. Make sure your process is reliable and fast. Automation and good tests are key. For more information, dig into Continuous delivery.

Start practicing right now, be consistent, and as with almost everything, practice makes perfect. At Opbeat we try to get better with every release, and we are still learning.

If you want more insight on how you are doing, you should start tracking your releases. At Opbeat, we use (fanfare) Opbeat. We can see how big our releases are, and time-to-release is especially helpful in understanding how well you are doing. See the image in the top for an example.

About the author
Ron Cohen is co-founder and CTO at Opbeat. A founding member of Django Copenhagen, Ron enjoys sharing his experiences with code and ops around the world.
You can follow him on Twitter or GitHub.