Over the last few years a movement seems to have emerged which is leading the development industry towards rethinking attitudes about how we release software, popularised by the Big Doggs like Facebook and Twitter. The old idea of planned releases staggered over months or years is being removed, with more and more people instead choosing to release not just early and often, but faster and faster while still holding on to quality. I think this change has been driven by the instant delivery nature of the web, and a proliferation of easily available and high quality tools and libraries.
But moving fast means having a workflow that helps, and doesn’t hurt. As a software company, at Box UK we are subject to a whole bunch of pressures which affect our demands for stability and flexibility in our codebases. These include…
When planning our strategy for source control, and how we funnel changes from development to production servers, we need to support these, and any other demands, as best we can. After all, we’re lazy devs, and will do anything to avoid hard work!
When I joined Box UK, in times of yore, we’d just moved our core product from being versioned with Accurev to the (then) decidedly more hip Subversion. We standardised that, and our other products, on the mainstream model of having an unstable mainline trunk (to which we could commit code at any time), and periodically cutting release branches which would be tested, QA’d, etc… These release branches would then only usually receive bug fixes, and be the stable point which we could make release tags from.
Stable release branch model
This (in my mind at least, feel free to disagree in the comments) is pretty much the standard model for managing the development and release of a software product, and has worked reasonably well for us since we adopted it.
When we’re working on new features our general workflow is:
We also of course make smaller bug fix releases while the project cycle continues.
You’re probably already thinking about how your process differs or improves on this, but it’s a pretty well worn route and one I’d bet a large majority of software companies use (with different problems for different implementations!). As “reasonably well” as we find this works however, it’s not without its problems.
1. Inaccurate testing / QA
The first issue that can occur is that the development and initial QA is done on a codebase that doesn’t actually reflect the state of what will ultimately make its way to the client for release. Whether you’re using Subversion, Git, or something else it doesn’t matter. The development code is two steps away from the live code.
2. Traffic jams
The next gotcha is that when multiple projects are going out at around the same time, after they’re merged to release, any delay in go live (additional requirements, change in client priorities, etc…) means they can block the path for other projects. We could merge stuff back out of release, but there could be more commits in there, and it’s basically not just a PITA but a problem that shouldn’t happen in the first place. Planning can help but this is always a potential headache. This problem is even more frustrating when you need to release quick support fixes.
3. Commit and run
Having an unstable trunk creates the temptation to commit code here when you’re not sure what to do with it. We assume it’ll get tested when the next release branch is made. But if it gets forgotten it could cause problems later.So, while we’ve been successfully using this model in all our projects for the last few years, it isn’t without its problems. Some can be minimised with discipline and toolage (note 1), but even then there are issues inherent in this model that can’t be totally removed. So can we do better?
As an experiment over the last few months we’ve moved some of our codebases onto a stable trunk model to see if it can help us. This model has the following general workflow:
Stable trunk model
The general rule being if it’s in trunk it’s on LIVE. Using this model we can remove the three main problems that I mentioned earlier:
As I said, we haven’t been using this model for very long, and only on a subset of our products. But one problem I can see that could possibly occur with larger codebases is merging fixes back to previous releases for clients that can’t just upgrade to latest. If this situation occurred we’d need to look at changing the tags we make from trunk to be more like branches that we could cherry pick fixes to – à la note 2.
This new model fits in well with the continuous delivery technique, and has been popularised by high profile companies like Github and Etsy. We’ve found it can be a more natural fit for the way we work, and it allows us to go faster. After successful trials we’re now rolling this out as the standard model for a large portion of our products; with active projects always on it’ll take a little time but we’re feeling confident that this is better for us.
Over the last year or two at Box UK we’ve also been making strides towards automating our development and release process, so the hope is in the future to hook this up with our CI (Continuous Integration) and possibly make more use of continuous deployment.
I’m sure people have created a gazillion ways of doing this, each with their own pros and cons. How do you manage code for release at your company, what problems have you faced, and what changes have you made to deal with this? Leave your own stories in the comments!