Fighting with Fowler on Continuous Integration

Published by marco on

Updated by marco on

The article Continuous Integration by Martin Fowler makes many interesting points. It is a compendium of know-how about CI by one of the industry heavyweights, who’s been using it for a long time.

While I found a lot of what he had to say interesting, I did wonder how applicable CI is for the kinds of teams that I know and work with. He makes several statements toward that end that pretty severely limit the applicability of what he calls “true CI” for many, if not most, teams.

I think he should have started his article with a very clear delineation for which kinds of organizations this kind of process is appropriate or efficient. In leaving it out, he seems to suggest that it’s the best for everyone, but at the end of the article, he lists what are, for me, quite severe restrictions. For example,

“Continuous Integration is more suited for team working full-time on a product […]”
“[…] elite teams deployed to production more rapidly, more frequently, and had a dramatically lower incidence of failure when they made these changes.”
“If a team attempts Continuous Integration without a strong test suite, they will run into all sorts of trouble”

I don’t get the impression that Fowler is discussing a dream scenario toward which one works, but rather what he considers to be the absolute minimum process that anyone should be utterly embarrassed about themselves for not already having. I didn’t see a single sentence in this 40-page, at-times repetitive document about how to actually get there from here—or whether that’s really appropriate for many projects that people who read Martin Fowler might be working on.

I wonder about the wisdom of prioritizing integration seemingly above all else.

Below are citations from the long paper, with my comments interleaved.

“This contrast isn’t the result of an expensive and complex tool. The essence of it lies in the simple practice of everyone on the team integrating frequently, at least daily, against a controlled source code repository. This practice is called “Continuous Integration” (or it’s called “Trunk-Based Development”).”

He says this a lot, but I never hear about the costs. Is there no amount of time lost on integrations that is too high a price? Is there no task that he doesn’t break down into a million pieces in order to accommodate this style of work? Is there no efficiency lost by making each task into 1-hour chunks of coding that the entire team then integrates? Is that what we’re doing now?

“This will consist of both altering the product code, and also adding or changing some of the automated tests. During that time I run the automated build and tests frequently. After an hour or so I have the moon logic incorporated and tests updated.”

I’m quite fed up with reading this kind of optimistic bulls%!t. What kind of programmers are these who can accomplish major work in one hour? Or are the tasks that Fowler can conceive of all so simple that they can be accomplished in an hour? I’m very suspicious about these kinds of statements. It reminds me of game developers in the 90s talking about how they’d “written the whole engine in a weekend”, but then the game still took five more years to deliver.

“Some people do keep the build products in source control, but I consider that to be a smell − an indication of a deeper problem, usually an inability to reliably recreate builds. It can be useful to cache build products, but they should always be treated as disposable, and it’s usually good to then ensure they are removed promptly so that people don’t rely on them when they shouldn’t.”

Sure. But—priorities. Your product is not the pipeline. It’s your product. You can’t make everything a slave to the process. Remember to fix that which you can fix quickly, but to focus on your own priorities. Don’t polish a build so that Martin Fowler is happy, if it’s going to make your customers wait a lot longer for their release.

“The tests act as an automated check of the health of the code base, and while tests are the key element of such an automated verification of the code, many programming environments provide additional verification tools. Linters can detect poor programming practices, and ensure code follows a team’s preferred formatting style, vulnerability scanners can find security weaknesses. Teams should evaluate these tools to include them in the verification process.”

“Everyone Pushes Commits To the Mainline Every Day

“No code sits unintegrated for more than a couple of hours.”

This feels completely divorced from reality, but maybe I just “don’t get it.”

“If everyone pushes to the mainline frequently, developers quickly find out if there’s a conflict between two developers. The key to fixing problems quickly is finding them quickly. With developers committing every few hours a conflict can be detected within a few hours of it occurring, at that point not much has happened and it’s easy to resolve. Conflicts that stay undetected for weeks can be very hard to resolve.”

I agree with the last sentence, but at what cost? It feels like you’re going to spend so much time committing and integrating. How is finding out if you have conflicts the highest-priority task your team has?

“Full mainline integration requires that developers push their work back into the mainline. If they don’t do that, then other team members can’t see their work and check for any conflicts.”

Who finishes anything non-trivial in an hour? I can’t escape the feeling that one-hour chunks is almost too granular, that this size was chosen because it aids integration. While that’s a noble goal, I wonder how appropriate it is for many tasks, and to what degree the shape of the process affects the size of the solution set.

“Since there’s only a few hours of changes between commits, there’s only so many places where the problem could be hiding. Furthermore since not much has changed we can use Diff Debugging to help us find the bug.”

But don’t you waste time hunting bugs that would have gone away by themselves if the process weren’t so frenetic? If you rebase everything, then you’ll still encounter every integration conflict. If you merge, though, you can skip many of those interim integrations because subsequent changes might have obviated prior ones that might have caused conflicts.

Instead of testing occasional version, you end up testing absolutely everything you do as if it were a release candidate. I’m not convinced that there’s no downside to that. I feel like it’s a waste of time if applied so mindlessly.

“Often people initially feel they can’t do something meaningful in just a few hours, but we’ve found that mentoring and practice helps us learn.”

I don’t know who you’re working with, but I wonder how useful is that? How useful is it to tailor your entire process to ruthlessly chopping up your work into tiny segments? What if that’s not how some people work? What if they can’t learn? Fire ‘em?

“Continuous Integration can only work if the mainline is kept in a healthy state. Should the integration build fail, then it needs to be fixed right away. As Kent Beck puts it: “nobody has a higher priority task than fixing the build”.”

You goal ends up being running to run the process, rather than to build the product. This sounds more and more like a cult.

“If the secondary build detects a bug, that’s a sign that the commit build could do with another test. As much as possible we want to ensure that any later-stage failure leads to new tests in the commit build that would have caught the bug, so the bug stays fixed in the commit build.”

“A team should thus automatically check for new versions of dependencies and integrate them into the build, essentially as if they were another team member. This should be done frequently, usually at least daily, depending on the rate of change of the dependencies.”

This seems like another thing that becomes a higher priority than building the product itself. Daily dependency check seems like overkill, but it’s automated, so who cares? He’s just running builds all the time, like we don’t have a climate crisis.

“if we rename a database field, we first create a new field with the new name, then write to both old and new fields, then copy data from the exisitng old fields, then read from the new field, and only then remove the old field. We can reverse any of these steps, which would not be possible if we made such a change all at once. Teams using Continuous Integration often look to break up changes in this way, keeping changes small and easy to undo.”

“Virtual environments make it much easier than it was in the past to do this. We run production software in containers, and reliably build exactly the same containers for testing, even in a developer’s workspace. It’s worth the effort and cost to do this, the price is usually small compared to hunting down a single bug that crawled out of the hole created by environment mismatches.”

I agree with this part, without qualification. At least as a goal.

“Being able to automatically revert also reduces a lot of the tension of deployment, encouraging people to deploy more frequently and thus get new features out to users quickly. Blue Green Deployment allows us to both make new versions live quickly, and to roll back equally quickly if needed, by shifting traffic between deployed versions.”

What about data schemas? What about if you don’t have a product that deploys on a web server or app store? I understand that there are solutions to this, but I wonder how great a fit they are to many teams? If your team is accustomed to SQL programming—or if you already have a suite of products that use SQL databases—then how worthwhile to your business is it to prioritize moving away from SQL to a local DB like SQLite, a NoSQL document store like RavenDB, or even to a completely different back-end like Rama?

“Continuous Integration effectively eliminates delivery risk. The integrations are so small that they usually proceed without comment. An awkward integration would be one that takes more than a few minutes to resolve.”

It sounds like very much like it prioritizes eliminating delivery risk over all else. It is only applicable to products built in this way from the beginning.

“Having to put work on a new feature aside to debug a problem found in an integration test [or] feature finished two weeks ago saps productivity.”

So does constantly integrating, though! It can be noise. It’s like the noise of micro-reviewing AI responses. You have to figure out the sweet spot for your team and iterate toward that goal, always ensuring that your team can deliver even if the dream process is not already in place. Make a diagram of all the facets and discuss a plan for your project. Pragmatic. Realistic.

“They found that elite teams deployed to production more rapidly, more frequently, and had a dramatically lower incidence of failure when they made these changes. The research also finds that teams have higher levels of performance when they have three or fewer active branches in the application’s code repository, merge branches to mainline at least once a day, and don’t have code freezes or integration phases.”

What if you don’t have an elite team?

“A two week refactoring session may greatly improve the code, but result in long merges because everyone else has been spending the last two weeks working with the old structure. This raises the costs of refactoring to prohibitive levels. Frequent integration solves this dilemma by ensuring that both those doing the refactoring and everyone else are regularly synchronizing their work.”

Some refactoring can’t just be done in mini bites like that. Sometimes, you work on a POC that takes more time to verify. Now what? Throw it away and build it from scratch in bite-sized pieces? Or integrate a long-lived branch, which is verboten?

I’m working on a sweeping change to the way solutions are configured. It involves changing packages and versions in four different solutions. Should I have merged to master everywhere and involved the whole team in my project? That sounds stupid. Sure, it takes longer to verify and integrate in one big chunk, but it has the advantage that it didn’t make upgrading the solution format the number-one priority for all developers for a sprint or two.

“[…] teams that spend a lot of effort keeping their code base healthy deliver features faster and cheaper. Time invested in writing tests and refactoring delivers impressive returns in delivery speed, and Continuous Integration is a core part of making that work in a team setting.”

For non-legacy projects. Continuous delivery can only really work for web-based products or apps. A lot of other products have to be deployed to processes that aren’t as easy to update five times a day.

“Continuous Integration is more suited for team working full-time on a product, as is usually the case with commercial software. But there is much middle ground between the classical open-source and the full-time model. We need to use our judgment about what integration policy to use that fits the commitment of the team.”

That is the first time that he’s conceded that maybe there are use cases to which this whole article doesn’t apply very well.

“If a team attempts Continuous Integration without a strong test suite, they will run into all sorts of trouble because they don’t have a mechanism for screening out bugs. If they don’t automate, integration will take too long, interfering with the flow of development.”

No kidding. You need some serious test coverage to continuously integrate and deploy. I also wonder about the size of the product you can legitimately do this. Can you imagine if your test suite takes ten minutes to run and you integrate three or four times per day? Can you imagine how much time you’re not developing software because you’re integrating someone else’s code? I understand that this happens eventually, but I wonder about the wisdom of prioritizing integration seemingly above all else.

“Continuous Integration is about integrating code to the mainline in the development team’s environment, and Continuous Delivery is the rest of the deployment pipeline heading to a production release.”

This is a good definition and I wonder that he rewrote this whole essay and didn’t put this right at the top.

“Continuous Integration ensures everyone integrates their code at least daily to the mainline in version control. Continuous Delivery then carries out any steps required to ensure that the product is releasable to product[ion] whenever anyone wishes. Continuous Deployment means the product is automatically released to production whenever it passes all the automated tests in the deployment pipeline.”

Also excellent definitions that make the distinction clear. Continuous Delivery is the one that many teams could strive for, even if they will never be able to do Continuous Delivery. The question is: at what cost?

“Those who do Continuous Integration deal with this by reframing how code review fits into their workflow.”

Well, that’s an interesting statement. Integration trumps review? Get your code in there and review later? Trust in your tests? Are you kidding me? You should review design, as well as implementation. If everyone’s coding and committing and pushing in hours, when do they review? Is the idea to have people communicate with each other only when they’ve already built something?