Anti Patterns of Continuous Integration

Continuous Integration(CI) is not a new terminology in software development. It is known for making software development easier by providing rapid feedback and reducing integration problems.

By Definition:

Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early.

By integrating regularly, you can detect errors quickly, and locate them more easily. — https://www.thoughtworks.com/continuous-integration

However, there are times when people say that CI is not working in a particular project because of team size/project structure/complexity. In this blog, I will talk about the common anti-patterns might lead to a situation which makes people feel that CI is not working for the project.

What is an Anti Pattern?

An anti-pattern is just like a pattern, except that instead of a solution it gives something that looks superficially like a solution but isn’t one.

— Andrew Koenig (https://martinfowler.com/bliki/AntiPattern.html )

According to the authors of Design Patterns, there must be at least two key elements present to formally distinguish an actual anti-pattern from a simple bad habit, bad practice, or bad idea:

* A commonly used process, structure, or pattern of action that despite initially appearing to be an appropriate and effective response to a problem, has more bad consequences than good ones.

* Another solution exists that is documented, repeatable, and proven to be effective

Below are the common anti-patterns of Continuous Integration.

Infrequent Checkin

This goes against the basics of continuous integration. In this case, the code stays on local laptops for a very long time without getting committed and checked in. The most prevalent reason is a large number of changes needed for the whole feature to finish. Since the check-ins are not frequent, the integration is delayed. While the feature is being developed, other developers check-in their code. The longer this feature takes the more code is checked in by others. This leads to a lot of merge conflicts because of the number are changes done.

It leads to tremendous integration effort because huge changes from different developers will lead to many merge conflicts. These conflicts are not easy to resolve because of the number of changes done.

The simplest solution to overcome this is, to split the feature into smaller independent tasks. At the end of every task, there should be a commit and check-in. Not only does it help in faster and hassle-free integration, but it also helps to have more specific commit messages. This can be useful if a specific task needs to be reverted.

In the image below you can see a possible list of tasks:

Possible list of tasks for a feature

Feature Branches

The core idea behind the Feature Branches is that all feature development should take place in a dedicated branch. This encapsulation makes it easy for multiple developers to work on features without disturbing the main codebase.

In a project, if feature branches never get changes from the mainline(the main branch for the repository) until it is finished, the project is not anywhere close to following Continuous Integration. In fact, this approach can be called continuous isolation.

The term continuous isolation came from a tweet and was expanded upon by Paul Hammant. Thoughtworks later identified the practice as “CI Theatre”.

An isolated feature branch leads to a lot of conflicts because, by the time the feature is completed, the mainline will have tons of changes. In case of huge changes by a number of developers, everyone now needs to sit together and merge, otherwise, some important changes might get missed while merging.

Image Source: https://www.martinfowler.com/bliki/FeatureBranch.html

The image above shows a repository containing many long-lived feature branches.The Branch by “Professor Plum” has got local commits P1, P2..P5 and she has been taking changes from mainline at points P1–2 and p3–4. Finally at P1–5 she merges it to mainline.The branch by “Reverend Green” which started at the same time has local commits G1, G2..G6 and has got changes from mainline at G1–2, G3 which are before P1–5. Now he decides to merge it to mainline and to do that he needs to incorporate all the changes done by “Professor Plum”. Since there are a lot of commits it will be difficult to integrate all of them.

The solution to the problem above is simple, avoid long living feature branches. There should be only one branch as a single source of truth. All the development work starts with checking out from this branch and should be pushed to the same. This single branch is commonly known as the trunk (usually master in case of Git). This development practice is known as Trunk Based Development since all the development happens against the trunk.

In projects, which need to have Pull Requests approval before the merge, this issue can be addressed by using short-lived branches per task. These branches get merged once the task is completed and do not need to exist until the whole feature gets completed.

Image Source: https://www.martinfowler.com/bliki/FeatureBranch.html

Above Image shows the use of short-lived branches. Reverend Green makes a local commit G1 and merges it to the mainline. Professor Plum makes a local commit P1 and before she can merge it into mainline, she needs to take commit G1, merge it locally with her changes and then push. Reverend Green will do the same when he needs to push the next commit. It is a series of multiple small merges and there is no need for a huge merge event.

Broken Build

Well, A Broken build is not a problem. That is one of the use cases of the build pipeline, To give early feedback when something goes wrong. The problem arises when it stays broken for a long time.

This leads to blocking because there shouldn’t be any other check-in until the build is fixed. Check-ins on a failing build will create more files, more changes, and more dependencies that make the detection and isolation of the defect difficult.

As stated above, the broken build is not the problem, so there is no solution to it. It’s just a discipline to make it a top priority to fix a broken build. If fixing the build requires a lot of time, reverting the commit is a valid option to fix the build.

Build Time

The whole point of continuous integration is to have fast feedback. A long-running build increases the wait time for developers after check-in since they need to wait for the build to finish before moving on. This can lead to infrequent check-ins to avoid the long wait time multiple times.

One reason for huge build time is running all sort of checks and tests as part of the build. To overcome this problem, the build should be configured to just compile the code and run quick unit tests. The other time-consuming checks/tests can happen as next steps in a pipeline fashion, which are not needed for the rapid feedback yet need to be tested.

The idea is to find a balance between tests/checks and build time so a build is stable enough to move on to the next task.

Image Source: https://i.stack.imgur.com/kXpHf.png

The image above shows a test pyramid, which segregates tests in 4 parts. Unit Tests, Integration Tests, Acceptance Tests, and UI or End2End Tests. The quantity of tests decreases and the run time increases as we move up in the pyramid.

The image below shows a pipeline which runs the above tests. As part of the build, only the Unit Tests will be run which take very less time. Everything else will run later but developers can move on to the next task once build is green.

A demo pipeline which runs different kind of tests in different stages

Build Feedback

This first step to fix a broken build is being aware that the build is broken. If the build is broken but no one is aware of it, everyone will be happy and continue working on the broken build.

There could be simple solutions for this:

• There could be a CI dashboard, showing the status of the build.
• The CI servers can be configured to notify teams with means like mail, slack. As soon as there is a notification about build being broken, that should become the priority,

The image below shows notifications from gocd-slack-bot about the build status.

Image Source: http://blog.ashwanthkumar.in/2015/02/gocd-slack-build-notifier.html

The horror of spam notifications

As mentioned above, the CI server can be configured to notify about build status. Now let’s think of a big team making a tremendous number of check-ins. If every 2 minutes there is a notification about build passing, it will lead to alert fatigue. People become inundated with build notifications and start ignoring the notifications. This makes the notifications ineffective because one might miss a notification about build failure.

To solve this, the CI server should be configured to not notify for every check-in. It should be configured to notify only when a new check-in fails, that’s when people should focus on the build, so it gets fixed. The other useful notification is when a new check-in fixes a broken build since it will enable others to check-in their code who were waiting for the build to be fixed.

Summary

• Small and independent commits frequently to the remote repository are the key to hassle-free continuous integration.
• Say no to l̶o̶n̶g̶-̶l̶i̶v̶e̶d̶ ̶f̶e̶a̶t̶u̶r̶e̶ ̶b̶r̶a̶n̶c̶h̶e̶s̶. Follow Trunk Based Development or short-lived branches for small tasks.
• The broken build should be the top priority. No other check-ins until the build is fixed.
• Reduce build time. It helps to get fast feedback if something goes wrong.
• Setup dashboard, a notification mechanism to inform team members about the build status.
• Only send important notifications from CI to avoid spamming.

Above are the major anti-patterns, which I have seen in the projects I have worked. I would love to hear about your experience too.

Liked reading it, let others know by clapping, sharing. Any feedback is appreciated.