How To Choose Your CI Tool

Software creation is a complicated process at the best of times; wouldn’t you agree? Given that, any process, or tool which reduces said complication is a valuable thing.

It’s for this reason that Continuous Integration (CI) has become increasingly accepted among professional software developers over several decades.

CI becomes even more valuable when you consider the range of tangible benefits which it brings; these include:

• Reduction in defect cost
• Increased visibility throughout the development process
• Reduced time from feature development to feature deployment
• Reduction in software bugs
• Reduction in errors related to deployments across multiple software environments
• Reduced risk when refactoring code
• Simplification of the software development process

Ultimately, all these lead to reduced software development costs. However, just because it brings such a broad and diverse list of benefits, doesn’t make choosing the correct solution an easy thing to do.

It, in fact, raises several questions:

• How do you decide on the right Continuous Integration tool; one that’s right for your organization?
• How do you choose one that’s the best fit for your team?
• How do you pick one that has the right mix of features, pricing, automation, extensibility, and so on?

I can’t answer these questions for you — only you can do that. But what I can do is to step you through eleven categories (presented in no particular order) which can help you make the assessment for yourself.

By the time you are finished reading, you will have more than enough information to make an informed choice, as well as to ask further questions before making a commitment.

Open Source or Proprietary Software

Let’s start off by weighing up open-source versus proprietary models. Open source has been around for many years and has long since shaken off its hippie connotation. Open source also underpins so much of modern computing, so much so that we’d be lost without it.

If you need an open source CI tool, there are a number to choose from, including: Gradle, Jenkins and Gitlab-CI.

But that doesn’t make it the right choice in every situation. It has the advantage of being low cost, if not free — and that can be enticing. It also provides complete code transparency, where anyone can review the quality and applicability of the source code.

What’s more, if the software doesn’t exactly provide what you need, when you have suitably skilled developers, you can either modify it, create a patch for it, or perhaps even create a plugin for it, so that it will do just what you need. All that’s needed are the skills and the time.

However, perhaps a proprietary solution is a better choice. Here are some reasons why; open source solutions will need to run on-premises. Given that there’ll be tangible costs, including setup and maintenance by suitably knowledgeable systems administrators.

However, by choosing a proprietary solution you often get commercial support as part of the package. Support is often, though not always, backed by a team who are intimately familiar with the product; such as CircleCI, TeamCity, Codeship, and Bamboo.

This kind of arrangement can often be of equal value to the software itself, especially if you encounter a tricky situation, or your existing tools don’t quite play well with others. In these situations, while you can’t modify the code, you can have someone on-hand, at little to no notice, to help resolve the situation.

Which advantages best fits your organization’s needs?

Support

Just because you’ve chosen an open-source tool doesn’t mean that you won’t get support however. It’s quite a common practice for the original developers behind an open-source product to form a company tasked with providing commercial support for it.

An excellent example is CloudBees which provides professional Jenkins support or Zend which provides commercial support and additional products for PHP, the open-source software language. In both of these cases (as well as for other open-source tools), you can download the software with no obligation to take up paid, professional support.

If you or your team are capable of supporting, extending, or patching the product, then you can use it free of charge indefinitely. But, should you need it, a vendor is there to provide support. But support goes beyond having someone to call or email in case of an emergency or tricky situation.

Regardless of whether the tool is proprietary or open-source, what options do they provide to help you help yourself? Do they have a Twitter (or other social media) account, where you are notified about outages, bugs, and new releases?

Do they have a community forum where you can engage with other users? Do they have online documentation and video-based training? Do they have a bug tracker where you can report issues? Make sure you consider these options before you make your choice.

Software-as-a-Service or Self-hosted

Now that we’ve considered open-source versus proprietary software, what kind of implementation works best? Would a Software-as-a-Service (SaaS), such as shippable or Codeship work best? Or would it be better, even being mandated by your organization, to go with a self-hosted solution, such as Jenkins or CruiseControl?

This question has interesting implications. Perhaps you work in an organization which mandates that the solution has to be hosted on-site, or in a company-controlled data center. Maybe you have an over-zealous security team who gets nervous when talk of external, off-site solutions comes up.

Perhaps you’re in an industry which has some form of legal requirements imposed upon them which necessitates an on-premise solution. If so, it’s good to know that you’re well served either way. But there are other factors to consider. Do you know where the solution is hosted? Is it in a data center in another country, subject to a different jurisdiction? Can you get access to the server in the event of an outage?

If you choose a hosted solution and you later want to migrate to a self-hosted option, how easy is it to get a copy of your data? This isn’t to slight SaaS solutions. I use several and am very happy with them. But these considerations are worth bearing in mind.

But let’s contrast these with self-hosted solutions. How much effort is involved in setting up a self-hosted solution? How many developer or sys-admin/devops hours are required to get the product up and running, and then to support it on a regular basis? What’s the cost-benefit analysis tell you, when you compare the two? Both options have their advantages. But both have their downsides as well.

Pricing Model

What about the product model? If you’re using an open-source tool there’s no direct cost which can be attributed to the product, such as purchase price, per-seat license(s), obligatory commercial support and so on.

However, your organization will still need to allocate staffing resources to learn the ins and outs of the tool, to get it installed and appropriately configured. On top of thisthere’s the ongoing support costs.

This might be a minimalist task. It could be quite involved also, based on how the installation and upgrade process works, what your staff retention rate and staffing levels are, for example. However, even if you have a paid solution, there will still be an element of education involved, both for developers and for those tasked with playing the role of support liaison

So, before you choose an open-source tool because it’s free to download or hosted off in “the cloud”, keep in mind the other factors that contribute to the total cost of ownership. Which model best suits your organization’s budget?

Operating System Requirements

This ties in nicely with the previous category. Let’s say, for example, that you’ve narrowed down your choice of solutions to a list of three, and of those three, you’re hooked on one in particular. Then, right as you’re about to make the decision to put it forward as the preferred choice, you notice that it’s only available on Windows. However, all of your servers run Linux.

If you chose this solution, excellent as it is, you’d be adding a, potentially, significant cost to your budget. Not just because (in this particular case) of the Windows licensing costs, but because of the extra support and staffing required to maintain the server.

If it were internal only, then perhaps the cost wouldn’t be so great. But if the server was public-facing because a component of your workforce was remote, then it introduces a security risk as well, which you’d have to have appropriately capable systems administrators on-hand to manage.

Setting the competing operating system issue aside, let’s say that the solution was available for your platform of choice. How much memory, disk space, swap space, free file nodes, and more does it need?

Is it a well written, and highly optimized solution which doesn’t need a particularly powerful server to support? Or is it a much resource hungry solution which needs top-of-the-line, newer hardware to power it?

This might seem absurd. But I’ve been in organizations which, while the official resources appeared modest, even feasible, when the solution was deployed, the reality of the situation was far different. It turned out on more than one occasion that the officially published resource requirements were far too modest and we were bitten quite badly.

Integration with Existing Tools

This one should be a core consideration unless you’re getting setup for the first time. The last thing that you want to do is to have to change the existing tools to match a new one. So ensure that your CI server integrates properly.

Having said that, to the best of my knowledge all CI servers should be quite accommodating out of the box, or at least able to be extended via plugins. In this respect, I’m primarily speaking about version control tools, such as Git, Mercurial, and Subversion. However, the languages available to provide custom extensions and plugins can also be a consideration.

Quick question: do the tools your assessing integrate with your existing ones?

Product Features & Extensibility

This is a hard one to quantify, as how many features are enough? Do you need everything as well as the kitchen sink? Or do you just need a few. Then there’s the longer-term perspective to consider. Perhaps you’re setting up a CI workflow for the first time, so you don’t know what you need yet, outside of a few basics.

However, as the workflow becomes more embedded and familiar, knowledge and awareness will grow. Given that, it is good to have a path for growth. At the start you may only need integration with an existing VCS along with a build history.

But, as time progresses you may come to need code quality tracking, simplified system maintenance, and the ability to integrate with a bug tracker, such as Jira. Try to brainstorm a list of all the features you may potentially need, then list them in order of priority.

Build Feedback

This can be a bit of a gray area, at least when everything is working well. In these times feedback might seem like an optional extra. Why would you want to be notified of the fact that the build is succeeding? You know that, as the commits are pushed and the changes are rolled out across the respective environments. That process is notification enough, right?

But what about when things go wrong? What about issues which indicate something more critical is in the process of happening? This can be anything from user account limits being reached, disc spacing being exhausted, to cumulative errors which lead to a build failure.

Then there’s the ability to provide feedback and reporting in different formats, regardless of if that’s a successful build, or broken one. Not everyone has a browser open all the time.

Not everyone wants to or is in a position to. So what about choosing a tool which provides a range of notification options. These can include notifications via email, Slack channels, remote servers, logging servers, instant messenger, and more.

Learning Curve

I alluded to this one earlier. How long will it take development and support staff to come up to speed with the software? Does the software have a well-designed interface (whether on the command-line or in some form of GUI), one that’s intuitive and easy to master, such as Codeship, which you can see below?

Or is it a cumbersome, non-intuitive mess of an interface? I’m not advocating for either a command-line or a GUI here. Regardless of which interface the product supports — both ideally — the interface has to be conducive to a modest, even small, learning curve.

If the learning curve’s too high, there’s a very real possibility that the tool will be avoided by the development team, whether passively or actively, eradicating nearly all the benefits of Continuous Integration.

Even better, does the tool require any learning at all — for the developers that is? If this learning curve can be reduced, as close to zero as possible, and administration requirements are fairly modest over the longer term, then you have the ideal solution.

Training & Documentation

This goes hand in hand with the previous point. The marketing information boasts of it being the be-all and end-all product for your Continuous Integration needs. But, once you’ve purchased, downloaded, or created an account with it, there’s next to no documentation and optional training on offer, then I’d be getting worried.

It’s almost inevitable that something will go wrong, or that something about the software will not be clear, to at least one person on the team. If information’s not forthcoming about how to work with the product, how to learn it, then you could find yourself rapidly reassessing your decision and looking to spurn it for another choice.

On the flip-side, if the documentation’s extensive, yet either poorly written or hard to navigate, then it may leave you in the same position, as the information is all be worthless.

As an example of good documentation, as well as a great first impression, have a look at how TeamCity, by JetBrains gets you started, immediately after downloading the product in the screenshot above. There are clear links to installation instructions, the administration reference material, along with a How-To reference.

What’s more, by clicking “Docs & Demos” you’re directed to a host of video-based training, a bug-tracker, community forum, and product Twitter account. If nothing else, this is very helpful, as well as very reassuring.

Does the tool you’re assessing provide this level of professional documentation?

Popularity

I’d like to finish up with this one. While there is never a clear correlation between popularity and quality (or success), it can be a good indicator of both the health and likely longevity of a product. The last thing that you want to do is to get on-board with a product which is clearly becoming extinct.

So, if you can, look for a gauge of its popularity. This is harder for some products than others. But some good ways to do so are looking for the number of downloads on GitHub, activity on Twitter and social media, activity in product forums, and attention in more traditional media.

In Conclusion

And those are eleven categories to help you make an informed choice of the correct continuous integration tool for your organization or team. While I could have gone further, I felt that the categories covered provided sufficient detail, without overwhelming you with too much information.

As Continuous Integration is such an integral part of creating quality software, I hope that you do find a solution that’s the right mix for your organization and gain the subsequent benefits. One last thing; Continuous Integration isn’t a set and forget situation. It’s a living process that will evolve over time.

So, please remember to revisit these categories periodically to ensure and reassess whether the workflow you’ve setup could do with further tuning or change.