Yet another comparison of CI/CD products on the market

2022-02-28|By Dawid Butrykowski|Code

On the market, we have a lot of options when it comes to CI and CD toolkit. Open source, freemium, paid… Enterprise model, self-managed, software as a service… We could count them indefinitely, assigning certain product or solution names to the right groups. A true abundance of tools, and even greater abundance of articles, comparing those against each other. So, why would i write yet another comparison article if there's already plenty of those?


The answer is simple. I couldn't find any good article about features comparing 3 or 4 most often recommended solutions. So, here we are, to help you understand and find the right solution without burning time on reading docs and making tables. I already did the table service for you (pun intended), dig in.

It's too many solutions... What to do, how to live?

When I first searched for the phrase "Ranking of CI platforms 2021" I've found no less than 22 solutions with a bunch of advantages, disadvantages and small description made for each one. No answer, which has feature A or B. Which has paid CI/CD infrastructure? Even a simple link to docs. NOTHING!

How to pick a solution? How to do a practical comparation between a group of them, if you don't know what to ask about, don't have enough information and time is quite short? Well, It's pretty simple, try those few easy steps.

First, you need to create a base list of things most needed from the solution, from business perspective. There's a few useful questions for this stage:

  • Do we require managed, or self-managed infrastructure?
  • Is a connector for our current VCS (Version Control System) there?
  • Are there any integrations with our tool stack (Jira, Slack, etc.)?
  • Can we build code on our own runners/workers?

Secondly, you need to ask developers, which features would they expect to have on such a platform:

  • Will there be any mean of integration with bots (Dependabot)?
  • Will there be any possibility of manual retriggers?
  • Would we be able to configure notifications on hooks?

Finally, You need to compare business and developers' needs with your questions:

  • How many workers I can deploy without additional costs?
  • Which features are free?
  • Which tool has automation for Kubernetes (if we are using it)?
  • Which tool has configuration with cloud objects (GCS, S3, etc.) out-of-the-box?

Basing on gathered information, you can narrow your search area down. If you are looking for SaaS platforms with self-managed runners, we already did it for you. If you are looking for self-managed CI/CD infrastructure, less so, but still, take a look at the rules prepared below, because a few solutions chosen for this comparison are available as self-managed as well.

Long story short about chosen tools

GitLab CI

Solution available either as SaaS or/and to implement in private infrastructure, both in free plans. They have many tools built by themselves to cover the area of SAST, DAST, vulnerabilities scanning, registries and more. Easy to use tools, to help with migration and update processes. Built-in connections with Jira, GCP, AWS.


GitLab pricing (At the time of this article's release)

GitLab pricing (At the time of this article's release)

Gitlab docs are available here

Drone CI

Solution available in two editions: Community and Enterprise. The first one is for manual implementation on private infrastructure. It integrates seamlessly with popular Source Control Management providers. Designed especially for cloud solutions. Has limits like missing option to create library of pipelines and reusing it between repos.


Drone CI pricing (At the time of this article's release)

Drone CI pricing (At the time of this article's release)

If you're interested in Drone CI docs, see here

Circle CI

Solution available either as SaaS, or to implement in private infrastructure, both in free plans. Installation on private infrastructure seems to be hard, but well documented for several possible options (GKE, EKS, etc.). Big advantage of this platform is its own library of ready, working job templates called orbs. It's also possible to make such orb yourself.


Circle CI pricing (At the time of this article's release)

Circle CI pricing (At the time of this article's release)

Circle CI documentation is available here

BuildKite

A newcomer, i would say - the freshest solution that i know about. Platform where you run pipelines is always hosted by BuildKite, however your Runners are deployed on your own infrastructure. This is the main thing that highlights this solution, about which the creators openly boast. In the highest plan (which isn't much more expensive than the others) all useful things like audit logs, SSO, et cetera are available.


BuildKite pricing (At the time of this article's release)

BuildKite pricing (At the time of this article's release)

Take a look at their docs here

And now, the great table

What means what?

⛔ - Not working/available
? - Not clearly described; might be too hard to efficiently implement (time > work)
⚠️ - Not available in every version (Might be included in premium but not in free, in enterprise but not basic, etc.)
✅ - Working/available

RuleGitlab CIDrone CICircle CIBuildKite
SaaS for VCS⚠️
(GitHub)
⚠️
(GitHub, Bitbucket)
External Workers⚠️
Commits containing [skip ci] are skipped
Trigger configuration (on event, branch, etc.)
Scheduled pipelines
Notifications about pipeline status
Built-in pre hooks configuration
Parallelism
Separate conditionals for pipelines and jobs
Jira integration
Confluence integration
Slack integration
SonarQube integration
ArgoCD integration
Runner cache in GCS
Runner cache in S3
Has official Terraform provider
Library of pipelines?
Audit log⚠️
DIND not needed to build image
CI Linting⚠️
*(more info here)
YAML
Starlark
Jsonnet
Available metrics exposure,
shipping build information to Datadog

Where is GitHub?

One might ask: Where is the most commonly used CI tool at the moment, GitHub Actions?


It is intentionally absent from this list. GitHub is not a complete CI/CD platform, which would allow you to configure more complex aspects of your workflow. Primary advantages of GitHub are simplicity of configuration and many built-in connections with popular tools like Spacelift or DependaBot, but that's it. From disadvantages, for example, you can't simply retrigger a single job from a workflow, or share a packed bunch of pipeline templates.

Recommendation

Before that, let me emphasize something - we didn't describe all existing tools and ecosystems for CI/CD here. Merely some of the most popular/interesting ones. I also did not mention open source platforms like Jenkins (oh, and by the way, Jenkins = 🤢) or Spinnaker, because they were outside the scope of this article. My main focus was on SaaS platforms, where you don't need to care about health of infrastructure and platform. Only pipelines, runners and permissions gradation.

In my humble opinion, out of all those tools on the market, I chose GitLab CI.


Why? It's simple, yet powerful. I think GitLab has the most stable infrastructure and their solutions feel most developed from all those to me, despite some misunderstandings in docs. You can create flows with dynamic changeable variables and rules for pipelines, manage pipelines, repositories, repository permissions, variables in repositories or group scope and much, much more by IaC.


References