On the market, we have a lot of options when it comes to CI and CD toolkit. Open source, freemium, paid… Enterprise model, self-managed, software as a service… We could count them indefinitely, assigning certain product or solution names to the right groups. A true abundance of tools, and even greater abundance of articles, comparing those against each other. So, why would i write yet another comparison article if there's already plenty of those?
The answer is simple. I couldn't find any good article about features comparing 3 or 4 most often recommended solutions. So, here we are, to help you understand and find the right solution without burning time on reading docs and making tables. I already did the table service for you (pun intended), dig in.
It's too many solutions... What to do, how to live?
When I first searched for the phrase "Ranking of CI platforms 2021" I've found no less than 22 solutions with a bunch of advantages, disadvantages and small description made for each one. No answer, which has feature A or B. Which has paid CI/CD infrastructure? Even a simple link to docs. NOTHING!
How to pick a solution? How to do a practical comparation between a group of them, if you don't know what to ask about, don't have enough information and time is quite short? Well, It's pretty simple, try those few easy steps.
First, you need to create a base list of things most needed from the solution, from business perspective. There's a few useful questions for this stage:
- Do we require managed, or self-managed infrastructure?
- Is a connector for our current VCS (Version Control System) there?
- Are there any integrations with our tool stack (Jira, Slack, etc.)?
- Can we build code on our own runners/workers?
Secondly, you need to ask developers, which features would they expect to have on such a platform:
- Will there be any mean of integration with bots (Dependabot)?
- Will there be any possibility of manual retriggers?
- Would we be able to configure notifications on hooks?
Finally, You need to compare business and developers' needs with your questions:
- How many workers I can deploy without additional costs?
- Which features are free?
- Which tool has automation for Kubernetes (if we are using it)?
- Which tool has configuration with cloud objects (GCS, S3, etc.) out-of-the-box?
Basing on gathered information, you can narrow your search area down. If you are looking for SaaS platforms with self-managed runners, we already did it for you. If you are looking for self-managed CI/CD infrastructure, less so, but still, take a look at the rules prepared below, because a few solutions chosen for this comparison are available as self-managed as well.
Long story short about chosen tools
Solution available either as SaaS or/and to implement in private infrastructure, both in free plans. They have many tools built by themselves to cover the area of SAST, DAST, vulnerabilities scanning, registries and more. Easy to use tools, to help with migration and update processes. Built-in connections with Jira, GCP, AWS.
GitLab pricing (At the time of this article's release)
Gitlab docs are available here
Solution available in two editions: Community and Enterprise. The first one is for manual implementation on private infrastructure. It integrates seamlessly with popular Source Control Management providers. Designed especially for cloud solutions. Has limits like missing option to create library of pipelines and reusing it between repos.
Drone CI pricing (At the time of this article's release)
If you're interested in Drone CI docs, see here
Solution available either as SaaS, or to implement in private infrastructure, both in free plans. Installation on private infrastructure seems to be hard, but well documented for several possible options (GKE, EKS, etc.). Big advantage of this platform is its own library of ready, working job templates called orbs. It's also possible to make such orb yourself.
Circle CI pricing (At the time of this article's release)
Circle CI documentation is available here
A newcomer, i would say - the freshest solution that i know about. Platform where you run pipelines is always hosted by BuildKite, however your Runners are deployed on your own infrastructure. This is the main thing that highlights this solution, about which the creators openly boast. In the highest plan (which isn't much more expensive than the others) all useful things like audit logs, SSO, et cetera are available.
BuildKite pricing (At the time of this article's release)
Take a look at their docs here
And now, the great table
What means what?
⛔ - Not working/available
? - Not clearly described; might be too hard to efficiently implement (time > work)
⚠️ - Not available in every version (Might be included in premium but not in free, in enterprise but not basic, etc.)
✅ - Working/available
|Rule||Gitlab CI||Drone CI||Circle CI||BuildKite|
|SaaS for VCS||✅||⚠️|
|Commits containing [skip ci] are skipped||✅||✅||✅||✅|
|Trigger configuration (on event, branch, etc.)||✅||✅||✅||✅|
|Notifications about pipeline status||✅||✅||✅||✅|
|Built-in pre hooks configuration||✅||⛔||⛔||✅|
|Separate conditionals for pipelines and jobs||✅||⛔||✅||✅|
|Runner cache in GCS||✅||✅||⛔||✅|
|Runner cache in S3||✅||✅||⛔||✅|
|Has official Terraform provider||✅||⛔||⛔||✅|
|Library of pipelines||✅||⛔||✅||?|
|DIND not needed to build image||✅||⛔||⛔||✅|
*(more info here)
|Available metrics exposure, |
shipping build information to Datadog
Where is GitHub?
One might ask: Where is the most commonly used CI tool at the moment, GitHub Actions?
It is intentionally absent from this list. GitHub is not a complete CI/CD platform, which would allow you to configure more complex aspects of your workflow. Primary advantages of GitHub are simplicity of configuration and many built-in connections with popular tools like Spacelift or DependaBot, but that's it. From disadvantages, for example, you can't simply retrigger a single job from a workflow, or share a packed bunch of pipeline templates.
Before that, let me emphasize something - we didn't describe all existing tools and ecosystems for CI/CD here. Merely some of the most popular/interesting ones. I also did not mention open source platforms like Jenkins (oh, and by the way, Jenkins = 🤢) or Spinnaker, because they were outside the scope of this article. My main focus was on SaaS platforms, where you don't need to care about health of infrastructure and platform. Only pipelines, runners and permissions gradation.
In my humble opinion, out of all those tools on the market, I chose GitLab CI.
Why? It's simple, yet powerful. I think GitLab has the most stable infrastructure and their solutions feel most developed from all those to me, despite some misunderstandings in docs. You can create flows with dynamic changeable variables and rules for pipelines, manage pipelines, repositories, repository permissions, variables in repositories or group scope and much, much more by IaC.
- GitLab CI/CD Docs. (n. d.). https://docs.gitlab.com/ee/ci/ (accessed February 28, 2022).
- Drone CI Docs. (n. d.). https://docs.drone.io/ (accessed February 28, 2022).
- Circle CI Docs. (n. d.). https://circleci.com/docs/ (accessed February 28, 2022).
- BuildKite Docs. (n. d.). https://buildkite.com/docs (accessed February 28, 2022).