Method
How the tools are ranked
Eight criteria, weighted to 100. Every tool is scored on all eight, the same way, from evidence anyone can check. Last reviewed 9 November 2026.
The criteria and weights
| Criterion | Weight | The question it answers |
|---|---|---|
| Pipeline speed | 20% | How long does a typical change wait from push to a pass/fail verdict? |
| Caching | 10% | Are dependencies, build outputs and container layers already there when a job starts? |
| Parallelism | 10% | How easily can one change's work be split across many machines? |
| Hosting model | 10% | Can it run where your code and data have to run: managed, in your cloud, or on your own machines? |
| Pricing transparency | 10% | Can you work out next month's bill from the public pricing page alone? |
| Migration effort | 15% | How much of your existing pipeline do you have to rewrite to adopt it, and how easy is it to go back? |
| Ecosystem | 10% | How many ready-made integrations, actions or plugins can you use instead of writing your own? |
| Fit for agent-written changes | 15% | When coding agents open many more pull requests a day, does waiting time and cost grow with every change, or less than that? |
Why these weights
Pipeline speed carries the most weight (20%) because time from push to verdict is what a team feels on every change. Migration effort and fit for agent-written changes come next (15% each): a faster tool that takes a quarter to adopt rarely pays back, and teams where coding agents open most pull requests multiply every minute of CI by the number of changes. Caching, parallelism, hosting model, pricing transparency and ecosystem are 10% each: they matter to everyone, but none decides the choice alone.
How a total is worked out
Each criterion is scored 1 to 5. A score of 5 earns the criterion’s full weight, a 1 earns a fifth of it. The total is the sum, rounded, out of 100. For example, Graphide CI scores 5 on pipeline speed, 5 on caching, 5 on parallelism, 3 on hosting model, 5 on pricing transparency, 5 on migration effort, 4 on ecosystem and 5 on fit for agent-written changes, which comes to 94/100. Ties are broken by speed, then migration effort.
What earns a 1, a 3 and a 5
Pipeline speed (20%)
- 5: Removes repeated work on every run (warm caches, only the tests a change can reach) on top of fast machines.
- 3: Standard hosted machines; speed depends on how well you tune your own workflows.
- 1: Speed depends entirely on hardware and plugins you buy, run and tune yourself.
Evidence: Machine sizes and per-job start-up as documented by the vendor, what the tool does to avoid repeated work, and vendor-published benchmarks only when they name the workload and the baseline they were measured against.
Caching (10%)
- 5: Caches sit on or next to the runner and are warm before the job starts, with no cache steps to maintain.
- 3: A key-based cache you configure per workflow, restored over the network each run.
- 1: Caching is something you build and host yourself.
Evidence: Documented cache types, where the cache lives relative to the runner, size limits and what it costs.
Parallelism (10%)
- 5: Splits tests across runners for you, balanced by timing, with high concurrency.
- 3: A matrix or parallel keyword; you decide the shards and balance them yourself.
- 1: Parallel runs need extra agents and plugins you configure by hand.
Evidence: Documented matrix, sharding and test-splitting features, and concurrency limits per plan.
Hosting model (10%)
- 5: Managed, hybrid and self-hosted options, across Linux, Windows and macOS.
- 3: Managed only, or one operating system or architecture only.
- 1: One narrow option with no way to run near your own data.
Evidence: Documented hosting options (managed, hybrid, self-hosted, single-tenant), operating systems and CPU architectures supported.
Pricing transparency (10%)
- 5: Published per-minute or per-vCPU rates, with a way to cap spend.
- 3: Published, but through credits, seats and add-ons you have to convert yourself.
- 1: Sales-quoted only, or the real cost is infrastructure and staff time you must estimate.
Evidence: The vendor's public pricing page on the review date. A price we could not find published is listed as "not public", never estimated.
Migration effort (15%)
- 5: Change one line per job (the runner label); undo it the same way.
- 3: New config format, but an importer or close mapping exists.
- 1: Rewrite every pipeline and run new infrastructure.
Evidence: What the vendor's own setup docs ask you to change, for a team starting from GitHub Actions (the most common starting point), and whether a documented way back exists.
Ecosystem (10%)
- 5: A large public marketplace that most vendors publish into first.
- 3: A solid set of plugins or orbs for common tasks.
- 1: Few integrations; most steps are your own scripts.
Evidence: The size and upkeep of the tool's public marketplace or plugin index, and whether it can reuse another tool's ecosystem.
Fit for agent-written changes (15%)
- 5: Each change runs only the work it can affect, keeps the same verdict, and agents can drive it directly.
- 3: Fast and parallel, but every change still pays for the full suite.
- 1: Every change runs everything on capacity you have to grow by hand.
Evidence: Documented features that keep each change's verdict while doing less repeated work (test selection that falls back to running everything when unsure, warm caches), concurrency headroom, and whether agents can set it up and read results themselves (API or MCP).
Evidence rules
- Primary sources first. Every score cites what the vendor documents: its docs, its pricing page, its changelog. Each score’s reason is printed next to it on the full ranking.
- No invented numbers. We do not publish our own ratings, review counts, savings or benchmark results. A price we could not find published says “not public” or “quoted” and links to where to ask.
- Vendor claims are labelled as theirs. A speed-up a vendor reports is quoted with its workload and baseline (for example, “against GitHub-hosted runners”), never turned into a promise for your pipeline.
- Speed claims need a baseline. “Faster” only means something against a stated starting point. Our guide to faster GitHub Actions shows how to measure your own.
- The verdict must not change. Anything that makes CI faster by running less only scores well if a failing test still fails the job, and if it runs more, not less, when it cannot tell what a change affects.
- Migration is scored from GitHub Actions. It is the most common starting point; where a tool is native to another host (GitLab CI), the verdict says what that means for teams already there.
- Dated. Scores and prices are as of the review date shown on every page. When a vendor changes a price or a limit, the entry is updated and re-dated.
What is in scope
The ranking covers tools a team would pick to run its CI: full CI/CD platforms (GitHub Actions, GitLab CI, CircleCI, Buildkite, Jenkins) and runner layers that run existing GitHub Actions jobs faster (Graphide CI, Blacksmith, Depot). They are scored on the same criteria because a team choosing between “switch platforms” and “keep the platform, change the runners” is making one decision. Each entry says plainly which kind it is.