CI/CD Pipeline Architecture for High-Throughput Teams

If I want a CI/CD pipeline to handle more work, I don’t start by adding runners. I start by measuring flow end to end. The article’s core point is simple: pipeline speed is shaped by queue wait, runner start-up, job run time, gates, and deploy limits - not just by one fast build job.

In plain terms, this means I’d focus on five things first:

• Measure the right baseline: queue depth, queue age, execution time, provisioning time, runner use, lead time, deployment frequency, and failure or retry rates.
• Map the pipeline as a DAG: run only the jobs a change needs, start jobs as soon as dependencies finish, and keep approval gates visible.
• Split runner pools and queues by job type: for example, pull requests, releases, deployments, heavy builds, image builds, and restricted jobs.
• Scale from queue signals: queue age, backlog growth, and job arrival rate tell me more than CPU charts alone.
• Tighten execution flow: shard tests by duration, build once and promote the same immutable artefact, and size CPU and memory by job class.

A simple example from the piece makes the sizing point clear: if 120 jobs arrive per hour and each job uses a runner for 10 minutes, I need about 20 concurrent runner slots just to keep up with average demand - before I add room for peaks, retries, or cold starts.

What I like about this approach is that it keeps the focus on outcomes people feel: lead time, deployment rate, failure rate, and recovery time. In other words, I’d treat CI/CD as a flow system, not just a list of build steps.

::: @figure {CI/CD Pipeline Architecture: End-to-End Flow Optimization} :::

Building a high performant CI/CD platform through standardization - Mihir Vora & Kamlesh Vaghela

Model the pipeline as a dependency graph

Model the pipeline as a DAG, not a simple list of stages. Each job is a node. Each prerequisite is a directed edge. That means a job starts as soon as its dependencies finish, instead of waiting for an entire earlier stage to clear.

A practical flow looks like this: change event → workflow selection → affected-component detection → fast validation → build jobs → unit-test shards → integration and security checks → artefact aggregation → promotion → deployment. The key idea is simple: run only the jobs touched by the change. If a commit only affects the web application, the API and worker build jobs should stay idle unless the dependency model says they must run.

Map stages, gates and conditional paths

Running only affected jobs is the main way a graph cuts wasted work. Every job needs a clear role, plus a defined input/output contract, owner, run condition and failure outcome. Put fast, cheap checks first - formatting, linting, configuration validation and dependency checks - before anything costly starts. Build jobs come next, scoped to changed components where dependency analysis allows it.

From there, branch into parallel paths:

• unit-test shards
• integration tests
• end-to-end tests
• licence checks
• vulnerability scans
• policy checks

These paths don’t depend on one another, so they should run in parallel.

Use fan-in when one downstream job needs results from several branches. A common case is an artefact publication job waiting for all required test shards and security checks. That creates a diamond dependency: one preparation job splits into parallel work, then joins again at an aggregation or release gate. The aggregation job should show a clear pass/fail result without masking which branch or shard failed.

Conditional paths can save a lot of time, but they need to be conservative. A change to a shared API schema should trigger the schema producer, all known consumers and contract tests, even if the application files appear untouched. Keep conditional rules close to component ownership, and fall back to broader validation when dependency metadata is missing or uncertain. Manual approval gates should be explicit graph nodes with recorded owners, the approved artefact, the checked policy and a timeout, not invisible pauses in the middle of the pipeline.

That dependency map is what you use to shape runner placement and queue design.

Compare sequential, DAG and fan-out designs

The right topology depends on workload size and dependency structure.

More edges and larger fan-in can create new bottlenecks. So when you change the graph, don’t judge it by job count alone. Check it against change lead time, deployment frequency, change-failure rate and failed-deployment recovery time. That’s the scorecard that matters.

Once the topology is set, the next gain usually comes from matching parallelism to runner capacity.

Separate inter-job from intra-job parallelism

These are not the same thing, and mixing them up burns capacity. Inter-job parallelism runs independent jobs across runners. Intra-job parallelism uses multiple cores or threads inside a single job. Inter-job parallelism is limited by runner capacity and queue contention. Intra-job parallelism is limited by the job’s CPU, memory and tool-level scaling.

A pipeline with ten parallel jobs may still barely improve if each job runs a single-threaded build or waits on a serial database fixture. On the other hand, increasing workers inside one job can push up memory pressure and cut total throughput if it creates runner contention. Measure job duration and utilisation before changing either layer, and identify the critical path first. Only parallelise work on the critical path. Parallelism outside the critical path doesn’t cut lead time.

This graph also shows where runner pools, queue policy and autoscaling should vary by workload class.

Design runner pools, queues and autoscaling together

Runner placement, queue design and autoscaling are one design problem. The dependency graph tells you what can run. This section is about where it runs and how fast it can grow when demand jumps.

The best way to design all three is from the same workload model. For each job class, write down its arrival rate, usual duration, CPU and memory profile, network and security needs, acceptable queue time and business priority. That gives you a working map instead of a pile of guesses.

Place runner pools by workload class and access needs

A single general-purpose pool looks simple. In practice, it often turns into a bottleneck. One long compilation or end-to-end test can sit on a runner that a short pull-request validation job needed right away.

Split pools by job type to avoid that collision. In most setups, that means separating:

• general builds
• heavy compilation
• image builds
• integration tests
• end-to-end tests
• restricted-network jobs
• protected deployments

Route jobs with explicit capability declarations such as labels, tags, node selectors, taints and tolerations. Don’t lean on implicit runner selection and hope the right job lands in the right place.

Be direct with requirements like arm64, large-memory, private-network or production-deploy. And make the default pool unable to take privileged or production-scoped work.

Data locality matters too. A runner that sits far from source mirrors, dependency caches or artefact storage may look cheaper per hour, but it can still make the pipeline slower because checkout and upload take longer.

Those pool boundaries shape the queue design next.

Build queues that protect critical work

Use separate queues for pull requests, mainline validation, releases and deployments. If everything shares one backlog, urgent work gets buried.

Add fair-share limits across teams and repositories so one monorepo doesn’t eat the whole fleet. Then mix priority with ageing rules. A strict priority queue can leave lower-priority work waiting forever. Plain first-in-first-out has the opposite problem: an urgent release can get stuck behind a mountain of older test jobs.

Weighted shares plus an ageing rule work better. Jobs still have priority, but waiting time slowly increases a job’s effective priority. That keeps the queue moving without letting one class dominate forever.

Treat deployment queues as downstream gates, not as a generic backlog. If a queue stays long even when runners are free, the bottleneck is often queue wait, environment wait or execution delay. Extra runners only help with the first case. They won’t fix a serial dependency, a protected environment lock or a downstream system whose throughput is below the pipeline’s arrival rate.

That queue model should drive autoscaling thresholds, not the other way round.

Scale on queue signals, not guesswork

Don’t scale on CPU alone. Low CPU use doesn’t mean there’s spare capacity. The scheduler may still have no eligible runner for the jobs that are waiting.

Scale from queue signals instead:

• queue age
• queue length
• queue growth rate
• runner busy time
• job arrival rate

Autoscaling is a resource-allocation decision, not a CPU-efficiency exercise.

Set pool and account quotas, and alert on rejected scale-up operations and quota headroom, not just average utilisation. If scale-up fails because of limits, a tidy CPU chart won’t help you.

Keep some warm capacity for short feedback targets and protected deployments. Once capacity matches demand, the next wins usually come from test sharding, artefact promotion and job sizing.

Optimise execution, artefact flow and resource allocation

Once runner pools, queues and autoscaling are set up, the next gains come from how work moves inside those pools. Three things matter most: how much work you run in parallel, how artefacts move between stages, and whether each job gets the right amount of compute.

Increase safe parallelism and shard tests by duration

After runner supply is sorted, start with the critical path. Use the DAG to run only work that is genuinely independent in parallel. Keep jobs in serial order only where order matters: migrations, shared mutable test environments, integration dependencies, approval gates and production deploys. In practice, only a small number of steps need strict sequencing.

More parallelism only helps when it shortens the critical path. So put parallel effort into stages that sit on that path, not everywhere at once. [2]

File-count sharding is the easy starting point. But duration-based sharding usually does a better job of balancing long-running tests. Historical-duration sharding takes the longest tests and assigns them to the shortest shard first, using timings from the last several successful runs. Rework shard assignments after major test additions, framework changes or steady timing drift. And for tests with no history yet, use a fallback duration.

Sharded and matrixed suites also need separate handling for unstable tests. Put known flaky tests in a quarantine lane with their own retry budget. Retries must not bury the first failure - that first failure should still show up in the aggregated report.

Each shard should publish machine-readable results, logs, screenshots, coverage and diagnostics under names that won't clash. Then a small final aggregation job can merge everything, deduplicate retries, keep the first failure visible, and fail if any required shard failed or timed out.

Use matrix execution only when the same job actually needs checking across a bounded set of meaningful dimensions, such as supported Python versions, operating systems, database engines or CPU architectures. A smoke matrix is usually enough for pull requests. Run the full matrix on scheduled or release workflows. Before you add more cells, measure what each extra one finds against the queue delay and cost it adds.

Build once, promote the same artefact

Artefact discipline matters just as much as run speed. Build once, deploy many: use immutable artefacts, unique version IDs and externalised configuration. [9] Environment-specific configuration, credentials and endpoints should be injected at deployment time, not baked into separate builds. That way, the bytes tested in staging are the same bytes deployed to production. Rollback becomes dependable, and audit trails mean something.

Reference containers by their SHA-256 content digest instead of a mutable tag like latest. [3] When the software changes, produce a new build. Do not alter existing artefacts.

Caches stop you downloading the same things again and again. Artefacts, by contrast, are outputs produced by jobs and stored so later jobs can fetch them. [7][8]

Cache keys should include the OS, architecture, toolchain version and dependency lockfile. If a stale cache leads to a wrong build, that's both a reliability problem and a security problem. Cache correctness matters more than chasing a high hit rate. [5][6]

If network transfer is the bottleneck, store artefacts close to runners. Use encryption in transit and at rest, lock down publication and promotion permissions, and remove stale or unreferenced cache entries on their own.

Allocate CPU, memory and spend by job class

Once artefacts are stable, size each job class to fit the work it does. Right-sizing means matching resources to workload, not matching budget to guesswork. Base requests on observed p50 and p95 usage, not estimates, and set limits so one job cannot drain a whole runner.

Warm capacity fits pull-request validation and release promotion, where cold-start time takes a noticeable bite out of lead time. Ephemeral capacity works well for bursty or isolated workloads. Interruptible capacity makes sense for retry-safe scheduled work - nightly regression, compatibility matrices and non-blocking benchmarks - but not for a non-retryable migration or final release promotion. [4]

Every optimisation should tie back to a measurable outcome in £. If the time saved does not justify the extra spend, use a cheaper pool or lower concurrency.

Track at least:

• Queue time by runner pool and priority
• Job and critical-path duration
• Concurrency and runner utilisation
• Cache-hit ratio and cache-transfer time
• Shard imbalance, flake rate and retry volume
• Artefact-transfer time and storage volume
• Cost per pipeline, per successful deployment and per pull request in £
• Cost of idle warm capacity versus cold-start delay

Operate, tune and govern the pipeline

Use observability to find the real bottleneck

Start with the baseline metrics from the opening section and use them to find the slowest part of the system. A pipeline can look fine at job level while the actual delay sits in queueing, provisioning or approval gates. The point here is to diagnose the graph, queue and runner setup described earlier, not just the jobs themselves.

Split all measurements by repository, branch, workload class, runner pool, priority and stage. Then break end-to-end lead time into parts:

• arrival-to-queue time
• queue wait
• runner provisioning
• execution
• artefact transfer
• approval
• deployment intervals

Track each part on its own. Watch p50 and p95 for every segment, because averages smooth over the slow runs that people feel most. Your dashboards should also let you drill down from deployment to commit, pipeline, stage, job, runner, cache and artefact events. That makes it far easier to see whether the constraint sits in the graph, the queue, the runners or the artefacts.

The table below links common symptoms to likely causes and possible fixes. Treat it as a starting point, not a final checklist.

Alert on sustained trends, not one bad run. For example, if p95 queue wait breaches your service objective for several intervals in a row, that deserves attention. A single slow pipeline often does not.

Roll out changes in a practical sequence

Once you know where the bottleneck is, change one layer at a time. If you alter scheduling, caching and test topology all at once, you won't know what helped and what made things worse. Roll changes out by job class and pool, not across the whole platform in one sweep.

Begin with a baseline for throughput, lead time, reliability, resource use and cost by workload class. After that, split pools and queues where contention matters. Remove serial stages that do not need to exist. Add caching, or fix it, and check that it still behaves correctly. Apply duration-aware test sharding and watch for imbalance. Set rules for immutable artefacts. Add queue-aware autoscaling with warm capacity and spend caps. Then put dashboards, ownership and peak-demand load tests in place.

Before any broader release, verify the change under peak load. Use canary pools or roll out to a percentage of repositories for material changes. And always keep a rollback path. That one habit saves a lot of pain.

Conclusion: the design rules that keep throughput high

The right architecture is the one whose measured behaviour matches your workload, risk tolerance, latency objectives and budget. The loop is simple: measure → change → verify → govern. DORA's measures still work well as outcome indicators: deployment frequency and change lead time describe delivery throughput, while change-failure rate and failed-deployment recovery time describe delivery stability.[1]

Govern the system with the same care you use to design it. Give clear ownership to each runner pool, shared cache, artefact store and deployment environment. Keep pipeline definitions, queue policies, concurrency limits and retention rules in version control. Review performance and cost on a regular cadence, and keep a clear incident process for pipeline-wide failures. Pair every speed metric with a quality metric. A faster pipeline that skips controls or trims test coverage doesn't remove cost; it just pushes it downstream.

In practice, that means defining throughput with measurable flow and stability, modelling dependencies before adding workers, separating pools and queues by workload and trust boundary, scaling from queue signals, sharding by historical duration, promoting immutable artefacts, sizing resources to service objectives, and treating ownership, security, budgets and change control as part of the architecture itself.

FAQs