DevSecOps Pipeline Design: Shift-Left Validation | Hokstad Consulting

DevSecOps Pipeline Design: Shift-Left Validation

DevSecOps Pipeline Design: Shift-Left Validation

Most pipeline risk should be blocked before release, not during it. This guide shows how I would set up security checks across six linked stages: planning, pull request, CI, pre-production, release, and runtime, so teams can stop new high-risk issues early, keep shipping, and still keep a clear audit trail.

Here’s the core idea in plain terms:

  • Check early: run secrets scanning, SAST, dependency checks, and IaC scanning as soon as code changes
  • Block only when risk is high: leaked credentials, exploitable critical flaws, failed security tests, unsigned artefacts, and risky config should stop the line
  • Track the rest: lower-risk findings should go into tickets with owners, dates, and review
  • Keep proof: store scan results, approvals, SBOMs, signatures, provenance, and deployment records against the artefact digest
  • Use time-limited exceptions: every bypass needs an owner, reason, expiry date, and review
  • Match runtime to release rules: monitor drift, exposure, cert expiry, logging gaps, and rollback health after deployment

A few numbers stand out. Teams that shifted security earlier were 2x more likely to do well on software delivery measures, and teams that built security into development were 1.6x more likely to meet or beat business goals. At the same time, the 2024 DORA report linked more AI use with a 7.2% drop in delivery stability in the cases studied. So the message is simple: more automation does not remove the need for gates.

If I had to boil the whole article down to one short checklist, it would be this:

  • define stage owners and pass/fail rules early
  • protect branches and review pipeline code like production code
  • sign artefacts and promote the same immutable digest across environments
  • verify policy, approvals, SBOM, signature, and provenance before release
  • keep some controls non-bypassable
  • feed production findings back into backlog and test rules

In short: shift-left works when checks are early, gates are risk-based, and every exception leaves a paper trail.

::: @figure DevSecOps Shift-Left Pipeline: 6-Stage Security Validation Framework{DevSecOps Shift-Left Pipeline: 6-Stage Security Validation Framework} :::

Shift Left DevSecOps Pipeline on AWS (Jenkins + Trivy)

Define the workflow and validation model

Using the six-stage model from the introduction, map each control to the earliest stage where it can be checked with confidence: planning, design, coding, pull request, build, artefact promotion, pre-production, deployment and runtime. Then turn those stages into named owners, gates and retained evidence.

Map pipeline stages to security objectives and evidence

For each stage, define six things: the security objective, the automated check, the trigger, the blocking threshold, the evidence retained and the team in charge. That makes ownership clear and shows exactly where a gate applies.

Stage Security objective Check Trigger Blocking threshold Evidence retained Responsible team
Planning Identify threats and compliance needs Risk classification and threat model High-risk change raised No approved design for a high-risk change Threat model, decisions, approvals Security specialists, leads
Design Prevent insecure architecture and trust-boundary mistakes Architecture review, authentication and authorisation review, data-flow analysis High-risk change entering design No approved architecture or design review Design record, review outcome Security specialists, architects
Pull request Prevent defects and credential exposure Secret scanning, linting, SAST, unit tests, dependency checks Every PR opened or updated Confirmed secret, failed required test, new critical finding Immutable job log, scan results, reviewer approvals Developers, platform engineers
Build Reduce dependency and image risk SCA, SBOM generation, container and IaC scanning Every merge to a protected branch New critical vulnerability, policy violation, prohibited image Scan reports, SBOM, policy decision Platform engineers
Release Prove artefact integrity Signing, provenance recording, promotion checks Promotion to release candidate Unsigned artefact, missing required evidence Signature, digest, provenance, approval record Platform engineers, security specialists
Pre-production Validate behaviour in a realistic environment DAST, integration tests, configuration checks, approval for high-risk changes Deployment to pre-production Security regression or failed approval Test reports, deployment record, approval Operations, security specialists
Deployment Enforce environment controls Admission, configuration and compliance policies Every deployment event Privileged workload or insecure configuration Deployment record, policy decision, approval chain Operations, platform engineers
Runtime Detect residual risk Monitoring, vulnerability management and drift detection Continuous Severity- and exposure-based incident threshold Alert, ticket, remediation and rollback evidence Operations, security specialists

The main split here is between blocking checks and asynchronous checks. Secret scanning, unit tests, linting and lightweight SAST should finish within the pull-request feedback window. Full DAST, broad compliance reviews and full infrastructure scans can run in parallel against a preview environment.

Asynchronous does not mean optional. Every result still needs an owner, a severity rating and a clear route to resolution. That line helps teams decide which checks must block at once and which can run alongside delivery without being ignored.

Set security ownership and acceptance criteria early

Set ownership by stage before development begins.

  • Developers own secure code, fixes and security design notes.
  • Platform engineers own pipeline controls, secrets, identity and evidence retention.
  • Operations own deployment safety, monitoring and rollback.
  • Security specialists own standards, high-risk exceptions and detection tuning.

Define acceptance criteria in the ticket or architecture decision record. Cover authentication, authorisation and least privilege; input validation and output encoding; protection of personal or sensitive data in transit, at rest and in logs; approved dependency sources and version constraints; container and base-image requirements; infrastructure configuration; audit logging; and recovery or rollback behaviour.

For internet-facing services, services handling sensitive data, identity or payment components, major architectural changes and changes that materially expand privilege, require a documented threat model before implementation starts.

Record existing risk before turning on blocking gates

Capture the current state before you switch on new blocking gates. If you apply hard blocks to an existing codebase without preparation, builds can fail at once and teams may feel pushed to work around the controls.

Start by exporting current findings. Then remove false positives, classify issues by severity and exploitability, assign owners and create tracked remediation tickets with due dates.

After that, configure gates to block only new or worsened findings at first. Apply hard stops straight away to confirmed critical vulnerabilities with a believable exploitation path, exposed active credentials, failed security-critical tests and unsigned artefacts. Lower-severity historic findings can sit under an agreed remediation deadline, with escalation if deadlines slip.

The gate should show both new risk and total residual risk. Otherwise, the backlog fades into the background while delivery moves on.

Separate baseline controls from maturity improvements

Roll out controls in stages so delivery does not grind to a halt.

Baseline controls are the minimum needed for a working DevSecOps workflow: protected main and release branches, mandatory peer review, secret scanning, SAST, software composition analysis, IaC and container configuration checks, unit and integration tests, signed release artefacts, deployment approval for defined risk classes, and vulnerability ownership with remediation tracking. OWASP identifies these as foundational pipeline capabilities.[1]

Maturity improvements sit on top once that baseline is steady: signed provenance attestations for every build, policy-as-code across more repositories and environments, reproducible or hermetic builds, centralised evidence dashboards, admission controls, IAST and targeted DAST, automated dependency remediation, and automated remediation or rollback. OWASP describes higher-assurance policy enforcement through admission controls, authorisation chains, centralised metrics and recurring policy review.[2] Its environment-hardening guidance also separates baseline assessment from continuous drift detection and automated remediation.[4]

Use this baseline to define branch protection and pull-request checks next.

Enforce branch rules and pull request controls

With stages and owners in place, the next step is to control how code gets into protected branches. This is where policy stops being a document and starts being something the repository enforces at merge time.

Protect branches, reviews and repository permissions

Start with the default and release branches. Require a pull request for every merge, turn off direct pushes and force-pushes, and block branch deletion except through a separately authorised break-glass process. A merge should be allowed only when all status checks pass on the latest commit. [8]

For most repositories, one independent approval is enough. But some repos carry far more risk. If a repository includes production infrastructure, authentication code or cryptography, require extra approval from a named security or platform owner. CODEOWNERS rules help here by sending the right changes to the right people without relying on memory. Updates to .github/workflows/, Dockerfiles, Terraform definitions, authentication modules, dependency manifests and security policies should always go to the team that owns that area.

Access should follow least privilege for both people and machines. CI jobs should use short-lived credentials scoped to the exact task in front of them. Build, test, publish and deploy should each use separate identities, not one shared token with broad access. Turn on repository and organisation audit logs, then send high-value events such as permission changes, ruleset changes and merge records to your central monitoring system. [6]

Once you control entry, you can control the risk that gets through.

Run pull request checks that block real risk

The rule here is simple: block what creates immediate danger, track the rest.

A merge should be blocked when a check finds an exposed credential or private key, a reachable critical vulnerability, a failing security regression test, a severe infrastructure misconfiguration tied to public exposure or privilege escalation, or a policy breach such as an unauthorised production change. [1] Other findings should still be handled, but not every alert needs to stop delivery. Lower-confidence SAST alerts, non-exploitable dependency findings and accepted licence exceptions should create a tracked ticket with an owner, severity rating and remediation deadline instead of blocking the merge.

For speed, run cheap deterministic checks on every commit push:

  • linting
  • unit tests
  • secret scanning
  • changed-file SAST
  • dependency-diff analysis

Run broader checks in parallel so developers are not left waiting on a single long pipeline. That includes full SCA, IaC scanning, licence compliance and container checks. Keep the output tight and useful. Developers should see the file, rule, severity, fix and suppression path straight away, so they can deal with the issue instead of hunting for it.

Control pipeline code, dependency updates and emergency changes

CI workflow files, reusable actions, build images and policy-as-code are production-impacting code. Treat them that way. Require pull requests and specialist review for changes to these files, pin third-party actions to immutable digests instead of mutable version tags, and scan them for vulnerabilities and unexpected permission changes. [7] Untrusted pull-request code must not get access to production secrets, especially in public repositories or forks. Run those jobs without network or secret access, or require maintainer approval before they run. [9][7]

The table below shows the core checks, where they run and what evidence they keep.

Check type Execution point Typical blocking condition Owner Retained evidence
Secret scanning Every commit and pull request Confirmed credential, private key or token Developer and security team Finding, commit SHA, revocation record and resolution
SAST Changed files on push; full scan in CI Confirmed critical or high-confidence exploitable issue Development team Tool version, rule, report, triage decision
SCA and dependency review PR dependency diff and CI Introduced critical vulnerability or prohibited package or source Development and platform teams Lockfile diff, vulnerability report, exception ticket
IaC and policy scanning Pull request before merge Public exposure, excessive privilege or policy breach Platform/cloud owner Plan, scan output, policy decision, approval
Licence compliance Pull request and release build Prohibited licence where policy requires blocking Legal or compliance owner Dependency licence report and decision
Linting and unit tests Every commit push Broken code, unsafe pattern or failed security regression test Developer Logs, test report, commit SHA
Workflow and action validation Changes to CI or policy files Unpinned action, excessive token permission or unauthorised workflow behaviour Platform/security owner Diff, review, scan report, approval

Those same rules then move forward into build and release validation.

For dependency updates, let automated patch-level pull requests open and merge when all checks pass, the dependency comes from an approved source, the lockfile changes match expectations and no vulnerability threshold is crossed. Human review should still be required for major-version updates, cryptographic libraries, authentication components and any dependency with privileged runtime access.

Emergency fixes need a process set out before the pressure hits. Allow a break-glass change only for a documented incident. An authorised approver should record the reason and risk assessment, the bypass should apply only to the named actor and branch, and it should be turned off as soon as it has been used. Run the full check suite afterwards and close the exception within a defined period. The normal branch rules stay as the default. An emergency bypass is an auditable exception with a clear expiry, so teams can move fast when they have to without weakening the default rule set. These branch rules feed the build and release gates that follow.

Build, test and gate the release artefact

Merged code should become a signed artefact with evidence attached, not just a green build. And that evidence chain has to hold up during the build too.

Run layered CI checks on code, dependencies, containers and IaC

Start by compiling the code. After that, run tests and scans in parallel, then sign the artefact.

Security testing should cover:

  • authentication checks for invalid credentials, token expiry and session handling
  • authorisation checks to confirm one user cannot access another role’s resources
  • input validation for injection attempts and malformed payloads
  • regression tests for issues that were fixed before

Once the image or package is built, scan the built artefact itself for insecure code, weak dependencies, image flaws and IaC misconfigurations.

Generate the SBOM from the built artefact, not earlier in the process. Build-time dependency resolution can change what actually ships. NIST SP 800-204D recommends collecting and safeguarding provenance data for each software release, including information represented in an SBOM.[3]

Gate releases using risk thresholds and artefact integrity

A release should only be blocked for issues that put the shipped artefact at direct risk. That includes exploitable critical flaws in reachable code, failed mandatory security tests, leaked secrets, prohibited licences, and artefacts that are unsigned or appear tampered with.

Lower-severity findings are different. If there is no known exploit path, if the issue is an accepted false positive, or if other controls already cover the risk, the build should not stop. Instead, create a tracked ticket and move it through the normal risk process.

Build in an isolated, ephemeral environment from pinned source revisions and locked dependencies. Sign the artefact and its provenance only after every required check passes. Then verify both again at promotion time.[10][11]

Once the artefact is signed, the next step is to check whether the deployed service behaves the way it should.

Keep delivery moving with parallel checks and clear evidence

Parallel checks shorten feedback loops, but they do not remove the need for final verification.

After checkout, separate jobs can begin at the same time: unit tests, SAST, dependency analysis, IaC scanning and image preparation. Cache package manager downloads and scanner databases with cache keys tied to the lockfile, tool version and base-image digest. That keeps builds fast without making cache reuse sloppy.

For pull requests, use incremental scans to keep turnaround times down. Before the release candidate is finalised, run a full scan.

Store the commit ID, build definition, lockfiles, SBOM, scan results, test reports, digest, signature, provenance, approvals and exceptions against the artefact digest in an access-controlled, tamper-evident location. A rebuilt or substituted package must not be able to inherit the original approval.[3][5]

Security gate decision table

Control Evidence Blocking threshold Permitted exception Escalation owner
SAST Versioned scan report and finding status Critical exploitable defect in reachable release code Time-limited compensating control with risk acceptance Engineering and security owner
SCA and SBOM SBOM, dependency report and vulnerability match Critical exploitable dependency or prohibited component Pinned mitigation, vendor advisory or documented non-reachability Product risk owner
Container scan Image digest and scan report Critical vulnerability in the shipped image or unauthorised base image Time-limited base-image exception Platform security owner
Terraform and Helm Versioned IaC scan report and policy result Exposed secret, excessive privilege or unsafe production setting Reviewed, time-limited exception Cloud or platform owner
Tests Unit, integration, API, authentication and authorisation reports Any required security test fails Only with explicit risk acceptance and replacement validation Service owner
Artefact integrity Signature, digest and provenance attestation Missing, invalid or signature that cannot be verified No routine exception for production Release security owner
Approvals and policy Approval record and policy decision Required reviewer or segregation-of-duties check missing Emergency procedure only Change authority

Define artefact promotion rules across environments

Promote the same immutable digest through test, staging and production. Do not rebuild for each environment.

Why? Because a separate build creates a different supply-chain risk and breaks the chain of custody between what was tested and what was deployed. Environment-specific settings should be added at deployment time through a managed configuration or secret store, never baked into the package.

Promotion should require successful automated gates, a verified SBOM and provenance, a valid signature, and approvals that match the target environment’s risk level. Record the full link between the commit, build ID, package digest, deployment manifest, environment, approver and deployment timestamp.

That chain of custody then carries straight into deployment validation.

Validate deployment, manage exceptions and preserve flow

Validate deployed services before and after release

Check the same release evidence at the latest safe point, then check it again at deployment. Before anything reaches staging or production, verify IaC policy status, environment configuration, image digest, SBOM, signature, provenance and the approved change record. Then confirm that the deployed digest matches the digest that was tested. You should also check secrets, certificates, public exposure and the required logging setup. That same evidence should then drive canary and post-release checks.

In staging, run smoke, regression, DAST, API security, container, host, compliance and configuration checks. After a canary release, run health, database, migration, smoke and API auth/authz checks, then enforce pre-set pause or rollback thresholds. Once rollout begins, those same controls should continue into runtime drift detection.

Runtime monitoring needs to do more than scan for vulnerabilities. It should also watch for vulnerability exposure, dependency and image updates, unexpected configuration, IAM or firewall changes, privileged access, suspicious API activity, certificate expiry, logging coverage and service health. Tie drift detection back to earlier gates by comparing production behaviour with the approved artefact, policy and configuration baseline. When drift appears, an actionable remediation item should open automatically. Certificate expiry, broken rollback and lost telemetry should be treated as release-readiness failures, not routine operational noise.

Use temporary exceptions and emergency rules with audit trails

Every exception needs a time-limited audit trail. The record should include the finding or control ID, affected asset and environment, business and technical justification, exploitability and impact assessment, compensating controls, named approvers, remediation owner, expiry date and the full audit trail. Exceptions should expire automatically - after 14 or 30 days, for example - and reopen the finding unless a new approval is recorded. Use that record to decide whether the release should proceed, bypass, or roll back.

The table below shows what each release path needs and which controls must stay active at all times.

Exception and recovery matrix

Path Approvals Non-bypassable controls Evidence required Review deadline
Standard release Normal code-owner, service-owner and change approvals Secrets detection, signed artefact and digest verification, deployment authorisation, required security gates Test reports, scan results, SBOM, provenance, approvals, deployment record and monitoring confirmation Routine post-release review
Approved exception Named risk owner plus security or designated approver Secrets detection, artefact integrity, identity and deployment authorisation remain active Exception record, risk assessment, compensating controls, affected asset, expiry date and remediation ticket At expiry or within 30 days, whichever is earlier
Emergency release Incident commander or duty approver, with retrospective security approval Secrets detection, artefact integrity, least-privilege deployment and destination verification Incident ID, reason, approver, commands or pipeline run, artefact digest, tests performed and rollback plan Retrospective review within one business day
Rollback path Service owner or incident commander under the incident procedure Verified rollback artefact, deployment authorisation, secrets protection and environment checks Previous known-good version, rollback trigger, health evidence, user impact and follow-up actions Incident review within five business days

Keep secrets detection, artefact integrity, deployment authorisation and identity controls non-bypassable. Only low-criticality advisory scans should be allowed to fail open, and only under approved compensating controls.

Feed runtime findings back into the backlog

Each production finding should go into a central backlog with severity, affected service, asset owner, detection source, exploitability, due date and remediation status. Critical findings should trigger incident response. Repeated lower-risk findings should turn into engineering improvements, such as a new regression test, an updated policy rule, a secure default or better scanner configuration.

After a failed deployment or incident, run a post-incident review that covers detection time, alert quality, decision points, containment, rollback effectiveness and control gaps. Failed deployments should feed back into the same gates, tests and exceptions used to release the change. Track actionable-alert rate, false positives, acknowledgement time, containment time, remediation time and recurrence. If a rule is noisy, tune it with measured data, but require approval and regression testing before weakening any blocking policy. NIST identifies mean time to identify, rectify and recover as useful indicators for refining DevSecOps processes.[14] Retain alerts, tickets, approvals, exception history, deployment logs, scan outputs and remediation records for the period your organisation's contractual, regulatory and audit obligations require. Each backlog item should feed the next control change, test or policy update.

Conclusion: Measure flow, tune controls and extend governance where needed

Shift-left validation works best when security checks sit at the earliest useful stage, ownership is clear, exceptions are temporary and evidence is retained from end to end. Use the checklist below to confirm the pipeline works across the full release path.

  • Production promotion uses the same verified, signed artefact tested in staging.
  • Canary or progressive deployment, health checks and tested rollback are available.
  • Runtime vulnerability, drift, certificate, logging and attack-surface monitoring are active.
  • Critical controls fail closed and cannot be bypassed through an ordinary exception.
  • Each exception records the finding, asset, justification, risk, compensating controls, approver, owner, expiry and audit trail.
  • Runtime findings create owned backlog items with severity-based remediation targets.
  • Pipeline evidence is retained for the organisation's audit and regulatory requirements.

If your delivery scope includes AI-assisted code, AI agents or AI-enabled products, the same governance model applies to prompts, model and system configuration, tool permissions, sensitive-data handling, output validation and human review of production-impacting changes. OWASP's DevSecOps guidance explicitly includes AI/LLM security and AI governance among relevant pipeline controls.[1] The 2024 DORA report found that increased AI adoption was associated with an estimated 7.2% reduction in delivery stability in the environments studied[12][13] - a useful reminder that automation and AI assistance do not remove the need for release safeguards, small batches and explicit production controls.

For organisations that need support designing secure AI delivery workflows, automation or DevOps transformation, Hokstad Consulting specialises in AI strategy, implementation, automation and DevOps transformation.

FAQs

How do we start shift-left without slowing delivery?

Start with a phased approach instead of trying to change everything at once. Begin with quick checks that have a big payoff, like linting and basic security scans, and keep them short enough to finish in 2 to 3 minutes.

For new policies, use advisory mode first. That way, developers can see warnings without having their builds fail straight away. It gives people room to fix issues without turning the pipeline into a bottleneck.

On pull requests, keep scans light. Save the heavier test runs for later stages, where they won’t slow down day-to-day work as much. It also helps to use caching or incremental scanning so you only check code that’s actually changed.

Which security checks should block a release?

Block releases when automated security gates find high or critical vulnerabilities. That includes critical CVEs, missing SBOMs, unapproved container configurations, IaC misconfigurations, or unauthorised access patterns.

If something urgent comes up, allow only time-limited, documented waivers with formal approval. That way, any exception stays auditable.

How should we handle security exceptions safely?

Use a formal, controlled waiver process instead of sidestepping security gates. Any time-bound waiver or change ticket should need clear sign-off from security or compliance teams, along with a plain-English reason for the exception.

Track every waiver in version control, such as Git, so there’s a full audit trail. Files like .trivyignore or security-waivers.yaml should spell out the justification and include a mandatory expiry date.

Need help with your DevOps, cloud or AI plans?

Hokstad Consulting helps companies with DevOps transformation, cloud architecture and hands-on AI development — pragmatic consulting with measurable results.

Our services: DevOps on Retainer · Hosting & Cloud · AI Development & Strategy