Skip to content
CAMPUX Cloud Bootcamp Phase Four · Class Thirty-Nine
Phase Four — Operate, Secure & AI
Reading 15 min · Drills 4 · Part VII of IX
Class Thirty-Nine · Pipelines
Class Thirty-Nine · Part VII

Ship it dark, release it slowly

Deploying code and releasing a feature are two different acts with two different risks; this part gives you the strategies that control how new code reaches users and the feature flags that let it arrive switched off — so the two can fail, and be undone, independently.

§1

Two different acts, wearing one word

A deploy and a release feel like a single event — you push a button and something new is live — but they are two acts with two clocks and two blast radii, and confusing them is how a Tuesday afternoon becomes an outage. Deploying code is moving a new build onto the machines that run it. Releasing a feature is the moment a user can actually reach the new behaviour. For most of software's history these happened at the same instant, because the only way to expose new behaviour was to put the code that produced it into production — so the deploy was the release, and every deploy was therefore a bet on a large number of untested users hitting new code at once.1

This part is about pulling those two acts apart so they can fail — and be undone — independently. There are two levers. Deployment strategies control how the new code reaches the infrastructure: all at once, a slice at a time, or machine by machine, each buying you a different rollback story for a different price. Feature flags go further and separate the acts entirely: the code ships to production switched off — dark — and the feature is released later, to whom you choose, by flipping a value in a central store, with no deploy at all. Together they turn "release" from a leap you take once, holding your breath, into a dial you can turn up slowly and back down in seconds. That dial is the difference between an engineer a team lets near production and one it does not.

Shipping code is not releasing a feature.

§2

Four ways to let new code reach users

Every deployment strategy is an answer to one question: when the new version is wrong — and one day it will be — how fast, and how cheaply, can you take it back? A strategy that exposes all users at once has the simplest mechanics and the worst answer; a strategy that exposes a slice first has more moving parts and a far better one. There is no free option. You are choosing where to spend complexity so that a bad release costs minutes and a handful of users instead of an hour and all of them. The four below are the ones worth knowing by name, ranked roughly from blunt to surgical.

Table 1 — Four release strategies, and what each buys
StrategyHow it worksRollback storyCost & complexity
Blue-green Two identical production environments — one live (blue), one idle (green). Deploy the new version to green, test it in isolation, then switch all traffic to green at once. Instant and total: switch traffic back to blue, which is untouched and still running the old version. Rollback is a routing change, not a redeploy. High. You pay to run two full environments and must keep data and state compatible across the switch.
Canary Release the new version to a small slice of traffic — say 5% — while everyone else stays on the old one. Watch the signals; expand in increments if healthy, abandon if not. Good and cheap: a bad canary is pulled after harming a fraction of users, before the increment that would have exposed the rest. Medium. Needs traffic splitting and, to be worth anything, real monitoring to decide expand-or-abort.
Rolling Replace instances in batches — a few machines at a time — until the whole fleet runs the new version, keeping enough of the old serving traffic to stay up throughout. Partial and slower: halt the roll on a bad batch, but instances already updated must be rolled back one batch at a time. Low. No second environment; it is the default way most orchestrators replace instances.
Ring-based Concentric rings of users, widening outward: internal staff first, then early adopters, then everyone. Each ring must look healthy before the next opens — Microsoft's own progressive-exposure model. Contained by design: a fault caught in an inner ring never reaches the outer ones, and you halt the advance rather than undo a global release. Medium to high. Needs a way to map users to rings and the patience to let each ring bake.

Two of these are named strategies you configure; two are patterns you assemble. Blue-green and ring-based are shapes of intent — Azure gives you the pieces (deployment slots, traffic rules, ring definitions) and you compose them. Canary and rolling, as the next section shows, are keywords the Azure DevOps deployment job understands directly. Canary is the one to picture, because it is the one an interviewer will ask you to draw.

Fig. 1 · A canary release — a slice first, then a decision
A canary release routes a small slice of traffic to the new version, watches its signals, and either promotes it to full traffic or rolls it back. traffic v1 · current 95% of traffic v2 · canary 5% of traffic watch errors · latency healthy → promote 100% bad → roll back the slice is the whole point — a small, reversible bet
Draw this once by hand. The whole idea is the small red slice: the new version carries only 5% of traffic until the monitor says it is safe, so a bad release harms a fraction of users for minutes, not everyone for an hour. The decision to expand or abort is the strategy — the split is just plumbing.
§3

What the deployment job actually supports

In an Azure DevOps YAML pipeline, you request a strategy inside a deployment job — the special job type from Part F that records deployment history against an environment. The strategy: keyword accepts exactly three values: runOnce, rolling, and canary. Each one runs your steps through a fixed set of lifecycle hooks — preDeploy, deploy, routeTraffic, postRouteTraffic, and an on: failure / on: success pair — so the strategy decides how many times and against how much of the fleet those hooks fire.

runOnce is the plain one: every hook runs once, then success or failure. rolling replaces targets in batches sized by maxParallel — and, importantly, Azure DevOps currently supports the rolling strategy only against virtual-machine resources, not arbitrary services. canary takes an increments: list — for example [10, 20] — and iterates the deploy-and-watch hooks once per increment before promoting to the remainder, most naturally against Kubernetes or AKS. Notice what is not on that list: there is no blueGreen and no ring keyword. Blue-green on Azure is something you build yourself — most commonly with App Service deployment slots, deploying to a staging slot and then swapping it with production. Ring-based release is a practice, not a YAML value: you express it as a sequence of environments or user cohorts, each gated, following Microsoft's safe-deployment guidance. Knowing which of the four is a keyword and which is a pattern you assemble is exactly the distinction that separates someone who has read the docs from someone who has only heard the words.

The lifecycle hooks are where the strategy earns its keep. postRouteTraffic is the interesting one: it is the window where you monitor the version you just exposed — for the defined interval, before the next increment — and your on: failure hook is where the rollback lives. A canary strategy with an empty postRouteTraffic is not a canary; it is a slow big-bang that happens to deploy in increments while watching nothing.

§4

Feature flags: the release switch that isn't a deploy

Deployment strategies still couple the two acts loosely — the feature goes live as the code arrives, just to fewer people at a time. A feature flag severs them completely. The new code ships to production wrapped in a conditional that is switched off by default, so it sits there, deployed but dormant — a dark deployment — reaching no user until you decide otherwise. Releasing the feature is then a separate act: you flip the flag's value, and because the value lives outside the application, nothing rebuilds and nothing redeploys.

Feature flag
A named variable, typically boolean, that gates a block of code at runtime — the block runs only when the flag is on. Because the flag's value is read from an external store rather than compiled in, you can turn a feature on or off without touching, rebuilding, or redeploying the application.

On Azure, the store is Azure App Configuration and its feature management capability, read through a client-side feature manager library. Flags and their current states live centrally; the app asks the feature manager whether a flag is on, and the feature manager answers from App Configuration — optionally through filters that decide per request, so a flag can be on for 5% of users, or only for internal accounts, or only in one region. That single indirection is what buys you four things at once. It enables trunk-based development: unfinished work merges to main behind an off flag instead of rotting on a long-lived branch. It gives you an instant kill switch: a feature misbehaving in production is turned off in seconds, without the rebuild-and-redeploy that an incident least has time for. It supports gradual rollout, widening a flag's audience on a dial. And it makes A/B testing a configuration change: show variant A to half your users, B to the other half, and measure — no deploy per experiment.

The payoff compounds with the strategies above. A canary release controls which servers see the new code; a feature flag controls which users see the new behaviour — and the flag can be flipped off the instant a signal turns bad, faster than any deploy-based rollback could ever reverse it. The two are not rivals; the safest releases use both, so that a bad change can be pulled back at the infrastructure layer or the feature layer, whichever notices first.

§5

Pricing the peak while it is still calm

The value of everything in this part is that it is done before the risk arrives, not during it. A canary you wire up mid-incident is not a canary; it is a panic. A feature flag you add while the site is down is a redeploy you did not have time for. The whole discipline is front-loaded: you spend the calm week building the slice, the monitor, and the switch, so that when the dangerous change ships you already hold the two things an incident cannot manufacture — a small blast radius and a fast undo. Campux is about to learn this on the one date it cannot afford to learn it the hard way.

Case File · Campux Retail

A new checkout, shipped dark, six weeks before November

The storefront's busiest date is coming — and it wants a rewritten checkout

Campux's November traffic peak — the single date the storefront earns its year on — is six weeks out, and the product team wants the rewritten checkout live for it. Shipping a rewritten payment path into the busiest hour of the year is exactly the bet this part exists to unmake. So the reader does not ship it as a release; the reader ships it dark. The new checkout is deployed to production weeks early, behind an Azure App Configuration feature flag that is off for everyone, riding the same gated staging-to-production flow Part F built — proven in place, carrying no traffic, waiting.

Then the release, decoupled from the deploy, happens on the reader's schedule and not the calendar's. A week before the peak, on a calm afternoon, the flag opens the new checkout to 5% of traffic — a canary in users rather than servers — while App Insights watches conversion, errors, and latency. The numbers hold; the slice widens to 20%, then to everyone, each step a value change and not a deploy. And the guarantee that lets the reader sleep is the same value in reverse: if the new checkout stumbles at any point during the peak itself, the flag flips off in seconds and every shopper is back on the proven path, with no redeploy, no rollback pipeline, and no war room. The risky change was priced while nothing was on fire — which is the only time you can afford it.

That is the whole method of this part in one move: the deploy happened weeks early and quietly; the release happened slowly and reversibly; and the two were never the same event. An engineer who can say that sentence in an interview — "I ship the code dark and release the feature on a dial" — is describing the difference between a team that survives its own busiest day and one that gathers afterward to write the postmortem.

On the job

The person who turns the dial

You · Cloud Engineer · owns how change reaches users

When the risky release comes, the team looks to you for the answer to one question: how do we ship this without betting the day on it? You ship the code dark, put it behind a flag, open it to a slice, watch the signals, and widen or kill it on a dial — deploy weeks early, release on your schedule, undo in seconds. That is not a tool you learned; it is the judgement a team trusts with its busiest hour.

Class 39g

Examination

Four drills, then two situations. These test whether you can hold the deploy and the release apart under pressure — the one distinction this part is built on. The situations have no marking scheme — write your answer before you reveal the reasoning, or the exercise is worthless. Nothing is stored.

Drill 01Recall
The rewritten checkout is deployed to production, but no user can reach it because it sits behind a feature flag that is switched off. What has shipping it this way actually bought you?
Marked

B — you have split the deploy from the release. The code is already on the production machines, exercised by the real environment, but reaching no customer; the risky, time-pressured act — putting new behaviour in front of users — is now a value change you make later, to whom you choose, without a rebuild. A misses the entire point: dark code is not wasted, it is de-risked, because the deploy is proven while calm and the release is decoupled from it. C is dangerous nonsense — an off feature is untested in production, not incapable of failing; the moment you flip it on, it can. D invents behaviour that does not exist and would defeat the purpose: the release is a decision, not an automatic side effect of the next deploy.

Drill 02Recall
In an Azure DevOps YAML deployment job, which exact set of values does the strategy: keyword accept?
Marked

B — runOnce, rolling, canary. Nothing else. This is the line that separates having read the docs from having heard the words. Blue-green and ring-based are real strategies, but they are not strategy: keywords — you compose them yourself (blue-green most often with App Service deployment slots; rings as a gated sequence of environments or cohorts). So A and C, which offer blueGreen or ring, name things the YAML schema will reject at parse time. D's featureFlag confuses two layers entirely: feature flags live in your application and Azure App Configuration, not in the pipeline's deployment strategy. Reach for a keyword that does not exist in front of an interviewer and the credibility cost is immediate.

Drill 03Select three
You inherit Basecamp's estate of thirty-one pipelines in week one. Which three actions match the posture this track teaches?
Marked

New pipelines born as code, riskiest credential first, existing ones convert when touched. Those three are the whole posture: stop the hole deepening, de-risk the estate where it is most dangerous, and let effort land only where change is already happening. The two rejects are the two crusades the track names. Rewriting thirty-one working pipelines in a quarter is a project with real regression risk, zero feature output, and a success state invisible to everyone outside engineering — the exact shape of initiative that dies at week eight, half-migrated, worse than either end state. And leaving the estate as scenery lets the click-drift compound and surrenders the review seat at the moment fresh eyes are worth the most. Neither crusade nor surrender; the measured middle you can state in three sentences.

Drill 04Spot the error
A new engineer sketches how they will approach Basecamp's estate. One line inverts the order this track is built on. Which?
# my plan for the inherited pipelines
1.  Learn to read one pipeline end to end before I
    change any of them.
2.  Lock down pipeline security first — federate every
    service connection this week — then worry about
    whether the pipelines even have tests or gates.
3.  Add the missing test gate to the warehouse pipeline
    when I next touch it for a real change.
4.  Give the November storefront peak a canary and a
    feature flag before it arrives, not during it.
Marked

Line two — and the tell is the word "then". Federating credentials first is genuinely right and matches the posture (riskiest credential first). The error is the second half: treating a locked-down credential as if it made the tests and approval gates optional or secondary. A pipeline with a perfect federated identity and no test gate still ships broken code to production the moment someone merges it — the credential controls who can deploy, not whether what they deploy is any good. Security, quality gates, and approvals are three independent controls that sit beside each other; none substitutes for another, and "worry about that later" is how a pipeline ends up impeccably authenticated and completely untrustworthy. The other lines are the plan at its best: read before you change (A inverts it), add the gate when you touch the pipeline (C is §4's convert-when-touched), and price resilience before the peak, not during it (D inverts it — the whole point of 39g is doing it while calm).

Situation 01Write before you reveal
Week two at the new job. The estate: thirty-one inherited pipelines, all working, some clicked together in a portal. Your manager, fresh from a conference: "Should we just migrate everything to the modern way this quarter? I can get budget." What do you advise?
Budget for a rewrite of working machinery is also budget not spent on something. What does the estate actually need first — and what converts on its own schedule?
Reasoning

The trap is answering the question as asked, because the option it offers is a crusade. Thirty-one working pipelines rewritten in a quarter is a project with real regression risk, zero feature output, and a success state invisible to everyone outside engineering — the exact shape of initiative that gets cancelled at week eight with a third of the estate half-migrated, worse than either end state. Decline it politely — but do not waste the manager's energy, because a manager offering budget for pipeline hygiene is rare weather. Redirect it.

Give them the §4 posture with a risk-ranked first target. New pipelines born as code, from today, as policy — free, immediate, and it stops the hole deepening. Existing pipelines convert when touched, so effort lands only where change already happens. And the budget goes to the one thing worth doing proactively: the credential audit — enumerate every service connection, find the stored, subscription-scoped ones (there will be at least one, used by everything), and replace them with federated, resource-group-scoped connections. That work is invisible in a demo and priceless in an incident, it de-risks all thirty-one pipelines at once without rewriting any of them, and it produces a number for the manager's slide: "standing credentials in the deploy path: was 6, now 0."

Close by giving the conference its due, so the manager keeps their win. The instinct is right — the estate should trend toward pipelines-as-code; the correction is sequencing, not direction. Offer the milestone version: this quarter, the audit plus the top three riskiest pipelines converted; next quarter, reassess with data. The sentence for the meeting: the pipelines aren't the risk — the credentials under them are, and fixing credentials doesn't require touching a single pipeline that works.

Situation 02Write before you reveal
An interviewer says: "Walk me through what happens, end to end, when a developer on your team merges a change — how does it get to production safely?" You have two minutes. What do you say?
They are testing whether you have a structure or will improvise. Walk the five boxes of Figure 1, and name the gate before production out loud.
Reasoning

Name the shape before the steps, so they know you are not improvising. "It travels a pipeline, and the pipeline is always the same shape." Then walk it: the merge is the trigger; the build turns the source into an artifact once, so the exact thing tested is the exact thing shipped; the tests run and the pipeline refuses to proceed if they fail — that is the whole point, the machine says no so a tired human doesn't have to. The tested artifact is published, and deployed to staging automatically.

Then put the human gate where it belongs and mean it. "Production is a separate environment with an approval — a named person confirms, so no single merge silently reaches customers at 4pm on a Friday. The deploy itself is a strategy, not a leap: I'd roll it out to a slice first — a canary — watch the signals, and roll back in a command if they're bad. And nothing in that pipeline holds a stored secret; it authenticates to Azure with a federated identity scoped to just the resource group it deploys to." Five boxes, one gate, one safe rollback, one scoped credential.

Close on what the shape reveals about you. "The reason I answer it as a shape rather than a tool is that it's the same shape on Azure DevOps or GitHub Actions — I've built it on both. What's portable isn't the YAML dialect; it's the decisions: test before ship, a human before production, a credential that's scoped rather than trusted, a rollback that's one command." That reframes you from "person who knows one tool" to "engineer who owns delivery" — and it is simply true, which is why it survives the follow-up questions. That, one final time, is the method of this whole bootcamp: the gap was never the missing noun; it was the proof, and now you carry it.

Examination record · first attempt
0/4
Class Thirty-Nine · Complete
Retain this much

Five things worth carrying out of this hub

  1. A pipeline replaces a fragile person with a legible machine: a defined sequence that runs the same way every time, records what it did, and refuses to skip the step that matters because it is late. That is the whole value — repeatability and a record, not speed.
  2. Every pipeline shares one shape: trigger, build, test, package as an artifact, deploy to an environment through an approval. The nine parts of this track are that shape, elaborated and taught in order. Draw it until it is reflex.
  3. CI is merging often with every change built and tested; CD is keeping that build always ready to release through a repeatable deploy. The pipeline is the machine that makes both cheap enough to actually do.
  4. The discipline is platform-independent. Learn it here on Azure DevOps and you can read GitHub Actions on the first morning, and vice versa — because the nouns differ but the shape is the field's.
  5. Meeting an inherited estate: new pipelines born as code, existing ones convert when touched, riskiest credentials convert first regardless. Neither crusade nor surrender — the migration posture of this whole bootcamp, one last time.
Notes
  1. Treat any specific market-share claim about Azure DevOps versus GitHub with suspicion — the numbers are argued about and change yearly; the direction is settled in both halves: an enormous installed base still ships on Azure DevOps, and net-new investment flows toward GitHub. Both halves matter to your career, which is precisely why this track teaches the platform-independent discipline first and the Azure DevOps nouns second. Read the current positioning before repeating anyone's prediction of the platform's demise, including one you infer from this page.