Forcing Reliability Defects in Docker

Docker is the most confident line on a lot of resumes and the least examined file in a lot of repositories. Containerizing a service became a way to signal that you know what you are doing, and for a certain kind of engineer the signal is the point. The docker-compose.yml works on the laptop, it works in the demo, it ships, and the confidence that carried it there quietly becomes the reason nobody asks the one question that decides whether it is safe: what does this do when it breaks.

That question is where I spend my time, because it is where the entrances are. A system is rarely equally secure in every state, and the state nobody tested is almost always the weaker one. The engineers most sure of their setup are often the ones who have never once watched it fail, and the distance between that confidence and that blind spot is a place an attacker can stand.

The blind spot survives because the failure that opens it does not look like a security bug. It looks like an operations problem, and calling it a security problem means admitting the confident version of the story was incomplete. So it stays filed under flaky, unexamined, shipped, running in production under a green checkmark. Using it against a system starts with seeing the two kinds of failure as genuinely different things, which most people do not.

Two different failures

A correctness bug is a wrong answer. Given input X, the system returns Y when it should have returned Z. It lives in the domain of what the system computes. We test for it with assertions, we catch it with fuzzing and property checks, and when a security person finds one they already know the vocabulary: injection, logic flaw, broken access control.

A reliability defect is the absence of a correct answer in time. The system hangs, crashes, restarts, runs out of memory, times out, or returns a degraded result because a dependency it needed was slow or gone. It lives in the domain of whether and when the system answers at all. It gets triaged to the operations column. It gets the word flaky attached to it, which is engineering slang for we have decided not to think about this.

The reason that triage is a mistake is simple and almost never stated out loud. A system is rarely equally secure in every state. It is one thing when it is healthy and a different thing when it is degraded, and the degraded version is usually the weaker one.

Degraded states are usually weaker

Walk through what systems actually do when they come under stress, and the pattern is hard to unsee.

When an authorization dependency becomes unreachable, a great many systems fail open. The policy engine times out, the call throws, the catch block logs a warning and returns allow, because the engineer who wrote it was optimizing for availability and did not want a slow sidecar to take down the whole product. That is a reliability decision with a security consequence, and it has a name: CWE-636, failure to fail securely. The steady state checks the policy. The degraded state does not.

When a process crashes and something restarts it, there is a window during boot where it is listening before it is ready. It binds the port, then loads the config, then fetches the TLS material, then wires up authentication. For a few hundred milliseconds it is a server that answers without the controls that make it safe to answer. Steady state is locked. The restart window is not.

When a log pipeline backs up or a disk fills, logging degrades or stops. The application keeps serving, but it stops writing the record of what it served. Detection goes quiet while the system stays up. The steady state is observed. The degraded state is blind.

When a healthcheck only proves that a port is open, a process can report healthy while it is anything but, and a load balancer will keep sending real traffic to an instance that has lost the half of itself that matters.

None of these are logic bugs. Every one of them is a reliability defect. And every one of them moves the system from a stronger security posture to a weaker one, on a trigger that is often nothing more exotic than load, a slow network, or a dependency going away.

The red-team move is to force it

Here is where it stops being an availability conversation and becomes an attack one.

A reliability defect on its own is latent. It fires when the stars align, when traffic spikes at 3 a.m. or a deploy races a cache warm. The defender experiences it as bad luck. The thing that turns it from bad luck into a vulnerability is a reliable trigger. If an attacker can force the degraded state on demand, they are no longer waiting for the system to be weak. They are choosing the moment.

Say it plainly. Forcing a reliability defect converts the weakest state of a system from an occasional accident into a condition the attacker selects. You do not need the fail-open path to be reachable by luck. You need to be able to knock out the dependency whose absence opens it, whenever you want, and then walk through while it is open.

The classic red-team instinct is to ask what the system computes incorrectly. The more productive instinct, and the rarer one, is to ask how the system fails, and whether its failure mode is more or less privileged than its normal one. The second question is where the entrances are.

Why Docker is the richest ground for this

I am picking on Docker on purpose, because it is everywhere and its defaults are generous in a way that manufactures reliability defects for free. It is also where the confidence runs highest. The same engineer who will defend their container setup in a design review has usually never run it out of memory on purpose, and the defaults reward that incuriosity by working anyway, right up until someone makes them fail. Every example below runs against containers you own, in a lab you control. The point is to understand the mechanism well enough to find it and close it, not to point it at anyone.

Resource limits are off until you turn them on

By default a Docker container has no memory limit and no process-count limit. It can allocate until the host has nothing left, and it can spawn processes until the host runs out of PIDs. When memory runs out, the kernel out-of-memory killer wakes up and picks a victim by its own heuristic, and that victim is not guaranteed to be the greedy container. It can be a different container on the same host, or a process the host needed.

# The quiet version. Nothing here says "unlimited," which is exactly the problem.
services:
  app:
    image: your-app:latest
  sidecar-auth:
    image: your-policy-engine:latest

That file has a reliability defect in it and the defect is what is missing. There is no mem_limit, no pids_limit, no cpus. If a request can make app balloon its memory, or if app can be driven to fork, the OOM killer may take out sidecar-auth instead, and now you are in a very interesting situation: the thing enforcing policy is dead and the thing being policed is alive. That is CWE-400, uncontrolled resource consumption, and CWE-770, allocation without limits or throttling, turned into a targeted removal of a control. In MITRE ATT&CK terms the pressure is T1499, endpoint denial of service, but the goal is not the outage. The outage is the crowbar. The prize is the state the outage leaves behind.

The fix is boring and that is the point:

services:
  sidecar-auth:
    image: your-policy-engine:latest
    mem_limit: 256m
    pids_limit: 128
    restart: on-failure

Limits keep one container's bad day from becoming the host's, and they make the OOM killer's choice predictable instead of convenient for an attacker.

Fail-open is the one that matters most

The single highest-value version of this is the authorization sidecar that fails open. The composition is seductively clean. Your application calls a policy engine over the local network for every sensitive action. It is fast, it is decoupled, it looks like good architecture. The reliability defect lives in one catch block:

# The defect is the default, not the exception path people think about.
try:
    decision = policy_client.check(subject, action, resource, timeout=0.5)
except (Timeout, ConnectionError):
    log.warning("policy engine unreachable, allowing")   # CWE-636
    decision = ALLOW

In steady state this is a correct, well-behaved system. Under a degraded policy engine it is an open door, and the degradation is forceable. If the policy container has no resource limits, you make it fall over with the technique above. If it does have limits, you exhaust its connection pool, or you fill the small queue in front of it, or you simply win the race during its own restart. However you get there, the application keeps answering, and now it answers yes to everything. You did not defeat the authorization logic. You removed the conditions under which it runs.

Failing closed is the whole fix, and it costs you availability you have to decide you can afford:

except (Timeout, ConnectionError):
    log.error("policy engine unreachable, denying")
    decision = DENY

This exact chain runs in a lab I built to go with this piece, called Docker Crowbar. Two copies of one app sit behind one policy sidecar and differ by a single line, whether they fail open or closed. The sidecar carries one deliberate defect, a /balloon endpoint that allocates memory until the container's mem_limit lets Docker's OOM killer take it. One request fires the crowbar. In the measured run the vulnerable app flips an unauthorized transfer from denied to allowed the moment the sidecar goes silent, and the hardened app does not move. It is at github.com/ghostintheprompt/docker_crowbar, it runs against its own containers, and the decision it turns on is a pure function with a test that needs no Docker at all.

Forcing the restart window

restart: always is the most natural line in a compose file and it quietly creates a loop you can drive. If an attacker can crash the application on demand, with a request that trips an unhandled exception, an input that triggers a panic, or the memory pressure from the first section, then restart: always hands them a fresh boot sequence every single time. CWE-703, improper handling of an exceptional condition, meets an orchestrator that is trying to be helpful.

Every boot replays the startup window. If the service listens before it finishes wiring authentication, or regenerates a default credential when it finds no config, or opens a debug or metrics port early in boot, then each forced restart is a new attempt at that window. A race that is impossibly tight to hit once becomes trivial when you can re-roll it on demand. The defense is to make liveness and readiness distinct, to not accept traffic until the security-relevant setup is finished, and to use on-failure with a backoff instead of an unconditional loop that an attacker can pump.

Healthchecks that only prove a port is open

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:8080/"]
  interval: 10s

This proves the socket answers. It does not prove the database connection is alive, the policy cache is warm, or the auth middleware loaded. A container can degrade into a state where it serves but no longer enforces, and keep reporting healthy, and the orchestrator will keep routing production traffic to it. The reliability defect is a healthcheck that measures the wrong thing. The security consequence is traffic pinned to a degraded, under-enforcing instance. A readiness check that exercises the security-critical path, not just the port, closes it.

Logs that eat the disk and then go silent

Docker's default logging driver writes JSON files with no size cap unless you set one. Drive enough error output and the disk fills. When it does, logging stops, and on many systems so do the application's own writes, which pushes you back toward a fail-open or fail-crash path depending on how the code handles a failed write. The reliability defect is unbounded log growth. The security consequence is a window where the system operates without a record, which is the condition every intruder wants and few can arrange. Bounded, rotated logs shipped off the host to somewhere an attacker on the box cannot starve are the answer.

logging:
  driver: json-file
  options: { max-size: "10m", max-file: "3" }

The chain has stages

A single forced fail-open is a finding. The reason to care is that the primitive scales, and each stage hands more blast radius to whoever can pull the trigger on command.

Stage one: force the defect

Everything above is stage one. Starve a container that has no memory limit, let the OOM killer take the authorization sidecar, and the app it was protecting answers yes on its own. One request, one degraded host, one unauthorized action. It runs in a lab I built for this piece, Docker Crowbar: two copies of one app differ by a single line, whether they fail open or closed, and the forced OOM flips an unauthorized transfer from denied to allowed on the one that fails open while the other does not move.

Stage two: let CI amplify it

Point the same primitive at a build pipeline and the blast radius stops being one request. It becomes everyone who pulls the result. The vector is GitHub Actions cache poisoning. A pull_request_target workflow runs with the base repository's secrets and its default-branch cache, and when it builds untrusted fork code, usually inside Docker, that code can write a poisoned entry into the shared cache. A later trusted release job restores that cache by key and runs what it finds, holding the permissions to publish. TanStack lived through exactly this in May 2026, and Angular before it. The link back to stage one is not a metaphor: the poisoned write has to land while a build is running, and one way to force that window is to evict the existing entries by filling the cache, which is the same uncontrolled-resource-consumption primitive from the first section, CWE-400, aimed at a different target. Force the reliability condition, land the poison, let the trusted pipeline carry it to every consumer downstream. The mechanism runs, marker-only, in the Docker Crowbar stage-two lab. GitHub has spent the last year closing it, which is the surest sign it is real. The Actions runtime token can no longer write to the cache after a job ends, actions/checkout is being hardened against the fork-checkout pattern, and organizations can now enforce pinning every action to a commit SHA, the fix the industry reached for after the tj-actions/changed-files compromise leaked secrets from more than twenty thousand repositories.

Stage three: poison the thing that trusts the output

A poisoned artifact, cache entry, or build log is trusted by everything downstream of it, and trusted input crossing into a context that acts on it is the exact shape of a trust transition, the move at the center of the indirect-prompt-injection work I have been mapping scenario by scenario. For years the thing downstream was a human reading a log or a server loading a package. Now it is increasingly an agent: a model that reads CI output, a coding assistant that restores the cache, a review bot that ingests the diff. Feed that agent a poisoned build log and the supply-chain compromise becomes a prompt injection that rode in through a boundary everyone had already agreed to trust, surviving the summarization hop into the agent's context and riding the pipeline's own routing to get there. That is the second level of the poison. One carrier, two payloads: the code the pipeline executes, and the instructions the agent obeys. The teams standing up AI inside their CI right now are building that carrier at speed, and they are measuring the convenience.

How to hunt them, red and blue

The method is the same from both chairs, which is why I like it.

Enumerate the dependencies of the thing you care about. For each one, ask what happens to the protected system when that dependency is slow, then when it is gone. Write the answer down as a security statement, not an availability one: when the policy engine is unreachable, access is granted. When said out loud like that, the fail-open defaults announce themselves.

Then test the degraded path on purpose. Do not wait for it. Kill the sidecar and replay the request. Starve the container and watch what the OOM killer takes. Hold the dependency in a slow-loris state and see whether the timeout path allows or denies. Crash the app and probe the boot window. These are one-line experiments in a lab you own, and they tell you in minutes what a year of uneventful uptime will never reveal.

For the blue chair the fixes are unglamorous and they are all in this piece already. Set resource and process limits. Fail closed on the controls that matter and accept the availability cost deliberately. Separate liveness from a readiness check that exercises the security path. Bound and ship your logs. Replace the reflexive always restart with a backoff that an attacker cannot pump. None of it is clever, and none of it ships by default. Most teams will read this, agree with all of it, and change nothing, because the system looks fine right up until the day someone forces the state they never tested.

What competence actually looks like

A correctness bug tells you a system is wrong. A reliability defect tells you it is fragile, and fragility becomes a security property the moment someone can trigger it on demand. The failure modes are part of the attack surface, and most have never been looked at by anyone paid to think like an attacker, because they were filed under operations by people who were certain they did not need to look.

Containerizing a service is not competence. Knowing what your container does when it is starved, crashed, restarted, or cut off from the thing it trusts is competence, and it is rarer than the confidence that stands in for it. The engineers worth working with are the ones who will say out loud that their sidecar fails open, and then go test it. The ones who treat the question as an insult are the ones already carrying the defect into production, sure they are not.

So when the test passes and the process will not exit, when the container restarts a little too eagerly, when the sidecar times out and the request goes through anyway, do not reach for the word flaky. Ask what state the system is in, and whether it is one you would mind an adversary being able to summon. If the honest answer is yes, you have found a way in, and it has been sitting in the backlog, under a green checkmark, the whole time.