Three claims, three tests
Synced: the live state compared by Argo CD matches rendered desired state within the scope of that comparison. That is useful evidence about delivery. It is not a business request or proof that every external dependency works.
Ready: the configured readiness conditions are satisfied. A readiness probe may intentionally check only part of the application. Failure does not restart the container; startup and liveness have different jobs.
A successful request: one path worked with particular inputs, an identity and a point in time. That evidence is bounded too. The task is to choose a path representative of the operational decision.
Narrow the contradiction
Suppose the rollout is Synced and Pods are Ready, but a feature returns AccessDenied. My first move would not be another rollout. The evidence now asks a different question: which identity is calling which action on which resource? GitOps can successfully deliver the intended configuration even when that configuration lacks a necessary permission.
A timeout leads elsewhere: separate name resolution, connection establishment, TLS and destination state. The same visible application symptom can originate at different boundaries. A diagnosis should be allowed to change direction when evidence changes.
A correction needs two kinds of completion
First, the affected application path must work again. Second, the correction must survive the system’s ownership rules. A live patch can disappear at the next GitOps reconciliation. A change in Git can remain ineffective until it reaches the affected revision.
I therefore separate effect from persistence: repeat the request, identify the applied revision and observe the next normal reconciliation. A green status is one signal. Explaining why the failure should not recur is the result.
Reproduce it: readiness 200, business request 503
Reproducible learning example using synthetic data. Not an account of an employer or client project.
A green status can be correct and still be insufficient. In this example, an application successfully answers “/ready”. To answer “/report”, it also needs a dependency that denies access.
Run locally
Python 3 is sufficient. The script starts two short-lived HTTP servers bound only to 127.0.0.1, using available ports. It needs no cloud credentials, writes no files, and stops both servers itself. Download it, then run:
Download the Python example · readiness-boundary.py
python readiness-boundary.pyExample output
GET /ready: 200 ready
Dependency GET /records: 403 AccessDenied
GET /report: 503 dependency denied access
Result: readiness passed; the business request failedThe 200 status establishes only that the readiness response succeeded. Calling the modelled dependency directly narrows the failure to denied access; the application translates it into 503. That translation is a choice in this example, not general Kubernetes or AWS behaviour. A 403 alone does not identify a particular incorrect policy.
Read the real evidence
In a suitable Kubernetes/Argo CD environment, these read-only queries can start the investigation. “argocd”, “reporting”, and “reports” are example names to replace with those in the actual installation. These commands were not executed against a cluster for this local model.
kubectl -n argocd get application reports -o yaml
kubectl -n reporting get deployment reports -o yaml
kubectl -n reporting get pods -l app=reports -o wide
kubectl -n reporting logs deployment/reports --since=10m --timestamps=trueIn the Application object, distinguish revision, sync and health. In the Deployment, inspect what the readiness probe actually calls. With multiple Pods and containers, associate the failed request with the correct instance; the final command does not automatically cover every replica. Look for timestamps, correlation IDs, and the specific dependency failure in logs. Do not publish unreviewed internal logs.
The next check follows the boundary
For a real access denial, first establish the actual workload’s identity, action and resource. An administrator-level test does not answer that question. Nor should every external dependency automatically become part of readiness: if a shared dependency fails, every replica could be removed from service traffic at once. The probe must support the intended operational decision.
The script demonstrates an HTTP boundary, not a Kubernetes or Argo CD installation. It cannot establish a real Synced status or validate IAM permissions. Keeping those claims separate is the point of the example.