The question behind “cloud-native”
An agent calls a tool. The tool reports success, the model finishes its response, and a client can display “Done”. Which of those facts survives replacement of the process? That question determines whether another attempt is useful, redundant, or dangerous.
The starting point was Craig McLuckie’s article about cloud-native agent harnesses. Mecatl separates the agent loop from clients, execution environments, and storage. The article explicitly describes recovery at persisted turn boundaries and does not promise a transaction over in-flight external actions. This experiment asks a narrower question: what does the engine report when saving itself fails?
The inspected revision was commit 9edd9b751f22c4deb5f5d0b03308e77ff06ba5d2, dated 2 October 2026. The results apply to that revision and the engine configuration below, not automatically to every Mecatl deployment.
Success despite failed persistence
The Engine.save() method treats ordinary snapshot writes as best effort. On failure it emits a correlated warning once per run. It does not return the error to its caller. In the ordinary completion path, terminateComplete() calls this method and can subsequently still emit a successful result.
An existing test explicitly requires this behaviour: TestSaveFailureEmitsOneCorrelatedWarn injects an always-failing store and still expects StopEndTurn. Here, success means that the engine ended the model turn normally. It is not an acknowledgement of a durably committed business transaction.
Not every write path has this policy. The engine also has saveRequired() for particular transitions. The finding is therefore not that Mecatl ignores all storage failures; it is a specific property of the ordinary snapshot and completion path.
What actually happens in the experiment
The test first saves an empty, idle session in Mecatl’s in-memory store. A wrapper allows reads but then rejects every snapshot write. A deterministic model adapter calls a mutating test tool and subsequently returns “done”. The tool increments a counter outside the session snapshot.
On the first run, the counter reaches 1 and the engine reports end_turn. The test then loads the old snapshot and constructs a fresh engine. The snapshot still contains zero messages. The test deliberately resubmits the task; the scripted model adapter requests the same action, and the counter reaches 2.
The control uses a working store. After one run, it loads a completed session with four messages, while the counter is 1. This control does not resubmit the task. It confirms successful persistence; it does not claim that the engine would automatically prevent all duplicate actions on another prompt.
Six existing tests and both additional experiment tests passed with Go 1.27.1. The extra file is injected through a Go overlay, leaving production source unchanged. The download contains the full commands and observed results.
Three limits on the conclusion
First: the test uses the real engine, but a simulated model, an in-memory store, and a counter standing in for an external service. It does not kill a process or restart a pod. It does not exercise Redis, the complete HTTP/gRPC server, or its separate event log.
Second: the test requests the repeat. The finding demonstrates a possible duplicate effect when deliberately retrying from stale state. It demonstrates neither an automatic Mecatl retry nor a prediction of what a real language model would decide.
Third: a repaired conversation does not establish the external outcome. For certain unanswered tool calls, Session.Abandon() adds synthetic error results so that history contains valid call/result pairs again. It does not query the external service for the actual outcome. If the old snapshot does not contain the call at all, this repair cannot reconstruct it.
Make the operational policy explicit
For read-only research, continuing through a storage fault may be reasonable if degraded recoverability remains visible. For an agent that changes infrastructure, I would gate further mutations on an explicit persistence and execution policy. A generic “Done” should not collapse those states.
It helps to record three facts separately: the agent-run outcome, acknowledgement of saved state, and the external action’s outcome. The last can be unknown. A timeout or a missing snapshot does not mean that the action did not execute.
Where the target API supports idempotency keys, retries of the same logical action must reuse the same stable key. Where it does not, recovery needs service-specific lookup and reconciliation or human resolution. Checking before creating alone does not prevent concurrent requests. Stopping after a storage failure also cannot undo an action that already happened.
What I would verify before granting production write access
This experiment is a starting point for a larger qualification. For the actual server, store, and tool combination, I would inject snapshot and event-log write failures separately, load from a new process, and count external effects independently. I would then test interruption between external acknowledgement and local persistence, followed by two concurrent retries. Those tests have not been performed here.
Mecatl documents recovery limits and makes this policy visible in code. The useful finding is not a blanket product verdict: a successful agent result alone is insufficient evidence for a safe retry. The application needs to know which effect it has confirmed and which question remains open.
Reproduce the snapshot-write failure
Reproducible learning example using synthetic data. Not an account of an employer or client project.
The download contains both supplemental tests, a script targeting the pinned commit, and the observed output. It exercises Mecatl’s actual engine rather than a rewritten Python loop.
Run locally
Prerequisites: Go 1.27.1 (tested), Git, Python 3, and a POSIX shell. Unzip into an empty working directory. The script downloads the repository and Go dependencies; the tests themselves require no model API or credentials. It refuses to overwrite an existing mecatl directory.
Download the experiment · mecatl-snapshot-lab.zip
sh reproduce.shExample output
attempt=1 restored_state=idle restored_messages=0
attempt=1 result=end_turn external_effect_count=1
attempt=2 restored_state=idle restored_messages=0
attempt=2 result=end_turn external_effect_count=2
healthy_store: restored_state=completed restored_messages=4 external_effect_count=1These are the experiment’s log lines with Go test prefixes removed. Both tests passed. The counter outlives the reconstructed engine but is not a real external service. The scripted model adapter and deliberate task resubmission make the result deterministic.
Make the failure visible at the right boundary
For a production service, I would expose and correlate persistence health separately from task outcome. A persistence warning needs an explicit response: pause further mutations, verify recovery, or involve an operator. The appropriate response depends on what the tool can change.
Engine-level test with an injected storage fault. It establishes no production failure rate, automatic retry behaviour, real-model decision, or end-to-end mecak8s deployment guarantee.
Technical references
- Mecatl · Engine.save and completion path (pinned source)
- Mecatl · Existing persistence-failure test (pinned source)
- Mecatl · Session.Abandon and history repair (pinned source)
- Mecatl · Session continuity
- Craig McLuckie · The case for a cloud native agent harness
- Temporal · Activity idempotency
- AWS Builders’ Library · Making retries safe with idempotent APIs