AI-ENG-O — Action Verification - Planning, Execution, Observation & Recovery

The integration of generative artificial intelligence models into distributed computing environments introduces a significant architectural mismatch: the interface between probabilistic, non-deterministic decision engines and rigid, deterministic state machines. In agentic workflows, this mismatch manifests as a systemic vulnerability at the point of action execution. Historically, distributed systems have relied on transport-level or database-level execution acknowledgments (such as an HTTP 200 OK or a write-ahead log commit) as proof of successful state mutation. However, in high-dimensional agentic architectures, an action cannot be deemed complete when the model asserts its completion, or even when the invoking tool returns a successful execution response.
An action is not complete when the model says it is complete, or even when the tool returns success. An action is complete only when the system verifies the resulting state against the intended outcome and reconciles any discrepancy.
Action verification functions as the central truth-management layer of tool-using AI architectures. It bridges the gap between what an agent believes it has accomplished and what has actually changed within authoritative systems. The fundamental question is not “Did the tool return a success response?” but “Did the intended state change actually occur, in the correct system, to the correct object, under the correct tenant and permission scope, with the expected side effects, no hidden duplicates, no unresolved partial failures, and an auditable recovery path if anything diverged?”. Without this verification layer, probabilistic agents operate on ungrounded assumptions, leading to cascade failures, duplicate mutations, or silent data corruptions. This report establishes the architectural specification for verifying planning, execution, observation, and recovery in enterprise-grade agentic environments.

Canon Placement and Continuity

This report closes Volume 5: Agentic Systems and Tool-Using Architectures, whose primary concern is how static generative models are transformed into autonomous actors, and how to make those actions bounded, contract-governed, verified, and recoverable.
To maintain structural durability within the AI Engineering Systems Canon, this report establishes precise interfaces and clear boundaries with preceding and succeeding modules:

• AI-ENG-M (Agentic Orchestration) owns the decision-making loop. It determines whether the agent should plan, call a tool, continue, stop, or escalate based on the current context. It manages loop budgets and records raw execution traces.
• AI-ENG-N (Tool Contracts) owns the execution gateway. It determines whether a proposed tool call is syntactically valid, authorized, safe, and idempotent prior to execution. It manages deterministic wrappers, validation schemas, and structural output contracts.
• AI-ENG-O (Action Verification) owns the post-action truth. It determines what actually occurred after execution, whether the agent’s task state can be updated, whether the user can be notified of completion, and the exact recovery path that must execute if reality diverges from the model’s belief.

  +---------------------------------------------------------------------------------+  
  |                            Volume 5 Architectural Boundaries                    |  
  +---------------------------------------------------------------------------------+  
  |  AI-ENG-M: Agentic Orchestration                                                |  
  |  - Decision-making loop                                                         | 
  |  - Plan generation and sub-goal management                                      |  
  |  - Autonomy boundaries and loop budgets                                         |  
  +---------------------------------------------------------------------------------+  
                                         |  
                                         v  
  +---------------------------------------------------------------------------------+  
  |  AI-ENG-N: Tool Contracts                                                       |  
  |  - Parameter validation and deterministic wrappers                              |  
  |  - Security mapping and credentials injection                                   |  
  |  - Idempotency key generation and confirmation gates                            |  
  +---------------------------------------------------------------------------------+  
                                         |  
                                         v  
  +---------------------------------------------------------------------------------+  
  |  AI-ENG-O: Action Verification (Truth-Management Layer)                         |  
  |  - Post-execution state verification and readbacks                              |  
  |  - Discrepancy classification and state reconciliation                          |  
  |  - Rollback, compensation sagas, and forward recovery                           |  
  +---------------------------------------------------------------------------------+

Furthermore, this report inherits and extends doctrines from across the broader canon:

• AI-ENG-B (Context Tenure and State Governance): Reconciled system states are bound to specific context life spans and strict tenant security scopes to prevent data leakage during multi-turn verification loops.
• AI-ENG-C (Cost Attribution and System Margin): Post-action verification queries and polling loops consume model tokens and API quotas, requiring dedicated margin allocations to protect system throughput.
• AI-ENG-I (Release Manifest, Regression Control, and Replay): Verification strategies, polling rules, and recovery workflows are treated as version-controlled artifacts, validated via replayable execution histories to prevent behavioral regressions.
• AI-ENG-L (Serving Observability and Operational Fallback): Degraded verification states (such as an unreachable database replica) map directly to defined fallback pathways to maintain agent availability without sacrificing correctness.

Conceptual Glossary

To establish a uniform vocabulary for platform architects, agent engineers, and site reliability engineers (SREs), the fundamental terms of action verification are defined below.

The Core Doctrine of Post-Execution Truth

The architectural foundation of action verification is post-execution truth management. Agentic systems must distinguish between what was planned, what was proposed, what was validated, what was attempted, what was observed, what was verified, what was reconciled, and what may safely be reported.

The lifecycle is:

+--------------------------------------------------------------------------------
| POST-EXECUTION TRUTH LIFECYCLE
+--------------------------------------------------------------------------------
|
| [ Planned ]
|      |
|      v
| [ Proposed ]
|      |
|      v
| [ Validated ]
|      |
|      v
| [ Executed / Attempted ]
|      |
|      v
| [ Observed ]
|      |
|      v
| [ Verified ]
|      |
|      v
| [ Reconciled ]
|      |
|      v
| [ Reported ]
|
+--------------------------------------------------------------------------------
| Rule:
|   User-facing success may not outrun reconciliation.
|   Agent task state may not outrun verification.
|   Model belief may not outrun authoritative state.
+--------------------------------------------------------------------------------

Why Every Transition Matters

Observations Are Evidence, Not Reality

Tool observations are structured evidence. They are not automatically authoritative truth.

For example:

• An email service may return accepted, but delivery may later bounce, quarantine, or fail.
• A payment processor may return authorized, but capture or settlement may remain pending.
• A deployment tool may return a rollout ID, but the rollout may later fail health checks.
• A database wrapper may return success, but a later verification query may reveal a version conflict, trigger failure, or wrong tenant boundary.
• A queue publisher may return accepted, but the downstream consumer may fail or dead-letter the message.

The orchestrator must therefore treat every observation as an intermediate state until verification determines what actually happened.

ObservationObject.status = evidence
AuthoritativeReadback = verification source
ReconciliationResult = task-state update authority

Reporting Invariant

A system may report:

The model must never convert an accepted, pending, timeout, or unverified observation into a completed user-facing success statement.

Hallucinated Success and Assumed Completion

Hallucinated success is the central execution pathology of tool-using agentic systems. It occurs when a model, orchestrator, wrapper, or user-facing interface treats an intended, attempted, pending, failed, partial, or unverifiable action as successfully completed.

This failure does not always originate in model text generation. It is often an architectural failure: the system failed to separate execution from verification.

From an operational standpoint, hallucinated success corrupts the agent’s belief state. Once an agent proceeds from false success, subsequent plans can compound the error: sending follow-up messages, making duplicate charges, skipping required remediation, or writing poisoned memory.

Completion Rule

A model statement is not completion.
A dispatch attempt is not completion.
An HTTP success code is not completion.
An accepted queue message is not completion.
A normalized observation is not completion.

Completion occurs only when authoritative state satisfies the required
verification predicates and the reconciled result is committed to task state.

State Reconciliation Model

The State Reconciliation Model compares the intended outcome, requested operation, observed response, authoritative state, and believed task state. Its purpose is not to demand raw object equality. Real systems add IDs, timestamps, status transitions, derived fields, audit metadata, version numbers, provider-specific fields, and asynchronous states.

Therefore, action verification should evaluate predicates over authoritative state, not simplistic equality between intended state and full system state.

+--------------------------------------------------------------------------------
| STATE RECONCILIATION MODEL
+--------------------------------------------------------------------------------
|
| [ Intended Outcome ]
|   user goal, plan target, expected business result
|        |
|        v
| [ Requested Operation ]
|   validated tool payload, target resource, tenant scope, idempotency key
|        |
|        v
| [ Observed Response ]
|   normalized tool observation: success, accepted, pending, error, timeout
|        |
|        v
| [ Authoritative State ]
|   source-of-record readback, ledger query, delivery status, deployment health
|        |
|        v
| [ Reconciliation Result ]
|   success, partial, mismatch, pending, duplicate, compensated, unknown
|        |
|        v
| [ Believed Task State ]
|   updated only from reconciliation result
|
+--------------------------------------------------------------------------------
| Rule:
|   Authoritative state is evaluated against verification predicates.
|   Believed task state is updated only after reconciliation.
+--------------------------------------------------------------------------------

Let:

A successful reconciliation is:

RR = SUCCESS if all p in P evaluate true over SA, SR, SI, and scope metadata.

Examples of verification predicates:

SA.resource_id == SR.resource_id
SA.tenant_id == SR.tenant_id
SA.status in allowed_terminal_success_states
SA.amount == SR.amount
SA.currency == SR.currency
SA.version > pre_action_version
SA.actor_id == expected_actor_or_service
SA.idempotency_key == SR.idempotency_key
SA.created_at >= action_execution_time
SA.audit_log contains trace_id

State Layers

Observation Trust Ladder

+--------------------------------------------------------------------------------
| OBSERVATION TRUST LADDER
+--------------------------------------------------------------------------------
|
| Level 0: Raw Tool Output
|   stdout, stack traces, provider blobs, raw API errors
|   Status: unsafe for direct model reasoning
|
| Level 1: Normalized Observation
|   typed status, structured errors, trace ID, result payload
|   Status: safe evidence, not verified truth
|
| Level 2: Verified Observation
|   independent readback confirms that an event or resource exists
|   Status: execution likely occurred
|
| Level 3: Reconciled State
|   authoritative state satisfies required predicates
|   Status: safe to update task state
|
| Level 4: Audit-Confirmed State
|   reconciled state is bound to append-only ledger, approval records,
|   idempotency record, policy version, and trace evidence
|   Status: safe for regulated reporting and replay
|
+--------------------------------------------------------------------------------

Discrepancy Classification Model

When authoritative state does not satisfy the predicate set, the reconciliation engine classifies the discrepancy.

Verification Depth by Side-Effect Class

Verification depth should scale with side-effect class, data sensitivity, reversibility, cost, and user impact. Low-risk actions should not require heavyweight reconciliation. Critical mutations should never rely on a single transport success response.

Verification Depth Rules

Read-only does not mean no-risk.
Low-risk write does not mean stale replica success.
External write does not mean provider-accepted success.
Critical mutation does not mean "retry and hope."

Verification strategy must be defined in the tool contract and recorded in the action ledger. If a tool has no verification path, the system must treat its result as unverifiable and constrain what the agent may claim or do next.

Post-Action Check Pattern Library

Post-action verification patterns vary by target system. Each pattern defines how the system verifies that the intended result occurred in the correct resource, tenant, region, and authority scope.

1. Databases

• Validation Action: Issue a read-after-write query against the writer, primary, or strongly consistent read endpoint. When available, use commit tokens, row versions, transaction IDs, or Log Sequence Numbers (LSNs) to avoid stale read ambiguity.
• Verification Parameters: Confirm target row exists, tenant ID matches, actor/service ID matches, row version advanced, changed fields satisfy predicates, and audit log contains the trace ID.
• Failure Indicator: Missing row, stale version, wrong tenant, mismatched field values, lock timeout, or verification read that cannot guarantee freshness.

2. Filesystems

• Validation Action: Normalize and resolve the target path, then verify that the resolved path remains inside the allowed sandbox root.
• Verification Parameters: Confirm file exists, path is allowlisted, file size matches expected bytes, file hash matches payload, permissions are correct, and no symlink/path traversal escape occurred.
• Failure Indicator: Resolved path escapes sandbox, file missing, hash mismatch, unauthorized extension, unexpected permission mode, or destructive operation outside policy.

3. Email Services

• Validation Action: Query provider status using returned message ID, gateway ID, or idempotency key.
• Verification Parameters: Confirm recipient, sender, subject/body hash when appropriate, queued/sent/delivered status, bounce status, suppression-list status, and DKIM/DMARC alignment where available.
• Failure Indicator: Bounce, rejection, quarantine, suppression, wrong recipient, duplicate send, or delivery state unknown beyond SLA.

4. Calendar Systems

• Validation Action: Read the created or updated event by event ID from the authoritative calendar API.
• Verification Parameters: Confirm organizer, calendar ID, timezone, start/end time, attendees, recurrence, conferencing link, reminders, and event status.
• Failure Indicator: Missing event, wrong timezone, duplicate event, attendee invite failure, permission mismatch, or conflicting booking policy.

5. Payments and Refunds

• Validation Action: Query payment processor, internal ledger, and idempotency record.
• Verification Parameters: Confirm transaction ID, idempotency key, amount, currency, account, status, authorization/capture/refund state, and ledger entry.
• Failure Indicator: Decline, duplicate charge, amount mismatch, pending beyond SLA, settlement failure, refund mismatch, or provider/local ledger divergence.

6. CRM and Ticketing

• Validation Action: Fetch updated record and change history from authoritative CRM/ticketing API.
• Verification Parameters: Confirm object ID, tenant/account, field changes, assignee, status, comments, actor ID, workflow ID, and timestamp.
• Failure Indicator: Missing field update, wrong assignee, duplicate ticket, record lock, stale state, or missing change-history entry.

7. Deployment Pipelines

• Validation Action: Query deployment orchestrator, rollout controller, health checks, and error-budget monitor.
• Verification Parameters: Confirm artifact SHA, environment, region, rollout stage, health checks, replica readiness, error rate, rollback status, and approval ID.
• Failure Indicator: Crash loop, failed readiness, rollback triggered, wrong SHA, partial-region rollout, error-budget burn, or missing deployment approval.

8. Procurement Systems

• Validation Action: Query procurement register, approval workflow, vendor catalog, and budget ledger.
• Verification Parameters: Confirm PO ID, vendor, SKU, amount, currency, approval state, budget allocation, contract terms, and requester authority.
• Failure Indicator: Unauthorized vendor/SKU, price mismatch, budget breach, approval missing, duplicate PO, or rejected purchase.

9. Browser and Web Automations

• Validation Action: Inspect page state after action using selectors, URL assertions, downloaded artifacts, and server-side confirmation when possible.
• Verification Parameters: Confirm success page, stable URL, expected DOM element, submitted values, downloaded file hash, and absence of error banner.
• Failure Indicator: Selector timeout, CAPTCHA, navigation mismatch, form validation failure, wrong account context, or file hash mismatch.

10. Security and Infrastructure

• Validation Action: Query IAM, directory, policy engine, audit logs, and live token assertions.
• Verification Parameters: Confirm target principal, granted/removed scopes, policy diff, approval ID, actor ID, audit entry, and absence of overbroad wildcard permissions.
• Failure Indicator: Privilege escalation, wrong principal, missing audit event, wildcard policy, stale token, or policy mismatch.

11. Queue and Event Systems

• Validation Action: Confirm message publication, consumer acknowledgement, dead-letter status, and downstream state transition.
• Verification Parameters: Confirm topic/queue, message ID, idempotency key, consumer group, processing status, and resulting state.
• Failure Indicator: Message stuck, duplicate event, dead-letter, consumer failure, out-of-order processing, or missing downstream mutation.

12. Vector Indexing and Search

• Validation Action: Query index metadata and perform retrieval test using stable document ID or checksum.
• Verification Parameters: Confirm document ID, embedding version, index version, corpus snapshot, permission scope, and retrieval eligibility.
• Failure Indicator: Document missing, wrong embedding version, stale index, permission leakage, or retrieval mismatch.

Verification should be specific enough that a downstream auditor can answer: what was changed, where, by whom, under which authority, and how the system proved it.

Partial Failure Handling Model

Distributed actions rarely fit a simple success/failure binary. A robust agentic architecture must represent intermediate, partial, pending, unknown, compensated, and review-required states explicitly.

Action State Machine

+--------------------------------------------------------------------------------
| ACTION VERIFICATION STATE MACHINE
+--------------------------------------------------------------------------------
|
| [ Proposed ]
|      |
|      v
| [ Validated ]
|      |
|      v
| [ Executing ]
|      |
|      +--> transport/tool failure before side effect -----> [ Failed ]
|      |
|      +--> accepted but not terminal ----------------------> [ Accepted ]
|      |                                                       |
|      |                                                       v
|      |                                                  [ Pending ]
|      |                                                       |
|      |                                      +----------------+----------------+
|      |                                      |                                 |
|      |                                      v                                 v
|      |                                [ Committed ]                 [ Verification Timeout ]
|      |                                      |                                 |
|      |                                      v                                 v
|      |                              [ Reconciled Success ]           [ Unknown / Review Required ]
|      |
|      +--> authoritative mismatch -------------------------> [ Reconciliation Failed ]
|      |                                                       |
|      |                         +-----------------------------+-----------------------------+
|      |                         |                                                           |
|      |                         v                                                           v
|      |                 [ Compensating ]                                             [ Forward Recovery ]
|      |                         |                                                           |
|      |              +----------+----------+                                     +----------+----------+
|      |              |                     |                                     |                     |
|      |              v                     v                                     v                     v
|      |       [ Compensated ]      [ Compensation Failed ]              [ Reconciled Success ]   [ Review Required ]
|      |
|      +--> multi-step partial commit -----------------------> [ Partially Committed ]
|                                                              |
|                                      +-----------------------+-----------------------+
|                                      |                                               |
|                                      v                                               v
|                                [ Compensating ]                              [ Forward Recovery ]
|
| Terminal states:
|   Reconciled Success
|   Failed
|   Compensated
|   Rolled Back
|   Review Required
|   Abandoned
|
+--------------------------------------------------------------------------------

Technical System States and Transitions

Partial Failure Rule

Committed is not Partially Committed.
Partially Committed is not Success.
Accepted is not Committed.
Unknown is not Failure.
Compensated is not the same as Rolled Back.

The state machine must preserve these distinctions because each state implies different user messaging, retry permissions, compensation options, and audit obligations.

Rollback, Compensation, and Forward Recovery

When verification detects a discrepancy, the system must choose a recovery path based on the transaction boundary, side-effect class, reversibility, idempotency posture, and business risk.

+--------------------------------------------------------------------------------
| RECOVERY STRATEGY MODEL
+--------------------------------------------------------------------------------
|
| [ Verification Discrepancy Detected ]
|      |
|      v
| [ Classify Transaction Boundary ]
|      |
|      +--> local atomic transaction open
|      |       |
|      |       v
|      |   [ Rollback ]
|      |       abort transaction | verify no visible side effect | ledger rolled_back
|      |
|      +--> compensable pre-pivot action
|      |       |
|      |       v
|      |   [ Backward Compensation ]
|      |       run compensation actions in reverse order | verify compensated state
|      |
|      +--> pivot committed / irreversible action
|      |       |
|      |       v
|      |   [ Forward Recovery ]
|      |       complete remaining idempotent steps | reconcile final state
|      |
|      +--> unknown or unverifiable high-impact state
|      |       |
|      |       v
|      |   [ Hold and Escalate ]
|      |       freeze workflow | preserve trace | route to operator
|
+--------------------------------------------------------------------------------

Rollback

Rollback is preferred when the system still controls a local transaction boundary. It restores the prior state before intermediate changes become externally visible.

Compensation

Compensation is used when a committed side effect cannot be rolled back atomically but can be logically reversed through another action.

Compensation is not time travel. It creates a new event that reverses or offsets the prior event. Audits must show both.

Forward Recovery

Forward recovery is used when rollback or compensation is impossible, unsafe, legally inappropriate, or more harmful than completing the workflow.

The Pivot Transaction

The pivot transaction is the point of no return in a saga.

[ Compensable Step 1 ]
      |
      v
[ Compensable Step 2 ]
      |
      v
[ Pivot Transaction ]
      |
      v
[ Retryable Step 3 ]
      |
      v
[ Retryable Step 4 ]
      |
      v
[ Reconciled Success ]

Recovery Strategy Matrix

Recovery Decision Tables

Recovery decisions must be deterministic. The model may explain a discrepancy, but recovery policy should map discrepancy type, side-effect class, transaction boundary, and operational condition to a predefined action.

Recovery Selection Rule

Prefer rollback when state is local and uncommitted.
Prefer compensation when prior side effects are reversible.
Prefer forward recovery after pivot or irreversible commit.
Prefer hold-and-escalate when state is unknown, high-impact, or security-sensitive.
Never retry a mutating operation blindly.

Action Ledger and Auditability

The Action Ledger is an append-only record of action intent, execution, verification, reconciliation, and recovery. Runtime traces describe service hops and latency. The Action Ledger describes what changed, what was proven, and how the system decided what to do next.

Action Ledger Entry Schema

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://canon.ai-eng.org/v5/action-ledger-entry.schema.json",
  "title": "ActionLedgerEntry",
  "type": "object",
  "required": [
    "action_id",
    "workflow_run_id",
    "tenant_id",
    "principal_id",
    "tool_contract",
    "policy_context",
    "side_effect_class",
    "intended_outcome",
    "requested_operation",
    "execution",
    "verification",
    "reconciliation",
    "timestamps",
    "trace"
  ],
  "additionalProperties": false,
  "properties": {
    "action_id": {
      "type": "string"
    },
    "workflow_run_id": {
      "type": "string"
    },
    "tenant_id": {
      "type": "string"
    },
    "principal_id": {
      "type": "string"
    },
    "tool_contract": {
      "type": "object",
      "required": [
        "name",
        "version",
        "schema_version",
        "wrapper_version"
      ],
      "additionalProperties": false,
      "properties": {
        "name": {
          "type": "string"
        },
        "version": {
          "type": "string"
        },
        "schema_version": {
          "type": "string"
        },
        "wrapper_version": {
          "type": "string"
        }
      }
    },
    "policy_context": {
      "type": "object",
      "required": [
        "autonomy_boundary_version",
        "approval_policy_version",
        "verification_policy_version",
        "recovery_policy_version"
      ],
      "additionalProperties": false,
      "properties": {
        "autonomy_boundary_version": {
          "type": "string"
        },
        "approval_policy_version": {
          "type": "string"
        },
        "verification_policy_version": {
          "type": "string"
        },
        "recovery_policy_version": {
          "type": "string"
        }
      }
    },
    "side_effect_class": {
      "type": "string",
      "enum": [
        "READ_ONLY",
        "EPHEMERAL_WRITE",
        "LOW_RISK_INTERNAL",
        "MEDIUM_RISK_WRITE",
        "HIGH_RISK_EXTERNAL",
        "CRITICAL_MUTATION"
      ]
    },
    "idempotency": {
      "type": "object",
      "required": [
        "required",
        "key_hash",
        "request_hash",
        "status"
      ],
      "additionalProperties": false,
      "properties": {
        "required": {
          "type": "boolean"
        },
        "key_hash": {
          "type": [
            "string",
            "null"
          ]
        },
        "request_hash": {
          "type": [
            "string",
            "null"
          ]
        },
        "status": {
          "type": [
            "string",
            "null"
          ],
          "enum": [
            "PENDING",
            "COMPLETED",
            "FAILED_RETRYABLE",
            "FAILED_FINAL",
            "COMPENSATED",
            "EXPIRED",
            null
          ]
        }
      }
    },
    "approval": {
      "type": "object",
      "required": [
        "required",
        "approval_id",
        "approver_id",
        "payload_hash",
        "expires_at"
      ],
      "additionalProperties": false,
      "properties": {
        "required": {
          "type": "boolean"
        },
        "approval_id": {
          "type": [
            "string",
            "null"
          ]
        },
        "approver_id": {
          "type": [
            "string",
            "null"
          ]
        },
        "payload_hash": {
          "type": [
            "string",
            "null"
          ]
        },
        "expires_at": {
          "type": [
            "string",
            "null"
          ],
          "format": "date-time"
        }
      }
    },
    "intended_outcome": {
      "type": "object",
      "required": [
        "target_resource",
        "expected_predicates"
      ],
      "additionalProperties": false,
      "properties": {
        "target_resource": {
          "type": "string"
        },
        "expected_predicates": {
          "type": "array",
          "items": {
            "type": "string"
          }
        }
      }
    },
    "requested_operation": {
      "type": "object",
      "required": [
        "validated_payload_hash",
        "target_resource",
        "operation_kind"
      ],
      "additionalProperties": false,
      "properties": {
        "validated_payload_hash": {
          "type": "string"
        },
        "target_resource": {
          "type": "string"
        },
        "operation_kind": {
          "type": "string"
        }
      }
    },
    "execution": {
      "type": "object",
      "required": [
        "status",
        "observation_pointer",
        "attempt_count"
      ],
      "additionalProperties": false,
      "properties": {
        "status": {
          "type": "string",
          "enum": [
            "NOT_EXECUTED",
            "EXECUTING",
            "ACCEPTED",
            "PENDING",
            "COMMITTED",
            "FAILED",
            "UNKNOWN"
          ]
        },
        "observation_pointer": {
          "type": [
            "string",
            "null"
          ]
        },
        "attempt_count": {
          "type": "integer",
          "minimum": 0
        }
      }
    },
    "verification": {
      "type": "object",
      "required": [
        "status",
        "source",
        "query_pointer",
        "verified_state_pointer"
      ],
      "additionalProperties": false,
      "properties": {
        "status": {
          "type": "string",
          "enum": [
            "NOT_REQUIRED",
            "NOT_STARTED",
            "PENDING",
            "VERIFIED",
            "FAILED",
            "TIMEOUT",
            "UNVERIFIABLE"
          ]
        },
        "source": {
          "type": [
            "string",
            "null"
          ]
        },
        "query_pointer": {
          "type": [
            "string",
            "null"
          ]
        },
        "verified_state_pointer": {
          "type": [
            "string",
            "null"
          ]
        }
      }
    },
    "reconciliation": {
      "type": "object",
      "required": [
        "status",
        "discrepancy_class",
        "recovery_decision"
      ],
      "additionalProperties": false,
      "properties": {
        "status": {
          "type": "string",
          "enum": [
            "NOT_STARTED",
            "RECONCILED_SUCCESS",
            "RECONCILED_PARTIAL",
            "RECONCILED_FAILURE",
            "UNKNOWN",
            "COMPENSATED",
            "ROLLED_BACK",
            "REVIEW_REQUIRED"
          ]
        },
        "discrepancy_class": {
          "type": [
            "string",
            "null"
          ]
        },
        "recovery_decision": {
          "type": [
            "string",
            "null"
          ]
        }
      }
    },
    "recovery": {
      "type": "object",
      "required": [
        "recovery_action_id",
        "compensation_action_id",
        "rollback_action_id",
        "incident_id",
        "review_id"
      ],
      "additionalProperties": false,
      "properties": {
        "recovery_action_id": {
          "type": [
            "string",
            "null"
          ]
        },
        "compensation_action_id": {
          "type": [
            "string",
            "null"
          ]
        },
        "rollback_action_id": {
          "type": [
            "string",
            "null"
          ]
        },
        "incident_id": {
          "type": [
            "string",
            "null"
          ]
        },
        "review_id": {
          "type": [
            "string",
            "null"
          ]
        }
      }
    },
    "timestamps": {
      "type": "object",
      "required": [
        "proposed_at",
        "validated_at",
        "executed_at",
        "verified_at",
        "reconciled_at"
      ],
      "additionalProperties": false,
      "properties": {
        "proposed_at": {
          "type": "string",
          "format": "date-time"
        },
        "validated_at": {
          "type": [
            "string",
            "null"
          ],
          "format": "date-time"
        },
        "executed_at": {
          "type": [
            "string",
            "null"
          ],
          "format": "date-time"
        },
        "verified_at": {
          "type": [
            "string",
            "null"
          ],
          "format": "date-time"
        },
        "reconciled_at": {
          "type": [
            "string",
            "null"
          ],
          "format": "date-time"
        }
      }
    },
    "trace": {
      "type": "object",
      "required": [
        "trace_id",
        "parent_span_id",
        "replay_bundle_id"
      ],
      "additionalProperties": false,
      "properties": {
        "trace_id": {
          "type": "string"
        },
        "parent_span_id": {
          "type": [
            "string",
            "null"
          ]
        },
        "replay_bundle_id": {
          "type": [
            "string",
            "null"
          ]
        }
      }
    }
  }
}

Trace versus Ledger

A trace answers: “What services did the request pass through?”

An action ledger answers: “What did the system intend to change, what actually changed, who approved it, how was it verified, and what recovery path executed?”

Tool-Result Grounding and Agent State Updates

To prevent model belief drift, the orchestrator must separate the model’s generation context from verified task state. The model may remember that it proposed or attempted an action. It may not treat that action as complete until reconciliation commits the result.

+--------------------------------------------------------------------------------
| TOOL-RESULT GROUNDING MODEL
+--------------------------------------------------------------------------------
|
| [ Model Execution Context ]
|   "I should charge the card."
|      |
|      v
| [ Tool Proposal ]
|   proposed payload, not state truth
|      |
|      v
| [ Deterministic Wrapper ]
|   validates, authorizes, executes, observes
|      |
|      v
| [ Observation Object ]
|   evidence: success, error, accepted, pending, timeout
|      |
|      v
| [ Verification Layer ]
|   source-of-record readback, provider status, ledger query
|      |
|      v
| [ Reconciliation Result ]
|   success, partial, pending, failed, compensated, unknown
|      |
|      v
| [ Structured Task State ]
|   updated only from reconciliation result
|      |
|      v
| [ Next Model Context ]
|   injected with verified state, not raw self-belief
|
+--------------------------------------------------------------------------------

State Update Rules

Strict Status Reporting Invariant

If verification is pending, report pending.
If state is unknown, report unknown.
If action partially completed, report partial.
If compensation occurred, report compensated.
If success is reconciled, only then report complete.

Planning Adaptation

After each verification result, the next planning step must be grounded in the reconciled task state.

The model’s context should contain verified state summaries, trace pointers, and unresolved discrepancies. It should not contain raw logs that invite the model to invent a happy ending. The audit department has enough hobbies already.

Action Verification Observability and Regression Control

Action verification must be observable, measurable, replayable, and regression-tested. A verification layer that silently fails is worse than no verification layer because it creates false confidence.

Platform Metrics

Trace Linkage and Context Propagation

Every action should link:

model inference span
  -> tool validation span
    -> deterministic wrapper span
      -> tool execution span
        -> verification span
          -> reconciliation span
            -> recovery span, if needed

Required trace attributes include:

Regression Control and Release Gates

Verification logic is production logic. Changes to predicates, polling rules, source selection, recovery policy, observation schema, or ledger schema can break agent behavior.

Release gates should include:

Release Invariant

No verifier release should promote unless historical trace replay,
canary metrics, failure injection, and ledger compatibility checks pass.

Action verification is not a passive logging layer. It is the truth-maintenance system that protects the agent from believing its own press release.

Cross-Canon Handoff Map

The truth-management patterns established in AI-ENG-O provide critical interfaces across the AI Engineering Systems Canon. AI-ENG-O receives proposed and executed actions from orchestration and tool-contract layers, then returns verified, reconciled state to downstream planning, audit, governance, telemetry, evaluation, fallback, and incident systems.

The durable handoff is this:

AI-ENG-O exports post-action truth:
what actually happened, how it was verified, whether it reconciled,
what recovery path executed, and what the agent may safely believe next.

Durable Architectural Principles

To ensure reliability across tool-using agent implementations, systems must adhere to the following principles.

I. Decouple Execution from Verification

An execution response is an acknowledgment of an attempt, not proof of completed state change. The system must treat the observed tool response as evidence. Task state remains locked until verification confirms the resulting authoritative state.

II. Verify Predicates, Not Naive Equality

Authoritative state rarely equals intended state byte-for-byte. Verification should evaluate explicit predicates over resource ID, tenant scope, status, amount, version, actor, idempotency key, audit trail, and business-specific success conditions.

III. Bypass Lag and Caches During Verification

Verification reads must use source-of-record, writer/primary, strongly consistent, or approved verification endpoints. Stale replicas and caches can produce false failures or false confirmations.

IV. Represent Intermediate States Explicitly

Accepted, pending, partial, committed, compensated, rolled back, unknown, and review-required states must be represented explicitly. Flattening these states into success or failure corrupts planning and reporting.

V. Prevent Duplicate Side Effects with Idempotency

Timeouts and retries must not create duplicate mutations. Side-effecting actions require idempotency posture proportional to risk, and unknown mutating states must block blind retries until reconciled.

VI. Use Deterministic Recovery Policies

Failure recovery must be driven by policy-defined decision tables mapped to discrepancy classes, transaction boundaries, and side-effect classes. The model may assist with explanation, but it should not improvise rollback, compensation, or forward-recovery policy.

VII. Treat Unknown State as a First-Class State

Unknown is not success. Unknown is not failure. Unknown means the system cannot safely claim completion or repeat the mutation until reconciliation, review, or recovery establishes truth.

VIII. Ground Model Context in Reconciled State

The model’s next planning step must receive verified task state, unresolved discrepancy state, and trace pointers. It must not proceed from historical generation logs or assumed completion.

IX. Do Not Let User-Facing Status Outrun Truth

A user-facing success message may be emitted only after reconciliation. Pending, partial, compensated, failed, and unknown states must be reported honestly.

X. Make Verification Replayable and Auditable

Action verification must produce ledger entries, trace spans, policy versions, verification evidence, recovery decisions, and terminal states sufficient for audit, replay, incident response, and regression testing.

XI. Verification Logic Is Release Logic

Changes to verification predicates, recovery tables, ledger schemas, or polling policies must pass replay, canary, failure-injection, and compatibility gates before production promotion.

The final invariant:

The agent may act only through contracts.
The system may believe only verified state.
The user may be told only reconciled truth.

Attribution

Part of Stunspot’s Guide to AI Systems — The AI Engineering Systems Canon.

Created by Sam “stunspot” Walker / Collaborative Dynamics.

Repository: https://github.com/Stunspot/stunspots-guide-to-ai-systems
Stunspot: https://stunspot.com
Collaborative Dynamics: https://www.collaborative-dynamics.com
Discord: https://discord.gg/stunspot

Licensed under CC BY-NC-SA 4.0 unless otherwise stated.
Commercial use, resale, paid redistribution, inclusion in commercial training products, or incorporation into paid knowledge-base products requires prior written permission.

← Back to Canon Map