43.1 Introduction
The previous chapter established the AI orchestration layer for controlled model inference.
The next architectural step is the AI agent layer.
A conventional AI request normally follows:
User → Model → Response
An agentic system can instead follow:
Goal → Plan → Retrieve → Reason → Propose Action → Verify → Execute → Observe → Continue
This additional capability creates substantial engineering and security requirements.
An AI agent should therefore not be designed as:
“Give the model access to everything and let it decide what to do.”
Instead, the application should define explicit boundaries around:
- what the agent can see,
- what the agent can remember,
- what tools it can call,
- what parameters it can provide,
- what actions require approval,
- what actions are prohibited,
- how long a task may run,
- how many steps are allowed,
- and how every action is recorded.
The fundamental architecture is:
User Goal
↓
Agent Orchestrator
↓
Planner
↓
Task State
↓
Tool Authorization
↓
Tool Proposal
↓
Validation
↓
Approval Policy
↓
Tool Execution
↓
Observation
↓
Verifier
↓
Next Step / Completion
43.2 What Is an AI Agent?
For this architecture, an AI agent is a system in which a model can participate in a multi-step workflow by:
- interpreting a goal,
- creating or selecting a plan,
- requesting information,
- proposing tool operations,
- receiving tool results,
- evaluating those results,
- and continuing until the task reaches a defined completion condition.
The model itself is not the entire agent.
A safer definition is:
Agent
=
Model
+
Orchestrator
+
State
+
Tools
+
Policies
+
Verification
This distinction is extremely important.
The application remains responsible for authorization and execution.
43.3 Agent Versus Chatbot
A chatbot might perform:
Question
↓
Answer
An agent might perform:
Goal
↓
Plan
↓
Retrieve Information
↓
Analyze
↓
Prepare Action
↓
Request Approval
↓
Execute
↓
Verify
The second architecture requires significantly more controls.
43.4 Agent Architecture
A complete agent subsystem can be organized as:
AGENT SYSTEM
│
┌───────────┴───────────┐
▼ ▼
Agent Session Agent Policy
│ │
▼ ▼
Planner Permissions
│ │
▼ │
Task State │
│ │
▼ │
Tool Selection ◄──────────────┘
│
▼
Tool Proposal
│
▼
Validation
│
▼
Approval Check
│
▼
Tool Execution
│
▼
Observation
│
▼
Verifier
│
└──────────→ Next Step
43.5 Agent Session
Every agent workflow should have an explicit session.
Conceptual structure:
AgentSession
├── id
├── userId
├── projectId
├── taskId
├── agentType
├── status
├── createdAt
└── updatedAt
Possible statuses include:
CREATED
RUNNING
WAITING_APPROVAL
PAUSED
COMPLETED
FAILED
CANCELLED
EXPIRED
This makes long-running workflows manageable.
43.6 Agent Task
A task represents the objective the agent is trying to accomplish.
Example:
Task:
"Summarize the uploaded research documents and prepare a structured report."
The task should have:
taskId
goal
userId
projectId
status
priority
createdAt
deadline
The goal should be treated as data associated with the task, not as an unrestricted instruction to the entire application.
43.7 Task State Machine
Agent tasks should use explicit state transitions.
Example:
CREATED
↓
PLANNING
↓
EXECUTING
↓
WAITING_APPROVAL
↓
EXECUTING
↓
VERIFYING
↓
COMPLETED
Failure can transition to:
FAILED
Cancellation can transition to:
CANCELLED
Invalid state transitions should be rejected by the application.
43.8 Why Explicit State Matters
Without persistent state, an agent can lose track of:
- what it already did,
- which tool was used,
- what result was returned,
- which step is awaiting approval,
- whether an operation already succeeded.
Persistent state allows the workflow to resume safely.
43.9 Agent Steps
A task can contain multiple steps.
For example:
Task
│
├── Step 1: Find relevant documents
├── Step 2: Extract evidence
├── Step 3: Compare findings
├── Step 4: Draft report
└── Step 5: Request review
Each step should have its own status.
Recommended states:
PENDING
RUNNING
WAITING
COMPLETED
FAILED
SKIPPED
CANCELLED
43.10 Planning
The planner converts a goal into an executable sequence.
Conceptually:
Goal
↓
Planner
↓
Plan
↓
Step 1
Step 2
Step 3
...
However, plans should not automatically authorize actions.
A plan is a proposal.
The application must independently enforce permissions.
43.11 Plan Validation
Before execution, the system should validate the plan.
Checks may include:
- permitted tools,
- project scope,
- data access,
- maximum steps,
- action sensitivity,
- approval requirements,
- resource limits.
Conceptually:
Generated Plan
↓
Policy Validator
↓
Allowed?
┌───┴───┐
Yes No
↓ ↓
Execute Reject/Revise
43.12 Bounded Planning
Agents should operate within explicit limits.
Example limits:
maxSteps
maxRuntime
maxToolCalls
maxRetrievalResults
maxTokens
maxConcurrentActions
This prevents an accidental loop from running indefinitely.
A bounded agent is easier to debug and safer to operate.
43.13 Tool Registry
Tools should be explicitly registered.
Conceptually:
ToolRegistry
├── searchDocuments
├── readDocument
├── summarizeDocument
├── createDraft
└── requestApproval
The model should only see tools that the current task is permitted to use.
43.14 Tool Definition
A tool should have metadata.
Example:
ToolDefinition
├── name
├── description
├── inputSchema
├── sensitivity
├── requiredPermission
├── requiresApproval
└── enabled
The input schema should be machine-validatable.
43.15 Tool Sensitivity Classes
A useful model is:
READ
WRITE
EXTERNAL_ACTION
ADMINISTRATIVE
For example:
Read document
→ READ
Create internal draft
→ WRITE
Send external message
→ EXTERNAL_ACTION
Change account permissions
→ ADMINISTRATIVE
Higher-risk categories should receive stronger controls.
43.16 Least Privilege
An agent should receive only the minimum capabilities required for its task.
For example:
Document Summarization Agent
may need:
READ_DOCUMENT
but does not need:
DELETE_DOCUMENT
CHANGE_USER_ROLE
This follows the principle of least privilege.
43.17 Tool Calling Lifecycle
A tool call should follow:
Model Proposal
↓
Parse
↓
Schema Validation
↓
Permission Check
↓
Resource Ownership Check
↓
Risk Classification
↓
Approval Check
↓
Execute
↓
Record Result
The model must never bypass these stages.
43.18 Tool Arguments Are Untrusted
The model may generate malformed or dangerous parameters.
Therefore:
Model Arguments
↓
Schema Validation
↓
Business Validation
↓
Authorization
↓
Execution
For example, a model proposing an operation on a project does not prove that the user owns or can access that project.
43.19 Resource-Level Authorization
Tool permissions should operate at both capability and resource levels.
For example:
Permission:
READ_DOCUMENT
is not sufficient.
The application must additionally verify:
Can this user read Document X?
Therefore:
Tool Permission
+
Resource Authorization
↓
Allowed Operation
43.20 Human Approval
Some operations should require explicit human approval.
Examples include:
- external communication,
- irreversible changes,
- sensitive data operations,
- account changes,
- high-impact actions.
The workflow becomes:
Agent Proposal
↓
Approval Required
↓
Human Review
↓
Approve / Reject
↓
Execution
The approval should be associated with the exact proposed action.
43.21 Approval Must Be Specific
An approval should not mean:
“The user approved everything this agent might do.”
Instead, approval should refer to a specific action.
For example:
Action:
Create draft report from Project X documents
Scope:
Project X
Requested by:
Agent Task 123
Status:
Pending Approval
This creates a much stronger authorization boundary.
43.22 Approval Expiration
Approvals should not remain valid forever.
An approval record can contain:
approvalId
taskId
stepId
requestedAt
expiresAt
approvedAt
approvedBy
status
If the task changes materially, a new approval may be required.
43.23 Dry-Run Mode
Agents should support a dry-run mode.
Instead of executing actions:
Agent
↓
Plan
↓
Tool Proposals
↓
Preview
The user can inspect the proposed workflow.
Example:
1. Read three documents.
2. Compare their findings.
3. Generate a draft.
4. No external action will be performed.
This is valuable during development and testing.
43.24 Simulation Mode
A stronger testing environment can simulate tools.
For example:
Real Tool
can be replaced by:
Mock Tool
The agent receives realistic-looking results without modifying real resources.
This enables safe evaluation of planning behavior.
43.25 Agent Memory
An agent may require memory across steps.
However, memory should be categorized.
Possible categories:
Task State
Conversation Context
Retrieved Evidence
Persistent User Memory
Tool Results
These should not be merged into one uncontrolled context store.
Each memory type should have different retention and authorization rules.
43.26 Tool Results as Untrusted Data
Tool output must also be treated as untrusted input.
A tool might return:
External Content
that contains instructions attempting to influence the agent.
Therefore:
Tool Result
does not automatically become:
Agent Instruction
The orchestrator should maintain the same instruction/data separation introduced in the RAG architecture.
43.27 External Data and Agent Hijacking
An agent may retrieve content from external sources.
That content can contain text designed to manipulate downstream model behavior.
The architecture should therefore assume:
External Data = Untrusted
The model can analyze the data, but the data should not automatically change:
- permissions,
- tool access,
- system policy,
- approval state,
- security configuration.
43.28 Agent Loop
A controlled agent loop can be represented as:
while task_not_complete:
observe_state()
create_or_select_next_step()
validate_step()
if approval_required:
wait_for_approval()
execute_authorized_action()
record_observation()
verify_result()
update_state()
The application—not the model—should control the loop.
43.29 Maximum Step Limit
Every agent task should have a maximum number of steps.
Example:
maxSteps = configured limit
When the limit is reached:
Task
↓
LIMIT_REACHED
↓
Pause / Fail Safely
This prevents infinite loops.
43.30 Maximum Tool Calls
Similarly:
maxToolCalls
should be tracked.
A model repeatedly requesting the same tool should not be allowed to consume unlimited resources.
The system can detect patterns such as:
same tool
+
same arguments
+
repeated failures
and terminate or pause the workflow.
43.31 Time Limits
Long-running agents require deadlines.
Example:
createdAt
deadline
When the deadline is exceeded:
RUNNING
↓
EXPIRED
The system should safely stop further execution.
43.32 Idempotency
Some tool operations may be retried.
If an operation is not idempotent, a retry could cause duplication.
For example:
Create Resource
could accidentally create two resources if the first operation succeeded but its response was lost.
Therefore, tool execution should support idempotency keys where appropriate.
Conceptually:
taskId + stepId + attempt
can help uniquely identify an operation.
43.33 Transactional Tool Execution
Where possible, state changes should be transactional.
For example:
Validate
↓
Authorize
↓
Execute
↓
Record Result
The application should define what happens if execution succeeds but result recording fails.
This requires careful consistency design.
43.34 Agent Observability
Every step should be observable.
A useful trace includes:
taskId
stepId
toolCallId
model
timestamp
duration
status
Additional metadata can include:
retrievalCount
tokenUsage
approvalStatus
retryCount
Avoid recording unnecessary sensitive content.
43.35 Agent Audit Trail
High-impact agent actions should create audit events.
Example:
AGENT_TASK_CREATED
AGENT_PLAN_CREATED
AGENT_TOOL_REQUESTED
AGENT_APPROVAL_REQUESTED
AGENT_APPROVAL_GRANTED
AGENT_APPROVAL_REJECTED
AGENT_TOOL_EXECUTED
AGENT_TASK_COMPLETED
AGENT_TASK_FAILED
This creates an operational history.
43.36 Agent Cancellation
Users should be able to cancel long-running tasks.
The system should support:
RUNNING
↓
CANCEL_REQUESTED
↓
STOPPING
↓
CANCELLED
The implementation should ensure that cancellation does not leave partially completed operations in an inconsistent state.
43.37 Agent Pause and Resume
Long-running tasks may need to pause.
Reasons include:
- human approval,
- temporary service outage,
- scheduled execution,
- resource constraints.
The state can become:
PAUSED
and later:
RESUMED
Persistent state is therefore essential.
43.38 Verification Layer
An agent should not blindly assume that a tool result means the task succeeded.
The verification layer can ask:
Did the action succeed?
Is the result complete?
Does it satisfy the task requirement?
Is the result consistent with expected state?
Conceptually:
Tool Result
↓
Verifier
↓
Valid?
┌──┴──┐
Yes No
↓ ↓
Next Retry/Recover
Step
43.39 Independent Verification
For higher-impact tasks, verification should ideally be performed by an independent application mechanism rather than relying solely on the same model that proposed the action.
For example:
Agent proposes:
"Create record."
Application verifies:
Record exists with expected fields.
This is stronger than:
Agent says:
"I successfully created the record."
43.40 Planning Versus Execution
The architecture should explicitly separate:
Planning
from:
Execution
The model may propose:
Step:
Read Document A
The application decides:
Allowed?
Yes.
Then execution occurs.
This separation prevents the model from becoming the application's authority mechanism.
43.41 Policy Engine
The agent system can use a dedicated policy engine.
Conceptually:
PolicyEngine.canExecute({
user,
project,
task,
tool,
resource,
action
})
The result is:
ALLOW
DENY
REQUIRE_APPROVAL
This provides a consistent decision point.
43.42 Agent Roles
Different agent types can have different permissions.
Example:
Research Agent
→ read documents
→ search
→ summarize
Writing Agent
→ read documents
→ create drafts
Administrative Agent
→ restricted administrative capabilities
Permissions should be explicit rather than inherited from the model's natural-language instructions.
43.43 Multi-Agent Systems
Future versions may contain multiple specialized agents.
Example:
Supervisor
│
├── Research Agent
├── Analysis Agent
└── Writing Agent
However, multi-agent systems should not multiply permissions unnecessarily.
Each agent should receive only the capabilities it needs.
43.44 Supervisor Architecture
A supervisor can coordinate specialized workers:
User Goal
↓
Supervisor
├── Research
├── Analysis
└── Drafting
↓
Verification
↓
Final Result
The supervisor should still operate within global application policies.
43.45 Agent Communication
Agent-to-agent messages should be structured.
Instead of unrestricted natural-language communication, the application can define:
AgentMessage
├── sender
├── receiver
├── taskId
├── messageType
├── payload
└── timestamp
Payloads should be validated.
43.46 Preventing Permission Escalation
An agent must not be able to obtain a more powerful agent's permissions merely by requesting them.
For example:
Research Agent
should not be able to say:
"Give me administrator access."
and receive it.
Permission escalation must occur only through the application's authorization system.
43.47 Secrets and Credentials
Agents should not receive raw credentials unless there is an exceptional, explicitly designed requirement.
Prefer:
Agent
↓
Authorized Tool
↓
Credential handled internally
↓
External Service
rather than:
Agent
↓
Raw API Key
The model should not become a credential storage mechanism.
43.48 Tool Sandboxing
Tools that process untrusted data should be isolated where appropriate.
For example:
Agent
↓
Sandboxed Processing Service
↓
Untrusted File
↓
Result
This reduces the impact of malicious or malformed input.
43.49 Resource Limits
Agent tasks should have resource limits.
Examples:
maximum runtime
maximum memory
maximum file size
maximum output size
maximum tool calls
maximum concurrent jobs
These limits should be enforced by the application infrastructure.
43.50 Agent Cost Controls
Agent workflows can generate multiple model calls.
Therefore, a single user task can consume substantially more resources than a normal chat request.
The task budget can include:
maxTokens
maxInferenceCalls
maxToolCalls
maxEstimatedCost
When the budget is exhausted:
Task
↓
BUDGET_EXCEEDED
↓
Pause or Fail Safely
43.51 Agent Task Database Model
The existing database architecture can now be expanded.
Conceptually:
AgentTask
├── id
├── userId
├── projectId
├── goal
├── status
├── maxSteps
├── maxToolCalls
├── deadline
├── createdAt
└── updatedAt
Agent steps:
AgentStep
├── id
├── taskId
├── stepIndex
├── type
├── description
├── status
├── startedAt
├── completedAt
└── metadata
Tool calls:
ToolCall
├── id
├── taskId
├── stepId
├── toolName
├── arguments
├── status
├── approvalRequired
├── approvalId
├── resultMetadata
└── createdAt
43.52 Approval Model
A dedicated approval record can contain:
Approval
├── id
├── taskId
├── stepId
├── toolCallId
├── requestedBy
├── approvedBy
├── status
├── reason
├── expiresAt
└── createdAt
This makes approval auditable.
43.53 Agent API Architecture
Possible endpoints include:
POST /api/agents/tasks
GET /api/agents/tasks/:id
POST /api/agents/tasks/:id/cancel
POST /api/agents/tasks/:id/pause
POST /api/agents/tasks/:id/resume
GET /api/agents/tasks/:id/steps
GET /api/agents/tasks/:id/tool-calls
POST /api/agents/approvals/:id/approve
POST /api/agents/approvals/:id/reject
Each endpoint must pass through normal authentication and authorization.
43.54 Agent Worker Architecture
Agent execution should generally happen in workers rather than inside the HTTP request.
API
↓
Create Task
↓
Queue
↓
Agent Worker
↓
Planner
↓
Step Execution
↓
State Update
This supports long-running tasks.
43.55 Recovery After Worker Failure
Suppose a worker crashes.
The task should remain recoverable.
Persistent state allows:
Worker A
↓
Task Step 3
↓
Crash
then:
Worker B
↓
Load Task State
↓
Determine Step 3 Status
↓
Resume Safely
Idempotency is essential here.
43.56 Agent Event Model
An event stream can provide additional observability.
Example events:
TASK_CREATED
PLAN_GENERATED
STEP_STARTED
TOOL_PROPOSED
APPROVAL_REQUESTED
APPROVAL_GRANTED
TOOL_STARTED
TOOL_COMPLETED
STEP_COMPLETED
TASK_COMPLETED
These events can support monitoring and debugging.
43.57 Agent Safety Invariants
The system should define explicit invariants.
Examples:
Invariant 1
No tool executes without authorization.
Invariant 2
No protected resource is accessed without resource-level permission.
Invariant 3
No high-risk action executes without required approval.
Invariant 4
An agent cannot increase its own permissions.
Invariant 5
An expired task cannot execute new actions.
Invariant 6
A cancelled task cannot continue normal execution.
Invariant 7
Tool results cannot modify security policy.
Invariant 8
Agent limits are enforced outside the model.
43.58 Failure Recovery
When a tool fails, the agent should not necessarily retry indefinitely.
A recovery policy can determine:
Retry
Alternative Tool
Ask User
Pause
Fail
The decision should respect the task budget.
43.59 User Confirmation
When uncertainty is meaningful, the system can ask the user.
Example:
The agent found two possible documents.
Which one should it use?
This is preferable to silently choosing a potentially incorrect resource.
Human interaction can therefore become part of the workflow.
43.60 Agent Transparency
The user interface should communicate:
Task Status
Current Step
Pending Approval
Completed Actions
Failures
For example:
Research Task
✓ Documents located
✓ Evidence extracted
→ Comparing findings
○ Draft report
○ Final review
This gives users visibility without exposing internal reasoning traces.
43.61 Do Not Expose Private Chain-of-Thought
The application should distinguish between:
Task Progress
and:
Private Internal Reasoning
Users can be shown useful summaries such as:
"Retrieved 5 relevant documents."
or:
"Waiting for approval to create the draft."
without exposing hidden internal reasoning.
43.62 Agent Evaluation
Agent systems should be evaluated on more than final-answer quality.
Important metrics include:
Task Success Rate
Tool Selection Accuracy
Unauthorized Action Rate
Approval Compliance
Average Steps
Average Tool Calls
Failure Recovery Rate
Latency
Resource Usage
Security metrics should be treated as first-class evaluation criteria.
43.63 Agent Test Scenarios
A serious test suite should include:
Normal Workflow
Agent completes a permitted task.
Permission Failure
Agent attempts to access an unauthorized resource.
Expected:
DENY
Approval Workflow
Agent requests a sensitive action.
Expected:
WAITING_APPROVAL
Tool Failure
A tool becomes unavailable.
Expected:
Controlled Recovery
Infinite Loop
Agent repeatedly proposes the same action.
Expected:
Step/Tool Limit Reached
Malicious External Content
Retrieved data contains instructions attempting to control the agent.
Expected:
Content remains untrusted data.
43.64 Agent Red-Team Evaluation
Agent security testing should evaluate the complete system rather than only the language model.
Test categories can include:
Prompt Manipulation
Indirect Instructions
Permission Boundary Testing
Tool Argument Manipulation
Cross-Project Access
Approval Bypass Attempts
Loop Abuse
Resource Exhaustion
Data Leakage
Testing should remain within controlled environments.
43.65 Agent Security Architecture
The complete secure flow becomes:
USER
│
▼
AGENT TASK
│
▼
AUTHORIZATION
│
▼
PLANNER
│
▼
PLAN VALIDATOR
│
▼
STEP STATE
│
▼
TOOL PROPOSAL
│
▼
SCHEMA VALIDATOR
│
▼
RESOURCE AUTHZ
│
▼
POLICY ENGINE
│
┌────────┴────────┐
▼ ▼
ALLOW REQUIRE APPROVAL
│ │
│ HUMAN
│ │
│ APPROVE / DENY
│ │
└────────┬────────┘
▼
TOOL EXECUTION
│
▼
OBSERVATION
│
▼
VERIFICATION
│
▼
STATE UPDATE
│
▼
NEXT STEP / COMPLETE
43.66 Recommended Agent Design Philosophy
The central philosophy should be:
The model proposes; the application decides; the tool executes; the verifier checks.
This provides a clean division of responsibility.
Model
Understands and proposes.
Orchestrator
Controls workflow.
Policy Engine
Authorizes.
Tool Layer
Executes.
Verifier
Checks results.
Audit Layer
Records what happened.
43.67 Chapter Summary
This chapter transformed the AI architecture from a simple inference system into a controlled agent framework.
The system now contains:
Agent Sessions
Agent Tasks
Agent Steps
Planning
Tool Registry
Tool Validation
Resource Authorization
Approval Workflows
Task Budgets
State Machines
Verification
Cancellation
Pause/Resume
Audit Events
Worker Execution
Failure Recovery
The most important security boundary is:
A model-generated plan is not an authorization decision.
The agent may propose:
"Perform action X."
but the application must independently determine:
Is X permitted?
Is the resource accessible?
Does X require approval?
Is the task within its limits?
Should the action execute?
The resulting architecture is therefore:
Goal → Plan → Validate → Authorize → Approve → Execute → Verify → Record
This establishes the foundation for the next subsystem: AI safety evaluation, red-team testing, adversarial robustness, prompt-injection testing, multilingual and multimodal safety, and measurable security benchmarks.
END OF CHAPTER 43
Implementation snippet — tool definition
export type ToolRisk = "READ" | "WRITE" | "EXTERNAL_ACTION" | "ADMINISTRATIVE";
export type ToolDefinition = {
name: string;
description: string;
risk: ToolRisk;
requiredPermission: string;
requiresApproval: boolean;
enabled: boolean;
inputSchema: unknown;
};
Implementation snippet — policy decision
type PolicyDecision = "ALLOW" | "DENY" | "REQUIRE_APPROVAL";
export async function authorizeToolCall(input: {
userId: string;
projectId: string;
tool: ToolDefinition;
}): Promise {
const hasPermission = await permissionService.hasPermission(
input.userId,
input.tool.requiredPermission
);
if (!hasPermission) {
return "DENY";
}
if (input.tool.requiresApproval) {
return "REQUIRE_APPROVAL";
}
return "ALLOW";
}
Implementation snippet — bounded agent loop
export async function executeAgentTask(taskId: string) {
const task = await taskRepository.get(taskId);
if (!task) {
throw new Error("Task not found");
}
if (task.status === "CANCELLED" || task.status === "EXPIRED") {
return;
}
if (task.stepCount >= task.maxSteps) {
await taskRepository.markFailed(taskId, "MAX_STEPS_REACHED");
return;
}
const nextStep = await planner.createNextStep(task);
const decision = await policyEngine.evaluate(nextStep);
if (decision === "DENY") {
await taskRepository.markFailed(taskId, "POLICY_DENIED");
return;
}
if (decision === "REQUIRE_APPROVAL") {
await approvalService.request(taskId, nextStep);
return;
}
await executor.run(nextStep);
await taskRepository.markStepCompleted(taskId, nextStep.id);
}
This article was originally published by DEV Community and written by Black Shadow Team ©.
Read original article on DEV Community