Custom AI Agent Development: Architecture, Scope, and Acceptance

Custom AI agent development is not “add a model to an app.” It is the design of a software system that can interpret a task, choose tools, read and change state, and decide what to do next within explicit limits. The model matters, but so do identity, permissions, tool contracts, evaluation, observability, and recovery.

That distinction changes the buying question. Before asking which framework or model to use, determine whether the work needs model-directed action at all. Then turn the business process into a testable requirements contract. A good custom agent is not defined by an impressive demo; it is defined by which work it completes, which actions it cannot take, how failures surface, and what the owner receives at handover.

This article owns the product and engineering specification: what to build and how to know it is ready. If you are comparing agencies, freelancers, contract structure, or a paid pilot, use the companion guide to AI agent development services.

First decide whether the work needs an agent

Anthropic distinguishes workflows, where code determines the execution path, from agents, where a model dynamically directs its process and tool use. Its guidance is deliberately conservative: begin with the simplest system that works, because agentic flexibility usually adds cost and latency (Anthropic, “Building effective agents”).

Use a deterministic workflow when:

• inputs and valid states are known;
• rules can describe the routing;
• exceptions are limited and enumerable;
• the same steps should run in the same order;
• a script, form, or automation already produces the required result.

Consider an agent when:

• requests arrive in varied, unstructured forms;
• the system must inspect evidence before choosing a tool or next step;
• tool results change the plan;
• exceptions cannot be encoded economically as a decision tree;
• stopping, asking for clarification, or escalating is part of correct behavior.

An LLM call is not automatically an agent. A support classifier that returns a label may need a model but not autonomous tool selection. A system that investigates an issue, reads account history, chooses a remedy, prepares a response, and requests approval before changing a record is agentic.

The strongest pre-build question is: what decision must the model make that ordinary software should not? If the answer is unclear, prototype the workflow before commissioning a production agent.

Write a requirements contract, not a feature wish list

“An agent that handles customer operations” is not implementable scope. A requirements contract should let a product owner, engineer, security reviewer, and evaluator recognize the same system.

Specify outcomes rather than UI activity. “Open the CRM and update the opportunity” is a procedure. “The approved opportunity record contains the verified meeting outcome, with an attributable audit entry and no change to billing fields” is an outcome that can be tested.

Include negative requirements. The agent must not invent missing fields, follow instructions found in untrusted documents, send externally without approval, or retry a write indefinitely. These constraints become both policy checks and evaluation cases.

A production architecture has more than a model

A useful design review should expose the following layers:

1. Task interface: where requests arrive and how identity is established.
2. Orchestrator or harness: the loop that supplies context, routes tool calls, enforces limits, and returns a result.
3. Model policy: model selection, prompts, reasoning settings, fallbacks, and version controls.
4. Tool layer: narrow interfaces for search, retrieval, computation, and actions.
5. State layer: run state, business records, approved long-term memory, and checkpoints.
6. Policy layer: authentication, authorization, validation, approvals, and hard prohibitions.
7. Evaluation and observability: traces, outcome checks, cost and latency measures, alerts, and review queues.
8. Execution boundary: sandboxing, network rules, secrets, deployment environments, and recovery.

A framework can provide parts of the harness, but a framework name is not an architecture. Ask what remains visible and portable underneath it: prompts, tool schemas, traces, evaluation fixtures, state, and policy decisions. Anthropic notes that frameworks can accelerate setup while hiding prompts and responses behind abstraction, which makes incorrect assumptions harder to debug (Anthropic's architecture guidance).

Start with one agent unless evaluation proves that additional specialized agents improve the outcome. Multi-agent systems introduce more handoffs, duplicated context, coordination failures, latency, and traces to interpret.

Tools are contracts, not connectors

An agent can only act through the interfaces it receives. Each tool needs:

• a single clear purpose and precise input schema;
• the minimum data and actions required;
• authentication tied to a service or user identity;
• server-side validation of dangerous parameters;
• timeout, rate-limit, and retry behavior;
• idempotency for writes that might repeat;
• an explicit result shape, including partial failure;
• a test double or sandbox for evaluation.

Separate read tools from write tools. “CRM access” is too broad; reading an account, drafting a note, applying an approved note, changing ownership, and issuing a refund should be different capabilities. The application—not a natural-language prompt—should enforce hard limits such as maximum refund amount, allowed recipient domains, or forbidden directories.

Tool descriptions also affect performance. Anthropic's tool engineering guidance recommends testing whether agents understand each tool's purpose and expected usage, not merely whether the underlying API works (Anthropic, “Writing effective tools for agents”).

Separate run state, business state, and memory

“The agent needs memory” is too vague for a design.

• Run state records progress within one job: completed steps, tool results, approvals, and checkpoints.
• Business state lives in systems of record: tickets, orders, documents, or repository branches.
• Long-term memory stores selected information across runs: preferences, resolved conventions, or learned facts.

Each has different ownership and retention. Run state should support resuming without repeating side effects. Business state should change only through validated tools. Long-term memory needs rules for provenance, correction, deletion, and expiry. A transcript is useful for audit and debugging, but it should not silently become trusted memory.

Prefer source retrieval over copying every fact into a memory store. If the authoritative value is already in the CRM or policy repository, fetch it at task time. Persist only what has a clear owner and reason to outlive the run.

Permissions and security are part of the product

Agent risk depends on both what the model can infer and what the system lets it do. NIST's Generative AI Profile organizes lifecycle work around govern, map, measure, and manage, a useful structure for assigning risk owners and documenting controls (NIST AI 600-1).

Build an action-permission matrix:

OpenAI's business guidance likewise recommends restricted data sources, confirmation before real-world actions, refusal of high-risk actions, resistance to malicious prompts, and audit trails (OpenAI's business guide to agents).

Test prompt injection in retrieved pages and files, malicious tool output, data exfiltration attempts, privilege escalation, and approval bypass. Use separate environments and credentials. Secrets should never be embedded in prompts or logs. For regulated or sensitive work, the business owner, privacy, legal, and security functions—not the development vendor alone—must approve the deployment boundary.

Evaluation should begin before implementation

An agent can say it completed a task when the required state never changed. Evaluate the outcome, the trajectory, and the operating cost.

Anthropic defines an agent trial as a task run whose transcript includes outputs, tool calls, intermediate results, and state changes. Its evaluation guidance recommends combining code-based checks, model-based grading, and human review, while reading traces rather than trusting one aggregate score (Anthropic, “Demystifying evals for AI agents”).

Create a versioned test set from:

• normal successful cases;
• ambiguous or incomplete requests;
• tool timeouts, malformed responses, and rate limits;
• duplicate events and interrupted runs;
• untrusted instructions in retrieved content;
• actions that require approval;
• cases the agent must refuse or escalate;
• previous production failures.

For each task, define the starting environment, allowed tools, success state, forbidden changes, maximum resource budget, and grading method. Run repeated trials where output variability matters. Track task success, unauthorized-action rate, escalation quality, human intervention, latency, token and tool cost, and recovery success.

Acceptance should use thresholds by task and risk class, not a single universal “accuracy” number. A drafting workflow can tolerate a visible mistake; an agent changing financial records cannot.

Define the delivery package before development starts

Source code alone is not a handover. A production-ready custom agent should deliver:

Also document exclusions: data cleanup, user interface work, compliance certification, penetration testing, third-party subscriptions, ongoing model usage, and post-launch support may not be included unless named.

Deployment must assume partial failure

Agent runs are longer and more stateful than ordinary requests. A browser can change, a model call can time out, a tool can succeed while its response is lost, or a human approval can arrive hours later.

Production design should include:

• bounded retries with backoff;
• idempotency keys for external writes;
• durable checkpoints before consequential actions;
• explicit states such as pending approval, blocked, failed, and completed;
• cancellation and timeout behavior;
• a dead-letter or manual-review queue;
• release versioning for model, prompt, tools, and policy;
• rollback for software and a compensating-action plan for business changes;
• alerts based on outcomes and unauthorized behavior, not only server uptime.

Recovery must be tested. Ask the team to interrupt a run after a write, restore a previous release, rotate a credential, replay a failed event, and show that the system neither loses work nor duplicates an action.

Build, buy, or choose a hybrid

Do not equate “custom” with “from scratch.” A hybrid often preserves commodity capabilities—identity, model access, workspace, scheduling, or observability—while custom code implements the proprietary decision, tool, or control. Build only the layer that produces differentiated value or satisfies a boundary the product cannot.

Prototype before committing to irreversible architecture. The prototype should answer the highest-risk question: can the model make the required decision, can the tool perform the action safely, or can the outcome be evaluated reliably?

How to accept a vendor-built agent

This is where engineering acceptance meets the broader build, buy, or hybrid decision. Contracting decides whom to hire and how to structure the engagement. Technical acceptance decides whether the delivered system is yours to operate.

Before sign-off, require the supplier to demonstrate:

1. the agreed evaluation suite running against the release candidate;
2. correct refusals, escalations, and approval pauses;
3. scoped credentials and attributable audit logs;
4. recovery from timeout, duplicate event, partial tool failure, and model unavailability;
5. cost and latency from representative workloads;
6. versioned prompts, models, tools, policy, and deployment configuration;
7. export of traces, evaluation data, state, and source under the agreed ownership terms;
8. a rollback exercise and an owner-led runbook drill.

Do not accept “human in the loop” without specifying who, at which action, with what evidence, and what happens if no one responds. Do not accept “monitoring” that shows only uptime while hiding task failures. Do not accept a demo dataset as the production acceptance set.

The practical decision

Custom AI agent development makes sense when a validated business task needs model-directed decisions, existing products cannot meet the integration or control boundary, and the owner is prepared to operate the result as a changing software system.

The development plan should begin with a requirements contract and evaluation cases, not a framework choice. The architecture should make tools, state, permissions, and failure recovery visible. The final delivery should include the evidence and operational assets needed to change models, debug incidents, and resume work without the original builder.

If the workflow is still unclear, configure or prototype first. If the workflow is standard, buy. If only one layer is proprietary, use a hybrid. Build the full custom system only when its control and differentiation justify owning every layer.