Evaluating LLM agents: building the design phase right

You built an agent and it works, sometimes. Visualisation, tracing and unit tests are what turn that into something you can change without fear.

Published: 15 Jan 2026
Reading: 9 min
Tags: agents
Originally: Codemancers

You've built your LLM agent. It works... sometimes. But how do you know it's working? How do you catch regressions before they hit production?

Welcome to the design phase of agent development. In this guide we build a real agent from scratch and walk through the complete workflow: visualisation, tracing, and testing.

What we're building

An Order Status Agent: a bot that checks order status and handles cancellations. Straightforward enough to understand, complex enough to need proper testing.

The flow:

User Query → Classify Intent → Check Status OR Cancel Order → Generate Response

Step 1: build the LangGraph agent

First, the agent structure. LangGraph gives us a graph with conditional routing based on user intent.

src/agent.ts

import { StateGraph, START, END, MemorySaver } from '@langchain/langgraph';
import { z } from 'zod/v4';
 
// Define the state schema
const OrderState = z.object({
  userQuery: z.string(),
  intent: z.enum(['check_status', 'cancel_order', 'unknown']),
  orderId: z.string().optional(),
  orderStatus: z.string().optional(),
  response: z.string(),
});
 
type OrderStateType = z.infer<typeof OrderState>;
 
// Node: Classify user intent
function classifyIntent(state: OrderStateType): Partial<OrderStateType> {
  const query = state.userQuery.toLowerCase();
 
  if (query.includes('cancel')) {
    return { intent: 'cancel_order' };
  } else if (query.includes('status') || query.includes('where')) {
    return { intent: 'check_status' };
  }
  return { intent: 'unknown' };
}
 
// Node: Extract order ID from query
function extractOrderId(state: OrderStateType): Partial<OrderStateType> {
  const match = state.userQuery.match(/ORD-\d+/i);
  return { orderId: match ? match[0].toUpperCase() : undefined };
}
 
// Node: Check order status (mock implementation)
function checkStatus(state: OrderStateType): Partial<OrderStateType> {
  // In production, this would call your order service
  const mockStatuses: Record<string, string> = {
    'ORD-123': 'Shipped - Arriving tomorrow',
    'ORD-456': 'Processing - Expected ship date: Jan 20',
    'ORD-789': 'Delivered',
  };
 
  const status = state.orderId
    ? mockStatuses[state.orderId] || 'Order not found'
    : 'No order ID provided';
 
  return { orderStatus: status };
}
 
// Node: Cancel order (mock implementation)
function cancelOrder(state: OrderStateType): Partial<OrderStateType> {
  if (!state.orderId) {
    return { orderStatus: 'Cannot cancel: No order ID provided' };
  }
  return { orderStatus: `Order ${state.orderId} has been cancelled` };
}
 
// Node: Generate final response
function generateResponse(state: OrderStateType): Partial<OrderStateType> {
  if (state.intent === 'unknown') {
    return {
      response:
        'I can help you check order status or cancel orders. Please include your order ID (e.g., ORD-123).',
    };
  }
  return { response: state.orderStatus || 'Unable to process your request' };
}
 
// Edge: Route based on intent
function routeByIntent(state: OrderStateType): string {
  switch (state.intent) {
    case 'check_status':
      return 'checkStatus';
    case 'cancel_order':
      return 'cancelOrder';
    default:
      return 'generateResponse';
  }
}
 
// Build the graph
const createOrderGraph = () => {
  return new StateGraph(OrderState)
    .addNode('classifyIntent', classifyIntent)
    .addNode('extractOrderId', extractOrderId)
    .addNode('checkStatus', checkStatus)
    .addNode('cancelOrder', cancelOrder)
    .addNode('generateResponse', generateResponse)
    .addEdge(START, 'classifyIntent')
    .addEdge('classifyIntent', 'extractOrderId')
    .addConditionalEdges('extractOrderId', routeByIntent, {
      checkStatus: 'checkStatus',
      cancelOrder: 'cancelOrder',
      generateResponse: 'generateResponse',
    })
    .addEdge('checkStatus', 'generateResponse')
    .addEdge('cancelOrder', 'generateResponse')
    .addEdge('generateResponse', END);
};
 
// Export for use
export const graph = createOrderGraph().compile();
export { createOrderGraph, OrderState };

Now we have a working agent. But how do we know it actually works correctly?

Step 2: visualise with LangGraph Studio

Before diving into tests and traces, watch the agent run. LangGraph Studio gives you a visual interface to step through the graph.

Install the CLI:

npm install -g @langchain/langgraph-cli

Create a langgraph.json config in the project root:

langgraph.json

{
  "dependencies": ["."],
  "graphs": {
    "order_agent": "./src/agent.ts:graph"
  },
  "env": ".env"
}

Create a .env:

.env

LANGSMITH_API_KEY=lsv2_your_api_key_here

Launch the dev server:

langgraph dev

Studio then opens at https://smith.langchain.com/studio/?baseUrl=http://127.0.0.1:2024.

What you get:

Watch nodes light up as they execute, in real time.
Inspect state at each step: exactly what intent was classified, what orderId was extracted.
Hot-reload your code and see changes immediately.
Re-run from any checkpoint, testing different paths without starting over.

Try running "Where is my order ORD-123?" and watch the flow go classifyIntent → extractOrderId → checkStatus → generateResponse.

This visual feedback is invaluable for understanding why your agent behaves a certain way, before you start writing tests.

Step 3: set up tracing with LangSmith

Studio is great for development, but you need traces to understand production behaviour. LangSmith captures every execution as a detailed trace.

export LANGCHAIN_TRACING_V2=true
export LANGCHAIN_API_KEY=your-api-key
export LANGCHAIN_PROJECT=order-agent

Every invocation is now captured with:

Input/output: the full request and response.
Latency: how long each node took.
Node execution order: the path through your graph.
State changes: what changed at each step.

Step 4: write unit tests with Vitest

Fast, deterministic tests are what make a CI pipeline worth having.

npm install -D vitest

Test the full agent

tests/agent.test.ts

import { test, expect } from 'vitest';
import { MemorySaver } from '@langchain/langgraph';
import { createOrderGraph, OrderState } from '../src/agent';
 
test('check order status for valid order', async () => {
  const graph = createOrderGraph();
  const checkpointer = new MemorySaver();
  const compiledGraph = graph.compile({ checkpointer });
 
  const result = await compiledGraph.invoke(
    {
      userQuery: 'What is the status of ORD-123?',
      intent: 'unknown',
      response: '',
    },
    { configurable: { thread_id: '1' } }
  );
 
  expect(result.intent).toBe('check_status');
  expect(result.orderId).toBe('ORD-123');
  expect(result.response).toContain('Shipped');
});
 
test('handle cancellation requests', async () => {
  const graph = createOrderGraph();
  const checkpointer = new MemorySaver();
  const compiledGraph = graph.compile({ checkpointer });
 
  const result = await compiledGraph.invoke(
    {
      userQuery: 'Please cancel ORD-456',
      intent: 'unknown',
      response: '',
    },
    { configurable: { thread_id: '2' } }
  );
 
  expect(result.intent).toBe('cancel_order');
  expect(result.response).toContain('cancelled');
});

Test individual nodes

LangGraph exposes each node via graph.nodes, so you can test them in isolation:

test('classifyIntent detects cancellation requests', async () => {
  const graph = createOrderGraph();
  const compiledGraph = graph.compile();
 
  // Test the classifyIntent node directly
  const result = await compiledGraph.nodes['classifyIntent'].invoke({
    userQuery: 'I want to cancel my order',
    intent: 'unknown',
    response: '',
  });
 
  expect(result.intent).toBe('cancel_order');
});
 
test('extractOrderId extracts valid order IDs', async () => {
  const graph = createOrderGraph();
  const compiledGraph = graph.compile();
 
  const result = await compiledGraph.nodes['extractOrderId'].invoke({
    userQuery: 'Check ORD-789 status',
    intent: 'check_status',
    response: '',
  });
 
  expect(result.orderId).toBe('ORD-789');
});

Test partial execution

For complex graphs, test specific sections using updateState and interruptAfter:

test('execute only status check path', async () => {
  const graph = createOrderGraph();
  const checkpointer = new MemorySaver();
  const compiledGraph = graph.compile({ checkpointer });
 
  // Simulate state as if we've already classified and extracted
  await compiledGraph.updateState(
    { configurable: { thread_id: '3' } },
    {
      userQuery: 'Check my order',
      intent: 'check_status',
      orderId: 'ORD-123',
      response: '',
    },
    'extractOrderId' // State as if extractOrderId just completed
  );
 
  // Resume and stop after checkStatus
  const result = await compiledGraph.invoke(null, {
    configurable: { thread_id: '3' },
    interruptAfter: ['checkStatus'],
  });
 
  expect(result.orderStatus).toBe('Shipped - Arriving tomorrow');
});

The complete design phase

1. BUILD      → Create your LangGraph agent
2. VISUALIZE  → Debug in LangGraph Studio
3. TRACE      → Capture executions with LangSmith
4. TEST       → Write Vitest unit tests for CI/CD
5. ITERATE    → Fix issues, repeat

Each tool serves a different purpose:

| Tool | Purpose | When to use | | --- | --- | --- | | LangGraph Studio | Visual debugging | Development, understanding flow | | LangSmith traces | Observability | Production monitoring, debugging | | Vitest | Regression testing | CI/CD, fast feedback |

The design phase isn't about achieving perfection on day one. It's about building the infrastructure to iterate towards it. With visualisation, tracing and testing in place, you're not flying blind any more.

Resources

From state to edges: how LangGraph connects the dots→