Testing & evals

Save a real conversation as a scenario, replay it against an assistant — with its current instructions or a candidate rewrite — and score how stable the answers are across repeated passes. That's the Testing & evals connector (?domain=sales_evals): seven tools for regression-testing a sales assistant the way you'd regression-test code.

You want to…	Call
Save a conversation as a replayable test	eval_scenario_save — from a real chat, the test chat, or inline messages
See / inspect / remove saved tests	eval_scenario_list / eval_scenario_get / eval_scenario_delete
Replay once against the live assistant	eval_run_start with the defaults (runs: 1, judge: "none")
Try a candidate prompt without touching the live assistant	eval_run_start with instruction_override
Measure answer stability	eval_run_start with runs: 2–5 and judge: "similarity"
Watch progress & read results	eval_run_get — poll it; partial results while running
Stop a run	eval_run_cancel — best effort, between turns

Every parameter of every tool is in the Sales tools reference → Testing & evals.

Three facts to keep in mind:

• Eval traffic is free. Every turn runs as test-chat traffic — real tokens are recorded as test-chat usage, never billed as customer messages. Replay a 30-turn conversation five times and your message balance doesn't move.
• Runs are async. eval_run_start returns a run_id immediately and the run executes in the background — poll eval_run_get for progress and results.
• One run at a time per account. Starting a second run while one is queued or running returns an error naming the active run — poll it or cancel it first.

Where it runs

Like every Sales Platform area, Testing & evals is available two ways:

• To an agent — connect the Testing & evals card in the From your sales platform section of Agents → Connectors, then toggle it on for the agent. Now you can just ask: "replay chat 8412 against this new prompt and tell me if the answers hold up."
• From Claude Code / Cursor — the same tools as an MCP server (click Connect AI tools on the card for your copyable config):

https://api.mychatbot.app/api/mcp/sales-management?account_id=<account_id>&domain=sales_evals

See MCP & connectors for the access key and client setup.

The workflow

The loop is always the same four steps:

1. Capture. Take a conversation worth protecting — a real chat where the assistant did well (or fumbled), or a test-chat session you just ran — and save it as a scenario with eval_scenario_save. Only the user turns are kept; the assistant's side is what a run produces. Optionally add reference answers — what the assistant should say per turn — to unlock scoring against them.
2. Replay. eval_run_start with the defaults is a plain replay: one pass over the turns against the assistant as currently configured, recording every response and every tool call.
3. Compare a candidate. Pass instruction_override with a rewritten system prompt to run the same turns against the candidate text. The live assistant is never modified — customers keep talking to the current version while you test. (skills_override does the same for skills: pass a slug list to pin the selection, [] to run with no skills at all.)
4. Check stability, then iterate. Answers you liked once can still be flaky. Set runs: 3 (up to 5) plus judge: "similarity" and every iteration replays the turns as an independent conversation; the judge scores how consistently the assistant answered each turn across iterations — and, when the scenario has reference answers, how close each answer came to them. Edit the prompt, re-run, repeat. When the candidate wins, ship it with assistant_update_instructions.

Scenarios are the regression suite

A scenario is saved per account and replayable forever — eval_scenario_save upserts by name, so "delivery-pricing" always means the latest version of that test. Build a handful of scenarios for the conversations that matter (pricing objections, delivery questions, refund policy) and re-run them after every instruction change.

Worked example: fix a fumbled delivery answer

The assistant fumbled a delivery-cost conversation. Capture that chat, test a rewritten prompt against it, and check the fix is stable — without customers ever seeing the draft.

1. Save the chat as a scenario. Take the chat id from list_chats (Conversations connector) and add reference answers for the turns you care about:

// eval_scenario_save
{
  "name": "delivery-pricing",
  "description": "Customer asks about delivery cost and timing; bot must quote 80 UAH and 1-2 days",
  "assistant_id": "42",
  "from_chat_id": "8412",
  "reference": [
    "",
    "Delivery is 80 UAH by Nova Poshta, 1-2 business days.",
    ""
  ]
}

{
  "scenario_id": 17,
  "name": "delivery-pricing",
  "turns": 3,
  "updated": false,
  "message": "Scenario saved. Run it with eval_run_start{scenario_id}."
}

2. Start a run with the candidate prompt. Three iterations, similarity judge, and the rewritten instructions in instruction_override:

// eval_run_start
{
  "assistant_id": "42",
  "scenario_id": 17,
  "runs": 3,
  "judge": "similarity",
  "instruction_override": "You are the sales assistant for Kvitka Flowers…\n\nDelivery: always quote 80 UAH via Nova Poshta, 1-2 business days. Never say 'it depends' — if the customer's city is unusual, quote the standard rate and offer to confirm."
}

{
  "run_id": 41,
  "status": "queued",
  "turns": 3,
  "runs": 3,
  "judge": "similarity",
  "message": "Eval run started in the background. Poll eval_run_get{run_id} for progress and results."
}

3. Poll. While the run executes, eval_run_get {"run_id": 41} returns status: "running" plus the turns completed so far (with a note that results are partial). When it finishes:

{
  "run_id": 41,
  "assistant_id": 42,
  "status": "done",
  "runs": 3,
  "judge": "similarity",
  "instruction_override_used": true,
  "source": { "type": "scenario", "scenario_id": 17, "turns": 3 },
  "iterations": [
    {
      "iteration": 1,
      "turns": [
        {
          "turn": 2,
          "user_message": "how much is delivery to Lviv?",
          "response": "Delivery to Lviv is 80 UAH via Nova Poshta and takes 1-2 business days.",
          "tool_calls": [
            { "name": "get_faq", "result_preview": "Delivery: Nova Poshta, 80 UAH…", "result_bytes": 412 }
          ]
        }
      ]
    }
  ],
  "metrics": {
    "judge": "similarity",
    "per_turn": [
      { "turn": 1, "consistency": 0.91 },
      { "turn": 2, "consistency": 0.88, "vs_reference": 0.83 },
      { "turn": 3, "consistency": 0.79 }
    ],
    "overall_consistency": 0.86,
    "overall_vs_reference": 0.83
  },
  "created_at": "2026-08-07T09:14:02Z",
  "finished_at": "2026-08-07T09:16:47Z"
}

(Transcripts abbreviated — you get every turn of every iteration, and include_tool_details: true adds each tool call's arguments.)

4. Read the metrics. Scores are 0–1, higher = more similar:

• consistency — how alike the three iterations' answers to that turn were (mean pairwise token overlap). A low turn is where the assistant improvises — tighten the instructions there.
• vs_reference — how close the answers came to your reference for that turn. Only turns with a non-empty reference get one.
• The overall_* values average the per-turn scores; consistency needs runs ≥ 2 to exist at all.

Turn 2 answers 0.83 against the reference and holds 0.88 consistency — the fix works and holds up. If it looks good, apply the candidate for real with assistant_update_instructions, and keep delivery-pricing in the suite to re-run after the next prompt change.

The judge is deterministic

similarity is token-overlap math (language-agnostic — Ukrainian works as well as English), not an LLM grader. It costs nothing extra and the same transcripts always score the same. Treat it as a stability and drift signal, and read the transcripts for quality judgments.

What runs for real during a run

Every turn of a run is a test conversation, so it follows the test-chat rule: reads run live, writes are simulated.

• Knowledge, product and web searches, CRM and calendar lookups run against live data, so a scenario can check that the assistant grounds its answer in what the business actually has.
• Tools that write — bookings, CRM deals and leads, orders, payment links, pipeline status and labels, operator handoff — are simulated. The transcript shows the tool answering "Simulated in test mode: … nothing was created or changed", and the assistant continues as if the write had worked. Nothing reaches the connected system.
• Tools from a connected MCP server run only if the server marks them read-only (the standard readOnlyHint). Any other MCP tool is not executed: the assistant receives an error saying the source is unavailable in test mode and is expected to tell the user so. A scenario that needs such a tool will show that error in tool_calls, not fabricated data.

So a scenario that validates a docs-grounded or catalog-grounded answer works as long as the knowledge source is one of the platform's own or an MCP server that declares its reads. If a run shows "not available in test mode" for a source you expected to be read, the server behind it does not declare the hint; ask its maintainer to add it and reopen the server's tool list under Integrations so the platform records it.

Limits & mechanics

Turns per scenario / run	max 30 user turns — longer chat sources are cut to the first 30 (flagged truncated); longer inline lists are rejected
Iterations per run	runs 1–5, default 1; each iteration is an independent conversation from a clean history
Concurrency	one queued-or-running run per account
Run deadline	15 minutes wall clock — a run that exceeds it flips to error, completed turns retained
Statuses	queued → running → done / error / cancelled
Cancel	best effort between turns — the current turn finishes first; completed turns stay readable
Cost	free — recorded as test-chat usage, never billed as customer messages
Results size	transcripts are capped (~2 MB per run); oversize results are trimmed with an explicit results_truncated marker, never silently
Deleting things	deleting a scenario keeps past runs' transcripts readable

See also

• Sales tools reference → Testing & evals — all seven tools with full parameters
• Connectors cheat sheet — connecting the Testing & evals area and toggling it per agent
• MCP & connectors — using ?domain=sales_evals from Claude Code / Cursor
• Sales tools reference → test_chat_send — the synchronous single-message test chat the eval runner is built on
• Working with the Sales Platform — the operator/runtime split this workflow lives in