Testing & evals

Save a real conversation as a scenario, replay it against an assistant — with its current instructions or a candidate rewrite — and score how stable the answers are across repeated passes. That's the Testing & evals connector (?domain=sales_evals): seven tools for regression-testing a sales assistant the way you'd regression-test code.

You want to…	Call
Save a conversation as a replayable test	eval_scenario_save — from a real chat, the test chat, or inline messages
See / inspect / remove saved tests	eval_scenario_list / eval_scenario_get / eval_scenario_delete
Replay once against the live assistant	eval_run_start with the defaults (runs: 1, judge: "none")
Try a candidate prompt without touching the live assistant	eval_run_start with instruction_override
Measure answer stability	eval_run_start with runs: 2–5 and judge: "similarity"
Watch progress & read results	eval_run_get — poll it; partial results while running
Stop a run	eval_run_cancel — best effort, between turns
Test a client who sends a photo	test_chat_send with images, or a scenario turn {content, images} — see Rehearse a client photo

Every parameter of every tool is in the Sales tools reference → Testing & evals.

Three facts to keep in mind:

• Eval traffic is free. Every turn runs as test-chat traffic — real tokens are recorded as test-chat usage, never billed as customer messages. Replay a 30-turn conversation five times and your message balance doesn't move.
• Runs are async. eval_run_start returns a run_id immediately and the run executes in the background — poll eval_run_get for progress and results.
• One run at a time per account. Starting a second run while one is queued or running returns an error naming the active run — poll it or cancel it first.

Where it runs

Like every Sales Platform area, Testing & evals is available two ways:

• To an agent — connect the Testing & evals card in the From your sales platform section of Agents → Connectors, then toggle it on for the agent. Now you can just ask: "replay chat 8412 against this new prompt and tell me if the answers hold up."
• From Claude Code / Cursor — the same tools as an MCP server (click Connect AI tools on the card for your copyable config):

https://api.mychatbot.app/api/mcp/sales-management?account_id=<account_id>&domain=sales_evals

See Sales Platform MCP for the access key and client setup.

The workflow

The loop is always the same four steps:

1. Capture. Take a conversation worth protecting — a real chat where the assistant did well (or fumbled), or a test-chat session you just ran — and save it as a scenario with eval_scenario_save. Only the user turns are kept; the assistant's side is what a run produces. Optionally add reference answers — what the assistant should say per turn — to unlock scoring against them.
2. Replay. eval_run_start with the defaults is a plain replay: one pass over the turns against the assistant as currently configured, recording every response and every tool call.
3. Compare a candidate. Pass instruction_override with a rewritten system prompt to run the same turns against the candidate text. The live assistant is never modified — customers keep talking to the current version while you test. (skills_override does the same for skills: pass a slug list to pin the selection, [] to run with no skills at all.)
4. Check stability, then iterate. Answers you liked once can still be flaky. Set runs: 3 (up to 5) plus judge: "similarity" and every iteration replays the turns as an independent conversation; the judge scores how consistently the assistant answered each turn across iterations — and, when the scenario has reference answers, how close each answer came to them. Edit the prompt, re-run, repeat. When the candidate wins, ship it with assistant_update_instructions.

Scenarios are the regression suite

A scenario is saved per account and replayable forever — eval_scenario_save upserts by name, so "delivery-pricing" always means the latest version of that test. Build a handful of scenarios for the conversations that matter (pricing objections, delivery questions, refund policy) and re-run them after every instruction change.

Worked example: fix a fumbled delivery answer

The assistant fumbled a delivery-cost conversation. Capture that chat, test a rewritten prompt against it, and check the fix is stable — without customers ever seeing the draft.

1. Save the chat as a scenario. Take the chat id from list_chats (Conversations connector) and add reference answers for the turns you care about:

// eval_scenario_save
{
  "name": "delivery-pricing",
  "description": "Customer asks about delivery cost and timing; bot must quote 80 UAH and 1-2 days",
  "assistant_id": "42",
  "from_chat_id": "8412",
  "reference": [
    "",
    "Delivery is 80 UAH by Nova Poshta, 1-2 business days.",
    ""
  ]
}

{
  "scenario_id": 17,
  "name": "delivery-pricing",
  "turns": 3,
  "updated": false,
  "message": "Scenario saved. Run it with eval_run_start{scenario_id}."
}

2. Start a run with the candidate prompt. Three iterations, similarity judge, and the rewritten instructions in instruction_override:

// eval_run_start
{
  "assistant_id": "42",
  "scenario_id": 17,
  "runs": 3,
  "judge": "similarity",
  "instruction_override": "You are the sales assistant for Kvitka Flowers…\n\nDelivery: always quote 80 UAH via Nova Poshta, 1-2 business days. Never say 'it depends' — if the customer's city is unusual, quote the standard rate and offer to confirm."
}

{
  "run_id": 41,
  "status": "queued",
  "turns": 3,
  "runs": 3,
  "judge": "similarity",
  "message": "Eval run started in the background. Poll eval_run_get{run_id} for progress and results."
}

3. Poll. While the run executes, eval_run_get {"run_id": 41} returns status: "running" plus the turns completed so far (with a note that results are partial). When it finishes:

{
  "run_id": 41,
  "assistant_id": 42,
  "status": "done",
  "runs": 3,
  "judge": "similarity",
  "instruction_override_used": true,
  "source": { "type": "scenario", "scenario_id": 17, "turns": 3 },
  "iterations": [
    {
      "iteration": 1,
      "turns": [
        {
          "turn": 2,
          "user_message": "how much is delivery to Lviv?",
          "response": "Delivery to Lviv is 80 UAH via Nova Poshta and takes 1-2 business days.",
          "tool_calls": [
            { "name": "get_faq", "result_preview": "Delivery: Nova Poshta, 80 UAH…", "result_bytes": 412 }
          ]
        }
      ]
    }
  ],
  "metrics": {
    "judge": "similarity",
    "per_turn": [
      { "turn": 1, "consistency": 0.91 },
      { "turn": 2, "consistency": 0.88, "vs_reference": 0.83 },
      { "turn": 3, "consistency": 0.79 }
    ],
    "overall_consistency": 0.86,
    "overall_vs_reference": 0.83
  },
  "created_at": "2026-08-07T09:14:02Z",
  "finished_at": "2026-08-07T09:16:47Z"
}

(Transcripts abbreviated — you get every turn of every iteration, and include_tool_details: true adds each tool call's arguments.)

4. Read the metrics. Scores are 0–1, higher = more similar:

• consistency — how alike the three iterations' answers to that turn were (mean pairwise token overlap). A low turn is where the assistant improvises — tighten the instructions there.
• vs_reference — how close the answers came to your reference for that turn. Only turns with a non-empty reference get one.
• The overall_* values average the per-turn scores; consistency needs runs ≥ 2 to exist at all.

Turn 2 answers 0.83 against the reference and holds 0.88 consistency — the fix works and holds up. If it looks good, apply the candidate for real with assistant_update_instructions, and keep delivery-pricing in the suite to re-run after the next prompt change.

The judge is deterministic

similarity is token-overlap math (language-agnostic — Ukrainian works as well as English), not an LLM grader. It costs nothing extra and the same transcripts always score the same. Treat it as a stability and drift signal, and read the transcripts for quality judgments.

Rehearse a client photo

Clients often send only a photo — a screenshot of a product, an Instagram post, a picture of what they want. A test turn can carry that photo, and the assistant sees it as a picture, exactly as it does when the photo arrives from Telegram or Instagram. It also gets the photo's URL, so it can pass the photo to an image search tool the way the live assistant does.

• Find the photo. get_chat_messages (Conversations connector) returns a photo message with images — the photo URLs — next to its text. A photo-only message has images and no content.
• Try it once. Pass those URLs to test_chat_send (Assistants connector) in images. message is optional when images is given. Up to 4 photos per message.
• Keep it as a regression test. Save the whole chat with eval_scenario_savefrom_chat_id — photos stay with their turns — or write the turn inline as {content, images}:

// eval_scenario_save
{
  "name": "photo-lookup",
  "assistant_id": "42",
  "messages": [
    { "images": ["https://static.mychatbot.app/chat-images/….jpg"] },
    "Is it available in size M?"
  ]
}

In eval_run_get, a replayed photo turn shows its URLs under user_images.

Put the URL in images, not in the text

A URL pasted into message (or into a plain-text scenario turn) reaches the assistant as text — it cannot see the picture. Photo URLs from get_chat_messages can be passed as they are, no re-upload needed. Other links must open without a login: if a photo cannot be fetched, the platform usually removes it and the assistant answers from the text alone, and with some models the turn fails instead. A failed test chat turn stays in the history — run test_chat_start before you retry.

What runs for real during a run

Every turn of a run is a test conversation, so it follows the test-chat rule: reads run live, writes are simulated.

• Knowledge, product and web searches, CRM and calendar lookups run against live data, so a scenario can check that the assistant grounds its answer in what the business actually has.
• Tools that write — bookings, CRM deals and leads, orders, payment links, pipeline status and labels, operator handoff — are simulated. The transcript shows the tool answering "Simulated in test mode: … nothing was created or changed", and the assistant continues as if the write had worked. Nothing reaches the connected system.
• Tools from a connected MCP server run only if the server marks them read-only (the standard readOnlyHint). Any other MCP tool is not executed: the assistant receives an error saying the source is unavailable in test mode and is expected to tell the user so. A scenario that needs such a tool will show that error in tool_calls, not fabricated data.

So a scenario that validates a docs-grounded or catalog-grounded answer works as long as the knowledge source is one of the platform's own or an MCP server that declares its reads. If a run shows "not available in test mode" for a source you expected to be read, the server behind it does not declare the hint; ask its maintainer to add it and reopen the server's tool list under Integrations so the platform records it.

Limits & mechanics

Turns per scenario / run	max 30 user turns — longer chat sources are cut to the first 30 (flagged truncated); longer inline lists are rejected
Photos per turn	max 4 public http(s) image URLs per user turn (test_chat_send images, inline {content, images})
Iterations per run	runs 1–5, default 1; each iteration is an independent conversation from a clean history
Concurrency	one queued-or-running run per account
Run deadline	15 minutes wall clock — a run that exceeds it flips to error, completed turns retained
Statuses	queued → running → done / error / cancelled
Cancel	best effort between turns — the current turn finishes first; completed turns stay readable
Cost	free — recorded as test-chat usage, never billed as customer messages
Results size	transcripts are capped (~2 MB per run); oversize results are trimmed with an explicit results_truncated marker, never silently
Deleting things	deleting a scenario keeps past runs' transcripts readable

See also

• Sales tools reference → Testing & evals — all seven tools with full parameters
• Connectors cheat sheet — connecting the Testing & evals area and toggling it per agent
• Sales Platform MCP — using ?domain=sales_evals from Claude Code, Codex, or Cursor
• Sales tools reference → test_chat_send — the synchronous single-message test chat the eval runner is built on
• Working with the Sales Platform — the operator/runtime split this workflow lives in