Testing & validating search

How to prove your catalog is actually searchable: lock down known-good results with regression cases, inspect the live index to see exactly which attributes and values are filterable, and confirm filters fire after a re-index.

You do this from two places:

• In the app — Knowledge Base → Products shows each catalog's indexing status and lets you connect AI tools to it.
• Over the Product MCP — connect the catalog to Claude Code, Cursor, or any MCP client and call the same search tools your agent uses, so you can script exact checks.

Cheat sheet

One catalog per search

Every product source you connect (a feed URL, a spreadsheet, KeyCRM, Sitniks, …) becomes one catalog, and every search runs against exactly one catalog:

• In the app you're always looking at a single catalog's card under Knowledge Base → Products.
• Over MCP, the Server address you copy from Connect AI tools is already scoped to your account and that one catalog — so the search tools never ask you which catalog to use; the address decides.

In the app

Knowledge Base → Products → open a catalog's ⋯ menu → Connect AI tools. Copy the Server address (and the ready-made Claude Code / Cursor setup) for the catalog you want to test. https://app.mychatbot.app/knowledge-base

The setup you copy looks like this — one server address, plus a one-line install for the tool you use:

# Claude Code — run in your terminal
claude mcp add --transport http product-search "https://product.mychatbot.app/mcp/<account>/<catalog>/stream"

// Cursor — add to ~/.cursor/mcp.json (or .cursor/mcp.json in your project)
{ "mcpServers": { "product-search": { "url": "https://product.mychatbot.app/mcp/<account>/<catalog>/stream" } } }

Inspect what the index knows (over MCP)

Once a catalog is connected, these read-only tools let you inspect what the index actually holds before and after an import. They're the same tools your agent uses, so running them yourself is the closest thing to what a shopper's message triggers.

Tool (over MCP)	What it does	Use it to
get_all_categories	Lists every category in the catalog	Confirm the catalog indexed at all; grab a category to drill into
get_category_attributes	For one category, lists the attributes you can filter on	Check the attributes you plan to filter on are genuinely filterable
get_category_attribute_values	For one attribute, shows the real values present	Discover exact values to filter on (brand names, colours, sizes…)
get_available_filters	For a category, returns each attribute's values with counts	Prove a filter is live — apply one and watch the counts narrow
semantic_product_search	Runs a real search (text and/or image), optionally with filters	Run the exact query a regression case checks

Check indexing status (in the app)

There's no status tool — indexing progress lives on the catalog's card.

In the app

Knowledge Base → Products → your catalog shows an Indexing status panel with a progress bar and a Refresh button. (KeyCRM / Sitniks catalogs show the same panel on the Integrations page.) https://app.mychatbot.app/knowledge-base

Status you see	Meaning	What to do
Preparing / preprocessing feed…, Parsing products…, Generating embeddings…, Indexing products…	Still working	Keep waiting — the panel auto-refreshes
Completed — N products / M categories	Done	Safe to verify searches and filters
No active indexing / an error	Stuck or failed	Re-run it (see Confirming filters work after a re-index)

Wait for Completed before you trust any search or filter result — a half-built index looks exactly like a ranking or filter bug.

Validation workflow

Step	Do this	Pass signal
1. Catalog present	get_all_categories	Expected categories appear with non-empty products
2. Attributes filterable	get_category_attributes on a category	The attributes you plan to filter on are listed
3. Real values exist	get_category_attribute_values for each	The value list is non-empty and matches your vendor data
4. Facets non-empty	get_available_filters with your attributes	Each requested attribute returns values and counts
5. Filter fires	semantic_product_search with a filter	Results come back and applying the filter narrows the facet counts
6. Known hits hold	Run your regression cases	Every expected product id lands in the top-N window

Known-hits regression pattern

The core idea: pin a query to the product ids that must come back for it, then check that every expected id shows up within the first N results. This catches ranking, category-selection, and merge regressions before they reach shoppers.

A case is four things:

• Which catalog it runs against (record it, so the case always hits the right catalog).
• The query (exactly as a shopper — or the bug report — phrased it).
• The product ids that must appear in the results.
• The window — how many top results to check.

Pick the window by query type:

Query type	Window to check	Why
Exact code / SKU lookup (MKRM4177-1)	Just the top few	The exact variants must be right at the top
Natural-language query ("engine for Honda Accord")	A wider top 10–20	Ordering is allowed to shift, but the target products must still surface

To run a case, call semantic_product_search over that catalog's MCP address with the query, read back the product ids, and confirm your expected ids are within the window. Keep the cases in a living list you re-run after any catalog change:

Catalog	Query	Must return	Window
Auto-parts	MKRM4177-1	the 3 exact SKU variants	top few
Auto-parts	Двигатель HONDA ACCORD	3 specific engine listings	top 10

Build cases from real bugs

Every regression case should capture a real production query or a bug you fixed — a code lookup that returned the wrong variant, or a kitchen query that surfaced wardrobes. That way any future ranking change re-triggers the exact failure it must not reintroduce.

Only meaningful against a fully-indexed catalog

Known-hits checks only mean something when the catalog is fully built. Before you treat a miss as a ranking regression, confirm the catalog shows Completed in its Indexing status panel — a partial index looks exactly like a ranking bug.

Inspecting the index

Before you trust a filter, ask the index what it actually knows. The three inspection tools compose top-down: category → its attributes → the values of one attribute → faceted counts.

// 1. What attributes can I filter on in this category?
get_category_attributes({ "category_id": "<a category from get_all_categories>" })

// 2. What real values does one attribute have?
get_category_attribute_values({ "category_id": "…", "attribute_name": "brand" })

// 3. Faceted value + count breakdown
get_available_filters({ "category_id": "…", "facet_attributes": ["brand", "price"] })

Use facets to prove a filter is live

get_available_filters is the fastest confidence check: apply a filter (say a price range) and the brand counts should drop. If applying a filter changes the totals, that attribute is genuinely indexed and filterable — not just listed somewhere in the raw product data.

Confirming filters work after a re-index

Some catalogs that were first indexed a long time ago can have filters that silently return nothing even though the products are there — the searchable attribute data behind those filters was never built the modern way. A fresh, from-scratch re-index fixes it, because it rebuilds those searchable attributes.

Re-index, then verify:

1. Trigger a from-scratch re-index. There's no one-click "rebuild" toggle. Either:

   • Remove the catalog and re-connect it — a freshly connected catalog re-indexes from scratch. Its catalog id changes, so re-point any agent tools and re-copy the MCP Server address from Connect AI tools.
   • Or contact MyChatBot support with your account and catalog, and support runs a full rebuild service-side.

   (Toggling the Products switch off/on or clicking Refresh re-validates the feed, but may not force a full from-scratch rebuild.)

2. Wait for Completed in the Indexing status panel, then re-run the inspection tools — get_category_attributes → get_category_attribute_values → get_available_filters. Facets that were empty before should now return values and counts.

3. Run a filtered search and confirm results come back:

   // A filtered search over the Product MCP
   semantic_product_search({
     "query": "wireless drill",
     "filters": [
       { "attribute": "brand", "operator": "eq",  "value": "Bosch" },
       { "attribute": "price", "operator": "lte", "value": 200 }
     ]
   })

If indexing gets stuck

If a rebuild stalls, the Indexing status panel eventually shows failed / No active indexing rather than spinning forever. Re-run it (remove & re-connect, or ask support), then resume the verification steps.

Exact-match vs range filters

Equals / one-of filters (is, is one of) work broadly. Range and comparison filters (greater than, less than, starts with) resolve reliably only inside a specific leaf category. If a range filter returns nothing on a broad or top-level category, that's expected — scope it to a specific leaf category instead. See Filters, attributes & params for the full operator list.

Very old catalogs

The oldest catalogs handle filters differently under the hood. A from-scratch re-index still helps, but confirm with get_category_attributes on a leaf category. If filters still misbehave, contact MyChatBot support — MyChatBot tunes search relevance per catalog and can adjust it for yours.

Best practices

Do

• Pin a regression case the moment you fix a search bug, using the exact failing query and the ids it should return — and record which catalog it belongs to so it runs against the right one.
• Keep code/SKU cases on the tight top-few window and natural-language cases on a wider top 10–20.
• Inspect with the live tools (get_category_attributes, get_available_filters) rather than trusting the raw feed — an attribute can be visible in the product data yet not actually filterable.
• Wait for Completed in the Indexing status panel before verifying, and re-run the inspection tools after every from-scratch re-index to confirm facets came back non-empty.

Don't

• Don't assert exact result ordering for natural-language queries — check set membership within a window; ranking is allowed to shift.
• Don't verify filters while the catalog is still Preparing / Parsing / Generating embeddings / Indexing — a half-built index reads like a filter or ranking bug.
• Don't debug a "filter returns nothing" as a query bug before ruling out a stale or partial index — the fix is usually a from-scratch re-index (remove & re-connect, or ask support).
• Don't filter with range operators on broad categories — scope to a specific leaf category, where range filters resolve.

Test it

1. Connect a catalog to your MCP client (Knowledge Base → Products → ⋯ → Connect AI tools), then call get_all_categories and note a leaf category.
2. Call get_category_attributes on it, then get_category_attribute_values for one attribute to grab a real value.
3. Call get_available_filters for that attribute and record the counts.
4. Re-call get_available_filters with a filter applied — the counts should narrow. If they don't change (or the facet was empty), trigger a from-scratch re-index (remove & re-connect the catalog, or ask MyChatBot support), wait for Completed in the Indexing status panel, and repeat.
5. Add the query + expected ids (with the catalog) to your regression list so the win is permanent.

See also

• Filters, attributes & params — full operator matrix and how filterable attributes are built.
• Search tools reference — every search tool over MCP and what it does.
• Getting inventory in — feeds, connecting a catalog, and re-indexing.
• Troubleshooting & best practices — stale index, name pollution, and other production failure modes.