The product-discovery test protocol
How Tenuvio tests what AI assistants say about a store's products, from catalog sample and prompts to controls, run records and scoring.
Published
Version 0.1Status: draftResults that use it: none yet
A test asks AI assistants the same prompts about a fixed set of products. It compares each answer with the shop’s own page at that moment, and scores it on the five checks below. Every published result links to its run records.
Two kinds of test
| Kind | Uses | Store named? |
|---|---|---|
| Observational | Public product pages and answers only. | Only if the store agrees. |
| Authorized | Also the merchant's feed and analytics, under a written agreement. | Only if the merchant agrees. |
Only an authorized test can tie a change in answers to a fix, because only then is the catalog’s before-and-after state known.
Where the answers come from
Each run is labeled with how it was collected, and the two kinds are never added together.
| Route | How | Caveat |
|---|---|---|
| Manual | A person asks in the assistant's own app or site, signed out where possible, and records the answer. | Only where the assistant's terms allow a person to collect answers. |
| API | The same question through the vendor's developer API, via a measurement panel shared with Tenuvio's sister publications. | API answers can differ from what shoppers see in the app. |
Perplexity’s terms (updated January 23, 2026) bar any automated or manual process that collects information from its service, so Tenuvio doesn’t test Perplexity’s app. No assistant is tested with bots or scrapers, which Microsoft’s Copilot terms, for one, forbid.
The catalog sample
10 to 20 products from one store, fixed before the first run. The sample includes different price points, at least one product with variants, and at least one also sold elsewhere.
For each product, a capture records what the shop’s page says and where each value was read. It covers the name, brand, GTIN or MPN, variant, price, availability and URL. A run is compared with the nearest capture taken within 2 hours of it.
Prompts
| Family | Asks | Example form |
|---|---|---|
| Named shop | Does the assistant know what this shop sells? | "Does [shop] sell [product type]? What does it cost?" |
| Named product | Does it identify the product and say where to buy it? | "Where can I buy [brand] [product]?" |
| Open need | Is the product recommended when no name is given? | "Recommend [product type] for [need] under [price]." |
Prompts are saved verbatim before the first run and never edited during a test.
Conditions
Every run records the assistant, its mode and any model name shown. It also records the time in UTC, the country, the language and the route, manual or API. Each run starts a new chat. Each prompt runs 3 times per assistant per run day, on at least 2 run days at least 7 days apart.
A test runs from one country, because shopping features differ by market: several, such as Copilot Checkout, are offered only in the US. Manual runs are signed out where the assistant allows it. Where it needs an account, a fresh one with memory and personalization turned off is used, and those runs are reported apart.
Scoring
| Check | Possible scores |
|---|---|
| Identity | Correct · wrong variant · different product · not mentioned |
| Price | Matches the page · matches an earlier price · different · not stated |
| Availability | Correct · wrong · not stated |
| Link | Shop's product page · checkout in the assistant · shop's other page · marketplace · other site · no link |
| Recommended | Product recommended · shop named without the product · neither |
The close calls:
- Price and availability are scored only for the right product. A wrong variant still gets a link score.
- A price must match in currency and amount. A range never counts as a match.
- With several links, the best one for the shop counts, judged by where it lands after redirects.
- Results the interface labels as sponsored are scored apart from the rest.
A person codes which catalog product each mention is; a script does the rest, so the same records always give the same scores. Results are counts, such as “price matched in 7 of 18 runs”, never one score per store.
Evidence labels
| Label | Meaning |
|---|---|
| Observed | Seen in at least one run, with its record. |
| Reproduced | Seen in at least two thirds of runs on each of 2 run days. |
| Unverified | Reported elsewhere, not seen in a Tenuvio run. |
| Unknown | Not tested, or the records don't settle it. |
A likely cause, such as a stale feed price, stays labeled as an inference. It becomes a finding only when an authorized test changes that one thing and the answers change with it.
Limits
Answers change as assistants are updated, so a result holds for its run dates. One store’s sample says little about other stores. Test prompts aren’t a sample of what shoppers ask.
Changes to this method
- : First draft (v0.1). Not yet used for a published result.
Sources
- Perplexity Terms of Service, section on prohibited uses, clause (i), Perplexity AI. Accessed .
- Microsoft Copilot terms of use, Microsoft. Accessed .