All insights
XINDAR INSIGHT

Compare AI Search Platforms with Paired Question Records

Reconstruct paired mention records across AI search surfaces,compare two fictional tables with identical marginal rates,and preserve missingness and sampling assumptions.

Sources checked November 17, 2026

A monitoring report says a brand appeared in 55% of answers on one AI search platform and 45% on another. A buyer considering where to concentrate content work needs the underlying question records. The two percentages conceal whether the platforms disagreed on twenty questions or sixty.

When the same comparison units receive a binary assessment on both platforms, a paired analysis preserves that relationship. McNemar's test examines the balance between the two directions of disagreement. Applying it to AI visibility requires a defensible comparison unit, consistent entity coding and an appropriate sampling design. NIST describes the paired binary setting and its independence assumptions. NIST McNemar documentation

The examples below are fictional. Platforms A and B are teaching labels, and every count was constructed for the calculation. The examples assume independent pairs drawn under a common binary-outcome model. They provide a way to inspect a report's method before interpreting an observed platform difference.

Give each comparison a common key

Start with the decision the comparison should support. A team might want to know whether its fixed set of procurement questions produces different brand-mention rates across two specified consumer search surfaces during a defined collection round.

Create a comparison-unit key before collection. It joins two observations made under the stated pairing rule. Each actual answer still receives its own observation ID. The common key might identify the question version and collection round, while the separate observations retain their respective platform, session and timestamp.

The pairing rule needs enough detail to survive an audit. Record the exact question, language, account context, selected mode, visible model label, collection period and available location evidence. Set any permissible time separation in advance. A question asking about American suppliers supplies question context. The collection record supplies evidence about the actual environment.

Matching by row position is fragile. A timeout can remove an answer from one export and shift every subsequent row. Matching by question text alone also becomes ambiguous when the same question appears on several days. An explicit key makes the intended relationship recoverable.

Define the binary outcome before reading the totals

For this tutorial, a positive outcome means that the completed answer body mentions the verified target company. A negative outcome means that the completed body contains no qualifying mention under the declared entity rule. Retain the matched passage and the reason for each positive classification.

Citation URLs, recommendation language and factual accuracy require additional fields. A source URL containing a company name can support a citation record. Whether the answer itself names that company is a separate coding decision. Define the outcome narrowly enough that another reviewer can reproduce it from the retained answer.

The review must also handle aliases and ambiguous names. Freeze the entity rule for the comparison, record its version and inspect borderline matches. If the rule changes, preserve the original classifications and document which answers were recoded. Otherwise, a difference between exports may partly reflect a change in the classifier.

Completion status belongs alongside the label. Preserve login interruptions, unavailable answers and collection failures as their actual states. Eligibility for the paired calculation can then be determined from the records.

Reconstruct the four cells

Consider the first fictional record of 100 complete pairs. The row identifies Platform A's result and the column identifies Platform B's result.

Record 1B mentions targetB has no target mentionA total
A mentions target401555
A has no target mention54045
B total4555100

Both platforms mention the target in 40 pairs. Both omit it in another 40. The remaining 20 pairs are discordant, with 15 positive only on A and five positive only on B.

A's observed rate is 55/100. B's observed rate is 45/100. The paired difference is ten percentage points, calculated as (15 − 5) / 100. The concordant cells contribute to the two marginal rates and the total denominator. The directional difference comes from the discordant cells.

Now consider a second fictional record, again with 100 complete independent pairs.

Record 2B mentions targetB has no target mentionA total
A mentions target203555
A has no target mention252045
B total4555100

The observed marginal rates and their ten-point gap are identical. This time, 60 pairs disagree. A is positive alone in 35, and B is positive alone in 25. A report retaining only the marginal percentages loses the information needed to distinguish these records.

Calculate the exact paired comparison

Under the equal-marginal-probability null model for independent, identically distributed pairs, a discordant pair is equally likely to point in either direction. Conditional on the number of discordant pairs, the directional count follows a binomial distribution with probability 0.5.

Let b be the A-only count and c the B-only count. Set d = b + c and k = min(b, c). The exact two-sided calculation used here is min(1, 2 × P[X ≤ k]), with X binomial with d trials and probability 0.5. The published statsmodels implementation uses this doubled lower tail and caps the result at one. statsmodels implementation

Fictional recordA-onlyB-onlyDiscordant pairsExact two-sided p-value
Record 1155200.041389
Record 23525600.245061

These values were calculated with integer binomial coefficients and exact rational arithmetic, then rounded for display. The underlying values are approximately 0.0413894653 and 0.2450608330. The calculation demonstrates the method for these constructed records.

A p-value describes how extreme the directional imbalance is under the specified null model and assumptions. Its interpretation depends on the analysis design, including any predeclared decision threshold and the number of comparisons being examined. Report the observed gap alongside the result so readers can assess its business scale.

Record the chosen implementation explicitly. The statsmodels API distinguishes exact binomial calculation from a chi-square approximation and applies its continuity-correction option to the approximate calculation. SciPy also documents a binomial testing interface. The libraries were consulted as documentation for this article. The numerical examples were computed independently with Python's standard library. statsmodels API, SciPy binomial-test API

Keep incomplete pairs visible

Suppose a fictional collection plan schedules 110 pairs. Both answers are available for 100. Six additional units contain an A answer while B is unavailable, and four contain a B answer while A is unavailable.

The paired table contains the 100 complete pairs. Coverage of scheduled pairs is therefore 100/110, approximately 90.91%. The six and four incomplete units stay in the collection ledger with their observed content and failure states. The analysis identifies its complete-pair population explicitly.

This matters commercially when collection availability differs by surface or question type. The retained ledger lets a reviewer ask whether excluded units were disproportionately technical questions, long answers or a particular account context. The report should explain how that selection affects the population to which its result applies.

Keep a flow from scheduled unit to submitted observation, completion, eligibility and classification. Someone checking the table should be able to identify every included pair and every scheduled unit outside it.

Check whether the pairs support inference

NIST lists mutual independence of the pairs among the test's assumptions. Repeating a small set of questions can create dependence across observations, even when every row has a unique timestamp. A stable question-specific preference, shared session history or a common collection-day disturbance can influence several pairs together. NIST assumptions

A table remains a descriptive summary when those dependencies exist. For inference, specify the sampling units and obtain a method that represents the clustering or repeated-measures design. The operational question is which observations can reasonably contribute independent information under the proposed model.

Pairing also has a purpose. Comparing two platforms on different question sets introduces question composition into the observed gap. A common planned panel helps align that composition. It still leaves platform mode, account context, collection timing and other environmental differences to be documented and interpreted.

For a deliberately chosen buying-question panel, describe the conclusion as a result for that panel and collection design. Broader market coverage needs a sampling justification. A larger answer count alone supplies limited information about how well the panel represents buyers.

Test a monitoring export with a small replay

Ask a vendor to provide two complete answers for one declared comparison unit, their separate observation IDs and the common matching key. Locate the target passage on each screen, then find those same passages in the exported files. Confirm the coding rule, missing-state handling and collection context before scaling the review.

Semvi by Xindar's product team has confirmed consumer-web sampling for ChatGPT and Gemini and export of complete answers and citation URLs. A buyer can use the paired replay described here to examine the actual account's exported records. Ask the team to demonstrate the available identifiers and fields required for the proposed comparison. Semvi monitoring scope

A practical acceptance sequence can be written into the pilot.

  1. Register the question panel, comparison-unit definition, surfaces and outcome rule.
  2. Retain each submitted observation with its completion state and collection context.
  3. Reconstruct the four cells directly from matched, eligible records.
  4. Recalculate both marginal rates, the paired difference and the chosen test result.
  5. Review incomplete pairs, dependencies and the scope of the resulting conclusion.

Acceptance should produce an inspectable table and a record-level explanation for every count. The buyer can then decide whether the observed difference warrants an editorial investigation, a collection change or a larger, appropriately designed study.

Questions buyers ask

Can two reports show the same percentages and different paired results?

Yes. The two fictional tables both show 55% versus 45%, while their discordant counts differ. Retaining the four cells makes that difference visible.

What happens when the two directional counts are equal?

For the exact doubled-tail convention described here, the p-value is capped at one. The observed paired rate difference is zero. Keep the concordant counts and overall collection coverage in the report as well.

Can an unavailable answer receive the no-mention label?

Eligibility and completion rules must be declared first. In this article's design, a no-mention label requires a completed answer. An unavailable observation retains its missing or failure state and falls outside the complete-pair table.

Does a paired platform difference establish a content improvement?

A platform comparison describes outcomes under its collection design. Evaluating an intervention requires a design that addresses the intervention, timing, comparison conditions and other plausible causes of change.

Which export fields should the procurement team inspect?

Inspect the common comparison key, separate observation IDs, exact question version, complete answers, target passages, coding version, completion states and collection context. Reconstructing one pair from screen to export gives the buyer a concrete starting point for acceptance.

Back to insightsMarkdown version