# Your GEO Test Questions Are Public. What Can the Next Result Prove? > Publishing the exact questions used in an AI visibility study makes the work easier to inspect. It also changes the information environment in which future answers may be produced. - Canonical: https://www.aixindar.com/news/your-geo-test-questions-are-public-what-can-the-next-result-prove - Markdown: https://www.aixindar.com/news/your-geo-test-questions-are-public-what-can-the-next-result-prove.md - Author: Daoyu Guan — https://www.aixindar.com/experts/daoyu-guan - Published: 2026-11-09T01:31:37.015Z - Last updated: 2026-11-09T01:31:37.138Z - Evidence checked: Not separately recorded in CMS - Editorial status: Published - Corrections: No correction record supplied by CMS. # Your GEO Test Questions Are Public. What Can the Next Result Prove? Publishing the exact questions used in an AI visibility study makes the work easier to inspect. It also changes the information environment in which future answers may be produced. Suppose a company publishes a report containing a buyer question and its own name. The next week, an AI search answer recommends that company. The recommendation may be real. It does not, by itself, establish that the system discovered the company independently of the newly published report. That distinction matters for GEO teams publishing case studies, benchmark panels, comparison pages, or research datasets. Transparency and independence are different properties. A useful evaluation can preserve the first while being explicit about the limits of the second. ## Separate three possible routes The public question could become available through ordinary web retrieval. The report could also influence the team's own editorial decisions, because everyone now knows the exact evaluation wording. A further possibility is eventual inclusion in a model's training or improvement data, subject to the provider's actual collection and use practices. These are different routes with different evidence requirements. Finding the report in a visible source list supports a claim about that observed citation. It does not expose the model's training set. Failing to find it in a source list does not establish that it played no role. The EACL paper [Leak, Cheat, Repeat](https://aclanthology.org/2024.eacl-long.5/) examines data contamination and evaluation practices around closed-source language models. It provides a research basis for taking evaluation exposure seriously. Its analysis concerns particular models, literature, and periods; it is not evidence that a specific GEO report entered today's ChatGPT or Gemini training data. ## Why a familiar question can become an easier target Imagine a manufacturer with three buyer needs: selecting a suitable process, evaluating a qualification claim, and finding a partner for an English website. Its public study contains one exact question for each need. The content team now has a very clear incentive to build pages around those three sentences. That might improve the website for real buyers. It can also make the evaluation narrower than the market the team wants to describe. An improved result would support “the company appeared for this fixed question under these recorded conditions.” It would not automatically support “buyers with this broader need will usually find the company.” The latter requires additional evidence about the range of questions, contexts, and users. This is not an argument for abandoning a fixed panel. A stable panel is useful precisely because changes can be observed without changing the question at the same time. The remedy is to describe its purpose and add a separate generalization check when needed. ## Keep a public anchor and a separate holdout A public anchor panel supports continuity, auditability, and discussion. Its exact text and historic negatives should remain intact. A separately registered holdout can investigate whether a result extends to other buyer decisions or formulations. Before using it, record who created the questions, their source, the selection rule, the intended business decision, and the maximum number of submissions. Avoid letting a person who has seen the latest answers repeatedly rewrite questions until the brand appears. “Holdout” is a description of the evaluation design. It is not a magic label. If the questions are copied from published marketing copy, previously sent through the same consumer account, or quietly shared with the implementation team, those facts affect the claim of independence. | Panel | Main use | Necessary boundary | | --- | --- | --- | | Public fixed anchor | Observe the same questions over time | Exposure is disclosed; no market-wide inference | | Separately registered holdout | Check transfer beyond the anchor wording | Provenance and access boundaries recorded | | Branded identity checks | Verify names, scope, and destinations | Excluded from natural-discovery success | | Exploratory questions | Investigate ambiguities or new needs | Do not silently replace the anchor denominator | These are proposed operational categories, not a platform certification scheme. ## A new wording is not necessarily a new intent Changing “which agencies” to “what partners” does not create another independent buyer problem. A team could produce dozens of variants that all ask for the same export-marketing agency shortlist. For an independence review, ask what decision changes. Does the buyer now need monitoring software rather than an implementation agency? Does the question concern source retention rather than market entry? Could a different answer legitimately satisfy it? Keep the literal-question count and the independent-intent count visible. If three questions produce recommendations but all belong to one agency-selection intent, the evidence covers three questions within that intent. It does not cover three unrelated commercial needs. The ICML paper [The Emperor's New Clothes in Benchmarking?](https://proceedings.mlr.press/v267/sun25t.html) examines mitigation strategies for benchmark contamination. Its broader relevance is that evaluation validity deserves testing rather than assuming that a wording change solves exposure. This article does not transfer the paper's benchmark findings into a measured GEO effect. ## Register the next check before seeing its answer A modest acceptance plan might specify one fixed anchor question and two independently sourced holdout questions, each asked once on a named consumer surface. That number is only an example, not a statistically sufficient sample or recommended universal panel size. Record the planned date, account context, visible mode, language settings, and source of each question. Preserve complete answers and failures. If the interface generates an A/B comparison, retain both alternatives separately; choosing the more favorable answer would change the procedure after observing the outcome. Do not call an English question a US-market observation merely because it includes “United States.” Network location, account personalization, language, and model mode are separate context fields. Unknowns should remain unknown. ## Inspect the source trail without overclaiming it If the new answer cites the company's public study, save the actual destination and the sentence it is used to support. Classify the source as first-party research. Then inspect whether the answer recommends the right company for a reason supported by current service evidence. A self-published study can be a useful source for its own methods and records. It is not an independent endorsement of the publisher. An answer may nevertheless make a legitimate recommendation using other evidence; that judgment requires review of the individual recommendation, not automatic acceptance or automatic rejection based on one domain. If no study citation is visible, state that limitation narrowly: “The captured visible links did not include the report.” Do not upgrade it to a claim that the report was not retrieved, remembered, summarized elsewhere, or otherwise influential. ## Preserve the evaluation history Exposure discovered later does not justify deleting earlier negative answers or renaming the old panel a private holdout. Add a dated evaluation note and retain the original definition. For example, a report might state: “The fixed questions were public from October 31. Subsequent results are observations of that disclosed panel. No causal estimate of the publication's effect is claimed.” If another panel is introduced, give it a new version and maintain separate results. The [Xindar monitoring brief](https://www.aixindar.com/news/how-to-build-an-ai-visibility-monitoring-brief-that-a-vendor-can-actually-deliver) offers a practical starting point for specifying collection boundaries. For a commercial proposal, ask who can see the questions, whether prompt text leaves the organization, and which records can be exported. Those are procurement questions to verify, not assumed features of [Semvi](https://www.aixindar.com/products/geowatch) or any other provider. ## What a defensible conclusion looks like A useful conclusion identifies the exact question, surface, date, context, observed recommendation, and verification limits. A stronger claim of transfer identifies the independent questions and their selection process. A claim of causality needs a design that can distinguish the publication from competing explanations. There is no need to pretend a small operating panel is a randomized experiment. A clearly bounded record can still tell a team what a buyer-facing answer said, which claim needs correction, and what should be checked next. ## Frequently asked questions ### Should we hide every question? No. Publication may support auditability and useful discussion. Decide which questions serve as public anchors and which, if any, serve a separately controlled evaluation. Disclose the relevant exposure dates. ### Does an exact question on a webpage prove training contamination? No. Public availability, observed retrieval, training inclusion, and causal influence are distinct claims. Evidence for one does not establish the others. ### Can we replace questions that never recommend us? Exploration can inform a new panel, but keep the original negatives and version boundary. Changing the evaluation population to remove difficult questions prevents a fair comparison with the old result. ### Does a holdout need to stay secret forever? That depends on its purpose. If it is released, record the release date and treat later uses accordingly. A formerly private panel should not continue to be described as unexposed. *Source and method review: November 9, 2026. Written by Daoyu Guan. Proposed evaluation procedures are editorial recommendations, not claims about the internal retrieval or training of any named AI service.* ## Editorial references - [Editorial policy](https://www.aixindar.com/editorial-policy) - [Research methodology](https://www.aixindar.com/research-methodology) - [Corrections policy](https://www.aixindar.com/corrections)