ApexaRank.ai
PROMPT INTELLIGENCE

How to Choose the Right Prompts for LLM Tracking, and Measure Them Properly

A hundred prompts run once each makes an impressive chart and a weak decision.

Good tracking needs the right questions and a method that separates a pattern from noise.

Choose fewer prompts, and measure them properly.

How to build a prompt list from real buyer questions, what answer variation does to your numbers, and how many runs a claim actually needs.

By ApexaRank.aiOctober 20268 min read
Isometric illustration of a phone showing a chat conversation, representing the questions people ask AI assistants

It’s easy to build an AI visibility dashboard that looks impressive and says very little.

Load a hundred prompts. Run each one once on a few AI platforms. Count the brand mentions, turn them into a percentage and call it “AI share of voice”.

The hard question comes later, when a marketing manager asks: “What should we do differently because of this number?”

Usually the dashboard isn’t the problem. The prompt list doesn’t represent a clear business question, and the measurement doesn’t say what it can and can’t support. Good tracking needs both the right questions and a method that separates a real pattern from a noisy answer.

Start With the Buyer’s Decision, Not Your Homepage Keywords

Take a company selling appointment software to clinics. The obvious thing to track is “best appointment software”. But a clinic manager comparing options is more likely to ask:

“How can we reduce missed appointments without adding a receptionist?”
“What should a three-location clinic look for in scheduling software?”
“Can appointment systems send WhatsApp reminders?”
“What alternatives to [competitor] support several doctors?”
“Can we migrate our patient appointment history without losing records?”

These sit at different stages of the same purchase. One names the problem, one tests requirements, one compares vendors. Track only variations of “best appointment software” and you’ll miss the questions where your product is most relevant.

Divide Your Prompts Into Four Intent Families

Intent familyExampleWhat you’re trying to learn
Problem“How do clinics reduce no-shows?”Is the brand present in category education?
Requirements“Scheduling software with multiple doctors and WhatsApp reminders”Do answers understand the relevant capabilities?
Comparison“Alternatives to [competitor] for small clinics”Is the brand in a meaningful shortlist?
Branded“Does [your product] integrate with our clinic system?”Are brand-specific descriptions accurate?

You don’t need equal numbers in each group. A young brand may need more discovery questions; an established one may watch comparisons and inaccurate descriptions more closely. What matters is that every prompt has a reason to be there.

Collect Questions From Customers Before Expanding Them With AI

Start with what’s already inside the business: sales and demo notes, onboarding objections, support tickets, site search, Search Console, reviews, and the criteria buyers use in RFPs.

Then use prompt discovery tools to find related phrasings and missing questions. Treat their output as candidates to review. Third-party estimates of “AI prompt volume” aren’t verified query logs from the platforms, so if one drives a priority, label it as an estimate.

Score Each Candidate Before Tracking It

Give each candidate a score from 1 to 3 in four categories:

CriterionScore 1Score 3
Buyer relevanceLoosely related to the categoryDirectly tied to a customer problem or decision
Business valueGeneral awareness, little clear valueTied to evaluation, trust or a commercial goal
Visibility gapNo meaningful problem foundCompetitors appear while you’re absent or misdescribed
FixabilityNo practical next stepA page, technical issue, fact or legitimate source you can improve

It’s a planning tool, not a prediction of what an AI will cite. Then ask one more question: if this prompt changed next month, who on the team would act on it? If nobody, it can wait.

The Problem Most Dashboards Underestimate: Answer Variation

AI assistants don’t return the same recommendations every time. The same wording can give different lists, and a reworded question can change them even more.

A May 2026 preprint by Jack and colleagues tested commercial recommendations with thousands of same-prompt and paraphrase runs on selected OpenAI and Anthropic models (arXiv). It measured Jaccard similarity, the overlap between sets of recommended brands, where 1.0 means identical.

Comparison in the studyRecommendation-set similarity
Same prompt, rerun0.50–0.61
Cosmetic rewording, similar intent0.288
Rewording that adds constraints0.135
Source: Jack et al. (2026), arXiv preprint. Figures describe the study’s own prompts, models and settings.

These figures describe that study, not every assistant. The lesson still carries: exact phrasing matters, and one answer is a weak base for a trend claim.

What Does “12 Mentions Out of 20” Really Mean?

Say your brand appears in 12 of 20 monitored responses. That’s 60% of your sample. It isn’t 60% of AI search in your category, because your prompts are a hand-picked list, not a random sample of everything buyers ask.

What repeated runs can tell you is how likely one prompt, under fixed conditions, is to name you. Rerun the same prompt with the same settings and each run is close to an independent draw, so a standard margin of error applies. Worst case, at 95% confidence:

Runs of one promptMargin of error
10±31 points
20±22 points
50±14 points
100±10 points

So 12 of 20 means somewhere around 38% to 82% for that prompt. That’s enough to spot a brand that’s missing everywhere. It isn’t enough to claim a 5-point gain since last month. Keep statements like this at the prompt or intent-family level, and say how you sampled.

Make the Tracking List Repeatable

Three rules that make trends defensible
Track intent families, not one magic phrase. Use a few natural phrasings for each important decision, and report the individual prompts and the family.

Keep a stable comparison set. If every prompt changes between periods, the trend means little. Review coverage periodically and log each change.

Record the conditions. Platform or surface, date, country, language, account or personalisation settings, and whether web search was on. Don’t merge unlike observations.

More runs won’t fix a biased prompt list. Better coverage comes first.

Look Beyond One Mention Percentage

For each important intent family, check:

Mention: did the answer name your brand?
Citation: was your domain linked, or did a third-party source carry the answer?
Competitive context: which other brands were named, and how?
Accuracy: were your capabilities, limits and positioning described correctly?
Consistency: does the pattern hold when the question is rephrased or rerun?

Position can help when an answer gives an ordered list, but “first brand mentioned” isn’t a standard ranking across platforms. Name the metric for what it measures. And when a number improves after a content edit, check model changes, competitor updates and normal variation before crediting the edit.

A First-Month Workflow

Week 1: Talk to sales and support, collect real buyer concerns, and draft your main intent families.
Week 2: Shortlist the highest-value questions, pick a few natural phrasings, set platforms and locations, and decide how often you’ll check.
Week 3: Read the answers. Note gaps: missing requirements, unclear comparisons, stale facts, inaccurate descriptions, or confirmed access issues.
Week 4: Choose two or three changes with named owners. Record what changed and when, and keep tracking.

The outcome should be a work plan, not just a prettier graph.

What Good Prompt Tracking Gives You

The goal isn’t the biggest prompt library. It’s a clear picture of where your brand belongs in a buyer’s decision, how AI assistants describe it today, and which gaps your team can realistically close.

Start With Ten Real Questions

Prompt Intelligence in ApexaRank.ai helps you discover relevant questions, monitor brand and competitor visibility, and connect what you find to actionable work. Start with ten questions from real customer conversations before you expand the list.

EXPLORE PROMPT INTELLIGENCE

References

  1. Jack et al. (2026), Paraphrase Brittleness in Production Retrieval-Augmented Commercial Recommendation (arXiv preprint)

Ready to See How Your Brand Shows Up in AI?

See where you stand, uncover visibility gaps, and discover what to improve next.

CHECK YOUR AI VISIBILITY

Get your first visibility score in minutes.

SHARE THIS ARTICLE
Copied!
KEEP EXPLORING

Related Insights

STAY AHEAD OF AI SEARCH

Get Practical AI Search Insights

Useful ideas on AI visibility, AEO, GEO, content strategy, prompt intelligence, and the changing search landscape.

No noise. Just useful insights.