Our Methodology

1. What we fetch and how

For each domain we request five URLs, in parallel, from our server (not from your browser):

• / (the homepage)
• /robots.txt
• /sitemap.xml
• /llms.txt
• /llms-full.txt

One audit checks a whole site's front door, not every page. These files and the homepage control whether AI agents can reach, read and understand the site as a whole. We don't crawl through every internal page, so this is not a page-by-page SEO crawl.

Each request uses a desktop Chrome user-agent string, follows redirects, and is abandoned after 10 seconds. We read at most 3 MB of each response. There are no automatic retries: if a request times out or errors, that is what we record. A slow or flaky server can therefore get a worse result on one run than another; re-running the audit is the retry.

We treat HTTP 403, HTTP 429 or a recognised bot-challenge page as "blocked". If the homepage cannot be fetched or is blocked, the audit produces no score — the report shows the failure and the evidence instead. Every request, its status and the time it was made are kept in the audit's fetch log.

Scheduled checks and our crawler

Eligible Pro, Team and Agency sites are re-audited automatically (daily by default). Competitors have a separate per-project allowance (Pro 10, Team 15, Agency 25) and run only when you select Refresh comparison or run a manual check; they never receive scheduled background monitoring. Those scheduled checks identify themselves honestly with this user-agent: Mozilla/5.0 (compatible; AI-Primed-Monitor/1.0; +https://ai-primed.com/methodology#crawler). They fetch the same five URLs, on the site owner's request, at most a few times a day. They also request the homepage once per tested agent (GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Applebot, bingbot), sending that agent's published user-agent token followed by our own crawler name. Those requests come from our servers, so rules based on an agent's own network addresses are not observed.

Scheduled checks are compared only with earlier scheduled checks, so differences in user-agent never show up as a change. Timeouts, network errors, DNS failures and 5xx/429 responses are retried with growing delays and never reported as a change; a block (403 or challenge) must repeat on a second attempt before it is recorded.

2. The six categories and their weights

Each category starts at 100 and loses points for every finding, more for higher severity. The overall score is the weighted sum of the categories. Weights depend on the detected site type:

Commerce readiness only counts if you sell online. If you don't, that 20 points is spread across the other five categories so your score reflects what actually matters for your site. The per-finding deductions are listed on the docs page.

3. Grade bands

A 92 and above · B 80–91 · C 65–79 · D 50–64 · F below 50.

4. What "fails outright" means

A site fails outright when its homepage either refused our request (blocked) or did not answer at all (unreachable). In both cases there is nothing for an agent to read, so we do not compute category scores and the report shows no score ring or number.

A homepage that redirects to a different domain does not fail; we name the destination and offer to audit it instead.

A failed fetch is stored with the status "blocked" or "unreachable" and no score — never a score of 0. It counts toward the fail-outright rate, but not toward the average, median or grade distribution. Until 9 October 2026 these audits were stored as 0, which pulled the published average and median down; see the changelog.

5. How the public sample is drawn

• Unit: one domain, using its most recent completed audit. Repeat audits of a domain replace, not add to, its entry.
• Cohorts: every audit is tagged user, public_sample, seed or test. Imported lists, bulk audits and batch jobs are seed; automated checks are test. Public figures include all four cohorts.
• Included: scored sites, plus blocked and unreachable sites, which count as failing outright and carry no score.
• Excluded: earlier re-audits of the same domain, audits that did not complete, and homepages that redirected to another domain. Seed and test cohorts currently remain in these public headline figures; the peer comparisons below exclude tests.
• Size today: 250,310 domains, of which 78,121 could be scored. These figures are live.
• Date range: included audits run from 7 October 2026 to 24 October 2026.
• Publication guard: headline figures are withheld ("figures under review") when the sample has fewer than 500 scored audits or a median score of 0. The share of sites failing outright no longer withholds figures; it is always reported alongside them.
• Source of domains: sites users chose to audit and imported domain lists. This is not a random sample of the web, and we cannot claim it is representative.

Peer comparisons in reports and portfolios

We take the latest completed, blocked or unreachable audit per case-insensitive domain, with audit ID breaking timestamp ties. Only if that latest audit is completed, scored and reachable does the domain enter peer score distributions; we do not fall back to older scores after a failure or test. Failed fetches, redirected homepages and test audits are excluded; imported-list audits are included and their count is disclosed.

Comparisons match the scoring site type recorded on each audit: online store, content site, local business or SaaS. Legacy audits use the commerce weight actually applied: positive means online store, zero means content site. Historical scoring-version IDs are not recorded, so we cannot guarantee all older audits used identical engine rules. No industry classification is currently recorded, so we do not claim industry-specific benchmarks or silently substitute other site types.

A percentile is shown only with at least 500 distinct scored domains of the matching type. This conservative display threshold limits small-sample volatility (one site contributes at most 0.2 percentage points), but does not make the non-random sample representative. Smaller groups still show their sample count and median; empty groups say no comparison is available.

“Higher than” means strictly lower scores divided by all scored sites in that peer set, multiplied by 100, rounded to one decimal place. Ties are not counted as lower. The median is the middle score (the mean of the two middle scores for an even sample). Each scored category has its own median and sample count; commerce is not compared for non-stores. Category gaps are your score minus the peer median, ordered most negative first.

Snapshots refresh on demand after one hour; each comparison shows the actual computation time, not the view time. The reference on score-history charts is the current median for the latest audit's site type, not a historical median. These comparisons describe our audit dataset, not AI recommendations, rankings or future outcomes.

6. Known limitations

• Our probe is browser-like. We read robots.txt rules for individual AI agents, but we do not send requests as each AI bot, so we do not verify that a firewall actually lets GPTBot, ClaudeBot or others through.
• llms.txt is an emerging convention, not a ratified standard. Agents may or may not read it.
• Agentic commerce standards are early. We report on them, but we do not treat their absence as a failure.
• Site-type detection is heuristic, based on product links and structured data. It can be wrong; owners can correct it on the results page.
• Structured-data checks read the homepage only, not every page. A site with good Product schema on product pages but none on the homepage will be under-credited.
• We see the HTML before JavaScript runs. Agents that render JavaScript may see more than we do.
• One run is one moment. Outages, rate limits and geographic blocking can make a single result unrepresentative.

7. Changelog

1. 2026-10-09

   Failed fetches (blocked or unreachable) are stored with no score instead of 0, and every audit is tagged with a cohort. Public figures now use only user audits and the public sample; imported seed lists, bulk and batch runs, and tests are excluded. A publication guard withholds headline figures when the sample has fewer than 500 scored audits, more than 40% failing outright, or a median of 0.

   Effect on scores: Individual scores of reachable sites unchanged. Blocked and unreachable audits lost their stored 0. The public sample shrank to user audits only, so headline figures are withheld until it passes the guard.

2. 2026-10-08

   Benchmark statistics rewritten to compute in a single pass.

   Effect on scores: No change to any score. Public figures can now be refreshed across the full sample; before this, refreshes timed out and published figures lagged the audits actually run.

3. 2026-10-08

   Homepages we could not reach (DNS failure, connection refused, timeout) are now included in the public sample and counted as failing outright. Previously only firewall-blocked homepages were counted that way; unreachable ones were left out.

   Effect on scores: Individual scores unchanged. Public figures shifted: the sample grew and the share of sites failing outright and grading F rose.

4. 2026-10-07

   Site-type corrections: owners can re-score an audit as a different site type, with the previous and corrected score both kept.

   Effect on scores: Only the corrected audit changes, and only when its owner applies a correction.

5. 2026-10-07

   Scoring v2: commerce is scored only for online stores (its weight is redistributed for other site types); missing llms.txt, entity schema and server-rendered content carry extra weight; a blocked or unreachable homepage fails outright instead of receiving category scores.

   Effect on scores: Non-store sites stopped losing points for commerce checks. Sites missing the extra-weight items scored lower. Blocked sites moved to failing outright.

8. Disputing a result

• Open the report and check the fetch log and evidence. They show exactly what we received.
• If the failure looks temporary, re-run the audit. The new result replaces the old one in the public sample.
• If the site type is wrong, correct it on the results page. Both scores are kept.
• If your firewall blocked us, see how to allow AI agents, then re-run.
• If you believe a check itself is wrong, tell us which finding and why. We will review it and record any engine change and its effect in the changelog above.