5 November 2026, on publication. No site holds the GSERP mark, so no mark is affected.
Checker
Version 0.2.0, dated 5 November 2026
Editor
Swomi Group Inc.
Status
In public before version 1.0; see Versions. Changes are in the changelog.
1 Scope
GSERP 0.2 sets out how a site presents itself to four kinds of reader. In this standard the words mean:
Classic search
Engines that crawl the web and list results with links and snippets.
AI search
Search that writes an answer from pages it has indexed, with links to them.
Answer engines
Assistants and agents that fetch a page when a person asks about it.
Swomi's own search
The search that Swomi runs. It is not open yet.
It tests two things a site controls: the files of a release, the folder or zip of files a site is deployed from, and the way the live site answers requests. It does not test what the pages say, how fast they load, how secure the server is, or how accessible the pages are beyond their language and image text. It promises nothing about ranking or citation.
Other words used: the domain is the site's bare host name, such as example.com; a page and its address are defined in 3.3; a page for search is a page not kept out of search by PAGE-1.
2 Conformance
Each criterion has one of three levels. The words MUST and SHOULD are used as RFC 2119 defines them:[1] MUST is an absolute requirement, and SHOULD allows a different choice when its effects are understood.
Required
Written with MUST. A result can be FAIL, and one FAIL means the site does not meet the standard. Some required criteria also give a WARN for advice, such as a long title.
Advised
Written with SHOULD. A result can be WARN, never FAIL.
Information
Sorts pages, reads a configuration file or records a choice that is the site's own, and never fails.
Each result is PASS, WARN or FAIL. A criterion can give one result for the site, or one per page, per address or per crawler. The checker ends with a verdict line, for example GSERP release example.com: CERTIFIED (190 pass, 2 warn, 0 fail) [checker 0.2.0]. CERTIFIED means that no result failed, and NOT CERTIFIED that at least one did. That word is a test result: it is not the GSERP mark, which is granted only by the process in How certification works.
A site meets GSERP 0.2 when its release and its live site each give no FAIL.
3 How the checker runs
3.1 Two modes
Release mode reads a site's release with the site's domain, before the site goes live. It runs the groups CFG, FILE, ROBOTS, SITEMAP, NOTFOUND, LLMS, PAGE, LD and LINKS, in that order, except that SITEMAP-5 and SITEMAP-6 are tested page by page, together with the page groups. Live mode reads the deployed site over the internet by its domain. It runs LIVE, and ROBOTS on the live robots.txt. Each criterion below names the modes it runs in.
3.2 The release root
A release is a folder or a zip file. A zip is unpacked first. When it holds exactly one folder and no index.html at its top, that folder is the root; otherwise its top is.
3.3 Pages and their addresses
Every file named index.html in the release is a page. The root's index.html has the address https://<domain>/; one in a folder has https://<domain>/<folder>/, always with the final slash. Other HTML files, such as 404.html, are not pages. Pages are tested in the order of their paths, and the detail of each page result begins with the page's path. The domain is read in lower case.
3.4 The existence rule
Wherever a criterion asks whether a path exists in the release, the checker first decodes percent-escapes. A path that ends in / exists if <path>index.html is a file. Any other path exists if it is a file, or if <path>/index.html is.
3.5 Requests in live mode
Every request is a GET. Redirects are not followed, so a redirect is reported as its own status; the one exception is a redirect of robots.txt, followed only to find the rules the checker obeys. Each request waits up to 20 seconds. The checker identifies itself with the user agent GSERP-check/0.2.0 (+https://swomi.ai/), whose product token in robots.txt is GSERP-check. A request that gets no answer is reported as no response, with the error.
The checker obeys robots.txt (LIVE-8). Its first request is for /robots.txt, and every later request follows the rules there for GSERP-check, or for the * group. If they close the site to the checker, it reports robots.txt, says plainly that it was asked to stop, and stops. It fails nothing for that.
3.6 Output
The checker lists every WARN and FAIL with its label and detail, then the verdict line, which is always the last line and ends with the checker's version in square brackets, [checker 0.2.0]. PASS results are listed on request. Its exit status is 0 when no result failed, 1 when one did, and 2 when it was called wrongly.
4 Criteria
54 criteria in 10 groups: 31 required, 20 advised and 3 for information. Each has an id, the label the checker prints, its level and the modes it runs in. The groups follow the checker's label prefixes, in the order it runs them. Every detail shown under Results is an example.
Paths served by the site's application are declared
Checker label: config: .gserp.json
Requires
Nothing: this file is optional. A release may carry a file named .gserp.json at its root: a JSON object whose app_paths member lists the paths that the site's application serves instead of files in the release, such as a sign-in path.
Why
Some sites answer a few paths from an application rather than from files, so a release cannot contain those pages. Declaring them lets the link test (LINKS-2) pass over links to them, without hiding any other broken link.
Test method
If the file exists at the release root, the checker reads it as JSON and takes its app_paths list, or an empty list if the member is absent. A link then counts as an app path when its path equals one of the listed paths, ignoring a final slash, or starts with one of them followed by /. The live check does not read this file.
Results
PASS
The file was read. The detail lists the app paths. Example detail: app paths: /login, /app
No result is reported when the file is absent. A file that is not valid JSON stops the check with an error, and the release does not pass.
Since
Checker 0.1.1. Before it, links to paths served by an application were reported as broken.
The llms.txt proposal places a short guide for agents at /llms.txt.[6] No large search engine requires one: Google says no special AI text files are needed to appear in its AI features.[7] GSERP 0.2 still requires the file so that agents and answer engines that do read it find the whole site, Swomi's own search among them. A later version is proposed to make it optional.
Test method
The checker looks for a file with exactly this name at the release root.
The site's icon at the usual address. Search results can show a site's icon next to its pages when the icon is a file a crawler can fetch.[8]PAGE-13 tests that each page points to one.
Test method
The checker looks for a file with exactly this name at the release root.
Results
PASS
The file is there.
FAIL
It is not. Example detail: missing
Since
Checker 0.1
4.3robots.txt robots:
Who may read the site, its owner's choice about AI training, and where its sitemap is.
6 criteria: 2 required, 3 advised, 1 for information
Table 1. The fifteen user agents of ROBOTS-1 and ROBOTS-4, in the checker's three lists
Agent
Operator
Used for, as the operator documents it
Web search and general agents, ROBOTS-1: must be open
The site root is open to search and answer crawlers
Checker label: robots: <agent>, one result for each agent in the first two lists of table 1
Requires
robots.txt MUST NOT disallow the root path / for any of the ten user agents in the first two lists of table 1: the general group *, the two web search crawlers and the seven AI search and answer agents. The five AI training crawlers and tokens are not part of this criterion: whether they are open is the site owner's choice (ROBOTS-4).
Why
robots.txt is how a site tells crawlers what they may fetch.[2] Each name in the table is the token its operator documents for robots.txt.[9][10][11][12][13][14][15] A site closed at the root cannot be found or read by that reader, and GSERP is about being found. Two operators note that their user-initiated agents may not follow robots.txt;[11][12] the checker tests them like the others.
Test method
The checker reads robots.txt into groups. A User-agent line starts a new group when it follows a rule line (or is the first line); consecutive User-agent lines share one group. Allow and Disallow lines add rules to the current group; rules before the first User-agent line are ignored. Text after # is ignored, lines without a colon are ignored, and field names are read in any case. For each agent, the checker takes every group whose User-agent value equals the agent's name, ignoring case; if there is none, every group for *. Among those groups' rules it keeps the ones whose path pattern matches /, reading * as any run of characters and a final $ as the end of the path; empty rules are skipped. The longest pattern wins, and Allow wins a tie. The root is open if no rule matches or the winning rule is Allow. This follows RFC 9309.[2]
Results
PASS
The root is open to this agent.
FAIL
A rule closes the root to this agent. Example detail: root is disallowed
Since
Checker 0.1. Changed in checker 0.2.0: the five AI training crawlers and tokens moved to ROBOTS-4. Before, a site that closed any of them failed here.
robots.txt SHOULD name each of the twelve AI search, answer and training agents of table 1 in a User-agent line.
Why
Naming each agent states the site's choice for each one, instead of leaving it to the * group. Operators document their agents as separate settings that a site can open or close one by one.[11][12]
Test method
The checker collects every User-agent value in the file and checks that each of the twelve names is among them, ignoring case. A name must be written in full: a group for another agent whose name merely contains it does not count.
Results
PASS
All twelve are named.
WARN
Some are not. The detail lists them. Example detail: not named: Applebot-Extended, CCBot
robots.txt MUST contain the line Sitemap: https://<domain>/sitemap.xml, with the site's bare domain.
Why
The Sitemap line tells crawlers where the sitemap is, and its value must be a full URL.[3][4] RFC 9309 lets crawlers read records beyond its own rules, the sitemap among them.[2]
Test method
The checker finds every line that starts, after any spaces, with sitemap: in any case, and takes the text after it up to the first space. One of them must equal https://<domain>/sitemap.xml exactly. A sitemap named at another address, such as the www. host, http:, or another file name, does not count.
Results
PASS
The exact line is there.
FAIL
It is not. Example detail: expected Sitemap: https://example.com/sitemap.xml
Checker label: robots: <agent>, one result for each agent in the AI training list of table 1
Requires
Nothing. Opening or closing the root path / to any of the five AI training crawlers and tokens in the last list of table 1 is the site owner's choice, and either way the result is a pass. What GSERP asks is that the choice be declared clearly and consistently (ROBOTS-6).
Why
A site's content is its owner's, and so is the choice to let it train models. These agents collect pages for training, or control that use; they do not decide whether a site can be found. Google says that Google-Extended does not affect inclusion or ranking in Google Search, and Apple that Applebot-Extended does not crawl pages.[9][14] The search and answer crawlers that find and cite a site stay under ROBOTS-1.
Test method
The checker resolves each agent's rules exactly as in ROBOTS-1 and reports whether the root is open or closed. This criterion never fails and never warns.
Results
PASS
The root is open to this agent.
PASS
The root is closed to this agent: the owner's choice. Example detail: root is disallowed: the owner's choice
Since
Checker 0.2.0 (new). Before checker 0.2.0 these five agents were part of ROBOTS-1, and closing one of them failed.
Where robots.txt has Content-Signal lines, each SHOULD be written as Cloudflare's Content Signals Policy defines it: a comma-separated list of the signals search, ai-input and ai-train, each set to yes or no, such as Content-Signal: search=yes, ai-train=no. A site does not need these lines.
Why
The Content Signals Policy adds a line to robots.txt that says how content may be used after it has been accessed: for search, as input to AI models, or for training or fine-tuning them. A yes grants a use, a no withholds it, and a signal left out neither grants nor restricts it. The signals are preferences, not technical measures against scraping.[16] GSERP reads them so that a report shows the owner's stated preferences beside the access rules.
Test method
The checker reads every line whose field name is Content-Signal, in any case, in the group where it appears; a line before the first User-agent line is kept for the whole file. These lines never start or end a group, so they do not change how ROBOTS-1 reads the rules. The value is split at commas, and each part must be a signal name and a value joined by =, in any case. The search and ai-input values are reported, not tested; ai-train is used by ROBOTS-6.
Results
PASS
Every part was understood. The detail lists the signals by group. Example detail: *: search=yes, ai-input=yes, ai-train=no
WARN
A part was not understood. The detail lists it, then what was read. Example detail: not understood: ai-trian=no; read: *: search=yes
No result is reported when robots.txt has no Content-Signal line.
The choice about AI training is declared clearly and consistently
Checker label: robots: training preference
Requires
robots.txt SHOULD declare the site's choice about AI training, and SHOULD declare it consistently. It is declared when robots.txt names at least one of the five AI training agents of table 1 in a User-agent line, or carries an ai-train value in a Content-Signal line (ROBOTS-5). Which choice the site makes is its own: allowing and refusing pass alike (ROBOTS-4).
Why
A crawler can follow only a choice it can read. Naming an agent states the site's choice for that agent instead of leaving it to the * group (ROBOTS-2), and an ai-train signal states it for the use itself.[16] When the two say opposite things, no reader can tell which one the owner meant.
Test method
If robots.txt names no training agent and has no ai-train value, the choice is not declared. Otherwise, for each training agent that robots.txt names, the checker takes the ai-train values that speak for it: those in a group that names it, or else those in a * group or before the first group. It is a contradiction when the value is yes and the agent's root is closed, when it is no and the root is open, or when one agent gets both values; so is a * group, or the part before the first group, that says both. Agents that robots.txt does not name are not compared: for them the * group's rules are a general rule, not a statement about training. The detail names the first contradiction found.
Results
PASS
Declared, with no contradiction. The detail sums up the choice. Example detail: robots.txt disallows all 5 training agents; Content-Signal ai-train=no
WARN
Not declared. Example detail: training preference not declared: no training agent named, no Content-Signal ai-train
WARN
Declared in ways that contradict each other. Example detail: Content-Signal ai-train=yes, but robots.txt disallows GPTBot
Since
Checker 0.2.0 (new)
4.4Sitemap sitemap:
The list of pages the site wants found, and nothing else.
sitemap.xml MUST be well-formed XML in the sitemaps.org 0.9 format, with a <loc> in every <url>.
Why
The sitemap protocol is XML, declares its namespace, and requires a <loc> for every URL.[4]
Test method
The checker parses the file as XML and reads each url element directly under the root, in the namespace http://www.sitemaps.org/schemas/sitemap/0.9, with its loc. Elements in any other namespace are not read, so a sitemap without the namespace lists no pages and every page then fails SITEMAP-5. Sitemap index files are not followed.
Results
PASS
The file parses. The detail gives the number of URLs. Example detail: 6 URLs
FAIL
It does not parse, or a url has no loc. The detail gives the parser's message. Example detail: mismatched tag: line 4, column 2
Every address in the sitemap MUST resolve to a page or file in the release.
Why
A sitemap should hold the pages a site wants in search results.[5] An address that leads nowhere sends crawlers to an error instead.
Test method
For each address on the site, the checker takes its path and decodes percent-escapes. A path that ends in / exists if <path>index.html is a file in the release; any other path exists if it is a file, or if <path>/index.html is. This is the existence rule used throughout (3.4).
Results
FAIL
An address does not exist in the release. Example detail: https://example.com/old-page/
A page kept out of search (PAGE-1) MUST NOT be listed in the sitemap.
Why
A sitemap is for the pages a site wants in search results,[5] and a noindex page asks to be left out of them.[17] Listing it gives readers two opposite instructions.
Test method
For each page kept out of search, the checker looks for its address among the <loc> values.
Results
FAIL
A page kept out of search is listed. Example detail: https://example.com/report/
No result is reported otherwise.
Since
Checker 0.1.1. Added together with the change to PAGE-1.
4.5Not-found page 404:
The page served for an address that does not exist.
An error page must not be indexed as if it were content. A page that says it does not exist while it is served as a success is what Google calls a soft 404.[18]noindex keeps a page out of the index,[17] and LIVE-6 tests that the live site also answers with a real 404 status.
Test method
The checker reads 404.html and looks for the text noindex anywhere in it, in lower case. It does not check that the text is in a robots meta tag.
Results
PASS
The text is there.
FAIL
It is not. Example detail: 404 page must be noindex
Since
Checker 0.1
4.6llms.txt llms.txt:
A short guide to the site for agents and answer engines.
llms.txt SHOULD link every address in the sitemap.
Why
llms.txt lists a site's content as Markdown links.[6] Listing every page gives an agent the whole site in one file.
Test method
For each <loc> in the sitemap, the checker looks for the text (<address>), the address in brackets as a Markdown link target, anywhere in the file. Titles and descriptions are not checked.
Results
PASS
Every address is linked.
WARN
Some are not. The detail lists them. Example detail: not listed: https://example.com/about/
Since
Checker 0.1
4.7Each page page:
What every page says about itself in its head: language, title, description, canonical address, previews and icon.
15 criteria: 7 required, 7 advised, 1 for information
Each page is either for search or kept out of it on purpose
Checker label: page: indexable, page: kept out of search
Requires
Nothing beyond a clear choice. A page carries a robots meta tag with noindex when it should stay out of search, such as a private report; otherwise it is a page for search.
Why
noindex is the documented way to keep a page out of search results,[17] and a page kept out on purpose should not be judged as a search page.
Test method
The checker reads the page's head, the text before </head>. It takes the first <meta> whose name is robots in any case, and reads its content. If that contains noindex, written in lower case, the page is kept out of search: only LINKS-1 and LINKS-2 apply to it, and SITEMAP-6 keeps it out of the sitemap. Otherwise every criterion in the page, structured data and links groups applies. Meta tags for one crawler, such as name="googlebot", and the X-Robots-Tag HTTP header are not read.
Results
PASS
A page for search, reported as page: indexable.
PASS
A page kept out of search on purpose, reported as page: kept out of search. Example detail: noindex on purpose; not checked as a search page
Since
Checker 0.1. Changed in checker 0.1.1. Before, a page with noindex failed (page: indexable, "robots meta says noindex").
The head MUST contain a <title> with text. The title SHOULD be 70 characters or fewer.
Why
Google asks for a title in the <title> element of every page.[20] It sets no maximum length and shortens long titles to fit the screen;[20] the 70-character advice is GSERP's own, so that the whole title is more likely to show.
Test method
The checker takes the first <title>…</title> in the head, written without attributes, decodes HTML entities and trims spaces, then counts characters.
Results
PASS
A title of 70 characters or fewer.
WARN
A title longer than 70 characters. Example detail: 74 chars
FAIL
No title, or an empty one. Example detail: missing
Each page for search MUST have a title that no other page has.
Why
Google asks for distinct text that describes each page, and warns against titles that differ in only one detail.[20] Readers that cite a page by its title need it to name one page.
Test method
Titles are compared exactly, after decoding and trimming, across the pages for search in order of their paths. The second page with a title already seen fails, and the detail names the first.
Results
FAIL
The title repeats another page's. Example detail: same as /about/
The head MUST contain <meta name="description"> with text. It SHOULD be 50 to 170 characters long.
Why
Google sometimes uses the meta description for the snippet under a result.[21] It sets no limit on length and truncates snippets as needed;[21] the 50 to 170 range is GSERP's own advice, long enough to say what the page is and short enough to show whole.
Test method
The checker takes the first <meta> in the head whose name is description in any case, decodes HTML entities in its content, and counts characters.
Results
PASS
A description of 50 to 170 characters.
WARN
Shorter or longer. Example detail: 182 chars
FAIL
No description, or an empty one. Example detail: missing
The head MUST contain <link rel="canonical"> whose href is the page's own full address, such as https://example.com/about/.
Why
The canonical link names the preferred address for a page's content, and it is the address that is shown.[22] Google recommends a canonical link on the canonical page itself, with an absolute address.[22] It is also the address an answer should cite.
Test method
The checker takes the first <link> in the head whose rel contains the word canonical and compares its href with the page's address (3.3) as text. A relative address, http:, the www. host or a missing final slash all fail.
Results
PASS
The canonical is the page's own address.
FAIL
It is missing or different. The detail gives both. Example detail: want https://example.com/about/, got none
The head SHOULD contain <meta property="og:title">, og:description and og:type, each with content.
Why
The Open Graph protocol lets a page become a rich object, such as a link preview. og:title and og:type are among its required properties and og:description is recommended.[23]
Test method
For each of the three, the checker takes the first <meta> whose property is that name and checks that its content is not empty. Each is reported on its own.
When og:image is an address on https://<domain>/, that file MUST be in the release.
Why
A preview image that does not exist leaves every preview of the page without its picture.
Test method
The checker takes the path of the address and applies the existence rule (3.4). An image on another host is not checked, and nothing is reported for it.
Results
PASS
The file is in the release.
FAIL
It is not. Example detail: https://example.com/og.png not in release
The head SHOULD contain <meta name="twitter:card">.
Why
The tag names the preview layout, such as summary_large_image, for services that read it. No source is cited for it: its publisher's documentation could not be opened when this version was published.
Test method
The checker takes the first <meta> whose name is twitter:card in any case and checks that it has content.
The head MUST contain at least one <link rel="icon"> whose href is a file, not a data: address, and every such file named by a path from the root MUST be in the release.
Why
Google asks for a <link rel="icon"> that points to an icon file its crawler can fetch, at an address that stays the same.[8] A data: address is not a file at an address, so GSERP does not count it.
Test method
The checker collects every <link> in the head whose rel contains the word icon, so rel="icon" and rel="shortcut icon" count and rel="apple-touch-icon" does not. It drops data: addresses. At least one must remain, and each remaining href that starts with / must pass the existence rule (3.4). Full addresses and relative paths are not looked up. The test runs on every page for search, not only the home page.
Results
PASS
An icon file is named and present.
FAIL
None is named, only a data: address is, or a named file is missing. Example detail: search engines need a crawlable favicon file, not a data: URI
The level-one heading is the top of a page's heading structure and names its main content.[24] No engine requires exactly one, which is why this is advice.
Test method
The checker counts <h1 followed by a space or >, in lower case, anywhere in the file.
Results
PASS
Exactly one.
WARN
None, or more than one. Example detail: 2 h1 elements
A block that does not parse says nothing to any reader.
Test method
The checker parses each block. From each that parses it collects the nodes: if the block is a list, each of its objects; for each object, the members of its @graph, or the object itself when it has no @graph. These nodes are what LD-3 to LD-7 read.
Results
FAIL
A block does not parse. One failure per block, with the parser's message. Example detail: Expecting ',' delimiter: line 1 column 212
The structured data MUST include a node whose url is the page's own address and whose @type is WebPage, any of its eleven direct subtypes, or WebSite (table 2).
Why
Statements about a page need a node for the page itself to hang from. Schema.org's WebPage has eleven direct subtypes for particular kinds of page.[26]
Test method
Among the collected nodes, one must have a url equal to the page's address and a @type, written as a string or a list, that includes one of the thirteen types in table 2.
Results
PASS
Such a node is there.
FAIL
It is not. Example detail: no WebPage/ProfilePage node for this URL
Since
Checker 0.1. Changed in checker 0.1.1: every direct subtype of WebPage is accepted. Before, only WebPage, ProfilePage, AboutPage, CollectionPage and WebSite were.
The structured data SHOULD include an Organization or a Person node.
Why
Organization markup helps engines tell one organization from another and understand who it is.[29] A reader that cites a page needs to know whose page it is.
Test method
The checker looks for a node whose @type includes exactly Organization or Person. Subtypes such as Corporation are not recognised in 0.1.
Results
PASS
One is there.
WARN
Neither is. Example detail: no Organization or Person
Every node reference, an object holding only @id, SHOULD point to a node that the same page defines.
Why
In JSON-LD, @id identifies a node, and a node reference is an object that has only the @id key.[30] A reference to a node the page never defines leaves a reader of that page with a gap in the graph.
Test method
The checker walks every collected node and everything inside it. An object whose only key is @id is a reference; any other object with an @id defines that identifier. References with no matching definition are reported.
Results
PASS
Every reference resolves.
WARN
Some do not. The detail lists them. Example detail: unresolved: https://example.com/#org
sameAs SHOULD list only pages that identify the same entity, such as its profile pages. Related things, such as a founder's company or a patent, belong in other properties, such as founder or creator.
Why
Schema.org defines sameAs as the address of a reference page that unambiguously indicates the item's identity.[31] Google's example for an organization is a profile on another site.[29]
Test method
For each collected node that has a url, the checker reads its sameAs list. An address counts against the node when its host, ignoring www., differs from the host of the node's url, and it matches none of these patterns for identity pages: sec.gov, linkedin.com, twst.com/bio, wikidata, wikipedia, x.com, twitter.com, facebook.com, instagram.com, youtube.com, crunchbase. Nodes without a url are not tested. Write sameAs as a list (see 5).
Results
WARN
A node's sameAs names another entity. The detail gives the node's name and the first such address. Example detail: Example Ltd: sameAs should identify the same entity: https://example.org/
No result is reported otherwise.
Since
Checker 0.1
4.9Links links:
Links a crawler can follow, and that lead somewhere.
The page MUST NOT contain an <a> element that carries a data-go attribute and no href.
Why
Google can reliably follow a link only when it is an <a> element with an href.[32] A link that works only through a script is invisible to a crawler.
Test method
The checker counts <a> tags that have data-go= and no href=. This applies to every page, kept out of search or not. Other links that work only through scripts are not detected in 0.1.
Results
PASS
None found.
FAIL
Some found. The detail gives the number. Example detail: 2 <a data-go> without href
Every link from a page to another place on the same site MUST resolve to a page or file in the release, or to a path the site's application serves (CFG-1).
Why
Crawlers find pages by following links,[32] and a link to a missing page leads them to an error that is never indexed.[18]
Test method
The checker reads the href of every <a> written in double quotes. It passes over addresses that start with http:, https:, mailto:, tel:, #, //, data: or javascript:, except full addresses on https://<domain>/, which it treats as links within the site. It removes the fragment and the query, skips an empty path, resolves a relative path against the page's folder, passes over declared app paths, then applies the existence rule (3.4). Fragments are not checked. This applies to every page, kept out of search or not.
Results
PASS
Every link resolves.
FAIL
Some do not. The detail lists them. Example detail: broken: /old-page/, /login
Since
Checker 0.1. Changed in checker 0.1.1: declared app paths are passed over.
4.10The live site live:
How the site answers on the internet once it is deployed, read only as its robots.txt allows.
/robots.txt, /sitemap.xml and /llms.txt MUST each answer 200.
Why
These are the files readers fetch first, and a release that has them (FILE) must also serve them.
Test method
One request each, not followed. robots.txt is the live check's first request (LIVE-8), and that answer is the one tested here; the sitemap and llms.txt are requested only where robots.txt lets the checker fetch them. The live robots.txt, with any byte-order mark removed, then goes through ROBOTS-1 to ROBOTS-6. The addresses in the live sitemap are read as the text between each <loc> and </loc>, for LIVE-7. The live llms.txt is fetched but its content is not tested.
Results
PASS
200.
FAIL
Any other status, or no answer. Example detail: got 404
Since
Checker 0.1. Changed in checker 0.2.0: robots.txt is requested first, and the other two files only where it allows the checker.
Nothing is required of a site for its own sake: robots.txt decides what the checker may fetch, and a site that closes itself to the checker is not failed for it. For the live criteria to be tested, robots.txt SHOULD let the user agent GSERP-check fetch the site.
Why
robots.txt is how a site tells crawlers what they may fetch,[2] and the checker is a crawler like any other. A site's choice to close itself to the checker is respected, and the report says plainly what was not tested.
Test method
The live check's first request is for /robots.txt, which RFC 9309 always allows.[2] If it redirects, the checker follows the redirects, up to ten, to find the file. An answer of 200 gives the rules: those of the groups for the product token GSERP-check, or else of the * group, resolved as in ROBOTS-1 but for each address's path and query instead of only /. An answer from 300 to 499 after the redirects means no rules apply. No answer, or an answer of 500 or more, means robots.txt is unreachable, which RFC 9309 treats as a complete disallow.[2] The same rules are applied to the http:// and www. addresses of LIVE-3 and LIVE-4. If / is closed, the checker reports LIVE-5 for robots.txt and the ROBOTS group from it, then this result, and stops. Otherwise it skips each address that is closed to it, and lists them.
Results
PASS
robots.txt let the checker fetch every address it needed. Example detail: robots.txt allows GSERP-check
WARN
robots.txt closes the site to the checker, or cannot be read. The live check stopped there, and the other live criteria were not tested. Example detail: robots.txt disallows GSERP-check: the live crawl stopped here, as it asks; the other live criteria were not tested
WARN
Some addresses were closed to the checker and not fetched. Example detail: robots.txt disallows GSERP-check for 1 address, not fetched: /private/
When robots.txt is unreachable, LIVE-5 fails for it, as it did before: the site fails because robots.txt does not answer, not because the checker stopped.
Since
Checker 0.2.0 (new)
5 Known limits of checker 0.2.0
The test methods above say exactly what the checker does, limits included. These are the limits worth knowing before relying on a result.
robots.txt: only the root path / is tested (ROBOTS-1, ROBOTS-4). Rules that close deeper paths are not judged. Groups apply only to the exact agent name.
Content signals: Cloudflare's policy does not say whether a Content-Signal line speaks only for its own group. The checker reads an agent's own group first, then the * group (ROBOTS-6). Only agents that robots.txt names are compared with the signal. The search and ai-input values are reported, not tested (ROBOTS-5).
Kept out of search: only the first robots meta tag is read, and only noindex in lower case (PAGE-1). Meta tags for one crawler and the X-Robots-Tag header are not read. On the not-found page, the text noindex anywhere is enough (NOTFOUND-1).
Titles: a <title> written with attributes is not found (PAGE-3).
Structured data: subtypes of Organization and Person are not recognised (LD-5). A sameAs written as a single address instead of a list is read letter by letter and gives a false warning, and the pattern x.com also matches any address that contains those characters (LD-7). Fixes for both are proposed for a later version.
Links: only href values in double quotes are read, and fragments are not checked (LINKS-2). Of the links that work only through scripts, only <a data-go> is detected (LINKS-1).
Sitemap: only /sitemap.xml is read; sitemap index files are not followed (SITEMAP-1).
Live mode: page tests other than the canonical link are not repeated live, and the live llms.txt is not read (LIVE-5, LIVE-7). The checker does not pace its requests. The rules of https://<domain>/robots.txt are applied to the http:// and www. addresses too, whose own robots.txt files are not read, and paths are matched as requested, without decoding percent-escapes (LIVE-8). A live check that robots.txt stopped tested only robots.txt.
Configuration: a .gserp.json that is not valid JSON stops the check (CFG-1).
6 Proposed for a later version
These are planned, not part of GSERP 0.2 and not tested by checker 0.2.0. Each will be published here as a criterion, with its test method and level, before any check uses it.
Crawl and index
The site answers crawlers, not only its robots.txt: no firewall or delivery network blocks them. Pages are indexable with snippets allowed, and nosnippet or max-snippet is used only on purpose. An IndexNow key file and working submissions. The site verified in the main webmaster tools, with their AI reports read where they exist, which the owner attests.
Page quality
Real authors with their backgrounds. A note on how a page was made where readers expect one, AI use included. No scaled near-duplicate pages, scraped content, expired-domain abuse, date-bumping or back-button hijacking. No hidden text or instructions aimed at AI answers. Facts stated directly, names used consistently, and the key information near the top.
Structured data
Valid, matching the visible text, and using only features the engines currently support, with deprecated types flagged.
Provenance and disclosure
AI-generated images labelled with the IPTC digital source type trainedAlgorithmicMedia, and C2PA content credentials where used. Chatbots say that they are AI. Labels for AI-generated content where the law requires them.
Speed and access
Core Web Vitals from public field data; HTTPS; pages that work on mobile; no intrusive interstitials.
llms.txt
Optional: tested for its format only when present, instead of required by FILE-4.
The checker itself
A public page for the checker as a crawler, with its user agent and the addresses it fetches from, published before the free check's first request. The checker pacing its requests.
7 Changelog
GSERP 0.2, with checker 0.2.0
A minor version: criteria were changed and added. It takes effect on publication, 5 November 2026; no site holds the GSERP mark, so no mark is affected. GSERP 0.1 is kept unchanged at its own address.
AI training is the site owner's choice. Opening or closing the five AI training crawlers and tokens never fails. They leave ROBOTS-1, which now covers the general group, the web search crawlers and the AI search and answer agents, for the new ROBOTS-4, which records the choice and always passes. Before, closing any of them failed ROBOTS-1. Closing a search crawler, or an AI search and answer agent, still fails, as before: GSERP is about being found.
The choice is declared clearly. The new ROBOTS-6 warns when robots.txt names no training agent and has no ai-train content signal, and when the two contradict each other, such as ai-train=yes with GPTBot closed.
Content signals are read. The new ROBOTS-5 reads the Content-Signal lines of Cloudflare's Content Signals Policy and reports them, with a warning for a part it cannot read.
The live check obeys robots.txt. It asks for robots.txt first and follows its rules for GSERP-check. When a site closes itself to the checker, the report says so plainly, the live check stops, and nothing fails for it (new LIVE-8, LIVE-5).
The version shows everywhere. The user agent is now GSERP-check/0.2.0 (+https://swomi.ai/), and the verdict line ends with [checker 0.2.0].
No site that met GSERP 0.1 fails GSERP 0.2: every change removes a failure or adds advice. A release check now reports one more result, ROBOTS-6, and one more again where robots.txt has content signals; a live check also reports LIVE-8.
Proposed for a later version, and not part of 0.2:
A sameAs written as a single address is read letter by letter and gives a false warning (LD-7).
The pattern x.com in LD-7 also matches any address that contains those characters.
The list of identity pages that LD-7 accepts in sameAs, which includes twst.com/bio, to be reviewed.
GSERP 0.1, published
First publication, on gserp.co. The text is written from checker 0.1.1, one criterion for each check it reports. No test changed.
Checker 0.1.1
Three changes:
Pages kept out of search on purpose. A page whose robots meta tag says noindex is now accepted as kept out of search: only its links are tested, and it must not be in the sitemap (PAGE-1, SITEMAP-5, new SITEMAP-6). Before, any such page failed page: indexable, and every page had to be in the sitemap.
App paths in .gserp.json. A release can declare the paths its application serves, and links to them are no longer reported as broken (new CFG-1, LINKS-2).
Every WebPage subtype. The page node may now be any of the eleven direct subtypes of WebPage, such as ContactPage or FAQPage (LD-3). Before, only WebPage, ProfilePage, AboutPage, CollectionPage and WebSite counted.
None of these makes a release fail that passed before, with one exception: a .gserp.json that is not valid JSON now stops the check.
Checker 0.1
First version, used to test site releases before they were deployed. It was not published as a separate document; it differs from 0.1.1 only in the three points above.
References
Numbered in the order they are first cited on this page; each entry links back to the places that cite it. Every source was opened on 5 November 2026.