SEO & GEO

Resolve every known AI/LLM crawler against a site's robots.txt and report which may fetch a given path. Covers OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot), Anthropic (ClaudeBot, Claude-User, Claude-SearchBot), Google (Google-Extended, Googlebot, Google-CloudVertexBot), Perplexity, Apple, Meta, Amazon, Mistral, Common Crawl, ByteDance and others. Three things this gets right that a naive robots.txt reader does not: 1. **Training vs citation.** Blocking GPTBot stops training; blocking OAI-SearchBot stops you being *cited* in ChatGPT search. Most people want the first, not the second. Blocked citation-critical bots are called out separately. 2. **Which blocks are actually enforceable.** Perplexity-User, ChatGPT-User and meta-externalfetcher are documented by their own vendors as ignoring or possibly ignoring robots.txt. A "blocked" verdict for those is advisory, and is reported as such rather than as a clean block. 3. **Vendor quirks.** Apple documents that when robots.txt has no Applebot group but does have a Googlebot group, Applebot follows the Googlebot rules — so the effective verdict differs from the literal one. Each crawler also carries its provenance: whether the token comes from first-party vendor documentation or only from community aggregators. Vendors that publish no crawler token at all (xAI/Grok, Microsoft Copilot) are listed separately, because absence of a rule cannot be read as allowed or blocked. Args: - site (string): domain or any URL on it. - path (string): path to test (default '/'). - include_deprecated (boolean): include retired tokens (default false). - response_format ('markdown' | 'json'): output format (default 'markdown'). Returns: { crawlers[{token, vendor, purpose, allowed, via_wildcard, matched_rule, respects_robots_txt, compliance_note, provenance, quirk}], allowed_count, blocked_count, blocked_citation_critical[], unenforceable_blocks[], undocumented_vendors[], findings[] }. Example: "Can ChatGPT and Perplexity crawl example.com?" -> ai_crawler_access(site="example.com").

Fetch all four host/scheme variants of a domain — http/https × apex/www — and confirm they converge on a single canonical URL. Divergence is the classic cause of a homepage competing with itself in the index. Also reports whether plain HTTP is upgraded to HTTPS, whether canonicalisation uses permanent (301/308) rather than temporary (302/307) redirects, and which variants do not serve content at all. Args: - site (string): a domain such as 'example.com' (www and scheme are ignored). - response_format ('markdown' | 'json'): output format (default 'markdown'). Returns: { domain, variants[{variant, reachable, status, final_url, hop_count, redirect_statuses[]}], canonical_url, converges, distinct_endpoints[], forces_https, score, grade, findings[] }. Example: "Do all versions of example.com redirect to one URL?" -> canonical_host_check(site="example.com").

Measure the page's main content: word count, sentence and paragraph counts, Flesch reading ease with a plain-language reading level, estimated reading time, text-to-HTML ratio, thin-content detection, and the top non-stopword terms with their density (English and Spanish stopwords are both filtered). Content is read from the <main>/<article> landmark when present, so navigation and footer chrome do not inflate the counts. Args: - url (string): the page to check. - response_format ('markdown' | 'json'): output format (default 'markdown'). Returns: { word_count, sentence_count, paragraph_count, avg_words_per_sentence, reading_ease, reading_level, reading_time_minutes, thin_content, text_to_html_ratio, used_content_landmark, top_terms[{term, count, density}], score, grade, findings[] }. Example: "Is the content on https://example.com/post too thin?" -> content_analysis(url="https://example.com/post").

Score how readily an AI answer engine (ChatGPT, Claude, Perplexity, Gemini, Copilot) can fetch, parse and cite this page. Weighted across: AI crawler access (25), server-rendered content (20), structured data (15), extractable structure (15), authorship & entity signals (10), freshness (8) and content depth (7). Two things this catches that a classic SEO tool does not: - Pages that rank fine in Google but are invisible to AI assistants, because most AI crawlers do not execute JavaScript and the content only appears after hydration. - robots.txt rules that block AI *search* crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) — the ones that build citation indexes — as opposed to the *training* crawlers people usually mean to block. Args: - url (string): the page to audit. - response_format ('markdown' | 'json'): output format (default 'markdown'). Returns: { geo{score, grade, signals[], top_recommendations[]}, rendering, crawler_access, llms_txt, robots }. Example: "Is https://example.com/guide ready to be cited by ChatGPT?" -> geo_audit(url="https://example.com/guide"). Note: llms.txt presence is reported but deliberately NOT scored — it is a community proposal with no committed vendor support, and Google has stated it does not use it.

Extract the full h1–h6 outline and evaluate it: how many h1s, whether levels are skipped (h2 followed by h4), empty heading tags, and how many headings are phrased as questions — the last being a strong signal for featured snippets and AI citations. Args: - url (string): the page to check. - response_format ('markdown' | 'json'): output format (default 'markdown'). Returns: { headings[{level, text, skips_level}], h1_count, h1_text[], level_skips, empty_headings, question_headings[], outline, score, grade, findings[] }. Example: "Show me the heading outline of https://example.com/guide" -> heading_structure(url="https://example.com/guide").

Validate a page's `<link rel="alternate" hreflang>` annotations: language/region code validity (BCP-47), the required self-referencing entry, the x-default fallback, and duplicate codes. With check_reciprocity=true it fetches each alternate and confirms it links back to this page — non-reciprocal hreflang is silently ignored by Google, and it is impossible to detect from one page in isolation. Args: - url (string): the page to check. - check_reciprocity (boolean): verify alternates link back (default false). - response_format ('markdown' | 'json'): output format (default 'markdown'). Returns: { declared_lang, entries[{hreflang, href, valid_code, is_self, reciprocates}], has_x_default, self_referencing, duplicate_codes[], invalid_codes[], findings[] }. Example: "Is hreflang set up correctly on https://example.com/es/pagina?" -> hreflang_check(url="https://example.com/es/pagina", check_reciprocity=true).

Audit every <img> on the page: missing alt attributes (an accessibility failure and a lost image-search signal), decorative alt="" usage, missing width/height (which causes layout shift, a Core Web Vitals factor), lazy-loading adoption, and how many images use modern formats (WebP/AVIF) versus legacy JPEG/PNG. <picture> sources are counted as modern delivery. Args: - url (string): the page to check. - response_format ('markdown' | 'json'): output format (default 'markdown'). Returns: { total_images, missing_alt, decorative_alt, missing_dimensions, lazy_loaded, modern_format, legacy_format, images[], score, grade, findings[] }. Example: "Which images on https://example.com are missing alt text?" -> image_seo_check(url="https://example.com").

Audit a page's outbound links: the internal/external split, rel attributes (nofollow, sponsored, ugc), links with no anchor text at all, generic anchor text ("click here", "leer más") that carries no topical signal, and the distribution of external domains. Optionally sample-verifies that links actually resolve, retrying with GET when a server rejects HEAD. Args: - url (string): the page to audit. - check_broken (boolean): verify links resolve (default false). - sample_size (number): how many links to verify (default 25). - response_format ('markdown' | 'json'): output format (default 'markdown'). Returns: { total_links, internal_links, external_links, nofollow_links, empty_anchor_text, generic_anchor_text[], external_domains[{domain, count}], checked_count, broken[], score, grade, findings[] }. Example: "Are there broken links on https://example.com/resources?" -> link_audit(url="https://example.com/resources", check_broken=true).

Check whether a site publishes /llms.txt and validate it against the llmstxt.org proposal: a required H1 title, an optional blockquote summary, and H2-delimited lists of `- [name](url): notes` links. Also detects /llms-full.txt. **Important context this tool always reports:** llms.txt is a community proposal from September 2024, not an adopted standard. No major AI vendor has documented that its crawlers read llms.txt from third-party sites, and Google has publicly stated it does not support it. Publishing one is cheap and may help human readers and some documentation tooling, but it does not earn AI visibility on its own — robots.txt access, structured data and server-rendered content do. Note also that `llms-full.txt` is a de-facto convention popularised by docs tooling, not part of the proposal. Use this tool to answer "do they publish one, and is it well-formed?" — not as evidence that a site is or is not AI-optimised. Args: - site (string): domain or any URL on it. - response_format ('markdown' | 'json'): output format (default 'markdown'). Returns: { found, status, full_variant_found, bytes, title, has_summary_blockquote, sections[], link_count, spec_compliant, adoption_status, findings[] }. Example: "Does example.com publish an llms.txt?" -> llms_txt_check(site="example.com").

Inspect a page's head tags: title, meta description, canonical, robots directives (meta AND the X-Robots-Tag header), html lang, charset, viewport and favicon. Flags length problems, missing or duplicated tags, and anything that makes the page non-indexable. Args: - url (string): the page to check. - response_format ('markdown' | 'json'): output format (default 'markdown'). Returns: { title, title_length, description, description_length, canonical, canonical_is_self, meta_robots, x_robots_tag, indexable, followable, lang, charset, viewport, score, grade, findings[] }. Example: "Are the meta tags on https://example.com correct?" -> meta_tags_check(url="https://example.com").

Follow a URL's redirect chain hop by hop, reporting each status code and target. Flags long chains (which waste crawl budget), redirect loops, temporary 302/307 redirects where a permanent 301/308 belongs, and chains that do not end on HTTPS. Args: - url (string): the starting URL. - response_format ('markdown' | 'json'): output format (default 'markdown'). Returns: { final_url, final_status, hops[{url, status, location}], hop_count, https_upgrade, ends_https, has_loop, has_temporary_redirect, elapsed_ms, findings[] }. Example: "Where does http://example.com/old-page end up?" -> redirect_trace(url="http://example.com/old-page").

Determine whether a page's content exists in the server HTML, or only appears after JavaScript runs. This matters more for AI visibility than for classic SEO: Googlebot renders JavaScript, but GPTBot, ClaudeBot, PerplexityBot and CCBot largely do not. A client-rendered page can rank perfectly well in Google and still be completely invisible to every AI assistant — this tool is how you catch that. Detects unhydrated SPA shells (empty #root / #app / #__next containers), reports how many words survive without JS, and flags documents dominated by inline script bytes. Args: - url (string): the page to check. - response_format ('markdown' | 'json'): output format (default 'markdown'). Returns: { renders_without_js, server_text_words, script_bytes, html_bytes, spa_shell_detected, framework_hint, findings[] }. Example: "Can ChatGPT actually read https://example.com/app?" -> render_check(url="https://example.com/app").

Fetch and parse a site's robots.txt per RFC 9309. Reports every user-agent group with its Allow/Disallow rules, the declared sitemaps, and any lines that could not be parsed. Flags the two failures that silently deindex a site: a wildcard `Disallow: /`, and a robots.txt that returns 5xx (which Google treats as "disallow everything"). Args: - site (string): domain or any URL on it, e.g. 'example.com'. - response_format ('markdown' | 'json'): output format (default 'markdown'). Returns: { found, status, group_count, sitemaps[], blocks_everything, groups[{agents[], rules[], crawl_delay}], parse_warnings[], findings[] }. Example: "What does example.com's robots.txt allow?" -> robots_txt_check(site="example.com"). For AI-crawler specifics use `ai_crawler_access` instead — it resolves each known AI bot against these rules.

Fetch a page once and audit it across seven weighted sections — meta tags & social preview, heading structure, content quality, structured data, image SEO, links and crawlability — returning a 0–100 score, an A–F grade and a prioritised fix list. This is the tool to start with for any "how is this page doing for SEO?" question; drill into the single-purpose tools afterwards for detail. Args: - url (string): the page to audit. - include_geo (boolean): also score AI answer-engine readiness (default false). - check_broken_links (boolean): sample-verify that links resolve (default false). - response_format ('markdown' | 'json'): output format (default 'markdown'). Returns: { score, grade, indexable, sections[{id, label, score, grade, weight, issues[]}], top_recommendations[], geo, findings[] }. Example: "Audit the SEO of https://example.com/pricing" -> seo_audit(url="https://example.com/pricing"). Note: a noindex page or a site-wide robots.txt block caps the score, because nothing else matters until that is fixed. Errors: returns an error if the URL is unreachable, non-HTML, or returns an HTTP error.

Discover, fetch and validate an XML sitemap. Finds it via the robots.txt `Sitemap:` directive, then falls back to /sitemap.xml, /sitemap_index.xml and /sitemap-index.xml. Handles sitemap indexes (following children) and gzipped sitemaps. Validates: URL count against the 50,000 limit, uncompressed size against 50 MiB, <lastmod> presence and W3C-datetime validity, URLs pointing off-origin, http:// URLs, and duplicates. Args: - site (string): domain or any URL on it. - sitemap_url (string, optional): explicit sitemap URL. - follow_children (number): child sitemaps of an index to follow (default 3). - response_format ('markdown' | 'json'): output format (default 'markdown'). Returns: { found, type, url_count, child_sitemaps[], with_lastmod, invalid_lastmod[], newest_lastmod, off_origin_urls[], exceeds_url_limit, discovered_via, score, grade, findings[] }. Example: "Check the sitemap for example.com" -> sitemap_check(site="example.com").

Validate the tags that build link-preview cards on X, LinkedIn, Facebook, Slack, WhatsApp and Discord: og:title, og:description, og:image, og:url, og:type, og:site_name and the twitter:* family. Optionally verifies the preview image actually loads, and flags the classic bug of a relative og:image URL (social scrapers require absolute URLs). Args: - url (string): the page to check. - check_image (boolean): verify the og:image resolves (default true). - response_format ('markdown' | 'json'): output format (default 'markdown'). Returns: { open_graph{}, twitter{}, og_image_url, og_image_reachable, og_image_status, score, grade, findings[] }. Example: "Why does my link preview look broken on LinkedIn?" -> social_preview_check(url="https://example.com/post").

Extract and validate JSON-LD, microdata and RDFa. Reports every @type found, flags JSON-LD blocks that fail to parse (those are invisible to search engines), and checks recognised types against Google's rich-result requirements — required properties that are missing, plus recommended ones worth adding. Covers Article/BlogPosting/NewsArticle, Product, FAQPage, HowTo, Recipe, Event, Organization, LocalBusiness, Person, WebSite, BreadcrumbList, VideoObject, JobPosting, Course, Review and AggregateRating. Args: - url (string): the page to check. - response_format ('markdown' | 'json'): output format (default 'markdown'). Returns: { json_ld_blocks, microdata_items, parse_errors[], items[{type, properties[], missing_required[], missing_recommended[], valid}], types_found[], has_organization, has_breadcrumb, score, grade, findings[] }. Example: "Does https://example.com/product have valid Product schema?" -> structured_data_check(url="https://example.com/product").