# NotePlus — crawler policy. # Public, indexable surfaces: marketing pages, the public library # (/library/...), /explore, and the Exams store (/exams/...). # Everything app-internal either 302s to login or is noise for crawlers; # keep crawl budget on the pages that convert. # # IMPORTANT — how the groups below work: a crawler obeys only the single most # specific User-agent group that matches its name, and ignores every other group # including "*". So each named group repeats the same Disallow list. Adding a # path to "*" alone will NOT block it for any bot named further down. User-agent: * Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /actuator/ Disallow: /t/ Disallow: /geo-check Disallow: /swagger-ui/ Disallow: /api-docs/ # Deliberately NOT disallowed: # /library?q=… keyword searches are handled with a noindex meta tag instead. # Blocking them here would stop crawlers reading that tag, which # leaves the URLs eligible to be indexed from external links — # noindex only works on a page the crawler is allowed to fetch. # /library/*/*/study same reasoning: those pages carry noindex,follow so their # outbound links still pass value back to the detail pages. # ============================================================================= # Answer engines (AEO) — ALLOWED # ============================================================================= # These crawlers build the indexes behind AI answers that cite a source and link # back to it. Allowing them is how NotePlus becomes a citable answer to "free # HSC chemistry flashcards" inside ChatGPT, Perplexity and AI Overviews. # See /llms.txt for a machine-readable summary of what is here. # OpenAI — search index for ChatGPT, and user-initiated page fetches. User-agent: OAI-SearchBot Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /actuator/ Disallow: /t/ Disallow: /geo-check Disallow: /swagger-ui/ Disallow: /api-docs/ User-agent: ChatGPT-User Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /actuator/ Disallow: /t/ Disallow: /geo-check Disallow: /swagger-ui/ Disallow: /api-docs/ # Perplexity — search index and user-initiated fetches. User-agent: PerplexityBot Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /actuator/ Disallow: /t/ Disallow: /geo-check Disallow: /swagger-ui/ Disallow: /api-docs/ User-agent: Perplexity-User Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /actuator/ Disallow: /t/ Disallow: /geo-check Disallow: /swagger-ui/ Disallow: /api-docs/ # Anthropic — search indexing and user-initiated fetches (NOT ClaudeBot, which is # the bulk training crawler and is blocked below). User-agent: Claude-SearchBot Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /actuator/ Disallow: /t/ Disallow: /geo-check Disallow: /swagger-ui/ Disallow: /api-docs/ User-agent: Claude-User Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /actuator/ Disallow: /t/ Disallow: /geo-check Disallow: /swagger-ui/ Disallow: /api-docs/ # Google-Extended controls whether content can ground Gemini and AI Overviews. # It does not affect normal Google Search ranking either way — blocking it only # removes NotePlus from AI answers. User-agent: Google-Extended Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /actuator/ Disallow: /t/ Disallow: /geo-check Disallow: /swagger-ui/ Disallow: /api-docs/ # ============================================================================= # Bulk training crawlers — BLOCKED # ============================================================================= # These collect content for model training corpora with no attribution and no # referral traffic back. Blocking them costs no AEO visibility, because none of # them power a citing answer engine. User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: meta-externalagent Disallow: / # /sitemap.xml is a sitemap index; it points at /sitemap-pages.xml and the paged # per-type files (/sitemap-deck-0.xml, …). Only the index needs declaring here. Sitemap: https://www.noteplus.io/sitemap.xml