# 2026-10-26 Paul: complete good bot allow list, agreed by Perplexity, ChatGPT and Grok (2 of 3 vote; see job folder CSV).
# Search engines, AI search and assistants, AI training, ad checks, link previews, feed readers, uptime monitors and safe reputable SEO tools may crawl.
# All other bots: stay out. Skipped on purpose: Amazonbot (training; Amzn-SearchBot and Amzn-User kept, agreed ChatGPT + Perplexity 26 Oct), Bytespider, TikTokSpider, DeepSeekBot, YandexBot, Baiduspider, Sogou, Omgilibot, ImagesiftBot, MegaIndex, BLEXBot, Barkrowler, ia_archiver.
User-agent: AdsBot-Google
User-agent: AdsBot-Google-Mobile
User-agent: AdsBot-Google-Mobile-Apps
User-agent: APIs-Google
User-agent: FeedFetcher-Google
User-agent: Google-Agent
User-agent: Google-CloudVertexBot
User-agent: Google-Extended
User-agent: Google-GeminiNotebook
User-agent: Google-InspectionTool
User-agent: Google-NotebookLM
User-agent: Google-Pinpoint
User-agent: Google-Read-Aloud
User-agent: Google-Site-Verification
User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: Googlebot-News
User-agent: Googlebot-Video
User-agent: GoogleMessages
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: GoogleProducer
User-agent: Mediapartners-Google
User-agent: Storebot-Google
User-agent: AdIdxBot
User-agent: Bingbot
User-agent: BingPreview
User-agent: MicrosoftPreview
User-agent: msnbot
User-agent: ChatGPT-User
User-agent: GPTBot
User-agent: OAI-AdsBot
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: ClaudeBot
User-agent: Perplexity-User
User-agent: PerplexityBot
User-agent: GrokAgent
User-agent: GrokBot
User-agent: xAI-Grok
User-agent: xAI-SearchBot
User-agent: AI2Bot
User-agent: AI2Bot-DeepResearchEval
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: Applebot
User-agent: Applebot-Extended
User-agent: Bravebot
User-agent: CCBot
User-agent: cohere-ai
User-agent: cohere-training-data-crawler
User-agent: Coherebot
User-agent: Diffbot
User-agent: Diffbot-User
User-agent: DuckAssistBot
User-agent: DuckDuckBot
User-agent: FacebookBot
User-agent: facebookcatalog
User-agent: facebookexternalhit
User-agent: kagi-fetcher
User-agent: Kagibot
User-agent: meta-externalads
User-agent: meta-externalagent
User-agent: Meta-ExternalFetcher
User-agent: meta-webindexer
User-agent: MistralAI-Index
User-agent: MistralAI-Training
User-agent: MistralAI-User
User-agent: MojeekBot
User-agent: PetalBot
User-agent: Qwantbot
User-agent: Qwantify
User-agent: SeznamBot
User-agent: Slurp
User-agent: Yeti
User-agent: YouBot
User-agent: AhrefsBot
User-agent: AhrefsSiteAudit
User-agent: DataForSeoBot
User-agent: DotBot
User-agent: Mangools
User-agent: MJ12bot
User-agent: rogerbot
User-agent: RSiteAuditor
User-agent: Screaming Frog SEO Spider
User-agent: SemrushBot
User-agent: SemrushBot-BA
User-agent: SemrushBot-BM
User-agent: SemrushBot-COUB
User-agent: SemrushBot-CT
User-agent: SemrushBot-ESI
User-agent: SemrushBot-FT
User-agent: SemrushBot-OCOB
User-agent: SemrushBot-SA
User-agent: SemrushBot-SI
User-agent: SemrushBot-SWA
User-agent: SEOkicks-Robot
User-agent: SEOptimer
User-agent: SERankingBacklinksBot
User-agent: serpstatbot
User-agent: similarweb
User-agent: SimilarwebBot
User-agent: SISTRIX
User-agent: SiteAuditBot
User-agent: Sitebulb
User-agent: SplitSignalBot
User-agent: Ubersuggest
User-agent: archive.org_bot
User-agent: BetterUptimeBot
User-agent: Chrome-Lighthouse
User-agent: Discordbot
User-agent: Embedly
User-agent: Facebot
User-agent: Feedbin
User-agent: Feedly
User-agent: FeedlyBot
User-agent: GTmetrix
User-agent: Iframely
User-agent: Inoreader
User-agent: Lighthouse
User-agent: LinkedInBot
User-agent: Mastodon
User-agent: NewsBlur
User-agent: Pingdom
User-agent: Pinterestbot
User-agent: redditbot
User-agent: SEBot-WA
User-agent: SkypeUriPreview
User-agent: Slackbot
User-agent: Slackbot-LinkExpanding
User-agent: Snapchat
User-agent: StatusCake
User-agent: TelegramBot
User-agent: Tumblr
User-agent: Twitterbot
User-agent: UptimeRobot
User-agent: W3C-checklink
User-agent: W3C_Validator
User-agent: WAVE
User-agent: WhatsApp
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-login.php
Disallow: /?s=
Disallow: /search/
Crawl-delay: 2

User-agent: *
Disallow: /

Sitemap: https://aitoolsforeverybody.com/sitemap_index.xml