# robots.txt for barrenjoey.com # Owner: robots # Last reviewed: 2026-10 # robots.txt is a request to well-behaved crawlers. It is not a security control. # --------------------------------------------------------------- # 1. AI training crawlers: blocked # --------------------------------------------------------------- User-agent: Amazonbot User-agent: anthropic-ai User-agent: Applebot-Extended User-agent: Bytespider User-agent: CCBot User-agent: Claude-Web User-agent: ClaudeBot User-agent: cohere-ai User-agent: cohere-training-data-crawler User-agent: Cotoyogi User-agent: Diffbot User-agent: FacebookBot User-agent: FishBot User-agent: Google-Extended User-agent: GoogleOther User-agent: GPTBot User-agent: ICC-Crawler User-agent: imageSpider User-agent: KangarooBot User-agent: KimiBot User-agent: LCC User-agent: Meta-ExternalAgent User-agent: MistralAI-Training User-agent: NavuBot User-agent: PanguBot User-agent: SBIntuitionsBot User-agent: TikTokSpider User-agent: Timpibot Disallow: / # --------------------------------------------------------------- # 2. SEO and scraper crawlers with no benefit to Barrenjoey # --------------------------------------------------------------- User-agent: AhrefsBot User-agent: Barkrowler User-agent: BLEXBot User-agent: DataForSeoBot User-agent: DotBot User-agent: MJ12bot User-agent: PetalBot User-agent: serpstatbot Disallow: / # --- AI fetch bots without verifiable IPs - DISALLOWED (name-only matching is spoofable) --- User-agent: alphalens-bot User-agent: amazon-QBusiness User-agent: Amzn-User User-agent: AndiBot User-agent: AzureAI-SearchBot User-agent: Bravebot User-agent: Brightbot User-agent: CledaraBot User-agent: Cloudflare-AI-Search User-agent: Cloudflare-AI-Search-External User-agent: EasyScan User-agent: Element451Bot User-agent: HarkBot User-agent: HIFIBot User-agent: Instapaper User-agent: KernelSearchBot User-agent: LinerBot User-agent: meta-externalfetcher User-agent: meta-webindexer User-agent: Nava User-agent: PhindBot User-agent: QATechBot User-agent: ShapBot User-agent: TongyiBot User-agent: TwinAgent User-agent: YiyanBot User-agent: YouBot Disallow: / # --- Automation / rent-by-the-hour scraping platforms - DISALLOWED --- # Apify/Firecrawl-style services let any paying account masquerade; not AI search. --- User-agent: ApifyBot User-agent: ApifyWebsiteContentCrawler User-agent: Devin User-agent: ExaBot User-agent: FirecrawlAgent User-agent: Kimi-User User-agent: MakeBot User-agent: Retool Disallow: / # --- Third-party data brokers / resellers - DISALLOWED --- User-agent: AdpResearchBot User-agent: amazon-kendra User-agent: atlassian-bot User-agent: BorderxBot User-agent: CitibotSiteCrawler User-agent: CloudflareBrowserRenderingCrawler User-agent: Novellum User-agent: omgili User-agent: SelectikaScraper User-agent: WARDBot User-agent: webzio-extended User-agent: ygs-scraper-bot Disallow: / User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=no Crawl-delay: 10 Disallow: /admin/ Disallow: /lucee/ Disallow: /config/ Sitemap: https://barrenjoey.com/sites/bstar/sitemap.xml