# robots.txt for writecraft.io # https://www.robotstxt.org/ # ============================================================ # Search engines (SEO) # ============================================================ User-agent: Googlebot Allow: / Disallow: /api/ User-agent: Bingbot Allow: / Disallow: /api/ User-agent: DuckDuckBot Allow: / Disallow: /api/ User-agent: Applebot Allow: / Disallow: /api/ User-agent: Yahoo! Slurp Allow: / Disallow: /api/ # ============================================================ # Social media & link preview bots (for shared post previews) # ============================================================ User-agent: facebookexternalhit Allow: / Disallow: /api/ User-agent: Twitterbot Allow: / Disallow: /api/ User-agent: TelegramBot Allow: / Disallow: /api/ User-agent: Slackbot Allow: / Disallow: /api/ User-agent: Discordbot Allow: / Disallow: /api/ User-agent: Line Allow: / Disallow: /api/ # ============================================================ # Block AI training crawlers # ============================================================ User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / # ============================================================ # Block all other bots # ============================================================ User-agent: * Disallow: /