# robots.txt for Disallow all Crawler User-agent: * Disallow: / Allow: /ads.txt Allow: /robots.txt