# Allow all crawlers User-agent: * Allow: / Disallow: /generate/ Disallow: /69a108540f48a5139ea77608640a40ae.html