User-agent: * Crawl-delay: 1 Disallow: /cgi-bin/ Disallow: /bin/ Disallow: /java/ Disallow: /capat/ Disallow: /functions/ Disallow: /user/ Disallow: /tests/ User-agent: MLBot Disallow: /