# Learning Market robots.txt User-agent: * Allow: / Crawl-delay: 30 Request-rate: 1/10