User-agent: * Request-rate: 1/10 # maximum rate is one page every 10 seconds Crawl-delay: 10 # 10 seconds between page requests