Synonyms of Robots.txt
Primary definition: A text file placed on a website to instruct web crawlers and robots about which pages to crawl or not to crawl.
Primary definition: A text file placed on a website to instruct web crawlers and robots about which pages to crawl or not to crawl.