an alphabetical list of words relating to a specific subject, text, or dialect.
A text file placed in the root directory of your site to share important directions with search engine crawlers. If available, it is the first page read by a search engine while visiting a site.
Use it to indicate parts of your site you don’t want accessed by search engine crawlers, the location of your sitemap files and also to block particular crawlers for visiting your site complete.
If you block some pages inside robots.txt, they will gradually be removed from search engines.