Robots exclusion standard

Topic | v1 | created by jjones |

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. Robots are often used by search engines to categorize websites. Not all robots cooperate with the standard; email harvesters, spambots, malware and robots that scan for security vulnerabilities may even start with the portions of the website where they have been told to stay out. The standard can be used in conjunction with Sitemaps, a robot inclusion standard for websites.


subtopic of Search engine optimization

Search engine optimization (SEO) is the process of growing the quality and quantity of website traffi...

Edit details Edit relations Attach new author Attach new topic Attach new resource

treated in XML Sitemaps: The Most Misunderstood Tool in the SEO's Toolbox

8.0 rating 5.0 level 9.0 clarity 5.0 background – 1 rating

XML sitemaps are a powerful tool for SEOs, but are often misunderstood and misused. Michael Cottam ex...