Sitemap Autodiscovery in robots.txt
Posted by Hendry Lee on 04/16/07 in Google, Internal Structure, MSN, Yahoo
If you create sitemap files to get search engines better crawl your sites, then I have a good news for you. Now there is a features called “Sitemap Autodiscovery.”
What is it and what it does?
Sitemap autodiscovery, as the name implies, is the feature for search engine crawlers to find the URL to the sitemap for the crawled site. The implementation is actually in the robots.txt file, usually to control access from search engine spiders to site files and directories, among other things.
Here is how you use it:
Sitemap: <location_of_the_sitemap>
Currently, not all search engine robots support this directive or consume sitemaps yet, but soon all will benefit from this. It is a good idea if you use sitemap, to add a new directive to your robots.txt.
For now though, keep doing the old methods of submitting sitemap files:
- By using search engine’s submission interface.
- By pinging search engines through HTTP request.
Sitemaps.org is a site for the Sitemap protocol. The latest version, Sitemap 0.90 is offered under the terms of the Attribution-ShareAlinke Creative Common License and has wide adoption, including support form Google, Yahoo! and Microsoft.

Post a Comment