Robots.txt > Free SEO Tools

Robots.txt

Tools for creating and analyzing robots.txt files to make sure your robots text file is working

Robots.txt - Categories

Articles

Search engines update - sitemaps auto submission

he top 4 search engines now support Sitemaps auto submission In November 2006, Google, Yahoo and Microsoft agreed to support the Sitemap protocol. The Sitemap protocol is a standard that can help you to submit all of your web pages to search engines. Last week, Ask.com joined the team so that now all four major search engines support the Sitemap protocol. In addition, search engines can now automatically detect your Sitemaps file.

Robots.txt - Google News

Marchex Gets UDRP Win with Clara.com, Robot Blocking Justified - Domain Name Wire (blog)
Marchex Gets UDRP Win with Clara.com, Robot Blocking JustifiedDomain Name Wire (blog)Company defends generic domain name and explains legitimate reasons for blocking robots.txt from parked domain names. Marchex subsidiary MDNH has ...

Rupert Murdoch ready to sue Google? - CNET
Rupert Murdoch ready to sue Google?CNET"OK, you guys know that you can just add a robots.txt file to get us to control where we index, and stop completely? And you chose not to do so? ...and more »

Solving Website Indexing Problems - Search Engine Land (blog)
Solving Website Indexing ProblemsSearch Engine Land (blog)Robots.txt file excludes that page/folder by mistake. For whatever reason, you may haveexcluded a folder or section of the site during design or testing, ...and more »

Google Webmaster Tools: Cutting Through the Mystery Part 1 - Drop Ship (press release) (blog)
Google Webmaster Tools: Cutting Through the Mystery Part 1Drop Ship (press release) (blog)This is your robots.txt file command center. Among other things, you can use this area as a sort of testing ground for tweaking your robots.txt commands and ...and more »

Five common SEO mistakes made by e-commerce sites - Econsultancy (blog)
Five common SEO mistakes made by e-commerce sitesEconsultancy (blog)From time to time we come across misuse of the robots.txt which wrongly blocks important parts of the site and lately misuse of the rel=canonical tag is ...and more »

feed

Robots.txt