SEO Tips on How to Properly Implement Robots.txt File

Author: audreyabbey | Posted: 24.06.2012

Designing your site to be search engine-friendly from the bottom up, making sure that your site’s content is crawled and indexed correctly is one purpose of SEO and one great way to achieve this is to properly implement a powerful file called “robots.txt” as this is considered as one of the important keys for your site to rank well in search results pages.

What is a Robots.txt file?

The robots.txt file, also known as Robot Exclusion Standard or the Robots Exclusion Protocol, is a simple text file that professional SEOs or webmasters create in your web site that will tell search engine robots how to crawl and index pages on your website.

By default, search engine bots crawl everything possible unless they are forbidden from doing so. When these search engine bots visits a site, they always crawl first the robots.txt file located in the root directory of the site before crawling the entire web site. The robots.txt file is then parsed or analyzed, and will instruct the robot as to which pages, files or directories are not to be crawled.

Robots.txt Dos and Don'ts

Using robots.txt in stopping the search engines from indexing certain files or directories on a website and allowing others for SEO purposes are done for many different good reasons.

Here's the do’s for robots.txt:

• Check all of the directories in your website. In all likelihood, there are directories in your site that you do not want the search engines to index. This may include directories like /cgi-bin/, /wp-admin/, /cart/, /scripts/, and others that might include sensitive data.

• Stop certain directories of your site that might include duplicate content from being indexed by the search engines. For example, there are some websites that have "print versions" of web pages and articles that allow visitors to print them easily. Make sure that you only allow one version of your content to be indexed by the search engines.

• Make sure that all barriers are removed that will stop the search engines from indexing the main content of your website.

• Check for certain files on your site that you do not want the search engines to index such as certain scripts, or files that might contain email addresses, phone numbers, or other sensitive data.

Here's the don’t for robots.txt:

• Avoid making use of comments in your robots.txt file.

• Do not make a list of all your files in the robots.txt file.

• By listing the files, this would allow people to find files that you don't want them to find.

• The "/allow" command is not part of the robots.txt file, so there's no reason for it to be added to the robots.txt file.



Article was printed from http://www.articleside.com/seo-articles/seo-tips-on-how-to-properly-implement-robots-txt-file.htm
Click here to return original format