Search Engine Facts
Search Engine Facts

Read our back issues

May 2017

December 2009

November 2009

October 2009

September 2009

August 2009

July 2009

June 2009

May 2009

April 2009

March 2009

February 2009

January 2009

December 2008

November 2008

October 2008

September 2008

August 2008

July 2008

June 2008

May 2008

April 2008

March 2008

February 2008

January 2008

December 2007

November 2007

October 2007

September 2007

August 2007

July 2007

June 2007

May 2007

April 2007

March 2007

February 2007

January 2007

December 2006

December 2006

November 2006

October 2006

September 2006

August 2006

July 2006

June 2006

May 2006

April 2006

March 2006

February 2006

Januray 2006

December 2005

November 2005

October 2005

September 2005

August 2005

July 2005

June 2005

May 2005

August 2005

March 2005

February 2005

January 2005

December 2004

November 2004

October 2004

September 2004

August 2004

July 2004


Home   Contact   Privacy policy    Partner sites

Google, Yahoo, the X-Robots directive and your website rankings

Last week, Yahoo announced that they now support the X-Robots-Tag in the HTTP header. This new tag allows you to influence how Google and Yahoo index your website pages.

What is the X-Robots-Tag?

Google introduced the new X-Robots-Tag directive in 2007 to allow webmasters to control access to non-web page documents, such as Adobe PDF files, video and audio files.

The X-Robots-Tag is included in the HTTP header of a document. The HTTP header is the initial reply of a server to a query. It contains information about the document that follows, including content type, creation date, character set, encodings, etc.

The new X-Robots-Tag allows webmasters to add information about search engine indexing to the HTTP header.

Which commands are supported by the X-Robots-Tag?

At this time, the X-Robots-Tag supports the following commands:

  • X-Robots-Tag: NOINDEX (Use this tag if you don't want to show the URL in Google's search results)

  • X-Robots-Tag: NOARCHIVE (Use this tag if you don't want to see a Cache link in the search result pages for the document)

  • X-Robots-Tag: NOSNIPPET (Use this tag if you don't want to display a summary in the search result pages.)

  • X-Robots-Tag: NOFOLLOW (Use this tag if you don't want Googe and Yahoo to index the links in the page.)

The X-Robots-Tag is currently supported by Google and Yahoo. Other search engines don't support the tag yet.

How to add the X-Robots-Tag to your documents

It depends on your server and on the method you use to create your web pages how you can add additional tags to the HTTP header. If your web server uses Apache, you can use the htaccess file to modify your HTTP headers.

In general, you shouldn't mess with the HTTP header if you're not 100% sure what you're doing. A broken HTTP header can keep search engines away so that no search engine will index your website.

Instead of using the X-Robots-Tag, you can also add meta tags with the same functionality to your web pages:

<meta name="robots" content="noarchive">
<meta name="robots" content="nosnippet">
<meta name="robots" content="nofollow">
<meta name="robots" content="noindex">

You can edit these tags directly in the head part of your web pages. They have the same effect as the corresponding X-Robots-Tags explained above.

Do you need the X-Robots-Tag or the corresponding meta tags?

If you don't want to limit access to your documents of if you only have normal web pages on your website then you don't need the X-Robots-Tag or the meta tags.

The X-Robots-Tag has been designed to restrict access to your documents. That means that it does not help to increase search engine rankings.

The new X-Robots-Tag and the meta robots tag allow you to restrict access to your documents. If you want to make sure that search engines index all of your web pages then you have to optimize your web pages.

Search engine robots must be able to easily parse your web pages and they must be able to find the right elements on your web pages. If you optimize the right web page elements with the right keywords, your web page will get high search engine rankings for these keywords.

Copyright - Internet marketing and search engine ranking software

Home   Contact   Privacy policy    Partner sites
December 2007 search engine articles