White House site hides Iraq-related pages

As revealed by Atrios, the White House is using its robots.txt file to hide dozens of folders with iraq in their names from Google and other search engines.

An example of the content affected is the White House Special Report Operation Iraqi Freedom. Neither the main page nor any of the other pages in this report are cached by Google.

For those unfamiliar with robots.txt, it's part of the Robots Exclusion Protocol, a way to tell search engines which parts of your site they should never index.

Add a Comment

All comments are moderated before publication. These HTML tags are permitted: <p>, <b>, <i>, <a>, and <blockquote>. This site is protected by reCAPTCHA (for which the Google Privacy Policy and Terms of Service apply).