White House site hides Iraq-related pages
As revealed by Atrios
, the White House is using its robots.txt
file to hide dozens of folders with iraq
in their names from Google and other search engines.
An example of the content affected is the White House Special Report Operation Iraqi Freedom. Neither the main page nor any of the other pages in this report are cached by Google.
For those unfamiliar with robots.txt, it's part of the Robots Exclusion Protocol, a way to tell search engines which parts of your site they should never index.
Add a Comment