Daily Archives: April 17, 2017

Robots.txt meant for search engines don’t work well for web archives

Robots.txt files were invented 20+ years ago to help advise “robots,” mostly search engine web crawlers, which sections of a web site should be crawled and indexed for search. Many sites use their robots.txt files to improve their SEO (search … Continue reading

Posted in Announcements, News | 34 Comments