Select Page

Ignoring Robots

by | Apr 7, 2005 | Blog, Entrepreneur

I’ve recently been playing around with a perl script which sits in the /cgi-bin/ of a site. The idea is that the script is to catch robots that are not obeying the robots.txt files found in the domains root. The robot.txt is a useful file (if obeyed) in that...