mirror of
https://github.com/SlackBuildsOrg/slackbuilds
synced 2024-09-28 15:21:00 +02:00
8 lines
488 B
Text
8 lines
488 B
Text
This module parses /robots.txt files as specified in "A Standard for
|
|
Robot Exclusion", at <http://www.robotstxt.org/wc/norobots.html>
|
|
Webmasters can use the /robots.txt file to forbid conforming robots
|
|
from accessing parts of their web site.
|
|
The parsed files are kept in a WWW::RobotRules object, and this
|
|
object provides methods to check if access to a given URL is
|
|
prohibited. The same WWW::RobotRules object can be used for one
|
|
or more parsed /robots.txt files on any number of hosts.
|