mirror of
https://github.com/Ponce/slackbuilds
synced 2024-11-28 10:02:43 +01:00
9 lines
488 B
Text
9 lines
488 B
Text
|
This module parses /robots.txt files as specified in "A Standard for
|
||
|
|
Robot Exclusion", at <http://www.robotstxt.org/wc/norobots.html>
|
||
|
|
Webmasters can use the /robots.txt file to forbid conforming robots
|
||
|
|
from accessing parts of their web site.
|
||
|
|
The parsed files are kept in a WWW::RobotRules object, and this
|
||
|
|
object provides methods to check if access to a given URL is
|
||
|
|
prohibited. The same WWW::RobotRules object can be used for one
|
||
|
|
or more parsed /robots.txt files on any number of hosts.
|