mirror of
https://github.com/Ponce/slackbuilds
synced 2024-10-13 08:43:42 +02:00
11 lines
520 B
Text
11 lines
520 B
Text
|
This module parses /robots.txt files as specified in "A Standard for
|
||
|
Robot Exclusion", at <http://www.robotstxt.org/wc/norobots.html>
|
||
|
Webmasters can use the /robots.txt file to forbid conforming robots
|
||
|
from accessing parts of their web site.
|
||
|
The parsed files are kept in a WWW::RobotRules object, and this
|
||
|
object provides methods to check if access to a given URL is
|
||
|
prohibited. The same WWW::RobotRules object can be used for one
|
||
|
or more parsed /robots.txt files on any number of hosts.
|
||
|
|
||
|
This requires perl-uri-escape.
|