mirror of
https://github.com/Ponce/slackbuilds
synced 2024-10-03 07:54:34 +02:00
10 lines
520 B
Text
10 lines
520 B
Text
This module parses /robots.txt files as specified in "A Standard for
|
|
Robot Exclusion", at <http://www.robotstxt.org/wc/norobots.html>
|
|
Webmasters can use the /robots.txt file to forbid conforming robots
|
|
from accessing parts of their web site.
|
|
The parsed files are kept in a WWW::RobotRules object, and this
|
|
object provides methods to check if access to a given URL is
|
|
prohibited. The same WWW::RobotRules object can be used for one
|
|
or more parsed /robots.txt files on any number of hosts.
|
|
|
|
This requires perl-uri-escape.
|