dc_crawler.OwnRobots._Ruleset Class Reference

Inheritance diagram for dc_crawler.OwnRobots._Ruleset:

Collaboration diagram for dc_crawler.OwnRobots._Ruleset:

Public Member Functions
def	__init__ (self)
def	__str__ (self)
def	__unicode__ (self)
def	add_robot_name (self, bot)
def	add_allow_rule (self, path)
def	add_disallow_rule (self, path)
def	is_not_empty (self)
def	is_default (self)
def	does_user_agent_match (self, user_agent)
def	is_url_allowed (self, url, syntax=GYM2008)

Public Attributes
	robot_names
	rules
	crawl_delay

Static Public Attributes
int	ALLOW = 1
int	DISALLOW = 2

Detailed Description

_Ruleset represents a set of allow/disallow rules (and possibly a 
crawl delay) that apply to a set of user agents.
  
Users of this module don't need this class. It's available at the module
level only because RobotExclusionRulesParser() instances can't be 
pickled if _Ruleset isn't visible a the module level.    

Definition at line 187 of file OwnRobots.py.

Constructor & Destructor Documentation

__init__()

def dc_crawler.OwnRobots._Ruleset.__init__

(

self

)

Definition at line 198 of file OwnRobots.py.

  def __init__(self):
    self.robot_names = [ ]
    self.rules = [ ]
    self.crawl_delay = None

Member Function Documentation

__str__()

def dc_crawler.OwnRobots._Ruleset.__str__

(

self

)

Definition at line 203 of file OwnRobots.py.

  def __str__(self):
    s = self.__unicode__()
    if PY_MAJOR_VERSION == 2:
      s = s.encode("utf-8")

    return s

Here is the call graph for this function:

__unicode__()

def dc_crawler.OwnRobots._Ruleset.__unicode__

(

self

)

Definition at line 210 of file OwnRobots.py.

  def __unicode__(self):
    d = { self.ALLOW : "Allow", self.DISALLOW : "Disallow" }

    s = ''.join( ["User-agent: %s\n" % name for name in self.robot_names] )

    if self.crawl_delay:
      s += "Crawl-delay: %s\n" % self.crawl_delay
  
    s += ''.join( ["%s: %s\n" % (d[rule_type], path) for rule_type, path in self.rules] )
  
    return s

Here is the caller graph for this function:

add_allow_rule()

def dc_crawler.OwnRobots._Ruleset.add_allow_rule	(		self,
			path
	)

Definition at line 225 of file OwnRobots.py.

  def add_allow_rule(self, path):
    self.rules.append((self.ALLOW, _unquote_path(path)))
  

Here is the call graph for this function:

add_disallow_rule()

def dc_crawler.OwnRobots._Ruleset.add_disallow_rule	(		self,
			path
	)

Definition at line 228 of file OwnRobots.py.

  def add_disallow_rule(self, path):
    self.rules.append((self.DISALLOW, _unquote_path(path)))
  

Here is the call graph for this function:

add_robot_name()

def dc_crawler.OwnRobots._Ruleset.add_robot_name	(		self,
			bot
	)

Definition at line 222 of file OwnRobots.py.

  def add_robot_name(self, bot):
    self.robot_names.append(bot)
  

does_user_agent_match()

def dc_crawler.OwnRobots._Ruleset.does_user_agent_match	(		self,
			user_agent
	)

Definition at line 237 of file OwnRobots.py.

  def does_user_agent_match(self, user_agent):
    match = False
  
    for robot_name in self.robot_names:
      # MK1994 says, "A case insensitive substring match of the name 
      # without version information is recommended." MK1996 3.2.1 
      # states it even more strongly: "The robot must obey the first
      # record in /robots.txt that contains a User-Agent line whose 
      # value contains the name token of the robot as a substring. 
      # The name comparisons are case-insensitive."
      match = match or (robot_name == '*') or (robot_name.lower() in user_agent.lower())
              
    return match

is_default()

def dc_crawler.OwnRobots._Ruleset.is_default

(

self

)

Definition at line 234 of file OwnRobots.py.

  def is_default(self):
    return bool('*' in self.robot_names)

is_not_empty()

def dc_crawler.OwnRobots._Ruleset.is_not_empty

(

self

)

Definition at line 231 of file OwnRobots.py.

  def is_not_empty(self):
    return bool(len(self.rules)) and bool(len(self.robot_names))

is_url_allowed()

def dc_crawler.OwnRobots._Ruleset.is_url_allowed	(		self,
			url,
			syntax = GYM2008
	)

Definition at line 251 of file OwnRobots.py.

  def is_url_allowed(self, url, syntax=GYM2008):
    allowed = True

    # Schemes and host names are not part of the robots.txt protocol, 
    # so  I ignore them. It is the caller's responsibility to make 
    # sure they match.
    _, _, path, parameters, query, fragment = urllib_urlparse(url)
    url = urllib_urlunparse(("", "", path, parameters, query, fragment))

    url = _unquote_path(url)

    done = False
    i = 0
    while not done:
      rule_type, path = self.rules[i]

      if (syntax == GYM2008) and ("*" in path or path.endswith("$")):
        # GYM2008-specific syntax applies here
        # http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40360
        if path.endswith("$"):
          appendix = "$"
          path = path[:-1]
        else:
          appendix = ""
        parts = path.split("*")
        pattern = "%s%s" % (".*".join([re.escape(p) for p in parts]), appendix)
        if re.match(pattern, url):
          # Ding!
          done = True
          allowed = (rule_type == self.ALLOW)
      else:  
        # Wildcards are either not present or are taken literally.
        if url.startswith(path):
          # Ding!
          done = True
          allowed = (rule_type == self.ALLOW)
          # A blank path means "nothing", so that effectively 
          # negates the value above. 
          # e.g. "Disallow:   " means allow everything
          if not path:
            allowed = not allowed

      i += 1
      if i == len(self.rules):
        done = True
        
    return allowed

Here is the call graph for this function:

Member Data Documentation

ALLOW

int dc_crawler.OwnRobots._Ruleset.ALLOW = 1

static

Definition at line 195 of file OwnRobots.py.

crawl_delay

dc_crawler.OwnRobots._Ruleset.crawl_delay

Definition at line 201 of file OwnRobots.py.

DISALLOW

int dc_crawler.OwnRobots._Ruleset.DISALLOW = 2

static

Definition at line 196 of file OwnRobots.py.

robot_names

dc_crawler.OwnRobots._Ruleset.robot_names

Definition at line 199 of file OwnRobots.py.

rules

dc_crawler.OwnRobots._Ruleset.rules

Definition at line 200 of file OwnRobots.py.

---

The documentation for this class was generated from the following file:

• sources/hce/dc_crawler/OwnRobots.py