dc_crawler.Fetcher.URLLibFetcher Class Reference

Inheritance diagram for dc_crawler.Fetcher.URLLibFetcher:

Collaboration diagram for dc_crawler.Fetcher.URLLibFetcher:

Public Member Functions
def	open (self, url, kwargs)
Public Member Functions inherited from dc_crawler.Fetcher.BaseFetcher
def	__init__ (self)
def	open (self, url, method='get', headers=None, timeout=100, allow_redirects=True, proxies=None, auth=None, data=None, log=None, allowed_content_types=None, max_resource_size=None, max_redirects=CONSTS.MAX_HTTP_REDIRECTS_LIMIT, filters=None, executable_path=None, depth=None, macro=None)
def	should_have_meta_res (self)
def	getDomainNameFromURL (self, url, default='')

Additional Inherited Members
Static Public Member Functions inherited from dc_crawler.Fetcher.BaseFetcher
def	init (dbWrapper=None, siteId=None)
def	get_fetcher (typ, dbWrapper=None, siteId=None)
Public Attributes inherited from dc_crawler.Fetcher.BaseFetcher
	connectionTimeout
	logger
Static Public Attributes inherited from dc_crawler.Fetcher.BaseFetcher
	fetchers = None
int	TYP_NORMAL = 1
int	TYP_DYNAMIC = 2
int	TYP_URLLIB = 5
int	TYP_CONTENT = 6
int	TYP_AUTO = 7
float	CONNECTION_TIMEOUT = 1.0

Detailed Description

Definition at line 1511 of file Fetcher.py.

Member Function Documentation

open()

def dc_crawler.Fetcher.URLLibFetcher.open	(		self,
			url,
			kwargs
	)

Definition at line 1523 of file Fetcher.py.

  def open(self, url, **kwargs):
    import urllib2

    if 'logger' in kwargs['logger']:
      log = kwargs['logger']
    else:
      log = logger
    allowed_content_types = kwargs['allowed_content_types']
    # max_resource_size = kwargs["max_resource_size"]

    res = Response()
    log.debug("url: <%s>", url)
    response = None
    try:
      response = urllib2.urlopen(url)
      headers_info = response.info()
      if headers_info is not None:
        if headers_info.type in allowed_content_types:
          if response is not None:
            # res.encoding = impl_res.encoding
            # res.cookies = requests.utils.dict_from_cookiejar(impl_res.cookies)
            res.url = response.geturl()
            res.status_code = response.getcode()
            content_response = response.read()
            res.unicode_content = content_response
            res.str_content = content_response
            res.rendered_unicode_content = content_response
            res.content_size = len(content_response)
            headers = {}
            headers["content-length"] = res.content_size
            headers["content-type"] = headers_info.type
            res.headers = headers
            history = []
            res.redirects = history
          else:
            log.debug("URLLib return empty response.")
        else:
          log.debug("Content-Type not allowed. headers_info.type: %s", str(headers_info.type))
      else:
        log.debug("URLLib info is empty.")
    except urllib2.HTTPError, err:
    # except Exception, err:
      log.debug("Exception <%s>", str(err.code))

    return res



# # external Fetcher
#
#

---

The documentation for this class was generated from the following file:

• sources/hce/dc_crawler/Fetcher.py