dc_crawler.Fetcher.ContentFetcher Class Reference

Inheritance diagram for dc_crawler.Fetcher.ContentFetcher:

Collaboration diagram for dc_crawler.Fetcher.ContentFetcher:

Public Member Functions
def	open (self, url, kwargs)
Public Member Functions inherited from dc_crawler.Fetcher.BaseFetcher
def	__init__ (self)
def	open (self, url, method='get', headers=None, timeout=100, allow_redirects=True, proxies=None, auth=None, data=None, log=None, allowed_content_types=None, max_resource_size=None, max_redirects=CONSTS.MAX_HTTP_REDIRECTS_LIMIT, filters=None, executable_path=None, depth=None, macro=None)
def	should_have_meta_res (self)
def	getDomainNameFromURL (self, url, default='')

Additional Inherited Members
Static Public Member Functions inherited from dc_crawler.Fetcher.BaseFetcher
def	init (dbWrapper=None, siteId=None)
def	get_fetcher (typ, dbWrapper=None, siteId=None)
Public Attributes inherited from dc_crawler.Fetcher.BaseFetcher
	connectionTimeout
	logger
Static Public Attributes inherited from dc_crawler.Fetcher.BaseFetcher
	fetchers = None
int	TYP_NORMAL = 1
int	TYP_DYNAMIC = 2
int	TYP_URLLIB = 5
int	TYP_CONTENT = 6
int	TYP_AUTO = 7
float	CONNECTION_TIMEOUT = 1.0

Detailed Description

Definition at line 1574 of file Fetcher.py.

Member Function Documentation

open()

def dc_crawler.Fetcher.ContentFetcher.open	(		self,
			url,
			kwargs
	)

Definition at line 1586 of file Fetcher.py.

  def open(self, url, **kwargs):
    try:
      localBuf = base64.b64decode(kwargs["inputContent"])
    except TypeError:
      localBuf = kwargs["inputContent"]
    res = Response()
    res.content_size = len(localBuf)
    res.headers = {}
    res.redirects = []
    res.status_code = 200
    res.url = url
    res.encoding = SimpleCharsetDetector().detect(localBuf)
    if res.encoding is None:
      res.encoding = "utf-8"
    res.unicode_content = localBuf
    res.str_content = localBuf
    res.rendered_unicode_content = localBuf

    return res



# #The Response class
# represents an web page response

---

The documentation for this class was generated from the following file:

• sources/hce/dc_crawler/Fetcher.py