dc_crawler.Fetcher.BaseFetcher Class Reference

Inheritance diagram for dc_crawler.Fetcher.BaseFetcher:

Collaboration diagram for dc_crawler.Fetcher.BaseFetcher:

Public Member Functions
def	__init__ (self)
def	open (self, url, method='get', headers=None, timeout=100, allow_redirects=True, proxies=None, auth=None, data=None, log=None, allowed_content_types=None, max_resource_size=None, max_redirects=CONSTS.MAX_HTTP_REDIRECTS_LIMIT, filters=None, executable_path=None, depth=None, macro=None)
def	should_have_meta_res (self)
def	getDomainNameFromURL (self, url, default='')

Static Public Member Functions
def	init (dbWrapper=None, siteId=None)
def	get_fetcher (typ, dbWrapper=None, siteId=None)

Public Attributes
	connectionTimeout
	logger

Static Public Attributes
	fetchers = None
int	TYP_NORMAL = 1
int	TYP_DYNAMIC = 2
int	TYP_URLLIB = 5
int	TYP_CONTENT = 6
int	TYP_AUTO = 7
float	CONNECTION_TIMEOUT = 1.0

Detailed Description

Definition at line 50 of file Fetcher.py.

Constructor & Destructor Documentation

__init__()

def dc_crawler.Fetcher.BaseFetcher.__init__

(

self

)

Definition at line 65 of file Fetcher.py.

  def __init__(self):
    self.connectionTimeout = self.CONNECTION_TIMEOUT
    self.logger = None

Member Function Documentation

get_fetcher()

def dc_crawler.Fetcher.BaseFetcher.get_fetcher (   typ,   dbWrapper = None,   siteId = None  )

static

Definition at line 121 of file Fetcher.py.

  def get_fetcher(typ, dbWrapper=None, siteId=None):
    if not BaseFetcher.fetchers:
      BaseFetcher.init(dbWrapper, siteId)
    if typ in BaseFetcher.fetchers:
      return BaseFetcher.fetchers[typ]
    else:
      raise BaseException("unsupported fetch type:%s" % (typ,))

getDomainNameFromURL()

def dc_crawler.Fetcher.BaseFetcher.getDomainNameFromURL	(		self,
			url,
			default = ''
	)

Definition at line 142 of file Fetcher.py.

  def getDomainNameFromURL(self, url, default=''):
    ret = default

    urlParts = urlsplit(url)
    if len(urlParts) > 1:
      ret = urlParts[1]

    return ret


# # Check redirects hook
#
#

Here is the caller graph for this function:

init()

def dc_crawler.Fetcher.BaseFetcher.init (   dbWrapper = None,   siteId = None  )

static

Definition at line 71 of file Fetcher.py.

  def init(dbWrapper=None, siteId=None):
    # enumerate content_types we don't want to fetch
    BaseFetcher.prohibited_conten_types = ["audio/mpeg", "application/pdf"]

    BaseFetcher.fetchers = {
        BaseFetcher.TYP_NORMAL : RequestsFetcher(dbWrapper, siteId),
        BaseFetcher.TYP_DYNAMIC: SeleniumFetcher(),
        BaseFetcher.TYP_URLLIB: URLLibFetcher(),
        BaseFetcher.TYP_CONTENT: ContentFetcher()
    }

Here is the call graph for this function:

Here is the caller graph for this function:

open()

def dc_crawler.Fetcher.BaseFetcher.open	(		self,
			url,
			method = 'get',
			headers = None,
			timeout = 100,
			allow_redirects = True,
			proxies = None,
			auth = None,
			data = None,
			log = None,
			allowed_content_types = None,
			max_resource_size = None,
			max_redirects = CONSTS.MAX_HTTP_REDIRECTS_LIMIT,
			filters = None,
			executable_path = None,
			depth = None,
			macro = None
	)

Definition at line 109 of file Fetcher.py.

           macro=None):
    if headers is None:
      headers = {}
    del url, method, headers, timeout, allow_redirects, proxies, auth, data, log, allowed_content_types, \
        max_resource_size, max_redirects, filters, executable_path, depth, macro

Here is the caller graph for this function:

should_have_meta_res()

def dc_crawler.Fetcher.BaseFetcher.should_have_meta_res

(

self

)

Definition at line 133 of file Fetcher.py.

  def should_have_meta_res(self):

    return False

Member Data Documentation

CONNECTION_TIMEOUT

float dc_crawler.Fetcher.BaseFetcher.CONNECTION_TIMEOUT = 1.0

static

Definition at line 60 of file Fetcher.py.

connectionTimeout

dc_crawler.Fetcher.BaseFetcher.connectionTimeout

Definition at line 66 of file Fetcher.py.

fetchers

dc_crawler.Fetcher.BaseFetcher.fetchers = None

static

Definition at line 52 of file Fetcher.py.

logger

dc_crawler.Fetcher.BaseFetcher.logger

Definition at line 67 of file Fetcher.py.

TYP_AUTO

int dc_crawler.Fetcher.BaseFetcher.TYP_AUTO = 7

static

Definition at line 58 of file Fetcher.py.

TYP_CONTENT

int dc_crawler.Fetcher.BaseFetcher.TYP_CONTENT = 6

static

Definition at line 57 of file Fetcher.py.

TYP_DYNAMIC

int dc_crawler.Fetcher.BaseFetcher.TYP_DYNAMIC = 2

static

Definition at line 55 of file Fetcher.py.

TYP_NORMAL

int dc_crawler.Fetcher.BaseFetcher.TYP_NORMAL = 1

static

Definition at line 54 of file Fetcher.py.

TYP_URLLIB

int dc_crawler.Fetcher.BaseFetcher.TYP_URLLIB = 5

static

Definition at line 56 of file Fetcher.py.

---

The documentation for this class was generated from the following file:

• sources/hce/dc_crawler/Fetcher.py