dc_crawler.Fetcher Namespace Reference

Classes
class	BaseFetcher
class	ContentFetcher
class	RequestsFetcher
class	Response
class	SeleniumFetcher
class	SimpleCharsetDetector
class	URLLibFetcher

Functions
def	checkRedirectsHook (r, args, kwargs)

Variables
	logger = logging.getLogger(APP_CONSTS.LOGGER_NAME)
int	MAX_CONTENT_SIZE_FOR_CHARDET = 5000000

Detailed Description

HCE project, Python bindings, Distributed Tasks Manager application.
web page fetchers.

@package: dc
@file Fetcher.py
@author madk, bgv <developers.hce@gmail.com>
@link: http://hierarchical-cluster-engine.com/
@copyright: Copyright &copy; 2013-201 IOIX Ukraine
@license: http://hierarchical-cluster-engine.com/license/
@since: 0.1

Function Documentation

checkRedirectsHook()

def dc_crawler.Fetcher.checkRedirectsHook	(		r,
			args,
			kwargs
	)

Definition at line 155 of file Fetcher.py.

def checkRedirectsHook(r, *args, **kwargs):
  logger.debug('r.url = ' + str(r.url))
  logger.debug('args = ' + str(args))
  logger.debug('kwargs = ' + str(kwargs))
  logger.debug('type(r): %s,  r = %s', str(type(r)), varDump(r))


# # Fetcher base on the requests module, cann't execute javascript
#
#

Here is the call graph for this function:

Variable Documentation

logger

dc_crawler.Fetcher.logger = logging.getLogger(APP_CONSTS.LOGGER_NAME)

Definition at line 44 of file Fetcher.py.

MAX_CONTENT_SIZE_FOR_CHARDET

int dc_crawler.Fetcher.MAX_CONTENT_SIZE_FOR_CHARDET = 5000000

Definition at line 47 of file Fetcher.py.