dc_crawler.HTTPRedirectResolver.HTTPRedirectResolver Class Reference

Inheritance diagram for dc_crawler.HTTPRedirectResolver.HTTPRedirectResolver:

Collaboration diagram for dc_crawler.HTTPRedirectResolver.HTTPRedirectResolver:

Classes
class	RedirectProperty

Public Member Functions
def	__init__ (self, propertyString=None, fetchType=BaseFetcher.TYP_NORMAL, dbWrapper=None, siteId=None, connectionTimeout=CONSTS.CONNECTION_TIMEOUT)
def	getRedirectProperty (self, propertyDict)
def	resolveRedirectUrl (self, url, headers, timeout=None, allowRedirects=True, proxies=None, auth=None, postData=None, maxRedirects=RedirectProperty.DEFAULT_VALUE_MAX_REDIRECTS, filters=None)

Static Public Member Functions
def	isAllowedUrl (url, patterns)

Public Attributes
	fetchType
	dbWrapper
	siteId
	connectionTimeout

Static Public Attributes
string	ERROR_INITIALIZATION = "Initialization class '%s' was failed. Error: %s"
string	ERROR_BAD_TYPE_PROPERTY_VALUE = "Wrong type (%s) of property value: %s"
string	ERROR_BAD_TYPE_HEADERS_VALUE = "Wrong type (%s) of headers: %s"
string	ERROR_BAD_PROPERTY_VALUE = "Not support value '%s' for property '%s'"
string	ERROR_BAD_STATUS_CODE_VALUE = "Not allowed status code '%s'. Allowed list: %s"
	redirectProperty = HTTPRedirectResolver.RedirectProperty()
	methodName
	urlPatternList
	maxRedirects
	typesList

Private Member Functions
def	__loadProperty (self, propertyString)
def	__repairHeaders (self, headers)
def	__resolveRedirect (self, url, headers, method, timeout=None, allowRedirects=True, proxies=None, auth=None, postData=None, maxRedirects=RedirectProperty.DEFAULT_VALUE_MAX_REDIRECTS, filters=None, redirectProperty=None)
def	__fetch (self, url, headers, method, timeout=None, allowRedirects=True, proxies=None, auth=None, postData=None, maxRedirects=RedirectProperty.DEFAULT_VALUE_MAX_REDIRECTS, filters=None, fetchType=BaseFetcher.TYP_NORMAL)

Detailed Description

Definition at line 32 of file HTTPRedirectResolver.py.

Constructor & Destructor Documentation

__init__()

def dc_crawler.HTTPRedirectResolver.HTTPRedirectResolver.__init__	(		self,
			propertyString = None,
			fetchType = BaseFetcher.TYP_NORMAL,
			dbWrapper = None,
			siteId = None,
			connectionTimeout = CONSTS.CONNECTION_TIMEOUT
	)

Definition at line 78 of file HTTPRedirectResolver.py.

               connectionTimeout=CONSTS.CONNECTION_TIMEOUT):
    self.redirectProperty = self.__loadProperty(propertyString)
    self.fetchType = fetchType
    self.dbWrapper = dbWrapper
    self.siteId = siteId
    self.connectionTimeout = connectionTimeout

Here is the call graph for this function:

Member Function Documentation

__fetch()

def dc_crawler.HTTPRedirectResolver.HTTPRedirectResolver.__fetch (   self,   url,   headers,   method,   timeout = None,   allowRedirects = True,   proxies = None,   auth = None,   postData = None,   maxRedirects = RedirectProperty.DEFAULT_VALUE_MAX_REDIRECTS,   filters = None,   fetchType = BaseFetcher.TYP_NORMAL  )

private

Definition at line 325 of file HTTPRedirectResolver.py.

                         fetchType=BaseFetcher.TYP_NORMAL):
    # variable for result
    ret = None
    fetcher = BaseFetcher.get_fetcher(fetchType, self.dbWrapper, self.siteId)
    fetcher.connectionTimeout = self.connectionTimeout

    res = fetcher.open(url=url, method=method, headers=headers, timeout=timeout,
                       allow_redirects=allowRedirects, proxies=proxies, auth=auth, data=postData, log=logger,
                       max_redirects=maxRedirects, filters=filters)

    if res.url is not None:
      ret = res.url

    return ret

__loadProperty()

def dc_crawler.HTTPRedirectResolver.HTTPRedirectResolver.__loadProperty (   self,   propertyString  )

private

Definition at line 147 of file HTTPRedirectResolver.py.

  def __loadProperty(self, propertyString):
    # variable for result
    redirectProperty = HTTPRedirectResolver.RedirectProperty()

    if propertyString is not None:
      try:
        if not isinstance(propertyString, basestring) or propertyString == "":
          raise Exception(self.ERROR_BAD_TYPE_PROPERTY_VALUE % (str(type(propertyString)), varDump(propertyString)))

        propertyDict = json.loads(propertyString)
        redirectProperty = self.getRedirectProperty(propertyDict)

      except Exception, err:
        logger.error(self.ERROR_INITIALIZATION, self.__class__.__name__, str(err))
        logger.info(getTracebackInfo())

    return redirectProperty

Here is the call graph for this function:

Here is the caller graph for this function:

__repairHeaders()

def dc_crawler.HTTPRedirectResolver.HTTPRedirectResolver.__repairHeaders (   self,   headers  )

private

Definition at line 195 of file HTTPRedirectResolver.py.

  def __repairHeaders(self, headers):
    if isinstance(headers, dict):
      for key, value in headers.items():
        headers[key] = ';'.join(value.split())

Here is the call graph for this function:

__resolveRedirect()

def dc_crawler.HTTPRedirectResolver.HTTPRedirectResolver.__resolveRedirect (   self,   url,   headers,   method,   timeout = None,   allowRedirects = True,   proxies = None,   auth = None,   postData = None,   maxRedirects = RedirectProperty.DEFAULT_VALUE_MAX_REDIRECTS,   filters = None,   redirectProperty = None  )

private

Definition at line 274 of file HTTPRedirectResolver.py.

                         redirectProperty=None):
    # variable for result
    ret = None
    logger.debug("type(redirectProperty) = %s", str(type(redirectProperty)))
    if isinstance(redirectProperty, HTTPRedirectResolver.RedirectProperty):
      logger.debug("type is GOOD!!!")

      # check is allowed url for processing by pattern list
      if HTTPRedirectResolver.isAllowedUrl(url, redirectProperty.urlPatternList):

        if redirectProperty.methodName == HTTPRedirectResolver.RedirectProperty.METHOD_NAME_HEAD:
          # method 'HEAD' execution
          ret = self.__fetch(url=url, method=HTTPRedirectResolver.RedirectProperty.METHOD_NAME_HEAD,
                             headers=headers, timeout=timeout, allowRedirects=allowRedirects, proxies=proxies,
                             auth=auth, postData=postData, maxRedirects=maxRedirects, filters=filters,
                             fetchType=self.fetchType)

        elif redirectProperty.methodName == HTTPRedirectResolver.RedirectProperty.METHOD_NAME_SGET:
          # method 'GET' for static fetcher type execution
          ret = self.__fetch(url=url, method=HTTPRedirectResolver.RedirectProperty.METHOD_NAME_GET,
                             headers=headers, timeout=timeout, allowRedirects=allowRedirects, proxies=proxies,
                             auth=auth, postData=postData, maxRedirects=maxRedirects, filters=filters,
                             fetchType=BaseFetcher.TYP_NORMAL)

        elif self.redirectProperty.methodName == HTTPRedirectResolver.RedirectProperty.METHOD_NAME_DGET:
          # method 'GET' for dynamic fetcher type execution
          ret = self.__fetch(url=url, method=HTTPRedirectResolver.RedirectProperty.METHOD_NAME_GET,
                             headers=headers, timeout=timeout, allowRedirects=allowRedirects, proxies=proxies,
                             auth=auth, postData=postData, maxRedirects=maxRedirects, filters=filters,
                             fetchType=BaseFetcher.TYP_DYNAMIC)

    return ret

Here is the call graph for this function:

Here is the caller graph for this function:

getRedirectProperty()

def dc_crawler.HTTPRedirectResolver.HTTPRedirectResolver.getRedirectProperty	(		self,
			propertyDict
	)

Definition at line 90 of file HTTPRedirectResolver.py.

  def getRedirectProperty(self, propertyDict):

Here is the caller graph for this function:

isAllowedUrl()

def dc_crawler.HTTPRedirectResolver.HTTPRedirectResolver.isAllowedUrl (   url,   patterns  )

static

Definition at line 172 of file HTTPRedirectResolver.py.

  def isAllowedUrl(url, patterns):
    # variable for result
    ret = True
    try:
      if isinstance(patterns, list):
        for pattern in patterns:
          if isinstance(pattern, basestring):
            ret = False
            if re.search(pattern, url, re.UNICODE + re.IGNORECASE) is not None:
              logger.debug("pattern: '%s' allowed for '%s'", str(pattern), str(url))
              ret = True
              break

    except Exception, err:
      logger.error(str(err))

    return ret

resolveRedirectUrl()

def dc_crawler.HTTPRedirectResolver.HTTPRedirectResolver.resolveRedirectUrl	(		self,
			url,
			headers,
			timeout = None,
			allowRedirects = True,
			proxies = None,
			auth = None,
			postData = None,
			maxRedirects = RedirectProperty.DEFAULT_VALUE_MAX_REDIRECTS,
			filters = None
	)

Definition at line 216 of file HTTPRedirectResolver.py.

                         maxRedirects=RedirectProperty.DEFAULT_VALUE_MAX_REDIRECTS, filters=None):

    # variable for result
    ret = None

    logger.debug("Input url: %s \nheaders: %s", str(url), varDump(headers))
#     logger.debug("method name: %s, max redirects = %s, redirect codes: %s",
#                  str(self.redirectProperty.methodName), str(self.redirectProperty.maxRedirects),
#                  str(self.redirectProperty.typesList))

#     self.__repairHeaders(headers) # remove in future because it's wrong logic
#     logger.debug("headers: %s", varDump(headers))

    try:

      ret = self.__resolveRedirect(url=url, method=self.redirectProperty.methodName, headers=headers, timeout=timeout,
                                   allowRedirects=allowRedirects, proxies=proxies, auth=auth, postData=postData,
                                   maxRedirects=maxRedirects, filters=filters,
                                   redirectProperty=self.redirectProperty)

      for urlPatternElem in self.redirectProperty.urlPatternList:
        logger.debug("urlPatternElem: %s", varDump(urlPatternElem))
        logger.debug("type(urlPatternElem) = %s", str(type(urlPatternElem)))

        if isinstance(urlPatternElem, HTTPRedirectResolver.RedirectProperty):
          res = self.__resolveRedirect(url=url, method=self.redirectProperty.methodName, headers=headers,
                                       timeout=timeout, allowRedirects=allowRedirects, proxies=proxies, auth=auth,
                                       postData=postData, maxRedirects=maxRedirects, filters=filters,
                                       redirectProperty=urlPatternElem)
          if res is not None:
            ret = res

    except CrawlerFilterException:
      logger.debug("Url '%s' should be skipped.", str(url))
    except (requests.exceptions.RequestException, Exception), err:
      logger.debug("Resolve redirect url failed: %s", str(err))
      logger.info(Utils.getTracebackInfo())

    return ret

Here is the call graph for this function:

Here is the caller graph for this function:

Member Data Documentation

connectionTimeout

dc_crawler.HTTPRedirectResolver.HTTPRedirectResolver.connectionTimeout

Definition at line 83 of file HTTPRedirectResolver.py.

dbWrapper

dc_crawler.HTTPRedirectResolver.HTTPRedirectResolver.dbWrapper

Definition at line 81 of file HTTPRedirectResolver.py.

ERROR_BAD_PROPERTY_VALUE

string dc_crawler.HTTPRedirectResolver.HTTPRedirectResolver.ERROR_BAD_PROPERTY_VALUE = "Not support value '%s' for property '%s'"

static

Definition at line 39 of file HTTPRedirectResolver.py.

ERROR_BAD_STATUS_CODE_VALUE

string dc_crawler.HTTPRedirectResolver.HTTPRedirectResolver.ERROR_BAD_STATUS_CODE_VALUE = "Not allowed status code '%s'. Allowed list: %s"

static

Definition at line 40 of file HTTPRedirectResolver.py.

ERROR_BAD_TYPE_HEADERS_VALUE

string dc_crawler.HTTPRedirectResolver.HTTPRedirectResolver.ERROR_BAD_TYPE_HEADERS_VALUE = "Wrong type (%s) of headers: %s"

static

Definition at line 38 of file HTTPRedirectResolver.py.

ERROR_BAD_TYPE_PROPERTY_VALUE

string dc_crawler.HTTPRedirectResolver.HTTPRedirectResolver.ERROR_BAD_TYPE_PROPERTY_VALUE = "Wrong type (%s) of property value: %s"

static

Definition at line 37 of file HTTPRedirectResolver.py.

ERROR_INITIALIZATION

string dc_crawler.HTTPRedirectResolver.HTTPRedirectResolver.ERROR_INITIALIZATION = "Initialization class '%s' was failed. Error: %s"

static

Definition at line 36 of file HTTPRedirectResolver.py.

fetchType

dc_crawler.HTTPRedirectResolver.HTTPRedirectResolver.fetchType

Definition at line 80 of file HTTPRedirectResolver.py.

maxRedirects

dc_crawler.HTTPRedirectResolver.HTTPRedirectResolver.maxRedirects

static

Definition at line 124 of file HTTPRedirectResolver.py.

methodName

dc_crawler.HTTPRedirectResolver.HTTPRedirectResolver.methodName

static

Definition at line 101 of file HTTPRedirectResolver.py.

redirectProperty

dc_crawler.HTTPRedirectResolver.HTTPRedirectResolver.redirectProperty = HTTPRedirectResolver.RedirectProperty()

static

Definition at line 92 of file HTTPRedirectResolver.py.

siteId

dc_crawler.HTTPRedirectResolver.HTTPRedirectResolver.siteId

Definition at line 82 of file HTTPRedirectResolver.py.

typesList

dc_crawler.HTTPRedirectResolver.HTTPRedirectResolver.typesList

static

Definition at line 130 of file HTTPRedirectResolver.py.

urlPatternList

dc_crawler.HTTPRedirectResolver.HTTPRedirectResolver.urlPatternList

static

Definition at line 110 of file HTTPRedirectResolver.py.

---

The documentation for this class was generated from the following file:

• sources/hce/dc_crawler/HTTPRedirectResolver.py