dc_crawler.HTTPCookieResolver.HTTPCookieResolver Class Reference

Inheritance diagram for dc_crawler.HTTPCookieResolver.HTTPCookieResolver:

Collaboration diagram for dc_crawler.HTTPCookieResolver.HTTPCookieResolver:

Classes
class	Cookie
class	DomainProperty

Public Member Functions
def	__init__ (self, propertyString=None)
def	addCookie (self, url, cookie, strip=True)
def	getCookie (self, url, stage=STAGE_DEFAULT)

Public Attributes
	cookiesDict
	property

Static Public Attributes
int	STAGE_REGULAR = 1
int	STAGE_REDIRECT = 2
int	STAGE_ROBOTS = 4
int	STAGE_RSS = 8
int	STAGE_DEFAULT = STAGE_REGULAR | STAGE_REDIRECT | STAGE_ROBOTS | STAGE_RSS
string	DOMAIN_DEFAULT = '.*'
string	ERROR_BAD_TYPE_INPUT_PROPERTY = "Bad type (%s) of input property: %s"
string	ERROR_BAD_TYPE_FOUND_PROPERTY = "Bad type (%s) of found property: %s"
string	ERROR_INPUT_PROPERTY = "Input wrong properties: %s"
string	ERROR_INITIALIZATION = "Initialization class '%s' was failed. Error: '%s'"

Private Member Functions
def	__loadProperty (self, propertyString)
def	__stripUrl (self, url)
def	__extractPair (self, element)
def	__splitCookieString (self, cookieString)
def	__isAllowedCookie (self, url, cookieObj)
def	__extractDomainProperty (self, url)
def	__isAllowedStage (self, stage, propertyStage)
def	__getDomainProperty (self, properties, domain)

Detailed Description

Definition at line 25 of file HTTPCookieResolver.py.

Constructor & Destructor Documentation

__init__()

def dc_crawler.HTTPCookieResolver.HTTPCookieResolver.__init__	(		self,
			propertyString = None
	)

Definition at line 74 of file HTTPCookieResolver.py.

  def __init__(self, propertyString=None):
    self.cookiesDict = {}
    self.property = {self.DOMAIN_DEFAULT:{'stage':self.STAGE_DEFAULT}}

    if propertyString is not None and isinstance(propertyString, basestring):
      self.property = self.__loadProperty(propertyString)
      logger.debug("!!! self.property: %s", varDump(self.property))

Member Function Documentation

__extractDomainProperty()

def dc_crawler.HTTPCookieResolver.HTTPCookieResolver.__extractDomainProperty (   self,   url  )

private

Definition at line 238 of file HTTPCookieResolver.py.

  def __extractDomainProperty(self, url):
    # variables for result
    propertyStage = HTTPCookieResolver.STAGE_DEFAULT
    propertyCookie = None

    propertyObj = self.__getDomainProperty(self.property, url)
    # logger.debug('propertyObj: ' + str(propertyObj))

    if propertyObj is not None:
      if propertyObj.stage is not None:
        propertyStage = propertyObj.stage
      propertyCookie = propertyObj.cookie

    return propertyStage, propertyCookie

Here is the call graph for this function:

Here is the caller graph for this function:

__extractPair()

def dc_crawler.HTTPCookieResolver.HTTPCookieResolver.__extractPair (   self,   element  )

private

Definition at line 135 of file HTTPCookieResolver.py.

  def __extractPair(self, element):
    # variables for result
    name = value = ''
    pairElem = element.split('=')
    if len(pairElem) > 0:
      name = pairElem[0]

    if len(pairElem) > 1:
      value = pairElem[1]

    return name, value

Here is the caller graph for this function:

__getDomainProperty()

def dc_crawler.HTTPCookieResolver.HTTPCookieResolver.__getDomainProperty (   self,   properties,   domain  )

private

Definition at line 273 of file HTTPCookieResolver.py.

  def __getDomainProperty(self, properties, domain):
    # variable for result
    ret = None
    try:
      if not isinstance(properties, dict):
        raise Exception(self.ERROR_BAD_TYPE_INPUT_PROPERTY % (str(type(property)), str(property)))

      foundProperty = None
      if domain in properties:
        foundProperty = properties[domain]
      elif self.DOMAIN_DEFAULT in properties:
        foundProperty = properties[self.DOMAIN_DEFAULT]
      else:
        logger.debug(self.ERROR_INPUT_PROPERTY, varDump(property))

      if foundProperty is not None:
        if not isinstance(foundProperty, dict):
          raise Exception(self.ERROR_BAD_TYPE_FOUND_PROPERTY % (str(type(foundProperty)), str(foundProperty)))

        domainPropertyObj = HTTPCookieResolver.DomainProperty()
        for name, value in foundProperty.items():
          if hasattr(domainPropertyObj, name):
            setattr(domainPropertyObj, name, value)
        ret = domainPropertyObj

    except Exception, err:
      logger.error(str(err))

    return ret

Here is the call graph for this function:

Here is the caller graph for this function:

__isAllowedCookie()

def dc_crawler.HTTPCookieResolver.HTTPCookieResolver.__isAllowedCookie (   self,   url,   cookieObj  )

private

Definition at line 205 of file HTTPCookieResolver.py.

  def __isAllowedCookie(self, url, cookieObj):
    # variable for result
    ret = False

    isAllowedPath = isAllowedExpires = True

    # check 'path'
    if cookieObj.path is not None and re.search('.*' + cookieObj.path + '.*', url, re.UNICODE) is None:
      isAllowedPath = False

    # check 'expired'
    if cookieObj.expires is not None:
      expiresDatetime = DateTimeType.parse(cookieObj.expires, True, logger, False)
      if expiresDatetime is not None:
        expiresDatetime = DateTimeType.toUTC(expiresDatetime)
        currentDatetime = datetime.datetime.utcnow()
        currentDatetime = currentDatetime.replace(tzinfo=None)
        if currentDatetime > expiresDatetime:
          isAllowedExpires = False

    logger.debug("Is allowed = %s for path '%s'", str(isAllowedPath), str(cookieObj.path))
    logger.debug("Is allowed = %s for expired '%s'", str(isAllowedExpires), str(cookieObj.expires))

    if isAllowedPath and isAllowedExpires:
      ret = True

    return ret

Here is the caller graph for this function:

__isAllowedStage()

def dc_crawler.HTTPCookieResolver.HTTPCookieResolver.__isAllowedStage (   self,   stage,   propertyStage  )

private

Definition at line 259 of file HTTPCookieResolver.py.

  def __isAllowedStage(self, stage, propertyStage):
    # variable for result
    ret = False
    if stage & int(propertyStage):
      ret = True

    return ret

Here is the caller graph for this function:

__loadProperty()

def dc_crawler.HTTPCookieResolver.HTTPCookieResolver.__loadProperty (   self,   propertyString  )

private

Definition at line 86 of file HTTPCookieResolver.py.

  def __loadProperty(self, propertyString):
    # variable for result
    ret = None
    try:
      ret = json.loads(propertyString)
    except Exception, err:
      logger.error(self.ERROR_INITIALIZATION, self.__class__.__name__, str(err))

    return ret

Here is the caller graph for this function:

__splitCookieString()

def dc_crawler.HTTPCookieResolver.HTTPCookieResolver.__splitCookieString (   self,   cookieString  )

private

Definition at line 152 of file HTTPCookieResolver.py.

  def __splitCookieString(self, cookieString):
    # variable for result
    cookies = []
    # extract elements
    elementsList = cookieString.split(';')
    elements = []
    for element in elementsList:
      if element.count('=') > 1:
        endPos = element.rfind('=')
        begPos = element.rfind(' ', 0, endPos)

        first = element[:begPos]
        second = element[begPos:]
        first = first.strip(',')

        elements.append(first.strip())
        elements.append(second.strip())
      else:
        elements.append(element.strip())

    # logger.debug("!!! elements: %s", varDump(elements))
    cookieObj = HTTPCookieResolver.Cookie()
    for element in elements:
      name, value = self.__extractPair(element)
      # logger.debug("!!! name = %s, value = %s", name, value)

      if name.lower() not in HTTPCookieResolver.Cookie.supportNames and \
        cookieObj.name is not None and cookieObj.value is not None:
        cookies.append(cookieObj)
        cookieObj = HTTPCookieResolver.Cookie()

      if name.lower() in HTTPCookieResolver.Cookie.supportNames:
        if hasattr(cookieObj, name.lower()):
          setattr(cookieObj, name.lower(), value)
      elif cookieObj.name is None and cookieObj.value is None:
        cookieObj.name = name
        cookieObj.value = value

    if cookieObj.name is not None and cookieObj.value is not None:
      cookies.append(cookieObj)

    logger.debug("!!! cookies:")
    for cookie in cookies:
      logger.debug("%s", varDump(cookie))

    return cookies

Here is the call graph for this function:

Here is the caller graph for this function:

__stripUrl()

def dc_crawler.HTTPCookieResolver.HTTPCookieResolver.__stripUrl (   self,   url  )

private

Definition at line 124 of file HTTPCookieResolver.py.

  def __stripUrl(self, url):
    if isinstance(url, basestring) and url.count('?') > 0:
      url = url[:url.find('?')]

    return url

Here is the caller graph for this function:

addCookie()

def dc_crawler.HTTPCookieResolver.HTTPCookieResolver.addCookie	(		self,
			url,
			cookie,
			strip = True
	)

Definition at line 103 of file HTTPCookieResolver.py.

  def addCookie(self, url, cookie, strip=True):
    # logger.debug("!!! addCookie ENTER !!! cookiesDict: %s", varDump(self.cookiesDict))
    # logger.debug("!!! url: '%s', cookie: %s, type = %s, strip = %s",
    #              str(url), str(cookie), str(type(cookie)), str(strip))

    localUrl = self.__stripUrl(url) if strip else url
    # logger.debug("!!! localUrl: %s", str(localUrl))

    if isinstance(cookie, basestring):
      if cookie != "":
        self.cookiesDict[localUrl] = ';'.join(cookie.split())
    elif isinstance(cookie, dict):
      self.cookiesDict[localUrl] = ';'.join(['%s=%s' % (k, v) for k, v in cookie.iteritems()])

    # logger.debug("!!! addCookie LEAVE !!! cookiesDict: %s", varDump(self.cookiesDict))

Here is the call graph for this function:

getCookie()

def dc_crawler.HTTPCookieResolver.HTTPCookieResolver.getCookie	(		self,
			url,
			stage = STAGE_DEFAULT
	)

Definition at line 309 of file HTTPCookieResolver.py.

  def getCookie(self, url, stage=STAGE_DEFAULT):
    # variable for result
    ret = None

    logger.debug('!!! getCookie ENTER !!! url: ' + str(url))
    if url in self.cookiesDict.keys():
      if self.cookiesDict[url] is not None and self.cookiesDict[url] != "":
        logger.debug('!!! getCookie LEAVE !!! return: ' + str(';'.join(self.cookiesDict[url].split())))
        return ';'.join(self.cookiesDict[url].split())

    for localUrl, cookieString in self.cookiesDict.items():
      logger.debug("!!! localUrl = %s, cookieString: %s", str(localUrl), str(cookieString))

      propertyStage, propertyCookie = self.__extractDomainProperty(parseHost(localUrl))
      logger.debug("!!! propertyStage = %s, propertyCookie: %s", str(propertyStage), varDump(propertyCookie))
      logger.debug('is allowed stage: ' + str(self.__isAllowedStage(stage, propertyStage)))

      if self.__isAllowedStage(stage, propertyStage):
        if propertyCookie is None:  # not exist default cookie
          logger.debug('cookieString: ' + str(cookieString))

          cookies = self.__splitCookieString(cookieString)
          logger.debug('cookies: ' + varDump(cookies))
          logger.debug('localUrl: ' + str(localUrl))
          logger.debug('url: ' + str(url))

          resStr = ''
          resList = []
          for cookie in cookies:
            if cookie.domain is None or (cookie.domain is not None and \
                                         re.search('.*' + cookie.domain + '.*', url, re.UNICODE) is not None):

              logger.debug('is allowed: ' + str(self.__isAllowedCookie(url, cookie)))
              if self.__isAllowedCookie(url, cookie):
                resList.append(cookie.getData())

          # remove dublicate
          resList = list(set(resList))
          resStr = ''.join(resList)

          if resStr is not '':
            ret = resStr

        else:  # apply default value of cookie
          ret = propertyCookie

    logger.debug('return cookie: ' + str(ret))

    return ret

Member Data Documentation

cookiesDict

dc_crawler.HTTPCookieResolver.HTTPCookieResolver.cookiesDict

Definition at line 75 of file HTTPCookieResolver.py.

DOMAIN_DEFAULT

string dc_crawler.HTTPCookieResolver.HTTPCookieResolver.DOMAIN_DEFAULT = '.*'

static

Definition at line 33 of file HTTPCookieResolver.py.

ERROR_BAD_TYPE_FOUND_PROPERTY

string dc_crawler.HTTPCookieResolver.HTTPCookieResolver.ERROR_BAD_TYPE_FOUND_PROPERTY = "Bad type (%s) of found property: %s"

static

Definition at line 37 of file HTTPCookieResolver.py.

ERROR_BAD_TYPE_INPUT_PROPERTY

string dc_crawler.HTTPCookieResolver.HTTPCookieResolver.ERROR_BAD_TYPE_INPUT_PROPERTY = "Bad type (%s) of input property: %s"

static

Definition at line 36 of file HTTPCookieResolver.py.

ERROR_INITIALIZATION

string dc_crawler.HTTPCookieResolver.HTTPCookieResolver.ERROR_INITIALIZATION = "Initialization class '%s' was failed. Error: '%s'"

static

Definition at line 39 of file HTTPCookieResolver.py.

ERROR_INPUT_PROPERTY

string dc_crawler.HTTPCookieResolver.HTTPCookieResolver.ERROR_INPUT_PROPERTY = "Input wrong properties: %s"

static

Definition at line 38 of file HTTPCookieResolver.py.

property

dc_crawler.HTTPCookieResolver.HTTPCookieResolver.property

Definition at line 76 of file HTTPCookieResolver.py.

STAGE_DEFAULT

int dc_crawler.HTTPCookieResolver.HTTPCookieResolver.STAGE_DEFAULT = STAGE_REGULAR | STAGE_REDIRECT | STAGE_ROBOTS | STAGE_RSS

static

Definition at line 32 of file HTTPCookieResolver.py.

STAGE_REDIRECT

int dc_crawler.HTTPCookieResolver.HTTPCookieResolver.STAGE_REDIRECT = 2

static

Definition at line 28 of file HTTPCookieResolver.py.

STAGE_REGULAR

int dc_crawler.HTTPCookieResolver.HTTPCookieResolver.STAGE_REGULAR = 1

static

Definition at line 27 of file HTTPCookieResolver.py.

STAGE_ROBOTS

int dc_crawler.HTTPCookieResolver.HTTPCookieResolver.STAGE_ROBOTS = 4

static

Definition at line 29 of file HTTPCookieResolver.py.

STAGE_RSS

int dc_crawler.HTTPCookieResolver.HTTPCookieResolver.STAGE_RSS = 8

static

Definition at line 30 of file HTTPCookieResolver.py.

---

The documentation for this class was generated from the following file:

• sources/hce/dc_crawler/HTTPCookieResolver.py