dc_crawler.RefererHeaderResolver.RefererHeaderResolver Class Reference

Inheritance diagram for dc_crawler.RefererHeaderResolver.RefererHeaderResolver:

Collaboration diagram for dc_crawler.RefererHeaderResolver.RefererHeaderResolver:

Public Member Functions
def	__init__ (self, dbWrapper=None)
def	fetchParentUrl (self, siteId, parentMd5, dbWrapper)
def	resolveRefererHeader (self, headers, mode, url, siteId=None, parentMd5=None, dbWrapper=None)

Public Attributes
	dbWrapper

Static Public Attributes
int	MODE_NONE = 0
int	MODE_SIMPLE = 1
int	MODE_DOMAIN = 2
int	MODE_PARENT = 3
string	HEADER_NAME = "Referer"

Detailed Description

Definition at line 22 of file RefererHeaderResolver.py.

Constructor & Destructor Documentation

__init__()

def dc_crawler.RefererHeaderResolver.RefererHeaderResolver.__init__	(		self,
			dbWrapper = None
	)

Definition at line 31 of file RefererHeaderResolver.py.

  def __init__(self, dbWrapper=None):
    self.dbWrapper = dbWrapper

Member Function Documentation

fetchParentUrl()

def dc_crawler.RefererHeaderResolver.RefererHeaderResolver.fetchParentUrl	(		self,
			siteId,
			parentMd5,
			dbWrapper
	)

Definition at line 40 of file RefererHeaderResolver.py.

  def fetchParentUrl(self, siteId, parentMd5, dbWrapper):
    ret = None
    if siteId is not None and parentMd5 is not None and dbWrapper is not None:
      urlStatus = dc.EventObjects.URLStatus(siteId, parentMd5)
      urlStatus.urlType = dc.EventObjects.URLStatus.URL_TYPE_MD5
      drceSyncTasksCoverObj = DC_CONSTS.DRCESyncTasksCover(DC_CONSTS.EVENT_TYPES.URL_STATUS, [urlStatus])
      responseDRCESyncTasksCover = dbWrapper.process(drceSyncTasksCoverObj)
      row = responseDRCESyncTasksCover.eventObject
      if row is not None and len(row) > 0 and row[0] is not None:
        ret = row[0].url
    return ret

Here is the caller graph for this function:

resolveRefererHeader()

def dc_crawler.RefererHeaderResolver.RefererHeaderResolver.resolveRefererHeader	(		self,
			headers,
			mode,
			url,
			siteId = None,
			parentMd5 = None,
			dbWrapper = None
	)

Definition at line 61 of file RefererHeaderResolver.py.

  def resolveRefererHeader(self, headers, mode, url, siteId=None, parentMd5=None, dbWrapper=None):
    mode = int(mode)

    for headerName in headers:
      if headerName.lower() == self.HEADER_NAME.lower():
        logger.info(">>> Referer field already in dict headers")
        return

    if mode == self.MODE_NONE:
      pass
    elif mode == self.MODE_SIMPLE:
      headers[self.HEADER_NAME] = url
    elif mode == self.MODE_DOMAIN:
      headers[self.HEADER_NAME] = Utils.UrlParser.generateDomainUrl(url)
    elif mode == self.MODE_PARENT:
      parentUrl = self.fetchParentUrl(siteId, parentMd5, dbWrapper if dbWrapper is not None else self.dbWrapper)
      headers[self.HEADER_NAME] = parentUrl if parentUrl is not None else url

Member Data Documentation

dbWrapper

dc_crawler.RefererHeaderResolver.RefererHeaderResolver.dbWrapper

Definition at line 32 of file RefererHeaderResolver.py.

HEADER_NAME

string dc_crawler.RefererHeaderResolver.RefererHeaderResolver.HEADER_NAME = "Referer"

static

Definition at line 28 of file RefererHeaderResolver.py.

MODE_DOMAIN

int dc_crawler.RefererHeaderResolver.RefererHeaderResolver.MODE_DOMAIN = 2

static

Definition at line 26 of file RefererHeaderResolver.py.

MODE_NONE

int dc_crawler.RefererHeaderResolver.RefererHeaderResolver.MODE_NONE = 0

static

Definition at line 24 of file RefererHeaderResolver.py.

MODE_PARENT

int dc_crawler.RefererHeaderResolver.RefererHeaderResolver.MODE_PARENT = 3

static

Definition at line 27 of file RefererHeaderResolver.py.

MODE_SIMPLE

int dc_crawler.RefererHeaderResolver.RefererHeaderResolver.MODE_SIMPLE = 1

static

Definition at line 25 of file RefererHeaderResolver.py.

---

The documentation for this class was generated from the following file:

• sources/hce/dc_crawler/RefererHeaderResolver.py