Inheritance diagram for dc_crawler.DetectModified.DetectModified:

Collaboration diagram for dc_crawler.DetectModified.DetectModified:

Public Member Functions
def	__init__ (self, modifiedSettingsStr)

def	generateETagsString (self, eTags)

def	headersClearing (self, httpParams)

def	makeHTTPRequest (self, fetchType, httpParams)

def	resourceComparing (self, res)

def	getBehaviour (self)

def	isNotModified (self)

def	notModifiedStateProcessing (self, siteId, url, dbWrapper, defaultStatus=dc_event.URL.STATUS_CRAWLED, defaultUpdateUDate=True)

Public Attributes
	modifiedSettings

	lastModified

	eTags

	isResourceNotChanged

	prevContentLen

	prevContentMd5

	prevContentDate

Static Public Attributes
int	MODE_DISABLED = 0

int	MODE_ONE_REQUEST = 1

int	MODE_TWO_REQUESTS = 2

int	ALGO_AUTO = 0

int	ALGO_IF_NOT_MATCH = 1

int	ALGO_IF_MOFIFIED_SINCE = 2

int	ALGO_HEAD = 3

int	COMPARE_AUTO = 0

int	COMPARE_DATE = 1

int	COMPARE_CRC32 = 2

int	COMPARE_RAW_CONTENT = 3

int	BEHAVIOR_DEFAULT = 0

int	BEHAVIOR_CRAWLED_STATUS = 1

int	BEHAVIOR_PROCESSED_STATUS = 2

int	BEHAVIOR_SAVE_UDATE = 3

Detailed Description

Definition at line 36 of file DetectModified.py.

Constructor & Destructor Documentation

◆ init()

def dc_crawler.DetectModified.DetectModified.__init__	(	self,
		modifiedSettingsStr
	)

Definition at line 61 of file DetectModified.py.

   def __init__(self, modifiedSettingsStr):
     try:
       self.modifiedSettings = json.loads(modifiedSettingsStr)
     except Exception:
       self.modifiedSettings = None
     self.lastModified = None
     self.eTags = None
     self.isResourceNotChanged = False
     self.prevContentLen = None
     self.prevContentMd5 = None
     self.prevContentDate = None
 
 

Member Function Documentation

◆ generateETagsString()

def dc_crawler.DetectModified.DetectModified.generateETagsString	(	self,
		eTags
	)

Definition at line 78 of file DetectModified.py.

   def generateETagsString(self, eTags):
     ret = ""
     if eTags is not None:
       if isinstance(eTags, list):
         # for eTag in eTags:
         #  ret += '"'
         #  ret += str(eTag)
         #  ret += '",'
         # ret = ret.strip(',')
         ret = '","'.join(eTags)
       elif isinstance(eTags, basestring):
         ret = eTags
       ret = '"' + ret + '"'
     return ret
 
 

Here is the caller graph for this function:

◆ getBehaviour()

def dc_crawler.DetectModified.DetectModified.getBehaviour ( self )

Definition at line 200 of file DetectModified.py.

   def getBehaviour(self):
     if self.modifiedSettings is not None:
       return self.modifiedSettings["behavior"]
     else:
       return None
 
 

Here is the caller graph for this function:

◆ headersClearing()

def dc_crawler.DetectModified.DetectModified.headersClearing	(	self,
		httpParams
	)

Definition at line 94 of file DetectModified.py.

   def headersClearing(self, httpParams):
     dellKeys = []
     for key in httpParams["httpHeader"]:
       if key.lower() == "if-none-match" or key.lower() == "if-modified-since":
         dellKeys.append(key)
     for elem in dellKeys:
       del httpParams["httpHeader"][elem]
 
 

Here is the caller graph for this function:

◆ isNotModified()

def dc_crawler.DetectModified.DetectModified.isNotModified ( self )

Definition at line 210 of file DetectModified.py.

   def isNotModified(self):
     # variable for result
     ret = False
     logger.debug("!!! isNotModified() enter ... self.isResourceNotChanged = " + str(bool(self.isResourceNotChanged)))
 
     if self.isResourceNotChanged and self.modifiedSettings is not None and \
     self.modifiedSettings["behavior"] in [self.BEHAVIOR_DEFAULT, \
                                           self.BEHAVIOR_CRAWLED_STATUS, \
                                           self.BEHAVIOR_PROCESSED_STATUS, \
                                           self.BEHAVIOR_SAVE_UDATE]:
       ret = True
 
     logger.debug("!!! isNotModified() leave ... ret = " + str(bool(ret)))
     return ret
 
 

Here is the call graph for this function:

◆ makeHTTPRequest()

def dc_crawler.DetectModified.DetectModified.makeHTTPRequest	(	self,
		fetchType,
		httpParams
	)

Definition at line 103 of file DetectModified.py.

   def makeHTTPRequest(self, fetchType, httpParams):
     ret = None
     self.isResourceNotChanged = False
     if self.modifiedSettings is not None:
     #  logger.debug(">>> httpParams = " + str(httpParams))
     #  logger.debug(">>> expiredData = " + str(self.lastModified))
     #  logger.debug(">>> algorithm = " + str(self.modifiedSettings["algorithm"]))
       if httpParams is None:
         httpParams = {}
 
       self.headersClearing(httpParams)
 
       localMethod = "get"
       if self.modifiedSettings["algorithm"] == self.ALGO_AUTO:
         if self.eTags is not None and self.eTags is not "":
           eTagsString = self.generateETagsString(self.eTags)
           httpParams["httpHeader"]["if-none-match"] = eTagsString
 
         if self.lastModified is not None:
           httpParams["httpHeader"]["if-modified-since"] = self.lastModified
 
       elif self.modifiedSettings["algorithm"] == self.ALGO_IF_NOT_MATCH:
         if self.eTags is not None and self.eTags is not "":
           eTagsString = self.generateETagsString(self.eTags)
           httpParams["httpHeader"]["if-none-match"] = eTagsString
 
       elif self.modifiedSettings["algorithm"] == self.ALGO_IF_MOFIFIED_SINCE:
         if self.lastModified is not None:
           httpParams["httpHeader"]["if-modified-since"] = self.lastModified
 
       elif self.modifiedSettings["algorithm"] == self.ALGO_HEAD:
         if self.modifiedSettings["mode"] == self.MODE_ONE_REQUEST:
           raise Exception(">>> Error [algorithm == 3 and mode == 1] not compatible !!!")
         localMethod = "head"
 
       ret = BaseFetcher.get_fetcher(fetchType).open(httpParams["url"],
                                                     timeout=httpParams["httpTimeout"],
                                                     headers=httpParams["httpHeader"],
                                                     allow_redirects=httpParams["allowHttpRedirects"],
                                                     proxies=httpParams["proxies"], auth=httpParams["auth"],
                                                     data=httpParams["postData"], log=logger,
                                                     allowed_content_types=httpParams["processContentTypes"],
                                                     max_resource_size=httpParams["maxResourceSize"],
                                                     max_redirects=httpParams["maxHttpRedirects"],
                                                     method=localMethod)
 
     if ret is not None:
       self.resourceComparing(ret)
       if not self.isResourceNotChanged and self.modifiedSettings["mode"] == self.MODE_TWO_REQUESTS:
         self.headersClearing(httpParams)
         ret = BaseFetcher.get_fetcher(fetchType).open(httpParams["url"],
                                                       timeout=httpParams["httpTimeout"],
                                                       headers=httpParams["httpHeader"],
                                                       allow_redirects=httpParams["allowHttpRedirects"],
                                                       proxies=httpParams["proxies"], auth=httpParams["auth"],
                                                       data=httpParams["postData"], log=logger,
                                                       allowed_content_types=httpParams["processContentTypes"],
                                                       max_resource_size=httpParams["maxResourceSize"],
                                                       max_redirects=httpParams["maxHttpRedirects"])
 
     # if ret is not None and ret.request is not None and ret.request.headers is not None:
     #  logger.debug(">>> requests headers = " + str(ret.request.headers))
 
     return ret
 
 

Here is the call graph for this function:

◆ notModifiedStateProcessing()

def dc_crawler.DetectModified.DetectModified.notModifiedStateProcessing	(	self,
		siteId,
		url,
		dbWrapper,
		defaultStatus = `dc_event.URL.STATUS_CRAWLED`,
		defaultUpdateUDate = `True`
	)

Definition at line 235 of file DetectModified.py.

                                  defaultUpdateUDate=True):
     # variables for result
     status = defaultStatus
     updateUDate = defaultUpdateUDate
     if self.isResourceNotChanged:
       status = dc_event.URL.STATUS_UNDEFINED
       updateUDate = False
       if self.getBehaviour() == self.BEHAVIOR_CRAWLED_STATUS:
         status = dc_event.URL.STATUS_CRAWLED
       elif self.getBehaviour() == self.BEHAVIOR_PROCESSED_STATUS:
         urlContentObj = dc_event.URLContentRequest(siteId, url, dc_event.URLContentRequest.CONTENT_TYPE_PROCESSED)
         urlContentResponse = dbWrapper.urlContent([urlContentObj])
         if len(urlContentResponse.processedContents) > 0:
           status = dc_event.URL.STATUS_PROCESSED
       elif self.getBehaviour() == self.BEHAVIOR_SAVE_UDATE:
         updateUDate = True
 
     return status, updateUDate
 

Here is the call graph for this function:



Here is the caller graph for this function:

◆ resourceComparing()

def dc_crawler.DetectModified.DetectModified.resourceComparing	(	self,
		res
	)

Definition at line 169 of file DetectModified.py.

   def resourceComparing(self, res):
     self.isResourceNotChanged = False
     if self.modifiedSettings["compare"] == self.COMPARE_AUTO:
       if res.status_code == 304:
         self.isResourceNotChanged = True
     elif self.modifiedSettings["compare"] == self.COMPARE_DATE:
       if "last-modified" in res.headers and self.prevContentDate is not None:
         try:
           resDate = time.mktime(rfc822.parsedate(res.headers["last-modified"]))
           prevResDate = time.mktime(rfc822.parsedate(self.prevContentDate))
           if resDate <= prevResDate:
             self.isResourceNotChanged = True
         except Exception:
           raise Exception(">>> Bad data format - resDate -" + str(res.headers["last-modified"]) + " or prevResDate -" +
                           str(self.prevContentDate))
     elif self.modifiedSettings["compare"] == self.COMPARE_CRC32:
       if res.rendered_unicode_content is not None and \
       hashlib.md5(res.rendered_unicode_content).hexdigest() == self.prevContentMd5:
         self.isResourceNotChanged = True
     elif self.modifiedSettings["compare"] == self.COMPARE_RAW_CONTENT:
       if res.rendered_unicode_content is not None and len(res.rendered_unicode_content) == self.prevContentLen:
         self.isResourceNotChanged = True
 
 
 #  def raiseExceptionIfNotModified(self):
 #    if self.isResourceNotChanged and self.modifiedSettings is not None and \
 #    self.modifiedSettings["behavior"] in [self.BEHAVIOR_CRAWLED_STATUS, self.BEHAVIOR_PROCESSED_STATUS, \
 #                                          self.BEHAVIOR_SAVE_UDATE]:
 #      raise NotModifiedException("Detect resource not modified state")
 
 

Here is the caller graph for this function:

Member Data Documentation

◆ ALGO_AUTO

int dc_crawler.DetectModified.DetectModified.ALGO_AUTO = 0

static

Definition at line 42 of file DetectModified.py.

◆ ALGO_HEAD

int dc_crawler.DetectModified.DetectModified.ALGO_HEAD = 3

static

Definition at line 45 of file DetectModified.py.

◆ ALGO_IF_MOFIFIED_SINCE

int dc_crawler.DetectModified.DetectModified.ALGO_IF_MOFIFIED_SINCE = 2

static

Definition at line 44 of file DetectModified.py.

◆ ALGO_IF_NOT_MATCH

int dc_crawler.DetectModified.DetectModified.ALGO_IF_NOT_MATCH = 1

static

Definition at line 43 of file DetectModified.py.

◆ BEHAVIOR_CRAWLED_STATUS

int dc_crawler.DetectModified.DetectModified.BEHAVIOR_CRAWLED_STATUS = 1

static

Definition at line 53 of file DetectModified.py.

◆ BEHAVIOR_DEFAULT

int dc_crawler.DetectModified.DetectModified.BEHAVIOR_DEFAULT = 0

static

Definition at line 52 of file DetectModified.py.

◆ BEHAVIOR_PROCESSED_STATUS

int dc_crawler.DetectModified.DetectModified.BEHAVIOR_PROCESSED_STATUS = 2

static

Definition at line 54 of file DetectModified.py.

◆ BEHAVIOR_SAVE_UDATE

int dc_crawler.DetectModified.DetectModified.BEHAVIOR_SAVE_UDATE = 3

static

Definition at line 55 of file DetectModified.py.

◆ COMPARE_AUTO

int dc_crawler.DetectModified.DetectModified.COMPARE_AUTO = 0

static

Definition at line 47 of file DetectModified.py.

◆ COMPARE_CRC32

int dc_crawler.DetectModified.DetectModified.COMPARE_CRC32 = 2

static

Definition at line 49 of file DetectModified.py.

◆ COMPARE_DATE

int dc_crawler.DetectModified.DetectModified.COMPARE_DATE = 1

static

Definition at line 48 of file DetectModified.py.

◆ COMPARE_RAW_CONTENT

int dc_crawler.DetectModified.DetectModified.COMPARE_RAW_CONTENT = 3

static

Definition at line 50 of file DetectModified.py.

◆ eTags

dc_crawler.DetectModified.DetectModified.eTags

Definition at line 67 of file DetectModified.py.

◆ isResourceNotChanged

dc_crawler.DetectModified.DetectModified.isResourceNotChanged

Definition at line 68 of file DetectModified.py.

◆ lastModified

dc_crawler.DetectModified.DetectModified.lastModified

Definition at line 66 of file DetectModified.py.

◆ MODE_DISABLED

int dc_crawler.DetectModified.DetectModified.MODE_DISABLED = 0

static

Definition at line 38 of file DetectModified.py.

◆ MODE_ONE_REQUEST

int dc_crawler.DetectModified.DetectModified.MODE_ONE_REQUEST = 1

static

Definition at line 39 of file DetectModified.py.

◆ MODE_TWO_REQUESTS

int dc_crawler.DetectModified.DetectModified.MODE_TWO_REQUESTS = 2

static

Definition at line 40 of file DetectModified.py.

◆ modifiedSettings

dc_crawler.DetectModified.DetectModified.modifiedSettings

Definition at line 63 of file DetectModified.py.

◆ prevContentDate

dc_crawler.DetectModified.DetectModified.prevContentDate

Definition at line 71 of file DetectModified.py.

◆ prevContentLen

dc_crawler.DetectModified.DetectModified.prevContentLen

Definition at line 69 of file DetectModified.py.

◆ prevContentMd5

dc_crawler.DetectModified.DetectModified.prevContentMd5

Definition at line 70 of file DetectModified.py.

The documentation for this class was generated from the following file:

sources/hce/dc_crawler/DetectModified.py

Public Member Functions

Public Attributes

Static Public Attributes

Detailed Description

Constructor & Destructor Documentation

◆ __init__()

Member Function Documentation

◆ generateETagsString()

◆ getBehaviour()

◆ headersClearing()

◆ isNotModified()

◆ makeHTTPRequest()

◆ notModifiedStateProcessing()

◆ resourceComparing()

Member Data Documentation

◆ ALGO_AUTO

◆ ALGO_HEAD

◆ ALGO_IF_MOFIFIED_SINCE

◆ ALGO_IF_NOT_MATCH

◆ BEHAVIOR_CRAWLED_STATUS

◆ BEHAVIOR_DEFAULT

◆ BEHAVIOR_PROCESSED_STATUS

◆ BEHAVIOR_SAVE_UDATE

◆ COMPARE_AUTO

◆ COMPARE_CRC32

◆ COMPARE_DATE

◆ COMPARE_RAW_CONTENT

◆ eTags

◆ isResourceNotChanged

◆ lastModified

◆ MODE_DISABLED

◆ MODE_ONE_REQUEST

◆ MODE_TWO_REQUESTS

◆ modifiedSettings

◆ prevContentDate

◆ prevContentLen

◆ prevContentMd5

◆ init()