dc_crawler.FetcherType.FetcherType Class Reference

Inheritance diagram for dc_crawler.FetcherType.FetcherType:

Collaboration diagram for dc_crawler.FetcherType.FetcherType:

Static Public Member Functions
def	getFromProperty (propertyValue, siteUrl, logger=None)
	Check the fetcher type by FETCHER_TYPE project property. More...

Detailed Description

Definition at line 21 of file FetcherType.py.

Member Function Documentation

getFromProperty()

def dc_crawler.FetcherType.FetcherType.getFromProperty (   propertyValue,   siteUrl,   logger = None  )

static

Check the fetcher type by FETCHER_TYPE project property.

Parameters

urlObj	- url request object
propertyValue	- json string with pattern rules
siteUrl	- site URL string with pattern rules. Only for debug purpose
logger	- instance of logger for log if necessary
isExtendLog	- boolean flag for allowed extend logging if True or only error message otherwise

Returns

extracted fetcher type

Definition at line 32 of file FetcherType.py.

  def getFromProperty(propertyValue, siteUrl, logger=None):
    #isExtendLog = False
    if logger is not None:
      logger.debug(siteUrl + ', FETCHER_TYPE: ' + propertyValue)

    fetchType = None
    try:
      fetcherProperties = json.loads(propertyValue)
      for pattern in fetcherProperties:
        match = re.search(pattern, siteUrl)
        if match:
          fetchType = fetcherProperties[pattern]
          # if fetchType in BaseFetcher.fetchers:
          if fetchType == BaseFetcher.TYP_DYNAMIC or fetchType == BaseFetcher.TYP_NORMAL:
            if logger is not None:
              logger.info(siteUrl + ', Fetch Type value: ' + str(fetchType))
          else:
            logger.debug(siteUrl + ', wrong Fetch Type number: ' + str(fetchType))
            fetchType = None
            continue
          break
    except Exception, ex:
      if logger is not None:
        logger.debug("Fetcher Type Exception: " + str(ex))
    return fetchType

---

The documentation for this class was generated from the following file:

• sources/hce/dc_crawler/FetcherType.py