dc_processor.custom_extractor.CustomExtractor Class Reference

Inheritance diagram for dc_processor.custom_extractor.CustomExtractor:

Collaboration diagram for dc_processor.custom_extractor.CustomExtractor:

Public Member Functions
def	__init__ (self, config, templ=None, domain=None, processorProperties=None)
def	extractTags (self, resource, reslut)
Public Member Functions inherited from dc_processor.base_extractor.BaseExtractor
def	__init__ (self, config, templ=None, domain=None, processorProperties=None)
def	__str__ (self)
def	__repr__ (self)
def	loadScraperProperties (self, scraperPropFileName)
def	isTagNotFilled (self, result, tagName)
def	isTagValueNotEmpty (self, tagValue)
def	tagValueElemValidate (self, tagValueElem, conditionElem)
def	tagValueValidate (self, tagName, tagValue)
def	addTag (self, result, tag_name, tag_value, xpath="", isDefaultTag=False, callAdjustment=True, tagType=None, allowNotFilled=False)
def	calculateMetrics (self, response)
def	rankReading (self, exctractorName)

Public Attributes
	name
Public Attributes inherited from dc_processor.base_extractor.BaseExtractor
	config
	processorProperties
	name
	rank
	process_mode
	modules
	data
	db_dc_scraper_db
	DBConnector
	imgDelimiter
	tagsValidator

Additional Inherited Members
Static Public Attributes inherited from dc_processor.base_extractor.BaseExtractor
	properties = None
dictionary	tag
dictionary	tagsMask

Detailed Description

Definition at line 27 of file custom_extractor.py.

Constructor & Destructor Documentation

__init__()

def dc_processor.custom_extractor.CustomExtractor.__init__	(		self,
			config,
			templ = None,
			domain = None,
			processorProperties = None
	)

Definition at line 36 of file custom_extractor.py.

  def __init__(self, config, templ=None, domain=None, processorProperties=None):
    try:
      BaseExtractor.__init__(self, config, templ, domain, processorProperties)
      logger.debug("Properties: %s", varDump(self.properties))
      self.name = "Custom extractor"
      # set module rank from module's properties
      self.rankReading(self.__class__.__name__)
      self.data["extractor"] = self.name
    except Exception as err:
      ExceptionLog.handler(logger, err, "Custom extractor constructor error: possible /tmp not permitted to write", (),
                           {ExceptionLog.LEVEL_NAME_ERROR:ExceptionLog.LEVEL_VALUE_DEBUG})
      raise

Here is the call graph for this function:

Member Function Documentation

extractTags()

def dc_processor.custom_extractor.CustomExtractor.extractTags	(		self,
			resource,
			reslut
	)

Definition at line 54 of file custom_extractor.py.

  def extractTags(self, resource, reslut):
    # support time execution limit
    signal.signal(signal.SIGALRM, signal_handler)
    if 'EXTRACTOR_CUSTOM_MAX_EXECUTION' in self.processorProperties:
      t = int(self.processorProperties['EXTRACTOR_CUSTOM_MAX_EXECUTION'])
    else:
      t = CONSTS.TIME_EXECUTION_LIMIT
    signal.alarm(t)
    logger.debug("Max execution time signal handler set timeout as: %s", str(t))

    try:
      if resource.raw_html is not None and isinstance(resource.raw_html, types.DictType):
        for key in resource.raw_html:
          localTagValue = resource.raw_html[key] if isinstance(resource.raw_html[key], types.ListType) else \
                          [str(resource.raw_html[key])]
          self.addTag(result=reslut, tag_name=key, tag_value=localTagValue)
    except IOError as err:
      ExceptionLog.handler(logger, err, "Custom extractor file error. It may be unsupported encoding like jp", (), \
                            {ExceptionLog.LEVEL_NAME_ERROR:ExceptionLog.LEVEL_VALUE_DEBUG})
    except Exception as err:
      ExceptionLog.handler(logger, err, "Custom extractor error", (), \
                            {ExceptionLog.LEVEL_NAME_ERROR:ExceptionLog.LEVEL_VALUE_DEBUG})
    return reslut

Member Data Documentation

name

dc_processor.custom_extractor.CustomExtractor.name

Definition at line 40 of file custom_extractor.py.

---

The documentation for this class was generated from the following file:

• sources/hce/dc_processor/custom_extractor.py