app.ResponseExtractor.ResponseExtractor Class Reference

ResponseExtractor Class main functional, class inherits from foundation.CementApp. More...

Inheritance diagram for app.ResponseExtractor.ResponseExtractor:

Collaboration diagram for app.ResponseExtractor.ResponseExtractor:

Classes
class	Meta

Public Member Functions
def	__init__ (self)
	constructor More...
def	setup (self)
	setup application More...
def	initLogger (self, configName)
	load log config file More...
def	readFile (self, templateFile)
	Read file. More...
def	writeFile (self, fileName, outBuffer)
	Write file. More...
def	run (self)
	run application More...
def	getNewsItem (self, contentObj, tags=None)
	Make output content by substitution of the template's parts. More...
def	getDefaultItem (self, contentObj, tags=None)
	Make output content by substitution of the template's parts. More...
def	getTagValueByName (self, tagName, item, responseFormat)
	Check is tag present in response item by the name. More...
def	parse (self, inputObject, inputFormat, maxItems=-1, startFrom=0, tags=None)
	Parse the input json and make output collection. More...
def	logResultedStatistics (self, inputObject, items)
	Detects the format of response object. More...
def	detectFormat (self, contentObj)
	Detects the format of response object. More...
def	jsonLoadsSafe (self, jsonString)
	Parse json and return dict if okay or None if not. More...
def	process (self, inputBuffer, inputFormat=FROMAT_AUTO, maxItems=-1, startFrom=0, tags=None)
	process main operations More...

Public Attributes
	logger
	batch
	exitCode
	initTagsUniqueHashConfig
	initTagsLimitsConfig
	extendedLog
	results
	itemDelimiter

Static Public Attributes
string	MSG_ERROR_PARSE_CMD_PARAMS = "Error parse command line parameters."
	Constants error messages used in class. More...
string	MSG_ERROR_EMPTY_CONFIG_FILE_NAME = "Config file name is empty."
string	MSG_ERROR_WRONG_CONFIG_FILE_NAME = "Config file name is wrong"
string	MSG_ERROR_LOAD_APP_CONFIG = "Error loading application config file."
string	MSG_ERROR_READ_LOG_CONFIG = "Error read log config file."
string	MSG_ERROR_PROCESSING_REQUEST = "Error processing input data."
string	CONFIG_OPTION_ITEM_DELIMITER = "itemDelimiter"
int	FROMAT_AUTO = -1
int	FROMAT_INTERNAL = 0
int	FROMAT_NEWS = 1
int	FROMAT_RSS_FEED = 2
int	RESULTS_FORMAT_CSV_LINE = 0
int	RESULTS_FORMAT_FIELD_LINE = 1
int	RESULTS_FORMAT_JSON = 2
string	CONFIG_OPTION_LOG = 'log'

Private Member Functions
def	__loadAppConfig (self, configName)
	load application config file More...

Detailed Description

ResponseExtractor Class main functional, class inherits from foundation.CementApp.

Definition at line 33 of file ResponseExtractor.py.

Constructor & Destructor Documentation

__init__()

def app.ResponseExtractor.ResponseExtractor.__init__

(

self

)

constructor

Definition at line 66 of file ResponseExtractor.py.

  def __init__(self):
    # call base class __init__ method
    foundation.CementApp.__init__(self)

    self.logger = None
    self.batch = None
    self.exitCode = APP_CONSTS.EXIT_SUCCESS
    self.initTagsUniqueHashConfig = ''
    self.initTagsLimitsConfig = ''
    self.extendedLog = False
    self.results = self.RESULTS_FORMAT_JSON
    self.itemDelimiter = "\n"

Member Function Documentation

__loadAppConfig()

def app.ResponseExtractor.ResponseExtractor.__loadAppConfig (   self,   configName  )

private

load application config file

Parameters

configName

- name of application config file

Returns

- log config file name

Definition at line 108 of file ResponseExtractor.py.

  def __loadAppConfig(self, configName):
    #variable for result
    confLogFileName = ""

    try:
      config = ConfigParser.ConfigParser()
      config.optionxform = str

      readOk = config.read(configName)

      if len(readOk) == 0:
        raise Exception(self.MSG_ERROR_WRONG_CONFIG_FILE_NAME + ": " + configName)

      if config.has_section(APP_CONSTS.CONFIG_APPLICATION_SECTION_NAME):
        confLogFileName = str(config.get(APP_CONSTS.CONFIG_APPLICATION_SECTION_NAME, self.CONFIG_OPTION_LOG))
        #Init response item delimiter
        self.itemDelimiter = str(config.get(APP_CONSTS.CONFIG_APPLICATION_SECTION_NAME,
                                            self.CONFIG_OPTION_ITEM_DELIMITER))
    except Exception, err:
      raise Exception(self.MSG_ERROR_LOAD_APP_CONFIG + ' ' + str(err))

    return confLogFileName

Here is the caller graph for this function:

detectFormat()

def app.ResponseExtractor.ResponseExtractor.detectFormat	(		self,
			contentObj
	)

Detects the format of response object.

Parameters

contentObj

response object

Returns

format code

Definition at line 379 of file ResponseExtractor.py.

  def detectFormat(self, contentObj):
    #self.logger.info("Type `%s`, contentObj:\n%s", str(type(contentObj)), str(contentObj))
    #Auto-detect format
    if isinstance(contentObj, dict):
      #Default internal format
      inputFormat = self.FROMAT_INTERNAL
    elif isinstance(contentObj, list) and len(contentObj) > 0:
      #News format
      inputFormat = self.FROMAT_NEWS
    else:
      #Unsupported format
      inputFormat = None

    return inputFormat

Here is the caller graph for this function:

getDefaultItem()

def app.ResponseExtractor.ResponseExtractor.getDefaultItem	(		self,
			contentObj,
			tags = None
	)

Make output content by substitution of the template's parts.

Parameters

contentObj	the object from response item in the default internal format
tags	- list of the tags names

Returns

the dict of tags and values

Definition at line 253 of file ResponseExtractor.py.

  def getDefaultItem(self, contentObj, tags=None):
    ret = {}

    if 'default' in contentObj and 'data' in contentObj['default'] and 'tagList' in contentObj['default']['data'] and\
     isinstance(contentObj['default']['data']['tagList'], list) and len(contentObj['default']['data']['tagList']) > 0:
      contentObj = contentObj['default']['data']['tagList'][0]
    else:
      raise Exception('Wrong format of the contentObj, structure checks not passed!')

    if self.extendedLog:
      self.logger.debug("Internal format item processing:\n%s", str(contentObj))

    #For each tag in the content object
    for tagItem in contentObj:
      tagName = tagItem['name']
      if tagName in tags:
        tagValue = self.getTagValueByName(tagName, contentObj, self.FROMAT_INTERNAL)
        ret[tagName] = tagValue

    return ret

Here is the call graph for this function:

Here is the caller graph for this function:

getNewsItem()

def app.ResponseExtractor.ResponseExtractor.getNewsItem	(		self,
			contentObj,
			tags = None
	)

Make output content by substitution of the template's parts.

Parameters

contentObj	the object from response item in News format
tags	- list of the tags names

Returns

the dict of tags and values

Definition at line 231 of file ResponseExtractor.py.

  def getNewsItem(self, contentObj, tags=None):
    ret = {}

    if self.extendedLog:
      self.logger.debug("News format item processing:\n%s", str(contentObj))

    if tags is None:
      ret = contentObj
    else:
      #For each tag in the content object
      for tagName in contentObj:
        if tagName in tags:
          ret[tagName] = self.getTagValueByName(tagName, contentObj, self.FROMAT_NEWS)

    return ret

Here is the call graph for this function:

Here is the caller graph for this function:

getTagValueByName()

def app.ResponseExtractor.ResponseExtractor.getTagValueByName	(		self,
			tagName,
			item,
			responseFormat
	)

Check is tag present in response item by the name.

Parameters

tagName	the name of the tag
item	- the one tags set item of the itemObject in the scraper response
responseFormat	- format of the scraper response

Returns

true if tag is present

Definition at line 281 of file ResponseExtractor.py.

  def getTagValueByName(self, tagName, item, responseFormat):
    ret = ''

    if responseFormat == self.FROMAT_NEWS:
      if tagName in item:
        ret = item[tagName].decode('string_escape')
      else:
        if self.extendedLog:
          self.logger.debug("Tag `%s` not found as News format, empty value assumed", tagName)
    elif responseFormat == self.FROMAT_INTERNAL:
      found = False
      for tag in item:
        if tag['name'] == tagName:
          found = True
          if len(tag['data']) > 0:
            ret = tag['data'][0].decode('string_escape')
          break
      if not found and self.extendedLog:
        self.logger.debug("Tag `%s` not found as internal format, empty value assumed", tagName)
    else:
      if self.extendedLog:
        self.logger.debug("Format %s not supported", str(responseFormat))

    return ret

Here is the call graph for this function:

Here is the caller graph for this function:

initLogger()

def app.ResponseExtractor.ResponseExtractor.initLogger	(		self,
			configName
	)

load log config file

Parameters

configName

- name of log rtc-finalizer config file

Returns

- None

Definition at line 136 of file ResponseExtractor.py.

  def initLogger(self, configName):
    try:
      if isinstance(configName, str) and len(configName) == 0:
        raise Exception(self.MSG_ERROR_EMPTY_CONFIG_FILE_NAME)

      logging.config.fileConfig(configName)

      #call rotation log files and initialization logger
      self.logger = Utils.MPLogger().getLogger()

    except Exception, err:
      raise Exception(self.MSG_ERROR_READ_LOG_CONFIG + ' ' + str(err))

Here is the call graph for this function:

Here is the caller graph for this function:

jsonLoadsSafe()

def app.ResponseExtractor.ResponseExtractor.jsonLoadsSafe	(		self,
			jsonString
	)

Parse json and return dict if okay or None if not.

Parameters

jsonString

json to pars

Returns

resulted dict

Definition at line 399 of file ResponseExtractor.py.

  def jsonLoadsSafe(self, jsonString):
    ret = None

    try:
      if jsonString is not None:
        ret = json.loads(jsonString)
    except Exception, err:
      self.logger.error("Error pars json: %s\n%s", str(err), jsonString)

    return ret

logResultedStatistics()

def app.ResponseExtractor.ResponseExtractor.logResultedStatistics	(		self,
			inputObject,
			items
	)

Detects the format of response object.

Parameters

inputObject
items	in results set

Definition at line 371 of file ResponseExtractor.py.

  def logResultedStatistics(self, inputObject, items):
    self.logger.debug("Items detected %s, output: %s", str(len(inputObject["itemsList"][0]["itemObject"])), str(items))

Here is the caller graph for this function:

parse()

def app.ResponseExtractor.ResponseExtractor.parse	(		self,
			inputObject,
			inputFormat,
			maxItems = -1,
			startFrom = 0,
			tags = None
	)

Parse the input json and make output collection.

Parameters

inputObject	json
inputFormat	of the inputObject
maxItems	- max items to process
startFrom	- processing start from item
tags	- list of tags names to get

Returns

the output buffer after all macro variables are substituted

Definition at line 315 of file ResponseExtractor.py.

  def parse(self, inputObject, inputFormat, maxItems=-1, startFrom=0, tags=None):
    ret = []

    i = 0
    s = 0

    #For all items in URLContent response accumulate doc and toc buffers
    for item in inputObject["itemsList"][0]["itemObject"]:
      if maxItems > -1 and i == maxItems:
        break

      if startFrom > 0 and s < startFrom:
        s += 1
        continue

      if len(item["processedContents"]) > 0:
        try:
          contentObj = json.loads(base64.b64decode(item["processedContents"][0]["buffer"]))
        except Exception, err:
          self.logger.error("Error get contentObj or cDate: %s, possible wrong json in buffer:\n%s", str(err),
                            str(item["processedContents"][0]["buffer"]))
          continue

        if inputFormat < 0:
          inputFormatLocal = self.detectFormat(contentObj)
          if inputFormatLocal is None:
            self.logger.info("Unsupported item object format or empty list:\n%s", str(contentObj))
            continue
        try:
          if inputFormatLocal == self.FROMAT_INTERNAL:
            #Process default internal object format
            item = self.getDefaultItem(contentObj, tags)
          elif inputFormatLocal == self.FROMAT_NEWS:
            #Process NEWS format of response
            item = self.getNewsItem(contentObj[0], tags)
          elif inputFormatLocal == self.FROMAT_RSS_FEED:
            #TODO: process RSS-FEED format of response
            continue
          ret.append(item)
          i += 1
        except (KeyboardInterrupt, SystemExit):
          raise
        except Exception, err:
          self.logger.error("Error process item: %s, contentObj:\n%s", str(err), str(contentObj))
          self.logger.debug("%s", Utils.getTracebackInfo())
          continue

    self.logResultedStatistics(inputObject, i)

    return ret

Here is the call graph for this function:

Here is the caller graph for this function:

process()

def app.ResponseExtractor.ResponseExtractor.process	(		self,
			inputBuffer,
			inputFormat = FROMAT_AUTO,
			maxItems = -1,
			startFrom = 0,
			tags = None
	)

process main operations

Parameters

inputBuffer	- te input buffer, supposes the json from DCC URL_CONTENT response
inputFormat	of the input buffer, including News, Template, RSS-Feed and so on
maxItems	- max items
startFrom	- start from item
tags	list of tags names

Returns

formatted string buffer

Definition at line 419 of file ResponseExtractor.py.

  def process(self, inputBuffer, inputFormat=FROMAT_AUTO, maxItems=-1, startFrom=0, tags=None):
    ret = ''

    try:
      self.logger.debug("Processing started, tags: %s", str(tags))

      inputObject = json.loads(inputBuffer)

      items = self.parse(inputObject, inputFormat, maxItems, startFrom, tags)

      #Make output string buffer
      if self.results == self.RESULTS_FORMAT_JSON:
        ret = json.dumps(items, indent=2, ensure_ascii=False)
      elif self.results == self.RESULTS_FORMAT_CSV_LINE:
        for item in items:
          buf = ''
          for tagName in item:
            buf += tagName + '=' + item[tagName] + ','
          if buf[:-1].strip() != '':
            ret += buf[:-1] + self.itemDelimiter.replace("\\n", "\n")
      elif self.results == self.RESULTS_FORMAT_FIELD_LINE:
        for item in items:
          buf = ''
          for tagName in item:
            buf += tagName + '=' + item[tagName] + "\n"
          if buf[:-1].strip() != '':
            ret += buf[:-1] + self.itemDelimiter.replace("\\n", "\n")
      else:
        pass

    except Exception, err:
      ExceptionLog.handler(self.logger, err, "Error:")
      raise Exception(self.MSG_ERROR_PROCESSING_REQUEST + ' ' + str(err))

    return ret

readFile()

def app.ResponseExtractor.ResponseExtractor.readFile	(		self,
			templateFile
	)

Read file.

Parameters

inFile

- name of file to read

Returns

- the buffer

Definition at line 154 of file ResponseExtractor.py.

  def readFile(self, templateFile):
    with open(templateFile, 'r') as f:
      ret = f.read()

    return ret

Here is the caller graph for this function:

run()

def app.ResponseExtractor.ResponseExtractor.run

(

self

)

run application

Definition at line 171 of file ResponseExtractor.py.

  def run(self):
    # call base class run method
    foundation.CementApp.run(self)

    startTime = time.time()

    if self.pargs.config:
      self.initLogger(self.__loadAppConfig(self.pargs.config))
    else:
      raise Exception(self.MSG_ERROR_LOAD_APP_CONFIG)

    if self.pargs.input:
      inputBuffer = self.readFile(self.pargs.input)
    else:
      inputBuffer = sys.stdin.read()

    if self.pargs.format:
      inputFormat = int(self.pargs.format)
    else:
      inputFormat = -1

    if self.pargs.maxitems:
      maxItems = int(self.pargs.maxitems)
    else:
      maxItems = -1
    if self.pargs.start:
      startFrom = int(self.pargs.start)
    else:
      startFrom = 0

    self.extendedLog = bool(int(self.pargs.extended))

    if self.pargs.tags:
      tags = self.pargs.tags.split(',')
    else:
      tags = None

    if self.pargs.results:
      self.results = int(self.pargs.results)

    # call processing
    outputBuffer = self.process(inputBuffer, inputFormat, maxItems, startFrom, tags)

    self.logger.info("Total time: %s", str(time.time() - startTime))

    if self.pargs.output:
      self.writeFile(self.pargs.output, outputBuffer)
    else:
      print outputBuffer
      sys.stdout.flush()

    # Finish logging
    self.logger.info(APP_CONSTS.LOGGER_DELIMITER_LINE)

Here is the call graph for this function:

setup()

def app.ResponseExtractor.ResponseExtractor.setup

(

self

)

setup application

Definition at line 80 of file ResponseExtractor.py.

  def setup(self):
    # call base class setup method
    foundation.CementApp.setup(self)
    self.args.add_argument('-c', '--config', action='store', metavar='config_file', help='config ini-file')
    self.args.add_argument('-i', '--input', action='store', metavar='input_json_file',
                           help='input json file of the URL_CONTENT response, if omitted the stdin read used')
    self.args.add_argument('-o', '--output', action='store', metavar='output_file, if omitted the stdout print used',
                           help='input file, if omitted the stdout write used')
    self.args.add_argument('-f', '--format', action='store', metavar='input_json_file_format',
                           help='input json file buffer format: -1 - auto (default if omitted) 0 - internal,' + \
                            ' 1 - news, 2 - rss-feed')
    self.args.add_argument('-m', '--maxitems', action='store', metavar='max_items',
                           help='max items number to read')
    self.args.add_argument('-s', '--start', action='store', metavar='start_from',
                           help='start from item')
    self.args.add_argument('-e', '--extended', action='store', metavar='extended',
                           help='extended log with additional debug information')
    self.args.add_argument('-t', '--tags', action='store', metavar='tags',
                           help='csv tags fields names list, all fields from response if omitted')
    self.args.add_argument('-r', '--results', action='store', metavar='results',
                           help='results format: 0 - csv fields names list one item per line, ' + \
                           '1 - fields list one field per line, 2 - json (default if omitted)')

writeFile()

def app.ResponseExtractor.ResponseExtractor.writeFile	(		self,
			fileName,
			outBuffer
	)

Write file.

Parameters

fileName	- name of file to write
outBuffer	- buffer to write

Definition at line 165 of file ResponseExtractor.py.

  def writeFile(self, fileName, outBuffer):
    with open(fileName, 'w') as f:
      f.write(outBuffer)

Here is the caller graph for this function:

Member Data Documentation

batch

app.ResponseExtractor.ResponseExtractor.batch

Definition at line 71 of file ResponseExtractor.py.

CONFIG_OPTION_ITEM_DELIMITER

string app.ResponseExtractor.ResponseExtractor.CONFIG_OPTION_ITEM_DELIMITER = "itemDelimiter"

static

Definition at line 43 of file ResponseExtractor.py.

CONFIG_OPTION_LOG

string app.ResponseExtractor.ResponseExtractor.CONFIG_OPTION_LOG = 'log'

static

Definition at line 55 of file ResponseExtractor.py.

exitCode

app.ResponseExtractor.ResponseExtractor.exitCode

Definition at line 72 of file ResponseExtractor.py.

extendedLog

app.ResponseExtractor.ResponseExtractor.extendedLog

Definition at line 75 of file ResponseExtractor.py.

FROMAT_AUTO

int app.ResponseExtractor.ResponseExtractor.FROMAT_AUTO = -1

static

Definition at line 45 of file ResponseExtractor.py.

FROMAT_INTERNAL

int app.ResponseExtractor.ResponseExtractor.FROMAT_INTERNAL = 0

static

Definition at line 46 of file ResponseExtractor.py.

FROMAT_NEWS

int app.ResponseExtractor.ResponseExtractor.FROMAT_NEWS = 1

static

Definition at line 47 of file ResponseExtractor.py.

FROMAT_RSS_FEED

int app.ResponseExtractor.ResponseExtractor.FROMAT_RSS_FEED = 2

static

Definition at line 48 of file ResponseExtractor.py.

initTagsLimitsConfig

app.ResponseExtractor.ResponseExtractor.initTagsLimitsConfig

Definition at line 74 of file ResponseExtractor.py.

initTagsUniqueHashConfig

app.ResponseExtractor.ResponseExtractor.initTagsUniqueHashConfig

Definition at line 73 of file ResponseExtractor.py.

itemDelimiter

app.ResponseExtractor.ResponseExtractor.itemDelimiter

Definition at line 77 of file ResponseExtractor.py.

logger

app.ResponseExtractor.ResponseExtractor.logger

Definition at line 70 of file ResponseExtractor.py.

MSG_ERROR_EMPTY_CONFIG_FILE_NAME

string app.ResponseExtractor.ResponseExtractor.MSG_ERROR_EMPTY_CONFIG_FILE_NAME = "Config file name is empty."

static

Definition at line 37 of file ResponseExtractor.py.

MSG_ERROR_LOAD_APP_CONFIG

string app.ResponseExtractor.ResponseExtractor.MSG_ERROR_LOAD_APP_CONFIG = "Error loading application config file."

static

Definition at line 39 of file ResponseExtractor.py.

MSG_ERROR_PARSE_CMD_PARAMS

string app.ResponseExtractor.ResponseExtractor.MSG_ERROR_PARSE_CMD_PARAMS = "Error parse command line parameters."

static

Constants error messages used in class.

Definition at line 36 of file ResponseExtractor.py.

MSG_ERROR_PROCESSING_REQUEST

string app.ResponseExtractor.ResponseExtractor.MSG_ERROR_PROCESSING_REQUEST = "Error processing input data."

static

Definition at line 41 of file ResponseExtractor.py.

MSG_ERROR_READ_LOG_CONFIG

string app.ResponseExtractor.ResponseExtractor.MSG_ERROR_READ_LOG_CONFIG = "Error read log config file."

static

Definition at line 40 of file ResponseExtractor.py.

MSG_ERROR_WRONG_CONFIG_FILE_NAME

string app.ResponseExtractor.ResponseExtractor.MSG_ERROR_WRONG_CONFIG_FILE_NAME = "Config file name is wrong"

static

Definition at line 38 of file ResponseExtractor.py.

results

app.ResponseExtractor.ResponseExtractor.results

Definition at line 76 of file ResponseExtractor.py.

RESULTS_FORMAT_CSV_LINE

int app.ResponseExtractor.ResponseExtractor.RESULTS_FORMAT_CSV_LINE = 0

static

Definition at line 50 of file ResponseExtractor.py.

RESULTS_FORMAT_FIELD_LINE

int app.ResponseExtractor.ResponseExtractor.RESULTS_FORMAT_FIELD_LINE = 1

static

Definition at line 51 of file ResponseExtractor.py.

RESULTS_FORMAT_JSON

int app.ResponseExtractor.ResponseExtractor.RESULTS_FORMAT_JSON = 2

static

Definition at line 52 of file ResponseExtractor.py.

---

The documentation for this class was generated from the following file:

• sources/hce/app/ResponseExtractor.py