app.ContentUpdater.ContentUpdater Class Reference

Inheritance diagram for app.ContentUpdater.ContentUpdater:

Collaboration diagram for app.ContentUpdater.ContentUpdater:

Classes
class	ConfigOptions
class	Meta

Public Member Functions
def	__init__ (self)
def	setup (self)
def	run (self)
def	getInputPickle (self)
def	unpickleInput (self, inputPickle)
def	createOutputPickle (self, outputBatch)
def	sendPickle (self, outputPickle)
def	process (self)
def	updateProcessedContents (self, inputBatch)
def	updateAttributesOnly (self, inputBatch)

Public Attributes
	exitCode
	logger
	dbWrapper
	errorMsg

Static Public Attributes
string	MSG_ERROR_EMPTY_CONFIG_FILE_NAME = "Config file name is empty."
string	MSG_ERROR_WRONG_CONFIG_FILE_NAME = "Config file name is wrong"
string	MSG_ERROR_LOAD_APP_CONFIG = "Error loading application config file."
string	MSG_ERROR_READ_LOG_CONFIG = "Error read log config file."
string	MSG_ERROR_MISSED_SECTION = "Missed mandatory section '%s'"
string	MSG_ERROR_DATABASE_OPERATION = "Database operation has error: %s"
string	MSG_ERROR_UPDATE_PROCESSED_CONTENTS = "Update processed contents has error: %s"
string	MSG_DEBUG_INPUT_PICKLE = "Input pickle: "
string	MSG_DEBUG_INPUT_UNPICKLED = "input unpickled: "
string	MSG_DEBUG_OUTPUT_BATCH = "Output batch: "
string	MSG_DEBUG_OUTPUT_PICKLE = "Output pickle: "
string	MSG_DEBUG_SEND_PICKLE = "Send pickle. Done."
string	ATTRIBUTE_ERROR_MESSAGE_NAME = 'errorMessage'

Private Member Functions
def	__initApp (self)
def	__loadAppConfig (self, configName)
def	__loadLogConfig (self, configName)
def	__createDBTasksWrapper (self, configName)

Detailed Description

Definition at line 40 of file ContentUpdater.py.

Constructor & Destructor Documentation

__init__()

def app.ContentUpdater.ContentUpdater.__init__

(

self

)

Definition at line 77 of file ContentUpdater.py.

  def __init__(self):
    # call base class __init__ method
    foundation.CementApp.__init__(self)
    self.exitCode = APP_CONSTS.EXIT_SUCCESS
    self.logger = None
    self.dbWrapper = None
    self.errorMsg = None

Member Function Documentation

__createDBTasksWrapper()

def app.ContentUpdater.ContentUpdater.__createDBTasksWrapper (   self,   configName  )

private

Definition at line 182 of file ContentUpdater.py.

  def __createDBTasksWrapper(self, configName):
    # variable for result
    dbTasksWrapper = None
    try:
      if configName == "":
        raise Exception(self.MSG_ERROR_EMPTY_CONFIG_FILE_NAME)

      config = ConfigParser.ConfigParser()
      config.optionxform = str

      readOk = config.read(configName)

      if len(readOk) == 0:
        raise Exception(self.MSG_ERROR_WRONG_CONFIG_FILE_NAME + ": " + configName)

      dbTasksWrapper = DBTasksWrapper(config)

    except Exception, err:
      raise Exception(self.MSG_ERROR_LOAD_APP_CONFIG + ' ' + str(err))

    return dbTasksWrapper

Here is the caller graph for this function:

__initApp()

def app.ContentUpdater.ContentUpdater.__initApp (   self )

private

Definition at line 111 of file ContentUpdater.py.

  def __initApp(self):
    if self.pargs.config:
      # load app config
      configOptions = self.__loadAppConfig(self.pargs.config)

      # load log config
      self.__loadLogConfig(configOptions.confLogFileName)

      # set attribute values of application
      self.dbWrapper = self.__createDBTasksWrapper(configOptions.dbTaskIniFile)

    else:
      raise Exception(self.MSG_ERROR_LOAD_APP_CONFIG)

    if self.pargs.error:
      self.errorMsg = str(self.pargs.error)

Here is the call graph for this function:

Here is the caller graph for this function:

__loadAppConfig()

def app.ContentUpdater.ContentUpdater.__loadAppConfig (   self,   configName  )

private

Definition at line 133 of file ContentUpdater.py.

  def __loadAppConfig(self, configName):
    # variable for result
    configOptions = None
    try:
      config = ConfigParser.ConfigParser()
      config.optionxform = str

      readOk = config.read(configName)

      if len(readOk) == 0:
        raise Exception(self.MSG_ERROR_WRONG_CONFIG_FILE_NAME + ": " + configName)

      if not config.has_section(APP_CONSTS.CONFIG_APPLICATION_SECTION_NAME):
        raise Exception(self.MSG_ERROR_MISSED_SECTION % str(APP_CONSTS.CONFIG_APPLICATION_SECTION_NAME))

      configOptions = ContentUpdater.ConfigOptions(
          str(config.get(APP_CONSTS.CONFIG_APPLICATION_SECTION_NAME,
                         ContentUpdater.ConfigOptions.CONTENT_UPDATER_OPTION_LOG)),
          str(config.get(APP_CONSTS.CONFIG_APPLICATION_SECTION_NAME,
                         ContentUpdater.ConfigOptions.CONTENT_UPDATER_OPTION_DB_TASK_INI)))

    except Exception, err:
      raise Exception(self.MSG_ERROR_LOAD_APP_CONFIG + ' ' + str(err))

    return configOptions

Here is the caller graph for this function:

__loadLogConfig()

def app.ContentUpdater.ContentUpdater.__loadLogConfig (   self,   configName  )

private

Definition at line 164 of file ContentUpdater.py.

  def __loadLogConfig(self, configName):
    try:
      if isinstance(configName, str) and len(configName) == 0:
        raise Exception(self.MSG_ERROR_EMPTY_CONFIG_FILE_NAME)

      logging.config.fileConfig(configName)

      # call rotation log files and initialization logger
      self.logger = Utils.MPLogger().getLogger()

    except Exception, err:
      raise Exception(self.MSG_ERROR_READ_LOG_CONFIG + ' ' + str(err))

Here is the call graph for this function:

Here is the caller graph for this function:

createOutputPickle()

def app.ContentUpdater.ContentUpdater.createOutputPickle	(		self,
			outputBatch
	)

Definition at line 230 of file ContentUpdater.py.

  def createOutputPickle(self, outputBatch):
    # self.logger.debug(self.MSG_DEBUG_OUTPUT_BATCH + varDump(outputBatch))
    outputPickle = pickle.dumps(outputBatch)
    # self.logger.debug(self.MSG_DEBUG_OUTPUT_PICKLE + str(outputPickle))

    return outputPickle

Here is the caller graph for this function:

getInputPickle()

def app.ContentUpdater.ContentUpdater.getInputPickle

(

self

)

Definition at line 209 of file ContentUpdater.py.

  def getInputPickle(self):
    inputPickle = sys.stdin.read()
    # self.logger.debug(self.MSG_DEBUG_INPUT_PICKLE + '\n' + str(inputPickle))

    return inputPickle

Here is the caller graph for this function:

process()

def app.ContentUpdater.ContentUpdater.process

(

self

)

Definition at line 251 of file ContentUpdater.py.

  def process(self):
    try:
      inputBatchObj = self.unpickleInput(self.getInputPickle())

      if self.errorMsg is None:
        self.updateProcessedContents(inputBatchObj)
      else:
        self.updateAttributesOnly(inputBatchObj)

      self.sendPickle(self.createOutputPickle(inputBatchObj))
    except Exception, err:
      self.logger.error(str(err))
      self.exitCode = APP_CONSTS.EXIT_FAILURE

Here is the call graph for this function:

Here is the caller graph for this function:

run()

def app.ContentUpdater.ContentUpdater.run

(

self

)

Definition at line 93 of file ContentUpdater.py.

  def run(self):
    # call base class run method
    foundation.CementApp.run(self)

    # call initialization application
    self.__initApp()

    # call internal processing
    self.process()

    # Finish logging
    self.logger.info(APP_CONSTS.LOGGER_DELIMITER_LINE)

Here is the call graph for this function:

sendPickle()

def app.ContentUpdater.ContentUpdater.sendPickle	(		self,
			outputPickle
	)

Definition at line 242 of file ContentUpdater.py.

  def sendPickle(self, outputPickle):
    sys.stdout.write(outputPickle)
    self.logger.debug(self.MSG_DEBUG_SEND_PICKLE)

Here is the caller graph for this function:

setup()

def app.ContentUpdater.ContentUpdater.setup

(

self

)

Definition at line 87 of file ContentUpdater.py.

  def setup(self):
    # call base class setup method
    foundation.CementApp.setup(self)

unpickleInput()

def app.ContentUpdater.ContentUpdater.unpickleInput	(		self,
			inputPickle
	)

Definition at line 219 of file ContentUpdater.py.

  def unpickleInput(self, inputPickle):
    inputUnpickled = pickle.loads(inputPickle)
    # self.logger.debug(self.MSG_DEBUG_INPUT_UNPICKLED + varDump(inputUnpickled))

    return inputUnpickled

Here is the caller graph for this function:

updateAttributesOnly()

def app.ContentUpdater.ContentUpdater.updateAttributesOnly	(		self,
			inputBatch
	)

Definition at line 357 of file ContentUpdater.py.

  def updateAttributesOnly(self, inputBatch):
    attributes = []

    for batchItem in inputBatch.items:
      self.logger.debug("batchItem: %s", varDump(batchItem))
      self.logger.debug("batchItem.urlContentResponse: %s", varDump(batchItem.urlContentResponse))

      try:
        # accumulate attributes
        attributes.append(Attribute(siteId=batchItem.siteId,
                                    name=self.ATTRIBUTE_ERROR_MESSAGE_NAME,
                                    urlMd5=batchItem.urlId,
                                    value=self.dbWrapper.dbTask.dbConnections[DB_CONSTS.PRIMARY_DB_ID].\
                                      escape_string(str(self.errorMsg))))

        self.logger.debug("Made attributes: %s", varDump(attributes))
      except Exception, err:
        self.logger.error("Make attributes error: %s", str(err))
        self.logger.debug(getTracebackInfo())

    try:
      # execute database operations
      affectDB = self.dbWrapper.affect_db
      self.dbWrapper.affect_db = True
      self.dbWrapper.putAttributes(attributes)
      self.dbWrapper.affect_db = affectDB

      self.logger.debug('Database operations executed...')

    except DatabaseException, err:
      self.logger.error(self.MSG_ERROR_DATABASE_OPERATION, str(err))
      self.logger.debug(getTracebackInfo())

updateProcessedContents()

def app.ContentUpdater.ContentUpdater.updateProcessedContents	(		self,
			inputBatch
	)

Definition at line 270 of file ContentUpdater.py.

  def updateProcessedContents(self, inputBatch):

    urlPuts = []
    attributes = []

    self.logger.debug("The processing of batch Id = %s started", str(inputBatch.id))

    for batchItem in inputBatch.items:
#       self.logger.debug("batchItem: %s", varDump(batchItem))
#       self.logger.debug("batchItem.urlContentResponse: %s", varDump(batchItem.urlContentResponse))

      if batchItem.urlContentResponse is not None:
        for processedContent in batchItem.urlContentResponse.processedContents:

          self.logger.debug("!!! processedContent: %s", varDump(processedContent, stringifyType=0))
          try:
            # create URLPut object
            putDict = {}
            putDict["id"] = batchItem.urlId
            putDict["data"] = processedContent
            putDict["cDate"] = SQLExpression("NOW()")

            urlPut = dc_event.URLPut(batchItem.siteId,
                                     batchItem.urlId,
                                     dc_event.Content.CONTENT_PROCESSOR_CONTENT,
                                     putDict)

            # accumulate URLPut objects
            urlPuts.append(urlPut)

          except Exception, err:
            self.logger.error(self.MSG_ERROR_UPDATE_PROCESSED_CONTENTS, str(err))
            self.logger.debug(getTracebackInfo())

        try:
#           self.logger.debug("type: %s, batchItem.urlContentResponse.attributes: %s",
#                             str(type(batchItem.urlContentResponse.attributes)),
#                             varDump(batchItem.urlContentResponse.attributes,
#                                     maxDepth=15))


          # accumulate attributes
          for attrJson in batchItem.urlContentResponse.attributes:
            attrDict = json.loads(attrJson)

#             self.logger.debug("type: %s, attrDict: %s", str(type(attrDict)), str(attrDict))

            attrValue = json.dumps(attrDict['value'], ensure_ascii=False, encoding='utf-8')

            attribute = Attribute(siteId=attrDict['siteId'],
                                  name=attrDict['name'],
                                  urlMd5=attrDict['urlMd5'],
                                  value=attrValue)

            attributes.append(attribute)

          if len(attributes) > 0:
            self.logger.debug("Made attributes: %s", varDump(attributes))

        except Exception, err:
          self.logger.error("Make attributes error: %s", str(err))
          self.logger.debug(getTracebackInfo())

    try:
      # execute database operations
      affectDB = self.dbWrapper.affect_db
      self.dbWrapper.affect_db = True
      self.dbWrapper.putURLContent(urlPuts)
      self.dbWrapper.putAttributes(attributes)
      self.dbWrapper.affect_db = affectDB

      self.logger.debug('Database operations executed...')

    except DatabaseException, err:
      self.logger.error(self.MSG_ERROR_DATABASE_OPERATION, str(err))
      self.logger.debug(getTracebackInfo())
    except Exception, err:
      self.logger.error(self.MSG_ERROR_DATABASE_OPERATION, str(err))
      self.logger.debug(getTracebackInfo())
      
    self.logger.debug("The processing of batch Id = %s finished", str(inputBatch.id))

Here is the call graph for this function:

Here is the caller graph for this function:

Member Data Documentation

ATTRIBUTE_ERROR_MESSAGE_NAME

string app.ContentUpdater.ContentUpdater.ATTRIBUTE_ERROR_MESSAGE_NAME = 'errorMessage'

static

Definition at line 56 of file ContentUpdater.py.

dbWrapper

app.ContentUpdater.ContentUpdater.dbWrapper

Definition at line 82 of file ContentUpdater.py.

errorMsg

app.ContentUpdater.ContentUpdater.errorMsg

Definition at line 83 of file ContentUpdater.py.

exitCode

app.ContentUpdater.ContentUpdater.exitCode

Definition at line 80 of file ContentUpdater.py.

logger

app.ContentUpdater.ContentUpdater.logger

Definition at line 81 of file ContentUpdater.py.

MSG_DEBUG_INPUT_PICKLE

string app.ContentUpdater.ContentUpdater.MSG_DEBUG_INPUT_PICKLE = "Input pickle: "

static

Definition at line 50 of file ContentUpdater.py.

MSG_DEBUG_INPUT_UNPICKLED

string app.ContentUpdater.ContentUpdater.MSG_DEBUG_INPUT_UNPICKLED = "input unpickled: "

static

Definition at line 51 of file ContentUpdater.py.

MSG_DEBUG_OUTPUT_BATCH

string app.ContentUpdater.ContentUpdater.MSG_DEBUG_OUTPUT_BATCH = "Output batch: "

static

Definition at line 52 of file ContentUpdater.py.

MSG_DEBUG_OUTPUT_PICKLE

string app.ContentUpdater.ContentUpdater.MSG_DEBUG_OUTPUT_PICKLE = "Output pickle: "

static

Definition at line 53 of file ContentUpdater.py.

MSG_DEBUG_SEND_PICKLE

string app.ContentUpdater.ContentUpdater.MSG_DEBUG_SEND_PICKLE = "Send pickle. Done."

static

Definition at line 54 of file ContentUpdater.py.

MSG_ERROR_DATABASE_OPERATION

string app.ContentUpdater.ContentUpdater.MSG_ERROR_DATABASE_OPERATION = "Database operation has error: %s"

static

Definition at line 47 of file ContentUpdater.py.

MSG_ERROR_EMPTY_CONFIG_FILE_NAME

string app.ContentUpdater.ContentUpdater.MSG_ERROR_EMPTY_CONFIG_FILE_NAME = "Config file name is empty."

static

Definition at line 42 of file ContentUpdater.py.

MSG_ERROR_LOAD_APP_CONFIG

string app.ContentUpdater.ContentUpdater.MSG_ERROR_LOAD_APP_CONFIG = "Error loading application config file."

static

Definition at line 44 of file ContentUpdater.py.

MSG_ERROR_MISSED_SECTION

string app.ContentUpdater.ContentUpdater.MSG_ERROR_MISSED_SECTION = "Missed mandatory section '%s'"

static

Definition at line 46 of file ContentUpdater.py.

MSG_ERROR_READ_LOG_CONFIG

string app.ContentUpdater.ContentUpdater.MSG_ERROR_READ_LOG_CONFIG = "Error read log config file."

static

Definition at line 45 of file ContentUpdater.py.

MSG_ERROR_UPDATE_PROCESSED_CONTENTS

string app.ContentUpdater.ContentUpdater.MSG_ERROR_UPDATE_PROCESSED_CONTENTS = "Update processed contents has error: %s"

static

Definition at line 48 of file ContentUpdater.py.

MSG_ERROR_WRONG_CONFIG_FILE_NAME

string app.ContentUpdater.ContentUpdater.MSG_ERROR_WRONG_CONFIG_FILE_NAME = "Config file name is wrong"

static

Definition at line 43 of file ContentUpdater.py.

---

The documentation for this class was generated from the following file:

• sources/hce/app/ContentUpdater.py