RTCPreprocessor Class content main functional for preprocessor for realtime crawling, class inherits from foundation.CementApp. More...

Inheritance diagram for dc_crawler.RTCPreprocessor.RTCPreprocessor:

[legend]

Collaboration diagram for dc_crawler.RTCPreprocessor.RTCPreprocessor:

[legend]

Classes
class	Meta

Public Member Functions
def	__init__ (self)
	constructor More...

def	setup (self)
	setup application More...

def	run (self)
	run application More...

def	getBatchFromInput (self)

def	cutBatch (self)

def	split (self, arr, count)

def	sendBatch (self)

def	getEnvVars (self)

def	process (self)

Public Attributes
	logger

	batch

	exitCode

	pickled_object

	envVars

Static Public Attributes
string	MSG_ERROR_PARSE_CMD_PARAMS = "Error parse command line parameters."
	Constants error messages used in class. More...

string	MSG_ERROR_EMPTY_CONFIG_FILE_NAME = "Config file name is empty."

string	MSG_ERROR_WRONG_CONFIG_FILE_NAME = "Config file name is wrong"

string	MSG_ERROR_LOAD_APP_CONFIG = "Error loading application config file."

string	MSG_ERROR_READ_LOG_CONFIG = "Error read log config file."

string	DRCE_NODE_NUMBER = "DRCE_NODE_NUMBER"
	Constans used in class. More...

string	DRCE_NODES_TOTAL = "DRCE_NODES_TOTAL"

int	ERROR_EMPTY_ENV_VARS = 2
	Constans as numeric for exit code. More...

string	PREPROCESSOR_OPTION_LOG = "log"
	Constans used options from config file. More...

Private Member Functions
def	__initApp (self)
	initialize application from config files More...

def	__loadAppConfig (self, configName)
	load application config file More...

def	__loadLogConfig (self, configName)
	load log config file More...

Detailed Description

RTCPreprocessor Class content main functional for preprocessor for realtime crawling, class inherits from foundation.CementApp.

Definition at line 34 of file RTCPreprocessor.py.

Constructor & Destructor Documentation

◆ init()

def dc_crawler.RTCPreprocessor.RTCPreprocessor.__init__ ( self )

constructor

Definition at line 60 of file RTCPreprocessor.py.

   def __init__(self):
     # call base class __init__ method
     foundation.CementApp.__init__(self)
 
     self.logger = None
     self.batch = None
     self.exitCode = APP_CONSTS.EXIT_SUCCESS
     self.pickled_object = None
     self.envVars = {self.DRCE_NODES_TOTAL: 1,
                     self.DRCE_NODE_NUMBER: 1}
 
 

Member Function Documentation

◆ __initApp()

def dc_crawler.RTCPreprocessor.RTCPreprocessor.__initApp ( self )

private

initialize application from config files

Parameters

- None

Returns: - None

Definition at line 97 of file RTCPreprocessor.py.

   def __initApp(self):
     if self.pargs.config:
       self.__loadLogConfig(self.__loadAppConfig(self.pargs.config))
     else:
       raise Exception(self.MSG_ERROR_LOAD_APP_CONFIG)
 
 

Here is the call graph for this function:

Here is the caller graph for this function:

◆ __loadAppConfig()

def dc_crawler.RTCPreprocessor.RTCPreprocessor.__loadAppConfig	(	self,
		configName
	)

private

load application config file

Parameters

configName - name of application config file

Returns: - log config file name

Definition at line 108 of file RTCPreprocessor.py.

   def __loadAppConfig(self, configName):
     #variable for result
     confLogFileName = ""
 
     try:
       config = ConfigParser.ConfigParser()
       config.optionxform = str
 
       readOk = config.read(configName)
 
       if len(readOk) == 0:
         raise Exception(self.MSG_ERROR_WRONG_CONFIG_FILE_NAME + ": " + configName)
 
       if config.has_section(APP_CONSTS.CONFIG_APPLICATION_SECTION_NAME):
         confLogFileName = str(config.get(APP_CONSTS.CONFIG_APPLICATION_SECTION_NAME, self.PREPROCESSOR_OPTION_LOG))
 
     except Exception, err:
       raise Exception(self.MSG_ERROR_LOAD_APP_CONFIG + ' ' + str(err))
 
     return confLogFileName
 
 

Here is the caller graph for this function:

◆ __loadLogConfig()

def dc_crawler.RTCPreprocessor.RTCPreprocessor.__loadLogConfig	(	self,
		configName
	)

private

load log config file

Parameters

configName - name of log rtc-finalizer config file

Returns: - None

Definition at line 134 of file RTCPreprocessor.py.

   def __loadLogConfig(self, configName):
     try:
       if isinstance(configName, str) and len(configName) == 0:
         raise Exception(self.MSG_ERROR_EMPTY_CONFIG_FILE_NAME)
 
       logging.config.fileConfig(configName)
 
       #call rotation log files and initialization logger
       self.logger = Utils.MPLogger().getLogger()
 
     except Exception, err:
       raise Exception(self.MSG_ERROR_READ_LOG_CONFIG + ' ' + str(err))
 
 

Here is the call graph for this function:

Here is the caller graph for this function:

◆ cutBatch()

def dc_crawler.RTCPreprocessor.RTCPreprocessor.cutBatch ( self )

Definition at line 152 of file RTCPreprocessor.py.

   def cutBatch(self):
     self.batch = (pickle.loads(self.pickled_object))
     self.logger.info("Before id:%s items: %s", str(self.batch.id), str(len(self.batch.items)))
     self.logger.debug("self.batch: %s", varDump(self.batch))
     items = self.batch.items
     if len(items) > 1:
       splitted_items = self.split(self.batch.items, int(self.envVars[self.DRCE_NODES_TOTAL]))
       self.logger.debug("Input items: %s", str(self.batch.items))
       self.logger.debug("Splitted items: %s", str(splitted_items))
       self.batch.items = splitted_items[int(self.envVars[self.DRCE_NODE_NUMBER]) - 1]
       self.logger.debug("Output items: %s", str(self.batch.items))
     self.logger.debug("Output batch: %s", varDump(self.batch))
     self.pickled_object = pickle.dumps(self.batch)
 
     self.logger.info("After id:%s items: %s", str(self.batch.id), str(len(self.batch.items)))
 

Here is the call graph for this function:

Here is the caller graph for this function:

◆ getBatchFromInput()

def dc_crawler.RTCPreprocessor.RTCPreprocessor.getBatchFromInput ( self )

Definition at line 148 of file RTCPreprocessor.py.

   def getBatchFromInput(self):
     self.pickled_object = sys.stdin.read()

Here is the caller graph for this function:

◆ getEnvVars()

def dc_crawler.RTCPreprocessor.RTCPreprocessor.getEnvVars ( self )

Definition at line 177 of file RTCPreprocessor.py.

   def getEnvVars(self):
     for key in self.envVars.keys():
       if key in os.environ and os.environ[key] != "":
         self.envVars[key] = os.environ[key]
         self.logger.debug("os.environ[%s]: set to <<%s>>" % (key, self.envVars[key]))
       else:
         self.logger.debug("os.environ[%s]: not set. Use default value: <<%s>>" % (key, self.envVars[key]))
         self.exitCode = self.ERROR_EMPTY_ENV_VARS
 
 

Here is the caller graph for this function:

◆ process()

def dc_crawler.RTCPreprocessor.RTCPreprocessor.process ( self )

Definition at line 187 of file RTCPreprocessor.py.

187 def process(self):

188 try:

189 self.getBatchFromInput()

190 self.getEnvVars()

191 if self.exitCode != self.ERROR_EMPTY_ENV_VARS:

192 self.logger.info("The batch possible will be reduced")

193 self.cutBatch()

194 else:

195 self.logger.info("The batch will not be reduced")

196 self.sendBatch()

197 except Exception:

198 self.exitCode = APP_CONSTS.EXIT_FAILURE

199

200

ftests.ftest_popen.process

process

Definition: ftest_popen.py:37

Here is the call graph for this function:

Here is the caller graph for this function:

◆ run()

def dc_crawler.RTCPreprocessor.RTCPreprocessor.run ( self )

run application

Definition at line 79 of file RTCPreprocessor.py.

   def run(self):
     # call base class run method
     foundation.CementApp.run(self)
 
     # call initialization application
     self.__initApp()
 
     # call internal processing
     self.process()
 
     # Finish logging
     self.logger.info(APP_CONSTS.LOGGER_DELIMITER_LINE)
 
 

Here is the call graph for this function:

◆ sendBatch()

def dc_crawler.RTCPreprocessor.RTCPreprocessor.sendBatch ( self )

Definition at line 172 of file RTCPreprocessor.py.

   def sendBatch(self):
     print self.pickled_object
     sys.stdout.flush()
 
 

Here is the caller graph for this function:

◆ setup()

def dc_crawler.RTCPreprocessor.RTCPreprocessor.setup ( self )

setup application

Definition at line 73 of file RTCPreprocessor.py.

   def setup(self):
     # call base class setup method
     foundation.CementApp.setup(self)
 
 

◆ split()

def dc_crawler.RTCPreprocessor.RTCPreprocessor.split	(	self,
		arr,
		count
	)

Definition at line 168 of file RTCPreprocessor.py.

   def split(self, arr, count):
     return [arr[i::count] for i in range(count)]

Here is the caller graph for this function:

Member Data Documentation

◆ batch

dc_crawler.RTCPreprocessor.RTCPreprocessor.batch

Definition at line 65 of file RTCPreprocessor.py.

◆ DRCE_NODE_NUMBER

string dc_crawler.RTCPreprocessor.RTCPreprocessor.DRCE_NODE_NUMBER = "DRCE_NODE_NUMBER"

static

Constans used in class.

Definition at line 44 of file RTCPreprocessor.py.

◆ DRCE_NODES_TOTAL

string dc_crawler.RTCPreprocessor.RTCPreprocessor.DRCE_NODES_TOTAL = "DRCE_NODES_TOTAL"

static

Definition at line 45 of file RTCPreprocessor.py.

◆ envVars

dc_crawler.RTCPreprocessor.RTCPreprocessor.envVars

Definition at line 68 of file RTCPreprocessor.py.

◆ ERROR_EMPTY_ENV_VARS

int dc_crawler.RTCPreprocessor.RTCPreprocessor.ERROR_EMPTY_ENV_VARS = 2

static

Constans as numeric for exit code.

Definition at line 48 of file RTCPreprocessor.py.

◆ exitCode

dc_crawler.RTCPreprocessor.RTCPreprocessor.exitCode

Definition at line 66 of file RTCPreprocessor.py.

◆ logger

dc_crawler.RTCPreprocessor.RTCPreprocessor.logger

Definition at line 64 of file RTCPreprocessor.py.

◆ MSG_ERROR_EMPTY_CONFIG_FILE_NAME

string dc_crawler.RTCPreprocessor.RTCPreprocessor.MSG_ERROR_EMPTY_CONFIG_FILE_NAME = "Config file name is empty."

static

Definition at line 38 of file RTCPreprocessor.py.

◆ MSG_ERROR_LOAD_APP_CONFIG

string dc_crawler.RTCPreprocessor.RTCPreprocessor.MSG_ERROR_LOAD_APP_CONFIG = "Error loading application config file."

static

Definition at line 40 of file RTCPreprocessor.py.

◆ MSG_ERROR_PARSE_CMD_PARAMS

string dc_crawler.RTCPreprocessor.RTCPreprocessor.MSG_ERROR_PARSE_CMD_PARAMS = "Error parse command line parameters."

static

Constants error messages used in class.

Definition at line 37 of file RTCPreprocessor.py.

◆ MSG_ERROR_READ_LOG_CONFIG

string dc_crawler.RTCPreprocessor.RTCPreprocessor.MSG_ERROR_READ_LOG_CONFIG = "Error read log config file."

static

Definition at line 41 of file RTCPreprocessor.py.

◆ MSG_ERROR_WRONG_CONFIG_FILE_NAME

string dc_crawler.RTCPreprocessor.RTCPreprocessor.MSG_ERROR_WRONG_CONFIG_FILE_NAME = "Config file name is wrong"

static

Definition at line 39 of file RTCPreprocessor.py.

◆ pickled_object

dc_crawler.RTCPreprocessor.RTCPreprocessor.pickled_object

Definition at line 67 of file RTCPreprocessor.py.

◆ PREPROCESSOR_OPTION_LOG

string dc_crawler.RTCPreprocessor.RTCPreprocessor.PREPROCESSOR_OPTION_LOG = "log"

static

Constans used options from config file.

Definition at line 51 of file RTCPreprocessor.py.

The documentation for this class was generated from the following file:

sources/hce/dc_crawler/RTCPreprocessor.py

Classes

Public Member Functions

Public Attributes

Static Public Attributes

Private Member Functions

Detailed Description

Constructor & Destructor Documentation

◆ __init__()

Member Function Documentation

◆ __initApp()

◆ __loadAppConfig()

◆ __loadLogConfig()

◆ cutBatch()

◆ getBatchFromInput()

◆ getEnvVars()

◆ process()

◆ run()

◆ sendBatch()

◆ setup()

◆ split()

Member Data Documentation

◆ batch

◆ DRCE_NODE_NUMBER

◆ DRCE_NODES_TOTAL

◆ envVars

◆ ERROR_EMPTY_ENV_VARS

◆ exitCode

◆ logger

◆ MSG_ERROR_EMPTY_CONFIG_FILE_NAME

◆ MSG_ERROR_LOAD_APP_CONFIG

◆ MSG_ERROR_PARSE_CMD_PARAMS

◆ MSG_ERROR_READ_LOG_CONFIG

◆ MSG_ERROR_WRONG_CONFIG_FILE_NAME

◆ pickled_object

◆ PREPROCESSOR_OPTION_LOG

◆ init()