dc.BatchTasksManagerProcess.BatchTasksManagerProcess Class Reference

Inheritance diagram for dc.BatchTasksManagerProcess.BatchTasksManagerProcess:

Collaboration diagram for dc.BatchTasksManagerProcess.BatchTasksManagerProcess:

Public Member Functions
def	__init__ (self, configParser, connectionBuilderLight=None)
def	on_poll_timeout (self)
def	setProcessBatch (self)
def	sendBatchTaskToDTM (self, batch)
def	getDTMTaskState (self, taskId)
def	processDTMTasksQueue (self)
def	isDTMTaskDead (self, state)
def	processFinishedBatch (self, taskBatch)
def	dtmdRequestExecute (self, requestEvent, timeout, maxTries=100)
def	sendURLUpdate (self, batchItemsList, batchId, batchState)
def	sendURLDelete (self, batchItemsList, batchId)
def	onURLUpdateResponse (self, event)
def	onURLDeleteResponse (self, event)
Public Member Functions inherited from app.BaseServerManager.BaseServerManager
def	__init__ (self, poller_manager=None, admin_connection=None, conectionLightBuilder=None, exceptionForward=False, dumpStatVars=True)
	constructor More...
def	addConnection (self, name, connection)
def	setEventHandler (self, eventType, handler)
	set event handler rewrite the current handler for eventType More...
def	send (self, connect_name, event)
	send event More...
def	reply (self, event, reply_event)
	wrapper for sending event in reply for event More...
def	poll (self)
	poll function polling connections receive as multipart msg, the second argument is pickled pyobj More...
def	process (self, event)
	process event call the event handler method that was set by user or on_unhandled_event method if not set More...
def	run (self)
def	is_connection_registered (self, name)
	check is a connection was registered in a instance of BaseServerManager i object More...
def	on_poll_timeout (self)
	function will call every time when ConnectionTimeout exception arrive More...
def	on_unhandled_event (self, event)
	function will call every time when arrive doesn't set handler for event type of event.evenType More...
def	build_poller_list (self)
def	clear_poller (self)
def	onAdminState (self, event)
	onAdminState event handler process admin SHUTDOWN command More...
def	onAdminFetchStatData (self, event)
	onAdminState event handler process admin command More...
def	onAdminSuspend (self, event)
	onAdminState event handler process admin command More...
def	getStatDataFields (self, fields)
	getStatDataFields returns stat data from storage More...
def	getSystemStat (self)
	getSystemStat returns stat data for system indicators: RAMV, RAMR and CPU More...
def	getConfigVarsFields (self, fields)
	getConfigVarsFields returns config vars from storage More...
def	onAdminGetConfigVars (self, event)
	onAdminGetConfigVars event handler process getConfigVars admin command, fill and return config vars array from internal storage More...
def	onAdminSetConfigVars (self, event)
	onAdminSetConfigVars event handler process setConfigVars admin command More...
def	setConfigVars (self, setConfigVars)
	processSetConfigVars sets config vars in storage More...
def	sendAdminReadyEvent (self)
	send ready event to notify adminInterfaceService More...
def	createLogMsg (self, event)
	from string message from event object More...
def	initStatFields (self, connect_name)
	add record in statFields More...
def	updateStatField (self, field_name, value, operation=STAT_FIELDS_OPERATION_ADD)
	update values of stat field - default sum More...
def	processSpecialConfigVars (self, name, value)
	send ready event to notify adminInterfaceService More...
def	getLogLevel (self)
	Get log level from first of existing loggers. More...
def	setLogLevel (self, level)
	Set log level for all loggers. More...
def	saveStatVarsDump (self)
	Save stat vars in json file. More...
def	loadStatVarsDump (self)
	Load stat vars in json file. More...
def	getStatVarsDumpFileName (self)
	Get stat vars file name. More...
def	createDBIDict (self, configParser)

Public Attributes
	serverName
	clientSitesManagerName
	dtmdConnection
	dtmTasksQueue
	processProcessingLastTs
	processBatchQueuelLastTs
Public Attributes inherited from app.BaseServerManager.BaseServerManager
	dumpStatVars
	poller_manager
	eventBuilder
	exit_flag
	pollTimeout
	connections
	event_handlers
	statFields
	stat fields container More...
	configVars
	exceptionForward

Static Public Attributes
int	DTM_TASK_CHECK_STATE_METHOD_STATUS = 0
int	DTM_TASK_CHECK_STATE_METHOD_STATE = 1
string	CONFIG_SERVER = "server"
string	CONFIG_DTMD_HOST = "DTMDHost"
string	CONFIG_DTMD_PORT = "DTMDPort"
string	CONFIG_DTMD_TIMEOUT = "DTMDTimeout"
string	CONFIG_POLLING_TIMEOUT = "PollingTimeout"
string	CONFIG_SITES_MANAGER_CLIENT = "clientSitesManager"
string	CONFIG_DRCE_PROCESSOR_APP_NAME = "DRCEProcessorAppName"
string	CONFIG_DRCE_DB_APP_NAME = "DRCEDBAppName"
string	CONFIG_PROCESS_PERIOD = "ProcessingPeriod"
string	CONFIG_PROCESS_MODE = "ProcessingMode"
string	CONFIG_BATCH_DEFAULT_MAX_TIME = "BatchDefaultMaxExecutionTime"
string	CONFIG_BATCH_MAX_URLS = "BatchDefaultMaxURLs"
string	CONFIG_BATCH_ORDER_BY_URLS = "BatchDefaultOrderByURLs"
string	CONFIG_BATCH_MAX_TASKS = "BatchDefaultMaxTasks"
string	CONFIG_BATCH_QUEUE_PERIOD = "BatchQueueProcessingPeriod"
string	CONFIG_BATCH_QUEUE_TASK_TTL = "BatchQueueTaskTTL"
string	CONFIG_BATCH_QUEUE_TASK_CHECK_METHOD = "BatchQueueTaskCheckStateMethod"
string	CONFIG_BATCH_DEFAULT_STARTER = "BatchTask_STARTER"
string	CONFIG_BATCH_WHERE_URLS = "BatchDefaultWhereURLs"
string	CONFIG_BATCH_WHERE_SITES = "BatchDefaultWhereSites"
string	CONFIG_BATCH_MAX_TIME = "BatchMaxExecutionTime"
string	CONFIG_BATCH_REMOVE_UNPROCESSED_ITEMS = "RemoveUnprocessedBatchItems"
string	CONFIG_BATCH_DEFAULT_STRATEGY_IO_WAIT_MAX = "BatchTask_IO_WAIT_MAX"
string	CONFIG_BATCH_DEFAULT_STRATEGY_CPU_LOAD_MAX = "BatchTask_CPU_LOAD_MAX"
string	CONFIG_BATCH_DEFAULT_STRATEGY_RAM_FREE_MIN = "BatchTask_RAM_FREE_MIN"
string	CONFIG_BATCH_DEFAULT_STRATEGY_STRATEGY_RDELAY = "BatchTask_RDELAY"
string	CONFIG_BATCH_DEFAULT_STRATEGY_RETRY = "BatchTask_RETRY"
string	CONFIG_BATCH_DEFAULT_STRATEGY_AUTOCLEANUP_TTL = "BatchTask_autocleanup_TTL"
string	CONFIG_BATCH_DEFAULT_STRATEGY_AUTOCLEANUP_DELETE_TYPE = "BatchTask_autocleanup_DeleteType"
string	CONFIG_BATCH_DEFAULT_STRATEGY_AUTOCLEANUP_DELETE_RETRIES = "BatchTask_autocleanup_DeleteRetries"
string	CONFIG_BATCH_DEFAULT_STRATEGY_AUTOCLEANUP_STATE = "BatchTask_autocleanup_State"
string	CONFIG_TASK_DTM_NAME_PROCESS = "BatchTaskDTMNameProcess"
string	CONFIG_TASK_DTM_TYPE_PROCESS = "BatchTaskDTMTypeProcess"
Static Public Attributes inherited from app.BaseServerManager.BaseServerManager
string	ADMIN_CONNECT_ENDPOINT = "Admin"
string	ADMIN_CONNECT_CLIENT = "Admin"
int	POLL_TIMEOUT_DEFAULT = 3000
int	STAT_FIELDS_OPERATION_ADD = 0
int	STAT_FIELDS_OPERATION_SUB = 1
int	STAT_FIELDS_OPERATION_SET = 2
int	STAT_FIELDS_OPERATION_INIT = 3
string	POLL_TIMEOUT_CONFIG_VAR_NAME = "POLL_TIMEOUT"
string	LOG_LEVEL_CONFIG_VAR_NAME = "LOG_LEVEL"
string	STAT_DUMPS_DEFAULT_DIR = "/tmp/"
string	STAT_DUMPS_DEFAULT_NAME = "%APP_NAME%_%CLASS_NAME%_stat_vars.dump"
dictionary	LOGGERS_NAMES = {APP_CONSTS.LOGGER_NAME, "dc", "dtm", "root", ""}

Detailed Description

Definition at line 41 of file BatchTasksManagerProcess.py.

Constructor & Destructor Documentation

__init__()

def dc.BatchTasksManagerProcess.BatchTasksManagerProcess.__init__	(		self,
			configParser,
			connectionBuilderLight = None
	)

Definition at line 95 of file BatchTasksManagerProcess.py.

  def __init__(self, configParser, connectionBuilderLight=None):
    super(BatchTasksManagerProcess, self).__init__()

    # Instantiate the connection builder light if not set
    if connectionBuilderLight is None:
      connectionBuilderLight = ConnectionBuilderLight()

    # Batches counter init in stat vars
    self.updateStatField(DC_CONSTS.BATCHES_PROCESS_COUNTER_TOTAL_NAME, 0, self.STAT_FIELDS_OPERATION_INIT)
    # Batches in queue counter init in stat vars
    self.updateStatField(DC_CONSTS.BATCHES_PROCESS_COUNTER_QUEUE_NAME, 0, self.STAT_FIELDS_OPERATION_SET)
    # Batches that fault processing counter init in stat vars
    self.updateStatField(DC_CONSTS.BATCHES_PROCESS_COUNTER_FAULT_NAME, 0, self.STAT_FIELDS_OPERATION_INIT)
    # Batches that not empty counter init in stat vars
    self.updateStatField(DC_CONSTS.BATCHES_PROCESS_COUNTER_FILLED_NAME, 0, self.STAT_FIELDS_OPERATION_INIT)
    # Batches urls total counter init in stat vars
    self.updateStatField(DC_CONSTS.BATCHES_PROCESS_COUNTER_URLS_NAME, 0, self.STAT_FIELDS_OPERATION_INIT)
    # Fault batches urls total counter init in stat vars
    self.updateStatField(DC_CONSTS.BATCHES_PROCESS_COUNTER_URLS_FAULT_NAME, 0, self.STAT_FIELDS_OPERATION_INIT)
    # Processing delete task requests fault counter name for stat variables
    self.updateStatField(DC_CONSTS.BATCHES_PROCESS_COUNTER_DELETE_FAULT_NAME, 0, self.STAT_FIELDS_OPERATION_INIT)
    # Processing check task requests fault counter name for stat variables
    self.updateStatField(DC_CONSTS.BATCHES_PROCESS_COUNTER_CHECK_FAULT_NAME, 0, self.STAT_FIELDS_OPERATION_INIT)
    # Processing batches fault TTL counter name for stat variables
    self.updateStatField(DC_CONSTS.BATCHES_PROCESS_COUNTER_FAULT_TTL_NAME, 0, self.STAT_FIELDS_OPERATION_INIT)
    # Processing batches cancelled counter name for stat variables
    self.updateStatField(DC_CONSTS.BATCHES_PROCESS_COUNTER_CANCELLED_NAME, 0, self.STAT_FIELDS_OPERATION_INIT)

    # Get configuration settings
    className = self.__class__.__name__
    self.serverName = configParser.get(className, self.CONFIG_SERVER)
    self.clientSitesManagerName = configParser.get(className, self.CONFIG_SITES_MANAGER_CLIENT)
    # Configuration settings for DTMD server interaction
    self.configVars[self.CONFIG_DTMD_HOST] = configParser.get(className, self.CONFIG_DTMD_HOST)
    self.configVars[self.CONFIG_DTMD_PORT] = configParser.get(className, self.CONFIG_DTMD_PORT)
    self.configVars[self.CONFIG_DTMD_TIMEOUT] = configParser.getint(className, self.CONFIG_DTMD_TIMEOUT)

    # Max URLs per batch
    self.configVars[self.CONFIG_BATCH_MAX_URLS] = configParser.getint(className, self.CONFIG_BATCH_MAX_URLS)
    # Set connections poll timeout, defines period of HCE cluster monitoring cycle, msec
    self.configVars[self.POLL_TIMEOUT_CONFIG_VAR_NAME] = configParser.getint(className, self.CONFIG_POLLING_TIMEOUT)
    # Set crawler task app name
    self.configVars[self.CONFIG_DRCE_PROCESSOR_APP_NAME] = \
      configParser.get(className, self.CONFIG_DRCE_PROCESSOR_APP_NAME)
    self.configVars[self.CONFIG_BATCH_DEFAULT_MAX_TIME] = \
      configParser.getint(className, self.CONFIG_BATCH_DEFAULT_MAX_TIME)

    # #TODO experemental
    # Max execution time for batch
    self.configVars[self.CONFIG_BATCH_MAX_TIME] = configParser.getint(className, self.CONFIG_BATCH_MAX_TIME)
    # Remove unprocessed items for batch
    self.configVars[self.CONFIG_BATCH_REMOVE_UNPROCESSED_ITEMS] = \
      configParser.getint(className, self.CONFIG_BATCH_REMOVE_UNPROCESSED_ITEMS)

    # Create connections and raise bind or connect actions for correspondent connection type
    serverConnection = connectionBuilderLight.build(TRANSPORT_CONSTS.SERVER_CONNECT, self.serverName)
    sitesManagerConnection = connectionBuilderLight.build(TRANSPORT_CONSTS.CLIENT_CONNECT, self.clientSitesManagerName)

    # Init the DTMD connection
    self.dtmdConnection = connectionBuilderLight.build(TRANSPORT_CONSTS.CLIENT_CONNECT,
                                                       self.configVars[self.CONFIG_DTMD_HOST] + ":" + \
                                                       self.configVars[self.CONFIG_DTMD_PORT],
                                                       TRANSPORT_CONSTS.TCP_TYPE)

    # Add connections to the polling set
    self.addConnection(self.serverName, serverConnection)
    self.addConnection(self.clientSitesManagerName, sitesManagerConnection)

    # Set event handler for URL_UPDATE_RESPONSE event
    self.setEventHandler(DC_CONSTS.EVENT_TYPES.URL_UPDATE_RESPONSE, self.onURLUpdateResponse)
    # Set event handler for URL_DELETE_RESPONSE event
    self.setEventHandler(DC_CONSTS.EVENT_TYPES.URL_DELETE_RESPONSE, self.onURLDeleteResponse)

    # Initialize the DTM tasks queue, the key is taskId, the value is the Batch object
    self.dtmTasksQueue = {}

    # Processing init
    self.configVars[self.CONFIG_DRCE_DB_APP_NAME] = configParser.get(className, self.CONFIG_DRCE_DB_APP_NAME)
    self.configVars[self.CONFIG_PROCESS_PERIOD] = configParser.getint(className, self.CONFIG_PROCESS_PERIOD)
    self.processProcessingLastTs = time.time()

    # The Batch default order by criterion to fetch URLs
    self.configVars[self.CONFIG_BATCH_ORDER_BY_URLS] = configParser.get(className, self.CONFIG_BATCH_ORDER_BY_URLS)
    # The Batch max tasks in batch queue, if limit reached new batch tasks will not be started; zero means unlimited
    self.configVars[self.CONFIG_BATCH_MAX_TASKS] = configParser.getint(className, self.CONFIG_BATCH_MAX_TASKS)
    # The Batch queue processing init
    self.configVars[self.CONFIG_BATCH_QUEUE_PERIOD] = configParser.getint(className, self.CONFIG_BATCH_QUEUE_PERIOD)
    self.processBatchQueuelLastTs = time.time()
    # The Batch queue task TTL, sec
    self.configVars[self.CONFIG_BATCH_QUEUE_TASK_TTL] = configParser.getint(className, self.CONFIG_BATCH_QUEUE_TASK_TTL)
    # The Batch queue tasks state check method, see ini file comments
    self.configVars[self.CONFIG_BATCH_QUEUE_TASK_CHECK_METHOD] = \
      configParser.getint(className, self.CONFIG_BATCH_QUEUE_TASK_CHECK_METHOD)
    # The Batch DRCE task starter name
    self.configVars[self.CONFIG_BATCH_DEFAULT_STARTER] = configParser.get(className, self.CONFIG_BATCH_DEFAULT_STARTER)
    # The Batch tasks's strategy configuration parameters for DTM service load
    self.configVars[self.CONFIG_BATCH_DEFAULT_STRATEGY_IO_WAIT_MAX] = \
      configParser.getint(className, self.CONFIG_BATCH_DEFAULT_STRATEGY_IO_WAIT_MAX)
    self.configVars[self.CONFIG_BATCH_DEFAULT_STRATEGY_CPU_LOAD_MAX] = \
      configParser.getint(className, self.CONFIG_BATCH_DEFAULT_STRATEGY_CPU_LOAD_MAX)
    self.configVars[self.CONFIG_BATCH_DEFAULT_STRATEGY_RAM_FREE_MIN] = \
      configParser.getint(className, self.CONFIG_BATCH_DEFAULT_STRATEGY_RAM_FREE_MIN)
    self.configVars[self.CONFIG_BATCH_DEFAULT_STRATEGY_STRATEGY_RDELAY] = \
      configParser.getint(className, self.CONFIG_BATCH_DEFAULT_STRATEGY_STRATEGY_RDELAY)
    self.configVars[self.CONFIG_BATCH_DEFAULT_STRATEGY_RETRY] = \
      configParser.getint(className, self.CONFIG_BATCH_DEFAULT_STRATEGY_RETRY)
    # The Batch DRCE tasks auto cleanup fields
    self.configVars[self.CONFIG_BATCH_DEFAULT_STRATEGY_AUTOCLEANUP_TTL] = \
                   configParser.getint(className, self.CONFIG_BATCH_DEFAULT_STRATEGY_AUTOCLEANUP_TTL)
    self.configVars[self.CONFIG_BATCH_DEFAULT_STRATEGY_AUTOCLEANUP_DELETE_TYPE] = \
                   configParser.getint(className, self.CONFIG_BATCH_DEFAULT_STRATEGY_AUTOCLEANUP_DELETE_TYPE)
    self.configVars[self.CONFIG_BATCH_DEFAULT_STRATEGY_AUTOCLEANUP_DELETE_RETRIES] = \
                   configParser.get(className, self.CONFIG_BATCH_DEFAULT_STRATEGY_AUTOCLEANUP_DELETE_RETRIES)
    self.configVars[self.CONFIG_BATCH_DEFAULT_STRATEGY_AUTOCLEANUP_STATE] = \
                   configParser.get(className, self.CONFIG_BATCH_DEFAULT_STRATEGY_AUTOCLEANUP_STATE)
    # The Batch default where criterion to fetch URLs
    self.configVars[self.CONFIG_BATCH_WHERE_URLS] = configParser.get(className, self.CONFIG_BATCH_WHERE_URLS)
    # The Batch default where criterion to fetch Sites
    self.configVars[self.CONFIG_BATCH_WHERE_SITES] = configParser.get(className, self.CONFIG_BATCH_WHERE_SITES)

    # Init the config processing mode and runtime values
    self.configVars[self.CONFIG_PROCESS_MODE] = configParser.getint(className, self.CONFIG_PROCESS_MODE)

    # Processing task DTM name
    self.configVars[self.CONFIG_TASK_DTM_NAME_PROCESS] = configParser.get(className, self.CONFIG_TASK_DTM_NAME_PROCESS)
    # Processing task DTM type
    self.configVars[self.CONFIG_TASK_DTM_TYPE_PROCESS] = configParser.getint(className,
                                                                             self.CONFIG_TASK_DTM_TYPE_PROCESS)

Here is the call graph for this function:

Member Function Documentation

dtmdRequestExecute()

def dc.BatchTasksManagerProcess.BatchTasksManagerProcess.dtmdRequestExecute	(		self,
			requestEvent,
			timeout,
			maxTries = 100
	)

Definition at line 545 of file BatchTasksManagerProcess.py.

  def dtmdRequestExecute(self, requestEvent, timeout, maxTries=100):
    ret = None
    if maxTries < 0:
      maxTries = 0

    try:
      # Send DTMD request
      self.dtmdConnection.send(requestEvent)

      for i in range(maxTries + 1):
        # Poll DTMD connection
        if self.dtmdConnection.poll(int(timeout)) == 0:
          logger.error("DTMD request timeout reached " + str(timeout) + "!")
          break
        else:
          # Recv DTMD response
          retEvent = self.dtmdConnection.recv()
          if retEvent != None:
            # Get response object
            if type(retEvent.eventObj) == type(dtm.EventObjects.EEResponseData(0)) or\
              type(retEvent.eventObj) == type(dtm.EventObjects.GeneralResponse()) or\
              isinstance(retEvent.eventObj, list):
              if retEvent.uid == requestEvent.uid:
                ret = retEvent.eventObj
                break
              else:
                logger.error("DTMD returned wrong object uid: " + str(retEvent.uid) + " but " + \
                             str(requestEvent.uid) + " expected, iteration " + str(i) + "!")
            else:
              logger.error("DTMD returned wrong object type: " + str(type(retEvent.eventObj)) + "!")
          else:
            logger.error("DTMD returned None event!")
    except Exception, e:
      logger.error("DTMD request execution exception: " + e.message + "!")

    logger.debug("The DTMD request finished!")

    return ret

Here is the call graph for this function:

Here is the caller graph for this function:

getDTMTaskState()

def dc.BatchTasksManagerProcess.BatchTasksManagerProcess.getDTMTaskState	(		self,
			taskId
	)

Definition at line 388 of file BatchTasksManagerProcess.py.

  def getDTMTaskState(self, taskId):
    taskState = None

    if self.configVars[self.CONFIG_BATCH_QUEUE_TASK_CHECK_METHOD] == self.DTM_TASK_CHECK_STATE_METHOD_STATUS:
      # Check the task status on DRCE EE hce-node
      logger.debug("Check state of taskId=" + str(taskId))
      checkTaskStateObj = dtm.EventObjects.CheckTaskState(taskId)
      checkStateEvent = self.eventBuilder.build(DTM_CONSTS.EVENT_TYPES.CHECK_TASK_STATE, checkTaskStateObj)
      eeResponseData = self.dtmdRequestExecute(checkStateEvent, self.configVars[self.CONFIG_DTMD_TIMEOUT])
      logger.debug("DTM CheckTaskState request finished, taskId=" + str(taskId))
      if eeResponseData is not None and type(eeResponseData) == type(dtm.EventObjects.EEResponseData(0)):
        taskState = eeResponseData.state
    else:
      # Get task status on DTM service
      logger.debug("Get status of taskId=" + str(taskId))
      getTasksStatusObj = dtm.EventObjects.GetTasksStatus([taskId])
      getTasksStatusEvent = self.eventBuilder.build(DTM_CONSTS.EVENT_TYPES.GET_TASK_STATUS, getTasksStatusObj)
      listTaskManagerFields = self.dtmdRequestExecute(getTasksStatusEvent, self.configVars[self.CONFIG_DTMD_TIMEOUT])
      logger.debug("DTM getTasksStatus request finished, taskId=" + str(taskId))
      if listTaskManagerFields is not None and isinstance(listTaskManagerFields, list):
        if len(listTaskManagerFields) > 0:
          taskState = listTaskManagerFields[0].fields["state"]
        else:
          # Set TASK_STATE_FINISHED state to push task to delete from queue
          taskState = dtm.EventObjects.EEResponseData.TASK_STATE_FINISHED
      else:
        logger.error("DTM getTasksStatus taskId=" + str(taskId) + " returned wrong data:\n" + \
                     Utils.varDump(listTaskManagerFields))

    return taskState

Here is the call graph for this function:

Here is the caller graph for this function:

isDTMTaskDead()

def dc.BatchTasksManagerProcess.BatchTasksManagerProcess.isDTMTaskDead	(		self,
			state
	)

Definition at line 509 of file BatchTasksManagerProcess.py.

  def isDTMTaskDead(self, state):
    ret = False
    if state == dtm.EventObjects.EEResponseData.TASK_STATE_FINISHED or\
       state == dtm.EventObjects.EEResponseData.TASK_STATE_CRASHED or\
       state == dtm.EventObjects.EEResponseData.TASK_STATE_TERMINATED or\
       state == dtm.EventObjects.EEResponseData.TASK_STATE_UNDEFINED or\
       state == dtm.EventObjects.EEResponseData.TASK_STATE_SET_ERROR or\
       state == dtm.EventObjects.EEResponseData.TASK_STATE_TERMINATED_BY_DRCE_TTL or\
       state == dtm.EventObjects.EEResponseData.TASK_STATE_SCHEDULE_TRIES_EXCEEDED:
      ret = True

    return ret

Here is the caller graph for this function:

on_poll_timeout()

def dc.BatchTasksManagerProcess.BatchTasksManagerProcess.on_poll_timeout

(

self

)

Definition at line 229 of file BatchTasksManagerProcess.py.

  def on_poll_timeout(self):
    logger.debug("Periodic iteration started.")
    try:
      # Process the Processing batch
      if self.configVars[self.CONFIG_PROCESS_PERIOD] > 0 and\
        time.time() - self.processProcessingLastTs > self.configVars[self.CONFIG_PROCESS_PERIOD]:
        self.processProcessingLastTs = time.time()
        if self.configVars[self.CONFIG_PROCESS_MODE] == 1:
          logger.debug("Processing batch cycle iteration started")
          if self.configVars[self.CONFIG_BATCH_MAX_TASKS] > len(self.dtmTasksQueue):
            if self.setProcessBatch():
              self.updateStatField(DC_CONSTS.BATCHES_PROCESS_COUNTER_TOTAL_NAME, 1, self.STAT_FIELDS_OPERATION_ADD)
          else:
            self.updateStatField(DC_CONSTS.BATCHES_PROCESS_COUNTER_CANCELLED_NAME, 1,
                                 self.STAT_FIELDS_OPERATION_ADD)
            logger.debug("Max processing batch tasks %s in queue reached, new batch is not created!",
                         str(len(self.dtmTasksQueue)))
        else:
          logger.debug("Processing batch disabled!")

      # Process the DRCE Batch tasks queue
      if self.configVars[self.CONFIG_BATCH_QUEUE_PERIOD] > 0 and\
        time.time() - self.processBatchQueuelLastTs > self.configVars[self.CONFIG_BATCH_QUEUE_PERIOD]:
        self.processBatchQueuelLastTs = time.time()
        logger.debug("Process DTM tasks queue!")
        # Process the DTM tasks queue
        self.processDTMTasksQueue()

    except IOError as e:
      del e
    except Exception as err:
      ExceptionLog.handler(logger, err, "Exception:")

    logger.debug("Periodic iteration finished.")

Here is the call graph for this function:

onURLDeleteResponse()

def dc.BatchTasksManagerProcess.BatchTasksManagerProcess.onURLDeleteResponse	(		self,
			event
	)

Definition at line 673 of file BatchTasksManagerProcess.py.

  def onURLDeleteResponse(self, event):
    try:
      logger.debug("Reply received on URL delete.")
      clientResponse = event.eventObj
      if clientResponse.errorCode == EventObjects.ClientResponse.STATUS_OK:
        if len(clientResponse.itemsList) > 0:
          for clientResponseItem in clientResponse.itemsList:
            if clientResponseItem.errorCode != EventObjects.ClientResponseItem.STATUS_OK:
              logger.error("URLDelete response error: " + str(clientResponseItem.errorCode) + " : " + \
                           clientResponseItem.errorMessage + ", host:" + clientResponseItem.host + ", port:" + \
                           clientResponseItem.port + ", node:" + clientResponseItem.node + "!")
        else:
          logger.error("URLDelete response empty list!")
      else:
        logger.error("URLDelete response error:" + str(clientResponse.errorCode) + " : " + clientResponse.errorMessage)
    except Exception as err:
      ExceptionLog.handler(logger, err, "Exception:")

onURLUpdateResponse()

def dc.BatchTasksManagerProcess.BatchTasksManagerProcess.onURLUpdateResponse	(		self,
			event
	)

Definition at line 651 of file BatchTasksManagerProcess.py.

  def onURLUpdateResponse(self, event):
    try:
      logger.debug("Reply received on URL update.")
      clientResponse = event.eventObj
      if clientResponse.errorCode == EventObjects.ClientResponse.STATUS_OK:
        if len(clientResponse.itemsList) > 0:
          for clientResponseItem in clientResponse.itemsList:
            if clientResponseItem.errorCode != EventObjects.ClientResponseItem.STATUS_OK:
              logger.error("URLUpdate response error: " + str(clientResponseItem.errorCode) + " : " + \
                           clientResponseItem.errorMessage + ", host:" + clientResponseItem.host + ", port:" + \
                           clientResponseItem.port + ", node:" + clientResponseItem.node + "!")
        else:
          logger.error("URLUpdate response empty list!")
      else:
        logger.error("URLUpdate response error:" + str(clientResponse.errorCode) + " : " + clientResponse.errorMessage)
    except Exception as err:
      ExceptionLog.handler(logger, err, "Exception:")

processDTMTasksQueue()

def dc.BatchTasksManagerProcess.BatchTasksManagerProcess.processDTMTasksQueue

(

self

)

Definition at line 424 of file BatchTasksManagerProcess.py.

  def processDTMTasksQueue(self):
    tmpQueue = {}

    logger.debug("Process batch tasks in queue:" + str(len(self.dtmTasksQueue)))
    self.updateStatField(DC_CONSTS.BATCHES_PROCESS_COUNTER_QUEUE_NAME, len(self.dtmTasksQueue),
                         self.STAT_FIELDS_OPERATION_SET)

    # Process the DTM tasks queue
    for taskId, taskBatch in self.dtmTasksQueue.items():
      ttl = self.configVars[self.CONFIG_BATCH_QUEUE_TASK_TTL]
      batchState = self.getDTMTaskState(taskId)
      if batchState != None:
        logger.debug("Process batch state " + str(batchState) + ", Id=" + str(taskId))
        if batchState == dtm.EventObjects.EEResponseData.TASK_STATE_FINISHED or\
           batchState == dtm.EventObjects.EEResponseData.TASK_STATE_CRASHED or\
           batchState == dtm.EventObjects.EEResponseData.TASK_STATE_TERMINATED or\
           batchState == dtm.EventObjects.EEResponseData.TASK_STATE_UNDEFINED or\
           batchState == dtm.EventObjects.EEResponseData.TASK_STATE_SET_ERROR or\
           batchState == dtm.EventObjects.EEResponseData.TASK_STATE_SCHEDULE_TRIES_EXCEEDED:
          # Delete task in DTM and task's data in EE (DRCE)
          deleteTaskObj = dtm.EventObjects.DeleteTask(taskId)
          deleteTaskEvent = self.eventBuilder.build(DTM_CONSTS.EVENT_TYPES.DELETE_TASK, deleteTaskObj)
          generalResponse = self.dtmdRequestExecute(deleteTaskEvent, self.configVars[self.CONFIG_DTMD_TIMEOUT])
          logger.debug("DTM DeleteTask request finished, taskId=" + str(taskId))
          if generalResponse is not None:
            if generalResponse.errorCode == dtm.EventObjects.GeneralResponse.ERROR_OK:
              logger.debug("DTM task deleted, taskId=" + str(taskId))
              # if batchState == dtm.EventObjects.EEResponseData.TASK_STATE_FINISHED:
              if self.isDTMTaskDead(batchState):
                logger.debug("batch:\n" + varDump(taskBatch) + "\n finished, taskId=" + str(taskId))
                self.processFinishedBatch(taskBatch)
              else:
                self.updateStatField(DC_CONSTS.BATCHES_PROCESS_COUNTER_FAULT_NAME, 1, self.STAT_FIELDS_OPERATION_ADD)
                logger.debug("batch:" + varDump(taskBatch) + " not finished, state= " + str(batchState))
                # TODO: Send update URLs of not finished batch on all nodes to get possibility to process them next time
                # self.sendURLUpdate(taskBatch.items, taskBatch.id, False)
            else:
              # Save this batch to check it later
              tmpQueue[taskId] = taskBatch
              self.updateStatField(DC_CONSTS.BATCHES_PROCESS_COUNTER_DELETE_FAULT_NAME, 1,
                                   self.STAT_FIELDS_OPERATION_ADD)
              logger.error("DTM delete task taskId=" + str(taskId) + " error: " + str(generalResponse.errorCode) + \
                           " : " + generalResponse.errorMessage + ", statuses:" + varDump(generalResponse))
          else:
            # Save this batch to check it later
            tmpQueue[taskId] = taskBatch
            self.updateStatField(DC_CONSTS.BATCHES_PROCESS_COUNTER_DELETE_FAULT_NAME, 1, self.STAT_FIELDS_OPERATION_ADD)
            logger.error("DTM delete task error: wrong response or timeout, taskId=" + str(taskId) + "!")
        else:
          logger.debug("DTM task Id=" + str(taskId) + " state=" + str(batchState))
          if time.time() - taskBatch.QueuedTs > ttl:
            # Terminate task and delete it's data request
            deleteTaskObj = dtm.EventObjects.DeleteTask(taskId)
            deleteTaskObj.action = dtm.EventObjects.DeleteTask.ACTION_TERMINATE_TASK_AND_DELETE_DATA
            deleteTaskEvent = self.eventBuilder.build(DTM_CONSTS.EVENT_TYPES.DELETE_TASK, deleteTaskObj)
            generalResponse = self.dtmdRequestExecute(deleteTaskEvent, self.configVars[self.CONFIG_DTMD_TIMEOUT])
            logger.error("DTM task Id=" + str(taskId) + " terminated and removed from queue by TTL:" + str(ttl))
            # TODO: Send update URLs of not finished batch on all nodes to get possibility to crawl them next time
            # self.sendURLUpdate(taskBatch.items, taskBatch.id, False)
            self.updateStatField(DC_CONSTS.BATCHES_PROCESS_COUNTER_FAULT_NAME, 1, self.STAT_FIELDS_OPERATION_ADD)
            self.updateStatField(DC_CONSTS.BATCHES_PROCESS_COUNTER_FAULT_TTL_NAME, 1, self.STAT_FIELDS_OPERATION_ADD)
          else:
            # Save this batch to check it later
            tmpQueue[taskId] = taskBatch
            logger.debug("DTM task Id=" + str(taskId) + " still in queue")
      else:
        logger.error("DTM check task state error: wrong response or timeout, taskId=" + str(taskId) + "!")
        if time.time() - taskBatch.QueuedTs > ttl:
          logger.error("DTM task Id=" + str(taskId) + " removed from queue by TTL:" + str(ttl))
        else:
          # Save this batch to check it later
          tmpQueue[taskId] = taskBatch
          self.updateStatField(DC_CONSTS.BATCHES_PROCESS_COUNTER_CHECK_FAULT_NAME, 1, self.STAT_FIELDS_OPERATION_ADD)
          logger.error("DTM task Id=" + str(taskId) + " saved in queue.")

    self.dtmTasksQueue = tmpQueue
    self.updateStatField(DC_CONSTS.BATCHES_PROCESS_COUNTER_QUEUE_NAME, len(self.dtmTasksQueue),
                         self.STAT_FIELDS_OPERATION_SET)
    logger.debug("The DTM tasks queue processing finished, batch tasks in queue " + str(len(self.dtmTasksQueue)))

Here is the call graph for this function:

Here is the caller graph for this function:

processFinishedBatch()

def dc.BatchTasksManagerProcess.BatchTasksManagerProcess.processFinishedBatch	(		self,
			taskBatch
	)

Definition at line 528 of file BatchTasksManagerProcess.py.

  def processFinishedBatch(self, taskBatch):
    # if self.configVars[self.CONFIG_CRAWLED_URLS_STRATEGY] == 0:
    #  self.sendURLUpdate(taskBatch.items, taskBatch.id, True)
    #  logger.debug("Send update URLs from batch for all foreign hosts by the Batch_Id")
    # else:
    #  self.sendURLDelete(taskBatch.items, taskBatch.id)
    #  logger.debug("Send delete URLs from batch for all foreign hosts by the Batch_Id")
    pass
    # TODO: update or delete URLs from returned batch

Here is the caller graph for this function:

sendBatchTaskToDTM()

def dc.BatchTasksManagerProcess.BatchTasksManagerProcess.sendBatchTaskToDTM	(		self,
			batch
	)

Definition at line 319 of file BatchTasksManagerProcess.py.

  def sendBatchTaskToDTM(self, batch):
    taskId = 0
    # Prepare NewTask object
    appName = self.configVars[self.CONFIG_DRCE_DB_APP_NAME]
    newTaskObj = dtm.EventObjects.NewTask(appName)
    newTaskObj.name = self.configVars[self.CONFIG_TASK_DTM_NAME_PROCESS]
    newTaskObj.type = self.configVars[self.CONFIG_TASK_DTM_TYPE_PROCESS]
    newTaskObj.setSessionVar("tmode", dtm.EventObjects.Task.TASK_MODE_ASYNCH)
    newTaskObj.setSessionVar("shell", self.configVars[self.CONFIG_BATCH_DEFAULT_STARTER])
    newTaskObj.setSessionVar("time_max", int(self.configVars[self.CONFIG_BATCH_DEFAULT_MAX_TIME]) * 1000)
    newTaskObj.setSessionVar("task_type", int(newTaskObj.type))
    # Configure task's strategy
    if self.configVars[self.CONFIG_BATCH_DEFAULT_STRATEGY_IO_WAIT_MAX] > 0:
      newTaskObj.setStrategyVar(dtm.EventObjects.Task.STRATEGY_IO_WAIT_MAX,
                                self.configVars[self.CONFIG_BATCH_DEFAULT_STRATEGY_IO_WAIT_MAX])
    if self.configVars[self.CONFIG_BATCH_DEFAULT_STRATEGY_CPU_LOAD_MAX] > 0:
      newTaskObj.setStrategyVar(dtm.EventObjects.Task.STRATEGY_CPU_LOAD_MAX,
                                self.configVars[self.CONFIG_BATCH_DEFAULT_STRATEGY_CPU_LOAD_MAX])
    if self.configVars[self.CONFIG_BATCH_DEFAULT_STRATEGY_RAM_FREE_MIN] > 0:
      newTaskObj.setStrategyVar(dtm.EventObjects.Task.STRATEGY_RAM_FREE,
                                self.configVars[self.CONFIG_BATCH_DEFAULT_STRATEGY_RAM_FREE_MIN])
    if self.configVars[self.CONFIG_BATCH_DEFAULT_STRATEGY_STRATEGY_RDELAY] > 0:
      newTaskObj.setStrategyVar(dtm.EventObjects.Task.STRATEGY_RDELAY,
                                self.configVars[self.CONFIG_BATCH_DEFAULT_STRATEGY_STRATEGY_RDELAY])
    if self.configVars[self.CONFIG_BATCH_DEFAULT_STRATEGY_RETRY] > 0:
      newTaskObj.setStrategyVar(dtm.EventObjects.Task.STRATEGY_RETRY,
                                self.configVars[self.CONFIG_BATCH_DEFAULT_STRATEGY_RETRY])
    # Set auto cleanup fields
    autoCleanupFields = {}
    if self.configVars[self.CONFIG_BATCH_DEFAULT_STRATEGY_AUTOCLEANUP_TTL] > -1:
      autoCleanupFields[dtm.EventObjects.Task.STRATEGY_AUTOCLEANUP_TTL] = \
                       int(self.configVars[self.CONFIG_BATCH_DEFAULT_STRATEGY_AUTOCLEANUP_TTL]) * 1000
    if self.configVars[self.CONFIG_BATCH_DEFAULT_STRATEGY_AUTOCLEANUP_DELETE_TYPE] > -1:
      autoCleanupFields[dtm.EventObjects.Task.STRATEGY_AUTOCLEANUP_DELETE_TYPE] = \
                       self.configVars[self.CONFIG_BATCH_DEFAULT_STRATEGY_AUTOCLEANUP_DELETE_TYPE]
    if self.configVars[self.CONFIG_BATCH_DEFAULT_STRATEGY_AUTOCLEANUP_DELETE_RETRIES] > -1:
      autoCleanupFields[dtm.EventObjects.Task.STRATEGY_AUTOCLEANUP_DELETE_RETRIES] = \
                       self.configVars[self.CONFIG_BATCH_DEFAULT_STRATEGY_AUTOCLEANUP_DELETE_RETRIES]
    if self.configVars[self.CONFIG_BATCH_DEFAULT_STRATEGY_AUTOCLEANUP_STATE] > -1:
      autoCleanupFields[dtm.EventObjects.Task.STRATEGY_AUTOCLEANUP_SSTATE] = \
                       self.configVars[self.CONFIG_BATCH_DEFAULT_STRATEGY_AUTOCLEANUP_STATE]
    if len(autoCleanupFields) > 0:
      newTaskObj.setStrategyVar(dtm.EventObjects.Task.STRATEGY_autoCleanupFields, autoCleanupFields)
    batch.id = newTaskObj.id
    drceSyncTasksCoverObj = DC_CONSTS.DRCESyncTasksCover(DC_CONSTS.EVENT_TYPES.URL_FETCH, [batch])
    newTaskObj.input = pickle.dumps(drceSyncTasksCoverObj)
    newTaskEvent = self.eventBuilder.build(DTM_CONSTS.EVENT_TYPES.NEW_TASK, newTaskObj)
    generalResponse = self.dtmdRequestExecute(newTaskEvent, self.configVars[self.CONFIG_DTMD_TIMEOUT])
    if generalResponse is not None:
      if generalResponse.errorCode == dtm.EventObjects.GeneralResponse.ERROR_OK:
        # New crawling Batch task set successfully
        taskId = newTaskObj.id
      else:
        # Some error of task operation
        logger.error("DTM set batch task error: " + str(generalResponse.errorCode) + " : " + \
                     generalResponse.errorMessage + ", statuses:" + varDump(generalResponse))
    else:
      logger.error("DTM set batch task response error, possible timeout!")

    # TODO: return the Task Id any case error or not to check it state later
    taskId = newTaskObj.id

    return taskId

Here is the call graph for this function:

Here is the caller graph for this function:

sendURLDelete()

def dc.BatchTasksManagerProcess.BatchTasksManagerProcess.sendURLDelete	(		self,
			batchItemsList,
			batchId
	)

Definition at line 626 of file BatchTasksManagerProcess.py.

  def sendURLDelete(self, batchItemsList, batchId):
    urlsList = []

    # Prepare URLs list to delete
    for batchItem in batchItemsList:
      sqlExpression = SQLExpression("`ParentMd5`<>'' AND `URLMd5`='" + str(batchItem.urlId) + "' AND `Batch_Id`<>" + \
                                     str(batchId))
      urlDelete = EventObjects.URLDelete(batchItem.siteId, None, EventObjects.URLStatus.URL_TYPE_URL,
                                         {EventObjects.URLFetch.CRITERION_WHERE:sqlExpression,
                                          EventObjects.URLFetch.CRITERION_LIMIT:1},
                                         reason=EventObjects.URLDelete.REASON_SELECT_TO_PROCESS_TTL)
      logger.debug("URLDelete: " + varDump(urlDelete))
      urlsList.append(urlDelete)

    # Make URLDelete event
    urlDeleteEvent = self.eventBuilder.build(DC_CONSTS.EVENT_TYPES.URL_DELETE, urlsList)
    # Send request URLDelete to SitesManager
    self.send(self.clientSitesManagerName, urlDeleteEvent)
    logger.debug("The URLDelete request to SitesManager sent!")

Here is the call graph for this function:

sendURLUpdate()

def dc.BatchTasksManagerProcess.BatchTasksManagerProcess.sendURLUpdate	(		self,
			batchItemsList,
			batchId,
			batchState
	)

Definition at line 591 of file BatchTasksManagerProcess.py.

  def sendURLUpdate(self, batchItemsList, batchId, batchState):
    urlsList = []

    # Prepare URLs list to update
    for batchItem in batchItemsList:
      # Set state value depends on update reason - crawled successfully or not
      if batchState is True:
        status = EventObjects.URL.STATUS_CRAWLED
        sqlExpression = SQLExpression("`URLMd5`='" + str(batchItem.urlId) + "' AND `Batch_Id` <>" + str(batchId) + \
                                      " AND `Status` NOT IN (" + str(EventObjects.URL.STATUS_CRAWLED) + \
                                      "," + str(EventObjects.URL.STATUS_SELECTED_PROCESSING) + ")")
      else:
        status = EventObjects.URL.STATUS_NEW
        sqlExpression = SQLExpression("`URLMd5`='" + str(batchItem.urlId) + "' AND `Status` IN (" +
                                      str(EventObjects.URL.STATUS_SELECTED_CRAWLING) + "," + \
                                      str(EventObjects.URL.STATUS_SELECTED_CRAWLING_INCREMENTAL) + ")")

      urlUpdate = EventObjects.URLUpdate(batchItem.siteId, batchItem.urlId, EventObjects.URLStatus.URL_TYPE_MD5,
                                         None, status)
      urlUpdate.criterions[EventObjects.URLFetch.CRITERION_WHERE] = sqlExpression
      logger.debug("URLUpdate: " + varDump(urlUpdate))
      urlsList.append(urlUpdate)

    # Make URLUpdate event
    urlUpdateEvent = self.eventBuilder.build(DC_CONSTS.EVENT_TYPES.URL_UPDATE, urlsList)
    # Send request URLUpdate to SitesManager
    self.send(self.clientSitesManagerName, urlUpdateEvent)
    logger.debug("The URLUpdate request to SitesManager sent!")

Here is the call graph for this function:

setProcessBatch()

def dc.BatchTasksManagerProcess.BatchTasksManagerProcess.setProcessBatch

(

self

)

Definition at line 269 of file BatchTasksManagerProcess.py.

  def setProcessBatch(self):
    ret = False

    try:
      # Create the URLUpdate object to set in progress of processing state for selected URLs
      urlUpdate = EventObjects.URLUpdate(0, "", EventObjects.URLStatus.URL_TYPE_MD5, None,
                                         EventObjects.URL.STATUS_SELECTED_PROCESSING)
      urlUpdate.tcDate = SQLExpression("NOW()")
      # Create URLFetch object with URLUpdate to update selected URLs state
      sCrit = "IFNULL((SELECT `Value` FROM `sites_properties` WHERE `Name`='PROCESS_WHERE_SITES'), " + \
              self.configVars[self.CONFIG_BATCH_WHERE_SITES] + ")"
      sitesCriterions = {EventObjects.URLFetch.CRITERION_WHERE: sCrit,
                         EventObjects.URLFetch.CRITERION_ORDER:"Priority DESC, TcDateProcess ASC"}
      uCrit = "IFNULL(%PROCESS_WHERE_URLS%, " + self.configVars[self.CONFIG_BATCH_WHERE_URLS] + ")"
      urlCriterions = {EventObjects.URLFetch.CRITERION_WHERE: uCrit,
                       EventObjects.URLFetch.CRITERION_ORDER: self.configVars[self.CONFIG_BATCH_ORDER_BY_URLS],
                       EventObjects.URLFetch.CRITERION_LIMIT: str(self.configVars[self.CONFIG_BATCH_MAX_URLS]),
                       EventObjects.URLFetch.CRITERION_SQL: {
                           "PROCESS_WHERE_URLS": "SELECT `Value` FROM `dc_sites`.`sites_properties` WHERE " + \
                         "`Name`='PROCESS_WHERE_URLS' AND `Site_Id`=\"%SITE_ID%\" LIMIT 1"  # pylint: disable=C0301
                       }
                      }
      siteUpdate = EventObjects.SiteUpdate(0)
      siteUpdate.tcDateProcess = EventObjects.SQLExpression("Now()")
      urlFetch = EventObjects.URLFetch(None, urlCriterions, sitesCriterions, urlUpdate, siteUpdate)
      urlFetch.algorithm = EventObjects.URLFetch.PROPORTIONAL_ALGORITHM
      urlFetch.maxURLs = self.configVars[self.CONFIG_BATCH_MAX_URLS]
      taskId = self.sendBatchTaskToDTM(urlFetch)
      if taskId > 0:
        logger.debug("DTM process batch was set, taskId=%s", str(taskId))
        # Insert the Batch object in to the queue
        urlFetch.QueuedTs = time.time()
        urlFetch.crawlerType = EventObjects.Batch.TYPE_PROCESS
        self.dtmTasksQueue[taskId] = urlFetch
        self.updateStatField(DC_CONSTS.BATCHES_PROCESS_COUNTER_QUEUE_NAME, len(self.dtmTasksQueue),
                             self.STAT_FIELDS_OPERATION_SET)
        ret = True
      else:
        logger.error("Error send process batch task to DTM!")

    except Exception as err:
      ExceptionLog.handler(logger, err, "Exception:")

    return ret

Here is the call graph for this function:

Here is the caller graph for this function:

Member Data Documentation

clientSitesManagerName

dc.BatchTasksManagerProcess.BatchTasksManagerProcess.clientSitesManagerName

Definition at line 126 of file BatchTasksManagerProcess.py.

CONFIG_BATCH_DEFAULT_MAX_TIME

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_BATCH_DEFAULT_MAX_TIME = "BatchDefaultMaxExecutionTime"

static

Definition at line 59 of file BatchTasksManagerProcess.py.

CONFIG_BATCH_DEFAULT_STARTER

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_BATCH_DEFAULT_STARTER = "BatchTask_STARTER"

static

Definition at line 66 of file BatchTasksManagerProcess.py.

CONFIG_BATCH_DEFAULT_STRATEGY_AUTOCLEANUP_DELETE_RETRIES

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_BATCH_DEFAULT_STRATEGY_AUTOCLEANUP_DELETE_RETRIES = "BatchTask_autocleanup_DeleteRetries"

static

Definition at line 80 of file BatchTasksManagerProcess.py.

CONFIG_BATCH_DEFAULT_STRATEGY_AUTOCLEANUP_DELETE_TYPE

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_BATCH_DEFAULT_STRATEGY_AUTOCLEANUP_DELETE_TYPE = "BatchTask_autocleanup_DeleteType"

static

Definition at line 79 of file BatchTasksManagerProcess.py.

CONFIG_BATCH_DEFAULT_STRATEGY_AUTOCLEANUP_STATE

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_BATCH_DEFAULT_STRATEGY_AUTOCLEANUP_STATE = "BatchTask_autocleanup_State"

static

Definition at line 81 of file BatchTasksManagerProcess.py.

CONFIG_BATCH_DEFAULT_STRATEGY_AUTOCLEANUP_TTL

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_BATCH_DEFAULT_STRATEGY_AUTOCLEANUP_TTL = "BatchTask_autocleanup_TTL"

static

Definition at line 78 of file BatchTasksManagerProcess.py.

CONFIG_BATCH_DEFAULT_STRATEGY_CPU_LOAD_MAX

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_BATCH_DEFAULT_STRATEGY_CPU_LOAD_MAX = "BatchTask_CPU_LOAD_MAX"

static

Definition at line 74 of file BatchTasksManagerProcess.py.

CONFIG_BATCH_DEFAULT_STRATEGY_IO_WAIT_MAX

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_BATCH_DEFAULT_STRATEGY_IO_WAIT_MAX = "BatchTask_IO_WAIT_MAX"

static

Definition at line 73 of file BatchTasksManagerProcess.py.

CONFIG_BATCH_DEFAULT_STRATEGY_RAM_FREE_MIN

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_BATCH_DEFAULT_STRATEGY_RAM_FREE_MIN = "BatchTask_RAM_FREE_MIN"

static

Definition at line 75 of file BatchTasksManagerProcess.py.

CONFIG_BATCH_DEFAULT_STRATEGY_RETRY

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_BATCH_DEFAULT_STRATEGY_RETRY = "BatchTask_RETRY"

static

Definition at line 77 of file BatchTasksManagerProcess.py.

CONFIG_BATCH_DEFAULT_STRATEGY_STRATEGY_RDELAY

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_BATCH_DEFAULT_STRATEGY_STRATEGY_RDELAY = "BatchTask_RDELAY"

static

Definition at line 76 of file BatchTasksManagerProcess.py.

CONFIG_BATCH_MAX_TASKS

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_BATCH_MAX_TASKS = "BatchDefaultMaxTasks"

static

Definition at line 62 of file BatchTasksManagerProcess.py.

CONFIG_BATCH_MAX_TIME

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_BATCH_MAX_TIME = "BatchMaxExecutionTime"

static

Definition at line 69 of file BatchTasksManagerProcess.py.

CONFIG_BATCH_MAX_URLS

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_BATCH_MAX_URLS = "BatchDefaultMaxURLs"

static

Definition at line 60 of file BatchTasksManagerProcess.py.

CONFIG_BATCH_ORDER_BY_URLS

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_BATCH_ORDER_BY_URLS = "BatchDefaultOrderByURLs"

static

Definition at line 61 of file BatchTasksManagerProcess.py.

CONFIG_BATCH_QUEUE_PERIOD

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_BATCH_QUEUE_PERIOD = "BatchQueueProcessingPeriod"

static

Definition at line 63 of file BatchTasksManagerProcess.py.

CONFIG_BATCH_QUEUE_TASK_CHECK_METHOD

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_BATCH_QUEUE_TASK_CHECK_METHOD = "BatchQueueTaskCheckStateMethod"

static

Definition at line 65 of file BatchTasksManagerProcess.py.

CONFIG_BATCH_QUEUE_TASK_TTL

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_BATCH_QUEUE_TASK_TTL = "BatchQueueTaskTTL"

static

Definition at line 64 of file BatchTasksManagerProcess.py.

CONFIG_BATCH_REMOVE_UNPROCESSED_ITEMS

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_BATCH_REMOVE_UNPROCESSED_ITEMS = "RemoveUnprocessedBatchItems"

static

Definition at line 70 of file BatchTasksManagerProcess.py.

CONFIG_BATCH_WHERE_SITES

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_BATCH_WHERE_SITES = "BatchDefaultWhereSites"

static

Definition at line 68 of file BatchTasksManagerProcess.py.

CONFIG_BATCH_WHERE_URLS

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_BATCH_WHERE_URLS = "BatchDefaultWhereURLs"

static

Definition at line 67 of file BatchTasksManagerProcess.py.

CONFIG_DRCE_DB_APP_NAME

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_DRCE_DB_APP_NAME = "DRCEDBAppName"

static

Definition at line 55 of file BatchTasksManagerProcess.py.

CONFIG_DRCE_PROCESSOR_APP_NAME

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_DRCE_PROCESSOR_APP_NAME = "DRCEProcessorAppName"

static

Definition at line 54 of file BatchTasksManagerProcess.py.

CONFIG_DTMD_HOST

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_DTMD_HOST = "DTMDHost"

static

Definition at line 48 of file BatchTasksManagerProcess.py.

CONFIG_DTMD_PORT

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_DTMD_PORT = "DTMDPort"

static

Definition at line 49 of file BatchTasksManagerProcess.py.

CONFIG_DTMD_TIMEOUT

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_DTMD_TIMEOUT = "DTMDTimeout"

static

Definition at line 50 of file BatchTasksManagerProcess.py.

CONFIG_POLLING_TIMEOUT

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_POLLING_TIMEOUT = "PollingTimeout"

static

Definition at line 51 of file BatchTasksManagerProcess.py.

CONFIG_PROCESS_MODE

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_PROCESS_MODE = "ProcessingMode"

static

Definition at line 57 of file BatchTasksManagerProcess.py.

CONFIG_PROCESS_PERIOD

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_PROCESS_PERIOD = "ProcessingPeriod"

static

Definition at line 56 of file BatchTasksManagerProcess.py.

CONFIG_SERVER

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_SERVER = "server"

static

Definition at line 47 of file BatchTasksManagerProcess.py.

CONFIG_SITES_MANAGER_CLIENT

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_SITES_MANAGER_CLIENT = "clientSitesManager"

static

Definition at line 52 of file BatchTasksManagerProcess.py.

CONFIG_TASK_DTM_NAME_PROCESS

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_TASK_DTM_NAME_PROCESS = "BatchTaskDTMNameProcess"

static

Definition at line 84 of file BatchTasksManagerProcess.py.

CONFIG_TASK_DTM_TYPE_PROCESS

string dc.BatchTasksManagerProcess.BatchTasksManagerProcess.CONFIG_TASK_DTM_TYPE_PROCESS = "BatchTaskDTMTypeProcess"

static

Definition at line 86 of file BatchTasksManagerProcess.py.

DTM_TASK_CHECK_STATE_METHOD_STATE

int dc.BatchTasksManagerProcess.BatchTasksManagerProcess.DTM_TASK_CHECK_STATE_METHOD_STATE = 1

static

Definition at line 44 of file BatchTasksManagerProcess.py.

DTM_TASK_CHECK_STATE_METHOD_STATUS

int dc.BatchTasksManagerProcess.BatchTasksManagerProcess.DTM_TASK_CHECK_STATE_METHOD_STATUS = 0

static

Definition at line 43 of file BatchTasksManagerProcess.py.

dtmdConnection

dc.BatchTasksManagerProcess.BatchTasksManagerProcess.dtmdConnection

Definition at line 154 of file BatchTasksManagerProcess.py.

dtmTasksQueue

dc.BatchTasksManagerProcess.BatchTasksManagerProcess.dtmTasksQueue

Definition at line 169 of file BatchTasksManagerProcess.py.

processBatchQueuelLastTs

dc.BatchTasksManagerProcess.BatchTasksManagerProcess.processBatchQueuelLastTs

Definition at line 182 of file BatchTasksManagerProcess.py.

processProcessingLastTs

dc.BatchTasksManagerProcess.BatchTasksManagerProcess.processProcessingLastTs

Definition at line 174 of file BatchTasksManagerProcess.py.

serverName

dc.BatchTasksManagerProcess.BatchTasksManagerProcess.serverName

Definition at line 125 of file BatchTasksManagerProcess.py.

---

The documentation for this class was generated from the following file:

• sources/hce/dc/BatchTasksManagerProcess.py