HCE Project Python language Distributed Tasks Manager Application, Distributed Crawler Application and client API bindings.  2.0.0-chaika
Hierarchical Cluster Engine Python language binding
dtm.TasksDataManager.TasksDataManager Class Reference

Class contents TasksDataManager module implementation. More...

Inheritance diagram for dtm.TasksDataManager.TasksDataManager:
Collaboration diagram for dtm.TasksDataManager.TasksDataManager:

Public Member Functions

def __init__ (self, config, connectionBuilder=None)
 constructor initialise all class variable and recieve config as param More...
 
def connectionInit (self)
 connectionInit method initializes internal inproc connection More...
 
def dbInit (self)
 dbInit method initializes internal database API More...
 
def dbiProcessing (self, data, eventType)
 dbiProcessing method Method contains main processing with database API More...
 
def getResponceEventType (self, eventType)
 Get responce event type method. More...
 
def eventProcessing (self, event)
 eventProcessing method Method contains main processing with incoming events More...
 
def badEventType (self, msg, event)
 eventProcessing method Method contains main error processing for incoming events More...
 
def onNewTask (self, event)
 Callbacks methods Event callbacks. More...
 
def onFetchTask (self, event)
 
def onUpdateTask (self, event)
 
def onDeleteTask (self, event)
 
def onInsertEEResponse (self, event)
 
def onFetchEEResponse (self, event)
 
def onDeleteEEResponse (self, event)
 
- Public Member Functions inherited from app.BaseServerManager.BaseServerManager
def __init__ (self, poller_manager=None, admin_connection=None, conectionLightBuilder=None, exceptionForward=False, dumpStatVars=True)
 constructor More...
 
def addConnection (self, name, connection)
 
def setEventHandler (self, eventType, handler)
 set event handler rewrite the current handler for eventType More...
 
def send (self, connect_name, event)
 send event More...
 
def reply (self, event, reply_event)
 wrapper for sending event in reply for event More...
 
def poll (self)
 poll function polling connections receive as multipart msg, the second argument is pickled pyobj More...
 
def process (self, event)
 process event call the event handler method that was set by user or on_unhandled_event method if not set More...
 
def run (self)
 
def is_connection_registered (self, name)
 check is a connection was registered in a instance of BaseServerManager i object More...
 
def on_poll_timeout (self)
 function will call every time when ConnectionTimeout exception arrive More...
 
def on_unhandled_event (self, event)
 function will call every time when arrive doesn't set handler for event type of event.evenType More...
 
def build_poller_list (self)
 
def clear_poller (self)
 
def onAdminState (self, event)
 onAdminState event handler process admin SHUTDOWN command More...
 
def onAdminFetchStatData (self, event)
 onAdminState event handler process admin command More...
 
def onAdminSuspend (self, event)
 onAdminState event handler process admin command More...
 
def getStatDataFields (self, fields)
 getStatDataFields returns stat data from storage More...
 
def getSystemStat (self)
 getSystemStat returns stat data for system indicators: RAMV, RAMR and CPU More...
 
def getConfigVarsFields (self, fields)
 getConfigVarsFields returns config vars from storage More...
 
def onAdminGetConfigVars (self, event)
 onAdminGetConfigVars event handler process getConfigVars admin command, fill and return config vars array from internal storage More...
 
def onAdminSetConfigVars (self, event)
 onAdminSetConfigVars event handler process setConfigVars admin command More...
 
def setConfigVars (self, setConfigVars)
 processSetConfigVars sets config vars in storage More...
 
def sendAdminReadyEvent (self)
 send ready event to notify adminInterfaceService More...
 
def createLogMsg (self, event)
 from string message from event object More...
 
def initStatFields (self, connect_name)
 add record in statFields More...
 
def updateStatField (self, field_name, value, operation=STAT_FIELDS_OPERATION_ADD)
 update values of stat field - default sum More...
 
def processSpecialConfigVars (self, name, value)
 send ready event to notify adminInterfaceService More...
 
def getLogLevel (self)
 Get log level from first of existing loggers. More...
 
def setLogLevel (self, level)
 Set log level for all loggers. More...
 
def saveStatVarsDump (self)
 Save stat vars in json file. More...
 
def loadStatVarsDump (self)
 Load stat vars in json file. More...
 
def getStatVarsDumpFileName (self)
 Get stat vars file name. More...
 
def createDBIDict (self, configParser)
 

Public Attributes

 moduleName
 
 servIndex
 
 eventBuilder
 
 connectionBuilder
 
 config
 
 dbi
 
- Public Attributes inherited from app.BaseServerManager.BaseServerManager
 dumpStatVars
 
 poller_manager
 
 eventBuilder
 
 exit_flag
 
 pollTimeout
 
 connections
 
 event_handlers
 
 statFields
 stat fields container More...
 
 configVars
 
 exceptionForward
 

Static Public Attributes

string TASK_DATA_MANAGER_SERV_NAME = "Server"
 
string TASK_DATA_MANAGER_SERV_CONFIG_NAME = "ServerName"
 
- Static Public Attributes inherited from app.BaseServerManager.BaseServerManager
string ADMIN_CONNECT_ENDPOINT = "Admin"
 
string ADMIN_CONNECT_CLIENT = "Admin"
 
int POLL_TIMEOUT_DEFAULT = 3000
 
int STAT_FIELDS_OPERATION_ADD = 0
 
int STAT_FIELDS_OPERATION_SUB = 1
 
int STAT_FIELDS_OPERATION_SET = 2
 
int STAT_FIELDS_OPERATION_INIT = 3
 
string POLL_TIMEOUT_CONFIG_VAR_NAME = "POLL_TIMEOUT"
 
string LOG_LEVEL_CONFIG_VAR_NAME = "LOG_LEVEL"
 
string STAT_DUMPS_DEFAULT_DIR = "/tmp/"
 
string STAT_DUMPS_DEFAULT_NAME = "%APP_NAME%_%CLASS_NAME%_stat_vars.dump"
 
dictionary LOGGERS_NAMES = {APP_CONSTS.LOGGER_NAME, "dc", "dtm", "root", ""}
 

Detailed Description

Class contents TasksDataManager module implementation.

Definition at line 37 of file TasksDataManager.py.

Constructor & Destructor Documentation

◆ __init__()

def dtm.TasksDataManager.TasksDataManager.__init__ (   self,
  config,
  connectionBuilder = None 
)

constructor initialise all class variable and recieve config as param

Definition at line 46 of file TasksDataManager.py.

46  def __init__(self, config, connectionBuilder=None):
47  BaseServerManager.__init__(self)
48  self.moduleName = self.__class__.__name__
49  self.servIndex = 1
50  self.eventBuilder = EventBuilder()
51  self.setEventHandler(EVENT.NEW_TASK, self.onNewTask)
52  self.setEventHandler(EVENT.FETCH_TASK_DATA, self.onFetchTask)
53  self.setEventHandler(EVENT.UPDATE_TASK, self.onUpdateTask)
54  self.setEventHandler(EVENT.DELETE_TASK_DATA, self.onDeleteTask)
55  self.setEventHandler(EVENT.INSERT_EE_DATA, self.onInsertEEResponse)
56  self.setEventHandler(EVENT.FETCH_EE_DATA, self.onFetchEEResponse)
57  self.setEventHandler(EVENT.DELETE_EE_DATA, self.onDeleteEEResponse)
58  self.connectionBuilder = connectionBuilder
59  self.config = config
60  self.dbi = None
61  self.connectionInit()
62  self.dbInit()
63 
64 
def __init__(self)
constructor
Definition: UIDGenerator.py:19

Member Function Documentation

◆ badEventType()

def dtm.TasksDataManager.TasksDataManager.badEventType (   self,
  msg,
  event 
)

eventProcessing method Method contains main error processing for incoming events

Definition at line 177 of file TasksDataManager.py.

177  def badEventType(self, msg, event):
178  errorStr = msg + str(event.eventType)
179  logger.error(LogFormatterEvent(event, [], errorStr))
180 # raise Exception(errorStr)
181 
182 
Here is the caller graph for this function:

◆ connectionInit()

def dtm.TasksDataManager.TasksDataManager.connectionInit (   self)

connectionInit method initializes internal inproc connection

Definition at line 67 of file TasksDataManager.py.

67  def connectionInit(self):
68  if self.connectionBuilder is None:
69  self.connectionBuilder = ConnectionBuilderLight()
70  try:
71  localConnection = self.connectionBuilder.build(transport.Consts.SERVER_CONNECT, \
72  self.config.get(self.moduleName, self.TASK_DATA_MANAGER_SERV_CONFIG_NAME))
73  self.addConnection(self.TASK_DATA_MANAGER_SERV_NAME + str(self.servIndex), localConnection)
74  self.servIndex = self.servIndex + 1
75  except ConfigParser.NoSectionError:
76  logger.error(">>> TasksDataManager can't read config - Section Error")
77  except ConfigParser.NoOptionError:
78  logger.error(">>> TasksDataManager can't read config - Option Error")
79 
80 
Here is the call graph for this function:
Here is the caller graph for this function:

◆ dbInit()

def dtm.TasksDataManager.TasksDataManager.dbInit (   self)

dbInit method initializes internal database API

Definition at line 83 of file TasksDataManager.py.

83  def dbInit(self):
84  if self.config != None:
85  dic = None
86  dic = dict(self.config.items(CONSTANTS.DB_CONFIG_SECTION))
87  if dic != None:
88  self.dbi = DBI(dic)
89 
90 

◆ dbiProcessing()

def dtm.TasksDataManager.TasksDataManager.dbiProcessing (   self,
  data,
  eventType 
)

dbiProcessing method Method contains main processing with database API

Definition at line 93 of file TasksDataManager.py.

93  def dbiProcessing(self, data, eventType):
94  # variable for result
95  ret = None
96 
97  if eventType == EVENT.NEW_TASK:
98  self.dbi.insert(data)
99  elif eventType == EVENT.FETCH_TASK_DATA:
100  ret = self.dbi.fetch(data, "id=%s" % data.id)
101  elif eventType == EVENT.UPDATE_TASK:
102  self.dbi.update(data, "id=%s" % data.id)
103  elif eventType == EVENT.DELETE_TASK_DATA:
104  self.dbi.delete(data, "id=%s" % data.id)
105  elif eventType == EVENT.INSERT_EE_DATA:
106  self.dbi.insert(data)
107  elif eventType == EVENT.FETCH_EE_DATA:
108  ret = self.dbi.fetch(data, "id=%s" % data.id)
109  elif eventType == EVENT.DELETE_EE_DATA:
110  self.dbi.delete(data, "id=%s" % data.id)
111 
112  return ret
113 
114 
Here is the caller graph for this function:

◆ eventProcessing()

def dtm.TasksDataManager.TasksDataManager.eventProcessing (   self,
  event 
)

eventProcessing method Method contains main processing with incoming events

Definition at line 143 of file TasksDataManager.py.

143  def eventProcessing(self, event):
144  serializeStr = pickle.dumps(event.eventObj)
145  if event.eventType == EVENT.NEW_TASK or \
146  event.eventType == EVENT.FETCH_TASK_DATA or \
147  event.eventType == EVENT.UPDATE_TASK or \
148  event.eventType == EVENT.DELETE_TASK_DATA:
149  data = TasksDataTable()
150  else:
151  data = EEResponsesTable() # pylint: disable=R0204
152  data.id = event.eventObj.id
153  data.data = serializeStr
154  dbiRet = None
155  retEventType = None
156  try:
157  retEventType = self.getResponceEventType(event.eventType)
158  dbiRet = GeneralResponse()
159  res = self.dbiProcessing(data, event.eventType)
160  if res is not None and len(res) > 0 and res[0] != None and hasattr(res[0], 'data') and res[0].data != None:
161  dbiRet = pickle.loads(str(res[0].data))
162 
163  except DBIErr as err:
164  logger.error("DB error: %s", str(err))
165  dbiRet.errorCode = err.errCode
166  dbiRet.errorMessage = "Some DB error in TasksDataManager.eventProcessing [" + str(err) + "]"
167  except Exception, err:
168  logger.error("Error: %s", str(err))
169  dbiRet.errorMessage = "Some error in TasksDataManager.eventProcessing [" + str(err) + "]"
170 
171  retEvent = self.eventBuilder.build(retEventType, dbiRet)
172  return retEvent
173 
174 
Here is the call graph for this function:
Here is the caller graph for this function:

◆ getResponceEventType()

def dtm.TasksDataManager.TasksDataManager.getResponceEventType (   self,
  eventType 
)

Get responce event type method.

Parameters
eventType- event type
Returns
responce event type

Definition at line 119 of file TasksDataManager.py.

119  def getResponceEventType(self, eventType):
120  #variable for result
121  retEventType = EVENT.GENERAL_RESPONSE
122 
123  if eventType == EVENT.NEW_TASK:
124  retEventType = EVENT.NEW_TASK_RESPONSE
125  elif eventType == EVENT.FETCH_TASK_DATA:
126  retEventType = EVENT.FETCH_TASK_DATA_RESPONSE
127  elif eventType == EVENT.UPDATE_TASK:
128  retEventType = EVENT.UPDATE_TASK_RESPONSE
129  elif eventType == EVENT.DELETE_TASK_DATA:
130  retEventType = EVENT.DELETE_TASK_DATA_RESPONSE
131  elif eventType == EVENT.INSERT_EE_DATA:
132  retEventType = EVENT.INSERT_EE_DATA_RESPONSE
133  elif eventType == EVENT.FETCH_EE_DATA:
134  retEventType = EVENT.FETCH_EE_DATA_RESPONSE
135  elif eventType == EVENT.DELETE_EE_DATA:
136  retEventType = EVENT.DELETE_EE_DATA_RESPONSE
137 
138  return retEventType
139 
140 
Here is the caller graph for this function:

◆ onDeleteEEResponse()

def dtm.TasksDataManager.TasksDataManager.onDeleteEEResponse (   self,
  event 
)

Definition at line 233 of file TasksDataManager.py.

233  def onDeleteEEResponse(self, event):
234  logger.debug(LogFormatterEvent(event, [], ">>> TasksDataManager [DELETE_EE_REQUEST] Handler start"))
235  if event.eventType != EVENT.DELETE_EE_DATA:
236  self.badEventType(">>> Wrong Event type [DELETE_EE_REQUEST] != ", event)
237  self.reply(event, self.eventProcessing(event))
238  logger.debug(LogFormatterEvent(event, [], ">>> TasksDataManager [DELETE_EE_REQUEST] Handler finish"))
239 
Here is the call graph for this function:

◆ onDeleteTask()

def dtm.TasksDataManager.TasksDataManager.onDeleteTask (   self,
  event 
)

Definition at line 209 of file TasksDataManager.py.

209  def onDeleteTask(self, event):
210  logger.debug(LogFormatterEvent(event, [], ">>> TasksDataManager [DELETE_TASK_DATA_REQUEST] Handler start"))
211  if event.eventType != EVENT.DELETE_TASK_DATA:
212  self.badEventType(">>> Wrong Event type [DELETE_TASK_DATA_REQUEST] != ", event)
213  self.reply(event, self.eventProcessing(event))
214  logger.debug(LogFormatterEvent(event, [], ">>> TasksDataManager [DELETE_TASK_DATA_REQUEST] Handler finish"))
215 
216 
Here is the call graph for this function:
Here is the caller graph for this function:

◆ onFetchEEResponse()

def dtm.TasksDataManager.TasksDataManager.onFetchEEResponse (   self,
  event 
)

Definition at line 225 of file TasksDataManager.py.

225  def onFetchEEResponse(self, event):
226  logger.debug(LogFormatterEvent(event, [], ">>> TasksDataManager [FETCH_EE_REQUEST] Handler start"))
227  if event.eventType != EVENT.FETCH_EE_DATA:
228  self.badEventType(">>> Wrong Event type [FETCH_EE_REQUEST] != ", event)
229  self.reply(event, self.eventProcessing(event))
230  logger.debug(LogFormatterEvent(event, [], ">>> TasksDataManager [FETCH_EE_REQUEST] Handler finish"))
231 
232 
Here is the call graph for this function:

◆ onFetchTask()

def dtm.TasksDataManager.TasksDataManager.onFetchTask (   self,
  event 
)

Definition at line 193 of file TasksDataManager.py.

193  def onFetchTask(self, event):
194  logger.debug(LogFormatterEvent(event, [], ">>> TasksDataManager [FETCH_TASK_REQUEST] Handler start"))
195  if event.eventType != EVENT.FETCH_TASK_DATA:
196  self.badEventType(">>> Wrong Event type [FETCH_TASK_REQUEST] != ", event)
197  self.reply(event, self.eventProcessing(event))
198  logger.debug(LogFormatterEvent(event, [], ">>> TasksDataManager [FETCH_TASK_REQUEST] Handler finish"))
199 
200 
Here is the call graph for this function:

◆ onInsertEEResponse()

def dtm.TasksDataManager.TasksDataManager.onInsertEEResponse (   self,
  event 
)

Definition at line 217 of file TasksDataManager.py.

217  def onInsertEEResponse(self, event):
218  logger.debug(LogFormatterEvent(event, [], ">>> TasksDataManager [INSERT_EE_REQUEST] Handler start"))
219  if event.eventType != EVENT.INSERT_EE_DATA:
220  self.badEventType(">>> Wrong Event type [INSERT_EE_REQUEST] != ", event)
221  self.reply(event, self.eventProcessing(event))
222  logger.debug(LogFormatterEvent(event, [], ">>> TasksDataManager [INSERT_EE_REQUEST] Handler finish"))
223 
224 
Here is the call graph for this function:

◆ onNewTask()

def dtm.TasksDataManager.TasksDataManager.onNewTask (   self,
  event 
)

Callbacks methods Event callbacks.

Definition at line 185 of file TasksDataManager.py.

185  def onNewTask(self, event):
186  logger.debug(LogFormatterEvent(event, [], ">>> TasksDataManager [NEW_TASK_REQUEST] Handler start"))
187  if event.eventType != EVENT.NEW_TASK:
188  self.badEventType(">>> Wrong Event type [NEW_TASK_REQUEST] != ", event)
189  self.reply(event, self.eventProcessing(event))
190  logger.debug(LogFormatterEvent(event, [], ">>> TasksDataManager [NEW_TASK_REQUEST] Handler finish"))
191 
192 
Here is the call graph for this function:

◆ onUpdateTask()

def dtm.TasksDataManager.TasksDataManager.onUpdateTask (   self,
  event 
)

Definition at line 201 of file TasksDataManager.py.

201  def onUpdateTask(self, event):
202  logger.debug(LogFormatterEvent(event, [], ">>> TasksDataManager [UPDATE_TASK_REQUEST] Handler start"))
203  if event.eventType != EVENT.UPDATE_TASK:
204  self.badEventType(">>> Wrong Event type [UPDATE_TASK_REQUEST] != ", event)
205  self.reply(event, self.eventProcessing(event))
206  logger.debug(LogFormatterEvent(event, [], ">>> TasksDataManager [UPDATE_TASK_REQUEST] Handler finish"))
207 
208 
Here is the call graph for this function:

Member Data Documentation

◆ config

dtm.TasksDataManager.TasksDataManager.config

Definition at line 59 of file TasksDataManager.py.

◆ connectionBuilder

dtm.TasksDataManager.TasksDataManager.connectionBuilder

Definition at line 58 of file TasksDataManager.py.

◆ dbi

dtm.TasksDataManager.TasksDataManager.dbi

Definition at line 60 of file TasksDataManager.py.

◆ eventBuilder

dtm.TasksDataManager.TasksDataManager.eventBuilder

Definition at line 50 of file TasksDataManager.py.

◆ moduleName

dtm.TasksDataManager.TasksDataManager.moduleName

Definition at line 48 of file TasksDataManager.py.

◆ servIndex

dtm.TasksDataManager.TasksDataManager.servIndex

Definition at line 49 of file TasksDataManager.py.

◆ TASK_DATA_MANAGER_SERV_CONFIG_NAME

string dtm.TasksDataManager.TasksDataManager.TASK_DATA_MANAGER_SERV_CONFIG_NAME = "ServerName"
static

Definition at line 41 of file TasksDataManager.py.

◆ TASK_DATA_MANAGER_SERV_NAME

string dtm.TasksDataManager.TasksDataManager.TASK_DATA_MANAGER_SERV_NAME = "Server"
static

Definition at line 40 of file TasksDataManager.py.


The documentation for this class was generated from the following file: