dc_crawler.DBTasksWrapper.DBTasksWrapper Class Reference

Inheritance diagram for dc_crawler.DBTasksWrapper.DBTasksWrapper:

Collaboration diagram for dc_crawler.DBTasksWrapper.DBTasksWrapper:

Public Member Functions
def	__init__ (self, cfgParser)
def	process (self, drceObject)
def	fieldsRecalculating (self, sites)
def	collectedURLsRecalculating (self, siteId)
def	customRequest (self, query, dbName, includeFieldsNames=dbi.EventObjects.CustomRequest.SQL_BY_INDEX)
def	urlUpdate (self, urlUpdateObject, criterionsWere=None, criterionsLimit=None, criterionsOrder=None)
def	urlStatus (self, urlStatusObject, useMd5Resolving=False)
def	urlContent (self, items)
def	putURLContent (self, urlPut_list)
def	urlNew (self, urlNewObject)
def	siteNewOrUpdate (self, siteObject, properties=None, filters=None, siteId=None, stype=dc.EventObjects.SiteUpdate)
def	siteStatus (self, siteStatusObject)
def	proxyStatus (self, proxyStatusObject)
def	proxyUpdate (self, proxyUpdateObject)
def	putAttributes (self, attributes)

Public Attributes
	dbTask
	fieldRecalculator
	rid
	affect_db

Detailed Description

Definition at line 25 of file DBTasksWrapper.py.

Constructor & Destructor Documentation

__init__()

def dc_crawler.DBTasksWrapper.DBTasksWrapper.__init__	(		self,
			cfgParser
	)

Definition at line 31 of file DBTasksWrapper.py.

  def __init__(self, cfgParser):
    self.dbTask = DBTasksManager(cfgParser)
    self.fieldRecalculator = DBFieldRecalculator()
    self.rid = 0
    self.affect_db = True

Member Function Documentation

collectedURLsRecalculating()

def dc_crawler.DBTasksWrapper.DBTasksWrapper.collectedURLsRecalculating	(		self,
			siteId
	)

Definition at line 59 of file DBTasksWrapper.py.

  def collectedURLsRecalculating(self, siteId):
    if self.affect_db:
      self.fieldRecalculator.updateCollectedURLs(siteId, self.dbTask.executeQuery)

customRequest()

def dc_crawler.DBTasksWrapper.DBTasksWrapper.customRequest	(		self,
			query,
			dbName,
			includeFieldsNames = dbi.EventObjects.CustomRequest.SQL_BY_INDEX
	)

Definition at line 69 of file DBTasksWrapper.py.

  def customRequest(self, query, dbName, includeFieldsNames=dbi.EventObjects.CustomRequest.SQL_BY_INDEX):
    ret = None
    if self.affect_db:
      customObject = dbi.EventObjects.CustomRequest(self.rid, query, dbName)
      customObject.includeFieldsNames = includeFieldsNames
      drceObject = dc.Constants.DRCESyncTasksCover(dc.Constants.EVENT_TYPES.SQL_CUSTOM, customObject)
      retDRCE = self.dbTask.process(drceObject)
      if retDRCE.eventType == dc.Constants.EVENT_TYPES.SQL_CUSTOM_RESPONSE:
        if retDRCE.eventObject.rid == self.rid:
          if retDRCE.eventObject.errString is None:
            ret = retDRCE.eventObject.result
          else:
            logger.error("SQL_CUSTOM_RESPONSE >>> Resonse error = " + retDRCE.eventObject.errString)
        else:
          logger.error("SQL_CUSTOM_RESPONSE >>> Wrong response rid")
      else:
        logger.error("SQL_CUSTOM_RESPONSE >>> Wrong response type")
    return ret

Here is the call graph for this function:

fieldsRecalculating()

def dc_crawler.DBTasksWrapper.DBTasksWrapper.fieldsRecalculating	(		self,
			sites
	)

Definition at line 46 of file DBTasksWrapper.py.

  def fieldsRecalculating(self, sites):
    if self.affect_db:
      fieldRecalculatingObjList = []
      for site in sites:
        fieldRecalculatingObjList.append(dc.EventObjects.FieldRecalculatorObj(site))
      drceObject = dc.Constants.DRCESyncTasksCover(dc.Constants.EVENT_TYPES.FIELD_RECALCULATE,
                                                   fieldRecalculatingObjList)
      retDRCE = self.dbTask.process(drceObject)
      del retDRCE

Here is the call graph for this function:

process()

def dc_crawler.DBTasksWrapper.DBTasksWrapper.process	(		self,
			drceObject
	)

Definition at line 40 of file DBTasksWrapper.py.

  def process(self, drceObject):
    return self.dbTask.process(drceObject)

Here is the caller graph for this function:

proxyStatus()

def dc_crawler.DBTasksWrapper.DBTasksWrapper.proxyStatus	(		self,
			proxyStatusObject
	)

Definition at line 223 of file DBTasksWrapper.py.

  def proxyStatus(self, proxyStatusObject):
    # variable for result
    ret = None
    if self.affect_db:
      if proxyStatusObject is None:
        logger.error("PROXY_STATUS: proxyObject is None!")
      else:
        if isinstance(proxyStatusObject, list):
          proxyStatusObjectList = proxyStatusObject
        else:
          proxyStatusObjectList = [proxyStatusObject]

        drceObject = dc.Constants.DRCESyncTasksCover(dc.Constants.EVENT_TYPES.PROXY_STATUS, proxyStatusObjectList)
        retDRCE = self.dbTask.process(drceObject)
        if retDRCE.eventType == dc.Constants.EVENT_TYPES.PROXY_STATUS_RESPONSE:
          ret = retDRCE.eventObject
        else:
          logger.error("PROXY_STATUS_RESPONSE: Wrong type of response object!")

    return ret

Here is the call graph for this function:

proxyUpdate()

def dc_crawler.DBTasksWrapper.DBTasksWrapper.proxyUpdate	(		self,
			proxyUpdateObject
	)

Definition at line 249 of file DBTasksWrapper.py.

  def proxyUpdate(self, proxyUpdateObject):
    # variable for result
    ret = None
    if self.affect_db:
      if proxyUpdateObject is not None:
        if isinstance(proxyUpdateObject, list):
          proxyUpdateObjectList = proxyUpdateObject
        else:
          proxyUpdateObjectList = [proxyUpdateObject]

        drceObject = dc.Constants.DRCESyncTasksCover(dc.Constants.EVENT_TYPES.PROXY_UPDATE, proxyUpdateObjectList)
        retDRCE = self.dbTask.process(drceObject)
        if retDRCE.eventType == dc.Constants.EVENT_TYPES.PROXY_UPDATE_RESPONSE:
          ret = retDRCE.eventObject
        else:
          logger.error("PROXY_UPDATE_RESPONSE >>> Wrong response type")
      else:
        logger.error("PROXY_UPDATE >>> Not enough incoming data")

    return ret

Here is the call graph for this function:

putAttributes()

def dc_crawler.DBTasksWrapper.DBTasksWrapper.putAttributes	(		self,
			attributes
	)

Definition at line 275 of file DBTasksWrapper.py.

  def putAttributes(self, attributes):

    attributesList = attributes if isinstance(attributes, list) else [attributes]

    drceSyncTasksCoverObj = dc.Constants.DRCESyncTasksCover(dc.Constants.EVENT_TYPES.ATTR_SET, attributesList)
    responseDRCESyncTasksCover = self.dbTask.process(drceSyncTasksCoverObj)
    if responseDRCESyncTasksCover.eventType != dc.Constants.EVENT_TYPES.ATTR_SET_RESPONSE:
      logger.error("Operation 'ATTR_SET' has error: Wrong response type")

putURLContent()

def dc_crawler.DBTasksWrapper.DBTasksWrapper.putURLContent	(		self,
			urlPut_list
	)

Definition at line 142 of file DBTasksWrapper.py.

  def putURLContent(self, urlPut_list):
    ret = None

    drceSyncTasksCoverObj = dc.Constants.DRCESyncTasksCover(dc.Constants.EVENT_TYPES.URL_PUT, urlPut_list)
    responseDRCESyncTasksCover = self.dbTask.process(drceSyncTasksCoverObj)
    if responseDRCESyncTasksCover.eventType == dc.Constants.EVENT_TYPES.URL_PUT_RESPONSE:
      for obj in responseDRCESyncTasksCover.eventObject:
        logger.debug("URL_PUT_RESPONSE: %s", varDump(obj))
    else:
      logger.error("URL_PUT_RESPONSE >>> Wrong response type")

    if responseDRCESyncTasksCover.eventType != dc.Constants.EVENT_TYPES.URL_PUT_RESPONSE:
      logger.error("URL_PUT_RESPONSE >>> Wrong response type")

    return ret

Here is the call graph for this function:

siteNewOrUpdate()

def dc_crawler.DBTasksWrapper.DBTasksWrapper.siteNewOrUpdate	(		self,
			siteObject,
			properties = None,
			filters = None,
			siteId = None,
			stype = dc.EventObjects.SiteUpdate
	)

Definition at line 175 of file DBTasksWrapper.py.

  def siteNewOrUpdate(self, siteObject, properties=None, filters=None, siteId=None, stype=dc.EventObjects.SiteUpdate):
    ret = []
    if self.affect_db:
      if stype == dc.EventObjects.Site:
        reqType = dc.Constants.EVENT_TYPES.SITE_NEW
        respType = dc.Constants.EVENT_TYPES.SITE_NEW_RESPONSE
      elif stype == dc.EventObjects.SiteUpdate:
        reqType = dc.Constants.EVENT_TYPES.SITE_UPDATE
        respType = dc.Constants.EVENT_TYPES.SITE_UPDATE_RESPONSE
      if siteObject is None:
        if siteId is not None:
          siteObject = stype(siteId)
          if properties is not None:
            siteObject.properties = properties
          if filters is not None:
            siteObject.filters = filters
      if siteObject is not None:
        drceObject = dc.Constants.DRCESyncTasksCover(reqType, siteObject)
        retDRCE = self.dbTask.process(drceObject)
        logger.debug("SITE_NEW_UPDATE_RESPONSE  retDRCE: " + varDump(retDRCE))
        if retDRCE.eventType == respType:
          ret = retDRCE.eventObject.statuses
        else:
          logger.error("SITE_NEW_UPDATE_RESPONSE >>> Wrong response type")
      else:
        logger.error("SITE_NEW_UPDATE >>> siteObject is None!")
    return ret

Here is the call graph for this function:

siteStatus()

def dc_crawler.DBTasksWrapper.DBTasksWrapper.siteStatus	(		self,
			siteStatusObject
	)

Definition at line 204 of file DBTasksWrapper.py.

  def siteStatus(self, siteStatusObject):
    ret = None
    if self.affect_db:
      if siteStatusObject is not None:
        drceObject = dc.Constants.DRCESyncTasksCover(dc.Constants.EVENT_TYPES.SITE_STATUS, siteStatusObject)
        retDRCE = self.dbTask.process(drceObject)
        if retDRCE.eventType == dc.Constants.EVENT_TYPES.SITE_STATUS_RESPONSE:
          ret = retDRCE.eventObject
        else:
          logger.error("SITE_STATUS_RESPONSE >>> Wrong response type")
      else:
        logger.error("SITE_STATUS >>> Not enough incoming data")
    return ret

Here is the call graph for this function:

urlContent()

def dc_crawler.DBTasksWrapper.DBTasksWrapper.urlContent	(		self,
			items
	)

Definition at line 131 of file DBTasksWrapper.py.

  def urlContent(self, items):
    ret = None
    drceSyncTasksCoverObj = dc.Constants.DRCESyncTasksCover(dc.Constants.EVENT_TYPES.URL_CONTENT, items)
    retDRCE = self.dbTask.process(drceSyncTasksCoverObj)
    if retDRCE.eventType == dc.Constants.EVENT_TYPES.URL_CONTENT_RESPONSE:
      ret = retDRCE.eventObject
    else:
      logger.error("URL_CONTENT_RESPONSE >>> Wrong response type")
    return ret

Here is the call graph for this function:

urlNew()

def dc_crawler.DBTasksWrapper.DBTasksWrapper.urlNew	(		self,
			urlNewObject
	)

Definition at line 159 of file DBTasksWrapper.py.

  def urlNew(self, urlNewObject):
    ret = 0
    if self.affect_db:
      if isinstance(urlNewObject, list):
        urlNewList = urlNewObject
      else:
        urlNewList = [urlNewObject]
      drceObject = dc.Constants.DRCESyncTasksCover(dc.Constants.EVENT_TYPES.URL_NEW, urlNewList)
      retDRCE = self.dbTask.process(drceObject)
      if retDRCE.eventType == dc.Constants.EVENT_TYPES.URL_NEW_RESPONSE:
        ret = len([i for i in retDRCE.eventObject.statuses if i == 0])
      else:
        logger.error("URL_NEW_RESPONSE >>> Wrong response type")
    return ret

Here is the call graph for this function:

Here is the caller graph for this function:

urlStatus()

def dc_crawler.DBTasksWrapper.DBTasksWrapper.urlStatus	(		self,
			urlStatusObject,
			useMd5Resolving = False
	)

Definition at line 117 of file DBTasksWrapper.py.

  def urlStatus(self, urlStatusObject, useMd5Resolving=False):
    ret = []
    if self.affect_db:
      if useMd5Resolving:
        urlStatusObject.urlType = dc.EventObjects.URLStatus.URL_TYPE_MD5
      drceObject = dc.Constants.DRCESyncTasksCover(dc.Constants.EVENT_TYPES.URL_STATUS, [urlStatusObject])
      retDRCE = self.dbTask.process(drceObject)
      if retDRCE.eventType == dc.Constants.EVENT_TYPES.URL_STATUS_RESPONSE:
        ret = retDRCE.eventObject
      else:
        logger.error("URL_STATUS_RESPONSE >>> Wrong response type")
    return ret

Here is the call graph for this function:

urlUpdate()

def dc_crawler.DBTasksWrapper.DBTasksWrapper.urlUpdate	(		self,
			urlUpdateObject,
			criterionsWere = None,
			criterionsLimit = None,
			criterionsOrder = None
	)

Definition at line 89 of file DBTasksWrapper.py.

  def urlUpdate(self, urlUpdateObject, criterionsWere=None, criterionsLimit=None, criterionsOrder=None):
    ret = 0
    if self.affect_db:
      if isinstance(urlUpdateObject, list):
        urlUpdateObjectList = urlUpdateObject
      else:
        urlUpdateObjectList = [urlUpdateObject]
        if criterionsWere is not None or criterionsLimit is not None or criterionsOrder is not None:
          urlUpdateObject.criterions = {}
        if criterionsWere is not None:
          urlUpdateObject.criterions[dc.EventObjects.URLFetch.CRITERION_WHERE] = criterionsWere
        if criterionsLimit is not None:
          urlUpdateObject.criterions[dc.EventObjects.URLFetch.CRITERION_LIMIT] = criterionsLimit
        if criterionsOrder is not None:
          urlUpdateObject.criterions[dc.EventObjects.URLFetch.CRITERION_ORDER] = criterionsOrder

      drceObject = dc.Constants.DRCESyncTasksCover(dc.Constants.EVENT_TYPES.URL_UPDATE, urlUpdateObjectList)
      retDRCE = self.dbTask.process(drceObject)
      if retDRCE.eventType == dc.Constants.EVENT_TYPES.URL_UPDATE_RESPONSE:
        if hasattr(retDRCE.eventObject.statuses, '__iter__') and len(retDRCE.eventObject.statuses) > 0 and \
        retDRCE.eventObject.statuses[0] is False:
          logger.error("URL_UPDATE_RESPONSE >>> Operation failure, look db-task log")
        ret = len([i for i in retDRCE.eventObject.statuses if i])
      else:
        logger.error("URL_UPDATE_RESPONSE >>> Wrong response type")
    return ret

Here is the call graph for this function:

Member Data Documentation

affect_db

dc_crawler.DBTasksWrapper.DBTasksWrapper.affect_db

Definition at line 35 of file DBTasksWrapper.py.

dbTask

dc_crawler.DBTasksWrapper.DBTasksWrapper.dbTask

Definition at line 32 of file DBTasksWrapper.py.

fieldRecalculator

dc_crawler.DBTasksWrapper.DBTasksWrapper.fieldRecalculator

Definition at line 33 of file DBTasksWrapper.py.

rid

dc_crawler.DBTasksWrapper.DBTasksWrapper.rid

Definition at line 34 of file DBTasksWrapper.py.

---

The documentation for this class was generated from the following file:

• sources/hce/dc_crawler/DBTasksWrapper.py