scraper_json_viewer Namespace Reference

Functions
def	processBatch ()

Variables
string	config_db_dir = "../data/dc_dbdata"

Detailed Description

HCE project, Python bindings, Distributed Tasks Manager application.
Event objects definitions.

@package: dc
@file scraper_json_viewer.py
@author Oleksii <developers.hce@gmail.com>
@link: http://hierarchical-cluster-engine.com/
@copyright: Copyright &copy; 2013-2014 IOIX Ukraine
@license: http://hierarchical-cluster-engine.com/license/
@since: 0.1

Function Documentation

processBatch()

def scraper_json_viewer.processBatch

(

)

Definition at line 27 of file scraper_json_viewer.py.

def processBatch():
  json = None
  # read pickled batch object from stdin and unpickle it
  input_pickled_object = sys.stdin.read()
  # print input_pickled_object
  input_data = (pickle.loads(input_pickled_object)).items[0]
  # print("Batch item: siteId: %s, urlId: %s" %(input_data.siteId, input_data.urlId))
  if len(input_data.siteId):
    db_name = config_db_dir + "/" + input_data.siteId + ".db"
  else:
    db_name = config_db_dir + "/0.db"
  con = lite.connect(db_name)
  with con:
    cur = con.cursor()
    query = "SELECT `data` FROM `articles` WHERE `id`='%s' order by `CDate` DESC LIMIT 1" % (input_data.urlId)
    cur.execute(query)
    json = cur.fetchone()
  print decode(json[0])

Here is the call graph for this function:

Variable Documentation

config_db_dir

string scraper_json_viewer.config_db_dir = "../data/dc_dbdata"

Definition at line 25 of file scraper_json_viewer.py.