Functions
def	processFullBatch (input_object)

Variables
list	url_list

string	PYTHON_BINARY = "/usr/bin/python"

string	CRAWLER_BINARY = "../../bin/crawler.py"

string	CFG = "--config=../../ini/crawler.ini"

	Results = namedtuple("Results", "exit_code output err")

	input_object = Batch(url_list)

def	result = processFullBatch(input_object)

	generalResponse = pickle.loads(result.output)

Detailed Description

HCE project, Python bindings, Distributed Tasks Manager application.
Event objects definitions.

@package: dc
@file ftest_dc_CrawlerTask_batch_processing.py
@author Oleksii <developers.hce@gmail.com>
@link: http://hierarchical-cluster-engine.com/
@copyright: Copyright &copy; 2013-2014 IOIX Ukraine
@license: http://hierarchical-cluster-engine.com/license/
@since: 0.1

Function Documentation

◆ processFullBatch()

def ftests.ftest_dc_CrawlerTask_batch_processing.processFullBatch ( input_object )

Definition at line 34 of file ftest_dc_CrawlerTask_batch_processing.py.

 def processFullBatch(input_object):
   input_pickled_object = pickle.dumps(input_object)
   process = Popen([PYTHON_BINARY, CRAWLER_BINARY, CFG], stdout=PIPE, stdin=PIPE)
   (output, err) = process.communicate(input=input_pickled_object)
   exit_code = process.wait()
   return Results(exit_code, output, err)
 
 

Variable Documentation

◆ CFG

string ftests.ftest_dc_CrawlerTask_batch_processing.CFG = "--config=../../ini/crawler.ini"

Definition at line 28 of file ftest_dc_CrawlerTask_batch_processing.py.

◆ CRAWLER_BINARY

string ftests.ftest_dc_CrawlerTask_batch_processing.CRAWLER_BINARY = "../../bin/crawler.py"

Definition at line 27 of file ftest_dc_CrawlerTask_batch_processing.py.

◆ generalResponse

ftests.ftest_dc_CrawlerTask_batch_processing.generalResponse = pickle.loads(result.output)

Definition at line 48 of file ftest_dc_CrawlerTask_batch_processing.py.

◆ input_object

ftests.ftest_dc_CrawlerTask_batch_processing.input_object = Batch(url_list)

Definition at line 44 of file ftest_dc_CrawlerTask_batch_processing.py.

◆ PYTHON_BINARY

string ftests.ftest_dc_CrawlerTask_batch_processing.PYTHON_BINARY = "/usr/bin/python"

Definition at line 26 of file ftest_dc_CrawlerTask_batch_processing.py.

◆ result

def ftests.ftest_dc_CrawlerTask_batch_processing.result = processFullBatch(input_object)

Definition at line 46 of file ftest_dc_CrawlerTask_batch_processing.py.

◆ Results

ftests.ftest_dc_CrawlerTask_batch_processing.Results = namedtuple("Results", "exit_code output err")

Definition at line 31 of file ftest_dc_CrawlerTask_batch_processing.py.

◆ url_list

list ftests.ftest_dc_CrawlerTask_batch_processing.url_list

Initial value:

 =  [
   "http://www.yomiuri.co.jp/sports/mlb/20140407-OYT1T50015.html?from=ytop_top"
   ]

Definition at line 21 of file ftest_dc_CrawlerTask_batch_processing.py.

Functions

Variables