HCE Project Python language Distributed Tasks Manager Application, Distributed Crawler Application and client API bindings.  2.0.0-chaika
Hierarchical Cluster Engine Python language binding
Functions | Variables
ftests.ftest_dc_ProcessorTask_batch_processing Namespace Reference

Functions

def processFullBatch (input_object)
 

Variables

 siteId1 = str(md5.new("http://www.yomiuri.co.jp").hexdigest())
 
 urlId1 = str(md5.new("http://www.yomiuri.co.jp/politics/20140422-OYT1T50105.html?from=ycont_top_txt").hexdigest())
 
 bItem1 = BatchItem(siteId1, urlId1)
 
 siteId2 = str(md5.new("http://www.asahi.com").hexdigest())
 
 urlId2 = str(md5.new("http://www.asahi.com/articles/ASG4Q4D0DG4QUTQP00Z.html").hexdigest())
 
 bItem2 = BatchItem(siteId2, urlId2)
 
 siteId3 = str(md5.new("http://mainichi.jp").hexdigest())
 
 urlId3 = str(md5.new("http://mainichi.jp/opinion/news/20140422k0000e070249000c.html").hexdigest())
 
 bItem3 = BatchItem(siteId3, urlId3)
 
 siteId4 = str(md5.new("http://sankei.jp.msn.com").hexdigest())
 
 urlId4 = str(md5.new("http://sankei.jp.msn.com/politics/news/140401/stt14040112470002-n1.htm").hexdigest())
 
 bItem4 = BatchItem(siteId4, urlId4)
 
 siteId5 = str(md5.new("http://www.jiji.com").hexdigest())
 
 urlId5 = str(md5.new("http://www.jiji.com/jc/zc?k=201403/2014033000221&rel=j&g=int&relid=1_1").hexdigest())
 
 bItem5 = BatchItem(siteId5, urlId5)
 
 siteId6 = str(md5.new("http://www.kyodo.co.jp").hexdigest())
 
 urlId6 = str(md5.new("http://www.kyodo.co.jp/release-news/2014-04-07_522084/").hexdigest())
 
 bItem6 = BatchItem(siteId6, urlId6)
 
 siteId7 = str(md5.new("http://www3.nhk.or.jp").hexdigest())
 
 urlId7 = str(md5.new("http://www3.nhk.or.jp/chihouhatsu/").hexdigest())
 
 bItem7 = BatchItem(siteId7, urlId7)
 
 siteId8 = str(md5.new("http://jp.reuters.com").hexdigest())
 
 urlId8 = str(md5.new("http://jp.reuters.com/article/topNews/idJPTYEA3700120140408").hexdigest())
 
 bItem8 = BatchItem(siteId8, urlId8)
 
 siteId9 = str(md5.new("http://www.nikkei.com").hexdigest())
 
 urlId9 = str(md5.new("http://www.nikkei.com/article/DGXNASGM0703U_Y4A400C1MM0000/?dg=1").hexdigest())
 
 bItem9 = BatchItem(siteId9, urlId9)
 
 siteId10 = str(md5.new("http://www.tokyo-np.co.jp").hexdigest())
 
 urlId10 = str(md5.new("http://www.tokyo-np.co.jp/s/article/2014042290135558.html").hexdigest())
 
 bItem10 = BatchItem(siteId10, urlId10)
 
list url_list
 
string PWD = "cd ../../bin"
 
string PYTHON_BINARY = "/usr/bin/python"
 
string CRAWLER_TASK_BINARY = "./crawler-task.py"
 
string CRAWLER_TASK_CFG = "--config=../ini/crawler-task.ini"
 
string PROCESSOR_TASK_BINARY = "./processor-task.py"
 
string PROCESSOR_TASK_CFG = "--config=../ini/processor-task.ini"
 
string PREPAIRER = "./prepairer.py"
 
string JSON_VIEWER = "./scraper_json_viewer.py"
 
 Results = namedtuple("Results", "exit_code, output, err")
 
 input_object = Batch(11, url_list)
 
def result = processFullBatch(input_object)
 
 generalResponse = pickle.loads(result.output)
 

Detailed Description

HCE project, Python bindings, Distributed Tasks Manager application.
Event objects definitions.

@package: dc
@file ftest_dc_ProcessorTask_batch_processing.py
@author Oleksii <developers.hce@gmail.com>
@link: http://hierarchical-cluster-engine.com/
@copyright: Copyright &copy; 2013-2014 IOIX Ukraine
@license: http://hierarchical-cluster-engine.com/license/
@since: 0.1

Function Documentation

◆ processFullBatch()

def ftests.ftest_dc_ProcessorTask_batch_processing.processFullBatch (   input_object)

Definition at line 165 of file ftest_dc_ProcessorTask_batch_processing.py.

165 def processFullBatch(input_object):
166  input_pickled_object = pickle.dumps(input_object)
167  # cmd = PWD + " && " +PREPAIRER +" | "+ PYTHON_BINARY + " " + PROCESSOR_TASK_BINARY + " " + PROCESSOR_TASK_CFG + " | " +JSON_VIEWER
168  cmd = PWD + " && " + PYTHON_BINARY + " " + PROCESSOR_TASK_BINARY + " " + PROCESSOR_TASK_CFG
169  # print "cmd: " + cmd
170  process = Popen(cmd, stdout=PIPE, stdin=PIPE, shell=True)
171  (output, err) = process.communicate(input=input_pickled_object)
172  # (output, err) = process.communicate(input="http://www.yomiuri.co.jp/politics/20140422-OYT1T50105.html?from=ycont_top_txt")
173  print output
174  exit_code = process.wait()
175  return Results(exit_code, output, err)
176 
177 
ftests.ftest_dc_ProcessorTask_batch_processing.processFullBatch
def processFullBatch(input_object)
Definition: ftest_dc_ProcessorTask_batch_processing.py:165
ftests.ftest_dc_ProcessorTask_batch_processing.Results
Results
Definition: ftest_dc_ProcessorTask_batch_processing.py:162

Variable Documentation

◆ bItem1

ftests.ftest_dc_ProcessorTask_batch_processing.bItem1 = BatchItem(siteId1, urlId1)

Definition at line 42 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ bItem10

ftests.ftest_dc_ProcessorTask_batch_processing.bItem10 = BatchItem(siteId10, urlId10)

Definition at line 133 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ bItem2

ftests.ftest_dc_ProcessorTask_batch_processing.bItem2 = BatchItem(siteId2, urlId2)

Definition at line 57 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ bItem3

ftests.ftest_dc_ProcessorTask_batch_processing.bItem3 = BatchItem(siteId3, urlId3)

Definition at line 66 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ bItem4

ftests.ftest_dc_ProcessorTask_batch_processing.bItem4 = BatchItem(siteId4, urlId4)

Definition at line 79 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ bItem5

ftests.ftest_dc_ProcessorTask_batch_processing.bItem5 = BatchItem(siteId5, urlId5)

Definition at line 90 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ bItem6

ftests.ftest_dc_ProcessorTask_batch_processing.bItem6 = BatchItem(siteId6, urlId6)

Definition at line 99 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ bItem7

ftests.ftest_dc_ProcessorTask_batch_processing.bItem7 = BatchItem(siteId7, urlId7)

Definition at line 108 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ bItem8

ftests.ftest_dc_ProcessorTask_batch_processing.bItem8 = BatchItem(siteId8, urlId8)

Definition at line 117 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ bItem9

ftests.ftest_dc_ProcessorTask_batch_processing.bItem9 = BatchItem(siteId9, urlId9)

Definition at line 125 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ CRAWLER_TASK_BINARY

string ftests.ftest_dc_ProcessorTask_batch_processing.CRAWLER_TASK_BINARY = "./crawler-task.py"

Definition at line 155 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ CRAWLER_TASK_CFG

string ftests.ftest_dc_ProcessorTask_batch_processing.CRAWLER_TASK_CFG = "--config=../ini/crawler-task.ini"

Definition at line 156 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ generalResponse

ftests.ftest_dc_ProcessorTask_batch_processing.generalResponse = pickle.loads(result.output)

Definition at line 186 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ input_object

ftests.ftest_dc_ProcessorTask_batch_processing.input_object = Batch(11, url_list)

Definition at line 180 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ JSON_VIEWER

string ftests.ftest_dc_ProcessorTask_batch_processing.JSON_VIEWER = "./scraper_json_viewer.py"

Definition at line 160 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ PREPAIRER

string ftests.ftest_dc_ProcessorTask_batch_processing.PREPAIRER = "./prepairer.py"

Definition at line 159 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ PROCESSOR_TASK_BINARY

string ftests.ftest_dc_ProcessorTask_batch_processing.PROCESSOR_TASK_BINARY = "./processor-task.py"

Definition at line 157 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ PROCESSOR_TASK_CFG

string ftests.ftest_dc_ProcessorTask_batch_processing.PROCESSOR_TASK_CFG = "--config=../ini/processor-task.ini"

Definition at line 158 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ PWD

string ftests.ftest_dc_ProcessorTask_batch_processing.PWD = "cd ../../bin"

Definition at line 153 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ PYTHON_BINARY

string ftests.ftest_dc_ProcessorTask_batch_processing.PYTHON_BINARY = "/usr/bin/python"

Definition at line 154 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ result

def ftests.ftest_dc_ProcessorTask_batch_processing.result = processFullBatch(input_object)

Definition at line 182 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ Results

ftests.ftest_dc_ProcessorTask_batch_processing.Results = namedtuple("Results", "exit_code, output, err")

Definition at line 162 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ siteId1

ftests.ftest_dc_ProcessorTask_batch_processing.siteId1 = str(md5.new("http://www.yomiuri.co.jp").hexdigest())

Definition at line 40 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ siteId10

ftests.ftest_dc_ProcessorTask_batch_processing.siteId10 = str(md5.new("http://www.tokyo-np.co.jp").hexdigest())

Definition at line 131 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ siteId2

ftests.ftest_dc_ProcessorTask_batch_processing.siteId2 = str(md5.new("http://www.asahi.com").hexdigest())

Definition at line 55 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ siteId3

ftests.ftest_dc_ProcessorTask_batch_processing.siteId3 = str(md5.new("http://mainichi.jp").hexdigest())

Definition at line 64 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ siteId4

ftests.ftest_dc_ProcessorTask_batch_processing.siteId4 = str(md5.new("http://sankei.jp.msn.com").hexdigest())

Definition at line 77 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ siteId5

ftests.ftest_dc_ProcessorTask_batch_processing.siteId5 = str(md5.new("http://www.jiji.com").hexdigest())

Definition at line 88 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ siteId6

ftests.ftest_dc_ProcessorTask_batch_processing.siteId6 = str(md5.new("http://www.kyodo.co.jp").hexdigest())

Definition at line 97 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ siteId7

ftests.ftest_dc_ProcessorTask_batch_processing.siteId7 = str(md5.new("http://www3.nhk.or.jp").hexdigest())

Definition at line 106 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ siteId8

ftests.ftest_dc_ProcessorTask_batch_processing.siteId8 = str(md5.new("http://jp.reuters.com").hexdigest())

Definition at line 115 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ siteId9

ftests.ftest_dc_ProcessorTask_batch_processing.siteId9 = str(md5.new("http://www.nikkei.com").hexdigest())

Definition at line 123 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ url_list

list ftests.ftest_dc_ProcessorTask_batch_processing.url_list
Initial value:
1 = [
2  bItem1,
3  bItem2,
4  bItem3,
5  bItem4,
6  bItem5,
7  bItem6,
8  bItem7,
9  bItem8,
10  bItem9,
11  bItem10
12  ]

Definition at line 139 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ urlId1

ftests.ftest_dc_ProcessorTask_batch_processing.urlId1 = str(md5.new("http://www.yomiuri.co.jp/politics/20140422-OYT1T50105.html?from=ycont_top_txt").hexdigest())

Definition at line 41 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ urlId10

ftests.ftest_dc_ProcessorTask_batch_processing.urlId10 = str(md5.new("http://www.tokyo-np.co.jp/s/article/2014042290135558.html").hexdigest())

Definition at line 132 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ urlId2

ftests.ftest_dc_ProcessorTask_batch_processing.urlId2 = str(md5.new("http://www.asahi.com/articles/ASG4Q4D0DG4QUTQP00Z.html").hexdigest())

Definition at line 56 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ urlId3

ftests.ftest_dc_ProcessorTask_batch_processing.urlId3 = str(md5.new("http://mainichi.jp/opinion/news/20140422k0000e070249000c.html").hexdigest())

Definition at line 65 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ urlId4

ftests.ftest_dc_ProcessorTask_batch_processing.urlId4 = str(md5.new("http://sankei.jp.msn.com/politics/news/140401/stt14040112470002-n1.htm").hexdigest())

Definition at line 78 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ urlId5

ftests.ftest_dc_ProcessorTask_batch_processing.urlId5 = str(md5.new("http://www.jiji.com/jc/zc?k=201403/2014033000221&rel=j&g=int&relid=1_1").hexdigest())

Definition at line 89 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ urlId6

ftests.ftest_dc_ProcessorTask_batch_processing.urlId6 = str(md5.new("http://www.kyodo.co.jp/release-news/2014-04-07_522084/").hexdigest())

Definition at line 98 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ urlId7

ftests.ftest_dc_ProcessorTask_batch_processing.urlId7 = str(md5.new("http://www3.nhk.or.jp/chihouhatsu/").hexdigest())

Definition at line 107 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ urlId8

ftests.ftest_dc_ProcessorTask_batch_processing.urlId8 = str(md5.new("http://jp.reuters.com/article/topNews/idJPTYEA3700120140408").hexdigest())

Definition at line 116 of file ftest_dc_ProcessorTask_batch_processing.py.

◆ urlId9

ftests.ftest_dc_ProcessorTask_batch_processing.urlId9 = str(md5.new("http://www.nikkei.com/article/DGXNASGM0703U_Y4A400C1MM0000/?dg=1").hexdigest())

Definition at line 124 of file ftest_dc_ProcessorTask_batch_processing.py.