HCE Project Python language Distributed Tasks Manager Application, Distributed Crawler Application and client API bindings.
2.0.0-chaika
Hierarchical Cluster Engine Python language binding
|
This is the complete list of members for dc_crawler.RobotsParser.RobotsParser, including all inherited members.
__init__(self, headers=None, isCacheUsing=False, robotsFileDir=None) | dc_crawler.RobotsParser.RobotsParser | |
cacheElement | dc_crawler.RobotsParser.RobotsParser | |
cacheElementKeys | dc_crawler.RobotsParser.RobotsParser | |
checkUrlByRobots(self, url, siteId=None, headers=None) | dc_crawler.RobotsParser.RobotsParser | |
headers | dc_crawler.RobotsParser.RobotsParser | |
HTTP_OK_CODE | dc_crawler.RobotsParser.RobotsParser | static |
initFiends(self, headers=None, isCacheUsing=False, robotsFileDir=None) | dc_crawler.RobotsParser.RobotsParser | |
loadRobots(self, url, siteId=None, additionHeaders=None, proxyName=None) | dc_crawler.RobotsParser.RobotsParser | |
localCrawlerDataStorage | dc_crawler.RobotsParser.RobotsParser | |
localParser | dc_crawler.RobotsParser.RobotsParser | |
ROBOTS_PATTERN | dc_crawler.RobotsParser.RobotsParser | static |
robotsFileDir | dc_crawler.RobotsParser.RobotsParser | |
USER_AGENT_HEADER_NAME | dc_crawler.RobotsParser.RobotsParser | static |