dc_processor.NewspaperWrapper.NewspaperWrapper Class Reference

Inheritance diagram for dc_processor.NewspaperWrapper.NewspaperWrapper:

Collaboration diagram for dc_processor.NewspaperWrapper.NewspaperWrapper:

Public Member Functions
def	__init__ (self, url, title=u'', source_url=u'', config=None, isLoadUrls=True, kwargs)
def	sort_images (self, imgs)
def	versionnedWrapper (self, methodName)
def	fetch_images (self)
def	set_top_img (self, src_url)

Public Attributes
	isLoadUrls

Detailed Description

Definition at line 22 of file NewspaperWrapper.py.

Constructor & Destructor Documentation

__init__()

def dc_processor.NewspaperWrapper.NewspaperWrapper.__init__	(		self,
			url,
			title = u'',
			source_url = u'',
			config = None,
			isLoadUrls = True,
			kwargs
	)

Definition at line 33 of file NewspaperWrapper.py.

  def __init__(self, url, title=u'', source_url=u'', config=None, isLoadUrls=True, **kwargs):
    super(NewspaperWrapper, self).__init__(url, title, source_url, config, **kwargs)
    self.isLoadUrls = isLoadUrls

Member Function Documentation

fetch_images()

def dc_processor.NewspaperWrapper.NewspaperWrapper.fetch_images

(

self

)

Definition at line 76 of file NewspaperWrapper.py.

  def fetch_images(self):
    if self.clean_doc is not None:  # pylint: disable=E1101
      meta_img_url = self.versionnedWrapper("get_meta_img_url")
      self.set_meta_img(meta_img_url)  # pylint: disable=E1101

    if self.clean_top_node is not None and not self.has_top_image():  # pylint: disable=E1101
      first_img = self.versionnedWrapper("get_first_img_url")
      self.set_top_img(first_img)

    if not self.has_top_image() and self.isLoadUrls:  # pylint: disable=E1101
      self.set_reddit_top_img()
    else:
      logger.debug(">>> not load urls")

    if self.isLoadUrls and self.clean_doc is not None:
      imgs = self.versionnedWrapper("get_img_urls")
      imgs = self.sort_images(imgs)
      self.set_imgs(imgs)

Here is the call graph for this function:

set_top_img()

def dc_processor.NewspaperWrapper.NewspaperWrapper.set_top_img	(		self,
			src_url
	)

Definition at line 99 of file NewspaperWrapper.py.

  def set_top_img(self, src_url):
    if self.isLoadUrls:
      super(NewspaperWrapper, self).set_top_img(src_url)
    else:
      logger.debug(">>> not load urls")

sort_images()

def dc_processor.NewspaperWrapper.NewspaperWrapper.sort_images	(		self,
			imgs
	)

Definition at line 42 of file NewspaperWrapper.py.

  def sort_images(self, imgs):
    img_dimensions = []
    for image in imgs:
      try:
        img_dimension = images.fetch_image_dimension(image, self.config.browser_user_agent)
      except Exception, err:
        logger.error("fetch_image_dimension: %s", str(err))
        img_dimension = None

      if img_dimension is None:
        img_dimensions.append({"dim": None, "img_url": image})
      else:
        img_dimensions.append({"dim": img_dimension[0] * img_dimension[1], "img_url": image})
    ret = [img["img_url"] for img in sorted(img_dimensions, key=lambda img: img["dim"], reverse=True)]
    return ret

Here is the caller graph for this function:

versionnedWrapper()

def dc_processor.NewspaperWrapper.NewspaperWrapper.versionnedWrapper	(		self,
			methodName
	)

Definition at line 64 of file NewspaperWrapper.py.

  def versionnedWrapper(self, methodName):
    argsResult = inspect.getargspec(getattr(self.extractor, methodName))
    if argsResult is not None and argsResult.args is not None and \
    len(argsResult.args) - (0 if argsResult.defaults is None else len(argsResult.defaults)) == 2 and \
    "article" in argsResult.args:
      ret = getattr(self.extractor, methodName)(self)  # pylint: disable=E1101
    else:
      ret = getattr(self.extractor, methodName)(self.url, self.clean_doc)  # pylint: disable=E1101
    return ret

Here is the caller graph for this function:

Member Data Documentation

isLoadUrls

dc_processor.NewspaperWrapper.NewspaperWrapper.isLoadUrls

Definition at line 35 of file NewspaperWrapper.py.

---

The documentation for this class was generated from the following file:

• sources/hce/dc_processor/NewspaperWrapper.py