如何在不下载文章的情况下使用Newspaper3k库

Question

假设我有新闻文章的本地副本。我怎样才能在这些文章上刊登报纸？根据文档，报纸库的正常使用如下所示：

from newspaper import Article
url = 'http://fox13now.com/2013/12/30/new-year-new-laws-obamacare-pot-guns-and-drones/'
article.download()
article = Article(url)
article.parse()
# ...

就我而言，我不需要从网页下载文章，因为我已经拥有该页面的本地副本。如何在网页的本地副本上使用报纸？

Answer 1

确实有一种官方方法可以解决这个问题，正如这里提到的

在程序中加载 html 后，您可以使用 set_html() 方法将其设置为 article.html

import newspaper
with open("file.html", 'rb') as fh:
    ht = fh.read()
article = newspaper.Article(url = ' ')
article.set_html(ht)
article.parse()

Answer 2

你可以，它只是有点笨拙。举个例子

import requests
from newspaper import Article
url = 'https://www.cnn.com/2019/06/19/india/chennai-water-crisis-intl-hnk/index.html'
# get sample html
r = requests.get(url)
# save to file
with open('file.html', 'wb') as fh:
    fh.write(r.content)
a = Article(url)
# set html manually
with open("file.html", 'rb') as fh:
    a.html = fh.read()
# need to set download_state to 2 for this to work
a.download_state = 2
a.parse()
# Now the article should be populated
a.text
# 'New Delhi (CNN) The floor...'

其中download_state来自 newspaper.article.py 中的代码段：

# /path/to/site-packages/newspaper/article.py
class ArticleDownloadState(object):
    NOT_STARTED = 0
    FAILED_RESPONSE = 1
    SUCCESS = 2
~snip~
# This is why you need to set that variable
class Article:
    def __init__(...):
        ~snip~
         # Keep state for downloads and parsing
        self.is_parsed = False
        self.download_state = ArticleDownloadState.NOT_STARTED
        self.download_exception_msg = None
    def parse(self):
        # will throw exception if download_state isn't 2
        self.throw_if_not_downloaded_verbose()
        self.doc = self.config.get_parser().fromstring(self.html)

作为替代方法，您可以重写该类以与 parse 函数执行相同的操作：

from newspaper import Article
import io
class localArticle(Article):
    def __init__(self, url, **kwargs):
        # set url to be file_name in __init__ if it's a file handle
        super().__init__(url if isinstance(url, str) else url.name, **kwargs)
        # set standalone _url attr so that parse will work as expected
        self._url = url
    def parse(self):
        # sets html and things for you
        if isinstance(self._url, str):
            with open(self._url, 'rb') as fh:
                self.html = fh.read()
        elif isinstance(self._url, (io.TextIOWrapper, io.BufferedReader)):
            self.html = self._url.read()
        else:
            raise TypeError(f"Expected file path or file-like object, got {self._url.__class__}")
        self.download_state = 2
        # now parse will continue on with the proper params set
        super(localArticle, self).parse()

a = localArticle('file.html') # pass your file name here
a.parse()
a.text[:10]
# 'New Delhi '
# or you can give it a file handle
with open("file.html", 'rb') as fh:
    a = localArticle(fh)
    a.parse()
a.text[:10]
# 'New Delhi '

Answer 3

我相信

你已经解决了这个问题，但报纸有能力处理本地存储的HTML文件。

from newspaper import Article
# Downloading the HTML for the article
url = 'http://fox13now.com/2013/12/30/new-year-new-laws-obamacare-pot-guns-and-drones/'
article = Article(url)
article.download()
article.parse()
with open('fox13no.html', 'w') as fileout:
   fileout.write(article.html)
# Read the locally stored HTML with Newspaper
with open("fox13no.html", 'r') as f:
   # note the URL string is empty
   article = Article('', language='en')
   article.download(input_html=f.read())
   article.parse()
   print(article.title) 
   New Year, new laws: Obamacare, pot, guns and drones