Newspaper3k在提取时过滤掉不好的URL

Question

需要一些帮助;)我已经设法从CNN新闻网站上抓取标题和内容，并把它放在一个。csv文件中。

现在带有url的列表(已经用另一个代码提取)有一些错误的url。这个代码非常简单，因为它只是扫描整个网站并返回所有的url。因此，列表中有一些坏的url(例如http://cnn.com/date/2021-10-17)而不是搜索这个列表并手动删除那些错误的url，我想知道这是否可以通过更改我的代码来跳过错误的url并继续下一个等等来解决。

示例代码:

import csv
from newspaper import Config
from newspaper import Article
from os.path import exists
USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Firefox/78.0'
config = Config()
config.browser_user_agent = USER_AGENT
config.request_timeout = 10
urls = ['https://www.cnn.com/2021/10/25/tech/facebook-papers/index.html', 'http://cnn.com/date/2021-10-17', 'https://www.cnn.com/entertainment/live-news/rust-shooting-alec-baldwin-10-25-21/h_257c62772a2b69cb37db397592971b58']
# the above normally would be where I refer to the .csv file with URLs
for url in urls:
article = Article(url, config=config)
article.download()
article.parse()
article_meta_data = article.meta_data
file_exists = exists('cnn_extraction_results.csv')
if not file_exists:
with open('cnn_extraction_results.csv', 'w', newline='') as file:
headers = ['article title', 'article text']
writer = csv.DictWriter(file, delimiter=',', lineterminator='n', fieldnames=headers)
writer.writeheader()
writer.writerow({'article title': article.title,
'article text': article.text})
else:
with open('cnn_extraction_results.csv', 'a', newline='') as file:
headers = ['article title', 'article text']
writer = csv.DictWriter(file, delimiter=',', lineterminator='n', fieldnames=headers)
writer.writerow({'article title': article.title,
'article text': article.text})

Answer 1

试试这个:

import csv
from os.path import exists
from newspaper import Config
from newspaper import Article
from newspaper import ArticleException
USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Firefox/78.0'
config = Config()
config.browser_user_agent = USER_AGENT
config.request_timeout = 10
urls = ['https://www.cnn.com/2021/10/25/tech/facebook-papers/index.html',
'http://cnn.com/date/2021-10-17',
'https://www.cnn.com/entertainment/live-news/rust-shooting-alec-baldwin-10-25-21/h_257c62772a2b69cb37db397592971b58']
for url in urls:
try:
article = Article(url, config=config)
article.download()
article.parse()
article_meta_data = article.meta_data
file_exists = exists('cnn_extraction_results.csv')
if not file_exists:
with open('cnn_extraction_results.csv', 'w', newline='') as file:
headers = ['article title', 'article text']
writer = csv.DictWriter(file, delimiter=',', lineterminator='n', fieldnames=headers)
writer.writeheader()
writer.writerow({'article title': article.title,
'article text': article.text})
else:
with open('cnn_extraction_results.csv', 'a', newline='') as file:
headers = ['article title', 'article text']
writer = csv.DictWriter(file, delimiter=',', lineterminator='n', fieldnames=headers)
writer.writerow({'article title': article.title,
'article text': article.text})
except ArticleException:
print('***FAILED TO DOWNLOAD***', url)