Newspaper3k仅在第一行导出为csv

Question

在" Life is complex "的帮助下，我成功地从CNN新闻网站上抓取了数据。从中提取的数据(url)保存在.csv文件(test1)中。请注意，这是手动完成的，因为它更容易做到!

from newspaper import Config
from newspaper import Article
from newspaper import ArticleException
import csv
USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Firefox/78.0'
config = Config()
config.browser_user_agent = USER_AGENT
config.request_timeout = 10
with open('test1.csv', 'r') as file:
csv_file = file.readlines()
for url in csv_file:
try:
article = Article(url.strip(), config=config)
article.download()
article.parse()
print(article.title)
article_text = article.text.replace('n', ' ')
print(article.text)
except ArticleException:
print('***FAILED TO DOWNLOAD***', article.url)

with open('test2.csv', 'a', newline='') as csvfile:
headers = ['article title', 'article text']
writer = csv.DictWriter(csvfile, lineterminator='n', fieldnames=headers)
writer.writeheader()
writer.writerow({'article title': article.title,
'article text': article.text})

使用上面的代码，我设法从url中抓取实际的新闻信息(标题和内容)，并将其导出到.csv文件。只有导出的问题是，它只导出最后一个标题和文本(因此我认为它一直覆盖第一行的信息)

如何获得csv文件中的所有标题和内容?

Answer 1

谢谢你的支持。

下面的代码应该可以帮助您解决CSV写入问题。如果没有，请告诉我，我会重新回答。

注:我将更新我的Newspaper3k概述文档，提供编写CSV文件的更多细节。

P.P.S.我现在正在写一个新的新闻刮板，因为Newspaper3k的开发已经结束了。我不确定我的代码的发布日期。

import csv
from newspaper import Config
from newspaper import Article
from os.path import exists
USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Firefox/78.0'
config = Config()
config.browser_user_agent = USER_AGENT
config.request_timeout = 10
urls = ['https://www.cnn.com/2021/10/25/tech/facebook-papers/index.html', 'https://www.cnn.com/entertainment/live-news/rust-shooting-alec-baldwin-10-25-21/h_257c62772a2b69cb37db397592971b58']
for url in urls:
article = Article(url, config=config)
article.download()
article.parse()
article_meta_data = article.meta_data
published_date = {value for (key, value) in article_meta_data.items() if key == 'pubdate'}
article_published_date = " ".join(str(x) for x in published_date)
file_exists = exists('cnn_extraction_results.csv')
if not file_exists:
with open('cnn_extraction_results.csv', 'w', newline='') as file:
headers = ['date published', 'article title', 'article text']
writer = csv.DictWriter(file, delimiter=',', lineterminator='n', fieldnames=headers)
writer.writeheader()
writer.writerow({'date published': article_published_date,
'article title': article.title,
'article text': article.text})
else:
with open('cnn_extraction_results.csv', 'a', newline='') as file:
headers = ['date published', 'article title', 'article text']
writer = csv.DictWriter(file, delimiter=',', lineterminator='n', fieldnames=headers)
writer.writerow({'date published': article_published_date,
'article title': article.title,
'article text': article.text})