我们是python的初学者。我们有一个链接/网站列表，上面有唐纳德·特朗普的言论。每一个环节都代表着整个采访/演讲等。我们现在想访问这些网站，抓取它们，并为每个链接创建一个文本文件。目前，我们的代码对2或3个链接都是这样做的，但随后只显示了这个错误：
Traceback (most recent call last):
File "C:UsersLotteAppDataLocalProgramsPythonPython37CodeCorpus_createScrapen und alle inhalte laden und speichern - zusammengefügt.py", line 79, in <module>
Trump=(tree.xpath('//div[@class="media topic-media-row mediahover "]/div[3]/div/div[2]/a')[item2].text_content())
IndexError: list index out of range
我们尝试了index元素，尝试了[0]，甚至忽略了它。什么也没用。然后，我们尝试只使用一个链接而不使用第一个循环来运行代码，这非常适合
import lxml
from lxml import html
from lxml.html import fromstring
import requests
import re
Linklist=['https://factba.se/transcript/donald-trump-remarks-briefing-room-border-security-january-3-2019', 'https://factba.se/transcript/donald-trump-remarks-cabinet-meeting-january-2-2019', 'https://factba.se/transcript/donald-trump-remarks-military-briefing-iraq-december-26-2018', 'https://factba.se/transcript/donald-trump-remarks-videoconference-troops-christmas-december-25-2018', 'https://factba.se/transcript/donald-trump-remarks-justice-reform-december-21-2018', 'https://factba.se/transcript/donald-trump-remarks-agriculture-bill-december-20-2018', 'https://factba.se/transcript/donald-trump-remarks-roundtable-school-safety-december-18-2018', 'https://factba.se/transcript/donald-trump-remarks-governors-elect-white-house-december-15-2018', 'https://factba.se/transcript/donald-trump-remarks-governors-elect-white-house-december-13-2018', 'https://factba.se/transcript/donald-trump-remarks-revitalization-council-executive-order-december-12-2018', 'https://factba.se/transcript/donald-trump-remarks-meeting-pelosi-schumer-december-11-2018', 'https://factba.se/transcript/donald-trump-remarks-bill-signing-genocide-december-11-2018', 'https://factba.se/transcript/donald-trump-remarks-chanukah-evening-reception-december-6-2018', 'https://factba.se/transcript/donald-trump-remarks-chanukah-afternoon-reception-december-6-2018', 'https://factba.se/transcript/donald-trump-remarks-bilat-china-xi-buenos-aires-december-1-2018', 'https://factba.se/transcript/donald-trump-remarks-bilat-germany-merkel-december-1-2018', 'https://factba.se/transcript/donald-trump-remarks-usmca-mexico-canada-buenos-aires-november-30-2018', 'https://factba.se/transcript/donald-trump-remarks-bilat-argentina-macri-november-30-2018', 'https://factba.se/transcript/donald-trump-remarks-bilat-morrison-australia-november-30-2018', 'https://factba.se/transcript/donald-trump-remarks-trilat-japan-india-abe-modi-november-30-2018']
for item in Linklist:
headers= {"User-Agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36"}
page = requests.get(item, headers=headers)
tree = html.fromstring(page.content)
#loads everything trump said
Text=[]
for item2 in range(len(tree.xpath('//div[@class="media topic-media-row mediahover "]'))):
Trump=(tree.xpath('//div[@class="media topic-media-row mediahover "]/div[3]/div/div[2]/a')[item2].text_content())
Text.append(Trump)
print(Text, 'n')
我们只想从的每一个环节中获得王牌话语
这是脚本的修改版本。
code.py:
from lxml import html
import requests
import re
from pprint import pprint
url_list = [
"https://factba.se/transcript/donald-trump-remarks-briefing-room-border-security-january-3-2019",
"https://factba.se/transcript/donald-trump-remarks-cabinet-meeting-january-2-2019",
"https://factba.se/transcript/donald-trump-remarks-military-briefing-iraq-december-26-2018",
"https://factba.se/transcript/donald-trump-remarks-videoconference-troops-christmas-december-25-2018",
"https://factba.se/transcript/donald-trump-remarks-justice-reform-december-21-2018",
"https://factba.se/transcript/donald-trump-remarks-agriculture-bill-december-20-2018",
"https://factba.se/transcript/donald-trump-remarks-roundtable-school-safety-december-18-2018",
"https://factba.se/transcript/donald-trump-remarks-governors-elect-white-house-december-15-2018",
"https://factba.se/transcript/donald-trump-remarks-governors-elect-white-house-december-13-2018",
"https://factba.se/transcript/donald-trump-remarks-revitalization-council-executive-order-december-12-2018",
"https://factba.se/transcript/donald-trump-remarks-meeting-pelosi-schumer-december-11-2018",
"https://factba.se/transcript/donald-trump-remarks-bill-signing-genocide-december-11-2018",
"https://factba.se/transcript/donald-trump-remarks-chanukah-evening-reception-december-6-2018",
"https://factba.se/transcript/donald-trump-remarks-chanukah-afternoon-reception-december-6-2018",
"https://factba.se/transcript/donald-trump-remarks-bilat-china-xi-buenos-aires-december-1-2018",
"https://factba.se/transcript/donald-trump-remarks-bilat-germany-merkel-december-1-2018",
"https://factba.se/transcript/donald-trump-remarks-usmca-mexico-canada-buenos-aires-november-30-2018",
"https://factba.se/transcript/donald-trump-remarks-bilat-argentina-macri-november-30-2018",
"https://factba.se/transcript/donald-trump-remarks-bilat-morrison-australia-november-30-2018",
"https://factba.se/transcript/donald-trump-remarks-trilat-japan-india-abe-modi-november-30-2018"
]
headers = {
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36"
}
media_row_xpath_marker = '//div[@class="media topic-media-row mediahover "]'
normal_xpath_marker = media_row_xpath_marker + "/div[3]/div/div[2]/a"
movieless_xpath_marker = media_row_xpath_marker + "/div[3]/div/div/a"
xpath_markers = [
normal_xpath_marker,
movieless_xpath_marker,
]
for url_index, url in enumerate(url_list):
page = requests.get(url, headers=headers)
tree = html.fromstring(page.content)
lines = []
media_row_list = tree.xpath(media_row_xpath_marker)
if media_row_list:
for xpath_marker in xpath_markers:
post_list = tree.xpath(xpath_marker)
if post_list:
lines = [item.text_content() for item in post_list]
break
#pprint(lines)
print("URL index: {0:02d} - Article count: {1:03d}".format(url_index, len(lines)))
备注：
问题是3rdURL与其他URL有点不同，如果你看一下，它没有YouTube，所以xpath不匹配。再加上缺少空列表测试，产生了上述异常。现在，正在尝试两种模式：
- movieless_path_marker-适用于"故障"页面
- normal_xpath_marker-它将对其余部分起作用(这是尝试的1st)
当一个模式触发某些结果时，只需忽略其余的(如果有的话)
- 我还重构了代码：
- 摆脱了一个循环(以及多次无用地执行的操作)
- 变量重命名
- 恒定提取
- 代码样式
- 其他细微变化
输出(显示每个URL的文章计数)：
(py_064_03.06.08_test0) e:WorkDevStackOverflowq054043232>"e:WorkDevVEnvspy_064_03.06.08_test0Scriptspython.exe" code.py URL index: 00 - Article count: 018 URL index: 01 - Article count: 207 URL index: 02 - Article count: 063 URL index: 03 - Article count: 068 URL index: 04 - Article count: 080 URL index: 05 - Article count: 051 URL index: 06 - Article count: 045 URL index: 07 - Article count: 014 URL index: 08 - Article count: 036 URL index: 09 - Article count: 022 URL index: 10 - Article count: 105 URL index: 11 - Article count: 020 URL index: 12 - Article count: 025 URL index: 13 - Article count: 028 URL index: 14 - Article count: 010 URL index: 15 - Article count: 012 URL index: 16 - Article count: 015 URL index: 17 - Article count: 005 URL index: 18 - Article count: 005 URL index: 19 - Article count: 006