我正在尝试使用Python和Chrome作为Selenium网络驱动程序来抓取Merriam-Webster的医学术语。到目前为止，这就是我所拥有的：
from os import path
from selenium import webdriver
# Adding an ad-blocker to Chrome to speed up page load times
options = webdriver.ChromeOptions()
options.add_extension(path.abspath("ublock-origin.crx"))
# Declaring the Selenium webdriver
driver = webdriver.Chrome(chrome_options = options)
# Fetching the "A" terms as a test set
driver.get("https://www.merriam-webster.com/browse/medical/a")
scraped_words = [] # The list that will hold each word
page_num = 1
while page_num < 55: # There are 54 pages of "A" terms
try:
for i in range(4): # There are 3 columns per page of words
column = "/html/body/div/div/div[5]/div[2]/div[1]/div/div[3]/ul/li[" + str(i) + "]/a"
number_of_words = len(driver.find_elements_by_xpath(column))
for j in range(number_of_words):
word = driver.find_elements_by_xpath(column + "[" + str(j) + "]")
scraped_words.append(word)
driver.find_element_by_class_name("fa-angle-right").click() # Next page
page_num += 1 # Increment page number to keep track of current page
except:
driver.close()
# Write out words to a file
with open("medical_terms.dict", "w") as text_file:
for i in range(len(scraped_words)):
text_file.write(str(scraped_words[i]))
text_file.write("n")
driver.close()
上面的代码获取所有项目，因为len(scraped_words)的输出是预期的数字。但是，由于我没有指定要获取元素的文本，因此我得到了元素标识符(我认为？(而不是文本。如果我决定使用word = driver.find_elements_by_xpath(column + "[" + str(j) + "]").text来指定我要获取元素的文本，则会收到以下错误：
Traceback (most recent call last):
File "mw_download.py", line 20, in <module>
number_of_words = len(driver.find_elements_by_xpath(column))
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 325, in find_elements_by_xpath
return self.find_elements(by=By.XPATH, value=xpath)
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 817, in find_elements
'value': value})['value']
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 256, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: no such session
(Driver info: chromedriver=2.31.488774 (7e15618d1bf16df8bf0ecf2914ed1964a387ba0b),platform=Mac OS X 10.12.6 x86_64)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "mw_download.py", line 27, in <module>
driver.close()
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 541, in close
self.execute(Command.CLOSE)
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 256, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: no such session
(Driver info: chromedriver=2.31.488774 (7e15618d1bf16df8bf0ecf2914ed1964a387ba0b),platform=Mac OS X 10.12.6 x86_64)
对我来说奇怪的是，我在运行之间更改的唯一代码在第 22 行，但错误消息指出了第 20 行。
任何帮助破译这里发生的事情以及我可以做些什么来修复它将不胜感激！
您只需要创建一个words列表来访问您的元素文本，更改：
word = driver.find_elements_by_xpath(column + "[" + str(j) + "]")
自：
word = [i.text for i in driver.find_elements_by_xpath(column + "[" + str(j) + "]")]
由于.find_elements_by_xpath将始终返回一个列表，因此直接访问.text将不起作用。