使用selenium(python)循环访问webelements列表以检索属性会产生StaleElementRefer

Question

我使用Selenium webdriver在一个网站上进行搜索，该网站为我提供了多个页面，每个页面上都有几个链接。我的目标是创建搜索产生的所有链接的列表。

它适用于第一页，但是当我在第二页上时，我尝试提取"href"属性的 for 循环会产生 StaleElementReferenceException。

我相信应该有一些方法可以使用WebDriverWait解决此问题，但我无法完全弄清楚具体如何。感谢您的任何建议。

links =[]
while True:
  result = driver.find_element_by_id('results')
# list of all relevant elements on the page
  docs = result.find_elements_by_xpath('//li[@class="row1"]')
  for d in docs:
# this line produces StaleElementReferenceException
    link = d.find_element_by_tag_name('a')
    links.append(link.get_attribute('href'))
# try block checks if "next page" button exists and clicks it (this works fine)
  try:
    next_page = WebDriverWait(driver, 30).until(EC.presence_of_element_located((By.XPATH, '//a[contains(@class,"la-TriangleRight")]')))
    driver.execute_script('arguments[0].click();', next_page)
# if "next page" button doesn't exist we break out of the loop
  except:
    break

编辑：错误信息：

StaleElementReferenceException            Traceback (most recent call last)
<ipython-input-6-ad6166a696e6> in <module>
     23 #         link = WebDriverWait(d, 30).until(EC.presence_of_element_located((By.XPATH, '//a')))
     24         link = d.find_element_by_tag_name('a')
---> 25         links.append(link.get_attribute('href'))
     26 
     27 #     for i in range(25):
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/webelement.py in get_attribute(self, name)
    139             attributeValue = self.parent.execute_script(
    140                 "return (%s).apply(null, arguments);" % getAttribute_js,
--> 141                 self, name)
    142         else:
    143             resp = self._execute(Command.GET_ELEMENT_ATTRIBUTE, {'name': name})
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py in execute_script(self, script, *args)
    634         return self.execute(command, {
    635             'script': script,
--> 636             'args': converted_args})['value']
    637 
    638     def execute_async_script(self, script, *args):
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py in execute(self, driver_command, params)
    319         response = self.command_executor.execute(driver_command, params)
    320         if response:
--> 321             self.error_handler.check_response(response)
    322             response['value'] = self._unwrap_value(
    323                 response.get('value', None))
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py in check_response(self, response)
    240                 alert_text = value['alert'].get('text')
    241             raise exception_class(message, screen, stacktrace, alert_text)
--> 242         raise exception_class(message, screen, stacktrace)
    243 
    244     def _value_or_default(self, obj, key, default):
StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
  (Session info: chrome=75.0.3770.100)

Answer 1

相反，对于每个循环，请尝试使用带有范围的 for 循环。

links =[]
while True:
  result = driver.find_element_by_id('results')
# list of all relevant elements on the page
  docs = result.find_elements_by_xpath('//li[@class="row1"]')
  for i in range(len(docs)):
# this line produces StaleElementReferenceException
    WebDriverWait(driver, 30).until(EC.staleness_of(docs[i]))
    link = docs[i].find_element_by_tag_name('a')
    links.append(link.get_attribute('href'))
    result = driver.find_element_by_id('results')
    docs = result.find_elements_by_xpath('//li[@class="row1"]')
# try block checks if "next page" button exists and clicks it (this works fine)
  try:
    next_page = WebDriverWait(driver, 30).until(EC.presence_of_element_located((By.XPATH, '//a[contains(@class,"la-TriangleRight")]')))
    driver.execute_script('arguments[0].click();', next_page)
# if "next page" button doesn't exist we break out of the loop
  except:
    break