如何从 span 标签中提取文本 3127 N University St， Peoria， IL 61604

Question

我正在尝试使用Python和Selenium开发一个网络爬虫。当我尝试使用以下代码解析页面时，返回了一个 false 元素。

from selenium import webdriver
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
capabilities = webdriver.DesiredCapabilities().FIREFOX
capabilities["marionette"] = True
binary = FirefoxBinary('C:/Program Files/Mozilla Firefox/firefox.exe')
driver = webdriver.Firefox(firefox_binary=binary, capabilities=capabilities, executable_path="C:\Users\19548\AppData\Local\Programs\Python\Python37\geckodriver.exe")
driver.get("https://www.google.com/search?sxsrf=ACYBGNT9OH8ZZcClzMK-BMwxesqsKeHyTg:1575693566606&q=google+maps+secure+dental&npsic=0&rflfq=1&rlha=0&rllag=41148676,-90063976,60206&tbm=lcl&ved=2ahUKEwjHtb_626LmAhXjzVkKHTpMCLAQtgN6BAgLEAQ&tbs=lrf:!1m4!1u3!2m2!3m1!1e1!1m5!1u15!2m2!15m1!1shas_1wheelchair_1accessible_1entrance!4e2!2m1!1e3!3sIAE,lf:1,lf_ui:4&rldoc=1#rlfi=hd:;si:16368180629414227255,l,Chlnb29nbGUgbWFwcyBzZWN1cmUgZGVudGFsIgOIAQFIxLbOi6yPgIAIWiYKDXNlY3VyZSBkZW50YWwQABABGAAYASINc2VjdXJlIGRlbnRhbA;mv:[[41.6797015,-86.9763612],[39.655607599999996,-90.7386324]]")
element=driver.find_element_by_xpath("""//*[@id="akp_tsuid2"]/div/div/div/div/div/div[1]/div/div[1]/div/div[2]/div/div[2]/div/div/span[2]""")
paragraphs=driver.find_element_by_xpath("""//*[@id="akp_tsuid2"]/div/div/div/div/div/div[1]/div/div[1]/div/div[2]/div/div[2]/div/div/span[2]""")
print (paragraphs.text)

Answer 1

要提取文本3127 N University St， Peoria， IL 61604， 美国您必须诱导WebDriverWaitvisibility_of_element_located()，您可以使用以下任一定位器策略：

• 使用CSS_SELECTOR和文本属性：

  driver.get('https://www.google.com/search?sxsrf=ACYBGNT9OH8ZZcClzMK-BMwxesqsKeHyTg:1575693566606&q=google+maps+secure+dental&npsic=0&rflfq=1&rlha=0&rllag=41148676,-90063976,60206&tbm=lcl&ved=2ahUKEwjHtb_626LmAhXjzVkKHTpMCLAQtgN6BAgLEAQ&tbs=lrf:!1m4!1u3!2m2!3m1!1e1!1m5!1u15!2m2!15m1!1shas_1wheelchair_1accessible_1entrance!4e2!2m1!1e3!3sIAE,lf:1,lf_ui:4&rldoc=1#rlfi=hd:;si:16368180629414227255,l,Chlnb29nbGUgbWFwcyBzZWN1cmUgZGVudGFsIgOIAQFIxLbOi6yPgIAIWiYKDXNlY3VyZSBkZW50YWwQABABGAAYASINc2VjdXJlIGRlbnRhbA;mv:[[41.6797015,-86.9763612],[39.655607599999996,-90.7386324]]')
  print(WebDriverWait(driver, 5).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.mod[data-attrid='kc:/location/location:address']>div>div>span:nth-child(2)"))).text)
  

• 使用XPATH和get_attribute()：

  driver.get('https://www.google.com/search?sxsrf=ACYBGNT9OH8ZZcClzMK-BMwxesqsKeHyTg:1575693566606&q=google+maps+secure+dental&npsic=0&rflfq=1&rlha=0&rllag=41148676,-90063976,60206&tbm=lcl&ved=2ahUKEwjHtb_626LmAhXjzVkKHTpMCLAQtgN6BAgLEAQ&tbs=lrf:!1m4!1u3!2m2!3m1!1e1!1m5!1u15!2m2!15m1!1shas_1wheelchair_1accessible_1entrance!4e2!2m1!1e3!3sIAE,lf:1,lf_ui:4&rldoc=1#rlfi=hd:;si:16368180629414227255,l,Chlnb29nbGUgbWFwcyBzZWN1cmUgZGVudGFsIgOIAQFIxLbOi6yPgIAIWiYKDXNlY3VyZSBkZW50YWwQABABGAAYASINc2VjdXJlIGRlbnRhbA;mv:[[41.6797015,-86.9763612],[39.655607599999996,-90.7386324]]')
  print(WebDriverWait(driver, 5).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='mod' and @data-attrid='kc:/location/location:address']/div/div//following::span[1]"))).get_attribute("innerHTML"))
  

• 控制台输出：

  3127 N University St, Peoria, IL 61604, United States
  

• 注意：您必须添加以下导入：

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  

Answer 2

我通常通过查找标签，然后找到所需的文本(例如"地址："，然后是实际的街道地址(来处理这样的定位器。这使得定位器更清晰、更易于阅读。

对于此处的地址，您可以使用 XPath

//a[.='Address']//following::span

解释

相关的 HTML 看起来像

<div class="zloOqf PZPZlf" data-dtype="d3ifr" data-local-attribute="d3adr" data-ved="2ahUKEwiF7tvpzKTmAhVPOq0KHUoBD9wQghwoADAEegQIARAh">
<span class="w8qArf">
<a class="fl" href="..." data-ved="2ahUKEwiF7tvpzKTmAhVPOq0KHUoBD9wQ6BMwBHoECAEQIg">Address</a>:
</span>
<span class="LrzXr">3127 N University St, Peoria, IL 61604</span>
</div>

因此，我们首先使用

//a[.='Address']

然后从那里我们找到第一个以下 SPAN 标签

//a[.='Address']//following::span

定位器就是这样。仅供参考，您在定位器中指定的越少(在合理范围内(，当页面发生变化时，中断的可能性就越小。

现在，您可以拉动该元素的.text来获得所需的内容。您可能需要添加等待，例如

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
...
driver.get(...)
paragraph = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, "//a[.='Address']//following::span")))
print(paragraph.text)

阅读更多关于 python waits。