硒搜索ID不总是工作?

Question

在过去，当一个网站"延迟加载"时，我经常会遇到问题。-

它帮助当我使用这样的搜索id

element = driver.find_element_by_id ("analyst-estimate")
driver.execute_script ("arguments[0].scrollIntoView();", element)

现在我看到这并不适用于每个站点

在以下站点上一切正常:

link = "https://www.gurufocus.com/stock/AAPL/summary"
options = Options ()
options.add_argument ('--headless')
options.add_experimental_option ('excludeSwitches', ['enable-logging'])
path = os.path.abspath (os.path.dirname (sys.argv[0]))
if platform == "win32": cd = '/chromedriver.exe'
elif platform == "linux": cd = '/chromedriver_linux'
elif platform == "darwin": cd = '/chromedriver'
driver = webdriver.Chrome (path + cd, options=options)
driver.get (link)  # Read link
time.sleep (2)  # Wait till the full site is loaded
element = driver.find_element_by_id ("analyst-estimate")
driver.execute_script ("arguments[0].scrollIntoView();", element)
time.sleep (1)

但在另一个网站(也有一个id -它根本不起作用)

link = "https://finance.yahoo.com/quote/MSFT/analysis?p=MSFT"
options = Options ()
options.add_argument ('--headless')
options.add_experimental_option ('excludeSwitches', ['enable-logging'])
path = os.path.abspath (os.path.dirname (sys.argv[0]))
if platform == "win32": cd = '/chromedriver.exe'
elif platform == "linux": cd = '/chromedriver_linux'
elif platform == "darwin": cd = '/chromedriver'
driver = webdriver.Chrome (path + cd, options=options)
driver.get (link)  # Read link
time.sleep (2)  # Wait till the full site is loaded
element = driver.find_element_by_id ("YDC-Col1")
# element = driver.find_element_by_id ("Col2-4-QuoteModule-Proxy")
# element = driver.find_element_by_id ("app")
driver.execute_script ("arguments[0].scrollIntoView();", element)
time.sleep (1)

为什么第二个网站不工作?这是完全相同的代码-为什么他找不到id -它存在于网页上?

Answer 1

在加载页面之前会弹出一个接受cookie的提示，你必须先点击它:

WebDriverWait(driver, 5).until(
EC.presence_of_element_located((By.NAME, "agree"))).click()
WebDriverWait(driver, 5).until(
EC.presence_of_element_located((By.ID, "YDC-Col1")))

在headless模式下测试一些东西之前，检查一下非headless模式下的实际行为，如果它只在headless模式下失败，请截图以了解失败期间网站的状态。

你可以截图如下:

try:
link = "https://finance.yahoo.com/quote/MSFT/analysis?p=MSFT"
options = ChromeOptions()
options.add_argument('--headless')
options.add_experimental_option('excludeSwitches', ['enable-logging'])
driver = webdriver.Chrome(options=options)
driver.get(link)  # Read link
time.sleep(2)  # Wait till the full site is loaded

element = driver.find_element_by_id("YDC-Col1")
# element = driver.find_element_by_id ("Col2-4-QuoteModule-Proxy")
# element = driver.find_element_by_id ("app")
driver.execute_script("arguments[0].scrollIntoView();", element)
time.sleep(1)
except:
driver.get_screenshot_as_file("a.jpeg")

Answer 2

Time.sleep()在等待页面加载时不是很稳定。切换到webdriver等待。而且它似乎不需要2秒加载。

wait = WebDriverWait(driver, 5)
wait.until(EC.presence_of_element_located((By.ID, "YDC-Col1")))

另外一个问题可能是使用headless而不设置窗口大小。

options.add_argument('--headless')
options.add_argument("--window-size=1920,1080")

进口

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC