我正在尝试访问此网站上的数据:https://vemcount.app/embed/widget/uOCRuLPangWo5fT?locale=en
到目前为止,我的代码如下:
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import TimeoutException
def configure_driver():
chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--use-fake-ui-for-media-stream")
driver = webdriver.Chrome(executable_path="C:\Users\uqiabde1\Downloads\chromedriver.exe", options = chrome_options)
return driver
def getNumber(driver):
# Step 1: Go to website
driver.get(f"https://vemcount.app/embed/widget/uOCRuLPangWo5fT?locale=en")
# wait for the element to load
try:
WebDriverWait(driver, 10).until(lambda s: s.find_element_by_id("flex items-center").is_displayed())
except TimeoutException:
print("TimeoutException: Element not found")
return None
# Step 2: Create a parse tree of page sources after searching
soup = BeautifulSoup(driver.page_source, "lxml")
# Step 3: Iterate over the search result and fetch the number
for number in soup.select("div.items-center"):
number_needed = "p span"
print({
"title": number.select_one(number_needed).text,
})
# create the driver object.
driver = configure_driver()
getNumber(driver)
# close the driver.
driver.close()
我在彩色打印机中得到以下错误
[0414/15051.086:信息:控制台(2(]";不允许启动AudioContext。用户在页面上做出手势后,必须恢复(或创建(它。https://goo[点]gl/7C7WLu";,来源:https://vemcount.app/build/embed.js?id=2ff0173dd78c5c1f99c6(2(
我不确定使用哪个chrome_option来绕过此错误。我试过一些,比如
--no-user-gesture-required
和
--disable-gesture-requirement-for-presentation
非常感谢您的帮助。谢谢
为了防止您有同样的查询或在动态web抓取方面遇到困难,我找到了一种不使用selenium
和bs4
抓取数据的方法。
我使用了更直接的player,它有一个非常受欢迎的inner_html()
函数,可以直接读取动态flex HTML代码。这是供参考的代码。
#part of the help to write the script I got from https://stackoverflow.com/questions/64303326/using-playwright-for-python-how-do-i-select-or-find-an-element
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch(slow_mo=1000)
page = browser.new_page()
page.goto('https://vemcount.app/embed/widget/uOCRuLPangWo5fT')
central = page.query_selector("p.w-full span");
print({'central': central.inner_html()})
browser.close()
如果有更好的方法,我非常乐意听取你的建议。
你的,
Python初学者。:(