尝试获取某些页面数据时出现AttributeErrors

Question

您好。我正试图从url中获取一些数据，然而，由于遇到错误，我的脚本只有几行可以工作。任何想法都可以。谢谢

from bs4 import BeautifulSoup
from urllib.request import Request, urlopen
import requests
header = {"User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:92.0) Gecko/20100101 Firefox/92.0"}
url = "https://www.pinksale.finance/#/pinklock/detail/0x5f7faccaff14ce5fe5ae2ff5bb1ea2fa1b7fc526?chain=BSC"
print ("Link:", url)
urlpage = requests.get(url, headers=header, timeout=10, allow_redirects=False)
site = BeautifulSoup(urlpage.content, 'html.parser')
item1 = site.find('div', class_='ant-list-item').get_text()
item2 = site.find('div', class_='Total Amount Locked').get_text()
item3 = site.find('div', class_='Total Values Locked').get_text()
print ("item1: ", item1)
print ("item2: ", item2)
print ("item3: ", item3)

电流输出：

Link: https://www.pinksale.finance/#/pinklock/detail/0x5f7faccaff14ce5fe5ae2ff5bb1ea2fa1b7fc526?chain=BSC
AttributeError: 'NoneType' object has no attribute 'get_text'

想要输出：

0x7A9b...c0b2    7.104591955602949963  2022.04.04 16:32 UTC
Total Amount Locked 7.10459195560294996
Total Values Locked $1,843

Answer 1

url完全依赖于JavaScript。所以你需要像硒这样的自动化。现在它正在按预期工作。你可以直接运行代码。

import time
from bs4 import BeautifulSoup
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
url ='https://www.pinksale.finance/#/pinklock/detail/0x5f7faccaff14ce5fe5ae2ff5bb1ea2fa1b7fc526?chain=BSC'
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.maximize_window()
driver.get(url)
time.sleep(8)
site = BeautifulSoup(driver.page_source, 'lxml')
driver.close()
item1 = ','.join([x.get_text().strip() for x in site.select('.ant-spin-container div ul li div.LockRecord_tvl__1cBpD')])
item2 = [x.get_text() for x in site.select('td.has-text-right')]
print ("item1: ", item1)
print ("item2: ", item2[0])
print ("item3: ", item2[1])

输出：

item1: 0x7A9b...c0b2,7.104591955602949963,2022.04.04 16:32 UTC      
item2:  7.104591955602949963
item3:  $1,711