requests.html .find方法返回空数组(css选择器没有问题)

Question

我正在试着抓取公海的排名页面。

下面的代码返回正确的HTML Element对象。但下一个返回一个空数组

# this one works fine
from requests_html import HTMLSession
url = 'https://opensea.io/rankings'
session = HTMLSession()
r = session.get(url)
collections = r.html.find('#main > div > div.sc-1xf18x6-0.enKNee > div > div:nth-child(3)')
print(collections)

两者的区别在于CSS选择器是否包含最后一个div

# this one works fine
from requests_html import HTMLSession
url = 'https://opensea.io/rankings'
session = HTMLSession()
r = session.get(url)
collections = r.html.find('#main > div > div.sc-1xf18x6-0.enKNee > div > div:nth-child(3) > div')
print(collections)

谢谢

Answer 1

我使用这个代码获得了所有100个数据。

search_url = 'https://opensea.io/rankings'
driver = webdriver.Chrome(executable_path=f"C:chromedriverchromedriver.exe")
driver.get(search_url)  
driver.maximize_window()
def RemoveExtraSpace(string):
new = ' '.join(string.split())
return newp = 1
while True:
try:
ele = WebDriverWait(driver, 4).until(EC.presence_of_element_located((By.XPATH, f'//div[text()="{p}"]'))).find_element(By.XPATH, '..').find_element(By.XPATH, '..').find_element(By.XPATH, '..').find_element(By.XPATH, '..')
print(RemoveExtraSpace(ele.text))
p+=1
driver.execute_script("window.scrollBy(0, arguments[0]);", 85)
except Exception as e:
print(str(e))
driver.quit()
break