PythonBeautifulSoup-表在按id抓取时不返回任何值

Question

我想从下面给定的url中抓取每日观察表https://www.wunderground.com/history/daily/in/chennai/VOMM/date/2017-1-1

我想使用表id进行刮擦。我正在使用此代码

from bs4 import BeautifulSoup
import requests
import lxml
url = 'https://www.wunderground.com/history/daily/in/chennai/VOMM/date/2017-1-1';
content = requests.get(url).content
soup = BeautifulSoup(content, 'lxml')
table = soup.find('table', {'id' : 'history-observation-table'})
print(table)

但这是无回报。我怎样才能擦桌子？

Answer 1

It动态页面，您可以使用来自URL的json数据，如

https://api.weather.com/v1/geocode/12.99361134/80.17694092/observations/historical.json?apiKey=*********&startDate=20170101&endDate=20170101&units=e

你可以看到真正的API键它在浏览器控制台->网络

或使用硒

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait 
driver = webdriver.Chrome()
driver.get("https://www.wunderground.com/history/daily/in/chennai/VOMM/date/2017-1-1")
table = WebDriverWait(driver, 15).until(lambda d: d.find_element_by_id('history-observation-table'))
print(table.text)