我正在尝试从url获取内容，并使用BeautyfulSoup解析响应.
这个url在加载时检索我最喜欢的监视列表项，问题是，当网站加载时，它需要几秒钟才能在表中显示数据，所以当我运行urlopen(my_url)时，响应没有表，因此我的解析方法失败。
我试图保持它简单，因为我正在学习的语言，所以我想使用的工具，我已经设置在我的代码，所以基于我有我想知道是否有一种方法等待，或检查当内容准备好了，我能够获取数据(表内容)。
下面是我的代码:from bs4 import BeautifulSoup as soup
from urllib.request import urlopen as ureq
from urllib.error import URLError, HTTPError
URL = 'url route goes here' # In compliance to SO rules I've removed the website path
def get_dom_from_url():
try:
u_client = ureq(URL)
html = u_client.read()
u_client.close()
except HTTPError as e:
print(f'There has been an HTTP ERROR: {e.code}')
except URLError as e:
print(f'There has been a problem reaching the URL. ERROR: {e.code}')
finally:
print('''
DOM loaded!
''')
return html
dom = soup(get_dom_from_url(), 'html.parser')
# Crawl the dom object and get the table thead element
col_names = [col.text for col in dom.table.thead.find_all('th')]
col_names = col_names[1:-2]
col_names
这是错误信息:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-102-625de133b2e2> in <module>
----> 1 col_names = [col.text for col in dom.table.thead.find_all('th')]
2 col_names = col_names[1:-2]
3 col_names
AttributeError: 'NoneType' object has no attribute 'thead'
当我加载没有路由的url时，上面的代码可以工作，但我需要它，因为我需要为我正在处理的ETL管道存储相同的数据。
如果只使用urllib没有办法实现这一点，我想听听你的建议。
实际上您不需要在这里使用Selenium。数据嵌入在<script>标签在一个有效的json格式的源html。只需要解析它:
import requests
from bs4 import BeautifulSoup
import json
import pandas as pd
url = 'https://coinmarketcap.com/watchlist/60321ee5b01cab343e1e37d6/'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
jsonStr = soup.find('script', {'id':'__NEXT_DATA__'}).text
jsonData = json.loads(jsonStr)
data = jsonData['props']['initialProps']['pageProps']['fetchedWatchlist']['cryptoCurrencies']
rows = []
for each in data:
quotes_row = each.pop('quotes')[0]
each.pop('tags')
if 'platform' in each.keys():
each.pop('platform')
each.update(quotes_row)
rows.append(each)
df = pd.DataFrame(rows)
输出:
print(df.to_string())
id name symbol slug status rank marketPairCount circulatingSupply totalSupply maxSupply lastUpdated dateAdded price volume24h marketCap percentChange1h percentChange24h percentChange7d
0 1 USD BTC bitcoin active 1 9717 1.863544e+07 1.863544e+07 2.100000e+07 2021-02-22T09:37:02.000Z 2013-04-28T00:00:00.000Z 55579.249971 5.656584e+10 1.035744e+12 -1.232746 -1.234765 16.978174
1 1027 USD ETH ethereum active 2 5982 1.147732e+08 1.147732e+08 NaN 2021-02-22T09:37:02.000Z 2015-08-07T00:00:00.000Z 1855.072456 2.450605e+10 2.129125e+11 -1.373583 -4.104364 5.315240
2 1839 USD BNB binance-coin active 3 469 1.545328e+08 1.705328e+08 1.705328e+08 2021-02-22T09:37:11.000Z 2017-07-25T00:00:00.000Z 272.095668 6.811884e+09 4.204770e+10 -2.381284 2.937286 109.533310
3 825 USD USDT tether active 4 10829 3.445054e+10 3.570817e+10 NaN 2021-02-22T09:37:08.000Z 2015-02-25T00:00:00.000Z 0.999576 1.087710e+11 3.443593e+10 -0.061248 -0.023795 -0.074917
4 6636 USD DOT polkadot-new active 5 145 9.103144e+08 1.045967e+09 NaN 2021-02-22T09:36:05.000Z 2020-08-19T00:00:00.000Z 37.503515 3.257901e+09 3.413999e+10 -1.327435 -2.635214 40.263648
5 2010 USD ADA cardano active 6 231 3.111248e+10 4.500000e+10 4.500000e+10 2021-02-22T09:37:09.000Z 2017-10-01T00:00:00.000Z 1.040491 6.621492e+09 3.237226e+10 -1.594681 -7.316003 25.951127
6 52 USD XRP xrp active 7 673 4.540403e+10 9.999083e+10 1.000000e+11 2021-02-22T09:38:03.000Z 2013-08-04T00:00:00.000Z 0.581321 1.102498e+10 2.639430e+10 -1.640063 11.286157 2.731301
7 2 USD LTC litecoin active 8 754 6.653055e+07 6.653055e+07 8.400000e+07 2021-02-22T09:38:02.000Z 2013-04-28T00:00:00.000Z 216.783950 6.530638e+09 1.442276e+10 -2.134667 -3.477237 5.932102
8 1975 USD LINK chainlink active 9 471 4.085096e+08 1.000000e+09 1.000000e+09 2021-02-22T09:37:11.000Z 2017-09-20T00:00:00.000Z 32.145503 1.885830e+09 1.313174e+10 -1.378857 -5.152372 -0.036835
9 1831 USD BCH bitcoin-cash active 10 581 1.866177e+07 1.866177e+07 2.100000e+07 2021-02-22T09:37:07.000Z 2017-07-23T00:00:00.000Z 679.047253 5.800439e+09 1.267222e+10 -1.298651 -0.162108 -1.595937
正如评论中提到的，urllib.request是相当古老的，Selenium可以处理javascript:
from selenium import webdriver
from bs4 import BeautifulSoup as soup
options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
wd = webdriver.Chrome('chromedriver',options=options)
wd.get("https://coinmarketcap.com/watchlist/60321ee5b01cab343e1e37d6")
dom = soup(wd.page_source, 'html.parser')
col_names = [col.text for col in dom.table.thead.find_all('th')]
col_names = col_names[1:-2]
col_names
输出:
['Name', 'Price', '24h', '7d', 'Market Cap', 'Volume', 'Circulating Supply']
我不确定你的目的是什么，但你也可以直接加载表到pandas而不使用BeautifulSoup:
import pandas as pd
df = pd.read_html(wd.page_source)[0]
df = df.iloc[:, 1:-2] # drop first and last two columns
df.head():
| | Name | Price | 24h | 7d | Market Cap | Volume | Circulating Supply |
|---:|:-----------------|:-----------|:------|:--------|:-------------------|:---------------------------------|:---------------------|
| 0 | Bitcoin1BTC | $57,515.75 | 3.67% | 17.00% | $1,069,904,656,718 | $65,190,502,8631,135,430 BTC | 18,634,643 BTC |
| 1 | Ethereum2ETH | $1,972.76 | 1.32% | 7.03% | $225,668,814,800 | $30,790,664,03815,658,144 ETH | 114,760,589 ETH |
| 2 | Binance Coin3BNB | $285.07 | 4.20% | 113.31% | $43,505,990,437 | $9,923,804,82135,249,242 BNB | 154,532,785 BNB |
| 3 | Polkadot4DOT | $39.69 | 4.23% | 39.72% | $36,117,982,640 | $5,555,973,830139,996,164 DOT | 910,079,701 DOT |
| 4 | Cardano5ADA | $1.14 | 8.25% | 31.00% | $35,603,496,069 | $10,299,401,9179,000,239,285 ADA | 31,112,484,646 ADA |