试图用python3和beautifulSoup抓取一个网站，但得到一个空列表

Question

我正在尝试创建一个简单的脚本，从网站上打印比特币的当前价格：https://www.bitcoinprice.com/

#This is a script that returns the price of one Bitcoin in US dollars
import requests
from bs4 import BeautifulSoup
url = "https://www.bitcoinprice.com/"
#Get the url to beautifulSoup format
res = requests.get(url)
res.raise_for_status()
soup = BeautifulSoup(res.text)
#get the price
element = soup.select("#price")

当我尝试打印"element"变量时，我得到这个：

<span id="price" style="font-size: 72px; font-weight: bold;"></span>

我相信这是由页面上的JavaScript引起的。有没有更好的方法可以从网站上获取价格？

Answer 1

是的，你就在这里。该值由Javascript不断更新，并且未被Scrapper捕获。
我建议您使用网站用于收集数据的后端API。
有一个开放的API，您可以在一定时间(假设1分钟(后点击并直接获取数据而不是报废。



GET https://min-api.cryptocompare.com/data/pricemultifull?fsyms=BTC&tsyms=USD&e=CCCAGGJSON 响应如下，

{
"RAW": {
"BTC": {
"USD": {
"TYPE": "5",
"MARKET": "CCCAGG",
"FROMSYMBOL": "BTC",
"TOSYMBOL": "USD",
"FLAGS": "4",
"PRICE": 6544.82,
"LASTUPDATE": 1539801288,
"LASTVOLUME": 0.004,
"LASTVOLUMETO": 25.74948,
"LASTTRADEID": "52589962",
"VOLUMEDAY": 27152.289698759938,
"VOLUMEDAYTO": 178354574.93607947,
"VOLUME24HOUR": 38151.0738459291,
"VOLUME24HOURTO": 250157534.15733647,
"OPENDAY": 6582.86,
"HIGHDAY": 6589.2,
"LOWDAY": 6511.6,
"OPEN24HOUR": 6541.77,
"HIGH24HOUR": 6598.58,
"LOW24HOUR": 6504.37,
"LASTMARKET": "Coinbase",
"CHANGE24HOUR": 3.0499999999992724,
"CHANGEPCT24HOUR": 0.04662346734903967,
"CHANGEDAY": -38.039999999999964,
"CHANGEPCTDAY": -0.5778643325241607,
"SUPPLY": 17327237,
"MKTCAP": 113403647262.34,
"TOTALVOLUME24H": 256273.81355038052,
"TOTALVOLUME24HTO": 1677731603.429824
}
}
},
"DISPLAY": {
"BTC": {
"USD": {
"FROMSYMBOL": "Ƀ",
"TOSYMBOL": "$",
"MARKET": "CryptoCompare Index",
"PRICE": "$ 6,544.82",
"LASTUPDATE": "Just now",
"LASTVOLUME": "Ƀ 0.004000",
"LASTVOLUMETO": "$ 25.75",
"LASTTRADEID": "52589962",
"VOLUMEDAY": "Ƀ 27,152.3",
"VOLUMEDAYTO": "$ 178,354,574.9",
"VOLUME24HOUR": "Ƀ 38,151.1",
"VOLUME24HOURTO": "$ 250,157,534.2",
"OPENDAY": "$ 6,582.86",
"HIGHDAY": "$ 6,589.20",
"LOWDAY": "$ 6,511.60",
"OPEN24HOUR": "$ 6,541.77",
"HIGH24HOUR": "$ 6,598.58",
"LOW24HOUR": "$ 6,504.37",
"LASTMARKET": "Coinbase",
"CHANGE24HOUR": "$ 3.05",
"CHANGEPCT24HOUR": "0.05",
"CHANGEDAY": "$ -38.04",
"CHANGEPCTDAY": "-0.58",
"SUPPLY": "Ƀ 17,327,237.0",
"MKTCAP": "$ 113.40 B",
"TOTALVOLUME24H": "Ƀ 256.27 K",
"TOTALVOLUME24HTO": "$ 1,677.73 M"
}
}
}
}

PRICE键保存所需的核心转发数据。