使用Python3，我想从vivino.com中的葡萄酒页面上抓取评论；社区评论"；，这就是我想要刮的。但是我的代码返回一个空数据。事实上，从requests.get(url)的内容来看，我看不到评论部分。有人能帮我找出原因并提出解决方案吗？
这是我的尝试：
from bs4 import BeautifulSoup
import requests
import time
url = "https://www.vivino.com/FR/en/dauprat-pauillac/w/3823873?year=2017&price_id=24797287"
headers = headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.106 Safari/537.36'}
data = requests.get(url, headers = headers) # this returns data but isn't capturing the section with review comments.
soup = BeautifulSoup(data.text, 'html.parser')
soup.find_all('div', id='all_reviews')
>>>
[]
使用API动态地检查负载。
response = requests.get('https://www.vivino.com/api/wines/3823873/reviews?per_page=4&year=2017'
reviews = response.json()
评论是从另一个URL动态加载的。你可以用这个例子来引导他们：
import re
import json
import requests
headers = {
"User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:89.0) Gecko/20100101 Firefox/89.0",
}
url = "https://www.vivino.com/FR/en/dauprat-pauillac/w/3823873?year=2017&price_id=24797287"
api_url = (
"https://www.vivino.com/api/wines/{id}/reviews?per_page=9999&year={year}"
) # <-- increased the number of reviews to 9999
id_ = re.search(r"/(d{5,})", url).group(1)
year = re.search(r"year=(d+)", url).group(1)
data = requests.get(api_url.format(id=id_, year=year), headers=headers).json()
# uncomment this to print all data:
# print(json.dumps(data, indent=4))
for r in data["reviews"]:
print(r["note"])
print("-" * 80)
打印：
Out the bottle - oak barrel bruja and cherry acidity, in decant - smooth toffee, vanilla, dark fruit and church air tannins and that cool mushroom midsection into a wonderful taste bud saturating peppered opacity of big barrelled Bordeaux. Never thought I would “leave” a wine to age, now want to get a boatload of these to bide a while, that nauseating term of “ageing potential” 🤮.
Sits deep and brooding in the decanter, could swear a gator breached briefly in slow lid blink.
Magical.
--------------------------------------------------------------------------------
Nice colour, mid depth, purple red at edge, blackcurrant fruit, some spice, little cherry.
Nice fresh entry; clean and pure fruit characters; bit of graphite and blackcurrant on the palate; nice texture.
--------------------------------------------------------------------------------
Dark ruby color. Blackberry, blueberry and tobacco. Medium body. Subtle and elegant.
--------------------------------------------------------------------------------
huge wine from a great region, palpable potential and structure. elegant tannins
--------------------------------------------------------------------------------
Lovly Paulliac , great price beautiful Bouquet
--------------------------------------------------------------------------------
A little young but if you decant for 2 hours it shines
--------------------------------------------------------------------------------
Great powerful Pauillac
--------------------------------------------------------------------------------
When I saw the low price, I had low expectations but this was a really powerful wine with a nice Cabernet backbone.
--------------------------------------------------------------------------------
Wonderful nose and still young... with loads of acid & tannins, this should age very well. Recommend waiting a few years to enjoy more fully.
--------------------------------------------------------------------------------
Volumes of ripe strawberry, tobacco, vanilla, pencil shavings, balsamic
Medium+ body
Intense, dry fruitiness, punchy acidity, powerful tannins
--------------------------------------------------------------------------------
earthy and dark fruits on the nose, purple hue, with dark fruits and oaky flavours. Lunch outside with Mum and Dad to celebrate Suzie and Jeremy’s wedding.
--------------------------------------------------------------------------------
Deep purple. Mint and vanilla nose with touch of mocha. Very attractive. Very good depth on palate with wonderful ripeness and extract. Really nice. Well done.
--------------------------------------------------------------------------------
Such an interesting wine.next door to paulliacs premier grand cru latour and Lynch bages.made by the owners of du glana..50 of a ha in the midst of excellence old vines and excellent itself.but just a paulliac!
--------------------------------------------------------------------------------
Earthy and Cohiba notes
--------------------------------------------------------------------------------
Fruit could be bolder, but well structured with ample tannins and graphite, mushroom and dark fruit notes. Elegant expression of the terroir.
--------------------------------------------------------------------------------
...and so on.