我试图使用美丽的汤刮多个页面，但代码不断返回每个页面相同的数据

Question

我正在尝试使用Python和beautiful soup在steam网站上抓取特价优惠。我试图使用for循环从多个页面刮数据。我附上了下面的Python代码。任何帮助都非常感谢。提前谢谢。

game_lis = set([])
for page in range(0,4):
page_url = "https://store.steampowered.com/specials#p=" +str(page)+"&tab=TopSellers"
#print(page_url)
steam_games = requests.get(page_url)
soup = BeautifulSoup(steam_games.text, 'lxml')
s_game_offers = soup.findAll('a', class_='tab_item')
print(page_url)
for game in s_game_offers:
title = game.find('div',class_='tab_item_name')
discount = game.find('div',class_='discount_pct')
game_lis.add(title.text)
print(title.text+":"+discount.text)

Answer 1

这个页面是通过JavaScript从不同的URL加载的，所以beautifulsoup看不到它。您可以使用下面的示例来加载不同的页面:

import requests
from bs4 import BeautifulSoup
api_url = "https://store.steampowered.com/contenthub/querypaginated/specials/TopSellers/render/"
params = {
"query": "",
"start": "0",
"count": "15",
"cc": "SK",  # <-- probably change code here
"l": "english",
"v": "4",
"tag": "",
}
for page in range(0, 4):
params["start"] = 15 * page
steam_games = requests.get(api_url, params=params)
soup = BeautifulSoup(steam_games.json()["results_html"], "lxml")
s_game_offers = soup.findAll("a", class_="tab_item")
for game in s_game_offers:
title = game.find("div", class_="tab_item_name")
discount = game.find("div", class_="discount_pct")
print(title.text + ":" + discount.text)
print("-" * 80)

打印:

F.I.S.T.: Forged In Shadow Torch:-10%
HITMAN 2 - Gold Edition:-85%
NieR:Automata™:-50%
Horizon Zero Dawn™ Complete Edition:-40%
Need for Speed™ Heat:-86%
Middle-earth: Shadow of War Definitive Edition:-80%
Batman: Arkham Collection:-80%
No Man's Sky:-50%
Legion TD 2 - Multiplayer Tower Defense:-20%
NieR Replicant™ ver.1.22474487139...:-35%
Days Gone:-20%
Mortal Kombat 11 Ultimate:-65%
Human: Fall Flat:-66%
Muse Dash - Just as planned:-30%
The Elder Scrolls Online - Blackwood:-50%
--------------------------------------------------------------------------------
The Elder Scrolls Online - Blackwood:-50%
Football Manager 2022:-10%
Age of Empires II: Definitive Edition:-33%
OCTOPATH TRAVELER™:-50%
DRAGON QUEST® XI S: Echoes of an Elusive Age™ - Definitive Edition:-35%
Witch It:-70%
Monster Hunter: World:-34%
NARUTO SHIPPUDEN: Ultimate Ninja STORM 4:-77%
MADNESS: Project Nexus:-10%
Mad Max:-75%
Outer Wilds:-40%
Middle-earth: Shadow of Mordor Game of the Year Edition:-75%
Age of Empires III: Definitive Edition:-33%
Ghostrunner:-60%
The Elder Scrolls® Online:-60%
--------------------------------------------------------------------------------
...