网络报废(电子商务)-空列表

Question

我正试图通过使用漂亮的汤来废弃一个电子商务网站。我注意到主页的HTML不完整。此外，当我试图找到几乎任何项目时，我的python列表都会变空。

这是代码：

import requests
import bs4
res = requests.get("https://www.example.com")
soup = bs4.BeautifulSoup(res.content,"lxml")
productlist = soup.find_all('div', class_='items-IW-')
print(productlist)

Answer 1

您在页面上看到的数据是通过JavaScript从外部URL加载的。您可以使用requests来模拟它。例如：

import json
import requests

url = "https://8r21li.a.searchspring.io/api/search/search.json"
params = {
"resultsFormat": "native",
"resultsPerPage": "24",
"page": "1",
"siteId": "8r21li",
"bgfilter.ss_category": "Women",
}
data = requests.get(url, params=params).json()
# uncomment to print all data:
# print(json.dumps(data, indent=4))
for r in data["results"]:
print("{:<50} {:<10}".format(r["name"], r["price"]))

打印：

KMDCore Polypro Long Sleeve Unisex Top             29.98     
Andulo Women's Rain Jacket v3                      139.98    
Epiq Women's 600 Fill Hooded Down Jacket           199.98    
KMDCore Polypro Unisex Long Johns                  29.98     
Heli Women's Hooded Lightweight Down Vest          70        
Accion driMOTION Low Cut Unisex Socks – 3Pk        31.49     
Pocket-it Women’s Two Layer Rain Jacket            70        
Unisex Thermo Socks 2Pk                            34.98     
NuYarn Ergonomic Unisex Hiking Socks               34.98     
KMDCore Polypro Short Sleeve Unisex Top            29.98     
Fyfe Unisex Beanie                                 17.49     
Makino Travel Skort                                79.98     
Miro Women’s 3/4 Pants                             69.98     
Ridge 100 Women’s Primaloft Bio Pullover           59.98     
Epiq Women's 600 Fill Down Vest                    149.98    
Epiq Women's 600 Fill Down Jacket                  179.98    
Flinders Women’s Pants                             89.98     
Flight Women’s Shorts                              59.98     
ANY-Time Sweats Joggers                            79.98     
Kamana Women’s Tapered Pants                       89.98     
NuYarn Ergonomic Quarter Crew Unisex Hiking Socks  31.49     
Bealey Women’s GORE-TEX Jacket                     190       
Winterburn Women’s 600 Fill Longline Down Coat     299.98    
Kathmandu Badge Unisex Beanie                      27.98