在 Python 中使用 Selenium 或 Request 不会读取所有 span 类

Question

为什么我在使用Request或Selenium抓取电子商务网站时没有得到所有的HTML代码？

所以，这是我的代码：

html = "https://www.tokopedia.com/p/fashion-anak-bayi/pakaian-anak-laki-laki/baju-tidur-anak-laki-laki?page=1&wholesale=true&goldmerchant=true&fcity=174,175,176,177,178,179"
header = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36 RuxitSynthetic/1.0 v6366394992 t38550 ath9b965f92 altpub',
"Upgrade-Insecure-Requests": "1",
"DNT": "1",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Encoding": "gzip, deflate"}
web = requests.get(html,headers=header, data=data)
pageweb = BeautifulSoup(web.content, 'html.parser')
#Get name and Location
store_loc = pageweb.find_all('span',{'class':'css-1kr22w3'})

但是，结果并没有出现在所有div类span中。如果我选择硒，那么答案仍然是一样的。

我要找的课并没有出现在所有的课上。

我使用这些代码来满足我对抓取和获取数据的好奇心。

Answer 1

试试这个：

import requests
from bs4 import BeautifulSoup
url = "https://www.tokopedia.com/p/fashion-anak-bayi/pakaian-anak-laki-laki/baju-tidur-anak-laki-laki?page=1&wholesale=true&goldmerchant=true&fcity=174,175,176,177,178,179"
headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36 RuxitSynthetic/1.0 v6366394992 t38550 ath9b965f92 altpub',
"Upgrade-Insecure-Requests": "1",
"DNT": "1",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Encoding": "gzip, deflate"
}
page = BeautifulSoup(requests.get(url, headers=headers).text, 'html.parser')
store_loc = page.find_all('span', {'class': 'css-1kr22w3'})
print(len(store_loc))
for tag in store_loc:
print(tag.text)

20项目的输出。

Jakarta Utara
Chloe Clozette
Jakarta Utara
Chloe Clozette
Jakarta Barat
AICOMFY Collection
...