BeautifulSoup只在每页有30个元素时返回20个元素

Question

我正在尝试刮数据(药物名称)来自此链接https://www.1mg.com/drugs-all-medicines此链接有841页，每页30个数据。但我的代码每页只能选择20个数据。我不知道是什么原因造成的，也不知道如何解决。这是我正在使用的代码。

import requests
import json
import io
from bs4 import BeautifulSoup
medicine_name = []
f = io.open('data.txt', 'a', encoding='utf-8')
for i in range(1,842):
url = "https://www.1mg.com/drugs-all-medicines?page=" + str(i)
r = requests.get(url)
HTMLcontent = r.content

soup = BeautifulSoup(HTMLcontent, 'html.parser')

json_data = json.loads(
soup.select_one("script").string
)
for data in json_data['itemListElement']:
medicine_name.append(data['name'])
f.write('n'+data['name'])
print("parsed --> " + str(len(medicine_name)) + " from page No. --> " + str(i) + "")
medicine_name = []

f.close()

我得到这样的输出:

PS E:PracticePython1mg Scraper> & D:/Python396/python.exe "e:/Practice/Python/1mg Scraper/tool.py"
parsed --> 20 from page No. --> 1
parsed --> 20 from page No. --> 2
parsed --> 20 from page No. --> 3
parsed --> 20 from page No. --> 4
parsed --> 20 from page No. --> 5
parsed --> 20 from page No. --> 6
parsed --> 20 from page No. --> 7
parsed --> 20 from page No. --> 8
parsed --> 20 from page No. --> 9
...................................
<-----------Upto------------------>
...................................
parsed --> 20 from page No. --> 837
parsed --> 20 from page No. --> 838
parsed --> 20 from page No. --> 839
parsed --> 20 from page No. --> 840
parsed --> 20 from page No. --> 841

我期望输出类似于

parsed --> 30 from page No. --> xxx

Answer 1

尝试指定User-AgentHTTP报头。如果没有它，服务器将返回不同类型的页面:

import json
import requests
from bs4 import BeautifulSoup
headers = {
"User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0"
}

for i in range(1, 842):
url = "https://www.1mg.com/drugs-all-medicines?page=" + str(i)
r = requests.get(url, headers=headers)
soup = BeautifulSoup(r.content, "html.parser")
json_data = json.loads(
soup.select_one('script[type="application/ld+json"]').string
)
for data in json_data["itemListElement"]:
print(data["name"])

每页打印30个产品:

Ascoril D Plus Syrup Sugar Free
Augmentin 625 Duo Tablet
Allegra 180mg Tablet
Azithral 500 Tablet
Ascoril LS Syrup
Avil 25 Tablet
Allegra 120mg Tablet
...and so on.