在Web抓取脚本上获取403错误

Question

我有一个web抓取脚本，最近遇到403错误。它只使用基本代码工作了一段时间，但现在已经遇到403个错误。我曾尝试使用用户代理来绕过这一点，它的作用非常短暂，但现在也出现了403错误。

有人知道如何重新运行这个脚本吗

如果有帮助的话，下面是一些上下文：脚本的目的是找出哪些艺术家在哪些Tidal播放列表上，为了这个问题的目的——我只包含了获取网站的代码片段，因为这就是错误发生的地方。

提前感谢！

基本代码如下：

baseurl = 'https://tidal.com/browse'
for i in platformlist:
url = baseurl+str(i[0])
tidal = requests.get(url)
tidal.raise_for_status()
if tidal.status_code != 200:
print ("Website Error: ", url)
pass
else:
soup = bs4.BeautifulSoup(tidal.text,"lxml")
text = str(soup)
text2 = text.lower()

使用用户代理：

user_agent_list = [
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Safari/605.1.15',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko/20100101 Firefox/77.0',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36',
]
url = 'https://tidal.com/playlist/1b418bb8-90a7-4f87-901d-707993838346'
for i in range(1,4):
#Pick a random user agent
user_agent = random.choice(user_agent_list)
#Set the headers 
headers = {'User-Agent': user_agent}
#Make the request
tidal = requests.get(url,headers=headers)

print("Request #%dnUser-Agent Sent:%snnHeaders Received by HTTPBin:"%(i,user_agent))
print(tidal.status_code)
print("-------------------")
#tidal = requests.get(webpage)
tidal.raise_for_status()
print(tidal.status_code)
#make webpage content legible
soup = bs4.BeautifulSoup(tidal.text,"lxml")
print(soup)
#turn bs4 type content into text
text = str(soup)
text2 = text.lower()

Answer 1

我想提出一个替代解决方案——一个不涉及BeautifulSoup的解决方案。

我访问了主页，点击了一个相册，同时记录了我的网络流量。我注意到我的浏览器向GraphQL API发出了HTTPPOST请求，后者接受一个自定义查询字符串作为POST负载的一部分，该负载指示响应数据的格式。响应是JSON，它包含我们用原始查询字符串请求的所有信息(在本例中，是播放列表中每首曲目的所有艺术家(。通常情况下，页面使用此API使用JavaScript异步填充自己，这是在浏览器中查看页面时通常会发生的情况。由于我们有API端点、请求头和POST有效负载，我们可以在Python中模拟该请求以获得JSON响应：

def main():
import requests
url = "https://tidal.com/browse/api"
headers = {
"accept": "application/json",
"accept-encoding": "gzip, deflate",
"content-type": "application/json",
"user-agent": "Mozilla/5.0"
}
query = """
query ($playlistId: String!) {
playlist(uuid: $playlistId) {
creator {
name
}
title
tracks {
albumID
albumTitle
artists {
id
name
}
id
title
}
}
}
"""
payload = {
"operationName": None,
"query": query,
"variables": {
"playlistId": "1b418bb8-90a7-4f87-901d-707993838346"
}
}
response = requests.post(url, headers=headers, json=payload)
response.raise_for_status()
playlist = response.json()["data"]["playlist"]
print("Artists in playlist "{}":".format(playlist["title"]))
for track_number, track in enumerate(playlist["tracks"], start=1):
artists = ", ".join(artist["name"] for artist in track["artists"])
print("Track #{} [{}]: {}".format(track_number, track["title"], artists))
return 0

if __name__ == "__main__":
import sys
sys.exit(main())

输出：

Artists in playlist "New Arrivals":
Track #1 [Higher Power]: Coldplay
Track #2 [i n t e r l u d e]: J. Cole
Track #3 [Fast (Motion)]: Saweetie
Track #4 [Miss The Rage]: Trippie Redd, Playboi Carti
Track #5 [In My Feelings (feat. Quavo & Young Dolph)]: Tee Grizzley, Quavo, Young Dolph
Track #6 [Thumbin]: Kash Doll
Track #7 [Tiempo]: Ozuna
...

您可以更改payload字典中的playlistId键值对，以获取任何播放列表的艺术家信息。

看看我发布的另一个答案，我将更深入地了解如何记录网络流量、查找API端点和模仿请求。