如何使用python从网站中的链接页面中提取数据

Question

我一直在尝试为数据分析项目从网页中抓取数据，并成功地从单个页面中获取了数据。

import requests
from bs4 import BeautifulSoup
import concurrent.futures
from urllib.parse import urlencode
from scraper_api import ScraperAPIClient

client = ScraperAPIClient('key')
results = client.get(url = "https://www.essex.ac.uk/course-search?query=&f.Level%7CcourseLevel=Undergraduate").text

print(results)

对于来自网站"；https://www.essex.ac.uk/course-search?query=&f.级别%7课程级别=本科生"；我需要在每个课程中导航，并从该页面获取一个名为持续时间的数据。

Answer 1

尝试以下操作：

client = ScraperAPIClient('key')
results = []
for i in range(10):
results.append(client.get(url = f"https://www.essex.ac.uk/course-search?query=&f.Level%7CcourseLevel=Undergraduate&start_rank={i}1").text)

print(results)

循环浏览10个结果页面，并将每个文本响应放入结果列表中

Answer 2

import requests
from bs4 import BeautifulSoup
import concurrent.futures
from urllib.parse import urlencode
from scraper_api import ScraperAPIClient
client = ScraperAPIClient('key')
total_pages = 12
for page_no in range(total_pages):
# you control this page_no variable.
# go to the website and see how the api go to the next page
# it depends on the 'start_rank' at the end of the URL
# for example start_rank=10, start_rank=20 will get you one page after another
rank = page_no * 10
results = client.get(url="https://www.essex.ac.uk/course-search?query=&f.Level%7CcourseLevel=Undergraduate&start_rank={0}".format(rank)).text
print(results)