运行我的刮板,我可以看到它刮擦了不必要的链接,而不是指向每所学校的链接。不过,我创建了正确的XPaths。该网站包含懒惰方法。也许需要抓住JSON回应。我尝试了:
import requests
from lxml import html
url = "http://www.boarding.org.au/find-a-school"
def LazyLoadWeb(address):
try :
page = requests.get(address, timeout=30)
except Exception:
print('timed out')
else:
tree = html.fromstring(page.text)
titles = tree.xpath('//div[contains(@class,"clearfix")]')
for title in titles:
links=title.xpath('.//a/@href')
for link in links:
print(link)
LazyLoadWeb(url)
您对JSON响应是正确的。该站点使用Ajax填充内容。您需要提出邮政请求,然后简单地从响应中解析JSON。
import requests
url = 'http://www.boarding.org.au/ajax-calls/GetSchoolsJson'
payload = {"state": 'null', "schoolType": 2, "orderMode": "ASC", "enableSchoolType": 'false', "loadAll": 'true'}
req = requests.post(url, json=payload)
data = req.json()
for i, item in enumerate(data, start=1):
print(i, item['URL'])
# 1 /schools/details/4/Abbotsleigh
# ...
# 189 /schools/details/83/Yirara-College