如何抓取网站抓取的所有价格

Question

就是这样，&这个看起来有点好笑…将打印前10个价格只从zillow。你怎么写这个来抓取页面上的所有条目呢?(如果你漂亮，还请告诉我如何将所有价格保存为变量列表<3)

通过XPATH (LXML)抓取所有项

from bs4 import BeautifulSoup
from lxml import etree
import requests
URL = "https://www.zillow.com/losangeles-ca/
HEADERS = {'User-Agent':'Mozilla/5.0'}

webpage = requests.get(URL, headers=HEADERS)
soup = BeautifulSoup(webpage.content, "html.parser")
dom = etree.HTML(str(soup))
x=1
while x<41:
print(dom.xpath('/html/body/div[1]/div[5]/div/div/div/div[1]/ul/li['+str(x)+']/article/div[1]/div[2]/div')[0].text)
x+=1

每个项目列表的差异是/li[x]/，因此这将循环遍历项目列表。

有更简单的方法吗?你会怎么写呢?

(即使不是xpath)

Answer 1

for x in soup.findAll('li'):
for y in x.findAll('article'):
for z in y.findAll('div', {'class': 'list-card-info'}):
try:
print('URL', z.a['href'])
for aa in z.findAll('div', {'class': 'list-card-heading'}):
for bb in aa.findAll('div', {'class': 'list-card-price'}):
print('PRICE', bb.get_text())
except TypeError:
pass