图像网页抓取工具工作不正确

Question

我正在构建一个网络抓取应用程序，该应用程序可以从网站上的每个广告中获取文本信息和图像。负责文本信息的代码段(正常工作(：

for a in soup.find_all("div", {"class": "c6e8ba5398--subtitle--UTwbQ"}):
a = a.text.split(',')
name.append(a[0])
sqr.append(a[1])
floor.append(a[2])
table['Name'] = name
table['Sqr'] = sqr
table['Floor'] = floor

负责图像的代码段：

for a in soup.find_all("div", {"class": "c6e8ba5398--container--Y5gG9"}):
photo.clear()
for b in a.find_all("img", {"class":"c6e8ba5398--image--3ua1b"}):
photo.append(b['src'])
photos.append(photo)

问题是，如果列出相同的广告，我会得到列表：

[['adv_1_img_1','adv_1_img_2','adv_1_img_3'],['adv_1_img_1','adv_1_img_2','adv_1_img_3']]

同时我需要这样的东西：

[['adv_1_img_1','adv_1_img_2','adv_1_img_3'],['adv_2_img_1','adv_2_img_2','adv_2_img_3']]

Answer 1

见下文(在将子列表添加到主照片列表之前，请参阅复制操作(

import requests
from bs4 import BeautifulSoup
import copy
r = requests.get('https://www.cian.ru/snyat-3-komnatnuyu-kvartiru/')
soup = BeautifulSoup(r.content, 'html.parser')
photos = []
photos_sub_list = []
for div in soup.find_all("div", {"class": "c6e8ba5398--container--Y5gG9"}):
photos_sub_list.clear()
for _photo in div.find_all("img", {"class": "c6e8ba5398--image--3ua1b"}):
photos_sub_list.append(_photo['src'])
photos.append(copy.deepcopy(photos_sub_list))
for sub_list in photos:
print(sub_list)

输出

['https://cdn-p.cian.site/images/8/239/617/kvartira-moskva-presnenskaya-naberezhnaya-716932893-4.jpg', 'https://cdn-p.cian.site/images/8/239/617/kvartira-moskva-presnenskaya-naberezhnaya-716932894-2.jpg', 'https://cdn-p.cian.site/images/8/239/617/kvartira-moskva-presnenskaya-naberezhnaya-716932895-2.jpg', 'https://cdn-p.cian.site/images/8/239/617/kvartira-moskva-presnenskaya-naberezhnaya-716932898-2.jpg', 'https://cdn-p.cian.site/images/8/239/617/kvartira-moskva-presnenskaya-naberezhnaya-716932896-2.jpg']
['https://cdn-p.cian.site/images2/7/393/857/kvartira-moskva-presnenskaya-naberezhnaya-758393739-4.jpg', 'https://cdn-p.cian.site/images2/7/393/857/kvartira-moskva-presnenskaya-naberezhnaya-758393740-2.jpg', 'https://cdn-p.cian.site/images2/7/393/857/kvartira-moskva-presnenskaya-naberezhnaya-758393742-2.jpg', 'https://cdn-p.cian.site/images2/7/393/857/kvartira-moskva-presnenskaya-naberezhnaya-758393743-2.jpg', 'https://cdn-p.cian.site/images2/7/393/857/kvartira-moskva-presnenskaya-naberezhnaya-758393744-2.jpg']
['https://cdn-p.cian.site/images2/8/453/557/kvartira-moskva-presnenskaya-naberezhnaya-755354822-4.jpg', 'https://cdn-p.cian.site/images2/8/453/557/kvartira-moskva-presnenskaya-naberezhnaya-755354822-2.jpg', 'https://cdn-p.cian.site/images2/8/453/557/kvartira-moskva-presnenskaya-naberezhnaya-755354824-2.jpg', 'https://cdn-p.cian.site/images2/8/453/557/kvartira-moskva-presnenskaya-naberezhnaya-755354820-2.jpg', 'https://cdn-p.cian.site/images2/8/453/557/kvartira-moskva-presnenskaya-naberezhnaya-755354823-2.jpg']
...

下面是代码的较短版本(但可读性不强(。

import requests
from bs4 import BeautifulSoup
r = requests.get('https://www.cian.ru/snyat-3-komnatnuyu-kvartiru/')
soup = BeautifulSoup(r.content, 'html.parser')
photos = [[img['src'] for img in div.find_all("img", {"class": "c6e8ba5398--image--3ua1b"})] for div in
soup.find_all("div", {"class": "c6e8ba5398--container--Y5gG9"})]
for sub_list in photos:
print(sub_list)