我无法抓取div一些文本"类=" "我认为文本会导致错误

Question

如何像(<div data-v-28872a74="" class="col-lg-10 col-md-10 col-sm-12 col-12 offset-lg-1 offset-md-1 offset-sm-0 offset-0">)一样抓取html。

我试过soup.find_all('div', class_ = 'col-lg-10 col-md-10 col-sm-12 col-12 offset-lg-1 offset-md-1 offset-sm-0 offset-0')，但输出只是[]。

实际代码:

div数据- v - 28872 - a74 =";">

import requests
from bs4 import BeautifulSoup as bs
url = 'https://remart.az/yasayis-kompleksi?cities=1&districts='
result = requests.get(url)
soup = bs(result.text, 'html.parser')
code= soup.find_all('div', class_ = 'col-lg-10 col-md-10  col-sm-12 col-12  offset-lg-1 offset-md-1 offset-sm-0 offset-0')
print(code)

第二个代码刮掉了url，但在下一个代码中我看到了同样的问题。

driver = webdriver.Chrome(r'C:Program Files (x86)chromedriver_win32chromedriver.exe')
driver.get('https://remart.az/yasayis-kompleksi?cities=1&districts=')
time.sleep(3)
aze = driver.find_element(By.XPATH, '//*[@id="app"]/div[2]/div[1]/div[2]/div[6]/button')

for a in range(1,2):
aze.click()
time.sleep(1)

soup = bs(driver.page_source, "html.parser")
aezexx = soup.find_all('div', class_ = 'bitem')
for parent in aezexx:
a_tag = parent.find("a")
URRL = a_tag.attrs['href']
print(URRL)

soup = bs(driver.page_source, "html.parser")
aezexx = soup.find_all('div', class_ = 'bitem')
for parent in aezexx:
a_tag = parent.find("a")
URRL = a_tag.attrs['href']
result = requests.get(URRL)
soup = bs(result.text, 'html.parser')
are = soup.find_all("div", class_ = 'bottom-panel-descripton cut-text')
for aes in are:
azzzz = aes.find_all('p')
print(azzzz) 

Answer 1

尝试:

import re
import json
import requests
import pandas as pd
from ast import literal_eval
url = "https://remart.az/yasayis-kompleksi?cities=1&districts="
html_doc = requests.get(url).text
data = re.search(r'window.__INITIAL_STATE__ = (".*")', html_doc).group(1)
data = json.loads(literal_eval(data))
df = pd.DataFrame(data)
del df["descr"]
df["city"] = df["city"].str["name"]
df["district"] = df["district"].str["name"]
print(df.head())

打印:

id                  name  status  company_id    end_date land_area  contact_person                                         website                      email                                                   phones  housing_count  block_count floor_count  apartment_count apartments_on_floor_count elevator_count                                       address  city_id  district_id orient_ids          lat          lng  underground_garage underground_garage_floor_count  underground_garage_place_count objects_floor_count objects_area infr_items infr_additional_items  credit interest_rate  maximum_installment_period minimum_initial_deposit  payment_graph  mortgage mortgage_interest_rate  mortgage_duration mortgage_initial_deposit partner_banks created_at updated_at  seen                 title                               description                                                                                                                                                                                                                                                                                                                                                                              keywords  kupcha  currency  village_id  metro_id  recommended  country_id foreign_price                                    image_cover                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             image_condo image_construction location_info infrastructure_info                                                                                        full_payment_comment credit_comment mortgage_comment documents_comment                  slug min_price min_price_apart  city      district
0  464  Golden Rose Boutique       1         296  2022-03-31      0.20             NaN                                            None                       None                                        [phone]            3.0          3.0           7             42.0                         2              1            Necef Nerimanov küç., 1979 məhellə        1            3       None         None         None                   0                           None                             NaN                None         None       None                  None       1          0.00                        12.0                   20.00              1         0                   8.00              240.0                    40.00          None       None       None   0.0  Golden Rose Boutique  Golden Rose Boutique - premium menziller  golden rose boutique, golden_rose_boutique, golden-rose-boutique, lalafo, korter, bina, yeniemlak, residence yaşayış kompleksi, yeniemlak, kreditlə yaşayış kompleksindəki mənzillər, yaşayış kompleksindəki mənzillərin qiyməti, yaşayış kompleksində mənzillərin alınması, kredit, ipoteka, Bakı, satış, yeni bina, yeni tikililər, mənzillər, mənzillər, otaqlar , modern, layihe       0         1         NaN       1.0            0           1          None  condos/September2021/xMhblXRcyffXvVk90AaK.jpg                                                                     ["condos/September2021/RadK9ymfGUCGzJqQsrpT.jpg","condos/September2021/F1JEiifMCxfUAPTFQATP.jpg","condos/September2021/qTRHpWZS9O3PczUBPMDr.jpg","condos/September2021/mzJp8VNyylp1GsPRyCWb.jpg","condos/September2021/59vJ3NLfluOjNmndQcj2.jpg","condos/September2021/hNrBMAuI04cyzxUrXHxc.jpg","condos/September2021/kzbi7vDumwFdvdprzS4i.jpg","condos/September2021/r90BM5Du1EGRzoQ6c3i5.jpg","condos/September2021/LVXvZeCNjOFij11wi5ag.jpg"]               None                                                                     <p>Mənzillərin 1 kvadrat metrinin nağd qiyməti 2450 manatdan başlayır</p>                                                    golden-rose-boutique   2450.00     237037.5000  Bakı  Nərimanov r.
1  463        Central Towers       1           9  2024-12-12      None             NaN                               srconstruction.az     [e-mail]                                     *1144, 050 988 11 44            3.0          3.0          16            178.0                       4-5              2                                                      1           12       None  40.38121100  49.82461600                   0                           None                             NaN                None         None       None                  None       0          None                         NaN                    None              1         1                  10.00               20.0                    30.00          None       None       None   0.0                                                                                                                                                                                                                                                                                                                                                                                                                                                             0         1         NaN      11.0            0           1          None  condos/September2021/6o8EAc6jDh99QjedN4ao.jpg                                                                     ["condos/September2021/zpLYa1KtFyZ7O56HyifJ.jpg","condos/September2021/qZctiglmI8WAPnTbjxAa.jpg","condos/September2021/FZvCsxXwtY6IGm1Mckkr.jpg","condos/September2021/ZIK9JgbiPr2k7p2pPRqk.jpg","condos/September2021/TTevd9TqrF6Zas25WIYl.jpg","condos/September2021/avYtaaEoT7cBkRADL5B7.jpg","condos/September2021/ocFd2JG7LbstKI12uCTY.jpg","condos/September2021/SF1p6C9nDzCBay94aBvl.jpg","condos/September2021/BeAlNNHu2Om5Btw4Od3p.jpg"]                 []                                                                                                                                                                                                        central-towers   1500.00     122830.0000  Bakı    Yasamal r.
2  462               SkyHome       1          18  2023-12-12      1.00             NaN  kristalabsheron.az/az/project/index/53/skyhome  [e-mail]                                                    *1544            3.0          3.0       16-18              NaN                      None              2                                                      1            5       None  40.38811800  49.81547200                   0                           None                             NaN                None         None       None                  None       0          1.00                         1.0                    1.00              1         1                  10.00               20.0                    20.00          None       None       None   0.0                                                                                                                                                                                                                                                                                                                                                                                                                                                             0         1         NaN       2.0            0           1          None     condos/August2021/KLh7WNZqsWizOytX6ABU.jpg                                                                                                                                                                                                                                                                                                                                                                                          ["condos/August2021/YkSXADyK9Q75mjasBwvJ.jpg","condos/August2021/hem8X1Mhq6loKwzPTkab.jpg","condos/August2021/nZY36EpVaixrOBNZKm26.jpg"]               None                                                                                                                                                                                                               skyhome   1750.00      94675.0000  Bakı     Nizami r.
3  461              Yuqa MTK       1         271  2021-08-08      None             NaN                                       resant.az  [e-mail]                                *4445,  [phone]            1.0          1.0          16             96.0                         6              2                                                      1            3       None  40.40601900  49.86819800                   0                           None                             NaN                None         None       None                  None       1          0.00                        24.0                   30.00              1         0                   None                NaN                     None          None       None       None   1.0                                                                                                                                                                                                                                                                                                                                                                                                                                                             1         1         NaN       NaN            0           1          None     condos/August2021/HtY9SPYpvCMy2AzSIYo6.jpg  ["condos/August2021/rSB7TAGpKy5bWG4YlpMa.jpg","condos/August2021/moXSg5i7ovKaM4Mxaxog.jpg","condos/August2021/HfTYzb3miKSLUp3nx6ZK.jpg","condos/August2021/WVQErnMZWlNIm08aCZxE.jpg","condos/August2021/AABftVGeTNkAcLOPjk20.jpg","condos/August2021/oj8f2wmWDmTxK2TvuJaX.jpg","condos/August2021/7VVuckWYC1pEquPmKX1c.jpg","condos/August2021/JC2T54WRukjFNoPRp63y.jpg","condos/August2021/erAitfcyK2LAdzZUuYwY.jpg","condos/August2021/hrMR8D5hlk0EAHmk18Y4.jpg","condos/August2021/jVldUFH35AssuMj0ZBdx.jpg"]               None                                                                                                                                                                                                              yuqa-mtk   2000.00     180000.0000  Bakı  Nərimanov r.
4  460               Zəfər 1       1         211  2023-09-09      None             NaN                              zefer1.rezidens.az           [e-mail]  [phone], [phone], [phone]            1.0          1.0          14             65.0                         5              2  Bakıxanov qəsəbəsi, S. Mehmandarov küçəsi, 5        1            7       None  40.38286700  49.96533800                   0                           None                             NaN                None         None       None                  None       1          0.00                        36.0                   50.00              1         0                   None                NaN                     None          None       None       None   1.0               Zəfər 1       Az mənzilli bina - Zəfər 1 layihəsi                 zəfər 1, zəfər_1, zəfər-1, biznes klass kompleksi, lalafo, korter, bina, yeniemlak, residence yaşayış kompleksi, yeniemlak, kreditlə yaşayış kompleksindəki mənzillər, yaşayış kompleksindəki mənzillərin qiyməti, yaşayış kompleksində mənzillərin alınması, kredit, ipoteka, Bakı, satış, yeni bina, yeni tikililər, mənzillər, mənzillər, otaqlar , modern, layihe       0         1         NaN      15.0            0           1          None     condos/August2021/7FKhwnY5qCd4W3owxKV1.jpg                                                                                                                                                                                              ["condos/August2021/8XZe2oTNyYEOUR6xDmx9.jpg","condos/August2021/WHfgTRvFM2lyYuPtvdSy.jpg","condos/August2021/ypBFEfcJLzjirVYs3QtD.jpg","condos/August2021/QrNYHSt3BvB7uRZ8SBFV.jpg","condos/August2021/QjXVTS03mGrSAwbJbknM.jpg","condos/August2021/uKtJRJoB9H2bQuWYqMcO.jpg","condos/August2021/pp7xqqrGoLznhxZ8pPJu.jpg"]               None                                    <p>Baxış istiqamətindən və mərtəbədən asılı olmayaraq qiymətlər 1,250 manatdan başlayır</p>n<p>&nbsp;</p>                                                                 zefer-1      0.00          0.0000  Bakı    Sabunçu r.