价格不会显示在 html 解析中

Question

我正在尝试将此对象的价格转换为变量并将其打印出来/放入CSV中。

这是我试图解析的html部分:

<span class="price" data-js-product-price="">
<span>$429.00 USD</span>
</span>

这是我的python代码(对不起，我对python有点陌生，我已经试着弄清楚了一段时间，所以抱歉，如果代码有点乱)

from bs4 import BeautifulSoup

url_to_scrape = "https://www.backfireboards.com/?gclid=CjwKCAjwjtOTBhAvEiwASG4bCGHPgmV4XjyqAIFrW0Lr0IiW0AvfTiC7sZ4E-HtM_qJ9k4ahAu2CHxoCH5YQAvD_BwE"
request_page = urlopen(url_to_scrape)
page_html = request_page.read()
request_page.close()
html_soup = BeautifulSoup(page_html, 'html.parser')
board_prices = html_soup.find_all('span', class_='price')
print("num of prices: " + str(len(board_prices)))
file_name = 'product.csv'
f = open(file_name,'w')
headers = 'Title, Price n'
f.write(headers)
i = 1
for price in board_prices:
currPrice = price.span.text
print(i)
i = i + 1
print(price)
print(currPrice)

f.close()

下面是我得到的错误:

Traceback (most recent call last):
File "/Users/isaiah/PycharmProjects/Web_scrape/main.py", line 26, in <module>
currPrice = price.span.text
AttributeError: 'NoneType' object has no attribute 'text'

我知道这不是一个文本对象，但是当我打印它时，不带。text，它会输出如下内容:

1
<span class="price" data-js-popup-cart-subtotal=""></span>
None
2
<span class="price" data-js-product-price=""><span></span></span>
<span></span>
3
<span class="price" data-js-product-price=""><span></span></span>
<span></span>
4
<span class="price"><span></span></span>
<span></span>
5
<span class="price" data-js-product-price=""><span></span></span>
<span></span>
6
<span class="price" data-js-product-price=""><span></span></span>
<span></span>

我不知道为什么429美元在对象中消失了，我对网络抓取之类的有点陌生。是不是有什么很简单的事情我也不知道?

此外，从我可以告诉网站实际上有8个价格在这个页面上列出，我是从board_prices对象只是大小6?有人能给我解释一下吗?

Answer 1

您在页面上看到的数据是从外部源加载的。要获取电动滑板的名称/价格，可以使用以下示例:

import requests
from bs4 import BeautifulSoup

url = "https://www.backfireboards.com/collections/electric-skateboards"
soup = BeautifulSoup(requests.get(url).content, "html.parser")
for h4 in soup.select(".product-collection__content h4"):
title = h4.get_text(strip=True)
price = h4.find_next(class_="price").contents[-1].text
print("{:<15} {}".format(price, title))

打印:

$429.00 USD     Backfire G2 Black with Super Power Hobbywing Motors and 96mm Wheels with 180 Days Warranty Especially Suitable for Beginners
$649.00 USD     Backfire G3 with Super Flexible Deck
$799.00 USD     Backfire Zealot S Belt Drive Electric Skateboard
$1,899.00 USD   Backfire Hammer Sledge
$1,199.00 USD   Backfire Hammer Belt Drive All Terrain Electric Skateboard
$649.00 USD     Backfire Zealot Belt Drive Electric Skateboard
$1,399.00 USD   Backfire Ranger X3 All Terrain Electric Skateboard with 1500W X2 Ultra High Power Ultra High Torque Motors and 12S High Voltage High Efficiency Electronic System
$999.00 USD     Backfire Ranger X2 All Terrain Electric Skateboard with 1200W X2 Ultra High Power Ultra High Torque Motors and 12S High Voltage High Efficiency Electronic System
$599.00 USD     Backfire Mini Super Portable Electric Skateboard Best for City Commute
$999.00 USD     Backfire G3 Plus with Carbon Fiber Deck and Ultra Long Range
$499.00 USD     Backfire ERA Electric Skateboard