div在使用BeautifulSoup刮取价格时返回空，其他所有内容(如标题、图片链接)都返回一个值.为什么会发生这种情

Question

我需要产品的价格。我去标题，图片链接。我想知道为什么div返回空，而div不是？

import requests, bs4, math
import urllib.request
res = requests.get('https://www.zara.com/ca/en/turtleneck-sweater-p07148300.html?v1=8389726&v2=1179974', 
headers={'User-agent': 'Mozilla/5.0 Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36'})
zara = bs4.BeautifulSoup(res.text,'html.parser')
price = zara.find('div', attrs={'class':"price _product-price"})
print(price)`

返回

<div class="price _product-price" data-qa-qualifier="product-price" tabindex="0"> </div>

div有的地方

<div class="price _product-price" tabindex="0" data-qa-qualifier="product-price">
<span>49.90 CAD</span>
</div>

Answer 1

选项1:

该表由Javascript生成，但在页面源中有该表的JSON数据。

要获取数据，您可以使用BeautifulSoup和json。我还使用了regex，并包含了更多的数据。这将迭代每个产品，但如果你注意到尺寸/sku之间没有价格变化，只需举第一个例子，而不是迭代。

import requests
import bs4
import json
import re
res = requests.get('https://www.zara.com/ca/en/turtleneck-sweater-p07148300.html?v1=8389726&v2=1179974', 
headers={'User-agent': 'Mozilla/5.0 Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36'})
zara = bs4.BeautifulSoup(res.text,'html.parser')
scripts = zara.find_all('script')
jsonObj = None
for script in scripts:
if '@context' in script.text:
jsonStr = script.text
jsonObj = json.loads(jsonStr)
for product in jsonObj:
name = product['name']
sku = product['sku']
price = product['offers']['price']
availabililty = product['offers']['availability'].split('/')[-1]
availabililty = [s for s in re.split("([A-Z][^A-Z]*)", availabililty) if s]
availabililty = ' '.join(availabililty)
print('Name: %s   SKU: %s   Price: %0.2f  Availability: %s' %(name, sku, float(price), availabililty))

输出：

Name: TURTLENECK SWEATER   SKU: 8389726-809-3   Price: 49.90  Availability: In Stock
Name: TURTLENECK SWEATER   SKU: 8389726-809-4   Price: 49.90  Availability: In Stock
Name: TURTLENECK SWEATER   SKU: 8389726-809-5   Price: 49.90  Availability: In Stock
Name: TURTLENECK SWEATER   SKU: 8389726-809-6   Price: 49.90  Availability: In Stock
Name: TURTLENECK SWEATER   SKU: 8389726-250-3   Price: 49.90  Availability: In Stock
Name: TURTLENECK SWEATER   SKU: 8389726-250-4   Price: 49.90  Availability: In Stock
Name: TURTLENECK SWEATER   SKU: 8389726-250-5   Price: 49.90  Availability: In Stock
Name: TURTLENECK SWEATER   SKU: 8389726-250-6   Price: 49.90  Availability: Out Of Stock
Name: TURTLENECK SWEATER   SKU: 8389726-401-3   Price: 49.90  Availability: In Stock
Name: TURTLENECK SWEATER   SKU: 8389726-401-4   Price: 49.90  Availability: In Stock
Name: TURTLENECK SWEATER   SKU: 8389726-401-5   Price: 49.90  Availability: In Stock
Name: TURTLENECK SWEATER   SKU: 8389726-401-6   Price: 49.90  Availability: Out Of Stock
Name: TURTLENECK SWEATER   SKU: 8389726-800-3   Price: 49.90  Availability: Out Of Stock
Name: TURTLENECK SWEATER   SKU: 8389726-800-4   Price: 49.90  Availability: In Stock
Name: TURTLENECK SWEATER   SKU: 8389726-800-5   Price: 49.90  Availability: Out Of Stock
Name: TURTLENECK SWEATER   SKU: 8389726-800-6   Price: 49.90  Availability: Out Of Stock

选项2:

由于页面是动态的，请使用Selenium模拟打开浏览器，让页面呈现，然后获取html。

import bs4 
from selenium import webdriver 
url = 'https://www.zara.com/ca/en/turtleneck-sweater-p07148300.html?v1=8389726&v2=1179974'
browser = webdriver.Chrome('C:chromedriver_win32chromedriver.exe')
browser.get(url)
res = browser.page_source
zara = bs4.BeautifulSoup(res,'html.parser')  
price = zara.find('div', attrs={'class':"price _product-price"})
print(price)
browser.close()

输出：

print(price)
<div class="price _product-price" data-qa-qualifier="product-price" tabindex="0"><span>49.90 CAD</span></div>

或者只获得没有标签的价格：

print(price.text)
49.90 CAD

选项3:

使用包请求html它似乎可以抓取一些由JavaScript渲染的文本，但我从未使用过它，因为它与我经常使用的jupyter Notebooks和Spyder相冲突，所以我会出现错误(我过去一直使用Selenium)。所以你可以自己尝试，但下面是文档中的一个例子。

from requests_html import HTMLSession
session = HTMLSession()
r = session.get('http://python-requests.org/')
r.html.render()
r.html.search('Python 2 will retire in only {months} months!')['months']

输出：

'<time>25</time>'