假设我正在查看此网页
https://openpaymentsdata.cms.gov/search/physicians/by-name-and-location?firstname=robert&lastname=b&city=Palo_Alto
我想提取医生档案的链接，但当我尝试网络抓取时，我找不到元素，即使使用CSS选择器也是如此。
from requests_html import HTMLSession
firstname = 'robert'
lastname = 'b'
city = 'Palo_Alto'
url = 'https://openpaymentsdata.cms.gov/search/physicians/by-name-and-location?firstname='
+ firstname + '&lastname=' + lastname + '&city=' + city
session = HTMLSession()
r = session.get(url)
sel = 'body > div.siteOuterWrapper > div.siteInnerWrapper > div.siteContentWrapper'
print(r.html.find(sel, first=True).text)
这一切都有效，直到我到达内容包装器，在那里我再也看不到任何元素了。为什么会这样？我看不到这个元素是什么原因？起初我以为这是因为Javascript，但这个库声称完全支持Javascripthttps://requests-html.kennethreitz.org/
下面的HTTP请求应该返回您要查找的数据。(在浏览器中执行F12>网络>XHR(
HTTP GET https://openpaymentsdata.cms.gov/resource/khdp-6xuy.json?%24select=%3Aid%2Cphysician_profile_id%2Cphysician_profile_last_name%2Cphysician_profile_middle_name%2Cphysician_profile_first_name%2Cphysician_profile_suffix%2Cphysician_profile_primary_specialty%2Cphysician_profile_address_line_1%2Cphysician_profile_address_line_2%2Cphysician_profile_city%2Cphysician_profile_state%2Cphysician_profile_province_name%2Cphysician_profile_country_name%2Cphysician_profile_zipcode%2Cphysician_profile_alternate_first_name1%2Cphysician_profile_alternate_last_name1%2Cphysician_profile_alternate_first_name2%2Cphysician_profile_alternate_last_name2%2Cphysician_profile_alternate_first_name3%2Cphysician_profile_alternate_last_name3%2Cphysician_profile_alternate_first_name4%2Cphysician_profile_alternate_last_name4%2Cphysician_profile_alternate_first_name5%2Cphysician_profile_alternate_last_name5%2Clocation&%24where=STARTS_WITH(UPPER(physician_profile_first_name)%2C%20%27ROBERT%27)%20AND%20STARTS_WITH(UPPER(physician_profile_last_name)%2C%20%27B%27)%20AND%20STARTS_WITH(UPPER(physician_profile_city)%2C%20%27PALO_ALTO%27)&%24order=physician_profile_last_name%20ASC%2Cphysician_profile_first_name%20ASC&%24limit=300
使用请求
print(requests.get('https://openpaymentsdata.cms.gov/resource/khdp-6xuy.json?%24select=%3Aid%2Cphysician_profile_id%2Cphysician_profile_last_name%2Cphysician_profile_middle_name%2Cphysician_profile_first_name%2Cphysician_profile_suffix%2Cphysician_profile_primary_specialty%2Cphysician_profile_address_line_1%2Cphysician_profile_address_line_2%2Cphysician_profile_city%2Cphysician_profile_state%2Cphysician_profile_province_name%2Cphysician_profile_country_name%2Cphysician_profile_zipcode%2Cphysician_profile_alternate_first_name1%2Cphysician_profile_alternate_last_name1%2Cphysician_profile_alternate_first_name2%2Cphysician_profile_alternate_last_name2%2Cphysician_profile_alternate_first_name3%2Cphysician_profile_alternate_last_name3%2Cphysician_profile_alternate_first_name4%2Cphysician_profile_alternate_last_name4%2Cphysician_profile_alternate_first_name5%2Cphysician_profile_alternate_last_name5%2Clocation&%24where=STARTS_WITH(UPPER(physician_profile_first_name)%2C%20%27ROBERT%27)%20AND%20STARTS_WITH(UPPER(physician_profile_last_name)%2C%20%27B%27)%20AND%20STARTS_WITH(UPPER(physician_profile_city)%2C%20%27PALO_ALTO%27)&%24order=physician_profile_last_name%20ASC%2Cphysician_profile_first_name%20ASC&%24limit=300').json())
输出
[{':id': 'row-9mfk-w6hd-ejup', 'physician_profile_id': '966387', 'physician_profile_last_name': 'BOCIAN', 'physician_profile_middle_name': 'C', 'physician_profile_first_name': 'ROBERT', 'physician_profile_primary_specialty': 'Allopathic & Osteopathic Physicians|Allergy & Immunology|Allergy', 'physician_profile_address_line_1': '795 EL CAMINO REAL', 'physician_profile_city': 'PALO ALTO', 'physician_profile_state': 'CA', 'physician_profile_country_name': 'UNITED STATES', 'physician_profile_zipcode': '94301-2302', 'physician_profile_alternate_first_name1': 'ROBERT', 'physician_profile_alternate_last_name1': 'BOCIAN'}]
您提到的站点从API获取数据-this。
您可以使用requests直接向该API发出GET请求并获取数据。
您可以使用Chrome Devtools找到API端点。