当使用Python3时，是否可以获得selenium headless chrome XHR响应

Question

我正在使用selenium google chrome headless来捕获一些内容，我试图做的是解析网页源代码并获得我想要的数据列表。我必须编写一些解析html和css代码来获得我想要的内容。代码如下：

@staticmethod
def fetch_music_download_url_impl(music_name: str):
chrome_driver_service = Service(ChromeDriverManager(chrome_type=ChromeType.GOOGLE).install())
chrome_options = Options()
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--headless")
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--remote-debugging-port=9230")
driver = webdriver.Chrome(service=chrome_driver_service,
options=chrome_options,
executable_path="/usr/local/bin/chromedriver")
try:
driver.maximize_window()
driver.get('http://example.cn/music/?page=audioPage&type=migu&name=' + music_name)
driver.implicitly_wait(5)
driver.find_element(By.CSS_SELECTOR, ".aplayer-list-download.iconfont.icon-xiazai").click()
urls = [a.get_attribute('href') for a in
driver.execute_script('return document.querySelectorAll(".modal-body a[href*='http']")')]
for url in urls:
if "listenSong.do" in url:
logger.info("fetched url:" + url)
FetchMusic.do_save_music_download_url(url)
except Exception as e:
logger.error("scrapy impl error", e)
finally:
driver.stop_client()
driver.close()
driver.quit()
chrome_driver_service.stop()

这段代码很有效，这只是一个演示，如果我想让它更完美，我必须写更多的代码来匹配数据。我想知道有没有可能用我想要的json内容来获得谷歌chrome无头XHR响应内容。所以我可以做更简单的步骤，只需解析json响应。这将更加简单可靠，节省资源。

我应该怎么做才能得到谷歌chrome无头XHR json响应？

Answer 1

我用selenium-wire这样做：

from seleniumwire import webdriver

然后使用这个代码：

@staticmethod
def fetch_search_xhr_response(driver: any):
for request in driver.requests:
if request.response and r'api/search' in request.url:
if request.response.status_code == http.client.OK:
search_response = request.response.body
json_response = json.loads(search_response)
if json_response["code"] == http.client.OK:
FetchMusic.handle_music_list(json_response["data"])