Selenium返回的html源与在浏览器中查看的不同

Question

我正试图使用Selenium通过单击该网站的"加载更多"按钮来加载下一页的结果。然而，selenium加载的html页面的源代码并没有显示(加载(浏览时可以看到的实际产品。这是我的代码：

from selenium import webdriver      
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
import os
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
#browser = webdriver.Firefox()#Chrome('./chromedriver.exe')
URL = "https://thekrazycouponlady.com/coupons-for/costco"
PATIENCE_TIME = 60
LOAD_MORE_BUTTON_XPATH = '//button[@class = "kcl-btn ng-scope"]/span' 
caps = DesiredCapabilities.PHANTOMJS
# driver = webdriver.Chrome(r'C:Python3seleniumwebdriverchromedriver_win32chromedriver.exe')
caps["phantomjs.page.settings.userAgent"] = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36"
driver = webdriver.PhantomJS(r'C:Python3seleniumwebdriverphantomjs-2.1.1-windowsbinphantomjs.exe',service_log_path=os.path.devnull,desired_capabilities=caps)
driver.get(URL)
while True:
try:
time.sleep(20)
html = driver.page_source.encode('utf-8')
print(html)
loadMoreButton = driver.find_element_by_xpath(LOAD_MORE_BUTTON_XPATH)

loadMoreButton.click()
except Exception as e:
print (e)
break
print ("Complete")
driver.quit()

不确定我是否可以在这里附上示例html文件以供参考。不管怎样，问题出在哪里？我如何加载与通过浏览器加载硒完全相同的页面？

Answer 1

这可能是由于使用了PhantomJS，因此不再对其进行维护，并从Selenium 3.8.1中弃用。请改用Chrome无头。

options = Options()
options.headless = True
driver = webdriver.Chrome(CHROMEDRIVER_PATH, chrome_options=options)