我需要从一堆使用JavaScript渲染的网页中提取文本。
下面的代码通常对我有效，只返回文本和行，这很好。
但是在某些页面它不起作用。
我如何使用RSelenium来提取"URL失败"显示网页?
library("tidyverse")
library("rvest")
library("RSelenium")
remDr <- remoteDriver(port = 4445L)
remDr$open()
# URL Works
url <- "https://www.td.com/ca/en/personal-banking/products/credit-cards/travel-rewards/rewards-visa-card/"
# URL Fails
# url <- "https://www.bmo.com/main/personal/credit-cards/bmo-cashback-mastercard/"
remDr$navigate(url)
pg <-
remDr$getPageSource()[[1]] %>%
read_html(encoding = "UTF-8") %>%
html_node(xpath = "//body") %>%
as.character() %>%
htm2txt::htm2txt()
remDr$close()
由@NadPat提出的解决方案
url <- "https://www.bmo.com/main/personal/credit-cards/bmo-cashback-mastercard/"
remDr$navigate(url)
text <- remDr$findElement(using = 'xpath', value = '/html')
text$getElementText()
Result for me:
Selenium message:a is null
Build info: version: '2.53.1', revision: 'a36b8b1', time: '2016-06-30 17:37:03'
System info: host: 'fe72a1de69e7', ip: '172.17.0.2', os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.0-84-generic', java.version: '1.8.0_91'
Driver info: driver.version: unknown
Error: Summary: UnknownError
Detail: An unknown server-side error occurred while processing the command.
class: org.openqa.selenium.WebDriverException
Further Details: run errorDetails method
对于失败的URL，正在读取某些内容，因为remDr$getPageSource()[[1]]返回:
[1] "<html xmlns="http://www.w3.org/1999/xhtml"><head><script>nnsitePrefix = 'BMO';nvar pageNameMapping = {};nn//channelDemonpageNameMapping["atm_en"]="channelDemo";npageNameMapping["atm_fr"]="channelDemo";nn//Every Day BankingnpageNameMapping["Personal"]="PERS";npageNameMapping["Bank Accounts"]="Bank-Accounts";npageNameMapping["Daily savings account"]="Premium-Rate-Savings";npageNameMapping["High Interest Savings Account"]="Smart-Saver";npageNameMapping["Chequing account"]="Primary-Chequing";npageNameMapping["Business Premium Rate Savings"]="Business Premium Rate Account";nn//CardsnpageNameMapping["Credit Cards"]="CC";nnn//MortgagesnpageNameMapping["Mortgages"]="MTG";npageNameMapping["Special Offers"]="Special-Offers";nn//Wealth ManagementnpageNameMapping["Wealth Management"]="Wealth";npageNameMapping["AdviceDirect"]="Advicedirect";nn//Online InvestingnpageNameMapping["Online Investing"]="ONL-INVS";npageNameMapping...
是否有什么问题，我如何设置RSelenium与Docker?
=======================
更新:我从docker中下载了最新版本的standalone-firefox，现在@NadPat的解决方案为我工作了。
docker pull selenium/standalone-firefox:latest
启动浏览器，
library(RSelenium)
driver = rsDriver(
port = 4841L,
browser = c("firefox"))
remDr <- driver[["client"]]
url <- "https://www.bmo.com/main/personal/credit-cards/bmo-cashback-mastercard/"
第一个方法,
remDr$navigate(url)
text <- remDr$findElement(using = 'xpath', value = '/html')
text$getElementText()
[[1]]
[1] "Skip navigationnPersonalnPrivate WealthnBusinessnCommercialnCapital MarketsnSearchnFind usnSupportnENnLoginnBank AccountsnCredit CardsnMortgagesnLoans & Lines of CreditnInvestmentsnFinancial PlanningnInsurancenWays to BanknAbout BMOnPersonalnCredit CardsnBMO CashBack MastercardnBMO CashBack® Mastercard®*nEnjoy the most cash back on groceries in Canada without paying an annual feenfootnoten*nFootnoten* Based on a comparison of the non-promotional groce
第二种方法,
text <- remDr$findElement(using = 'xpath', value = '//*[@id="main"]')
text$getElementText()
[[1]]
[1] "PersonalnCredit CardsnBMO CashBack MastercardnBMO CashBack® Mastercard®*nEnjoy the most cash back on groceries in Canada without paying an annual feenfootnoten*nFootnoten* Based on a comparison of the non-promotional grocery rewards earn rate on cash back credit cards with no annual fee as of June 1, 2021.nWelcome offernGet up to 5% cash back in your first 3 months‡‡ and a 1.99% introductory interest rate on balance transfers for 9 months with a 1% transfer fee.§§nAPPL