r-使用rvest迭代抓取多个链接时，查找标记的备用版本

Question

我正在从sec档案中抓取一些数据。每个xml文档都有一个基本形式：

<ns1:infoTable>
<ns1:nameOfIssuer>ACCENTURE PLC IRELAND</ns1:nameOfIssuer>
<ns1:titleOfClass>SHS CLASS A</ns1:titleOfClass>
<ns1:cusip>G1151C101</ns1:cusip>
<ns1:value>47837</ns1:value>
<ns1:shrsOrPrnAmt>
<ns1:sshPrnamt>183135</ns1:sshPrnamt>
<ns1:sshPrnamtType>SH</ns1:sshPrnamtType>
</ns1:shrsOrPrnAmt>
<ns1:investmentDiscretion>SOLE</ns1:investmentDiscretion>
<ns1:votingAuthority>
<ns1:Sole>0</ns1:Sole>
<ns1:Shared>0</ns1:Shared>
<ns1:None>183135</ns1:None>
</ns1:votingAuthority>
</ns1:infoTable>

然而，有时文件的格式是：

<infoTable>
<nameOfIssuer>2U INC</nameOfIssuer>
<titleOfClass>COM</titleOfClass>
<cusip>90214J101</cusip>
<value>340</value>
<shrsOrPrnAmt>
<sshPrnamt>8504</sshPrnamt>
<sshPrnamtType>SH</sshPrnamtType>
</shrsOrPrnAmt>
<investmentDiscretion>SOLE</investmentDiscretion>
<votingAuthority>
<Sole>8504</Sole>
<Shared>0</Shared>
<None>0</None>
</votingAuthority>
</infoTable>

因此，标签中唯一的区别是添加了"；ns1："；前缀

在抓取数据时，我能够找到这样的节点：

urll <- "https://www.sec.gov/Archives/edgar/data/1002152/000108514621000479/infotable.xml"
session %>% 
nod(urll) %>% 
scrape(verbose = FALSE) %>%
xml_ns_strip() %>% 
xml_find_all('ns1:infoTable')

或者对于没有ns1:前缀的备用标签

urll <- "https://www.sec.gov/Archives/edgar/data/1002672/000106299321000915/form13fInfoTable.xml"
session %>% 
nod(urll) %>% 
scrape(verbose = FALSE) %>%
xml_ns_strip() %>% 
xml_find_all('infoTable')

但是，当在多个链接上循环时，我不知道哪个xml文档将具有哪个标记。有没有一种方法可以通过用"0"指定节点来获得节点；或"；运算符或查找具有字符串匹配的标签来查找特定文本"；infoTable"；标签中？

我试过了：

session %>% 
nod(urll) %>% 
scrape(verbose = FALSE) %>%
xml_ns_strip() %>% 
xml_find_all(xpath = '//*[self::infoTable or self::ns1:infoTable]')

或

session %>% 
nod(urll) %>% 
scrape(verbose = FALSE) %>%
xml_ns_strip() %>% 
xml_find_all(xpath = "//*[contains(text(),'infoTable')]")

但这两种变体都不起作用。关于如何让它发挥作用，有什么建议吗？

提前谢谢。我使用礼貌，rvest，dplyr

Answer 1

在XPath表达式中考虑local-name()。下面使用httr和新的R4.1.0+管道|>:

library(xml2)
library(httr)
url <- "https://www.sec.gov/Archives/edgar/data/1002152/000108514621000479/infotable.xml"
info_tables <- httr::GET(url, user_agent("Mozilla/5.0")) |> 
httr::content(encoding="UTF-8") |>
xml2::xml_find_all(xpath = "//*[local-name()='infoTable']")

以及构建数据帧：

df_list <- lapply(info_tables, function(r) {
vals <- xml2::xml_children(r)

other_vals <- xml2::xml_find_all(r, "*") |>
xml2::xml_children()
child_df <- setNames(
c(xml2::xml_text(vals)), 
c(xml2::xml_name(vals))
) |> rbind() |> data.frame()

grand_df <- setNames(
c(xml2::xml_text(other_vals)), 
c(xml2::xml_name(other_vals))
) |> rbind() |> data.frame()

cbind.data.frame(child_df, grand_df)
})
final_df <- do.call(rbind.data.frame, df_list)
final_df
nameOfIssuer titleOfClass     cusip value shrsOrPrnAmt investmentDiscretion votingAuthority sshPrnamt sshPrnamtType Sole Shared   None
1         ACCENTURE PLC IRELAND  SHS CLASS A G1151C101 47837     183135SH                 SOLE        00183135    183135            SH    0      0 183135
2                  ALPHABET INC CAP STK CL A 02079K305 43695      24931SH                 SOLE         0024931     24931            SH    0      0  24931
3                     APPLE INC          COM 037833100  3229      24334SH                 SOLE         0024334     24334            SH    0      0  24334
4    BERKSHIRE HATHAWAY INC DEL         CL A 084670108  2783          8SH                 SOLE             008         8            SH    0      0      8
5           CANADIAN NATL RY CO          COM 136375102   218       1985SH                 SOLE          001985      1985            SH    0      0   1985
6  CHECK POINT SOFTWARE TECH LT          ORD M22465104 45505     342375SH                 SOLE        00342375    342375            SH    0      0 342375
7           CHURCH & DWIGHT INC          COM 171340102 42500     487221SH                 SOLE        00487221    487221            SH    0      0 487221
8  COGNIZANT TECHNOLOGY SOLUTIO         CL A 192446102 46076     562243SH                 SOLE        00562243    562243            SH    0      0 562243
9               CVS HEALTH CORP          COM 126650100 44311     648773SH                 SOLE        00648773    648773            SH    0      0 648773
10          DANAHER CORPORATION          COM 235851102 44200     198974SH                 SOLE        00198974    198974            SH    0      0 198974