在R中改进了从谷歌获取股票新闻数据的功能

Question

我已经编写了一个函数，用于从Google获取和解析给定股票代码的新闻数据，但我确信还有可以改进的方法。首先，我的函数返回一个GMT时区的对象，而不是用户当前时区的对象，如果传递的数字大于299，它就会失败(可能是因为google只返回每支股票300个故事)。这在某种程度上是对我自己关于堆栈溢出的问题的回应，并且很大程度上依赖于这篇博客文章。

tl;dr:如何改进这个函数?

 getNews <- function(symbol, number){
    # Warn about length
    if (number>300) {
        warning("May only get 300 stories from google")
    }
    # load libraries
    require(XML); require(plyr); require(stringr); require(lubridate);
    require(xts); require(RDSTK)
    # construct url to news feed rss and encode it correctly
    url.b1 = 'http://www.google.com/finance/company_news?q='
    url    = paste(url.b1, symbol, '&output=rss', "&start=", 1,
               "&num=", number, sep = '')
    url    = URLencode(url)
    # parse xml tree, get item nodes, extract data and return data frame
    doc   = xmlTreeParse(url, useInternalNodes = TRUE)
    nodes = getNodeSet(doc, "//item")
    mydf  = ldply(nodes, as.data.frame(xmlToList))
    # clean up names of data frame
    names(mydf) = str_replace_all(names(mydf), "value\.", "")
    # convert pubDate to date-time object and convert time zone
    pubDate = strptime(mydf$pubDate, 
                     format = '%a, %d %b %Y %H:%M:%S', tz = 'GMT')
    pubDate = with_tz(pubDate, tz = 'America/New_york')
    mydf$pubDate = NULL
    #Parse the description field
    mydf$description <- as.character(mydf$description)
    parseDescription <- function(x) {
        out <- html2text(x)$text
        out <- strsplit(out,'n|--')[[1]]
        #Find Lead
        TextLength <- sapply(out,nchar)
        Lead <- out[TextLength==max(TextLength)]
        #Find Site
        Site <- out[3]
        #Return cleaned fields
        out <- c(Site,Lead)
        names(out) <- c('Site','Lead')
        out
    }
    description <- lapply(mydf$description,parseDescription)
    description <- do.call(rbind,description)
    mydf <- cbind(mydf,description)
    #Format as XTS object
    mydf = xts(mydf,order.by=pubDate)
    # drop Extra attributes that we don't use yet
    mydf$guid.text = mydf$guid..attrs = mydf$description = mydf$link = NULL
    return(mydf) 
}

Answer 1

这是getNews函数的一个更短(可能更有效)的版本

  getNews2 <- function(symbol, number){
    # load libraries
    require(XML); require(plyr); require(stringr); require(lubridate);  
    # construct url to news feed rss and encode it correctly
    url.b1 = 'http://www.google.com/finance/company_news?q='
    url    = paste(url.b1, symbol, '&output=rss', "&start=", 1,
               "&num=", number, sep = '')
    url    = URLencode(url)
    # parse xml tree, get item nodes, extract data and return data frame
    doc   = xmlTreeParse(url, useInternalNodes = T);
    nodes = getNodeSet(doc, "//item");
    mydf  = ldply(nodes, as.data.frame(xmlToList))
    # clean up names of data frame
    names(mydf) = str_replace_all(names(mydf), "value\.", "")
    # convert pubDate to date-time object and convert time zone
    mydf$pubDate = strptime(mydf$pubDate, 
                     format = '%a, %d %b %Y %H:%M:%S', tz = 'GMT')
    mydf$pubDate = with_tz(mydf$pubDate, tz = 'America/New_york')
    # drop guid.text and guid..attrs
    mydf$guid.text = mydf$guid..attrs = NULL
    return(mydf)    
}

此外，您的代码中可能存在错误，因为我试图将其用于symbol = 'WMT'，它返回了一个错误。我认为getNews2也适用于WMT。看看它，让我知道它是否适合你。

p。description列仍然包含html代码。但是从中提取文本应该很容易。当我找到时间时，我会发布更新