警告查找R carret包中的相关性"Combination row and column is above the cut-off"，并且不获取和返回值

Question

我目前正在尝试根据它们的相关性过滤变量数据在我的Mac上的RStudio中使用R中的carret包。

到目前为止，我可以计算并打印数据集的相关性。但是，一旦应用findCorrelation方法，就不会返回任何数据。我只得到以下警告:

"组合行和列高于分界点，值=标记列"

library(caret)
preProcessAttributeClass <- function (data.convert) {
classe <- data.convert$classe
data.convert <- as.data.frame(sapply(data.convert,as.numeric))
data.convert$X.1 <- NULL
data.convert$X <- NULL
data.convert$user_name <- NULL
data.convert$raw_timestamp_part_1 <- NULL
data.convert$raw_timestamp_part_2 <- NULL
data.convert$cvtd_timestamp <- NULL
data.convert$new_window <- NULL
data.convert$num_window <- NULL
data.convert
}
data.train <- read.csv(file="training.csv",na.strings=c("NA",""))
data.train <- preProcessAttributeClass(data.train)
descrCor <- (cor(na.omit(data.train),use="complete.obs"))
highlyCorDescr <- findCorrelation(na.omit(descrCor), cutoff = .9,    verbose=TRUE,names=FALSE)

你知道我的问题可能是什么原因吗?

Answer 1

我认为问题出在你的关联矩阵上:

> class(na.omit(descrCor))
[1] "matrix"
> dim(na.omit(descrCor))
[1]   0 153

这些数据有很多缺失数据的列:

> pct_na <- unlist(lapply(data.train, function(x) mean(is.na(x))))
> summary(pct_na)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.0000  0.0000  0.9793  0.6401  0.9793  0.9793 

我将押注是否有~95%缺失的列是有用的，但他们阻止你得到一个有用的相关矩阵。我建议用更少的列做相关过滤器:

> sum(pct_na > .1)
[1] 100
> keepers <- data.train[,names(which(pct_na <= .1))]
> descrCor <- cor(keepers ,use="complete.obs")

其余大多数列要么没有相关性，要么相关性非常高:

> summary(descrCor[upper.tri(descrCor)])
     Min.   1st Qu.    Median      Mean   3rd Qu.      Max. 
-0.992000 -0.108800  0.001911  0.001667  0.088680  0.980900 

现在执行过滤器:

> highlyCorDescr <- findCorrelation(descrCor, cutoff = .9,    verbose=TRUE,names=FALSE)
Compare row 10  and column  1 with corr  0.992 
  Means:  0.266 vs 0.164 so flagging column 10 
Compare row 1  and column  9 with corr  0.925 
  Means:  0.247 vs 0.161 so flagging column 1 
Compare row 9  and column  4 with corr  0.928 
  Means:  0.229 vs 0.158 so flagging column 9 
Compare row 8  and column  2 with corr  0.966 
  Means:  0.24 vs 0.154 so flagging column 8 
Compare row 19  and column  18 with corr  0.918 
  Means:  0.089 vs 0.155 so flagging column 18 
Compare row 46  and column  31 with corr  0.914 
  Means:  0.099 vs 0.158 so flagging column 31 
Compare row 46  and column  33 with corr  0.933 
  Means:  0.081 vs 0.161 so flagging column 33 
All correlations <= 0.9 
> keep_these <- names(data.train)[!(names(data.train) %in% colnames(descrCor)[highlyCorDescr])]
> data.train.subset <- data.train[, keep_these]