将一台机器中的日志加载到使用 logstash 的另一台机器中的 elasticsearch 设置中

Question

我的日志和日志存储在一台 EC2 机器 (M1( 上运行，所以我使用以下配置读取放置在本地机器上的日志：

input { 
    file{
        path => "/path/to/logs/in/M1"
        start_position => "beginning"
    }
}

现在，我们在另一台EC2机器(M2(上运行了elasticsearch，我需要使用logstash将日志从M1传输到M2中的elasticsearch。我使用了以下输出配置：

output {
    stdout { codec => rubydebug }
    elasticsearch {
            hosts => "http://<M2 ip address>:9200"
        index => "logstash-%{+YYYY.MM.dd}"
    }
}

当我运行配置文件时，出现以下错误：

04:18:57.640 [[main]>worker0] WARN  logstash.outputs.elasticsearch - UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}
04:18:57.646 [[main]>worker0] ERROR logstash.outputs.elasticsearch - Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>2}
04:18:59.682 [[main]>worker0] WARN  logstash.outputs.elasticsearch - UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}
04:18:59.686 [[main]>worker0] ERROR logstash.outputs.elasticsearch - Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>4}
04:19:01.109 [Ruby-0-Thread-17: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:188] WARN  logstash.outputs.elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x1d08c988 URL:http://10.60.40.120:9200>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://10.60.40.120:9200][Manticore::ConnectTimeout] connect timed out"}
04:19:02.111 [Ruby-0-Thread-17: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:188] INFO  logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:url=>#<URI::HTTP:0x55444fcf URL:http://10.60.40.120:9200>, :healthcheck_path=>"/"}

我是洛格斯塔什的新手。任何帮助，不胜感激。

更新：

所以我在论坛中环顾四周，我得到了一个解决方案，它告诉我使用以下命令更新 logstash 输出：

sudo /usr/share/logstash/bin/logstash-plugin update logstash-output-elasticsearch

我还更新了 logstash 配置文件以包含用户名和密码：

output {
    stdout { codec => rubydebug }
    elasticsearch {
            hosts => ["<M2 ip address>"]
        user => 'username'
        password => 'changeme'
        index => "logstash-%{+YYYY.MM.dd}"
        manage_template => false
    }
}

现在我遇到了一个不同的错误。请求帮助：

09:16:21.305 [[main]>worker0] WARN  logstash.outputs.elasticsearch - Could not index event to Elasticsearch. {:status=>404, :action=>["index", {:_id=>nil, :_index=>"logstash-2017.04.17", :_type=>"Messagelog", :_routing=>nil}, 2017-04-17T10:06:11.348Z ip-10-60-40-201 No valid licenses found for COLL], :response=>{"index"=>{"_index"=>"logstash-2017.04.17", "_type"=>"Messagelog", "_id"=>nil, "status"=>404, "error"=>{"type"=>"index_not_found_exception", "reason"=>"no such index and [action.auto_create_index] ([.security,.monitoring*,.watches,.triggered_watches,.watcher-history*]) doesn't match", "index_uuid"=>"_na_", "index"=>"logstash-2017.04.17"}}}}

谢谢。

Answer 1

看起来您已经禁用了在弹性搜索上自动创建索引。默认情况下，elasticsearch 支持自动创建索引。删除

action.auto_create_index: -b*,+a*,-*

(无论模式是什么(在你的elasticsearch.yml中，你会很好。

此外，如果要接受以 l 开头的索引的自动创建，请使用模式 +l*。那就是通过添加

action.auto_create_index: +l*

阅读此内容以获取更多信息。