索引附件文件弹性搜索

Question

我输入了这个命令来索引Elasticsearch中的文档

创建索引

curl -X PUT "localhost:9200/test_idx_1x"

创建映射

curl -X PUT "localhost:9200/test_idx_1x/test_mapping_1x/_mapping" -d '{
  "test_mapping_1x": {
    "properties": {
      "my_attachments": {
        "type": "attachment"
      }
    }
  }
}'

索引此文档

curl -XPUT 'http://localhost:9200/test_idx_1x/test_mapping_1x/4' -d '{
  "post_date": "2009-11-15T14:12:12",
  "message": "test Elastic Search",
  "name": "N1"
}'

这三个命令都很好。但是当我输入这个命令:

curl -XPOST 'http://localhost:9200/test_idx_1x/test_mapping_1x/1' -d '{
  "post_date": "2009-11-15T14:12:12",
  "message": "trying out Elastic Search",
  "name": "N2",
  "my_attachments": {
    "type": "attachment",
    "_content_type": "text/plain",
    "file": "http://localhost:5984/my_test_couch_db_7/ID2/test.txt"
  }
}'

我收到这个错误信息:

{
  "error": "NullPointerException[null]",
  "status": 500
}

我把它改成;

curl -XPOST 'http://localhost:9200/test_idx_1x/test_mapping_1x/1bis' -d '{
  "post_date": "2009-11-15T14:12:12",
  "message": "trying out Elastic Search",
  "name": "N2",
  "my_attachments": {
    "type": "attachment",
    "_content_type": "text/plain",
    "_name": "/inf/bd/my_home_directory/test.txt"
  }
}'
curl -XPUT 'http://localhost:9200/test_idx_1x/test_mapping_1x/1' -d '{
  "post_date": "2009-11-15T14:12:12",
  "message": "trying out Elastic Search",
  "name": "N2",
  "my_attachments": {
    "file": "http://localhost:5984/my_test_couch_db_7/ID2/test.txt"
  }
}'
curl -XPUT 'http://localhost:9200/test_idx_1x/test_mapping_1x/1' -d '{
  "post_date": "2009-11-15T14:12:12",
  "message": "trying out Elastic Search",
  "name": "N2",
  "my_attachments": {
    "file": "http://localhost:5984/my_test_couch_db_7/ID2/test.txt",
    "_content_type": "text/plain"
  }
}'

输出是相同的错误。

我把它改成这样

curl -XPUT 'http://localhost:9200/test_idx_1x/test_mapping_1x/1' -d '{
  "user": "kimchy",
  "post_date": "2009-11-15T14:12:12",
  "message": "trying out Elastic Search",
  "name": "N2",
  "my_attachments": {
    "file": "http://localhost:5984/my_test_couch_db_7/ID2/test.txt",
    "_content_type": "text/plain",
    "content": "... base64 encoded attachment ..."
  }
}'

错误是

{
  "error": "MapperParsingException[Failed to parse]; nested: JsonParseException[Failed to decode VALUE_STRING as base64 (MIME-NO-LINEFEEDS): Illegal character '.' (code 0x2e) in base64 contentn at [Source: [B@159b3; line: 1, column: 241]]; ",
  "status": 400
}
curl -XPUT 'http://localhost:9200/test_idx_1x/test_mapping_1x/1' -d '{
  "post_date": "2009-11-15T14:12:12",
  "message": "trying out Elastic Search",
  "name": "N2",
  "my_attachments": "http://localhost:5984/my_test_couch_db_7/ID2/test.txt"
}'

我收到这个错误信息:

{
  "error": "MapperParsingException[Failed to parse]; nested: JsonParseException[Unexpected character ('h' (code 104)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')n at [Source: [B@1ae9565; line: 1, column: 132]]; ",
  "status": 400
}

如果我输入

curl -XPUT 'http://localhost:9200/test_idx_1x/test_mapping_1x/1' -d '{
  "post_date": "2009-11-15T14:12:12",
  "message": "trying out Elastic Search",
  "name": "N2",
  "my_attachments": "http://localhost:5984/my_test_couch_db_7/ID2/test.txt"
}'

我收到错误。我能理解

{
  "error": "MapperParsingException[Failed to parse]; nested: JsonParseException[Failed to decode VALUE_STRING as base64 (MIME-NO-LINEFEEDS): Illegal character ':' (code 0x3a) in base64 contentn at [Source: [B@1ffb7d4; line: 1, column: 137]]; ",
  "status": 400
}

我如何使用附加文件到ES以便ES可以索引它?

---

谢谢你的回答。当我输入这些命令时，我已经安装了附件插件。文本文件的内容是用Base64编码的，所以我不再对它进行编码。如果我不使用文件的路径，而是直接使用base64的内容，例如

curl -XPUT 'http://localhost:9200/test_idx_1x/test_mapping_1x/' -d '{
  "post_date": "2009-11-15T14:12:12",
  "message": "trying out Elastic Search",
  "name": "N2",
  "my_attachments": "file's content string encoded in base64"
}'

一切都很好，我已经成功地发布了文件，并在稍后搜索了它的内容。

但是如果我用path的文件替换它，我得到了否定的结果。所以我想知道如何在命令行中编码Base64文件，在ES索引的命令中(当然，我不想在键入第二个命令以在ES中索引它之前键入Base64命令来编码文件)。正如您的回答，我是否必须安装"Perl库"之类的东西才能执行您的命令?

Answer 1

http://es-cn.medcl.net/tutorials/2011/07/18/attachment-type-in-action.html

#!/bin/sh
coded=`cat fn6742.pdf | perl -MMIME::Base64 -ne 'print encode_base64($_)'`
json="{"file":"${coded}"}"
echo "$json" > json.file
curl -X POST "localhost:9200/test/attachment/" -d @json.file

Answer 2

首先，不要指定是否安装了attachment插件。如果没有，可以使用:

./bin/plugin -install mapper-attachments

你需要重新启动ElasticSearch来加载插件。

然后，像上面所做的那样，将一个字段映射为具有类型attachment:

curl -XPUT 'http://127.0.0.1:9200/foo/?pretty=1'  -d '
{
   "mappings" : {
      "doc" : {
         "properties" : {
            "file" : {
               "type" : "attachment"
            }
         }
      }
   }
}
'

当您尝试索引文档时，您需要用Base64编码文件的内容。您可以使用base64命令行实用程序在命令行上执行此操作。但是，要成为合法的JSON，还需要对新行进行编码，这可以通过通过Perl将base64的输出管道化来实现:

curl -XPOST 'http://127.0.0.1:9200/foo/doc?pretty=1'  -d '
{
   "file" : '`base64 /path/to/file | perl -pe 's/n/\n/g'`'
}
'

现在你可以搜索你的文件了:

curl -XGET 'http://127.0.0.1:9200/foo/doc/_search?pretty=1'  -d '
{
   "query" : {
      "text" : {
         "file" : "text to look for"
      }
   }
}
'

查看ElasticSearch附件类型。

Answer 3

这是一个完整的shell脚本实现:

file_path='/path/to/file'
file=$(base64 $file_path | perl -pe 's/n/\n/g')
curl -XPUT "http://eshost.com:9200/index/type/" -d '{
    "file" : "content" : "'$file'"
}'

Answer 4

有一个替代的解决方案-插件在http://elasticwarehouse.org。你可以上传二进制文件使用_ewupload?，读取新生成的ID，并使用此引用更新您的不同索引。

安装插件:

plugin -install elasticwarehouseplugin -u http://elasticwarehouse.org/elasticwarehouse/elasticsearch-elasticwarehouseplugin-1.2.2-1.7.0-with-dependencies.zip

重启集群，然后:

curl -XPOST "http://127.0.0.1:9200/_ewupload?folder=/myfolder&filename=mybinaryfile.bin" --data-binary @mybinaryfile.bin

示例响应:

{"id":"nWvrczBcSEywHRBBBwfy2g","version":1,"created":true}