如何让Zlib在Ruby中从S3流解压缩

Question

Ruby Zlib::GzipReader应该创建一个类似io的对象(必须有一个与IO#read行为相同的read方法)。

我的问题是，我不能得到这个io样的对象从AWS::S3库。据我所知，从它获得流的唯一方法是将块传递给S3Object#stream。

我已经试过了:

Zlib::GzipReader.new(AWS::S3::S3Object.stream('file', 'bucket'))
# Wich gaves me error: undefined method `read' for #<AWS::S3::S3Object::Value:0x000000017cbe78>

有谁知道我该怎么做吗?

Answer 1

一个简单的解决方案是将下载的数据写入StringIO，然后再将其读取出来:

require 'stringio'
io = StringIO.new
io.write AWS::S3::S3Object.value('file', 'bucket')
io.rewind
gz = Zlib::GzipReader.new(io)
data = gz.read
gz.close
# do something with data ...

一种更精细的方法是在流还在下载的时候开始膨胀压缩后的数据，这可以用IO.pipe来实现。像这样:

reader, writer = IO.pipe
fork do
  reader.close
  AWS::S3::S3Object.stream('file', 'bucket') do |chunk|
    writer.write chunk
  end
end
writer.close
gz = Zlib::GzipReader.new(reader)
while line = gz.gets
  # do something with line ...
end
gz.close

你也可以用Thread代替fork:

reader, writer = IO.pipe
thread = Thread.new do
  AWS::S3::S3Object.stream('file', 'bucket') do |chunk|
    writer.write chunk
  end
  writer.close
end
gz = Zlib::GzipReader.new(reader)
while line = gz.gets
  # do something with line
end
gz.close
thread.join