使用Deflatestream时避免复制压缩数据

Question

假设我们给了API函数F（流S）将流中包含的二进制数据放入数据库中。我想使用F将文件放入数据库中，但我想提前压缩数据。因此，我认为我可以做以下操作：

var fileStream= File.OpenRead(path);
using(var dstream = new DeflateStream(fileStream, CompressionLevel.Optimal))
   f(dstream);

但似乎DeflateStream仅写入流fileStream中，但在压缩时不会从中读取。在我发现的所有示例中，流的CopyTo方法用于压缩或解压缩。但这意味着我必须将压缩数据的副本保存在内存中，然后再将其传递给f：

：

var memoryStream = new MemoryStream();
using(var fileStream= File.OpenRead(path)) 
  using(var dstream = new DeflateStream(memoryStream, CompressionLevel.Optimal)) {
    fileStream.CopyTo(dstream);
    memoryStream.Seek(0, SeekOrigin.Begin);
    f(memoryStream);
  }    

有什么方法可以避免使用Memorystream？

更新为了某些评论员的持久性，我添加了一个完整的示例：

using System;
using System.IO;
using System.IO.Compression;
public class ThisWouldBeTheDatabaseClient {
  public void f(Stream s) {
    // some implementation I don't have access to
    // The only thing I know is that it reads data from the stream in some way.
    var buffer = new byte[10];
    s.Read(buffer,0,10);
  }
}
public class Program {
  public static void Main() {
    var dummyDatabaseClient = new ThisWouldBeTheDatabaseClient();
    var dataBuffer = new byte[1000];
    var fileStream= new MemoryStream( dataBuffer ); // would be "File.OpenRead(path)" in real case
    using(var dstream = new DeflateStream(fileStream, CompressionLevel.Optimal))
        dummyDatabaseClient.f(dstream);
  }
}

f的虚拟实现中的读取操作抛出一个例外： invalidoperationException：不支持从压缩流中读取。在评论中结束讨论，我认为无法进行所需的行为使用DeflateStream，但第三方库中有其他选择。

Answer 1

DeflateStream只是一个包装器，需要压缩数据的流。因此，您必须使用两个流。

有什么方法可以避免使用Memorystream？

是。

您需要流来存储临时数据而不会消耗（太多）内存。而是使用MemoryStream您可以使用临时文件为此。

对于懒惰的人（就像我一样），让我们创建一个类似于MemoryStream

public class TempFileStream : FileStream
{
    public TempFileStream() : base(
        path: Path.Combine(Path.GetTempPath(), Path.GetRandomFileName()),
        mode: FileMode.OpenOrCreate,
        access: FileAccess.ReadWrite,
        share: FileShare.None,
        bufferSize: 4096,
        options: FileOptions.DeleteOnClose | FileOptions.Asynchronous | FileOptions.Encrypted | FileOptions.RandomAccess)
    {
    }
}

这里重要的部分是FileOptions.DeleteOnClose，当您处置流时将删除临时文件。

然后使用它

using (var compressedStream = new TempFileStream())
{
    using (var deflateStream = new DeflateStream(
        stream: compressedStream,
        compressionLevel: CompressionLevel.Optimal,
        leaveOpen: true))
    using (var fileStream = File.OpenRead(path))
    {
        fileStream.CopyTo(deflateStream);
    }
    f(compressedStream);
}

Answer 2

您可以使用SharpCompress。它的DeflateStream允许您即时阅读压缩数据，这正是您想要的。

这是基于Rufo爵士的完整示例：

using System;
using System.IO;
using SharpCompress.Compressors;
using SharpCompress.Compressors.Deflate;
using System.Linq;
public class Program
{
    public static void Main()
    {
        var dataBuffer = Enumerable.Range(1, 50000).Select(e => (byte)(e % 256)).ToArray();
        using (var dataStream = new MemoryStream(dataBuffer))
        {
            // Note: this refers to SharpCompress.Compressors.Deflate.DeflateStream                
            using (var deflateStream = new DeflateStream(dataStream, CompressionMode.Compress))
            {
                ConsumeStream(deflateStream);
            }
        }
    }
    public static void ConsumeStream(Stream stream)
    {
        // Let's just prove we can reinflate to the original data...
        byte[] data;
        using (var decompressed = new MemoryStream())
        {
            using (var decompressor = new DeflateStream(stream, CompressionMode.Decompress))
            {
                decompressor.CopyTo(decompressed);
            }
            data = decompressed.ToArray();
        }
        Console.WriteLine("Reinflated size: " + data.Length);
        int errors = 0;
        for (int i = 0; i < data.Length; i++)
        {
            if (data[i] != (i + 1) % 256)
            {
                errors++;
            }
        }
        Console.WriteLine("Total errors: " + errors);
    }
}

或使用您的示例代码：

using System;
using System.IO;
using SharpCompress.Compressors;
using SharpCompress.Compressors.Deflate;
public class ThisWouldBeTheDatabaseClient {
  public void f(Stream s) {
    // some implementation I don't have access to
    // The only thing I know is that it reads data from the stream in some way.
    var buffer = new byte[10];
    s.Read(buffer,0,10);
  }
}
public class Program {
  public static void Main() {
    var dummyDatabaseClient = new ThisWouldBeTheDatabaseClient();
    var dataBuffer = new byte[1000];
    var fileStream= new MemoryStream( dataBuffer ); // would be "File.OpenRead(path)" in real case
    using(var dstream = new DeflateStream(
        fileStream, CompressionMode.Compress, CompressionLevel.BestCompression))
        dummyDatabaseClient.f(dstream);
  }
}

现在，这不会引发异常，并且将提供压缩数据。