This project is read-only.

Streaming compression

Sep 8, 2011 at 9:20 AM

Hi,

I have serialisation/deserialisation routines of the form:

 

 

public static void Serialize(Stream s, MyObject o) { ... }

public static void Deserialize(Stream s, MyObject o) { ... }

 

 

The basic implementation of Serialize is that it will instantiate a BinaryWriter around s and proceed to write a series of primitives (doubles, ints, floats) to it. Deserialize works in reverse, with a BinaryReader.

 

These methods are working fine but now I would like to compress. I've been able to achieve this with a simple:

 

MyObject o = ...;

var raw = new MemoryStream();

MyObject.Serialize(raw, o);

var compressor = new SevenZipCompressor();

compressor.CompressionLevel = CompressionLevel.Fast;

var compressed = new MemoryStream();

compressor.CompressStream(raw, compressed);

 

This has been fine, giving me a 5-fold reduction in size with a bearable performance increase. I was wondering if there was a way to further improve that by sidestepping to need to create full raw byte[] copy of of my object and instead writing the primitives directly to a compression stream (like you would with the standard .NET GZipStream).

I've experimented with the LzmaEncodeStream/LzmaDecodeStream but I can't figure out how (or if) I can tweak the compression level. 

 

MyObject o = ...

var compressed = new MemoryStream();

var lzma = new LzmaEncodeStream(compressed);

MyObject.Serializer.Serialise(lzma, o);

 

This works but produces a maybe 2% improvement in size and a 10-fold increase in time. I'd like to tweak this a little to trade back some of the size for faster serialisation.

After some digging I can see that LzmaEncoder uses a SevenZip.Sdk.Compression.Lzma.Encoder object, but I'm quickly overwhelmed by the details here. At the end of the day I'd like to avoid having to learn about the intricacies of compression and just have a tool that "works" (hence why I'm using SevenZipSharp instead of the raw 7zip sdk :).

Is there a way I can do what I want? I've entertained the idea of writing a FifoStream to interpose between the SevenZipCompressor and my own MyObject.Serializer.Serialize routine, but was wondering if I was missing something more obvious.