Support for Extracting/Compressing to Streams

Mar 15, 2009 at 6:32 AM
Edited Mar 15, 2009 at 6:34 AM
I would like to propose a new feature a new feature to the SevenZipSharp Managed Library. Adding support for extracting files to streams (i.e. MemoryStream) just like the DotNetZip Library at dotnetzip.codeplex.com.

More Power!

O.T.
I would like to thank you for developing this library! In the past, I have to use separate libraries for Extracting Zip and Rar archives in my program.
Coordinator
Mar 15, 2009 at 9:29 AM
Thanks for your hint, I will implement streaming in the 0.29 release today.
Coordinator
Mar 15, 2009 at 5:08 PM
Implemented. Ta-da :)
Mar 16, 2009 at 4:25 PM
I'm not really sure how your code is implemented, and if it is possible at all, but could you make a method that gets a stream as an input, and returns the compressed/decompressed stream an an output.

The reason for this is I want to have the ability to "stream-compress" or "stream-decompress" a file, without loading the whole file into memory. Disregarding max file sizes, imagine trying to compress a 4GB file. If this was implemented as a stream, the program would only read a part of the file at a time, compress it on the fly and write/send it to the destination, then read the next chunk..

Is that possible? :)

P.S. I've seen you used Seek(0, 0) in a method that used streams. What if the incoming stream is a WebResponseStream? You can't seek on them :P

Thanks in advance
Coordinator
Mar 17, 2009 at 1:21 AM
Never worked with such streams as a WebResponseStream, but it seems nothing will work without seeking. I will see what can be done though... Double-sided streaming is sure possible to implement, will make it in the next release.
Mar 17, 2009 at 2:38 AM
Make an assumption that when you receive a stream as an input, it's pointer is in the beginning of the stream (which should really be true in most cases). In this case you can remove Seek(0, 0) with no consequences.

Also try to use unly simple reads from the stream that don't specify a position - just read. This way it will work with streams where you can't control the pointer position - like web streams.

If you could test it too, that would be awesome! Find an http link to a zip file, then create a web request to this file, and get the response stream. Then feed that stream into the SevenZipExtractor, and try to read the archive to the end. If this will work, and will not throw any exceptions - I guess then it will work with close to any stream :)
Coordinator
Mar 17, 2009 at 3:22 PM
Edited Mar 17, 2009 at 4:21 PM
Thanks for your suggestions.

Alas for us! Any Archive,GetProperty() or Extract() COM call makes ISequentialInStream.Seek() calls to the input archive stream :( So the thing you are talking about seems impossible. I coded CompressStream(Stream inStream, Stream outStream, int inLength) and DecompressStream(...) functions, they do not call Seek I guess and use LZMA SDK. I am going to implement ExtractFile(Stream stream) routine, which will allow to extract files from a stream to a stream, no WebResponseStream support though.
The only way seems to convert WebResponseStream to a MemoryStream.

Update: I have an idea. I will study where Seek() positions to, but I feel all Seek()-s are called to the beginning of the stream. So, if I recreate WebResponseStream each time Seek() is called and Read data to the desired position, this will do the trick with little traffic penalty. Please, write an example of how do you create a WebResponseStream, so that I will be able to implement SevenZipExtractor.ExtractWebResponse(<the object which makes a WebResponseStream>) function.
Coordinator
Mar 17, 2009 at 10:53 PM
I will think about it in the planned 0.31 release.
Mar 18, 2009 at 7:25 AM
Hi,
 I want to implement 7-zip functionality in asp.net webapplications( code behind language is c#.net).
Is it possible to implement 7-zip functionality in webapplications?
Please suggest me the way of how to implement it?
Please give me the code if possible.

Thanks & Regards,
Sai Akshaya
Coordinator
Mar 18, 2009 at 9:23 AM
What do you mean? WebResponseStream? You can compress and extract everything on the server side with the existing code.
Mar 18, 2009 at 10:15 AM
How can we achieve "extracting the files" at server side(using asp.net)? Please give me link where the code is available.
Mar 18, 2009 at 10:21 AM
Maybe I am a bit rude (sorry for that) but if you're a developer, you have to code, not copy the code that people give you.
Secondly, you have to learn to READ and do some research! The code examples are available in BOTH the source and the discussions here!

Finally, if you have a question regarding this project, create a discussion thread with a relevant name, don't reuse a random discussion -.-
Sheesh!

P.S. Regarding your situation, in ASP.NET you will have to specify the full path to 7z.dll file, as by default the assembly location will be pointing to a location deep inside the Windows folder :P
The rest will generally work as in regular apps.
Mar 18, 2009 at 11:07 AM
Edited Mar 18, 2009 at 11:09 AM
This is an additional request, since my last post was a bit vague.

Will it be possible to add stream support (in function SevenZipExtractor.ExtractFile) forextracting individual files from archives just like the one below?

Public Sub ExtractFile(FileName as String, OutStream as Stream)
Public Sub ExtractFile(index as UInteger, OutStream as Stream)
Mar 18, 2009 at 12:24 PM
Hi markhor,
I missed the update to your post, so I didn't see your idea about WebRequests :(

To get a stream from a webrequest you can use this (the most simple and basic example :P ):
using System.Net;
[...]
WebRequest.Create("url_goes_here.zip").GetResponse().GetResponseStream()
This method will return a Stream object, but it's Length and Seek() methods will throw exceptions :)

I read your update again, and came up with the following idea:
Create a class that derives from a Stream and override it's Seek() and Length methods (I guess Length isn't used anywhere.. or almost anywhere crucial). I suppose that was your idea too.
I'm pretty sure that 7z.dll does Seek()s though, and not only to position 0.

As tuldok89 proposed earlier, in a method with stream input and stream output this you would only extract 1 file (otherwise you can't possible return a stream as a result really.. ). I would like to ask for a method that extracts a file from an archive, based on the file name or the index, and returns that file's uncompressed contents as a stream. Now that is a useful method!!!

So the method definitions would be (based on tuldok89's example, but in C# + supports multiple archive types
public Stream ExtractFile(String fileName, Stream archiveStream, InArchiveFormat format);
public Stream ExtractFile(uint index, Stream archiveStream, InArchiveFormat format);

The whole WebRequest thing with Seek() overrides doesn't have to be inside this library really, as it's outside of it's scope, I'd say, so I don't think there's any point in trying to do it (unless you're a nerd like me and want to do it, just for the fun). 

Thanks for your help :)
Coordinator
Mar 18, 2009 at 12:28 PM
tuldok89: yes, it will, in the 0.31 release.
Coordinator
Mar 18, 2009 at 12:55 PM
Hi gregshutdown, I was going to blame you for your previous post :)

My solution of the issue is adding the constructor SevenZipExtractor(WebRequest request, ...) and overriding InStream.Seek() method with some code like

private WebRequest _WebRequest;

public virtual void Seek(long offset, uint seekOrigin, IntPtr newPosition)
{
            if (offset < 0)
            {
                   long oldPosition = BaseStream.Position;
                   BaseStream.Close();
                   BaseStream = _WebRequest.GetResponse().GetResponseStream();
                   BaseStream.Read(temporary byte[] array, 0, oldPosition + offset);                  

            }
            else
            {
                   BaseStream.Read(temporary byte[] array, 0, offset); 
            }
            long Position = BaseStream.Position;
            if (newPosition != IntPtr.Zero)
                Marshal.WriteInt64(newPosition, Position);
}
Coordinator
Mar 18, 2009 at 12:58 PM
gregshutdown, deleted your last post because we had mutual misunderstanding.
Coordinator
Mar 18, 2009 at 7:05 PM
public Stream ExtractFile(uint index, Stream archiveStream) and everything was implemented in the 0.31 release. InArchiveFormat is always set in the SevenZipExtractor constructor.
Mar 20, 2009 at 2:13 PM
Thanks for implementing streams support! That saved me a lot of custom (unreliable?) cleanup code in my app.

More Power!