Misuse

I have some utilities to read and write zip files in an application I maintain. I’ve been periodically running into “buffer errors” in the GZipReader library of Zlib. I’ve been working around the errors in various ways but today I finally got stopped cold and had to figure out what was going on. It turns out that the RubyGems people have also seen this error and wrote some code to solve it. So I adapted their code to my purposes and I’ll share what I’ve got in hopes that someone else can use this stuff too.

The code is relatively straightforward. You open a gzip file as a regular File IO object in binary mode. You then use the low level gzip reader to pull the unzipped data into a StringIO object. Then you can stream that StringIO into an output file. Here are some assumptions:

unzipped_filename = string holding path/filename of where you’re saving the unzipped file
zipped_filename = string holding path/filename of where you’re gzipped file is located

    File::open(unzipped_filename, 'wb') do |unzipped_file|
      unzipped_file.binmode
      gz = File::open(zipped_filename, 'rb')
      gz.binmode
      begin
        zis = Zlib::GzipReader.new(gz)
        dis = zis.read
        io = StringIO.new(dis)
      ensure
        zis.finish if zis
      end
      unzipped_file.write(io.read)
      unzipped_file.flush
      unzipped_file.fsync
      gz.close
    end
  GC.start

That’s all there is to it! I call flush and fsync to make sure your file is actually on disk before leaving the routine. Also, GC.start will clean up a lot of ram right away that was used by this function if you’re dealing with extremely large files (I have a unit test for this against a random 100 megabyte ascii file that works now). Performance isn’t bad either!

Note: If you’re using Gnu-Win32 utils (at least) and you gzip a file using maximal compression, don’t expect Ruby’s Zlib to work correctly. I’ve found that using standard compression results in gzip files that Ruby can read more reliably (i.e. without “buffer error” problems).

Addendum 9/22/07: I’m still getting torqued on a few cases where Zlib is returning buffer error for gzipped files on Windows. I have no work around but the solutions above have reduced the # of errors I’m getting. I’m about ready to just shell out to `gzip` and forget about code-driven solutions..

Addendum 5/20/08: I have swapped out my Ruby code and now just shell to do all this:

`gzip -d #{zipped_filename}`

It seems to do the job. Also, zipping files has never caused a problem, it’s unzipping which often fails in Ruby win. Probably b/c in zip mode Ruby is feeding the OS at its pace, but in unzip mode, the OS is throwing bytes at Ruby.