Does anyone know how I can use gzip to zip a large log file on the fly. My simulation is currently logging a large file that I need for analysis at a later point. However the files are so huge that I may even run out of disk space. The content is mainly text so when compressed the files are smaller... I would like to gzip on the fly without having to dump it to a file and then create the archive. This is what I am doing right now and would like to not store foo.log... ./sim >foo.log 2>&1 and when its completed i do tar -cf - foo.log | gzip > foo.tar.gz Any suggetions?

Hmmm, it seems that this should be possible. I know that this is done to some extent by Apache/Mozilla for serving pages. http://www.mozilla.org/projects/apache/gzip/ Without being able to test this right now, what you could do is to first create a tar gzipped file, and then stream to the archive using tar. Look at the tar manual pages. I have similar issues with very large simulation output files, so this would be a useful thing to figure out!

I'm not really following what your trying to do. I think what you want is ./sim 2>&1 | tar -czf foo.tar.gz

Actually what I put in my previous post is wrong. Sorry about that.

If all you want is a gzip file of a single log file try ./sim 2>&1 | gzip -c > foo.gz Hope that works better than my first answer!