Skip to content

Instantly share code, notes, and snippets.

@mchirico
Created August 3, 2013 19:34
Show Gist options
  • Save mchirico/6147687 to your computer and use it in GitHub Desktop.
Save mchirico/6147687 to your computer and use it in GitHub Desktop.
Example of writing a gzip file in go (golang). This appends data to the same file every time it is run. It is important to Flush() the data before closing the file.
/*
This is an example of a golang gzip writer program,
which appends data to a file.
*/
package main
import (
"bufio"
"compress/gzip"
"log"
// "fmt"
"os"
)
type F struct {
f *os.File
gf *gzip.Writer
fw *bufio.Writer
}
func CreateGZ(s string) (f F) {
fi, err := os.OpenFile(s, os.O_WRONLY|os.O_APPEND|os.O_CREATE, 0660)
if err != nil {
log.Printf("Error in Create\n")
panic(err)
}
gf := gzip.NewWriter(fi)
fw := bufio.NewWriter(gf)
f = F{fi, gf, fw}
return
}
func WriteGZ(f F, s string) {
(f.fw).WriteString(s)
}
func CloseGZ(f F) {
f.fw.Flush()
// Close the gzip first.
f.gf.Close()
f.f.Close()
}
func main() {
f := CreateGZ("Append.gz")
WriteGZ(f, "gzipWriter.go created this file.\n")
CloseGZ(f)
}
@stokito
Copy link

stokito commented Sep 16, 2021

066 is rw-rw---- i.e. group can write to the file. It may be better to change to 0644 rw-r--r--
Also, except of embedded devices, CPUs time is not so big limit than IO so it makes sense to use a better compression level:

gf, _ := gzip.NewWriterLevel(f, flate.BestCompression)

Another possible improvement is to increase buffer size. gzip has a window up to 32 Kb but by default bufio has 4Kb.

// Gzip anyway have a 32Kb window so smaller buffer useless
bf:= bufio.NewWriterSize(gf, 32768)

But that also means that you can't make zcat/zgrep untill the buffer is filled. Also when process is killed you must flush and close the file otherwise all the data will be lost. You have to listen for SIGINT and SIGTERM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment