I'm quite confused as there seems to be multiple redundant ways to solve my problem (read a file, parse the content, serve it via http). Most people on stackoverflow would use bufio, but I just can't get the differences between this package and the Buffer type of bytes and just reading a file with the os methods. Also I don't know when and why I should choose those ways to do it, when I have the simple, but non-versatile, ioutils.ReadFile.
This is how I solved it currently:
func loadPage(path string, f os.FileInfo, err error) error {
if err != nil {
panic(err)
}
if !!f.IsDir() {
return nil
}
matched, err := filepath.Match("[0-9]-*.md", f.Name())
if err != nil {
panic(err)
}
if matched {
titleStart := strings.Index(f.Name(), "-") + 1
titleEnd := strings.LastIndex(f.Name(), ".md")
title := strings.ToLower(f.Name()[titleStart:titleEnd])
content, err := ioutil.ReadFile(path)
if err != nil {
panic(err)
}
pages[title] = bytes.NewBuffer(blackfriday.MarkdownCommon(content))
}
return nil
}
func loadPages() {
pages = make(map[string]*bytes.Buffer)
err := filepath.Walk(PAGEDIR, loadPage)
if err != nil {
panic(err)
}
}
RickyS at Feb 25, 2014 at 4:07 am ⇧ 
- os is for opening and closing files.
- bufio is for reading, writing and parsing, usually text files.
Ondekoza at Feb 25, 2014 at 12:04 pm ⇧ 
The only problem that might occur with ReadFile is that it might fail on extremely large files. But since your files are guaranteed to be markdown files which are probably written by a human writer, this is not a problem. Your solution seems to work for you.
I would use a regular expression to extract the title from the filename. https://github.com/StefanSchroeder/Golang-Regex-Tutorial
Carlos Castillo at Feb 25, 2014 at 12:58 pm ⇧ First of all, if you are presenting code, you should use playground links.
- You shouldn't be using panic in load, filepath.Walk will return the error that load returns if the error is not nil, and not filepath.SkipDir: http://play.golang.org/p/2bu5gWluri
- loadPages can panic, or log.Fatal, return the error to code that handles the error, or deal with the error itself (eg: try another path)
- You can store just the processed byte slice in the map, not a
*bytes.Buffer: http://play.golang.org/p/MjSi3dBHDr
- If you need an io.Reader type (which your example doesn't), you can create a bytes.Reader on the []byte instead, which is much more lightweight, and safer to use
- In the handler, with a byte-slice you can use
http.ResponseWriter.Write directly: http://play.golang.org/p/dKM6MZKdas
- Although the only thing that can fail is the write to the client (and so you can't give them an error message), you probably should log the error from the ResponseWriter
- Alternatively to using filepath.Walk, with filepath.Match to check
files, you could use filepath.Glob, to get the list of matching files and
then make your own for-loop.
- Upsides
- You don't need to write callback code
- Probably can fit it all in one function
- Downsides:
- No recursive traversal into subdirectories
- May not be an issue
- You don't get a os.FileInfo automatically (to test for dirs,
etc...)
- If the user has a directory that matches your pattern, it might make more sense to return the error from ioutil.ReadFile than to check and then skip it
- No recursive traversal into subdirectories
- Upsides
In your code, you are reading the contents of a directory once at startup, and storing the processed contents in memory. This means that you have to restart the program if you want to serve updated or new files, and that the memory for these files is always in use. Neither of these problems is critical, and I have no idea of you specific situation, so you don't need to change your program, but just be mindful of the limitations of the code you have written. Also your code doesn't distinguish the files "1-foo.md" from "2-foo.md" in the map, so the later will overwrite the former. This may have been your intent though...
Finally to answer your initial question, the packages you mention in the subject line work as follows w.r.t. your needs:
- io defines interfaces that handle streams of bytes (Reader, Writer, etc...) as well as functions that work generically with types implement these interfaces (eg: io.Copy)
- os defines types and functions that represent operating system
objects/functionality at a low level that is portable across all go
implementations.
- *os.File is a type that implements io.Reader, and io.Writer (among
others) which streams bytes to or from a file on disk
- It is useful if you don't want to read the whole file into memory, or are using io.Copy (or some other method) to stream data to/from the file
- It has the downside of being a lower level construct, meaning data must often be processed in loops (with error checks on each iteration), and that it must be manually managed (via Close())
- io/ioutil provides helper functions for some non-trivial file and io tasks
- ReadFile reads an entire file into memory (as a []byte) in a single
call
- It automatically allocates a byte slice of the correct size (no need to Read + append in a loop)
- It automatically closes the file
- It returns the first error that prevented it from working (so you only need a single error check)
- bufio provides wrapper types for io.Reader and io.Writer that buffer the input / output to improve efficiency
- The net/http package already buffers data for you (using bufio itself) so you don't need this package for that
- If you are reading a file in one or a few large steps, you probably don't need it either
- buffered input and output add some extra concerns
- bufio.Scanner is a nice utility type to efficiently read independent lines of text from an io.Reader
- *os.File is a type that implements io.Reader, and io.Writer (among
others) which streams bytes to or from a file on disk
- bytes provides helper functions and types for interacting with byte
slices ([]byte)
- bytes.Reader turns a []byte into a io.Reader (as well as an io.Seeker to rewind)
- bytes.Buffer uses []bytes to implement a reader/writer, it is useful when you want to use code that takes an io.Writer, and store the results in memory for use later
- the strings package provides analogous behaviour for go strings