In this talk we dive into the TeeReader, Pipe, and MultiWriter functions of Go’s io package and discuss when they can be used to solve common problems that nearly every Go developer will encounter.

Intro (rough draft):

I’m sure most of you have heard, but Go has an awesome standard library. The net/http package tends to get the spotlight because it helps illustrate just how comprehensive the standard library is, but I believe this sells the language short. Go’s standard lib isn’t just comprehensive; it is robust and incredibly well thought out, and this shows even when working with more common utility packages.

In this talk we are going to explore one of the packages that, in my opinion, doesn’t get enough love - the io package. Specifically, we are going to take a look at three functions - TeeReader(), Pipe(), and MultiWriter() - and how they all take advantage of the io.Reader and io.Writer interfaces to provide some incredibly powerful functionality out of the box. I decided to talk about these three because they are the ones that I find myself using most frequently, but hopefully this talk will inspire you to take some time to explore the rest of the standard library. There are plenty of gems there ready to be discovered.

Everything else is going to be bullet points with enough info to explain the talk without actually writing everything out. I also didn’t include all code snippets etc to go along with it, but all of this will have slides to go along, often with little code snippets to illustrate the point.

TeeReader

• I will start by discussing a common scenario where TeeReader is useful; eg when you are using http.Post() and want to read the response body while also making sure it gets written somewhere so that we can log it or access the entire body after the fact if we need to.

• One naive solution is to read the entire response body, write it somewhere.

go var buf bytes.Buffer res, err := http.Get("http://awesomesauce.com") if err != nil { // Handle the error } defer res.Body.Close() reader := bufio.NewReader(res.Body) reader.WriteTo(&buf) // Now we can read from buf

• We could then read from buf, as it implements the io.Reader interface, but this approach has a lot of downsides. For starters, we can’t stream the data, so if this was a websocket connection this approach would be rather awkward. What if we instead deferred writing the data until we needed to read it?

• To enable streaming we could do a few things. One option is to expected whoever reads the data to also write it to an alternate source, but that is bound to be bug-ridden.

• Another option is to read bits of data as it comes in, store it somewhere, and then pass those bits to our functions. This isn’t ideal though because our code that handles initiating requests also has to know about how to read and process that data. We don’t want to mix those two responsibilities.

• The ideal solution would be a wrapper around our reader that reads however much is requested, writes it to an alternate source, and then returns the newly read data. Enter TeeReader()!

• TeeReader() allows us to solve this problem by wrapping our reader (the http.Response’s Body) in a reader that will also write anything it reads to whatever writer we provide. As a result, we can easily log everything we read to std out, or store it elsewhere by simply providing a writer to the TeeReader() function.
• Example:

go res, err := http.Get("http://awesomesauce.com") if err != nil { // Handle the error! } defer res.Body.Close() body := io.TeeReader(res.Body, os.Stdout) // Use body exactly like you would have used res.Body

• All of our problems are solved, and we barely had to write any code. Sweetness!
• In short - use TeeReader() to create a reader that also writes the data to an alternate source. Just be careful - there isn’t any internal buffering and a write needs to occur before the read is completed, so if your provided writer is blocking it could cause issues.

Pipe

• Pipe() is one of those functions that looks and sounds pretty pointless at first glance despite actually being exceptionally useful. The basic idea is that io.Pipe() allows us to insert a virtual pipe between an io.Reader and its consumer.
• Imagine that your io.Reader was instead a pitcher of water, and it was “read” by pouring the water into the consumer’s glass. io.Pipe() in this metaphor would be like inserting a large glass tube between the two and pouring the pitcher into the tube and then letting the water run through the tube, out the other end, and into the consumer’s glass.

I need some visuals here, but I don’t have them for the proposal

• Sounds pretty useless right?

• Wrong! io.Pipe is incredibly useful when we want to limit memory consumption.

• At this point I will go into explaining an example of when io.Pipe() could be used to help us stream data. The most common example is doing an http.Post() with a multipart form and a large (> 1gb) file, so I’ll probably use that.
• What we really want to do is stream the file data rather than read it all into memory.

• io.Pipe() allows us to create a goroutine that reads from the file and uses a multipart writer to write the data do the “writer” end of the pipe. Then the “reader” end of the pipe (our http.Post) - which is running in another thread - can read the data we wrote and stream it.

• The big improvement here over writing the code yourself is that you don’t have to manage alternating between reading and writing. In fact, you don’t even have to think about it. By having a goroutine that writes, and another that reads (possibly the main thread), Go handles all of the logic necessary to alternate between the routines for you.
• In short, io.Pipe() allows us to avoid a for loop that alternates between reading and writing and and allows us to write our code almost as if we were just reading the entire file and writing it to a multipart writer.

MultiWriter

• TeeReader lets us read data while also handling writing it elsewehre for us
• MultiWriter lets us write data once, and handles writing it to multiple destinations for us
• Possible use cases include sending a message out to many subscribers in a Pub/Sub, writing data to both a log file and wherever else it is being used in code, or even writing data to a multipart writer (like in the io.Pipe example) and writing again to a status bar visualizer which would just count the total bytes written vs how many total are expected to write to calculate a percentage to show.

Many gems like these exist in the standard library

There are many gems like these in the standard library. You should explore the std lib and get a feel for what all is there. They are not only neat, but checking out the source can often teach you a thing or two.

A couple examples:

• the os package also has a file Pipe - https://golang.org/pkg/os/#Pipe
• filepath’s Walk function is one I found really interesting - https://golang.org/pkg/path/filepath/#Walk
• even math/rand’s Perm() function is a really neat tool for accessing slices in a random order.

I expect this talk to take ~15m + 5-10m in questions, but I haven’t ever given it so I need to do a few mock runs to evaluate timing. I can do this upon request - just let me know.

This is a very rough draft of the proposal and I hope to clean it up before the deadline, but I am submitting it now in case I am unable to. I only just came across the CFP on Twitter on Feb 27, and spent the day cleaning up and organizing notes that I had started for this talk in order to get it into the state it is in now.

If you do need any clarifications, or would like me to elaborate on anything or present this CFP in a better structure please let me know.

I don’t have much talking experience at the moment, but I do write a good bit about Go at https://www.calhoun.io/ and I am in the process of finishing up a book that teaches Web Development with Go that has screencasts, so I do have experience talking to an audience from behind a web camera and could provide some of those screencasts to help you evaluate me as a speaker.