Skip to content

Instantly share code, notes, and snippets.

@mattetti
Last active August 21, 2024 04:52
Show Gist options
  • Save mattetti/5914158 to your computer and use it in GitHub Desktop.
Save mattetti/5914158 to your computer and use it in GitHub Desktop.
Example of doing a multipart upload in Go (golang)
package main
import (
"bytes"
"fmt"
"io"
"log"
"mime/multipart"
"net/http"
"os"
"path/filepath"
)
// Creates a new file upload http request with optional extra params
func newfileUploadRequest(uri string, params map[string]string, paramName, path string) (*http.Request, error) {
file, err := os.Open(path)
if err != nil {
return nil, err
}
defer file.Close()
body := &bytes.Buffer{}
writer := multipart.NewWriter(body)
part, err := writer.CreateFormFile(paramName, filepath.Base(path))
if err != nil {
return nil, err
}
_, err = io.Copy(part, file)
for key, val := range params {
_ = writer.WriteField(key, val)
}
err = writer.Close()
if err != nil {
return nil, err
}
req, err := http.NewRequest("POST", uri, body)
req.Header.Set("Content-Type", writer.FormDataContentType())
return req, err
}
func main() {
path, _ := os.Getwd()
path += "/test.pdf"
extraParams := map[string]string{
"title": "My Document",
"author": "Matt Aimonetti",
"description": "A document with all the Go programming language secrets",
}
request, err := newfileUploadRequest("https://google.com/upload", extraParams, "file", "/tmp/doc.pdf")
if err != nil {
log.Fatal(err)
}
client := &http.Client{}
resp, err := client.Do(request)
if err != nil {
log.Fatal(err)
} else {
body := &bytes.Buffer{}
_, err := body.ReadFrom(resp.Body)
if err != nil {
log.Fatal(err)
}
resp.Body.Close()
fmt.Println(resp.StatusCode)
fmt.Println(resp.Header)
fmt.Println(body)
}
}
@nullne
Copy link

nullne commented Feb 4, 2017

@bacongobbler. thanks

@sclevine
Copy link

@bacongobbler @nullne @mattetti If you're interested in a streaming solution that doesn't require reading the entire file into memory at all (ex. for large files, or for production use in gateways), see here:
https://github.com/sclevine/cflocal/blob/49495238fad2959061bef7a23c6b28da8734f838/remote/droplet.go#L21-L58

(Worth noting: S3 doesn't accept chunked multi-part uploads, so you have to calculate the length ahead of time if that's where you're sending the file.)

@vicbaily528
Copy link

I found a problem:
When I simulated uploading a file in golang to the spring restful API, I found that the uploaded file was incorrect when I uploaded it using your method. Although it can get the data.
@PostMapping("/demo") public String postdemo(@RequestParam("userName") String userName, @RequestBody byte[] file) throws IOException { System.out.println(userName); if (file != null) { System.out.println(file.length); String jsonFilePath = "/local/code/12312312.zip"; File jsonFile = new File(jsonFilePath); FileOutputStream outputStream = FileUtils.openOutputStream(jsonFile); outputStream.write(file); outputStream.flush(); outputStream.close(); } return "you send post parame is " + userName; }

The header of the generated file will contain the following information

--45e6a34acb7e1b27165a75bf5052f2f4f00fc96dd04899d1bf66bd782fa1 Content-Disposition: form-data; name="file"; filename="pom.zip" Content-Type: application/octet-stream ����

If I use this code "request.Header.Add("Content-Type", writer.FormDataContentType())", then he will prompt me to get the parameters: username

@dezza
Copy link

dezza commented Dec 21, 2017

and then just test it:

	rr := httptest.NewRecorder()
	handler := http.HandlerFunc(API.Create)
	handler.ServeHTTP(rr, req)

	if status := rr.Code; status != http.StatusOK {
		t.Errorf("handler returned wrong status code: got %v want %v",
			status, http.StatusOK)
	}

@eoinahern
Copy link

any examples of posting a file along with a serialized struct?

@lusi1990
Copy link

@mkaz save my day

@naughtymonkey1010
Copy link

@mkaz thanks

@icefed
Copy link

icefed commented Mar 26, 2018

using io.pipe will be better?

@shuaihanhungry
Copy link

@e7
Copy link

e7 commented Jul 12, 2018

@themihai I'm wondering it too, do you have solution?

@tcr-ableton
Copy link

My two cents: using file.Stat is an triggers an unnecessary system call. You could use filepath.Base(path) in order to obtain the filename.

@ramadani
Copy link

great solution @bacongobbler, big thanks

@melissafzhang
Copy link

I'm having similar issue as @vicbaily528 where the content type is an octet stream. Any ideas how to get around this?

@jk2K
Copy link

jk2K commented Feb 25, 2019

var bodyContent []byte
resp.Body.Read(bodyContent)
resp.Body.Close()
fmt.Println(bodyContent)

not work,

body, _ := ioutil.ReadAll(res.Body)
fmt.Println(res)
fmt.Println(string(body))

work well

@yzhanginwa
Copy link

Didn't work at first. Just used wireshark and found there is no content-type in the request.

@omerkaya1
Copy link

omerkaya1 commented Apr 17, 2019

Thanks for this example.
It helped a lot.
However, if I may, I'd like to contribute to this discussion in the hope that it'll be of use to others and save them time.

As was already pointed out by @ghost when it comes to large files, it's inconvenient to use a buffer, as it consumes a lot of resources.

Luckily, there's a solution. After googling for a while I encountered this article where the idea of in-memory piping is leveraged for multipart uploading.
The advantages of this approach are tremendous in my point of view.
Another example can be found here.
I hope it'll be helpful.
Cheers.

@030
Copy link

030 commented May 31, 2019

@jacksonwbrito
Copy link

Thanks a lot!

I had the same problem as @vicbaily528 and @melissafzhang, after take a look at mime/multipart/writer.go I decide to replace the CreateFormFile function for a custom one:

func MyCreateFormFile(fieldname, filename, contentType string) (io.Writer, error) {
        h := make(textproto.MIMEHeader)
	h.Set("Content-Disposition", fmt.Sprintf(`form-data; name="%s"; filename="%s"`, escapeQuotes(fieldname), escapeQuotes(filename)))
	h.Set("Content-Type", escapeQuotes(contentType))
	return w.CreatePart(h)
}

this worked for me, I hope it can help you or others who have the same problem.

@prayuditb
Copy link

@mattetti, thanks for helpful gist and as @mkaz mentioned, adding Content-Type header works for me

@sebnyberg
Copy link

sebnyberg commented Aug 18, 2020

Yet another example:

func uploadFileMultipart(url string, path string) (*http.Response, error) {
	f, err := os.OpenFile(path, os.O_RDONLY, 0644)
	if err != nil {
		return nil, err
	}

	// Reduce number of syscalls when reading from disk.
	bufferedFileReader := bufio.NewReader(f)
	defer f.Close()

	// Create a pipe for writing from the file and reading to
	// the request concurrently.
	bodyReader, bodyWriter := io.Pipe()
	formWriter := multipart.NewWriter(bodyWriter)

	// Store the first write error in writeErr.
	var (
		writeErr error
		errOnce  sync.Once
	)
	setErr := func(err error) {
		if err != nil {
			errOnce.Do(func() { writeErr = err })
		}
	}
	go func() {
		partWriter, err := formWriter.CreateFormFile("file", path)
		setErr(err)
		_, err = io.Copy(partWriter, bufferedFileReader)
		setErr(err)
		setErr(formWriter.Close())
		setErr(bodyWriter.Close())
	}()

	req, err := http.NewRequest(http.MethodPut, url, bodyReader)
	if err != nil {
		return nil, err
	}
	req.Header.Add("Content-Type", formWriter.FormDataContentType())

	// This operation will block until both the formWriter
	// and bodyWriter have been closed by the goroutine,
	// or in the event of a HTTP error.
	resp, err := http.DefaultClient.Do(req)

	if writeErr != nil {
		return nil, writeErr
	}

	return resp, err
}

@faddat
Copy link

faddat commented Sep 27, 2020

thank you.

@donnol
Copy link

donnol commented Oct 15, 2020

Can I upload two or more files one times?

@mirisbowring
Copy link

mirisbowring commented Nov 1, 2020

@donnol - jep, just call the "createFormFile" function multiple times

@toannd96
Copy link

toannd96 commented Mar 5, 2021

@vikfrank
Copy link

Thanks a lot.. This is really useful.

@pablodz
Copy link

pablodz commented Aug 12, 2022

Really useful, it works

@pforpramit
Copy link

@sebnyberg Hello, just wanted to understand about the comment "This operation will block until both the formWriter...". Could you please explain, thank you!

@sebnyberg
Copy link

sebnyberg commented Dec 22, 2022

@sebnyberg Hello, just wanted to understand about the comment "This operation will block until both the formWriter...". Could you please explain, thank you!

Sure. The request reads from the provided reader until it returns io.EOF. For an io.Pipe, the reader-end will return io.EOF after the write-end is closed. That's what is meant by "blocking". If the bodyWriter is not closed, the the request will last forever (until conn timeout).

https://pkg.go.dev/io#PipeWriter.Close

Closing the formWriter isn't strictly necessary to send the request. However, the formWriters Close() writes a trailer to the multipart message that is required for the request to be valid.

https://pkg.go.dev/mime/multipart#Writer.Close

@pforpramit
Copy link

@sebnyberg Thank you very much! :)

@nirajchandak
Copy link

If uploading specific format like gzip or server expect content other than octet-stream, better to use writer.CreatePart instead of formWriter.CreateFormFile

	part, err := writer.CreatePart(textproto.MIMEHeader{
		"Content-Type": []string{"application/x-gzip"},
		"Content-Disposition": []string{fmt.Sprintf(`form-data; name="%s"; filename="%s"`,
			"file", fi.Name())},
	})

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment