Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Example of doing a multipart upload in Go (golang)
package main
import (
"bytes"
"fmt"
"io"
"log"
"mime/multipart"
"net/http"
"os"
"path/filepath"
)
// Creates a new file upload http request with optional extra params
func newfileUploadRequest(uri string, params map[string]string, paramName, path string) (*http.Request, error) {
file, err := os.Open(path)
if err != nil {
return nil, err
}
defer file.Close()
body := &bytes.Buffer{}
writer := multipart.NewWriter(body)
part, err := writer.CreateFormFile(paramName, filepath.Base(path))
if err != nil {
return nil, err
}
_, err = io.Copy(part, file)
for key, val := range params {
_ = writer.WriteField(key, val)
}
err = writer.Close()
if err != nil {
return nil, err
}
req, err := http.NewRequest("POST", uri, body)
req.Header.Set("Content-Type", writer.FormDataContentType())
return req, err
}
func main() {
path, _ := os.Getwd()
path += "/test.pdf"
extraParams := map[string]string{
"title": "My Document",
"author": "Matt Aimonetti",
"description": "A document with all the Go programming language secrets",
}
request, err := newfileUploadRequest("https://google.com/upload", extraParams, "file", "/tmp/doc.pdf")
if err != nil {
log.Fatal(err)
}
client := &http.Client{}
resp, err := client.Do(request)
if err != nil {
log.Fatal(err)
} else {
body := &bytes.Buffer{}
_, err := body.ReadFrom(resp.Body)
if err != nil {
log.Fatal(err)
}
resp.Body.Close()
fmt.Println(resp.StatusCode)
fmt.Println(resp.Header)
fmt.Println(body)
}
}
@themihai

This comment has been minimized.

Copy link

commented May 9, 2014

I'm wondering if it would be possible to upload a 'file' from memory (not from disk) ?

@mkaz

This comment has been minimized.

Copy link

commented May 30, 2014

Thanks for the helpful gist, I think the only example I saw with a file upload and passing parameters. One thing, you should add a Content-Type header to the request.

For example, the return of newfileUploadRequest() :

request := http.NewRequest("POST", uri, body)
request.Header.Add("Content-Type", writer.FormDataContentType())
return request
@wzhliang

This comment has been minimized.

Copy link

commented Sep 25, 2014

@mkaz that addition save a lot of my time!

@quxiao

This comment has been minimized.

Copy link

commented Jan 23, 2015

thanks! @mkaz

@sklyar

This comment has been minimized.

Copy link

commented Feb 20, 2015

Thanks. Do not tell me how to send multiple files at once?

@ystreibel

This comment has been minimized.

Copy link

commented Oct 6, 2015

👍 @mkaz thanks a lot

request, err := http.NewRequest("POST", uri, body)
request.Header.Add("Content-Type", writer.FormDataContentType())
return request, err
@xiconet

This comment has been minimized.

Copy link

commented Oct 24, 2015

Works, but memory usage is still high (at least in my hands), even using io.copy()

@filewalkwithme

This comment has been minimized.

Copy link

commented Nov 24, 2015

@mkaz, thanks, bro!!

@ZetaChow

This comment has been minimized.

Copy link

commented Jan 16, 2016

@mkaz thanks!!

@kreshikhin

This comment has been minimized.

Copy link

commented Feb 12, 2016

@mkaz thanks, you'd saved my day -))

@bigwheel

This comment has been minimized.

Copy link

commented Mar 3, 2016

@mkaz ty!

@skyleelove

This comment has been minimized.

Copy link

commented May 10, 2016

hi, this example is client.Could you show a example for server.Thanks.

@cherrot

This comment has been minimized.

Copy link

commented Nov 29, 2016

in resp.Body.Read(bodyContent), bodyContent should not be a 0-length slice. Instead, initialize it using:

var bodyContent = make([]byte, contentLengthOfThisRequest)
@ghost

This comment has been minimized.

Copy link

commented Jan 7, 2017

This reads the whole file to RAM which is sometimes impossible 😓

@bacongobbler

This comment has been minimized.

Copy link

commented Jan 31, 2017

Instead of reading the entire file into memory and then writing it to the multipart form, just open the file, defer file.Close() then call io.Copy. For example:

func newfileUploadRequest(uri string, params map[string]string, paramName, path string) (*http.Request, error) {
	file, err := os.Open(path)
	if err != nil {
		return nil, err
	}
	defer file.Close()

	fi, err := file.Stat()
	if err != nil {
		return nil, err
	}

	body := new(bytes.Buffer)
	writer := multipart.NewWriter(body)
	part, err := writer.CreateFormFile(paramName, fi.Name())
	if err != nil {
		return nil, err
	}
	io.Copy(part, file)

	...

This requires adding io to the package import list.

@nullne

This comment has been minimized.

Copy link

commented Feb 4, 2017

@bacongobbler. thanks

@sclevine

This comment has been minimized.

Copy link

commented Mar 25, 2017

@bacongobbler @nullne @mattetti If you're interested in a streaming solution that doesn't require reading the entire file into memory at all (ex. for large files, or for production use in gateways), see here:
https://github.com/sclevine/cflocal/blob/49495238fad2959061bef7a23c6b28da8734f838/remote/droplet.go#L21-L58

(Worth noting: S3 doesn't accept chunked multi-part uploads, so you have to calculate the length ahead of time if that's where you're sending the file.)

@vicbaily528

This comment has been minimized.

Copy link

commented Nov 7, 2017

I found a problem:
When I simulated uploading a file in golang to the spring restful API, I found that the uploaded file was incorrect when I uploaded it using your method. Although it can get the data.
@PostMapping("/demo") public String postdemo(@RequestParam("userName") String userName, @RequestBody byte[] file) throws IOException { System.out.println(userName); if (file != null) { System.out.println(file.length); String jsonFilePath = "/local/code/12312312.zip"; File jsonFile = new File(jsonFilePath); FileOutputStream outputStream = FileUtils.openOutputStream(jsonFile); outputStream.write(file); outputStream.flush(); outputStream.close(); } return "you send post parame is " + userName; }

The header of the generated file will contain the following information

--45e6a34acb7e1b27165a75bf5052f2f4f00fc96dd04899d1bf66bd782fa1 Content-Disposition: form-data; name="file"; filename="pom.zip" Content-Type: application/octet-stream ����

If I use this code "request.Header.Add("Content-Type", writer.FormDataContentType())", then he will prompt me to get the parameters: username

@dezza

This comment has been minimized.

Copy link

commented Dec 21, 2017

and then just test it:

	rr := httptest.NewRecorder()
	handler := http.HandlerFunc(API.Create)
	handler.ServeHTTP(rr, req)

	if status := rr.Code; status != http.StatusOK {
		t.Errorf("handler returned wrong status code: got %v want %v",
			status, http.StatusOK)
	}
@eoinahern

This comment has been minimized.

Copy link

commented Jan 16, 2018

any examples of posting a file along with a serialized struct?

@lusi1990

This comment has been minimized.

Copy link

commented Jan 17, 2018

@mkaz save my day

@naughtymonkey1010

This comment has been minimized.

Copy link

commented Feb 28, 2018

@mkaz thanks

@icefed

This comment has been minimized.

Copy link

commented Mar 26, 2018

using io.pipe will be better?

@shuaihanhungry

This comment has been minimized.

Copy link

commented Apr 25, 2018

@e7

This comment has been minimized.

Copy link

commented Jul 12, 2018

@themihai I'm wondering it too, do you have solution?

@tcr-ableton

This comment has been minimized.

Copy link

commented Sep 4, 2018

My two cents: using file.Stat is an triggers an unnecessary system call. You could use filepath.Base(path) in order to obtain the filename.

@ramadani

This comment has been minimized.

Copy link

commented Dec 31, 2018

great solution @bacongobbler, big thanks

@melissafzhang

This comment has been minimized.

Copy link

commented Feb 13, 2019

I'm having similar issue as @vicbaily528 where the content type is an octet stream. Any ideas how to get around this?

@jk2K

This comment has been minimized.

Copy link

commented Feb 25, 2019

var bodyContent []byte
resp.Body.Read(bodyContent)
resp.Body.Close()
fmt.Println(bodyContent)

not work,

body, _ := ioutil.ReadAll(res.Body)
fmt.Println(res)
fmt.Println(string(body))

work well

@yzhanginwa

This comment has been minimized.

Copy link

commented Mar 31, 2019

Didn't work at first. Just used wireshark and found there is no content-type in the request.

@omerkaya1

This comment has been minimized.

Copy link

commented Apr 17, 2019

Thanks for this example.
It helped a lot.
However, if I may, I'd like to contribute to this discussion in the hope that it'll be of use to others and save them time.

As was already pointed out by @ghost when it comes to large files, it's inconvenient to use a buffer, as it consumes a lot of resources.

Luckily, there's a solution. After googling for a while I encountered this article where the idea of in-memory piping is leveraged for multipart uploading.
The advantages of this approach are tremendous in my point of view.
Another example can be found here.
I hope it'll be helpful.
Cheers.

@030

This comment has been minimized.

Copy link

commented May 31, 2019

@jacksonwbrito

This comment has been minimized.

Copy link

commented Jun 5, 2019

Thanks a lot!

I had the same problem as @vicbaily528 and @melissafzhang, after take a look at mime/multipart/writer.go I decide to replace the CreateFormFile function for a custom one:

func MyCreateFormFile(fieldname, filename, contentType string) (io.Writer, error) {
        h := make(textproto.MIMEHeader)
	h.Set("Content-Disposition", fmt.Sprintf(`form-data; name="%s"; filename="%s"`, escapeQuotes(fieldname), escapeQuotes(filename)))
	h.Set("Content-Type", escapeQuotes(contentType))
	return w.CreatePart(h)
}

this worked for me, I hope it can help you or others who have the same problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.