Instantly share code, notes, and snippets.

Embed
What would you like to do?
Example of doing a multipart upload in Go (golang)
package main
import (
"bytes"
"fmt"
"io"
"log"
"mime/multipart"
"net/http"
"os"
"path/filepath"
)
// Creates a new file upload http request with optional extra params
func newfileUploadRequest(uri string, params map[string]string, paramName, path string) (*http.Request, error) {
file, err := os.Open(path)
if err != nil {
return nil, err
}
defer file.Close()
body := &bytes.Buffer{}
writer := multipart.NewWriter(body)
part, err := writer.CreateFormFile(paramName, filepath.Base(path))
if err != nil {
return nil, err
}
_, err = io.Copy(part, file)
for key, val := range params {
_ = writer.WriteField(key, val)
}
err = writer.Close()
if err != nil {
return nil, err
}
req, err := http.NewRequest("POST", uri, body)
req.Header.Set("Content-Type", writer.FormDataContentType())
return req, err
}
func main() {
path, _ := os.Getwd()
path += "/test.pdf"
extraParams := map[string]string{
"title": "My Document",
"author": "Matt Aimonetti",
"description": "A document with all the Go programming language secrets",
}
request, err := newfileUploadRequest("https://google.com/upload", extraParams, "file", "/tmp/doc.pdf")
if err != nil {
log.Fatal(err)
}
client := &http.Client{}
resp, err := client.Do(request)
if err != nil {
log.Fatal(err)
} else {
body := &bytes.Buffer{}
_, err := body.ReadFrom(resp.Body)
if err != nil {
log.Fatal(err)
}
resp.Body.Close()
fmt.Println(resp.StatusCode)
fmt.Println(resp.Header)
fmt.Println(body)
}
}
@themihai

This comment has been minimized.

Show comment
Hide comment
@themihai

themihai May 9, 2014

I'm wondering if it would be possible to upload a 'file' from memory (not from disk) ?

themihai commented May 9, 2014

I'm wondering if it would be possible to upload a 'file' from memory (not from disk) ?

@mkaz

This comment has been minimized.

Show comment
Hide comment
@mkaz

mkaz May 30, 2014

Thanks for the helpful gist, I think the only example I saw with a file upload and passing parameters. One thing, you should add a Content-Type header to the request.

For example, the return of newfileUploadRequest() :

request := http.NewRequest("POST", uri, body)
request.Header.Add("Content-Type", writer.FormDataContentType())
return request

mkaz commented May 30, 2014

Thanks for the helpful gist, I think the only example I saw with a file upload and passing parameters. One thing, you should add a Content-Type header to the request.

For example, the return of newfileUploadRequest() :

request := http.NewRequest("POST", uri, body)
request.Header.Add("Content-Type", writer.FormDataContentType())
return request
@wzhliang

This comment has been minimized.

Show comment
Hide comment
@wzhliang

wzhliang Sep 25, 2014

@mkaz that addition save a lot of my time!

wzhliang commented Sep 25, 2014

@mkaz that addition save a lot of my time!

@quxiao

This comment has been minimized.

Show comment
Hide comment
@quxiao

quxiao commented Jan 23, 2015

thanks! @mkaz

@sklyar

This comment has been minimized.

Show comment
Hide comment
@sklyar

sklyar Feb 20, 2015

Thanks. Do not tell me how to send multiple files at once?

sklyar commented Feb 20, 2015

Thanks. Do not tell me how to send multiple files at once?

@ystreibel

This comment has been minimized.

Show comment
Hide comment
@ystreibel

ystreibel Oct 6, 2015

👍 @mkaz thanks a lot

request, err := http.NewRequest("POST", uri, body)
request.Header.Add("Content-Type", writer.FormDataContentType())
return request, err

ystreibel commented Oct 6, 2015

👍 @mkaz thanks a lot

request, err := http.NewRequest("POST", uri, body)
request.Header.Add("Content-Type", writer.FormDataContentType())
return request, err
@xiconet

This comment has been minimized.

Show comment
Hide comment
@xiconet

xiconet Oct 24, 2015

Works, but memory usage is still high (at least in my hands), even using io.copy()

xiconet commented Oct 24, 2015

Works, but memory usage is still high (at least in my hands), even using io.copy()

@filewalkwithme

This comment has been minimized.

Show comment
Hide comment
@filewalkwithme

filewalkwithme commented Nov 24, 2015

@mkaz, thanks, bro!!

@ZetaChow

This comment has been minimized.

Show comment
Hide comment
@ZetaChow

ZetaChow commented Jan 16, 2016

@mkaz thanks!!

@kreshikhin

This comment has been minimized.

Show comment
Hide comment
@kreshikhin

kreshikhin Feb 12, 2016

@mkaz thanks, you'd saved my day -))

kreshikhin commented Feb 12, 2016

@mkaz thanks, you'd saved my day -))

@bigwheel

This comment has been minimized.

Show comment
Hide comment
@bigwheel

bigwheel commented Mar 3, 2016

@mkaz ty!

@skyleelove

This comment has been minimized.

Show comment
Hide comment
@skyleelove

skyleelove May 10, 2016

hi, this example is client.Could you show a example for server.Thanks.

skyleelove commented May 10, 2016

hi, this example is client.Could you show a example for server.Thanks.

@cherrot

This comment has been minimized.

Show comment
Hide comment
@cherrot

cherrot Nov 29, 2016

in resp.Body.Read(bodyContent), bodyContent should not be a 0-length slice. Instead, initialize it using:

var bodyContent = make([]byte, contentLengthOfThisRequest)

cherrot commented Nov 29, 2016

in resp.Body.Read(bodyContent), bodyContent should not be a 0-length slice. Instead, initialize it using:

var bodyContent = make([]byte, contentLengthOfThisRequest)
@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Jan 7, 2017

This reads the whole file to RAM which is sometimes impossible 😓

ghost commented Jan 7, 2017

This reads the whole file to RAM which is sometimes impossible 😓

@bacongobbler

This comment has been minimized.

Show comment
Hide comment
@bacongobbler

bacongobbler Jan 31, 2017

Instead of reading the entire file into memory and then writing it to the multipart form, just open the file, defer file.Close() then call io.Copy. For example:

func newfileUploadRequest(uri string, params map[string]string, paramName, path string) (*http.Request, error) {
	file, err := os.Open(path)
	if err != nil {
		return nil, err
	}
	defer file.Close()

	fi, err := file.Stat()
	if err != nil {
		return nil, err
	}

	body := new(bytes.Buffer)
	writer := multipart.NewWriter(body)
	part, err := writer.CreateFormFile(paramName, fi.Name())
	if err != nil {
		return nil, err
	}
	io.Copy(part, file)

	...

This requires adding io to the package import list.

bacongobbler commented Jan 31, 2017

Instead of reading the entire file into memory and then writing it to the multipart form, just open the file, defer file.Close() then call io.Copy. For example:

func newfileUploadRequest(uri string, params map[string]string, paramName, path string) (*http.Request, error) {
	file, err := os.Open(path)
	if err != nil {
		return nil, err
	}
	defer file.Close()

	fi, err := file.Stat()
	if err != nil {
		return nil, err
	}

	body := new(bytes.Buffer)
	writer := multipart.NewWriter(body)
	part, err := writer.CreateFormFile(paramName, fi.Name())
	if err != nil {
		return nil, err
	}
	io.Copy(part, file)

	...

This requires adding io to the package import list.

@nullne

This comment has been minimized.

Show comment
Hide comment
@nullne

nullne commented Feb 4, 2017

@bacongobbler. thanks

@sclevine

This comment has been minimized.

Show comment
Hide comment
@sclevine

sclevine Mar 25, 2017

@bacongobbler @nullne @mattetti If you're interested in a streaming solution that doesn't require reading the entire file into memory at all (ex. for large files, or for production use in gateways), see here:
https://github.com/sclevine/cflocal/blob/49495238fad2959061bef7a23c6b28da8734f838/remote/droplet.go#L21-L58

(Worth noting: S3 doesn't accept chunked multi-part uploads, so you have to calculate the length ahead of time if that's where you're sending the file.)

sclevine commented Mar 25, 2017

@bacongobbler @nullne @mattetti If you're interested in a streaming solution that doesn't require reading the entire file into memory at all (ex. for large files, or for production use in gateways), see here:
https://github.com/sclevine/cflocal/blob/49495238fad2959061bef7a23c6b28da8734f838/remote/droplet.go#L21-L58

(Worth noting: S3 doesn't accept chunked multi-part uploads, so you have to calculate the length ahead of time if that's where you're sending the file.)

@vicbaily528

This comment has been minimized.

Show comment
Hide comment
@vicbaily528

vicbaily528 Nov 7, 2017

I found a problem:
When I simulated uploading a file in golang to the spring restful API, I found that the uploaded file was incorrect when I uploaded it using your method. Although it can get the data.
@PostMapping("/demo") public String postdemo(@RequestParam("userName") String userName, @RequestBody byte[] file) throws IOException { System.out.println(userName); if (file != null) { System.out.println(file.length); String jsonFilePath = "/local/code/12312312.zip"; File jsonFile = new File(jsonFilePath); FileOutputStream outputStream = FileUtils.openOutputStream(jsonFile); outputStream.write(file); outputStream.flush(); outputStream.close(); } return "you send post parame is " + userName; }

The header of the generated file will contain the following information

--45e6a34acb7e1b27165a75bf5052f2f4f00fc96dd04899d1bf66bd782fa1 Content-Disposition: form-data; name="file"; filename="pom.zip" Content-Type: application/octet-stream ����

If I use this code "request.Header.Add("Content-Type", writer.FormDataContentType())", then he will prompt me to get the parameters: username

vicbaily528 commented Nov 7, 2017

I found a problem:
When I simulated uploading a file in golang to the spring restful API, I found that the uploaded file was incorrect when I uploaded it using your method. Although it can get the data.
@PostMapping("/demo") public String postdemo(@RequestParam("userName") String userName, @RequestBody byte[] file) throws IOException { System.out.println(userName); if (file != null) { System.out.println(file.length); String jsonFilePath = "/local/code/12312312.zip"; File jsonFile = new File(jsonFilePath); FileOutputStream outputStream = FileUtils.openOutputStream(jsonFile); outputStream.write(file); outputStream.flush(); outputStream.close(); } return "you send post parame is " + userName; }

The header of the generated file will contain the following information

--45e6a34acb7e1b27165a75bf5052f2f4f00fc96dd04899d1bf66bd782fa1 Content-Disposition: form-data; name="file"; filename="pom.zip" Content-Type: application/octet-stream ����

If I use this code "request.Header.Add("Content-Type", writer.FormDataContentType())", then he will prompt me to get the parameters: username

@dezza

This comment has been minimized.

Show comment
Hide comment
@dezza

dezza Dec 21, 2017

and then just test it:

	rr := httptest.NewRecorder()
	handler := http.HandlerFunc(API.Create)
	handler.ServeHTTP(rr, req)

	if status := rr.Code; status != http.StatusOK {
		t.Errorf("handler returned wrong status code: got %v want %v",
			status, http.StatusOK)
	}

dezza commented Dec 21, 2017

and then just test it:

	rr := httptest.NewRecorder()
	handler := http.HandlerFunc(API.Create)
	handler.ServeHTTP(rr, req)

	if status := rr.Code; status != http.StatusOK {
		t.Errorf("handler returned wrong status code: got %v want %v",
			status, http.StatusOK)
	}
@eoinahern

This comment has been minimized.

Show comment
Hide comment
@eoinahern

eoinahern Jan 16, 2018

any examples of posting a file along with a serialized struct?

eoinahern commented Jan 16, 2018

any examples of posting a file along with a serialized struct?

@lusi1990

This comment has been minimized.

Show comment
Hide comment
@lusi1990

lusi1990 commented Jan 17, 2018

@mkaz save my day

@naughtymonkey1010

This comment has been minimized.

Show comment
Hide comment
@naughtymonkey1010

naughtymonkey1010 commented Feb 28, 2018

@mkaz thanks

@znoodl

This comment has been minimized.

Show comment
Hide comment
@znoodl

znoodl Mar 26, 2018

using io.pipe will be better?

znoodl commented Mar 26, 2018

using io.pipe will be better?

@shuaihanhungry

This comment has been minimized.

Show comment
Hide comment
@shuaihanhungry

shuaihanhungry commented Apr 25, 2018

@e7

This comment has been minimized.

Show comment
Hide comment
@e7

e7 Jul 12, 2018

@themihai I'm wondering it too, do you have solution?

e7 commented Jul 12, 2018

@themihai I'm wondering it too, do you have solution?

@tcr-ableton

This comment has been minimized.

Show comment
Hide comment
@tcr-ableton

tcr-ableton Sep 4, 2018

My two cents: using file.Stat is an triggers an unnecessary system call. You could use filepath.Base(path) in order to obtain the filename.

tcr-ableton commented Sep 4, 2018

My two cents: using file.Stat is an triggers an unnecessary system call. You could use filepath.Base(path) in order to obtain the filename.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment