S3 Image Uploads via AWS-SDK with Golang

Uploading files and varying types of media can always be a headache. I was recently tasked with building out a media service; a dedicated Go service that would handle uploads to an s3 bucket. The documentation isn’t completely clear and most of the examples and tutorials are outdated, so I am posting a small snippet of basic working code using the AWS SDK

An Ultra-Basic GO AWS-SDK example for S3 Bucket image uploads

Define Credentials:

The first step to setting up the AWS-SDK is to define your credentials. Eventually you’ll want to do this using an environment variable or the /.aws/credentials file. For now we are going to use static credentials in code Do Not Upload To GitHub or use in Production or Development. Your Secret is like a password

awsAccessKey := "Insert Key Here" 
awsSecret := "Insert Secret Here" 
token := ""

Next we want to set our credentials into a credentials struct provided by the AWS-SDK. In this case we are using NewStaticCredentials, but only because we have statically set credentials, you might instead use NewEnvCredentials to pull in an environment variable

creds := credentials.NewStaticCredentials(awsAccessKey, awsSecret, token)

Next we want to call the .Get() method on our credentials struct. Get returns the credentials value or an error if the value cannot be retrieved. In this case, we don’t need the credential’s value, we just need to expose an error if there is one

_, err := creds.Get() 
if err != nil { 
 // handle error
}

Set up Configuration and S3 Instance

Next we need to set up a configuration object to pass to the SDK. In this case, we are going to use aws.NewConfig(), which returns a new config file to allow method chaining. We are going to chain this with our region and credentials

cfg := aws.NewConfig().WithRegion("us-west-1").WithCredentials(creds)

Next we need to create a new s3 instance that contains the client and session. s3.New() takes in two parameters, a ConfigProvider and an aws.Config struct

svc := s3.New(session.New(), cfg)

Prepping the file

Next we will prep the file. We need to convert it to []bytes and preferably pass along some metadata

First we will open the file (for this example to work, you’ll need to pass in a path to an actual file)

file, err := os.Open("test.jpg") 
if err != nil { 
  // handle error
} 
defer file.Close()

next we will determine the size of the file

fileInfo, _ := file.Stat() 
var size int64 = fileInfo.Size()

and create a buffer of the size of the file, set the file content to the buffer and then convert it to Bytes

buffer := make([]byte, size) 
file.Read(buffer) 
fileBytes := bytes.NewReader(buffer) 
fileType := http.DetectContentType(buffer)

Finally we make the set the path and put together the s3 request

path := "/media/" + file.Name() 
params := &s3.PutObjectInput{ 
  Bucket: aws.String("nameofBucketHere"), 
  Key: aws.String(path), 
  Body: fileBytes, 
  ContentLength: aws.Int64(size), 
  ContentType: aws.String(fileType), 
}

and submit it to AWS

resp, err := svc.PutObject(params) 
if err != nil { 
  // handle error
} 
fmt.Printf("response %s”, awsutil.StringValue(resp))

Final

import( 
  "github.com/aws/aws-sdk-go/aws" 
  "github.com/aws/aws-sdk-go/aws/awsutil" 
  "github.com/aws/aws-sdk-go/aws/credentials" 
  "github.com/aws/aws-sdk-go/service/s3" 
  "github.com/aws/aws-sdk-go/aws/session" 
) 
func main() { 
  aws_access_key_id := "Insert Key ID here" 
  aws_secret_access_key := "Insert Secret Here" 
  token := "" 
  creds := credentials.NewStaticCredentials(aws_access_key_id, aws_secret_access_key, token) 
  _, err := creds.Get() 
  if err != nil { 
    // handle error
  } 
  cfg := aws.NewConfig().WithRegion("us-west-1").WithCredentials(creds) 
  svc := s3.New(session.New(), cfg) 
  
  file, err := os.Open("test.jpg") 
  if err != nil { 
    // handle error
  } 
  defer file.Close() 
  fileInfo, _ := file.Stat() 
  size := fileInfo.Size() 
  buffer := make([]byte, size) // read file content to buffer 
 
  file.Read(buffer) 
  fileBytes := bytes.NewReader(buffer) 
  fileType := http.DetectContentType(buffer) 
  path := "/media/" + file.Name() 
  params := &s3.PutObjectInput{ 
    Bucket: aws.String("testBucket"), 
    Key: aws.String(path), 
    Body: fileBytes, 
    ContentLength: aws.Int64(size), 
    ContentType: aws.String(fileType), 
  } 
  resp, err := svc.PutObject(params) 
  if err != nil { 
    // handle error
  } 
  fmt.Printf("response %s", awsutil.StringValue(resp)) 
}

Receiving Images from a Form

A commenter recently asked about how to handle a more real world example where an image or piece of media is sent from a form. Luckily, it’s event easier because the data from the multipart request is already a buffer. You can use an abstracted GO method here

//r is an *http.Request object
file, h, err := r.FormFile(“data”) // or whatever the form key is
if err != nil {
  // handle error
}

From docs: https://golang.org/pkg/net/http/#Request.FormFile returns the first file for the provided form key. FormFile calls ParseMultipartForm and ParseForm if necessary.

the first return value (file) is the buffer, and the second return value (h) consists of several metadata fields (e.g. h.Header[“Content-Type”][0] will return the content-type, h.Filename will return the filename)

Originally published at questhenkart.com on February 3, 2016.