Skip to content

Instantly share code, notes, and snippets.

@coboshm
Created August 21, 2017 07:54
Show Gist options
  • Save coboshm/1c89bcc7bf2c9f9694e4984051474951 to your computer and use it in GitHub Desktop.
Save coboshm/1c89bcc7bf2c9f9694e4984051474951 to your computer and use it in GitHub Desktop.
Golang + Kinesis firehose
package main
import (
"log"
"encoding/json"
"fmt"
"os"
"math/rand"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/firehose"
)
// FakeEntity is just used for testing purposes
type FakeEntity struct {
ID int `json:"id"`
Name string `json:"name"`
Description string `json:"description"`
}
const maxUint = 4294967295
func main() {
log.Println("Begin")
streamName := "firehoseStream"
sess := session.Must(session.NewSession())
// Create a Firehose client with additional configuration
firehoseService := firehose.New(sess, aws.NewConfig().WithRegion("us-east-1"))
recordsBatchInput := &firehose.PutRecordBatchInput{}
recordsBatchInput = recordsBatchInput.SetDeliveryStreamName(streamName)
records := []*firehose.Record{}
for i := 0; i < 10; i++ {
data := FakeEntity{
ID: rand.Intn(maxUint),
Name: fmt.Sprintf("Name-%d", rand.Intn(maxUint)),
Description: fmt.Sprintf("Test-Description %d", rand.Intn(maxUint)),
}
b, err := json.Marshal(data)
if err != nil {
log.Printf("Error: %v", err)
os.Exit(1)
}
record := &firehose.Record{Data: b}
records = append(records, record)
}
recordsBatchInput = recordsBatchInput.SetRecords(records)
resp, err := firehoseService.PutRecordBatch(recordsBatchInput)
if err != nil {
fmt.Printf("PutRecordBatch err: %v\n", err)
} else {
fmt.Printf("PutRecordBatch: %v\n", resp)
}
log.Println("Done")
}
@leonardorifeli
Copy link

Hi @coboshm.

The firehose implementation in Nodejs, you can set an httpOption with time to get a timed-out and max retries. In golang do you know something to resolve that?

Tks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment