Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
code snippet to store invalid records in Google Cloud Storage
//get all error records per topic
final Map<String, GCSInvalidMessagesWrapper> topicMessagesMap = getMessagesToSerializePerTopic(errorRecords);
//serialize the messages in GCS for each topic - a file with all messages per topic is stored in GCS
topicMessagesMap.keySet().forEach(topicName -> {
final String fileName = UUID.randomUUID().toString();
final String pathPrefix = gcsBasePathPrefix + "/" + topicName + "/"
+ getFormattedDatePrefix(Instant.now()) + "/";
final BlobId blobId = BlobId.of(gcsBucket, pathPrefix + fileName);
final Map<String, String> metaDataMap = new HashMap<>();
....
final BlobInfo objectInfo = BlobInfo.newBuilder(blobId)
.setContentType("text/plain")
.setMetadata(metaDataMap).build();
final Blob objectCreated = gcsStore.create(objectInfo,
topicMessagesMap.get(topicName).getBytes());
});
log.info("Pushing {} records to GCS success?: {}", errorRecords.size(), true);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment