To simplify this process, we will use CFSSL - CloudFlare's PKI/TLS toolkit
brew install cfssl
Ankur Chauhan ankurcha
ankurcha
/ AvroToBigQuery.java
Created
October 23, 2015 05:29
Helper class to convert Apache Avro schema to BigQuery Table schema
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package com.mallo64.dataflow | |
| import com.google.api.client.json.GenericJson; | |
| import com.google.api.services.bigquery.model.TableCell; | |
| import com.google.api.services.bigquery.model.TableFieldSchema; | |
| import com.google.api.services.bigquery.model.TableRow; | |
| import com.google.api.services.bigquery.model.TableSchema; | |
| import com.google.cloud.dataflow.sdk.transforms.DoFn; | |
| import org.apache.avro.Schema; | |
| import org.apache.avro.specific.SpecificRecord; |
ankurcha
/ list-storage-transferjobs.py
Created
January 16, 2020 05:15
Simple python script to list running transfer jobs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import googleapiclient.discovery | |
| import warnings | |
| warnings.filterwarnings("ignore", "Your application has authenticated using end user credentials") | |
| storagetransfer = googleapiclient.discovery.build('storagetransfer', 'v1') | |
| def printTable(myDict, colList=None): | |
| """ Pretty print a list of dictionaries (myDict) as a dynamically sized table. | |
| If column names (colList) aren't specified, they will show in random order. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # runs linkerd in a daemonset, in linker-to-linker mode, with TLS for all calls | |
| # from linkerd to linkerd | |
| --- | |
| apiVersion: v1 | |
| kind: ConfigMap | |
| metadata: | |
| name: l5d-config | |
| data: | |
| config.yaml: |- | |
| admin: |
ankurcha
/ ViewRegistryPollerTransform.java
Created
January 5, 2017 20:33
Refreshing PCollectionView by polling view registry
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| public class ViewRegistryPollerTransform extends PTransform<PCollection<Row>, PCollection<MaterializedViewGenerator>> { | |
| // configuration | |
| private AnalyticsPipelineOptions opts; | |
| private Duration bucketDuration; | |
| public ViewRegistryPollerTransform(AnalyticsPipelineOptions opts, Duration bucketDuration) { | |
| this.opts = opts; | |
| this.bucketDuration = bucketDuration; | |
| } |
ankurcha
/ DiskOverflowReducer.java
Created
December 19, 2016 05:42
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import com.brightcove.rna.model.Events; | |
| import com.brightcove.rna.model.Row; | |
| import io.grpc.Context; | |
| import org.jetbrains.annotations.NotNull; | |
| import org.mapdb.*; | |
| import org.slf4j.Logger; | |
| import org.slf4j.LoggerFactory; | |
| import java.io.IOException; |
ankurcha
/ gist:c967597c4fce7df0d6fa0adcd6bedba3
Created
November 16, 2016 00:29
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [u'video'] 6418 | |
| [] 3741 | |
| [u'date'] 765 | |
| [u'day'] 694 | |
| [u'player', u'video'] 393 | |
| [u'country'] 274 | |
| [u'device_type'] 254 | |
| [u'country', u'video'] 176 | |
| [u'referrer_domain', u'video'] 176 | |
| [u'player'] 147 |
ankurcha
/ stacktrace.txt
Created
October 25, 2016 06:28
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| com.google.bigtable.repackaged.com.google.cloud.grpc.io.IOExceptionWithStatus: Error in response stream | |
| at com.google.bigtable.repackaged.com.google.cloud.grpc.scanner.ResultQueueEntry$ExceptionResultQueueEntry.getResponseOrThrow(ResultQueueEntry.java:88) | |
| at com.google.bigtable.repackaged.com.google.cloud.grpc.scanner.ResponseQueueReader.getNextMergedRow(ResponseQueueReader.java:95) | |
| at com.google.bigtable.repackaged.com.google.cloud.grpc.scanner.StreamingBigtableResultScanner.next(StreamingBigtableResultScanner.java:60) | |
| at com.google.bigtable.repackaged.com.google.cloud.grpc.scanner.StreamingBigtableResultScanner.next(StreamingBigtableResultScanner.java:34) | |
| at com.google.bigtable.repackaged.com.google.cloud.grpc.scanner.ResumingStreamingResultScanner.next(ResumingStreamingResultScanner.java:89) | |
| at com.google.bigtable.repackaged.com.google.cloud.grpc.scanner.ResumingStreamingResultScanner.next(ResumingStreamingResultScanner.java:35) | |
| at com.google.cloud.bigtable.dataflow.CloudBigtableIO$1.next(CloudBigta |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| STARTUP_MSG: java = 1.8.0_25 | |
| ************************************************************/ | |
| 14/10/28 06:27:07 INFO mapred.JobTracker: registered UNIX signal handlers for [TERM, HUP, INT] | |
| 14/10/28 06:27:08 FATAL mapred.JobTracker: java.lang.IllegalArgumentException: Does not contain a valid host:port authority: local | |
| at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:211) | |
| at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:163) | |
| at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:152) | |
| at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:2165) | |
| at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1764) | |
| at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1757) |
ankurcha
/ dataflow.dot
Created
June 12, 2016 00:07
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| strict digraph "StepGraph" { | |
| subgraph "cluster AnalyticsTransform" { | |
| style=filled; | |
| bgcolor=white; | |
| labeljust=left; | |
| tooltip="AnalyticsTransform"; | |
| label="AnalyticsTransform"; | |
| subgraph "cluster AnalyticsTransform/ActivityEvents" { | |
| style=filled; | |
| bgcolor=white; |
NewerOlder