Skip to content

Instantly share code, notes, and snippets.

View yuwtennis's full-sized avatar

Yu Watanabe yuwtennis

3 followers · 5 following
View GitHub Profile
@yuwtennis
yuwtennis / az-consumption
Last active December 29, 2022 03:08
SUBSCRIPTION_ID: str = os.environ.get("SUBSCRIPTION_ID", None)
SCOPE: str =f'/subscriptions/{SUBSCRIPTION_ID}'
logging.basicConfig(filename='./az-sdk-consumption-api-8-0-0-b1.log', level=logging.DEBUG, force=True)
current_timestamp = datetime.datetime.now(datetime.timezone.utc).strftime("%c %")
print(f"Querying Microsoft REST API. Started on {current_timestamp}")
consumption_client = ConsumptionManagementClient(
credential=DefaultAzureCredential(),
subscription_id=SUBSCRIPTION_ID
@yuwtennis
yuwtennis / elasticsearchio
Last active August 28, 2022 11:44
package elasticsearchio
import (
"fmt"
"github.com/apache/beam/sdks/v2/go/pkg/beam"
"github.com/apache/beam/sdks/v2/go/pkg/beam/core/typex"
"github.com/apache/beam/sdks/v2/go/pkg/beam/core/util/reflectx"
)
const (
@yuwtennis
yuwtennis / log
Created August 17, 2022 10:00
No expansion service for ElasticsearchIO
[ywatanabe@laptop-archlinux beam]$ ./gradlew :sdks:java:io:expansion-service:runExpansionService -PconstructionService.port=18089
Configuration on demand is an incubating feature.
> Task :sdks:java:io:expansion-service:runExpansionService
Starting expansion service at localhost:18089
Aug 17, 2022 6:49:19 PM org.apache.beam.sdk.expansion.service.ExpansionService loadRegisteredTransforms
beam:transform:org.apache.beam:kafka_read_with_metadata:v1: org.apache.beam.sdk.expansion.service.ExpansionService$ExternalTransformRegistrarLoader$1@663c9e7a
INFO: Registering external transforms: [beam:transform:org.apache.beam:kafka_read_with_metadata:v1, beam:transform:org.apache.beam:kafka_read_without_metadata:v1, beam:transform:org.apache.beam:kafka_write:v1, beam:external:java:generate_sequence:v1]
beam:transform:org.apache.beam:kafka_read_without_metadata:v1: org.apache.beam.sdk.expansion.service.ExpansionService$ExternalTransformRegistrarLoader$1@19e4653c
beam:transform:org.apache.beam:kafka_w
@yuwtennis
yuwtennis / TUTORIAL_ES_DATASTREAM.md
Last active June 9, 2022 05:58
Indexing document using data stream in elassticsearch

TBC

@yuwtennis
yuwtennis / TUTORIAL_ES_ALIAS.txt
Last active June 9, 2022 05:59
Indexing document using alias endpoint in elasticsearch
# ILM
# https://www.elastic.co/guide/en/elasticsearch/reference/7.17/ilm-rollover.html#ilm-rollover-primar-shardsize-ex
# rollover in 5 min
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover" : {
@yuwtennis
yuwtennis / RESULT.md
Last active April 8, 2022 10:51
Comparing sdk results for google admin reports api

For go

Source

See go.result for full log.

(venv) y-watanabe@LAPTOP-IG41EBJ5:~/repos/github/google-api-tutorials/sdks/go$ go run examples/quickstarts/admin/report/main.go &> go.result
(venv) y-watanabe@LAPTOP-IG41EBJ5:~/repos/github/google-api-tutorials/sdks/go$ grep Page go.result
2022/04/08 18:35:22 Page: 1, Received 1000 items
@yuwtennis
yuwtennis / filebeat.debug.yml
Last active April 1, 2022 02:34
###################### Filebeat Configuration Example #########################
# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html
# For more available modules and options, please see the filebeat.reference.yml sample
@yuwtennis
yuwtennis / KAFKA_CONNECT_TUTORIAL_WITH_JDBC.md
Last active March 13, 2022 10:34
Quick test with kafka connect using postgres consuming with kaka-python

This tutorial will guide through consuming message using jdbc source connector.
For consumer I used kafka python.

  • Step1 Download kafka
  • Step2 Boot up kafka cluster
  • Step3 Setup postgresql
  • Step4 Download jdbc driver
  • Step5 Boot up kafka connect
  • Step6 Start consuming
  • Step7 Insert event
@yuwtennis
yuwtennis / search.txt
Last active March 5, 2022 05:24
Example cost spike monitoring
GET invoices/_search
{
"size": 0,
"query": {"match_all": {} },
"aggs": {
"amount_by_month": {
"date_histogram": {
"field": "date",
"interval": "month",
"order": {
@yuwtennis
yuwtennis / build.gradle
Created February 23, 2022 10:28
spark-2_12.3.2.1
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* License); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0