TBC
Yu Watanabe yuwtennis
yuwtennis
/ DagOptions.java
Created
July 14, 2024 09:54
Practical example for setting custom options as beamTestPipelineOptions for TestPipeline in apache beam
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| public interface DagOptions extends PipelineOptions { | |
| @Description("Dag options") | |
| @Default.String("HELLOWORLD") | |
| String getDagType(); | |
| void setDagType(String dagType); | |
| } |
yuwtennis
/ az-consumption
Last active
December 29, 2022 03:08
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| SUBSCRIPTION_ID: str = os.environ.get("SUBSCRIPTION_ID", None) | |
| SCOPE: str =f'/subscriptions/{SUBSCRIPTION_ID}' | |
| logging.basicConfig(filename='./az-sdk-consumption-api-8-0-0-b1.log', level=logging.DEBUG, force=True) | |
| current_timestamp = datetime.datetime.now(datetime.timezone.utc).strftime("%c %") | |
| print(f"Querying Microsoft REST API. Started on {current_timestamp}") | |
| consumption_client = ConsumptionManagementClient( | |
| credential=DefaultAzureCredential(), | |
| subscription_id=SUBSCRIPTION_ID |
yuwtennis
/ elasticsearchio
Last active
August 28, 2022 11:44
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package elasticsearchio | |
| import ( | |
| "fmt" | |
| "github.com/apache/beam/sdks/v2/go/pkg/beam" | |
| "github.com/apache/beam/sdks/v2/go/pkg/beam/core/typex" | |
| "github.com/apache/beam/sdks/v2/go/pkg/beam/core/util/reflectx" | |
| ) | |
| const ( |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [ywatanabe@laptop-archlinux beam]$ ./gradlew :sdks:java:io:expansion-service:runExpansionService -PconstructionService.port=18089 | |
| Configuration on demand is an incubating feature. | |
| > Task :sdks:java:io:expansion-service:runExpansionService | |
| Starting expansion service at localhost:18089 | |
| Aug 17, 2022 6:49:19 PM org.apache.beam.sdk.expansion.service.ExpansionService loadRegisteredTransforms | |
| beam:transform:org.apache.beam:kafka_read_with_metadata:v1: org.apache.beam.sdk.expansion.service.ExpansionService$ExternalTransformRegistrarLoader$1@663c9e7a | |
| INFO: Registering external transforms: [beam:transform:org.apache.beam:kafka_read_with_metadata:v1, beam:transform:org.apache.beam:kafka_read_without_metadata:v1, beam:transform:org.apache.beam:kafka_write:v1, beam:external:java:generate_sequence:v1] | |
| beam:transform:org.apache.beam:kafka_read_without_metadata:v1: org.apache.beam.sdk.expansion.service.ExpansionService$ExternalTransformRegistrarLoader$1@19e4653c | |
| beam:transform:org.apache.beam:kafka_w |
yuwtennis
/ TUTORIAL_ES_DATASTREAM.md
Last active
June 9, 2022 05:58
Indexing document using data stream in elassticsearch
yuwtennis
/ TUTORIAL_ES_ALIAS.txt
Last active
June 9, 2022 05:59
Indexing document using alias endpoint in elasticsearch
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # ILM | |
| # https://www.elastic.co/guide/en/elasticsearch/reference/7.17/ilm-rollover.html#ilm-rollover-primar-shardsize-ex | |
| # rollover in 5 min | |
| PUT _ilm/policy/my_policy | |
| { | |
| "policy": { | |
| "phases": { | |
| "hot": { | |
| "actions": { | |
| "rollover" : { |
See go.result for full log.
(venv) y-watanabe@LAPTOP-IG41EBJ5:~/repos/github/google-api-tutorials/sdks/go$ go run examples/quickstarts/admin/report/main.go &> go.result
(venv) y-watanabe@LAPTOP-IG41EBJ5:~/repos/github/google-api-tutorials/sdks/go$ grep Page go.result
2022/04/08 18:35:22 Page: 1, Received 1000 items
yuwtennis
/ filebeat.debug.yml
Last active
April 1, 2022 02:34
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ###################### Filebeat Configuration Example ######################### | |
| # This file is an example configuration file highlighting only the most common | |
| # options. The filebeat.reference.yml file from the same directory contains all the | |
| # supported options with more comments. You can use it as a reference. | |
| # | |
| # You can find the full configuration reference here: | |
| # https://www.elastic.co/guide/en/beats/filebeat/index.html | |
| # For more available modules and options, please see the filebeat.reference.yml sample |
yuwtennis
/ KAFKA_CONNECT_TUTORIAL_WITH_JDBC.md
Last active
March 13, 2022 10:34
Quick test with kafka connect using postgres consuming with kaka-python
This tutorial will guide through consuming message using jdbc source connector.
For consumer I used kafka python.
- Step1 Download kafka
- Step2 Boot up kafka cluster
- Step3 Setup postgresql
- Step4 Download jdbc driver
- Step5 Boot up kafka connect
- Step6 Start consuming
- Step7 Insert event
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| GET invoices/_search | |
| { | |
| "size": 0, | |
| "query": {"match_all": {} }, | |
| "aggs": { | |
| "amount_by_month": { | |
| "date_histogram": { | |
| "field": "date", | |
| "interval": "month", | |
| "order": { |
NewerOlder