This project demonstrates how to put together a list of words for Apache Beam. This list of words is meant to be used to launch a Beam wordle game. The project has the following pieces:
scrape-beam-words.py.
The outputs of this pipeline are:
beam_wordle_words.txt*: A long list of words, separated by newline.beam_wordle_histogram.txt*: A file detailing the frequency in word
| #include <pthread.h> | |
| #include <stdio.h> | |
| #include <stdlib.h> | |
| #include <unistd.h> | |
| #define MAX_QUEUE_SIZE 10 | |
| typedef struct { | |
| int tag; | |
| char *message; |
| prebuilt_beam_sdk/beam_python_prebuilt_sdk | |
| IMAGES FOR REPO prebuilt_beam_sdk/beam_python_prebuilt_sdk | |
| IMAGES FOR REPO beam-sdk | |
| IMAGES FOR REPO beam_portability | |
| gcr.io/apache-beam-testing/prebuilt_beam_sdk/beam_python_prebuilt_sdk/cachegcr.io/apache-beam-testing/beam-sdk/beam_go_sdk gcr.io/apache-beam-testing/beam-sdk/beam_java11_sdk gcr.io/apache-beam-testing/beam-sdk/beam_java17_sdk gcr.io/apache-beam-testing/beam-sdk/beam_java8_sdk gcr.io/apache-beam-testing/beam-sdk/beam_python3.10_sdk gcr.io/apache-beam-testing/beam-sdk/beam_python3.6_sdk gcr.io/apache-beam-testing/beam-sdk/beam_python3.7_sdk gcr.io/apache-beam-testing/beam-sdk/beam_python3.8_sdk gcr.io/apache-beam-testing/beam-sdk/beam_python3.9_sdkgcr.io/apache-beam-testing/beam_portability/beam_flink1.10_job_server gcr.io/apache-beam-testing/beam_portability/beam_flink1.12_job_server gcr.io/apache-beam-testing/beam_portability/beam_flink1.13_job_server gcr.io/apache-beam-testing/beam_portability/beam_go_sdk gcr.io/apache-beam-testing/beam_portability/b |
This is a set of two sample Beam Pipelines that we use to calculate Pi with a monte carlo algorithm that relies on generating random points in an n-dimentional cube, and validating whether they are located within a circle in that space.
These examples are meant to showcase the use of Beam's Batched DoFns developed by @TheNeuralBit.
| apache-beam[aws,gcp,test] | |
| ipython |
| ## -*- coding: utf-8 -*- | |
| import json | |
| import sys | |
| import time | |
| import apache_beam as beam | |
| from apache_beam.io.fileio import CompressionTypes as ct | |
| from apache_beam.utils.options import PipelineOptions | |
| from beam_utils.sources import CsvFileSource |
.
| import subprocess | |
| tables = [ | |
| "Dictionary_codi", | |
| "Dictionary_item", | |
| "NFC_hits", | |
| "album", | |
| "apply", | |
| "banner", | |
| "board_comment", |
| #!/usr/bin/env python3 | |
| import sys | |
| import json | |
| import pickle | |
| def decode(dic, bitstr): | |
| res = [] | |
| length = bitstr.bit_length() - 1 | |
| if bitstr >> length != 1: |