Single User operations only are supported (no bulk)
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
def _hello_world_aggr(request, context): | |
""" | |
Aggregates the parameters to a single comma separated string. | |
:param request: iterable sequence of bundled rows | |
:return: string | |
""" | |
params = [] | |
logging.info('hello_world_aggr called') | |
# Iterate over bundled rows | |
for request_rows in request: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import com.google.cloud.dataflow.sdk.io.Read; | |
... | |
static class FormatToCanonical extends DoFn<Entity, String> { | |
@Override | |
public void processElement(ProcessContext c) { | |
create a json payload which consists of | |
some data extracted from the DS Entity | |
and the original Entity data in a 'raw_payload' field | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
public class Raw2Bigquery { | |
... | |
... | |
public static void main(String[] args) { | |
DataflowPipelineOptions options = PipelineOptionsFactory.create() | |
.as(DataflowPipelineOptions.class); | |
... | |
Pipeline mPipeline = Pipeline.create(options); | |
... |
###Benchmarking the system
We'll use tsung to benchmark the system.
####Tsung quick install
$ sudo apt-get install -y build-essential erlang gnuplot libtemplate-perl
$ sudo apt-get update
$ wget http://tsung.erlang-projects.org/dist/tsung-1.5.1.tar.gz
It is recommeded to read the 'Introduction' section in rest_api.md
before continuing here.
It is possible to connect to sigserv websockets with three URLs:
ws://HOST:PORT/v1/sub/GID/SID/TID1,TID2,TIDn?clear=0|1&timeout=T&idle_timeout=TI&verbose=0|1&aon=0|1
- Subscribe as GID + SID to topics TID1, TID2, ... TIDnws://HOST:PORT/v1/sub/GID/SID?clear=0|1&timeout=T&idle_timeout=TI&verbose=0|1&aon=0|1
- Subscribe as GID + SID
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
https://github.com/nivertech/sigserv/blob/master/doc/api/ws_api.md |