Skip to content

Instantly share code, notes, and snippets.

@jrhone
Created April 29, 2017 13:57
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save jrhone/1b586a0c2e0c788c80bf7affd75f70a2 to your computer and use it in GitHub Desktop.
Save jrhone/1b586a0c2e0c788c80bf7affd75f70a2 to your computer and use it in GitHub Desktop.
transporter issue
var source = mongodb({
"uri": "mongodb://user:pass@db:27017/mongo_app_develop?authSource=admin",
"ssl": true,
"bulk": false,
})
var sink = elasticsearch({
"uri": "http://ip:9200/mongo_app_develop"
})
t.Source("source", source, "/^role$/").Save("sink", sink, "role")
root@admin-develop:~# transporter run -log.level "debug" pipeline.js
DEBU[0000] failed to parse duration, , falling back to default timeout of 30s
INFO[0000] existing messages in commitlog, checking writer offsets... name=source newestOffset=662 oldestOffset=0 path=source type=mongodb
INFO[0000] offsetMap: map[user:563 user_following_user:659 role:661] name=sink path=source type=mongodb
INFO[0000] waiting for all children to resume... name=source path=source type=mongodb
INFO[0000] done waiting for all children to resume name=source path=source type=mongodb
INFO[0000] done checking for resume errors name=source path=source type=mongodb
INFO[0000] searching segments num_segments=1 offset=563
DEBU[0000] finding offset in segment offset=563 segment_index=0
INFO[0000] found offset position offset=563 position=600062
INFO[0000] searching segments num_segments=1 offset=659
DEBU[0000] finding offset in segment offset=659 segment_index=0
INFO[0000] found offset position offset=659 position=621974
INFO[0000] searching segments num_segments=1 offset=661
DEBU[0000] finding offset in segment offset=661 segment_index=0
INFO[0000] found offset position offset=661 position=622317
INFO[0000] starting with metadata map[user:{Msg:0xc4200163c0 Timestamp:0 Mode:0} user_following_user:{Msg:0xc420017680 Timestamp:0 Mode:0} role:{Msg:0xc4201f6180 Timestamp:0 Mode:0}] name=source path=source type=mongodb
INFO[0000] adaptor Starting... name=source path=source type=mongodb
INFO[0000] boot map[source:mongodb sink:elasticsearch] ts=1493473862878487415
INFO[0000] adaptor Listening... name=sink path="source/sink" type=elasticsearch
INFO[0000] starting Read func db="mongo_app_develop"
INFO[0000] SYNC Processing mongo-db-01-uet3s.mongodb.net:27017...
INFO[0000] Establishing new connection to mongo-db-01-uet3s.mongodb.net:27017 (timeout=5s)...
INFO[0000] collection count db="mongo_app_develop" num_collections=24
INFO[0000] skipping iteration... collection="response_drop" db="mongo_app_develop"
INFO[0000] adding for iteration... collection=role db="mongo_app_develop"
INFO[0000] skipping iteration... collection=spotify db="mongo_app_develop"
INFO[0000] done iterating collections db="mongo_app_develop"
INFO[0000] SYNC Processing mongo-db-00-uet3s.mongodb.net:27017...
INFO[0000] Establishing new connection to mongo-db-00-uet3s.mongodb.net:27017 (timeout=5s)...
INFO[0000] iterating... collection=role
INFO[0000] Establishing new connection to mongo-db-02-uet3s.mongodb.net:27017 (timeout=1h0m0s)...
INFO[0000] Connection to mongo-db-01-uet3s.mongodb.net:27017 established.
INFO[0000] SYNC Adding mongo-db-01-uet3s.mongodb.net:27017 to cluster as a slave.
INFO[0000] Connection to mongo-db-02-uet3s.mongodb.net:27017 established.
INFO[0000] Connection to mongo-db-00-uet3s.mongodb.net:27017 established.
INFO[0000] SYNC Adding mongo-db-00-uet3s.mongodb.net:27017 to cluster as a slave.
INFO[0000] iterating complete collection=role db="mongo_app_develop"
INFO[0000] Read completed db="mongo_app_develop"
INFO[0000] adaptor Start finished... name=source path=source type=mongodb
INFO[0000] closing session... name=source path=source type=mongodb
INFO[0000] session closed... name=source path=source type=mongodb
INFO[0000] adaptor Stopping... name=source path=source type=mongodb
INFO[0000] adaptor Stopped name=source path=source type=mongodb
INFO[0000] closing connection... name=source path=source type=mongodb
INFO[0000] connection closed... name=source path=source type=mongodb
INFO[0000] adaptor Stopping... name=sink path="source/sink" type=elasticsearch
INFO[0000] received stop, message buffer is empty, closing...
INFO[0000] adaptor Listen closed... name=sink path="source/sink" type=elasticsearch
INFO[0000] adaptor Stopped name=sink path="source/sink" type=elasticsearch
INFO[0000] closing connection... name=sink path="source/sink" type=elasticsearch
INFO[0000] connection closed... name=sink path="source/sink" type=elasticsearch
INFO[0000] closing writer... name=sink path="source/sink" type=elasticsearch
INFO[0000] closing BulkProcessor version=2 writer=elasticsearch
INFO[0000] writer closed... name=sink path="source/sink" type=elasticsearch
INFO[0000] metrics source records: 0 path=source ts=1493473863199973977
INFO[0000] metrics source/sink records: 0 path="source/sink" ts=1493473863199978823
INFO[0000] exit map[sink:elasticsearch source:mongodb] ts=1493473863199980020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment