Navigation Menu

Skip to content

Instantly share code, notes, and snippets.

@ajw0100
Created January 6, 2015 21:27
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save ajw0100/a515293a8967903de005 to your computer and use it in GitHub Desktop.
Save ajw0100/a515293a8967903de005 to your computer and use it in GitHub Desktop.
mongodb:
pkg:
- installed
service:
- running
- enable: true
- require:
- pkg: mongodb
mongo_data:
file:
- recurse
- name: /tmp/mongo_data
- source: salt://mongo/data
- user: {{ pillar['HOME_USER'] }}
- group: {{ pillar['HOME_GROUP'] }}
cmd:
- run
- name: |
gzip -d customer_reviews_1998.json.gz
mongoimport --db test --collection customer_reviews --type json \
--file customer_reviews_1998.json
- user: {{ pillar['HOME_USER'] }}
- group: {{ pillar['HOME_USER'] }}
- cwd: /tmp/mongo_data
- require:
- service: mongodb
- file: mongo_data
2015-01-06 21:15:43,980 [salt.state ][INFO ] Running state [mongodb] at time 21:15:43.980040
2015-01-06 21:15:43,981 [salt.state ][INFO ] Executing state pkg.installed for mongodb
2015-01-06 21:15:43,984 [salt.loaded.int.module.cmdmod][INFO ] Executing command ['apt-get', '-q', '-y', '-o', 'DPkg::Options::=--force-confold', '-o', 'DPkg::Options::=--force-confdef', 'install', 'mongodb'] in directory '/root'
2015-01-06 21:16:50,968 [salt.loaded.int.module.cmdmod][DEBUG ] output: Reading package lists...
Building dependency tree...
Reading state information...
The following extra packages will be installed:
libboost-dev libboost-filesystem1.54.0 libboost-program-options1.54.0
libboost-system1.54.0 libboost-thread1.54.0 libboost1.54-dev
libgoogle-perftools4 libpcrecpp0 libsnappy1 libstdc++-4.8-dev
libtcmalloc-minimal4 libunwind8 libv8-3.14.5 mongodb-clients mongodb-dev
mongodb-server
Suggested packages:
libboost-doc libboost1.54-doc libboost-atomic1.54-dev
libboost-chrono1.54-dev libboost-context1.54-dev libboost-coroutine.54-dev
libboost-date-time1.54-dev libboost-exception1.54-dev
libboost-filesystem1.54-dev libboost-graph1.54-dev
libboost-graph-parallel1.54-dev libboost-iostreams1.54-dev
libboost-locale1.54-dev libboost-log.54-dev libboost-math1.54-dev
libboost-mpi1.54-dev libboost-mpi-python1.54-dev
libboost-program-options1.54-dev libboost-python1.54-dev
libboost-random1.54-dev libboost-regex1.54-dev
libboost-serialization1.54-dev libboost-signals1.54-dev
libboost-system1.54-dev libboost-test1.54-dev libboost-thread1.54-dev
libboost-timer1.54-dev libboost-wave1.54-dev libboost1.54-tools-dev
libmpfrc++-dev libntl-dev libstdc++-4.8-doc
The following NEW packages will be installed:
libboost-dev libboost-filesystem1.54.0 libboost-program-options1.54.0
libboost-system1.54.0 libboost-thread1.54.0 libboost1.54-dev
libgoogle-perftools4 libpcrecpp0 libsnappy1 libstdc++-4.8-dev
libtcmalloc-minimal4 libunwind8 libv8-3.14.5 mongodb mongodb-clients
mongodb-dev mongodb-server
0 upgraded, 17 newly installed, 0 to remove and 71 not upgraded.
Need to get 22.4 MB of archives.
After this operation, 229 MB of additional disk space will be used.
Get:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-updates/main libboost-system1.54.0 amd64 1.54.0-4ubuntu3.1 [10.1 kB]
Get:2 http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-updates/main libboost-filesystem1.54.0 amd64 1.54.0-4ubuntu3.1 [34.2 kB]
Get:3 http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-updates/main libboost-program-options1.54.0 amd64 1.54.0-4ubuntu3.1 [115 kB]
Get:4 http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-updates/main libboost-thread1.54.0 amd64 1.54.0-4ubuntu3.1 [26.5 kB]
Get:5 http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty/main libpcrecpp0 amd64 1:8.31-2ubuntu2 [14.5 kB]
Get:6 http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty/main libunwind8 amd64 1.1-2.2ubuntu3 [48.3 kB]
Get:7 http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty/main libstdc++-4.8-dev amd64 4.8.2-19ubuntu1 [1050 kB]
Get:8 http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty-updates/main libboost1.54-dev amd64 1.54.0-4ubuntu3.1 [5682 kB]
Get:9 http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty/main libboost-dev amd64 1.54.0.1ubuntu1 [3076 B]
Get:10 http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty/main libtcmalloc-minimal4 amd64 2.1-2ubuntu1 [104 kB]
Get:11 http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty/main libgoogle-perftools4 amd64 2.1-2ubuntu1 [184 kB]
Get:12 http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty/universe libv8-3.14.5 amd64 3.14.5.8-5ubuntu2 [1189 kB]
Get:13 http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty/universe mongodb-dev amd64 1:2.4.9-1ubuntu2 [982 kB]
Get:14 http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty/main libsnappy1 amd64 1.1.0-1ubuntu1 [11.2 kB]
Get:15 http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty/universe mongodb-clients amd64 1:2.4.9-1ubuntu2 [9595 kB]
Get:16 http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty/universe mongodb-server amd64 1:2.4.9-1ubuntu2 [3312 kB]
Get:17 http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ trusty/universe mongodb amd64 1:2.4.9-1ubuntu2 [5124 B]
Fetched 22.4 MB in 46s (484 kB/s)
Selecting previously unselected package libboost-system1.54.0:amd64.
(Reading database ... 59568 files and directories currently installed.)
Preparing to unpack .../libboost-system1.54.0_1.54.0-4ubuntu3.1_amd64.deb ...
Unpacking libboost-system1.54.0:amd64 (1.54.0-4ubuntu3.1) ...
Selecting previously unselected package libboost-filesystem1.54.0:amd64.
Preparing to unpack .../libboost-filesystem1.54.0_1.54.0-4ubuntu3.1_amd64.deb ...
Unpacking libboost-filesystem1.54.0:amd64 (1.54.0-4ubuntu3.1) ...
Selecting previously unselected package libboost-program-options1.54.0:amd64.
Preparing to unpack .../libboost-program-options1.54.0_1.54.0-4ubuntu3.1_amd64.deb ...
Unpacking libboost-program-options1.54.0:amd64 (1.54.0-4ubuntu3.1) ...
Selecting previously unselected package libboost-thread1.54.0:amd64.
Preparing to unpack .../libboost-thread1.54.0_1.54.0-4ubuntu3.1_amd64.deb ...
Unpacking libboost-thread1.54.0:amd64 (1.54.0-4ubuntu3.1) ...
Selecting previously unselected package libpcrecpp0:amd64.
Preparing to unpack .../libpcrecpp0_1%3a8.31-2ubuntu2_amd64.deb ...
Unpacking libpcrecpp0:amd64 (1:8.31-2ubuntu2) ...
Selecting previously unselected package libunwind8.
Preparing to unpack .../libunwind8_1.1-2.2ubuntu3_amd64.deb ...
Unpacking libunwind8 (1.1-2.2ubuntu3) ...
Selecting previously unselected package libstdc++-4.8-dev:amd64.
Preparing to unpack .../libstdc++-4.8-dev_4.8.2-19ubuntu1_amd64.deb ...
Unpacking libstdc++-4.8-dev:amd64 (4.8.2-19ubuntu1) ...
Selecting previously unselected package libboost1.54-dev.
Preparing to unpack .../libboost1.54-dev_1.54.0-4ubuntu3.1_amd64.deb ...
Unpacking libboost1.54-dev (1.54.0-4ubuntu3.1) ...
Selecting previously unselected package libboost-dev.
Preparing to unpack .../libboost-dev_1.54.0.1ubuntu1_amd64.deb ...
Unpacking libboost-dev (1.54.0.1ubuntu1) ...
Selecting previously unselected package libtcmalloc-minimal4.
Preparing to unpack .../libtcmalloc-minimal4_2.1-2ubuntu1_amd64.deb ...
Unpacking libtcmalloc-minimal4 (2.1-2ubuntu1) ...
Selecting previously unselected package libgoogle-perftools4.
Preparing to unpack .../libgoogle-perftools4_2.1-2ubuntu1_amd64.deb ...
Unpacking libgoogle-perftools4 (2.1-2ubuntu1) ...
Selecting previously unselected package libv8-3.14.5.
Preparing to unpack .../libv8-3.14.5_3.14.5.8-5ubuntu2_amd64.deb ...
Unpacking libv8-3.14.5 (3.14.5.8-5ubuntu2) ...
Selecting previously unselected package mongodb-dev.
Preparing to unpack .../mongodb-dev_1%3a2.4.9-1ubuntu2_amd64.deb ...
Unpacking mongodb-dev (1:2.4.9-1ubuntu2) ...
Selecting previously unselected package libsnappy1.
Preparing to unpack .../libsnappy1_1.1.0-1ubuntu1_amd64.deb ...
Unpacking libsnappy1 (1.1.0-1ubuntu1) ...
Selecting previously unselected package mongodb-clients.
Preparing to unpack .../mongodb-clients_1%3a2.4.9-1ubuntu2_amd64.deb ...
Unpacking mongodb-clients (1:2.4.9-1ubuntu2) ...
Selecting previously unselected package mongodb-server.
Preparing to unpack .../mongodb-server_1%3a2.4.9-1ubuntu2_amd64.deb ...
Unpacking mongodb-server (1:2.4.9-1ubuntu2) ...
Selecting previously unselected package mongodb.
Preparing to unpack .../mongodb_1%3a2.4.9-1ubuntu2_amd64.deb ...
Unpacking mongodb (1:2.4.9-1ubuntu2) ...
Processing triggers for man-db (2.6.7.1-1) ...
Processing triggers for ureadahead (0.100.0-16) ...
Setting up libboost-system1.54.0:amd64 (1.54.0-4ubuntu3.1) ...
Setting up libboost-filesystem1.54.0:amd64 (1.54.0-4ubuntu3.1) ...
Setting up libboost-program-options1.54.0:amd64 (1.54.0-4ubuntu3.1) ...
Setting up libboost-thread1.54.0:amd64 (1.54.0-4ubuntu3.1) ...
Setting up libpcrecpp0:amd64 (1:8.31-2ubuntu2) ...
Setting up libunwind8 (1.1-2.2ubuntu3) ...
Setting up libstdc++-4.8-dev:amd64 (4.8.2-19ubuntu1) ...
Setting up libboost1.54-dev (1.54.0-4ubuntu3.1) ...
Setting up libboost-dev (1.54.0.1ubuntu1) ...
Setting up libtcmalloc-minimal4 (2.1-2ubuntu1) ...
Setting up libgoogle-perftools4 (2.1-2ubuntu1) ...
Setting up libv8-3.14.5 (3.14.5.8-5ubuntu2) ...
Setting up mongodb-dev (1:2.4.9-1ubuntu2) ...
Setting up libsnappy1 (1.1.0-1ubuntu1) ...
Setting up mongodb-clients (1:2.4.9-1ubuntu2) ...
Setting up mongodb-server (1:2.4.9-1ubuntu2) ...
Adding system user `mongodb' (UID 107) ...
Adding new user `mongodb' (UID 107) with group `nogroup' ...
Not creating home directory `/home/mongodb'.
Adding group `mongodb' (GID 113) ...
Done.
Adding user `mongodb' to group `mongodb' ...
Adding user mongodb to group mongodb
Done.
mongodb start/running, process 13093
Processing triggers for ureadahead (0.100.0-16) ...
Setting up mongodb (1:2.4.9-1ubuntu2) ...
Processing triggers for libc-bin (2.19-0ubuntu6.3) ...
2015-01-06 21:16:50,973 [salt.loaded.int.module.cmdmod][INFO ] Executing command "dpkg-query --showformat='${Status} ${Package} ${Version} ${Architecture}\n' -W" in directory '/root'
2015-01-06 21:16:51,055 [salt.state ][INFO ] Installed Packages:
libboost-dev changed from absent to 1.54.0.1ubuntu1
libpcrecpp0 changed from absent to 1:8.31-2ubuntu2
mongodb changed from absent to 1:2.4.9-1ubuntu2
libstdc++-dev changed from absent to 1
libboost-thread1.54.0 changed from absent to 1.54.0-4ubuntu3.1
libboost1.54-dev changed from absent to 1.54.0-4ubuntu3.1
libtcmalloc-minimal4 changed from absent to 2.1-2ubuntu1
libboost-program-options1.54.0 changed from absent to 1.54.0-4ubuntu3.1
mongodb-dev changed from absent to 1:2.4.9-1ubuntu2
mongodb-clients changed from absent to 1:2.4.9-1ubuntu2
mongodb-server changed from absent to 1:2.4.9-1ubuntu2
libgoogle-perftools4 changed from absent to 2.1-2ubuntu1
libstdc++-4.8-dev changed from absent to 4.8.2-19ubuntu1
libboost-system1.54.0 changed from absent to 1.54.0-4ubuntu3.1
libv8-3.14.5 changed from absent to 3.14.5.8-5ubuntu2
libsnappy1 changed from absent to 1.1.0-1ubuntu1
libunwind8 changed from absent to 1.1-2.2ubuntu3
libboost-filesystem1.54.0 changed from absent to 1.54.0-4ubuntu3.1
2015-01-06 21:16:51,055 [salt.state ][DEBUG ] Refreshing modules...
2015-01-06 21:16:51,113 [salt.state ][INFO ] Loading fresh modules for state activity
2015-01-06 21:16:51,383 [salt.utils.event ][DEBUG ] SaltEvent PUB socket URI: ipc:///var/run/salt/minion/minion_event_a4dcfaa704_pub.ipc
2015-01-06 21:16:51,383 [salt.utils.event ][DEBUG ] SaltEvent PULL socket URI: ipc:///var/run/salt/minion/minion_event_a4dcfaa704_pull.ipc
2015-01-06 21:16:51,384 [salt.utils.event ][DEBUG ] Sending event - data = {'_stamp': '2015-01-06T21:16:51.383569'}
2015-01-06 21:16:56,411 [salt.state ][INFO ] Completed state [mongodb] at time 21:16:56.410703
2015-01-06 21:16:56,412 [salt.state ][INFO ] Running state [mongodb] at time 21:16:56.412100
2015-01-06 21:16:56,412 [salt.state ][INFO ] Executing state service.running for mongodb
2015-01-06 21:16:56,422 [salt.loaded.int.module.cmdmod][INFO ] Executing command 'service mongodb status' in directory '/root'
2015-01-06 21:16:56,893 [salt.loaded.int.module.cmdmod][DEBUG ] output: mongodb start/running, process 13093
2015-01-06 21:16:56,907 [salt.state ][INFO ] Service mongodb is already enabled, and is in the desired state
2015-01-06 21:16:56,907 [salt.state ][INFO ] Completed state [mongodb] at time 21:16:56.906999
2015-01-06 21:16:56,908 [salt.state ][INFO ] Running state [/tmp/mongo_data] at time 21:16:56.907678
2015-01-06 21:16:56,908 [salt.state ][INFO ] Executing state file.recurse for /tmp/mongo_data
2015-01-06 21:17:04,587 [salt.fileclient ][DEBUG ] Fetching file from saltenv 'base', ** attempting ** 'salt://mongo/data/customer_reviews_1998.json.gz'
2015-01-06 21:17:09,226 [salt.fileclient ][INFO ] Fetching file from saltenv 'base', ** done ** 'mongo/data/customer_reviews_1998.json.gz'
2015-01-06 21:17:15,076 [salt.state ][INFO ] {'/tmp/mongo_data/customer_reviews_1998.json.gz': {'diff': 'New file', 'group': 'ubuntu', 'mode': '0644', 'user': 'ubuntu'}}
2015-01-06 21:17:15,077 [salt.state ][DEBUG ] Refreshing modules...
2015-01-06 21:17:15,206 [salt.state ][INFO ] Loading fresh modules for state activity
2015-01-06 21:17:15,459 [salt.utils.event ][DEBUG ] SaltEvent PUB socket URI: ipc:///var/run/salt/minion/minion_event_a4dcfaa704_pub.ipc
2015-01-06 21:17:15,459 [salt.utils.event ][DEBUG ] SaltEvent PULL socket URI: ipc:///var/run/salt/minion/minion_event_a4dcfaa704_pull.ipc
2015-01-06 21:17:15,460 [salt.utils.event ][DEBUG ] Sending event - data = {'_stamp': '2015-01-06T21:17:15.459930'}
2015-01-06 21:17:20,486 [salt.state ][INFO ] Completed state [/tmp/mongo_data] at time 21:17:20.486144
2015-01-06 21:17:20,488 [salt.state ][INFO ] Running state [gzip -d customer_reviews_1998.json.gz
mongoimport --db test --collection customer_reviews --type json \
--file customer_reviews_1998.json
] at time 21:17:20.487646
2015-01-06 21:17:20,488 [salt.state ][INFO ] Executing state cmd.run for gzip -d customer_reviews_1998.json.gz
mongoimport --db test --collection customer_reviews --type json \
--file customer_reviews_1998.json
2015-01-06 21:17:36,746 [salt.loaded.int.module.cmdmod][INFO ] Executing command 'gzip -d customer_reviews_1998.json.gz\nmongoimport --db test --collection customer_reviews --type json \\\n --file customer_reviews_1998.json\n' as user 'ubuntu' in directory '/tmp/mongo_data'
2015-01-06 21:18:40,449 [salt.loaded.int.module.cmdmod][ERROR ] Command 'gzip -d customer_reviews_1998.json.gz\nmongoimport --db test --collection customer_reviews --type json \\\n --file customer_reviews_1998.json\n' failed with return code: 255
2015-01-06 21:18:40,453 [salt.loaded.int.module.cmdmod][ERROR ] stderr: couldn't connect to [127.0.0.1] couldn't connect to server 127.0.0.1:27017
2015-01-06 21:18:40,453 [salt.loaded.int.module.cmdmod][ERROR ] retcode: 255
2015-01-06 21:18:40,454 [salt.state ][ERROR ] {'pid': 13128, 'retcode': 255, 'stderr': "couldn't connect to [127.0.0.1] couldn't connect to server 127.0.0.1:27017", 'stdout': ''}
2015-01-06 21:18:40,455 [salt.state ][INFO ] Completed state [gzip -d customer_reviews_1998.json.gz
mongoimport --db test --collection customer_reviews --type json \
--file customer_reviews_1998.json
] at time 21:18:40.454581
2015-01-06 21:18:40,987 [salt.loader ][DEBUG ] LazyLoaded .returner
➜ ~ sudo apt-get install mongodb
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
mongodb
0 upgraded, 1 newly installed, 0 to remove and 71 not upgraded.
Need to get 0 B/5,124 B of archives.
After this operation, 43.0 kB of additional disk space will be used.
Selecting previously unselected package mongodb.
(Reading database ... 71412 files and directories currently installed.)
Preparing to unpack .../mongodb_1%3a2.4.9-1ubuntu2_amd64.deb ...
Unpacking mongodb (1:2.4.9-1ubuntu2) ...
Setting up mongodb (1:2.4.9-1ubuntu2) ...
➜ ~ cd /tmp/mongo_data
➜ mongo_data mongoimport --db test --collection customer_reviews --type json \
> --file customer_reviews_1998.json
connected to: 127.0.0.1
Tue Jan 6 21:26:00.007 Progress: 17971466/261965676 6%
Tue Jan 6 21:26:00.012 40200 13400/second
Tue Jan 6 21:26:03.010 Progress: 35794194/261965676 13%
Tue Jan 6 21:26:03.010 80100 13350/second
Tue Jan 6 21:26:06.057 Progress: 53635788/261965676 20%
Tue Jan 6 21:26:06.058 120000 13333/second
Tue Jan 6 21:26:09.015 Progress: 70981831/261965676 27%
Tue Jan 6 21:26:09.015 158800 13233/second
Tue Jan 6 21:26:12.037 Progress: 83617883/261965676 31%
Tue Jan 6 21:26:12.038 187100 12473/second
Tue Jan 6 21:26:15.006 Progress: 102659908/261965676 39%
Tue Jan 6 21:26:15.011 229800 12766/second
Tue Jan 6 21:26:18.011 Progress: 120097979/261965676 45%
Tue Jan 6 21:26:18.011 269100 12814/second
Tue Jan 6 21:26:21.006 Progress: 137368471/261965676 52%
Tue Jan 6 21:26:21.007 308100 12837/second
Tue Jan 6 21:26:24.007 Progress: 147143256/261965676 56%
Tue Jan 6 21:26:24.007 330200 12229/second
Tue Jan 6 21:26:27.098 Progress: 166097299/261965676 63%
Tue Jan 6 21:26:27.098 373000 12433/second
Tue Jan 6 21:26:30.005 Progress: 177638067/261965676 67%
Tue Jan 6 21:26:30.006 399100 12093/second
Tue Jan 6 21:26:33.567 Progress: 195953277/261965676 74%
Tue Jan 6 21:26:33.568 440500 12236/second
Tue Jan 6 21:26:36.006 Progress: 211186005/261965676 80%
Tue Jan 6 21:26:36.007 474800 12174/second
Tue Jan 6 21:26:39.651 Progress: 230258873/261965676 87%
Tue Jan 6 21:26:39.652 518000 12333/second
Tue Jan 6 21:26:42.067 Progress: 244707357/261965676 93%
Tue Jan 6 21:26:42.068 550700 12237/second
Tue Jan 6 21:26:45.195 Progress: 258061227/261965676 98%
Tue Jan 6 21:26:45.196 581000 12104/second
Tue Jan 6 21:26:45.778 check 9 589859
Tue Jan 6 21:26:46.498 imported 589859 objects
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment