mchv

for bucket in $(aws s3api list-buckets --query 'Buckets[*].{Name:Name}' --output text)
do 
    
    echo "$bucket:"
    
    region=$(aws s3api get-bucket-location --bucket $bucket --query 'LocationConstraint' --output text | awk '{sub(/None/,"eu-west-1")}; 1')
    parts=$(aws s3api list-multipart-uploads --bucket $bucket --region $region --query 'Uploads[*].{Key:Key,UploadId:UploadId}' --output text)

    if [ "$parts" != "None" ]; then
        IFS=$'\n'

jkeirstead

kixorz

"UserData": {
  "Fn::Base64": { "Fn::Join":["", [
    "#!/bin/bash -ex\n",
    "apt-get update\n",
    "apt-get -y install python-setuptools\n",
    "mkdir aws-cfn-bootstrap-latest\n",
    "curl https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz | tar xz -C aws-cfn-bootstrap-latest --strip-components 1\n",
    "easy_install aws-cfn-bootstrap-latest\n",
    "/usr/local/bin/cfn-init --stack ", { "Ref":"AWS::StackName" }, " --resource WebServer", " --region ", { "Ref": "AWS::Region" }, "\n",
    "\n",

shevron

Copyright (c) 2015, Shahar Evron
All rights reserved.

Redistribution and use in source and binary forms, with or without modification, 
are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this 
list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice, 

joseph-rickert

#------------------------------------------------------------ 
# REVOLUTION ANALYTICS WEBINAR: INTRODUCTION TO R FOR DATA MINING
# February 14, 2013
# Joseph B. Rickert
# Technical Marketing Manager
#
# BIG DATA with RevoScaleR
#
# Copyright: Revolution Analytics

duydo

If you want, I can try and help with pointers as to how to improve the indexing speed you get. Its quite easy to really increase it by using some simple guidelines, for example:

- Use create in the index API (assuming you can).
- Relax the real time aspect from 1 second to something a bit higher (index.engine.robin.refresh_interval).
- Increase the indexing buffer size (indices.memory.index_buffer_size), it defaults to the value 10% which is 10% of the heap.
- Increase the number of dirty operations that trigger automatic flush (so the translog won't get really big, even though its FS based) by setting index.translog.flush_threshold (defaults to 5000).
- Increase the memory allocated to elasticsearch node. By default its 1g.
- Start with a lower replica count (even 0), and then once the bulk loading is done, increate it to the value you want it to be using the update_settings API. This will improve things as possibly less shards will be allocated to each machine.
- Increase the number of machines you have so