atcuno

The purpose of this document is to make recommendations on how to browse in a privacy and security conscious manner. This information is compiled from a number of sources, which are referenced throughout the document, as well as my own experiences with the described technologies.

I welcome contributions and comments on the information contained. Please see the How to Contribute section for information on contributing your own knowledge.

Table of Contents

• Motivation
• Audience
• TL;DR?
• Updates
• Technologies that Affect Security and Privacy

tillberg

function loggedAsyncAuto (obj, doneCb) {
  var globalStart;
  function time () {
    return new Date().getTime();
  }
  function elapsed (now) {
    var s = '' + (now - globalStart);
    return _.times(5 - s.length, function () { return ' '; }).join('') + s;
  }
  var keys = _.keys(obj);

gmarik

package main

import (
	"bufio"
	"fmt"
	"net"
)

type Client struct {
	incoming chan string

mrflip

Next Steps

• Measure time spend on index, flush, refresh, merge, query, etc. (TD - done)
• Take hot threads snapshots under read+write, read-only, write-only (TD - done)
• Adjust refresh time to 10s (from 1s) and see how load changes (TD)
• Measure time of a rolling restart doing disable_flush and disable_recovery (TD)
• Specify routing on query -- make it choose same node for each shard each time (MD)
• GC new generation size (TD)
• Warmers

• measure before/after of client query time with and without warmers (MD)

duydo

If you want, I can try and help with pointers as to how to improve the indexing speed you get. Its quite easy to really increase it by using some simple guidelines, for example:

- Use create in the index API (assuming you can).
- Relax the real time aspect from 1 second to something a bit higher (index.engine.robin.refresh_interval).
- Increase the indexing buffer size (indices.memory.index_buffer_size), it defaults to the value 10% which is 10% of the heap.
- Increase the number of dirty operations that trigger automatic flush (so the translog won't get really big, even though its FS based) by setting index.translog.flush_threshold (defaults to 5000).
- Increase the memory allocated to elasticsearch node. By default its 1g.
- Start with a lower replica count (even 0), and then once the bulk loading is done, increate it to the value you want it to be using the update_settings API. This will improve things as possibly less shards will be allocated to each machine.
- Increase the number of machines you have so