Skip to content

Instantly share code, notes, and snippets.

Avatar

Dieter Plaetinck Dieterbe

View GitHub Profile
View prepare-benchmarking.sh
#!/bin/bash
# useful articles:
# https://vstinner.github.io/journey-to-stable-benchmark-system.html
# http://perf.readthedocs.io/en/latest/system.html
# https://baiweiblog.wordpress.com/2017/11/02/how-to-set-processor-affinity-in-linux-using-taskset/
# https://wiki.archlinux.org/index.php/CPU_frequency_scaling#CPU_frequency_driver
if [[ $EUID > 0 ]]; then
echo "Please run as root"
exit 1
View bench.txt
$ go test -count=10 -run='^$' -bench=. -benchtime=100ms ./...
goos: linux
goarch: amd64
pkg: github.com/grafana/metrictank/api
BenchmarkFix1M-8 16 9189930 ns/op 1305.78 MB/s
BenchmarkFix1M-8 14 8155018 ns/op 1471.49 MB/s
BenchmarkFix1M-8 13 12086950 ns/op 992.81 MB/s
BenchmarkFix1M-8 13 9010493 ns/op 1331.78 MB/s
BenchmarkFix1M-8 16 10516333 ns/op 1141.08 MB/s
BenchmarkFix1M-8 16 9929228 ns/op 1208.55 MB/s
View metrictank-special-functions.md

as described in devdocs/expr.md, metrictank resets the consolidation function when it passes a "special" function. a special function is one that changes the nature of the data on a fundamental level. Such as summarize, perSecond, derivative, integral We explain the reasoning for that behavior with a few examples.

example 1

summarize(consolidateBy(statsd.count,"sum"),"24h","sum") 
  • the inputs to summarize, if we need to consolidate, can be summed, because we'll sum the data anyway
  • but if we want to see daily sums, and maxdatapoints dictates that we must consolidate at runtime we don't want the points summed together into buckets of arbitrary N-day-size buckets.
View analysis
sarama/8 rtm 3424 -> 0.030103
sarama/8 id 687540 -> 0.002537
sarama/8 ops 1009213 -> 0.004118
sarama/8 fng 3024891 -> 0.001198
sarama/8 fake-false 1000 -> 0.032249
sarama/8 fake-false 10000 -> 0.006223
sarama/8 fake-false 100000 -> 0.002262
sarama/8 fake-false 1000000 -> 0.000801
sarama/8 fake-false 10000000 -> 0.000325
sarama/8 fake-true 1000 -> 1.088978
View benchmarks
taskset --cpu-list 1,2 go test . -run='^$' -test.benchmem -bench .
goos: linux
goarch: amd64
pkg: github.com/grafana/metrictank/cluster/partitioner
BenchmarkPartitionerSarama-2 10000000 205 ns/op 0 B/op 0 allocs/op
BenchmarkJumpPartitionerMauro-2 5000000 271 ns/op 210 B/op 1 allocs/op
BenchmarkJumpPartitionerFnv-2 5000000 323 ns/op 8 B/op 1 allocs/op
BenchmarkJumpPartitionerMetro-2 20000000 101 ns/op 0 B/op 0 allocs/op
BenchmarkJumpPartitionerSip-2 10000000 145 ns/op 0 B/op 0 allocs/op
BenchmarkJumpPartitionerXxhash-2 20000000 99.4 ns/op 0 B/op 0 allocs/op
View unused
unused ./...
Unused has been deprecated. Please use staticcheck instead.
api/config.go:20:2: var logMinDur is unused (U1000)
cmd/mt-store-cat/match_test.go:5:6: type matchCase is unused (U1000)
cmd/mt-store-cat/match_test.go:40:6: func testmatch is unused (U1000)
cmd/mt-update-ttl/main_test.go:5:6: type testCase is unused (U1000)
cmd/mt-update-ttl/main_test.go:28:6: type completenessCase is unused (U1000)
cmd/mt-whisper-importer-reader/conversion_test.go:10:6: func testIncResolution is unused (U1000)
cmd/mt-whisper-importer-reader/conversion_test.go:397:6: func testDecResolution is unused (U1000)
cmd/mt-whisper-importer-reader/conversion_test.go:422:6: func getSimpleInData is unused (U1000)
View theme.sh
#!/bin/bash
style=$1
if [ "$style" != "LIGHT" -a "$style" != light -a "$style" != "dark" ]; then
echo "arg1 must be LIGHT, light or dark" >&2
exit 2
fi
# shell
# if we source this in terminals we can't just reload termite with proper colors,
View statsd-imho.txt
* etsy/statsd is the official statsd implementation, yet hasn't been maintained for more than 2 years.
(see also https://github.com/etsy/statsd/issues/649)
* dozens of forks and re-implementations exist in the community.
usually they bring one or more of extra features (e.g. graphite tags, security, ...),
easier deployment and better performance,
with various degrees of backwards compatibility
Which implementation is "best" depends on many factors, but often enough, it's not the official reference implementation.
IMHO, most of a statsd server's responsibility is catering to an organisations need of how they organize/name/.. their metrics.
So any implementation is very opinionated, and for etsy/statsd trying to cater to many different styles leads to a ballooning complexity
View carbon-relay-ng.ini.docker-dev-custom-cfg-kafka
## Global settings ##
# instance id's distinguish stats of multiple relays.
# do not run multiple relays with the same instance id.
# supported variables:
# ${HOST} : hostname
instance = "${HOST}"
max_procs = 2
admin_addr = "0.0.0.0:2004"
View carbon-relay-ng-memleak.sh
#!/bin/bash
log () {
echo "$(date +'%Y/%m/%d %H:%M:%S') $@"
}
for commit in $(cat commits); do
echo
log "doing $commit"
echo
You can’t perform that action at this time.