Skip to content

Instantly share code, notes, and snippets.

Avatar
🎧
Eat Sleep Code Repeat

Frank Hassanabad FrankHassanabad

🎧
Eat Sleep Code Repeat
View GitHub Profile
@FrankHassanabad
FrankHassanabad / doc_valuefields.test.md
Last active Jun 2, 2021
Mixing doc_valuefields with fields can lead to unexpected results
View doc_valuefields.test.md

I want to return @timestamp as a different format such as "YYYY" but am mixing together doc_valuefields with fields. Fields takes priority over doc_valuefields it looks like.

Ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html

We might have a slight bug here: https://github.com/elastic/kibana/blob/master/x-pack/plugins/security_solution/server/lib/detection_engine/signals/build_events_query.ts#L109-L125

Where we mix these two. I test this by trying override the date times below but it only works when we remove doc_valuefields and not mix it with fields

@FrankHassanabad
FrankHassanabad / per_test_const_keyword_alias.json
Last active Jun 1, 2021
Perf testing across const keyword and regular keyword and aliases
View per_test_const_keyword_alias.json
```json
# Adds an index for testing where we can test across:
# * keyword
# * const keyword
# * And a corner case where someone reindexes non-compatible const data into const data from _source.
DELETE const-logs-frank-delme-2
PUT const-logs-frank-delme-2
{
"mappings": {
"dynamic": "false",
@FrankHassanabad
FrankHassanabad / slow_data_stream_runtimefields.json
Created May 28, 2021
Slow queries using runtime fields to split against "data_stream.dataset" to create "event.module"
View slow_data_stream_runtimefields.json
# Create an index which has a runtime field to do a split against a "constant_keyword"
# of "data_stream.dataset" and use the first value found as the "event.module"
# I add an additional "host.name" type of keyword to compare
# * Profile of aggregating against the runtime field which does a split against the "constant_keyword"
# * Profile of aggregating against the const_keyword
# * Profile of aggregating against a normal keyword
# * Profile against a field alias against the "constant_keyword" field
DELETE const-logs-frank-delme-1
PUT const-logs-frank-delme-1
{
@FrankHassanabad
FrankHassanabad / queries_const_keyword.json
Last active May 28, 2021
Elastic Queries with aliases const keyword and non-const keyword
View queries_const_keyword.json
# Mapping with a "constant_keyword" for "data_stream.dataset"
# Mapping with an alias from "event.dataset" -> "data_stream.dataset"
DELETE const-keyword-frank-delme-1
PUT const-keyword-frank-delme-1
{
"mappings": {
"dynamic":"false",
"properties": {
"@timestamp": {
"type": "date"
@FrankHassanabad
FrankHassanabad / nested_kql_testing
Last active Jun 16, 2020
Tests around Nesting and KQL with references
View nested_kql_testing
#
# Create two mappings, one with nesting and one without:
#
# Optinally clean up any old index data
DELETE /delme-test-categories
DELETE /delme-test-categories2
# Without nesting
PUT /delme-test-categories
@FrankHassanabad
FrankHassanabad / test_saved_objects.sh
Created Feb 3, 2020
Test saved objects and duplicate issues part 1
View test_saved_objects.sh
#!/bin/sh
#
# Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
# or more contributor license agreements. Licensed under the Elastic License;
# you may not use this file except in compliance with the Elastic License.
#
set -e
./check_env_variables.sh
@FrankHassanabad
FrankHassanabad / check_empty.sh
Created Jan 30, 2020
Checks if keys are empty using jq
View check_empty.sh
for f in *.json ; do
echo "checking $f"
cat $f | jq .max_signals
done
View json_to_toml.sh
#!/bin/sh
# Download yj from:
# https://github.com/sclevine/yj/releases
# such as wget https://github.com/sclevine/yj/releases/download/v4.0.0/yj-macos
#
# Then chmod 755 ./yj-macos and chmod 755 toml_to_json.sh
# Go to your toml and run this
# toml_to_json.sh
@FrankHassanabad
FrankHassanabad / toml_to_json.sh
Last active Jan 15, 2020
Quick dirty toml to json converter which sorts the keys using jq and yj-macos
View toml_to_json.sh
#!/bin/sh
# Download yj from:
# https://github.com/sclevine/yj/releases
# such as wget https://github.com/sclevine/yj/releases/download/v4.0.0/yj-macos
#
# Then chmod 755 ./yj-macos and chmod 755 toml_to_json.sh
# Go to your toml and run this
# toml_to_json.sh
@FrankHassanabad
FrankHassanabad / sort_keys_jq.sh
Created Jan 14, 2020
Sorts JSON keys using jq
View sort_keys_jq.sh
#!/bin/sh
mkdir sorted
for f in ~/projects/kibana/x-pack/legacy/plugins/siem/server/lib/detection_engine/rules/prepackaged_rules/*.json ; do
echo $f
cat $f | jq . -S > sorted/$(basename -- "$f" .json).json
done