Skip to content

Instantly share code, notes, and snippets.

View stefnestor's full-sized avatar
🦖

Stef Nestor stefnestor

🦖
View GitHub Profile

Disable:

sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.metadata.mds.plist

Enable:

sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.metadata.mds.plist

@stefnestor
stefnestor / lnav_text_to_json.md
Created July 16, 2023 22:02
Exporting Multi-line Elastic Text Logs as JSON with LNAV

With example elasticsearch custom format (which handles processing multiple lines like Logstash's multilie codec for us) defined as

{"$schema": "https://lnav.org/schemas/format-v1.schema.json", "elasticsearch": {"title": "Elasticsearch Logs", "url": "https://github.com/elastic/elasticsearch", "description": "The log format for Elasticsearch", "file-pattern": ".*\\.log.*", "body-field": "message", "level-field": "level", "multiline": true, "regex": {"std": {"pattern": "^\\[(?<timestamp>[^\\]]+)\\]\\[(?<level>[^\\]]+) ?\\]\\[(?<class>[^\\]]+)\\] (?:\\[(?<hostname>[^\\]]*)\\])? ?(?:\\[(?<es_index>[^\\]]*)\\]\\[(?<shard>\\d+)\\]\\] )?(?<message>[^\\n]*)?\\n?(?<exception>(?:com|io|org|java)[^\\n]*)?\\n?(?<stacktrace>[\\s\\S]*)"} }, "level": {"error": "ERROR", "debug": "DEBUG", "warning": "WARN", "info": "INFO", "critical": "CRIT", "fatal": "FATAL"}, "opid-field
@stefnestor
stefnestor / elastic_cloud_apis.md
Created May 17, 2023 14:13
Elastic Cloud (ESS) API Summary Table

doc, generate api key

object doc description url
APM routes doc Reset secret tokens for your APM resources. POST /api/v1/deployments/{deployment_id}/apm/{ref_id}/_reset-token
Accounts doc Retrieve and update the current Elasticsearch Service account. GET /api/v1/account
PATCH /api/v1/account
PUT /api/v1/account
App Search (Read only) doc Enable read only mode on an App Search resource. GET /api/v1/deployments/{deployment_id}/appsearch/{ref_id}/read_only_mode
PUT /api/v1/deployments/{deployment_id}/appsearch/{ref_id}/read_only_mode
Authentication doc Mana

With ...

> SELECT id, color, name FROM shirts LIMIT 5

id   color     name
1    blue      Jane
2    yellow    Susan
3    green     Craig
4    yellow    Susan
5    blue      Susan
# docker-compose -f elastic_docker_singleNode_withSecurity.yaml -p "secure_elastic" up
version: '3'
services:
elasticsearch:
container_name: elasticsearch
image: docker.elastic.co/elasticsearch/elasticsearch:7.17.7
environment:
- bootstrap.memory_lock=true
- discovery.seed_hosts=elasticsearch
- discovery.type=single-node
@stefnestor
stefnestor / override_mapping_data_type.txt
Created December 22, 2022 16:42
Override Index Field's Mapping Data Type
# Runtime: https://www.elastic.co/guide/en/elasticsearch/reference/8.5/runtime-mapping-fields.html
DELETE test_index
PUT test_index
{ "mappings": { "properties": { "myKey": { "type": "text" }}}}
@stefnestor
stefnestor / kibana_feature_priveleges.md
Created November 12, 2022 21:21
Enable Feature-Specific, Kibana Rule Templates for Users/Roles

General rule access

doc If you want to use the alerting features in a Kibana app, you must have the appropriate feature privileges. For example, to create rules in Stack Management > Rules and Connectors, you must have all privileges for the Management > Stack Rules feature. To attach actions to the rule, you must also have read privileges for the Actions and Connectors feature. For more information on configuring roles that provide access to features, go to Feature privileges.

Specific rule type (via feature)

doc You must have all privileges for the appropriate Kibana features, depending on the consumer and rule_type_id of the rules you’re creating. For example, the Management > **Stac

@stefnestor
stefnestor / graphing_directory_files_stats.py
Created May 16, 2022 18:33
Graphing a Directory's File Statistics
import datetime
import math
import matplotlib.pyplot as plt # https://matplotlib.org
import mplcursors # https://mplcursors.readthedocs.io
import numpy as np # https://numpy.org/
import os
import sys
### usage notes
# Method to pull file statitics may depend on OS.
@stefnestor
stefnestor / lab_elasticsearch_cpu.md
Created May 8, 2022 00:33
What does Elasticsearch CPU report?

Checking the Node Stats *.cpu outputs compared to container/system CPU info.

http://localhost:9200/_nodes/stats?filter_path=nodes.*.os.cpu,nodes.*.os.cgroup.cpu,nodes.*.process.cpu.percent

This data backs JSON for CAT Nodes which uses os.cpu per code.

review

Docker

setup

Using Docker to run a local, basic Elasticsearch with Kibana setup: