Skip to content

Instantly share code, notes, and snippets.

View ycombinator's full-sized avatar

Shaunak Kashyap ycombinator

  • Elastic
  • San Jose, CA
View GitHub Profile
@ycombinator
ycombinator / keybase.md
Created June 11, 2021 19:35
keybase.md

Keybase proof

I hereby claim:

  • I am ycombinator on github.
  • I am ycombinator (https://keybase.io/ycombinator) on keybase.
  • I have a public key whose fingerprint is 88A0 9B75 97D9 3C72 4661 E601 2F4B ECBF E8DD 0F18

To claim this, I am signing this object:

@ycombinator
ycombinator / package-dashboard-id-prefixes.sh
Last active October 26, 2020 02:39
Prefix Kibana asset IDs with package names
#!/bin/bash
# Find dashboards to fix
to_fix=""
for p in packages/*; do
pkg=$(echo $p | cut -d"/" -f 2)
echo "Checking package $pkg..."
dashboard_filenames=$(find $p/kibana/dashboard -type f 2>/dev/null | cut -d "/" -f 5)
echo "Dashboards: "
@ycombinator
ycombinator / convert.py
Created July 9, 2020 00:42
Field aliasing
import yaml
def print_yaml(yaml_list):
print(yaml.safe_dump_all(yaml_list))
def make_yaml_list(tree):
yaml_list = []
for key in tree:
item = {
"name": key
@ycombinator
ycombinator / find_pr_jenkins_jobs.sh
Created January 28, 2020 17:53
Find Jenkins jobs for Beats PR
#!/bin/bash
# Usage:
# ./find_jenkins_job.sh PR_NUMBER
# Example:
# ./find_jenkins_job.sh 15790
set -e
NUM_JOBS_TO_SEARCH=100
@ycombinator
ycombinator / mb_mon_migration.md
Last active December 27, 2019 16:06
Metricbeat monitoring of stack products: migration plans

User perspective

Current state

As of: 7.5.0

  • ES:
    • Recommended: User installs and configures MB along side ES. MB monitors ES node and sends monitoring data directly to monitoring cluster, monitoring-* indices.
    • Alternative: User configures ES to send monitoring data to itself (aka production cluster). Production cluster forwards monitoring data to monitoring cluster, .monitoring-* indices.
  • Kibana:
  • Recommended: User installs and configures MB along side Kibana. MB monitors Kibana instance and sends monitoring data directly to monitoring cluster, monitoring-* indices.
#!/bin/bash
CHILD_PIDFILE=$PWD/child.pid
# Clean up old child, if any
if [ -f $CHILD_PIDFILE ]; then
OLD_CHILD_PID=$(cat $CHILD_PIDFILE)
ps $OLD_CHILD_PID >/dev/null
if [ $? -eq 0 ]; then
kill -9 $OLD_CHILD_PID
@ycombinator
ycombinator / adjusting_filebeat_es_logs.md
Last active July 22, 2019 14:58
Adjusting Filebeat when ES logs change

Background

Filebeat consumes Elasticsearch logs via its elasticsearch module. Specifically, for each type of Elasticsearch log (server, gc, deprecation, etc.) there is a corresponding fileset under the Filebeat elasticsearch module. This fileset is responsible for parsing the Elasticsearch log files into structured event that can then be shipped to Elasticsearch or other outputs.

So whenever the structure of Elasticsearch logs changes, the changes must be tested with the Filebeat elasticsearch module to ensure two things:

  • that the module can handle the new log structure, and
  • that the module can continue to handle the previous log structure (in case a user is running an older version of Elasticsearch against a newer version of Filebeat)

If necessary, the ingest pipeline used by the fileset to do the parsing should be updated.

Request

POST .monitoring-kibana-*/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "term": {
@ycombinator
ycombinator / instructions.md
Created April 2, 2019 14:19
Testing Monitoring Across Versions

Test if Monitoring Cluster version X is able to monitor Production Cluster version Y, where X > Y.

  1. Download and install Elasticsearch version X. This will be the Monitoring Cluster
  2. Download and install Elasticearch version Y. This will be the Production Cluster
  3. Download and install Kibana version Y. This will be the Production Kibana
  4. Start the Monitoring Cluster:
    bin/elasticsearch -E cluster.name=esmon -E node.name=esmon_1 -E http.port=9400
    
  5. Start the Production Cluster:
@ycombinator
ycombinator / input.svg
Created January 10, 2018 23:57
Logstash Pipeline Viewer: input plugin icon
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.