Skip to content

Instantly share code, notes, and snippets.

View xyu's full-sized avatar
:shipit:

Xiao Yu xyu

:shipit:
View GitHub Profile
@xyu
xyu / JndiLookup.java
Created January 7, 2022 19:57
Nerfed JndiLookup
package org.apache.logging.log4j.core.lookup;
import org.apache.logging.log4j.core.LogEvent;
import org.apache.logging.log4j.core.config.plugins.Plugin;
@Plugin(name = "jndi", category = StrLookup.CATEGORY)
public class JndiLookup extends AbstractLookup {
@Override
public String lookup(final LogEvent event, final String key) {
return null;
@xyu
xyu / es-function-score.js
Created April 11, 2016 01:23
Example of Elasticsearch function scoring with gaussian decay
{
"query": {
"function_score": {
"query": {
"multi_match": {
"query": "...",
"fields": [
"title", "content", "author",
"tag.name", "category.name"
],
#!/usr/bin/env python
import argparse
import json
import random
import sys
def get_topic( args, data ):
topic = data['partitions'][0]['topic']
for partition in data['partitions']:
@xyu
xyu / hive-nginx.hql
Created January 1, 2016 20:33
Hive query to create table for parsing Nginx logs
-- Table for raw NGINX logs
CREATE EXTERNAL TABLE IF NOT EXISTS `raw_nginx_logs` (
`remote_addr` string,
`remote_user` string,
`time_local` string,
`http_verb` string,
`url` string,
`http_ver` string,
`status` int,
`body_bytes_sent` int,
$content_url = content_url();
error_log( print_r( $response, true ), 3, $content_url . "/" . 'debug.log' );

Keybase proof

I hereby claim:

  • I am xyu on github.
  • I am xyu (https://keybase.io/xyu) on keybase.
  • I have a public key whose fingerprint is FA86 9979 BE1E 4985 6091 F3DE A4C9 7671 14FE E84B

To claim this, I am signing this object:

@xyu
xyu / es13.iad
Created May 2, 2014 18:24
Elasticsearch OOM Log
[2014-04-30 03:11:13,900][WARN ][action.deletebyquery ] [es13.global.search.iad.wordpress.com] Failed to perform deleteByQuery/shard on replica [global-test-0-40m-50m][205]
org.elasticsearch.transport.RemoteTransportException: [es13.global.search.sat.wordpress.com][inet[/207.198.101.96:9300]][deleteByQuery/shard/replica]
Caused by: org.elasticsearch.index.engine.RefreshFailedEngineException: [global-test-0-40m-50m][205] Refresh failed
at org.elasticsearch.index.engine.internal.InternalEngine.refresh(InternalEngine.java:736)
at org.elasticsearch.index.engine.internal.InternalEngine.refreshVersioningTable(InternalEngine.java:906)
at org.elasticsearch.index.engine.internal.InternalEngine.delete(InternalEngine.java:648)
at org.elasticsearch.index.shard.service.InternalIndexShard.deleteByQuery(InternalIndexShard.java:451)
at org.elasticsearch.action.deletebyquery.TransportShardDeleteByQueryAction.shardOperationOnReplica(TransportShardDeleteByQueryAction.java:148)
at org.elasticsearch.ac
@xyu
xyu / gist:ef3a5c9faf66109d8c24
Last active August 29, 2015 14:00
Elasticsearch OOM Log
[2014-04-30 03:07:50,254][WARN ][monitor.jvm ] [es13] [gc][old][690921][528] duration [2.1m], collections [3]/[2.1m], total [2.1m]/[4.7m], memory [17.8gb]->[14gb]/[29.8gb], all_pools {[young] [744.2mb]->[46.6mb]/[1.1gb]}{[survivor] [149.7mb]->[0b]/[149.7mb]}{[old] [16.9gb]->[14gb]/[28.5gb]}
[2014-04-30 03:11:14,113][WARN ][monitor.jvm ] [es13] [gc][old][690922][533] duration [3.3m], collections [5]/[3.3m], total [3.3m]/[8.1m], memory [14gb]->[14.3gb]/[29.8gb], all_pools {[young] [46.6mb]->[233.4mb]/[1.1gb]}{[survivor] [0b]->[149.7mb]/[149.7mb]}{[old] [14gb]->[13.9gb]/[28.5gb]}
[2014-04-30 03:11:55,994][WARN ][monitor.jvm ] [es13] [gc][old][690923][534] duration [41.8s], collections [1]/[41.8s], total [41.8s]/[8.8m], memory [14.3gb]->[14gb]/[29.8gb], all_pools {[young] [233.4mb]->[243.2kb]/[1.1gb]}{[survivor] [149.7mb]->[0b]/[149.7mb]}{[old] [13.9gb]->[14gb]/[28.5gb]}
[2014-04-30 03:12:37,752][WARN ][monitor.jvm ] [es13] [gc][old][690924][535] duration [41.6s
::: [es13.global.search.sat.wordpress.com][4W88FVN4Rh-TuGhVzqhYYQ][inet[/207.198.101.96:9300]]{dc=sat, parity=1, master=false}
56.8% (284.2ms out of 500ms) cpu usage by thread 'elasticsearch[es13.global.search.sat.wordpress.com][search][T#8]'
2/10 snapshots sharing following 27 elements
sun.nio.ch.NativeThread.current(Native Method)
sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)
sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:670)
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:169)
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:271)
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:137)