Skip to content

Instantly share code, notes, and snippets.

Dvir Volk dvirsky

  • Google
Block or report user

Report or block dvirsky

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
View safelog.h
#include <redismodule.h>
#include <stdarg.h>
extern RedisModuleCtx *logCtx_g;
// FFS let's have contstants for the logging levels and not hard code them
#define RS_LOG_WARNING "warning"
#define ...
View agg.c
typedef struct {
RSMultiKey *keys;
} AggregateLoadStep;
typedef struct {
const char *reducer;
CmdArg *args; // TODO: something better here...
const char *alias;
} AggregateGroupReduce;
View Aggregate.md

RediSearch Aggregations

[TOC]

Aggregations are a way to process the results of a search query, group, sort and transform them - and extract analytic insights from them. Much like aggregation queries in other databases and search engines, they can be used to create analytics report, or to perform Faceted Search style queries.

For example, indexing a web-server's logs, we can create report for unique users by hour, country or any other breakdown; or create different reports for errors, warnings, etc.

View cmdparse.md

CmdParse - Complex Redis Command Parsing

This is an API to help with parsing complex redis commands - where you need complex and recursive command structues. It can validate and semantically parse commands into a structured AST of sorts.

The main idea is that you can define a Schema for the command, detailing its structure, argument types and so on - and the API can automatically validate and parse incoming redis arguments.

It currently supports:

  • Named and positional arguments.
  • Required and optional arguments.
View aggregations.md

RediSearch Aggregation Engine

Main Idea

RediSearch can already index and retrieve large amounts of documents really fast. This is a proposal to allow aggregations on top of that - i.e to enable extracting statitstics and insights from data stored in Redis using RediSearch.

The idea is to perform a search on the RS index, load properties from the fetched documents, and perform calculations based on them - using grouping, sorting, and projection functions. These are composed as a pipeline, and reentrant.

Internally, the aggregation engine uses the same mechanism that loads normal search results (the result processor chain), only using a different set of result processors, which ultimately build result objects from the pipeline. If a normal search processing pipeline looks like filter -> score -> sort -> load documents -> serialize, an aggregation pipeline would look like: filter -> load properties -> group -> reduce -> project -> sort -> serialize.

@dvirsky
dvirsky / tokenize.py
Last active Jun 28, 2017
Pre-tokenize source code for searching
View tokenize.py
import re
import itertools
import sys
def snake_case_split(ident):
"""
Split a snake case identifier into words, returning the original ident and its splits as a list
"""
splits = filter(None, re.split('_', ident))
View module_style.c
#ifndef NS_HEADER_NAME_H_
#define NS_HEADER_NAME_H_
/* Enums are defined like data types */
typedef enum {
/* Options of enums are documented */
NSEnum_Foo,
/* Options of enums are documented */
@dvirsky
dvirsky / Makefile
Last active Jun 15, 2017
Minimal makefile for a redis module
View Makefile
DEBUGFLAGS = -g -ggdb -O2
ifeq ($(DEBUG), 1)
DEBUGFLAGS = -g -ggdb -O0
endif
# find the OS
uname_S := $(shell sh -c 'uname -s 2>/dev/null || echo not')
CFLAGS = -Wall -Wno-unused-function $(DEBUGFLAGS) -fPIC -std=gnu99 -D_GNU_SOURCE
CC:=$(shell sh -c 'type $(CC) >/dev/null 2>/dev/null && echo $(CC) || echo gcc')
View helloblock.c
/* Helloblock module -- An example of blocking command implementation
* with threads.
*
* -----------------------------------------------------------------------------
*
* Copyright (c) 2016, Salvatore Sanfilippo <antirez at gmail dot com>
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
@dvirsky
dvirsky / crc16_slottable.h
Last active Oct 14, 2019
A table of the shortest possible alphanumeric string that is mapped by redis' crc16 to any given cluster slot. the
View crc16_slottable.h
#ifndef _CRC16_TABLE_H__
#define _CRC16_TABLE_H__
/* A table of the shortest possible alphanumeric string that is mapped by redis' crc16
* to any given redis cluster slot.
*
* The array indexes are slot numbers, so that given a desired slot, this string is guaranteed
* to make redis cluster route a request to the shard holding this slot
*/
You can’t perform that action at this time.