Skip to content

Instantly share code, notes, and snippets.

View krestenkrab's full-sized avatar

Kresten Krab Thorup krestenkrab

View GitHub Profile
message Person {
required int32 id = 1;
required string name = 2;
optional string email = 3;
repeated Address addresses = 4;
extensions 10 to max;
}
message Address {
@krestenkrab
krestenkrab / hashtree_optimizations.md
Last active August 29, 2015 14:10
hash treee optimizations

hashtree.erl optimization ideas

On-disk format for segment data

The current on-disk format is that each KV is stored in the tree file as

<<$t,TreeId:22/binary,$s,Segment:64/integer,Key/binary>>

TreeId is 22 bytes (20 bytes from the vnode id, and 2 bytes for N value). The implementation in hashtree uses 1M segments, and the Segment is stored in a 64 bit unsiged. Finally, the Key is usually term_to_binary({Bucket,Key}).

// Representation of int32
class EInt32 extends EObject {
static final long MIN_INT = Integer.MIN_VALUE, MAX_LONG=Long.MAX_VALUE;
int value;
ENumber plus(EObject rhs) { return rhs.r_plus(value); }
ENumber r_plus(int lhs) {
long ret = lhs + this.value;
if(((ret + MIN_INT) & MAX_LONG) <= 0xFFFFFFFFL)
return new EInt32((int)ret);
else
The Erjang/OTP shell in action!
(For some reason it does not print the prompt; don't let yourself be confused by that).
krab$ ./erl.sh
Eshell V5.7.3 (abort with ^G)
erlang:display("hello world!").
"hello world!"
true
## Erjang session trying to compile a simple erlang file (see foo.erl below).
##
## The good news is, that the compiler (erlc hosten on erjang) runs, and the
## failure happens not during compilation but when the eshell command c("foo")
## tries to load the resulting beam file. That's a substantial chunk of erlang
## code running there.
##
## The bad news is that the written beam file is pretty bogus. Most likely has
## to do with bug(s) in binary assembly, or iolist() handling somewhere.
##
krab$ cat > foo.erl
-module(foo).
-export([main/0]).
main() -> version1.
^D
krab$ ./erl.sh
Eshell V5.7.3 (abort with ^G)
1> c("foo").
Krestens-MacBook-Pro:erjang krab$ java -jar erjang-0.1.jar
Eshell V5.7.3 (abort with ^G)
1> {ok,P} = prim_inet:open(tcp).
{ok,<port:474>}
2> prim_inet:close(P).
ok
3>
Eshell V5.7.3 (abort with ^G)
1> {ok,P} = inet_tcp:connect({74,125,79,104},80,[]).
{ok,<port:478>}
2> V1 = inet:send(P, "GET /\n").
ok
3> process_info(self(), messages).
{messages,[{tcp,<port:478>,
"HTTP/1.0 302 Found\r\nLocation: http://www.google.dk/\r\nCache-Control: private\r\nContent-Type: text/html; charset=UTF-8\r\nSet-Cookie: PREF=ID=343fdb84f9eb331b:TM=1270640853:LM=1270640853:S=AaVxF7QpxV4CDNz1; expires=Fri, 06-Apr-2012 11:47:33 GMT; path=/; domain=.google.com\r\nSet-Cookie: NID=33=tEmot8xGMHKZa-gsccbD7tDiXpsHaC24bKm7tSBYgdomt_nceXViDrTzEIn4eosoj7qhYAXVPYr7u60vKjW0UdE-FYN9sP3UmOz89XnvoY8qeVJu5ewjnMgKdmjB7uw8; expires=Thu, 07-Oct-2010 11:47:33 GMT; path=/; domain=.google.com; HttpOnly\r\nDate: Wed, 07 Apr 2010 11:47:33 GMT\r\nServer: gws\r\nContent-Length: 218\r\n\r\n<HTML><HEAD><meta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\">\n<TITLE>302 Moved</TITLE></HEAD><BODY>\n<H1>302 Moved</H1>\nThe document has moved\n<A HREF=\"http://www.google.dk/\">here</A>.\r\n</BODY
## LOG OF BEAM SESSION [with erl_dist debug output]
pc-226:~ krab$ /sw/bin/erl -name beam
Erlang R13B02 (erts-5.7.3) [source] [smp:2:2] [rq:2] [async-threads:0] [kernel-poll:false]
##
## This does " init ! { self(), get_arguments } " to remote Erlang VM.
##
Eshell V5.7.3 (abort with ^G)
%%
%% First complete ETS test suite run, with 37% success.
%%
pc-226:test_server krab$ java -d64 -server -Xss100m -da -jar ../erjang-0.1.jar -home $HOME
Eshell V5.7.3 (abort with ^G)
1> ts:run(ets).
*** Running [ets_SUITE,default]