Create a gist now

Instantly share code, notes, and snippets.

What would you like to do?
LuaJIT experiments in trying to grab memory outside of the low 32-bits of memory using jemalloc. Looks like jemalloc on Linux is a silver bullet!
#!/usr/bin/env luajit
-- iteratively allocates of `size` bytes using a specified allocator
-- intended to exercise malloc and jemalloc to see where they are allocating memory
-- requires the luajit-jemalloc module
local num_iters = 5
local alloc = 'malloc'
local size = arg[1] and tonumber(arg[1]) or 1000
if arg[2] and arg[2] == 'jemalloc' then alloc = 'jemalloc' end
io.stdout:write(string.format('%s: %d allocs of %d bytes\n',
alloc, num_iters, size
))
if alloc == 'jemalloc' then
local J = require 'jemalloc'
local ptrs = {}
for i = 0, num_iters do
local ptr = J.mallocx( size )
io.stdout:write(tostring(ptr), '\n')
if ptr then ptrs[#ptrs+1] = ptr end
end
for i = 1, #ptrs do
J.dallocx(ptrs[i])
end
else -- malloc
local ffi = require 'ffi'
local C = ffi.C
ffi.cdef([[
void * malloc(size_t size);
void free(void *ptr);
]])
local ptrs = {}
for i = 0, num_iters do
local ptr = C.malloc( size )
io.stdout:write(tostring(ptr), '\n')
if ptr then ptrs[#ptrs+1] = ptr end
end
for i = 1, #ptrs do
C.free(ptrs[i])
end
end

Results on OSX

It looks like both malloc and jemalloc allocate to the low 32-bits of memory. But jemalloc is a little bit better.

I think these two mailing list threads converge on why, but I don't totally understand what's going on....

Using jemalloc 3.5 and Luajit 2.1 alpha on Mavericks on a MacBook Air

./malloc_test.lua 1000 malloc
malloc:  5 allocs of 1000 bytes
cdata<void *>: 0x00103a30
cdata<void *>: 0x00103e20
cdata<void *>: 0x00104210
cdata<void *>: 0x00104600
cdata<void *>: 0x001049f0
cdata<void *>: 0x00104de0
$ DYLD_INSERT_LIBRARIES=/usr/local/Cellar/jemalloc/3.5.0/lib/libjemalloc.1.dylib  ./malloc_test.lua 1000 jemalloc
jemalloc:  5 allocs of 1000 bytes
cdata<void *>: 0x01419400
cdata<void *>: 0x01419800
cdata<void *>: 0x01419c00
cdata<void *>: 0x0141a000
cdata<void *>: 0x0141a400
cdata<void *>: 0x0141a800

Results on Ubuntu Linux

So it looks like jemalloc is a good solution for Linux... malloc is clearly allocating in the low space, but jemalloc, whether used as a malloc replacement or through the non-standard API is allocating memory at high locations.

Using jemalloc 3.5.0 (from chris-lea:redis-server PPA) and Luajit 2.1 alpha in a VM, running Ubuntu 12.04

$ ./malloc_test.lua 1000
malloc:  5 allocs of 1000 bytes
cdata<void *>: 0x00b08010
cdata<void *>: 0x00b08400
cdata<void *>: 0x00b087f0
cdata<void *>: 0x00b08be0
cdata<void *>: 0x00b08fd0
cdata<void *>: 0x00b093c0
LD_PRELOAD=/usr/lib/libjemalloc.so.1 ./malloc_test.lua 1000 jemalloc
jemalloc:  5 allocs of 1000 bytes
cdata<void *>: 0x7f6ca5c18400
cdata<void *>: 0x7f6ca5c18800
cdata<void *>: 0x7f6ca5c18c00
cdata<void *>: 0x7f6ca5c19000
cdata<void *>: 0x7f6ca5c19400
cdata<void *>: 0x7f6ca5c19800
# malloc API, but with jemalloc preloaded
LD_PRELOAD=/usr/lib/libjemalloc.so.1 ./malloc_test.lua 1000
malloc:  5 allocs of 1000 bytes
cdata<void *>: 0x7fd380017400
cdata<void *>: 0x7fd380017800
cdata<void *>: 0x7fd380017c00
cdata<void *>: 0x7fd380018000
cdata<void *>: 0x7fd380018400
cdata<void *>: 0x7fd380018800
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment