Skip to content

Instantly share code, notes, and snippets.

@bnyeggen
Last active May 31, 2021 12:06
Show Gist options
  • Star 8 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save bnyeggen/c679a5ea6a68503ed19f to your computer and use it in GitHub Desktop.
Save bnyeggen/c679a5ea6a68503ed19f to your computer and use it in GitHub Desktop.
Mmap more than 2GB of a file in Java
import java.io.RandomAccessFile;
import java.lang.reflect.Field;
import java.lang.reflect.Method;
import java.nio.channels.FileChannel;
import sun.nio.ch.FileChannelImpl;
import sun.misc.Unsafe;
@SuppressWarnings("restriction")
public class MMapper {
private static final Unsafe unsafe;
private static final Method mmap;
private static final Method unmmap;
private static final int BYTE_ARRAY_OFFSET;
private long addr, size;
private final String loc;
static {
try {
Field singleoneInstanceField = Unsafe.class.getDeclaredField("theUnsafe");
singleoneInstanceField.setAccessible(true);
unsafe = (Unsafe) singleoneInstanceField.get(null);
mmap = getMethod(FileChannelImpl.class, "map0", int.class, long.class, long.class);
unmmap = getMethod(FileChannelImpl.class, "unmap0", long.class, long.class);
BYTE_ARRAY_OFFSET = unsafe.arrayBaseOffset(byte[].class);
} catch (Exception e){
throw new RuntimeException(e);
}
}
//Bundle reflection calls to get access to the given method
private static Method getMethod(Class<?> cls, String name, Class<?>... params) throws Exception {
Method m = cls.getDeclaredMethod(name, params);
m.setAccessible(true);
return m;
}
//Round to next 4096 bytes
private static long roundTo4096(long i) {
return (i + 0xfffL) & ~0xfffL;
}
//Given that the location and size have been set, map that location
//for the given length and set this.addr to the returned offset
private void mapAndSetOffset() throws Exception{
final RandomAccessFile backingFile = new RandomAccessFile(this.loc, "rw");
backingFile.setLength(this.size);
final FileChannel ch = backingFile.getChannel();
this.addr = (long) mmap.invoke(ch, 1, 0L, this.size);
ch.close();
backingFile.close();
}
public MMapper(final String loc, long len) throws Exception {
this.loc = loc;
this.size = roundTo4096(len);
mapAndSetOffset();
}
//Callers should synchronize to avoid calls in the middle of this, but
//it is undesirable to synchronize w/ all access methods.
public void remap(long nLen) throws Exception{
unmmap.invoke(null, addr, this.size);
this.size = roundTo4096(nLen);
mapAndSetOffset();
}
public int getInt(long pos){
return unsafe.getInt(pos + addr);
}
public long getLong(long pos){
return unsafe.getLong(pos + addr);
}
public void putInt(long pos, int val){
unsafe.putInt(pos + addr, val);
}
public void putLong(long pos, long val){
unsafe.putLong(pos + addr, val);
}
//May want to have offset & length within data as well, for both of these
public void getBytes(long pos, byte[] data){
unsafe.copyMemory(null, pos + addr, data, BYTE_ARRAY_OFFSET, data.length);
}
public void setBytes(long pos, byte[] data){
unsafe.copyMemory(data, BYTE_ARRAY_OFFSET, null, pos + addr, data.length);
}
}
@drtimcooper
Copy link

drtimcooper commented Jul 6, 2016

I tried this on my 4.5Gb file. When reading beyond 500Mb, I got this:

A fatal error has been detected by the Java Runtime Environment:

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x000000007220a4c0, pid=13876, tid=14204

JRE version: Java(TM) SE Runtime Environment (7.0_79-b15) (build 1.7.0_79-b15)
Java VM: Java HotSpot(TM) 64-Bit Server VM (24.79-b02 mixed mode windows-amd64 compressed oops)
Problematic frame:
V [jvm.dll+0x22a4c0]

It would be FANTASTIC if you can get this class to work. You can email me on tim@edval.com.au if you think you can solve this problem.

@rednikotin
Copy link

There is an issue with map0 for windows implementation with the following conversion:

mapAddress = MapViewOfFile(
        mapping,             /* Handle of file mapping object */
        mapAccess,           /* Read and write access */
        highOffset,          /* High word of offset */
        lowOffset,           /* Low word of offset */
        (DWORD)len);         /* Number of bytes to map */

But the side effect is that if len is a multiple of 2^32 then the last param (dwNumberOfBytesToMap) is 0 and it works as if you has mapped the whole file.

@lapo-luchini
Copy link

Is unmmap0 unnecessary, or should better a finalize() method be implemented to force unmapping before GC?
(in doubt, I added it in my local copy)

@RockyLOMO
Copy link

cool

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment