Skip to content

Instantly share code, notes, and snippets.

@rampage644

rampage644/osv.md

Last active August 29, 2015 14:07
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save rampage644/60d31e3f5c53d22cbfe5 to your computer and use it in GitHub Desktop.
Save rampage644/60d31e3f5c53d22cbfe5 to your computer and use it in GitHub Desktop.
Download ZIP
Osv
Raw
osv.md

OSv + Impala status

  1. I think i get plan-fragment-executor-test run under OSv
  2. But it fails very quickly
  3. Problem is with tcmallocstatic. First, OSv doesn't support sbrk-based memory management. One has to tune tcmallocstatic not to use SbrkMemoryAllocator at all (comment #undef HAVE_SBRK in config.h.in). Second, it still fails with invalid opcode exception.

Issues

tcmallocstatic

  1. Dig into TLS and tls-relocations
  2. Link OSv TLS issues with this issues
  3. Test with simple example
  4. Dig into invalid opcode exception

I fixed relocation problem with this patch (it has been accepted and should be in upstream):

 arch/x64/arch-elf.cc | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/arch/x64/arch-elf.cc b/arch/x64/arch-elf.cc
index 473f935..86ef2de 100644
--- a/arch/x64/arch-elf.cc
+++ b/arch/x64/arch-elf.cc
@@ -84,7 +84,10 @@ bool object::arch_relocate_rela(u32 type, u32 sym, void *addr,
         *static_cast<u64*>(addr) = symbol(sym).symbol->st_value;
         break;
     case R_X86_64_TPOFF64:
-        *static_cast<u64*>(addr) = symbol(sym).symbol->st_value - get_tls_size();
+        if (sym)
+            *static_cast<u64*>(addr) = symbol(sym).symbol->st_value - get_tls_size();
+        else
+            *static_cast<void**>(addr) = _base + addend - get_tls_size();
         break;
     default:
         return false;

Upd: It didn't actually fixes the problem. It really should abort. (see https://groups.google.com/forum/#!topic/osv-dev/uSL7uo2Tn0c). It could be fixed with gperftools modification (comment out HAVE_TLS in config.h.in)

set_current_thread_data

  1. Check where symbol loader looks for this symbol
  2. Check for conflicting boost versions (OSv definitely includes boost symbols, but not that one)
  3. Recompile impala with newer boost Didn't help

I found out that this symbol is resolved at runtime (it has been looking for during resolve_pltgot procedure, so it should be lazy linking, thanks to Yaroslav). Compiling plan-fragment-executor-test with -Wl,-z,now option which tells to resolve all symbols at start time with no deferred symbol resolution helps a little. Now there are others symbols unresolved.

This issue is still not resolved. I use workaround (-z now) not to stop with it. Didn't get useful feedback from OSv forums.

stubbed symbols

I didn't try to subьit this patches to OSv.

  • mallinfo
  • pthread_attr_setschedpolicy/pthread_attr_getschedpolicy
  • pthread_attr_setschedparam/pthread_attr_getschedparam
  • pthread_mutexattr_setpshared
  • syslog
  • clone

There is also __sbrk symbol which i fixed at musl repo with this patch:

diff --git a/src/linux/sbrk.c b/src/linux/sbrk.c
index 3643765..43f0593 100644
--- a/src/linux/sbrk.c
+++ b/src/linux/sbrk.c
@@ -1,5 +1,6 @@
 #include <stdint.h>
 #include "syscall.h"
+#include "libc.h"
 
 void *sbrk(intptr_t inc)
 {
@@ -7,3 +8,5 @@ void *sbrk(intptr_t inc)
        if (inc && syscall(SYS_brk, cur+inc) != cur+inc) return (void *)-1;
        return (void *)cur;
 }
+
+weak_alias(sbrk, __sbrk);
\ No newline at end of file
diff --git a/src/regex/tre.h b/src/regex/tre.h
index 67cb9a8..9bfb63f 100644
--- a/src/regex/tre.h
+++ b/src/regex/tre.h
@@ -35,7 +35,9 @@
 
 #undef  TRE_MBSTATE
 
+#ifndef NDEBUG
 #define NDEBUG
+#endif
 
 #define TRE_REGEX_T_FIELD __opaque
 typedef int reg_errcode_t;

Patch for other symbols:

diff --git a/include/api/malloc.h b/include/api/malloc.h
index e69de29..a4f6537 100644
--- a/include/api/malloc.h
+++ b/include/api/malloc.h
@@ -0,0 +1,22 @@
+#ifdef __cplusplus
+extern "C" {
+#endif
+  
+struct mallinfo mallinfo (void);
+
+struct mallinfo {
+    int arena;     /* Non-mmapped space allocated (bytes) */
+    int ordblks;   /* Number of free chunks */
+    int smblks;    /* Number of free fastbin blocks */
+    int hblks;     /* Number of mmapped regions */
+    int hblkhd;    /* Space allocated in mmapped regions (bytes) */
+    int usmblks;   /* Maximum total allocated space (bytes) */
+    int fsmblks;   /* Space in freed fastbin blocks (bytes) */
+    int uordblks;  /* Total allocated space (bytes) */
+    int fordblks;  /* Total free space (bytes) */
+    int keepcost;  /* Top-most, releasable space (bytes) */
+};
+
+#ifdef __cplusplus
+}
+#endif
diff --git a/libc/build.mk b/libc/build.mk
index a04bfca..a5c646b 100644
--- a/libc/build.mk
+++ b/libc/build.mk
@@ -65,6 +65,8 @@ musl += ctype/__ctype_b_loc.o
 
 libc += errno/strerror.o
 
+musl += linux/sbrk.o
+
 musl += locale/catclose.o
 musl += locale/catgets.o
 musl += locale/catopen.o
@@ -428,8 +430,14 @@ libc += process/execve.o
 libc += process/execle.o
 musl += process/execv.o
 musl += process/execl.o
+musl += process/wait.o
 libc += process/waitpid.o
 
+musl += regex/regcomp.o
+musl += regex/regexec.o
+musl += regex/regerror.o
+musl += regex/tre-mem.o
+
 libc += arch/$(arch)/setjmp/setjmp.o
 libc += arch/$(arch)/setjmp/longjmp.o
 libc += arch/$(arch)/setjmp/sigrtmax.o
diff --git a/libc/pthread.cc b/libc/pthread.cc
index 3e31591..e1bf124 100644
--- a/libc/pthread.cc
+++ b/libc/pthread.cc
@@ -976,3 +976,32 @@ int pthread_attr_getaffinity_np(const pthread_attr_t *attr, size_t cpusetsize,
 
     return 0;
 }
+
+int pthread_attr_setschedpolicy(pthread_attr_t *attr, int policy)
+{
+    WARN_STUBBED();
+    return EINVAL;
+}
+int pthread_attr_getschedpolicy(const pthread_attr_t *attr, int *policy)
+{
+    WARN_STUBBED();
+    return EINVAL;
+}
+
+int pthread_attr_setschedparam(pthread_attr_t *attr,
+                               const struct sched_param *param)
+{
+    WARN_STUBBED();
+    return EINVAL;
+}
+int pthread_attr_getschedparam(const pthread_attr_t *attr,
+                               struct sched_param *param)
+{
+    WARN_STUBBED();
+    return EINVAL;
+}
+int pthread_mutexattr_setpshared(pthread_mutexattr_t *, int)
+{
+    WARN_STUBBED();
+    return EINVAL;
+}
\ No newline at end of file
diff --git a/libc/syslog.c b/libc/syslog.c
index be7cfb6..b27ddeb 100644
--- a/libc/syslog.c
+++ b/libc/syslog.c
@@ -71,3 +71,10 @@ void __syslog_chk(int priority, int flag, const char *message, ...)
     UNLOCK(lock);
 }
 
+void syslog(int priority, const char *message, ...)
+{
+    va_list ap;
+    va_start(ap, message);
+    __syslog_chk(priority, 0, message, ap);
+    va_end(ap);
+}
diff --git a/runtime.cc b/runtime.cc
index 5d58e82..c2b9d9a 100644
--- a/runtime.cc
+++ b/runtime.cc
@@ -30,6 +30,7 @@
 #include <stdarg.h>
 #include <xlocale.h>
 #include <cassert>
+#include <malloc.h>
 #include <sys/sysinfo.h>
 #include "processor.hh"
 #include <osv/debug.hh>
@@ -77,6 +78,10 @@ extern "C" {
     int mallopt(int param, int value);
     FILE *popen(const char *command, const char *type);
     int pclose(FILE *stream);
+    int clone(int (*fn)(void *), void *child_stack,
+                 int flags, void *arg, ...
+                 /* pid_t *ptid, struct user_desc *tls, pid_t *ctid */ );
+    struct mallinfo mallinfo(void);
 }
 
 void *__dso_handle;
@@ -211,6 +216,21 @@ int fork()
     return -1;
 }
 
+int clone(int (*fn)(void *), void *child_stack,
+                 int flags, void *arg, ...
+                 /* pid_t *ptid, struct user_desc *tls, pid_t *ctid */ )
+{
+    WARN_STUBBED();
+    return -1;
+}
+
+struct mallinfo mallinfo(void)
+{
+    WARN_STUBBED();
+    struct mallinfo st;
+    return st;
+}
+
 pid_t setsid(void)
 {
     WARN_STUBBED();

Wiki

ssh connection

If you're using VirualBox and VMWare images make sure you have set "host" network adapter. Then just connect as usual to 22 port.

If your're using local kvm/qemu instances with no extnernal networking connect to 2222 port as manual says. Use external networking to connect to 22 port as usual.

stubbing functions

Look for *.mk thorugh OSv source tree. Search for libc/build.mk. It defines libc target and assigns files to it. Either hack one of appropriate file or add your own.

Debugging techniques

  1. Look here.
  2. And here.
  3. When you encounter OSv fail and see backtrace remember that OSv hangs waiting you to connect with gdb and debug it.
  4. use mode=debug when building and -d switch when running OSv.
  5. OSv could be run with --verbose switch.
  6. Remember to use tracing when needed! (Read about it). Remember about -d switch if uisng debug version.

When you need to stop on breakpoint connect somehow to OSv, set all breakpoints and them restart VM with gdb command monitor system_reset.

How to upload files directly to image

cd build/release.x64
../../scripts/upload_manifest.py -o usr.img -m ../../impala.manifest

Because upload_manifest.py does cd ../.. inside. Use -Dkey=value to substitute %(key)s occurences in manifest.

Example of manifest file:

[manifest]
/usr/lib/plan-fragment-executor-test.so: /home/ramp/git/impala-cut/be/build/debug/runtime/plan-fragment-executor-test

/usr/lib/libboost_thread.so.1.54.0: %(miscbin)s/usr/lib64/libboost_thread.so.1.56.0

Upload like this:

cd build/release
../../scripts/upload_manifest.py -o usr.img -m ../../impala.manifest -Dmiscbin=/home/ramp/git/osv/external/misc.bin
cd ../..
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment