gistfile1.irclog

4:55 AM <Lindrian> guys
4:55 AM <Lindrian> anyone, please?
4:55 AM <Lindrian> Ive been asking for help for days >.<
5:19 AM <Lindrian> clb: do you have any idea??
5:20 AM <clb> Lindrian: sorry, no
5:20 AM <Lindrian> clb: if you happen to catch someone who does, could you ask them for me?
5:21 AM <Lindrian> you probably know more people using emscripten than i do
5:22 AM <clb> Lindrian: I've seen you asking for help for weeks now, I think you need to have the capability to research the problem field yourself and learn the required knowledge if there is nobody else that is familiar with the issue
5:23 AM <Lindrian>  i have been searching extensively
5:23 AM <Lindrian> i have asked countless people
5:23 AM <Lindrian> nobody knows of a good way to go python -> c/llvm
11:14 AM <Vito``> Lindrian: maybe there isn't one
11:14 AM <Vito``> Lindrian: it's entirely possible you need to solve your problem in a different way
11:14 AM <Vito``> Lindrian: and/or invent something new
11:16 AM <Lindrian> linkmauve said he has done it, but i havent been able to get in touch with him in a while
11:16 AM <Lindrian> I've found numba which I am reading up on
11:17 AM <Vito``> Lindrian: I assume my edited answer didn't work, then
11:18 AM <Lindrian> not sure how it would help
11:18 AM <Lindrian> i tried using --embed but that didnt seem to help
11:18 AM <Lindrian> ive also looked at pyjs
11:20 AM <Vito``> Lindrian: so "public" didn't help, as in this answer? http://stackoverflow.com/a/14280526/117088
11:20 AM <Lindrian> oh darn i missed that
11:20 AM <Lindrian> hold on
11:22 AM <Lindrian> my function name seems OK now, but compiling resulted in >9999 errors
11:22 AM <Lindrian> AssertionError: Error in js optimizer: 
11:23 AM <Lindrian> thanks Vito``, think im one step closer now
11:24 AM <Vito``> that was my edited answer
11:25 AM <Vito``> I haven't found anything else
11:25 AM <Vito``> do update SO with your solution, whatever you come up with
11:25 AM <Lindrian> right
11:25 AM <Lindrian> now i seem to be getting some other weird ass errors from emscripten
12:00 PM <Lindrian> Vito``: do you know how an emcc call should look when compiling a python source?
12:01 PM <Vito``> no, I haven't personally tried it
12:01 PM <Vito``> what does your emcc command look like?
12:01 PM <Vito``> and what errors are you getting?
12:01 PM <Lindrian> emcc -I./Include regex101_cython.c libpython2.7.so -O2 -s EXPORTED_FUNCTIONS="['_regex101_match']"
12:02 PM <Lindrian> forgot an -I i think..
12:05 PM <Lindrian> woot, i can see my regex function in a.out.js now
12:05 PM <Lindrian> magic :o
1:26 PM <Lindrian> In emscripten when I try to run my function I get: missing function: PyList_New 
1:26 PM <Lindrian> What did I do wrong? I compiled my file using the following command after converting it using cython: ~/emscripten/emcc -I./Include regex101_cython.c -Ilibpython2.7.so -O2 -s EXPORTED_FUNCTIONS="['_regex101_match']"

1:58 PM <azakai> Lindrian: either the function was not in the bitcode you gave to it - can use llvm-nm to see what is in there - or llvm dce removed it, might need to add it to EXPORTED_FUNCTIONS. also, probably you want a main in there?
1:58 PM <Lindrian> why do I need a main?
1:58 PM <azakai> well, what is the entry point to the code? regex101_match?
1:58 PM <Lindrian> yeah
1:58 PM <azakai> anything not reachable by that will be dce'd
1:58 PM <azakai> ok
1:59 PM <Lindrian> i use lists in that function
1:59 PM <Lindrian> to keep track of things
2:00 PM <Lindrian> im trying adding a Lib folder that i embed
2:03 PM <Lindrian> azakai: have you ever compiled python into js using emscripten
2:03 PM <Lindrian> ?
2:04 PM <azakai> Lindrian: the test suite includes a build of the python interpreter, see test_python
2:04 PM <azakai> that's what i did
2:05 PM <Lindrian> cant find test_python
2:05 PM <azakai> grep for test_python in tests/
2:06 PM <Lindrian> i see the python folder, no test in it though
2:06 PM <Lindrian> looking at github
2:08 PM <azakai> look at the test in tests/test_core.py called test_python
2:08 PM <azakai> and in tests/python/ there is the bitcode and all the instructions
2:08 PM <Lindrian> right
2:08 PM <Lindrian> im still getting the error
2:08 PM <Lindrian> I included List.py
2:08 PM <Lindrian> didnt fix the problem though
2:09 PM <Lindrian> missing function: PyList_New 
2:09 PM <azakai> did llvm-nm help?
2:09 PM <Lindrian> where do I run that?
2:09 PM <azakai> like normal nm. run it on the bitcode you pass into emcc.
2:09 PM <azakai> that will verify if is it in there
2:10 PM <Lindrian> T PyList_New
2:11 PM <Lindrian> seems like its there
2:11 PM <Lindrian> don't know that 'T' means
2:12 PM <azakai> run llvm-nm --help or nm --help
2:12 PM <azakai> i never remember what those mean either
2:13 PM <Lindrian> but it seems like its in the .so file
2:13 PM <Lindrian> i wonder why its getting dce'd
2:13 PM <azakai> then llvm dce must remove it. try EMCC_DEBUG=1 in the env when building, then you will get the bitcode files before and after dce
2:15 PM <Lindrian> hmm i dont see any other bitcode files?
2:15 PM <azakai> in /tmp/emscripten_temp
2:15 PM <azakai> that's where EMCC_DEBUG stores files
2:16 PM <Lindrian> sorry if im being a bother azakai 
2:16 PM <azakai> np
2:17 PM <Lindrian> i dont have such a folder
2:18 PM <Lindrian> i tried exported_functions but that did not work either
2:19 PM <Lindrian> still says its missing that function
2:20 PM <Lindrian> azakai: do you need to convert the python code to C using cython?
2:21 PM <Lindrian> because thats what im currently doing
2:21 PM <azakai> i never tried cython
2:21 PM <azakai> EMCC_DEBUG=1 ./emcc ...         should show lots of console output, and generate /tmp/emscripten_temp
2:21 PM <Lindrian> how did you go from .py to .js?
2:21 PM <azakai> i just built the python C interpreter and ran python inside that
2:22 PM <Lindrian> I have the python C interpreter in a .so file
2:24 PM <Lindrian> azakai: how else would I write my code?
2:24 PM <Lindrian> I thought you write python, convert using cython and compile using emcc
2:24 PM <azakai> no, i never used cython in my life
2:24 PM <Lindrian> so how do you compile a .py file?
2:24 PM <azakai> i just ported  the python c interpreter as i said above
2:24 PM <azakai> i never compiled .py files
2:24 PM <azakai> i let the compiled interpreter run them
2:24 PM <Lindrian> ah okay
2:24 PM <Lindrian> oh
2:25 PM <Lindrian> so you convert the entire interpreter to JS and then feed it the .py file?
2:25 PM <azakai> Lindrian: yes
2:25 PM <Lindrian> i see
2:25 PM <Lindrian> not viable for me though :(
2:25 PM <azakai> why?
2:26 PM <Lindrian> entire python interpreter would be huge
2:26 PM <Lindrian> i just want to access the regex module in python, nothing else
2:26 PM <azakai> using it from js?
2:26 PM <Lindrian> correct
2:26 PM <azakai> then why compile any .py files?
2:27 PM <azakai> make a C api to python regex, make sure it works natively, then port that to js
2:27 PM <Lindrian> I was told it would be easier to make a light wrapper
2:28 PM <azakai> in python?
2:28 PM <Lindrian> yeah
2:28 PM <azakai> i don't know anything about the internals. but surely a little c hacking can get you what you want there
2:29 PM <Lindrian> thats probably true...
2:29 PM <Lindrian> i just have no idea how to work with python in c
2:29 PM <azakai> i don't know how stable cython is
2:29 PM <azakai> doesn't it support just a subset of python?
2:30 PM <Lindrian> maybe
2:30 PM <azakai> did you get a native build of cython'd code + python regex to work?
2:31 PM <azakai> if not, i would start with that
2:31 PM <Lindrian> yeah the code works
2:31 PM <Lindrian> thats not the problem
2:31 PM <azakai> linking against libpython?
2:31 PM <Lindrian> yes yes
2:31 PM <Lindrian> it compiles just fine
2:31 PM <azakai> then that means you need to port all of libpython to js
2:31 PM <azakai> so you are not avoiding compiling the interpreter
2:32 PM <Lindrian> problem is  PyList_New is outside my function
2:32 PM <Lindrian> and thus I think its being removed
2:32 PM <azakai> well, try to just add it to EXPORTED_FUNCTIONS manually. or just build with -s LINKABLE=1 which avoids dce
2:32 PM <azakai> but those are not recommended approaches
2:32 PM <Lindrian> i tried exported functions, did not work
2:32 PM <azakai> then try -s LINKABLE=1
2:33 PM <Lindrian> new error : Uncaught TypeError: Property '_PyList_New' of object [object Object] is not a function
2:34 PM <azakai> compile error? run error?
2:34 PM <Lindrian> runtime error
2:35 PM <Lindrian> happens when i call my functions
2:35 PM <Lindrian> function*
2:35 PM <azakai> look in the source for those functions
2:35 PM <Lindrian> okay, what else?
2:36 PM <azakai> is that with LINKABLE?
2:36 PM <Lindrian> yep
2:36 PM <azakai> what is in the source for that funciton, that isn't a function? what causes that error?
2:38 PM <Lindrian> want to see my python code?
2:38 PM <azakai> sure
2:39 PM <Lindrian> http://pastebin.com/xaJ9wgdi
2:40 PM <azakai> i mean, the compiled code that gives that error
2:41 PM <azakai> standalone js file i can test in the shell would be best
2:41 PM <Lindrian> okay
2:41 PM <Lindrian> where does one pastebin 5k lines?
2:41 PM <azakai> perhaps dropbox or such
2:42 PM <Lindrian> https://gist.github.com/firasdib/64a32b87278f4ac2922e
2:42 PM <Lindrian> this seems to work
2:48 PM <Lindrian> see the error azakai ?
2:49 PM <azakai> Lindrian: yes. did you verify with llvm-nm that that function is in the input bitcode? and verify using EMCC_DEBUG=1 on a build with -s LINKABLE=1 that it is not dce'd?
2:51 PM <Lindrian> the llvm-nm shows that function is present
2:51 PM <Lindrian> im unable to get EMCC_DEBUG to work
2:51 PM <Lindrian> it doesnt output anything extra at all in console
2:51 PM <Lindrian> and no temp folder is made
2:51 PM <azakai> why not? show me the output you get for EMCC_DEBUG=1 ./emcc tests/hello_world.c
2:52 PM <Lindrian> huh, i get a bunch of output then
2:52 PM <Lindrian> EMCC_DEBUG=1 sudo ~/emscripten/emcc -I./Include regex101_cython.c -Ilibpython2.7.so -O2 -s EXPORTED_FUNCTIONS="['_regex101_match']" -s LINKABLE=1
2:52 PM <Lindrian> no output
2:55 PM <Lindrian> something wrong with that command?
2:55 PM <Lindrian> ah i see
2:55 PM <Lindrian> removing the sudo worked
2:56 PM <Lindrian> azakai: http://pastebin.com/kMGpvsFG
2:56 PM <azakai> you need the sudo before it perhaps
2:56 PM <Lindrian> yeah
2:56 PM <azakai> now do that and look at the bitcode files in the temp dir
2:56 PM <Lindrian> which one?
2:56 PM <azakai> see if your func is removed
2:57 PM <Lindrian> two .bc, one .ll
2:57 PM <azakai> the numbers say the order
2:57 PM <azakai> first bc is the input, second is after dce
2:57 PM <azakai> verify using nm that the func is in both
2:57 PM <Lindrian> ok
2:57 PM <azakai> then with grep that it is in the .ll
2:58 PM <Lindrian> they both have PyList_New
2:58 PM <azakai> and the .ll?
2:58 PM <azakai> Lindrian: send me all the emcc-* files
2:59 PM <Lindrian> llvm-nm: emcc-2-ll.ll: unrecognizable file type
2:59 PM <Lindrian> alright
2:59 PM <azakai> for the nm, just grep, not nm
3:01 PM <Lindrian> azakai: https://www.dropbox.com/sh/0qnsb192sayl8xn/E0yR6pkLLX
3:03 PM <azakai> Lindrian: s$ ~/Dev/clang+llvm-3.2-x86-linux-ubuntu-12.04/bin/llvm-nm emcc-0-basebc.bc  | grep PyList
3:03 PM <azakai>          U PyList_Append
3:03 PM <azakai>          U PyList_New
3:03 PM <azakai>          U PyList_Type
3:03 PM <azakai>          t __Pyx_PyList_Append
3:03 PM <azakai> U means undefined
3:03 PM <azakai> the bitcode input it receives does not contain that function
3:04 PM <azakai> if it *is* in the input to emcc, then linking removed it. linking of .a files can do that. see the faq entry on dead code
3:05 PM <Lindrian> its .so though
3:05 PM <azakai> then send me the .so files you are sending into emcc
3:05 PM <azakai> and i will try locally
3:05 PM <azakai> also the c files and everything else I need to build locally using your command from before
3:06 PM <Lindrian> aight buddy
3:07 PM <azakai> off to lunch, bbl
3:08 PM <Lindrian> azakai: https://www.dropbox.com/sh/0qnsb192sayl8xn/E0yR6pkLLX
3:08 PM <Lindrian> damnit, he just left
3:56 PM <Lindrian> poke me when you're back azakai :)

4:25 PM <azakai> Lindrian: your command does -Ilibpython etc., that is wrong. it needs to just be libpython.so
4:25 PM <Lindrian> ah okay
4:25 PM <azakai> Lindrian: so you were not linking libpython at all
4:25 PM <Lindrian> ill try again
4:27 PM <azakai> building that locally, it errors on having inline asm. you need to build it so it does not use inline x86 asm
4:27 PM <Lindrian> im not getting any errors though
4:27 PM <Lindrian> how do I rebuild without inline asm?
4:27 PM <azakai> i don't know, depends what flags python uses to enable that
4:27 PM <azakai> perhaps a configure flag
4:28 PM <Lindrian> oh boy
4:28 PM <Lindrian> i get so many errors now
4:28 PM <Lindrian> failed to compile entirely
4:29 PM <Lindrian> azakai: http://pastebin.com/qBMYLdf9
4:29 PM <azakai> that is the inline asm stuff. to see it, try to run the first js file in /tmp/emscripten_temp.
4:29 PM <azakai> it contains invalid js, which the js optimizer chokes on
4:29 PM <azakai> hence the parse error
4:29 PM <Lindrian> aha
4:30 PM <Lindrian> i followed your guide azakai 
4:30 PM <Lindrian> under /python/ on github
4:30 PM <azakai> perhaps the way you build, different code ends up used
4:31 PM <Lindrian> try it with your .so lib?
4:32 PM <azakai> worth trying
4:32 PM <azakai> anyhow, need to fix it during python compile
4:44 PM <azakai> Lindrian: i don't have time to try, sorry. but just look in the source code for asm() and find out which build flags trigger that
4:45 PM <Lindrian> source code for asm()? where do I find that?
4:45 PM <azakai> grep for asm
4:46 PM <Lindrian> where?
4:46 PM <Lindrian> in what file?
4:46 PM <azakai> all the python c files
4:56 PM <Lindrian> tons of matches azakai :(
4:56 PM <azakai> but probably just one or two flags to disable them all
4:56 PM <Lindrian> #define READ_TIMESTAMP(val) \
4:56 PM <Lindrian>     __asm__ __volatile__("rdtsc" : \
4:56 PM <Lindrian>                          "=a" (((int*)&(val))[0]), "=d" (((int*)&(val))[1]));
5:33 PM <Lindrian> what the hell
5:33 PM <Lindrian> im trying to follow the guide for python again and i keep getting Makefile:94: *** missing separator.  Stop.
5:35 PM <Lindrian> Any idea azakai ?
5:43 PM <Lindrian> regular configure + make works fine
5:43 PM <Lindrian> must be something weird with emscripten
5:50 PM <azakai> no idea, but if make gives an error, problem must be in the makefile
5:56 PM <Lindrian> azakai: i am literally following the instructions to the letter
5:57 PM <Lindrian> getting the same bloody error everytime
5:57 PM <Lindrian> ive tried maybe 15 times now
5:57 PM <azakai> i have no idea what could cause that. i think you just need to debug into the build system to see what the problem is
5:58 PM <Lindrian> gahh
5:58 PM <Lindrian> i have no clueeeeeeeee
5:59 PM <azakai> Lindrian: perhaps ask the python people if the problem is building python itself
6:02 PM <Lindrian> azakai: I did, its not
6:03 PM <azakai> Lindrian: doesn't look like an emscripten problem either if the makefile says "missing separator"
6:04 PM <Lindrian> ERROR    root: Executable to run not specified.
6:04 PM <Lindrian> thats what emscripten says
6:05 PM <azakai> just that?
6:05 PM <Lindrian> thats it
6:05 PM <Lindrian> just running emmake
6:05 PM <azakai> no idea what that is. you need to debug into it a little. grep for the error message, put some debug printouts. or reduce to a testcase and file an issue and i can take a look later
6:05 PM <Lindrian> i had the same problem a while back
6:05 PM <Lindrian> dont remember how i fixed it
6:09 PM <Lindrian> azakai: how do I add emscripten functions to path?
6:09 PM <Lindrian> so i dont have to refer to the directory all the time
6:09 PM <azakai> the same way you add anything to your system path, depends on your os
6:10 PM <Lindrian> ubuntu
6:10 PM <azakai> i don't know offhand. but a search should turn that up easily
6:10 PM <Lindrian> alright
6:14 PM <Lindrian> azakai: okay this is killing me
6:15 PM <Lindrian> azakai: ./configure and make works fine, emconfigure + emmake does not work. emconfigure to make does not work.
6:15 PM <Lindrian> emconfigure + make does not work***
6:17 PM <azakai> the main difference is that emconfigure/emmake make it use the emscripten system headers, not your local system's stuff
6:17 PM <azakai> but i don't see how that can cause errors like you mentioned
6:20 PM <Lindrian> i have no idea man
6:20 PM <Lindrian> im getting really pissed now
6:21 PM <azakai> take a break, might be something simple when you come back to it later
6:21 PM <azakai> helps me when i get annoyed at stuff like that
6:23 PM <Lindrian> intersting
6:23 PM <Lindrian> running _just_ emconfigure ./configure and then make works
6:26 PM <Lindrian> adding EMCONFIGURE_JS=1 breaks it
6:26 PM <Lindrian> any idea azakai ?
6:27 PM <azakai> EMCONFIGURE_JS is experimental, and off by default for that reason. not expected to work
6:27 PM <Lindrian> its in the guide man
6:27 PM <Lindrian> you should remove it
6:27 PM <azakai> perhaps it works for python then
6:27 PM <Lindrian> works now
6:27 PM <azakai> it worked when the guide was written
6:27 PM <Lindrian> -.-
6:27 PM <Lindrian> okay
6:27 PM <azakai> might want to use the revision when the guide was written, that could be safest
6:30 PM <Lindrian> im using head i beleive
6:30 PM <Lindrian> believe*
6:39 PM <Lindrian> yay
6:39 PM <Lindrian> i finally got it to work
6:39 PM <Lindrian> but my output file is 3.5mb
6:40 PM <Lindrian> doesnt look like there was any compression made on it though
6:40 PM <Lindrian> hmm
6:42 PM <Lindrian> file got a tiny bit smaller with -O3, but nothing worth mentioning
6:42 PM <Lindrian> file is till 3.3mb
6:43 PM <Lindrian> how on earth did this thing end up so massive
6:59 PM <Lindrian> now uh, my function takes PyObjects, how do I create those in javascript??