CMCDragonkai/nix_shell_unshare_nsenter_straced_pstree_lsof_fuser_cgroups_gdb_setsid.md Secret

## nix_shell_unshare_nsenter_straced_pstree_lsof_fuser_cgroups_gdb_setsid.md

      
    Raw
  

              nix_shell_unshare_nsenter_straced_pstree_lsof_fuser_cgroups_gdb_setsid.md
            
          
    nix-shell, unshare, nsenter, strace, pstree, lsof, fuser, cgroups, gdb, setsid

Find some way to combine all these tools together to show a superpowered debuggable development environment.
Note that unshare is really useful for creating a namespace for a nix-shell.
Where nix-shell gives you the environment variables.
unshare gives you the namespace.
If you want to re-enter the same namespace, probably nsenter with nix-shell may be useful.
Debugging the user-kernel boundary would be a matter of strace.
Using pstree gives you an idea of the process hierarchy within the nix-shell!
# this will show the process and thread tree rooted at 15688
pstree --show-pids --show-pgids --long 15688
# this will get all processes and threads that is part of the process group 15688
pgrep --pgroup 15688 --lightweight
# this straces all the processes that is part of the group 15688
# using -ff will automatically trace child threads as well
# and every thread will have its trace saved as trace.* where * is the PID/TID
strace -p $(pgrep --pgroup 15688 | paste --serial --delimiter=',') -ff -o trace
# this will show the file descriptor 47 in the process 15719 (lsof doesn't work with thread ids)
lsof -p 15719 -d 47
# ps doesn't allow selection to a thread id directly, you have to select by processes then use -T or -L to show threads
# we can use fuser to show what has something open
fuser --verbose /dev/pts/1

Also cpuset! Is probably useful for testing multiprocessing behaviour.
While unshare will put you into a namespace, for actually controlling computational resources like CPU, Memory.. etc. You need to use cgroups instead. Things like systemd and docker already manage cgroups for their respective services/containers. But if you are just using nix-shell interactively, you need to bring this together yourself, it gives you more power and flexibility at the cost of increased complexity (due to the depth of each dimensional axis of configuration space).
The way to solve such complexity and preserve functionality is through abtraction and functional composition. Whereas Docker and Systemd represents their walled-garden approach of encapsulation of managing this complexity.
The usage of gdb is complicated when the target is inside a container as debugging symbols cannot be accessed as the filesystem path is resolved within a chroot environment.
Note that alot of these tools comes from util-linux package.
Another cool tool is lsns which shows you all the namespaces available.
It sort of makes sense to build up a tool that can declaratively use all of these.
NixOS containers relies on the OS being NixOS. But Nix itself could be used to declare all of these.
But the problem is that the because we are not "building" a package, you need a separate tool to understand the declaration and create the necessary container.
Kind of like nix-shell --container or nix-container to do such a thing.
Another thing is that once we have containers we want to achieve resource isolation. But that is only for production.
While developing, we want to know how much resources things will grow to. This means we want to be able to monitor the processes/namespace that we are working against.
Consider tools like:

smem - to show memory usage (the difference between USS, PSS and RSS is very interesting)
benchmarking tools
profiling tools (is there a general profiling tool on a process that records the entire run characteristics?)
rr for record replay

The idea is to capture a "general profile" of any process or program as you run it. It should be able to work against a new process like: profile program. Or it can attach to an existing program like profile -p program.
Just like strace and gdb.
Once that is done, it becomes more possible to have better resource usage estimations for production.
Note that when running in a container, can pages be shared between different libraries across containers?
At first this seems like no. But if the container is built from the same layer, this may be true.
At the same time, you'd want to eventually be able to mount the container the /nix/store so that everybody shares
the same /nix/store. But that would be the same as a nix-shell namespace rather than a Docker container which also packages the dependencies statically. I'm not sure about layer sharing and whether layer sharing allows memory sharing.
https://ogavrisevs.github.io/2016/07/11/isolation-in-docker/
For nix-shell it sort of makes sense to do enableDebugging on your derivation. So that the compiled executables will produce the debug output. But that doesn't really work cause the nix-shell doesn't actually build your derivation. Instead you would eventually want to use nix-build and specifically apply the debugging parameter to it. That should be possible with -I and using the -p.
For interpreted environments, you want to do enableDebugging on the interpreter. This is debugging the interpreter not the code you wrote yourself. Except for Python which has the python extensions for gdb.
For compiled code, it's still possible to use gdb but they may need some work though.
See supported languages: https://sourceware.org/gdb/onlinedocs/gdb/Supported-Languages.html#Supported-Languages

https://github.com/snare/voltron
http://www.well-typed.com/blog/96/
https://web.archive.org/web/20180531125031/http://grapsus.net/blog/post/Low-level-Python-debugging-with-GDB


The only way to do this is to use docker exec which does ns-enter into the namespace, and runs a second process, where in this case its the gdb inside the container. This allows gdb inside the container to resolve the files. And potentially access debugging symbols there. Although even though I don't have any compiled with debugging symbols, I can still see some high level information though. Note that doing so uses a different PID inside the container. This requires the container to have gdb installed. You are basically creating a "debuggable container image". However another way is to have gdb installed somewhere else and then do volumes from mount. But this won't with the way Nix is setup with absolute paths. Damn.
If you want to be able to use gdb inside the container, that container requires higher privileges. You cannot give it higher privileges without restarting the container.

"warning: Target and debugger are in different PID namespaces; thread lists and other data are likely unreliable"

https://blog.wnohang.net/index.php/2015/05/05/debugging-docker-containers-with-gdb-and-nsenter/
So this shows us a better way is to use nsenter with a container that isn't privileged. And you don't need to restart. You do need gdb installed (or gdbserver) installed inside the container. Then use nsenter to get get gdb running.
There is branch of gdb with namespace support apparently.
To debug a nix build:

Use nix-build ... -K. Which will store the output of a build in the /tmp/.... Here you can see env-vars where you can create another shell that sources it and try out the commands. However you also have to assume the nixbld user to do it. This is sort of really low level. A better solution is 2.
Use nix-shell ....drv, get the failed derivation and run nix shell against it. This will run derivation but with everything "reified" (unlike using a Nix expression). Then you're in a properly setup environment. However to get the relevant files you have to execute the build stages like genericBuild... etc. Check if there's a echo "$checkPhase" and echo "$installCheckPhase". This is pretty useful.


The setsid program can create a new session. A new session corresponds to a new "tty" unlike a process group. Not sure what this could be used for.