Skip to content

Instantly share code, notes, and snippets.

@simonhf

simonhf/_libarchive-read-blocking-part-4.md

Created November 9, 2019 02:37
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save simonhf/a14903378e2318e0a6c773454f0cf91e to your computer and use it in GitHub Desktop.
Save simonhf/a14903378e2318e0a6c773454f0cf91e to your computer and use it in GitHub Desktop.
Download ZIP
Raw
_libarchive-read-blocking-part-4.md

Experiments with libarchive read blocking: Part 4

  • Note: This is a continuation from Part 3 [1].

[1] Experiments with libarchive read blocking: Part 3

How to figure out how much stack space the libarchive example3.c code needs?

  • Modify cwrap.pl to report the stack pointer on every log line.
  • Modify ./doit.sh to run example3.c with two archives, and show the SHA256 sums of the extracted files.
  • Rebuild example3.c and run via ./doit.sh.
  • We can see that the largest amount of stack space used is about 1,280 bytes.
  • At the deepest stack, the indentation of the log line shows that the call-tree is about 13 functions deep. Although obviously each function uses a variable amount of stack space during its invocation.
  • The deepest stack also happens to be one of the callback invocations for each archive file.
  • We can also see that at each callback the stack used varies a lot. So something to keep in mind: A more complex archive might increase the stack size? Something to investigate...
  • Note: The cwrap instrumentation itself is responsible for part of the stack size used, because the function depth doubles.
$ ./doit.sh
...
11d4fec99890b26d8ee8ebb9710ff36b665c5fb4653d43de2ea86a3d52dc7b3b  ./.libs/libarchive_test_1a.a
295ef327b417e4118d0329ef707d360b0bc40ce12be336d2ce006422dd593429  ./.libs/libarchive_test_1b.a
295ef327b417e4118d0329ef707d360b0bc40ce12be336d2ce006422dd593429  ./.libs/libarchive_test_2a.a
11d4fec99890b26d8ee8ebb9710ff36b665c5fb4653d43de2ea86a3d52dc7b3b  ./.libs/libarchive_test_2b.a

$ head -20 example3.cwrap.out
T25606 C0 SP    +0 0.000000s + archive_read_new() {
T25606 C0 SP   -64 0.000292s   + archive_entry_new2() {} = ? // #1
T25606 C0 SP    +0 0.000380s   } = ? // archive_read_new() #1
T25606 C0 SP    +0 0.000415s + archive_read_support_format_all() {
T25606 C0 SP   -80 0.000428s   + __archive_check_magic() {} = 0 // #1
T25606 C0 SP   -48 0.000478s   + archive_read_support_format_ar() {
T25606 C0 SP  -144 0.000490s     + __archive_check_magic() {} = 0 // #2
T25606 C0 SP  -208 0.000540s     + __archive_read_register_format() {
T25606 C0 SP  -384 0.000552s       + __archive_check_magic() {} = 0 // #3
T25606 C0 SP  -208 0.000602s       } = 0 // __archive_read_register_format() #1
T25606 C0 SP   -48 0.000801s     } = 0 // archive_read_support_format_ar() #1
T25606 C0 SP   -48 0.000879s   + archive_read_support_format_cpio() {
T25606 C0 SP  -144 0.000914s     + __archive_check_magic() {} = 0 // #4
T25606 C0 SP  -208 0.000971s     + __archive_read_register_format() {
T25606 C0 SP  -384 0.000983s       + __archive_check_magic() {} = 0 // #5
T25606 C0 SP  -208 0.001032s       } = 0 // __archive_read_register_format() #2
T25606 C0 SP   -48 0.001063s     } = 0 // archive_read_support_format_cpio() #1
T25606 C0 SP   -48 0.001093s   + archive_read_support_format_empty() {
T25606 C0 SP  -128 0.001105s     + __archive_check_magic() {} = 0 // #6
T25606 C0 SP  -192 0.001154s     + __archive_read_register_format() {

$ cat example3.cwrap.out | perl -lane '$lines++; if(m~SP\s*[\-|+]*(\d+)~){ $sp=$1; $sp_hi=$sp if($sp > $sp_hi); } sub END{ printf qq[- sp_hi=%d lines=%d\n], $sp_hi, $lines; }'
- sp_hi=1280 lines=13952

$ cat example3.cwrap.out | egrep "SP-1280"
T25606 C0 SP -1280 0.438052s                             - libarchive_read_callback(0=id) {} = 0=bytes_available
T25606 C0 SP -1280 0.580512s                             - libarchive_read_callback(1=id) {} = 0=bytes_available

$ cat example3.cwrap.out | egrep "libarchive_read_callback.*bytes_available"
T25606 C0 SP  -576 0.102593s               - libarchive_read_callback(0=id) {} = 7=bytes_available
T25606 C0 SP  -576 0.102968s               - libarchive_read_callback(0=id) {} = 7=bytes_available
T25606 C0 SP  -800 0.109330s                   - libarchive_read_callback(0=id) {} = 7=bytes_available
T25606 C0 SP  -800 0.109868s                   - libarchive_read_callback(0=id) {} = 262,144=bytes_available
T25606 C0 SP  -576 0.222140s               - libarchive_read_callback(1=id) {} = 7=bytes_available
T25606 C0 SP  -576 0.222447s               - libarchive_read_callback(1=id) {} = 7=bytes_available
T25606 C0 SP  -800 0.228260s                   - libarchive_read_callback(1=id) {} = 7=bytes_available
T25606 C0 SP  -800 0.228802s                   - libarchive_read_callback(1=id) {} = 262,144=bytes_available
T25606 C0 SP -1056 0.275702s                         - libarchive_read_callback(0=id) {} = 262,144=bytes_available
T25606 C0 SP -1056 0.303442s                         - libarchive_read_callback(0=id) {} = 262,144=bytes_available
T25606 C0 SP -1056 0.326784s                         - libarchive_read_callback(0=id) {} = 262,144=bytes_available
T25606 C0 SP -1056 0.350137s                         - libarchive_read_callback(0=id) {} = 262,144=bytes_available
T25606 C0 SP -1056 0.375164s                         - libarchive_read_callback(0=id) {} = 262,144=bytes_available
T25606 C0 SP -1056 0.399960s                         - libarchive_read_callback(0=id) {} = 116,967=bytes_available
T25606 C0 SP -1280 0.438052s                             - libarchive_read_callback(0=id) {} = 0=bytes_available
T25606 C0 SP -1056 0.466527s                         - libarchive_read_callback(1=id) {} = 262,144=bytes_available
T25606 C0 SP -1056 0.487312s                         - libarchive_read_callback(1=id) {} = 262,144=bytes_available
T25606 C0 SP -1056 0.505807s                         - libarchive_read_callback(1=id) {} = 262,144=bytes_available
T25606 C0 SP -1056 0.526057s                         - libarchive_read_callback(1=id) {} = 262,144=bytes_available
T25606 C0 SP -1056 0.548620s                         - libarchive_read_callback(1=id) {} = 262,144=bytes_available
T25606 C0 SP -1056 0.571182s                         - libarchive_read_callback(1=id) {} = 116,704=bytes_available
T25606 C0 SP -1280 0.580512s                             - libarchive_read_callback(1=id) {} = 0=bytes_available

Merging the coroutine code from part 3 with the libarchive example3.c code from part 2

  • Note: The id below is the archive number of either 0 or 1 and not the coroutine number.
  • Note: The Cn below is the coroutine number between 0 and 2 and not the archive number.
  • Note: coroutine 0 represents the regular program flow, while coroutines 1 and 2 are created for reading archives 0 and 1.
$ cp cogo-3-dummy-libarchive.c cor.c
$ cp example3.c example4.c
$ cp doit.sh doit4.sh
$ touch cor.h
$ # hack cor.[ch] to provide   our home-grown coroutines
$ # hack example4 to work with our home-grown coroutines; business logic alternates between archives and libarchive_read_callback() functions
$ # hack cwrap    to work with our home-grown coroutines
$ # hack doit4.sh to work with example4.c
$ ./doit4.sh
make  all-am
make[1]: Entering directory '/home/simon/20191016-libarchive/libarchive'
make[1]: Leaving directory '/home/simon/20191016-libarchive/libarchive'
Fri Nov  8 18:18:20 2019 0.099997 cwrap: gcc .. >                                                          cor.o.cwrap.cc.out;    8 functions,    1 warnings
Fri Nov  8 18:18:20 2019 0.189195 cwrap: gcc .. >                                                     example4.o.cwrap.cc.out;    4 functions,    3 warnings
Fri Nov  8 18:18:20 2019 0.241950 cwrap: gcc .. >                                                               .cwrap.cc.out;  121 objects  ,    0 libs
0=id - main_n_coroutine(1) {} // ready to unpack!
1=id - main_n_coroutine(2) {} // ready to unpack!
     - main_0_coroutine() {}
0=id - for archive file 0: read off disk, and archive_read_open()
0=id - attempting to open: test_1.tar.gz
0=id - read 1,689,852 bytes into id_buff_archive[0]
0=id - archive_read_new() {}
0=id - archive_read_open() {
1=id - for archive file 1: read off disk, and archive_read_open()
1=id - attempting to open: test_2.tar.gz
1=id - read 1,689,589 bytes into id_buff_archive[1]
1=id - archive_read_new() {}
1=id - archive_read_open() {
0=id   - libarchive_read_callback() {} = 7=bytes_available
1=id   - libarchive_read_callback() {} = 7=bytes_available
0=id   - libarchive_read_callback() {} = 7=bytes_available
1=id   - libarchive_read_callback() {} = 7=bytes_available
0=id   - libarchive_read_callback() {} = 7=bytes_available
1=id   - libarchive_read_callback() {} = 7=bytes_available
0=id   - libarchive_read_callback() {} = 262,144=bytes_available
0=id   } = ARCHIVE_OK
0=id - for archive file 0: archive_read_next_header() and archive_read_extract()
0=id - archive_read_next_header() {} = ARCHIVE_OK
0=id - archive_entry_pathname(entry) {} = .libs/libarchive_test_1a.a // file to extract
0=id - archive_read_extract() {
1=id   - libarchive_read_callback() {} = 262,144=bytes_available
1=id   } = ARCHIVE_OK
1=id - for archive file 1: archive_read_next_header() and archive_read_extract()
1=id - archive_read_next_header() {} = ARCHIVE_OK
1=id - archive_entry_pathname(entry) {} = .libs/libarchive_test_2a.a // file to extract
1=id - archive_read_extract() {
1=id   } = ARCHIVE_OK
1=id - archive_read_next_header() {} = ARCHIVE_OK
1=id - archive_entry_pathname(entry) {} = .libs/libarchive_test_2b.a // file to extract
1=id - archive_read_extract() {
0=id   - libarchive_read_callback() {} = 262,144=bytes_available
1=id   - libarchive_read_callback() {} = 262,144=bytes_available
0=id   - libarchive_read_callback() {} = 262,144=bytes_available
1=id   - libarchive_read_callback() {} = 262,144=bytes_available
0=id   - libarchive_read_callback() {} = 262,144=bytes_available
1=id   - libarchive_read_callback() {} = 262,144=bytes_available
0=id   - libarchive_read_callback() {} = 262,144=bytes_available
1=id   - libarchive_read_callback() {} = 262,144=bytes_available
0=id   - libarchive_read_callback() {} = 262,144=bytes_available
1=id   - libarchive_read_callback() {} = 262,144=bytes_available
0=id   - libarchive_read_callback() {} = 116,967=bytes_available
0=id   } = ARCHIVE_OK
0=id - archive_read_next_header() {} = ARCHIVE_OK
0=id - archive_entry_pathname(entry) {} = .libs/libarchive_test_1b.a // file to extract
0=id - archive_read_extract() {
1=id   - libarchive_read_callback() {} = 116,704=bytes_available
0=id   - libarchive_read_callback() {} = 0=bytes_available
0=id   } = ARCHIVE_OK
0=id - archive_read_next_header() {} = ARCHIVE_EOF
0=id - for archive file 0: archive_read_close() and archive_read_free()
0=id - archive_read_close() {}
0=id - archive_read_free() {}
0=id - main_n_coroutine(1) {} // ready to unpack!
1=id   - libarchive_read_callback() {} = 0=bytes_available
1=id   } = ARCHIVE_OK
1=id - archive_read_next_header() {} = ARCHIVE_EOF
1=id - for archive file 1: archive_read_close() and archive_read_free()
1=id - archive_read_close() {}
1=id - archive_read_free() {}
1=id - main_n_coroutine(2) {} // ready to unpack!
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 53.44    0.000799           0     14618           write
 44.35    0.000663          95         7           read
  2.21    0.000033           0     14359           gettid
  0.00    0.000000           0        14           open
  0.00    0.000000           0        13           close
  0.00    0.000000           0         9           fstat
  0.00    0.000000           0        18           mmap
  0.00    0.000000           0        12           mprotect
  0.00    0.000000           0         1           munmap
  0.00    0.000000           0         9           brk
  0.00    0.000000           0         7         7 access
  0.00    0.000000           0         1           execve
  0.00    0.000000           0         4           fcntl
  0.00    0.000000           0        12           umask
  0.00    0.000000           0         1           getrlimit
  0.00    0.000000           0         2           geteuid
  0.00    0.000000           0         1           arch_prctl
  0.00    0.000000           0         4           utimensat
------ ----------- ----------- --------- --------- ----------------
100.00    0.001495                 29092         7 total
T28688 C2 SP-20864 0.519121s               + archive_wstring_free() {} // #85
T28688 C2 SP-20864 0.519162s               + archive_string_free() {} // #301
T28688 C2 SP-20864 0.519186s               + archive_string_free() {} // #302
T28688 C2 SP-20864 0.519218s               + archive_string_free() {} // #303
T28688 C2 SP-20784 0.519243s               } // archive_mstring_clean() #81
T28688 C2 SP-20816 0.519269s             + archive_mstring_clean() {
T28688 C2 SP-20864 0.519278s               + archive_wstring_free() {} // #86
T28688 C2 SP-20864 0.519319s               + archive_string_free() {} // #304
T28688 C2 SP-20864 0.519344s               + archive_string_free() {} // #305
T28688 C2 SP-20864 0.519369s               + archive_string_free() {} // #306
T28688 C2 SP-20784 0.519395s               } // archive_mstring_clean() #82
T28688 C2 SP-20816 0.519420s             + archive_mstring_clean() {
T28688 C2 SP-20864 0.519429s               + archive_wstring_free() {} // #87
T28688 C2 SP-20864 0.519470s               + archive_string_free() {} // #307
T28688 C2 SP-20864 0.519495s               + archive_string_free() {} // #308
T28688 C2 SP-20864 0.519521s               + archive_string_free() {} // #309
T28688 C2 SP-20784 0.519546s               } // archive_mstring_clean() #83
T28688 C2 SP-20816 0.519571s             + archive_mstring_clean() {
T28688 C2 SP-20864 0.519580s               + archive_wstring_free() {} // #88
T28688 C2 SP-20864 0.519621s               + archive_string_free() {} // #310
T28688 C2 SP-20864 0.519646s               + archive_string_free() {} // #311
T28688 C2 SP-20864 0.519671s               + archive_string_free() {} // #312
T28688 C2 SP-20784 0.519697s               } // archive_mstring_clean() #84
T28688 C2 SP-20832 0.519722s             + archive_entry_copy_mac_metadata() {} // #16
T28688 C2 SP-20816 0.519747s             + archive_acl_clear() {} // #16
T28688 C2 SP-20816 0.519773s             + archive_entry_xattr_clear() {} // #12
T28688 C2 SP-20816 0.519798s             + archive_entry_sparse_clear() {} // #12
T28688 C2 SP-20768 0.519823s             } = ? // archive_entry_clear() #14
T28688 C2 SP-20688 0.519849s           } // archive_entry_free() #8
T28688 C2 SP-20720 0.519874s         + __archive_clean() {
T28688 C2 SP-20768 0.519884s           + archive_string_conversion_free() {} // #4
T28688 C2 SP-20720 0.519933s           } = 0 // __archive_clean() #4
T28688 C2 SP-20640 0.519980s         } = 0 // _archive_read_free() #2
T28688 C2 SP-20608 0.520006s       } = 0 // archive_free() #4
T28688 C2 SP-20576 0.520032s     } = 0 // archive_read_free() #2
T28688 C2 SP-20592 0.520068s   + cor_switch() {
T28688 C2 SP-20672 0.520078s     + cor_switch_internal() {
T28688 C2 SP-20720 0.520104s       - switch: 2 -> 0 // setjmp(2) for jump point b
T28688 C2 SP-20720 0.520145s       - switch: 2 -> 0 // longjmp(0)
... voodoo ...
T28688 C0 SP-30624 0.520171s             } // cor_switch_internal() #50
T28688 C0 SP-30576 0.520196s           - continued via jump point b
T28688 C0 SP-30624 0.520222s           + cor_sanity_check() {} // #53; co_stack_addr_main@-30,672 end1@-0 end2@-8,388,608
T28688 C0 SP-30544 0.520248s           } // cor_switch() #50
T28688 C0 SP-30432 0.520273s         } // main_0_coroutine() #1
T28688 C0 SP-30432 0.520299s       - coroutine 0 exiting; process shutting down shortly?
T28688 C0 SP  -256 0.520324s       } // cor_launch_internal() #3
T28688 C0 SP  -144 0.520350s     } // cor_launch() #3
T28688 C0 SP    +0 0.520376s   } // cor_launch_all() #1
T28688 C0 SP   +80 0.520401s - coroutines done!
11d4fec99890b26d8ee8ebb9710ff36b665c5fb4653d43de2ea86a3d52dc7b3b  ./.libs/libarchive_test_1a.a
295ef327b417e4118d0329ef707d360b0bc40ce12be336d2ce006422dd593429  ./.libs/libarchive_test_1b.a
295ef327b417e4118d0329ef707d360b0bc40ce12be336d2ce006422dd593429  ./.libs/libarchive_test_2a.a
11d4fec99890b26d8ee8ebb9710ff36b665c5fb4653d43de2ea86a3d52dc7b3b  ./.libs/libarchive_test_2b.a
$ cat example4.cwrap.out | wc -l
14413

Comparing example3.c with example4.c

Comparing order of libarchive_read_callback() at run-time

  • Below we can see that example3.c output shows blocks of processing due to the lack of async functionality.
  • Below we can see that example4.c output shows interleaving processing due to the new async-like functionality.
$ cat example3.cwrap.out | egrep "libarchive_read_callback.*bytes_available"
T25606 C0 SP  -576 0.102593s               - libarchive_read_callback(0=id) {} = 7=bytes_available
T25606 C0 SP  -576 0.102968s               - libarchive_read_callback(0=id) {} = 7=bytes_available
T25606 C0 SP  -800 0.109330s                   - libarchive_read_callback(0=id) {} = 7=bytes_available
T25606 C0 SP  -800 0.109868s                   - libarchive_read_callback(0=id) {} = 262,144=bytes_available
T25606 C0 SP  -576 0.222140s               - libarchive_read_callback(1=id) {} = 7=bytes_available
T25606 C0 SP  -576 0.222447s               - libarchive_read_callback(1=id) {} = 7=bytes_available
T25606 C0 SP  -800 0.228260s                   - libarchive_read_callback(1=id) {} = 7=bytes_available
T25606 C0 SP  -800 0.228802s                   - libarchive_read_callback(1=id) {} = 262,144=bytes_available
T25606 C0 SP -1056 0.275702s                         - libarchive_read_callback(0=id) {} = 262,144=bytes_available
T25606 C0 SP -1056 0.303442s                         - libarchive_read_callback(0=id) {} = 262,144=bytes_available
T25606 C0 SP -1056 0.326784s                         - libarchive_read_callback(0=id) {} = 262,144=bytes_available
T25606 C0 SP -1056 0.350137s                         - libarchive_read_callback(0=id) {} = 262,144=bytes_available
T25606 C0 SP -1056 0.375164s                         - libarchive_read_callback(0=id) {} = 262,144=bytes_available
T25606 C0 SP -1056 0.399960s                         - libarchive_read_callback(0=id) {} = 116,967=bytes_available
T25606 C0 SP -1280 0.438052s                             - libarchive_read_callback(0=id) {} = 0=bytes_available
T25606 C0 SP -1056 0.466527s                         - libarchive_read_callback(1=id) {} = 262,144=bytes_available
T25606 C0 SP -1056 0.487312s                         - libarchive_read_callback(1=id) {} = 262,144=bytes_available
T25606 C0 SP -1056 0.505807s                         - libarchive_read_callback(1=id) {} = 262,144=bytes_available
T25606 C0 SP -1056 0.526057s                         - libarchive_read_callback(1=id) {} = 262,144=bytes_available
T25606 C0 SP -1056 0.548620s                         - libarchive_read_callback(1=id) {} = 262,144=bytes_available
T25606 C0 SP -1056 0.571182s                         - libarchive_read_callback(1=id) {} = 116,704=bytes_available
T25606 C0 SP -1280 0.580512s                             - libarchive_read_callback(1=id) {} = 0=bytes_available

$ cat example4.cwrap.out | egrep "libarchive_read_callback.*bytes_available"
T28688 C1 SP-11120 0.142435s                 } = ? // libarchive_read_callback() #2; bytes_available=7
T28688 C2 SP-21120 0.143123s                 } = ? // libarchive_read_callback() #3; bytes_available=7
T28688 C1 SP-11120 0.143815s                 } = ? // libarchive_read_callback() #4; bytes_available=7
T28688 C2 SP-21120 0.147809s                 } = ? // libarchive_read_callback() #5; bytes_available=7
T28688 C1 SP-11344 0.151809s                     } = ? // libarchive_read_callback() #6; bytes_available=7
T28688 C2 SP-21344 0.152651s                     } = ? // libarchive_read_callback() #7; bytes_available=7
T28688 C1 SP-11344 0.153486s                     } = ? // libarchive_read_callback() #8; bytes_available=262,144
T28688 C2 SP-21344 0.187212s                     } = ? // libarchive_read_callback() #9; bytes_available=262,144
T28688 C1 SP-11600 0.236439s                           } = ? // libarchive_read_callback() #10; bytes_available=262,144
T28688 C2 SP-21600 0.260115s                           } = ? // libarchive_read_callback() #11; bytes_available=262,144
T28688 C1 SP-11600 0.285268s                           } = ? // libarchive_read_callback() #12; bytes_available=262,144
T28688 C2 SP-21600 0.306599s                           } = ? // libarchive_read_callback() #13; bytes_available=262,144
T28688 C1 SP-11600 0.325048s                           } = ? // libarchive_read_callback() #14; bytes_available=262,144
T28688 C2 SP-21600 0.347720s                           } = ? // libarchive_read_callback() #15; bytes_available=262,144
T28688 C1 SP-11600 0.369507s                           } = ? // libarchive_read_callback() #16; bytes_available=262,144
T28688 C2 SP-21600 0.392095s                           } = ? // libarchive_read_callback() #17; bytes_available=262,144
T28688 C1 SP-11600 0.416115s                           } = ? // libarchive_read_callback() #18; bytes_available=262,144
T28688 C2 SP-21600 0.438947s                           } = ? // libarchive_read_callback() #19; bytes_available=262,144
T28688 C1 SP-11600 0.461652s                           } = ? // libarchive_read_callback() #20; bytes_available=116,967
T28688 C2 SP-21600 0.486657s                           } = ? // libarchive_read_callback() #21; bytes_available=116,704
T28688 C1 SP-11824 0.497856s                               } = ? // libarchive_read_callback() #22; bytes_available=0
T28688 C2 SP-21824 0.509564s                               } = ? // libarchive_read_callback() #22; bytes_available=0

Comparing SHA256 sums

  • Below we can see that the extracted files have the same SHA256 sums with and without coroutines.
$ ./doit.sh
...
11d4fec99890b26d8ee8ebb9710ff36b665c5fb4653d43de2ea86a3d52dc7b3b  ./.libs/libarchive_test_1a.a
295ef327b417e4118d0329ef707d360b0bc40ce12be336d2ce006422dd593429  ./.libs/libarchive_test_1b.a
295ef327b417e4118d0329ef707d360b0bc40ce12be336d2ce006422dd593429  ./.libs/libarchive_test_2a.a
11d4fec99890b26d8ee8ebb9710ff36b665c5fb4653d43de2ea86a3d52dc7b3b  ./.libs/libarchive_test_2b.a

$ ./doit4.sh
...
11d4fec99890b26d8ee8ebb9710ff36b665c5fb4653d43de2ea86a3d52dc7b3b  ./.libs/libarchive_test_1a.a
295ef327b417e4118d0329ef707d360b0bc40ce12be336d2ce006422dd593429  ./.libs/libarchive_test_1b.a
295ef327b417e4118d0329ef707d360b0bc40ce12be336d2ce006422dd593429  ./.libs/libarchive_test_2a.a
11d4fec99890b26d8ee8ebb9710ff36b665c5fb4653d43de2ea86a3d52dc7b3b  ./.libs/libarchive_test_2b.a
Raw
cor.c
#include <assert.h>
#include <stdio.h>
#include <stdlib.h>
#include <setjmp.h>
#include <sys/resource.h>
#include <alloca.h>
#include <locale.h>
//#pragma GCC optimize ("O0") <-- note: using this pragma to force -O0 should work but it does not... resulting in seg faults etc :-(
#if defined __OPTIMIZE__
#pragma message "Need to compile " __FILE__ " with -O0 otherwise stack manipulation for coroutines can silently fail even if it appears to work... you have been warned!"
#pragma GCC error "aborting compile!"
#endif
#ifndef COR_ID
__thread int cor_id = 0;
#endif
static __thread jmp_buf * cor_id_jmp_buf;
static __thread char * * cor_id_stack_lo;
static __thread char * * cor_id_stack_hi;
static __thread int cor_max_coroutines ; // coroutine 0 + n coroutine stacks
static __thread int cor_max_coroutines_launched ; // n coroutine stacks
static __thread int cor_id_hi = 0 ;
static __thread int cor_stack_direction = 0 ; // set via stack_direction_get(); + for upwards and - for downwards
static __thread int cor_stack_size = 0 ;
static __thread int cor_stack_size_per_coroutine = 10000 ;
static __thread int cor_stack_corountine_partitions = 0 ;
static __thread int cor_coroutine_zero_exit = 0 ;
static __thread char * cor_stack_addr_main;
volatile __thread void * cor_stack_stick;
void cor_sanity_check(int id_new) {
char * stack_addr_this = (char *)&id_new;
/* cwrap_log_hint("; co_stack_addr_main@%s%'u end1@%s%'u end2@%s%'u",
(cor_stack_direction < 0) ? "-" : "+", abs(cor_stack_addr_main - stack_addr_this ) ,
(cor_stack_direction < 0) ? "-" : "+", abs(cor_stack_addr_main - cor_id_stack_lo[id_new]) ,
(cor_stack_direction < 0) ? "-" : "+", abs(cor_stack_addr_main - cor_id_stack_hi[id_new])); */
if ((stack_addr_this > cor_id_stack_lo[id_new])
|| (stack_addr_this < cor_id_stack_hi[id_new])) { // todo: get this to work with "up" stacks
printf("- %d: cor_sanity_check() // ERROR: current stack pointer is out of range!\n", cor_id); // todo: create an error / abort function
exit(1);
}
}
void cor_switch_internal(int id_old, int id_new) {
void *here = &cor_id_jmp_buf[id_old]; // https://www.pcjs.org/pubs/pc/reference/microsoft/kb/Q34921/
void *there = &cor_id_jmp_buf[id_new];
// cwrap_log("- switch: %d -> %d // setjmp(%d) for jump point b", id_old, id_new, id_old);
if (setjmp(here)) {
// note: putting instrumentation here appears to mess up the stack (although no seg fault!)
return; // note: setjmp() man page: "The stack context will be invalidated if the function which called setjmp() returns."
}
// cwrap_log("- switch: %d -> %d // longjmp(%d)\n... voodoo ...", id_old, id_new, id_new);
cor_id = id_new;
longjmp(there, 1);
}
void cor_switch(int id_old, int id_new) {
cor_switch_internal(id_old, id_new);
// note: return maybe from different coroutine, so cannot use function variables now!
// cwrap_log("- continued via jump point b");
cor_sanity_check(cor_id);
}
int cor_stack_direction_get(void * stack_addr_caller) {
int direction;
if (stack_addr_caller) {
// come here if 2nd call
void * stack_addr_callee = &stack_addr_caller;
int diff = stack_addr_callee - stack_addr_caller;
direction = (diff < 0) ? -1 : +1;
// cwrap_log_hint("; stack_addr_caller=%p stack_addr_callee=%p diff=%d result=%+d AKA %s", stack_addr_caller, stack_addr_callee, diff, direction, (-1 == direction) ? "down" : "up");
}
else {
// come here if 1st call
void * stack_addr_caller = &stack_addr_caller;
direction = cor_stack_direction_get(stack_addr_caller);
}
return(direction);
}
int cor_stack_size_get(void * argc_addr) {
cor_stack_addr_main = argc_addr;
struct rlimit l;
getrlimit(RLIMIT_STACK, &l);
// cwrap_log_hint("; result=%'ld bytes %s @ stack_addr_main=%p", l.rlim_cur, (cor_stack_direction < 0) ? "downward": "upward", cor_stack_addr_main);
char * cor_id_stack_end_1 = cor_stack_addr_main; // todo: get more precise stack addresses... from /proc ?
char * cor_id_stack_end_2 = cor_stack_addr_main + (cor_stack_direction * l.rlim_cur);
cor_id_stack_lo[0] = cor_id_stack_end_2 > cor_id_stack_end_1 ? cor_id_stack_end_2 : cor_id_stack_end_1;
cor_id_stack_hi[0] = cor_id_stack_end_2 > cor_id_stack_end_1 ? cor_id_stack_end_1 : cor_id_stack_end_2;
return(l.rlim_cur);
}
void cor_launch_internal(int id_old, int id_new, void (*thread_main)(void), const char * thread_main_name) {
char * stack_addr_this = (char *)&thread_main_name;
// todo: assert that id_old must be coroutine 0
cor_stack_corountine_partitions ++;
int size_of_all_partitions_so_far = (cor_stack_corountine_partitions * cor_stack_size_per_coroutine);
if (id_new) {
char * cor_id_stack_end_1 = stack_addr_this + (cor_stack_direction * ((0 + cor_stack_corountine_partitions) * cor_stack_size_per_coroutine));
char * cor_id_stack_end_2 = stack_addr_this + (cor_stack_direction * ((1 + cor_stack_corountine_partitions) * cor_stack_size_per_coroutine));
cor_id_stack_lo[id_new] = cor_id_stack_end_2 > cor_id_stack_end_1 ? cor_id_stack_end_2 : cor_id_stack_end_1;
cor_id_stack_hi[id_new] = cor_id_stack_end_2 > cor_id_stack_end_1 ? cor_id_stack_end_1 : cor_id_stack_end_2;
}
/* cwrap_log("- launch %d -> %d // co_stack_corountine_partitions=%d size_of_all_partitions_so_far=%d*%'d=%'d co_stack_addr_main@%s%'u end1@%s%'u end2@%s%'u%s",
id_old,
id_new,
cor_stack_corountine_partitions,
cor_stack_corountine_partitions,
cor_stack_size_per_coroutine,
size_of_all_partitions_so_far,
(cor_stack_direction < 0) ? "-" : "+", abs(cor_stack_addr_main - stack_addr_this ),
(cor_stack_direction < 0) ? "-" : "+", abs(cor_stack_addr_main - cor_id_stack_lo[id_new]),
(cor_stack_direction < 0) ? "-" : "+", abs(cor_stack_addr_main - cor_id_stack_hi[id_new]),
id_new ? "" : " <-- coroutine 0 stack skips past all others!"); */
// todo: assert that stack for this process is big enough for alloca()
cor_stack_stick = alloca(size_of_all_partitions_so_far); // ensure optimizer always keeps alloca() call
if (id_new) {
void *here = &cor_id_jmp_buf[id_old]; // https://www.pcjs.org/pubs/pc/reference/microsoft/kb/Q34921/
// cwrap_log("- launch %d -> %d // setjmp(%d) for jump point a\n... voodoo ...", id_old, id_new, id_old);
if (setjmp(here)) {
// note: putting instrumentation here appears to mess up the stack (although no seg fault!)
return; // note: setjmp() man page: "The stack context will be invalidated if the function which called setjmp() returns."
}
}
if (id_new) {
cor_id = id_new;
}
cor_sanity_check(cor_id);
thread_main();
// note: return maybe from different coroutine, so cannot use function variables now!
if (cor_id) {
printf("- %d: cor_launch_internal() // INTERNAL ERROR: program flow for coroutines > 0 should never come here!\n", cor_id);
abort(); // should never come here!
}
else {
// cwrap_log("- coroutine 0 exiting; process shutting down shortly?");
cor_coroutine_zero_exit = 1;
}
}
void cor_launch(int id_old, int id_new, void (*thread_main)(void), const char * thread_main_name) {
cor_launch_internal(id_old, id_new, thread_main, thread_main_name);
// note: return maybe from different coroutine, so cannot use function variables now!
if (cor_coroutine_zero_exit) {
// e.g. sub_main() returning to main() and soon process exit?
}
else {
// cwrap_log("- continued via jump point a");
cor_sanity_check(cor_id);
}
}
void cor_launch_all(
int coroutines ,
int coroutine_stack_size ,
void (* coroutine_n_main)(void),
const char * coroutine_n_main_name ,
void (* coroutine_0_main)(void),
const char * coroutine_0_main_name ) {
setlocale(LC_NUMERIC, "");
srand(25);
cor_max_coroutines_launched = coroutines ;
cor_max_coroutines = 1 + cor_max_coroutines_launched;
cor_id_jmp_buf = malloc(cor_max_coroutines * sizeof(*cor_id_jmp_buf));
cor_id_stack_lo = malloc(cor_max_coroutines * sizeof(*cor_id_stack_lo));
cor_id_stack_hi = malloc(cor_max_coroutines * sizeof(*cor_id_stack_hi));
cor_stack_size_per_coroutine = coroutine_stack_size;
cor_stack_direction = cor_stack_direction_get(NULL);
cor_stack_size = cor_stack_size_get(&coroutines);
// cwrap_log("- cor_stack_size_per_coroutine=%'d sizeof(jmp_buf)=%ld", cor_stack_size_per_coroutine, sizeof(jmp_buf));
// cwrap_log("- launching %d corountines in addition to coroutine 0", cor_max_coroutines_launched);
while (cor_id_hi < cor_max_coroutines_launched) {
cor_id_hi ++;
cor_launch(0, cor_id_hi, coroutine_n_main, coroutine_n_main_name);
}
// cwrap_log("- launched all corountines; launching dummy coroutine 0 for stack skipping purposes");
cor_launch(0, 0, coroutine_0_main, coroutine_0_main_name);
}
Raw
cor.h
#ifndef COR_H
#define COR_H
extern __thread int cor_id;
extern void cor_switch(int id_old, int id_new);
#endif /* COR_H */
Raw
cwrap.pl
# Copyright (c) 2012 Simon Hardy-Francis.
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
# http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html
# http://www.mindfruit.co.uk/2012/06/relocations-relocations.html
# http://www.akkadia.org/drepper/dsohowto.pdf
# $ time make CC="perl ~/work/20121228-cwrap/cwrap.pl gcc"
# todo: compile with only auxinfo for fast but summarized build
# todo: compile c only once with auxinfo, create c2 with mock functions, munge s files together == faster
use strict;
use Time::HiRes;
use File::Basename;
$| ++;
sub get_stem_and_ext {
my ($path_file_exit) = @_;
#debug printf qq[debug: (?, ?, ?)=fileparse(%s)\n], $path_file_exit;
my ( $f, $d, $s ) = fileparse($path_file_exit, "", qr/\.[^.]*/); # suffix maybe nothing or .something
$d = '' if ($d eq './');
my $stem;
$stem = $d if ($d ne ''); ; # e.g. 'connection' in 'connection.o'
$stem .= $f ; # e.g. 'connection' in 'connection.o'
my $ext = $s ; # e.g. '.o' in 'connection.o'
#debug printf qq[debug: ($f, $d, $s)=fileparse(%s) // %s=stem %s=ext\n], $path_file_exit, $stem, $ext;
return ($stem, $ext);
}
my $start_time = Time::HiRes::time;
# e.g. -DHAVE_CONFIG_H -I. -I../.. -DSHARE_DATADIR="\"/usr/local/share\"" -DLOCALSTATEDIR="\"/usr/local/var\"" -DBINDIR="\"/usr/local/bin\"" -I../../src/common -g -O2 -D_FORTIFY_SOURCE=2 -fstack-protector-all -Wstack-protector -fwrapv - -param ssp-buffer-size=1 -fPIE -Wall -fno-strict-aliasing -MT connection.o -MD -MP -MF . deps/connection.Tpo -c -o connection.o connection.c
my $arguments = "";
for ( my $c = 0; $c < @ARGV; $c++ ) {
my $tmp = $ARGV[$c];
$tmp =~ s~"~\\"~g;
$arguments .= ' ' . $tmp;
}
my $sub_end_hint ;
my $c2s ;
my $c2s2 ;
my $deps_file_stem ;
my $input_file ;
my $input_file_stem ;
my $input_file_ext ;
my $output_file ;
my $output_file_stem;
my $output_file_ext ;
my $output_file_log ;
my $s2o ;
if ( $ARGV[0] eq 'ar' ) {
$c2s = $arguments; # e.g. ar cru libtor.a buffers.o circuitbuild.o circuitlist.o circuituse.o command.o config.o connection.o connection_edge.o connection_or.o control.o cpuworker.o directory.o dirserv.o dirvote.o dns.o dnsserv.o geoip.o hibernate.o main.o microdesc.o networkstatus.o nodelist.o onion.o transports.o policies.o reasons.o relay.o rendclient.o rendcommon.o rendmid.o rendservice.o rephist.o router.o routerlist.o routerparse.o status.o config_codedigest.o
( $output_file ) = $c2s =~ m~\s+([^ ]+\.a\b)~ ; # e.g. 'libtor.a' in '-o libtor.a'
( $output_file_stem, # e.g. 'libtor' in 'libtor.a'
$output_file_ext ) = get_stem_and_ext($output_file) ; # e.g. '.a' in 'libtor.a'
( $output_file_log ) = $input_file_stem . $output_file_ext . '.cwrap.ar.out'; # e.g. 'libtor.a.cwrap.ar.out'
$sub_end_hint = sprintf qq[; %4u objects], scalar $arguments =~ s~(\.o)\b~$1~g;
}
elsif ( $ARGV[0] =~ m~^(gcc|g\+\+)$~ ) {
# -aux-info filename
# Output to the given filename prototyped declarations for all functions declared and/or defined in a translation unit, including those in header files. This option is silently ignored in any language other than C.
# Besides declarations, the file indicates, in comments, the origin of each declaration (source file and line), whether the declaration was implicit, prototyped or unprototyped (`I', `N' for new or `O' for old, respectively, in the first character after the line number and the colon), and whether it came from a declaration or a definition (`C' or `F', respectively, in the following character). In the case of function definitions, a K&R-style list of arguments followed by their declarations is also provided, inside comments, after the declaration.
# $ ctags -f - --format=2 --c++-kinds=+p --fields=+iaS --language-force=C c-example.c
# bar c-example.c /^int bar(int a) { return baz(a); }$/;" f signature:(int a)
# baz c-example.c /^int baz(int a) { return a ; }$/;" f signature:(int a)
# foo c-example.c /^int foo(int a) { return bar(a); }$/;" f signature:(int a)
# main c-example.c /^int main()$/;" f
# $ egrep "^\s+"call c-example.cwrap.3.s | egrep -i "(foo|bar|baz)"
# call baz
# call bar
# call foo
# call bar
# Consider using ctags for C++? e.g.:
# $ ctags -f - --format=2 --c++-kinds=+p --fields=+iaS --language-force=C++ cpp-example.cpp
# Foo cpp-example.cpp /^ Foo() {$/;" f class:Foo access:public signature:()
# Foo cpp-example.cpp /^class Foo$/;" c file:
# bar cpp-example.cpp /^int bar(int a) { return baz(a); }$/;" f signature:(int a)
# baz cpp-example.cpp /^int baz(int a) { return a ; }$/;" f signature:(int a)
# doitPrivate cpp-example.cpp /^ int doitPrivate(int a) { return bar(a); }$/;" f class:Foo file: access:private signature:(int a)
# doitPublic cpp-example.cpp /^ int doitPublic(int a) { return doitPrivate(a); }$/;" f class:Foo access:public signature:(int a)
# main cpp-example.cpp /^int main()$/;" f signature:()
# ~Foo cpp-example.cpp /^ ~Foo() {$/;" f class:Foo access:public signature:()
# How to understand C++ marking up? e.g.:
# Note: https://en.wikipedia.org/wiki/Name_mangling
# Note: https://stackoverflow.com/questions/41524956/gcc-c-name-mangling-reference
# Note: http://www.int0x80.gr/papers/name_mangling.pdf
# Note: http://demangler.com/
# $ egrep "^\s+"call cpp-example.cwrap.3.s | egrep -i "(foo|bar|baz)"
# call _Z3bazi
# call _Z3bari
# call _ZN3Foo11doitPrivateEi
# call _ZN3FooC1Ev
# call _ZN3Foo10doitPublicEi
# call _Z3bari
# call _ZN3FooD1Ev
#our $gcc = `which clang`; chomp $gcc; # clang doesn't support -aux-info
#our $gcc = `which gcc`; chomp $gcc;
$c2s = $arguments;
$c2s2 = $arguments;
( $deps_file_stem ) = $arguments =~ m~-MF (.*\.deps/[^\.]+).Tpo~ ; # e.g. '.deps/util' in '-MF .deps/util.Tpo'
( $input_file ) = $c2s =~ m~\s+([^ ]+\.cp*)~ ; # e.g. 'connection.c'
( $input_file_stem, # e.g. 'connection' in 'connection.c'
$input_file_ext ) = get_stem_and_ext($input_file) if ($input_file ne '') ; # e.g. '.c' in 'connection.c'
( $output_file ) = $c2s =~ m~\-o\s+([^ ]+)~ ; # e.g. 'connection.o' in '-o connection.o'
( $output_file_stem, # e.g. 'connection' in 'connection.o'
$output_file_ext ) = get_stem_and_ext($output_file) ; # e.g. '.o' in 'connection.o'
( $output_file_log ) = $input_file_stem . $output_file_ext . '.cwrap.cc.out'; # e.g. 'connection.o.cwrap.compile.out'
die qq[ERROR: not -o <file> detected in arguments: ] . $arguments if ( not $output_file );
$c2s =~ s~\s+\-MT\s+[^ ]+~~; # remove -MT util.o
$c2s =~ s~\s+\-MD\b~~ ; # remove -MD -MP
$c2s =~ s~\s+\-MP\b~~ ; # remove -MD -MP
$c2s =~ s~\s+\-MF\s+[^ ]+~~; # remove -MF .deps/util.Tpo
$c2s =~ s~ -o\s+$output_file_stem\.o\b~ -aux-info $input_file_stem.cwrap.1.aux -o /dev/null~; # NOTE: precompiled headers appear to be incompatible with -aux-info option :-(
$c2s2 =~ s~ -o\s+$output_file_stem\.o\b~ -fPIC -S -o $input_file_stem.cwrap.3.s~; # todo: do not add in -fPIC if it's already present
$c2s2 =~ s~ $input_file\b~ $input_file_stem.cwrap.2.c~;
$s2o = sprintf qq[as $input_file_stem.cwrap.4.s -o $output_file_stem.o] if ( $output_file_stem ); # -al
$sub_end_hint = sprintf qq[; %4u objects , %4u libs], scalar $arguments =~ s~(\.o)\b~$1~g, scalar $arguments =~ s~(\.a)\b~$1~g;
}
else {
die qq[ERROR: unrecognized first argument:] . $arguments;
}
my $arguments_short = sprintf qq[%-3s .. > %75s], $ARGV[0], $output_file_log;
sub END {
printf qq[%s %f cwrap: %s%s\n], scalar localtime, Time::HiRes::time - $start_time, $arguments_short, $sub_end_hint;
open ( my $fd_output_file_log, ">>", $output_file_log ) || die qq[];
printf $fd_output_file_log qq[%s %fs cwrap: %s%s\n], scalar localtime, Time::HiRes::time - $start_time, $arguments_short, $sub_end_hint;
close $fd_output_file_log;
}
$c2s .= ' -DCWRAP -include .cwrap.h';
$c2s2 .= ' -DCWRAP -include .cwrap.h';
$c2s .= ' >> ' . $output_file_log . ' 2>&1';
$c2s2 .= ' >> ' . $output_file_log . ' 2>&1';
$s2o .= ' >> ' . $output_file_log . ' 2>&1';
my $cwrap_log_not_first_time;
sub cwrap_log {
my ( $printf_format ) = shift;
open ( my $fd_output_file_log, defined $cwrap_log_not_first_time ? '>>' : '>', $output_file_log ) || die sprintf qq[ERROR: cannot open log file: %s: $!], $output_file_log;
printf $fd_output_file_log qq[%s %fs cwrap: ] . $printf_format . qq[\n], scalar localtime, Time::HiRes::time - $start_time, @_;
close $fd_output_file_log;
$cwrap_log_not_first_time = 1;
}
cwrap_log qq[debug: arguments :%s], $arguments ;
cwrap_log qq[debug: deps_file_stem :%s], $deps_file_stem ;
cwrap_log qq[debug: input_file :%s], $input_file ;
cwrap_log qq[debug: input_file_stem :%s], $input_file_stem ;
cwrap_log qq[debug: input_file_ext :%s], $input_file_ext ;
cwrap_log qq[debug: output_file :%s], $output_file ;
cwrap_log qq[debug: output_file_stem:%s], $output_file_stem;
cwrap_log qq[debug: output_file_ext :%s], $output_file_ext ;
die sprintf qq[ERROR: regex sanity check failed: output_file parts mismatch: '%s'=output_file ne ('%s'=output_file_stem . '%s'=output_file_exit)], $output_file, $output_file_stem, $output_file_ext if($output_file ne ($output_file_stem . $output_file_ext));
open ( my $fd_h_file, ">", ".cwrap.h" ) || die qq[];
printf $fd_h_file <<EOL;
#ifdef __cplusplus
extern "C" {
#endif
#define COR_ID (1)
extern __thread int cor_id;
extern void cwrap_log(const char * format, ...);
extern void cwrap_log(const char * format, ...);
extern void cwrap_log_push(int indent_direction, int no_append, const char * format, ...);
extern int cwrap_log_get_lines(void);
extern void cwrap_log_hint(const char * format, ...);
extern char * cwrap_log_hint_get(void);
#ifdef __cplusplus
}
#endif
EOL
close $fd_h_file;
if ( $output_file_ext eq '.o' ) {
cwrap_log qq[step 1: c->aux: running:%s], $c2s ;
}
elsif ( $output_file_ext eq '.a' ) {
cwrap_log qq[step 1: o->lib: running:%s], $c2s ;
}
else { # come here if linking exe
open ( my $fd_c_file, ">", $output_file_stem . '.cwrap.c' ) || die qq[];
printf $fd_c_file <<EOL;
#define _GNU_SOURCE /* See feature_test_macros(7) */
#include <unistd.h>
#include <sys/syscall.h> /* for syscall(), SYS_gettid */
#include <stdio.h>
#include <stdarg.h>
#include <sys/time.h>
#define CWRAP_LOG_LINE_MAX (1024)
#define COR_ID_MAX (10000)
__thread int cor_id = 0;
static char cwrap_log_spaces[] = " ";
static __thread char cwrap_log_line_hint[COR_ID_MAX][CWRAP_LOG_LINE_MAX];
static __thread int cwrap_log_line_hint_pos[COR_ID_MAX] = {0};
static __thread char cwrap_log_line[COR_ID_MAX][CWRAP_LOG_LINE_MAX];
static __thread int cwrap_log_line_pos[COR_ID_MAX] = {0};
__thread int cwrap_log_lines = 0;
static __thread int cwrap_log_indent[COR_ID_MAX] = {0}; // todo: make this dynamically grow to the necessary size; mremap() ?
static __thread void * cwrap_log_sp_start = NULL;
static FILE * cwrap_log_file = NULL;
static int cwrap_log_fd;
static double cwrap_log_time;
int cwrap_log_get_lines(void)
{
return cwrap_log_lines;
}
double cwrap_get_time_in_seconds(void)
{
struct timeval tv;
gettimeofday(&tv, NULL);
return (double)tv.tv_sec + 1.e-6 * (double)tv.tv_usec;
}
void cwrap_log_flush(void)
{
fprintf(cwrap_log_file, "%%s\\n", &cwrap_log_line[cor_id][0]);
cwrap_log_line_pos[cor_id] = 0;
fflush(cwrap_log_file);
//fsync(cwrap_log_fd); // todo: fsync intermittantly?
}
void cwrap_log_push_v(int indent_direction, int no_append, const char * format, va_list argument_list)
{
double time = cwrap_get_time_in_seconds();
if (no_append && (cwrap_log_line_pos[cor_id] > 0)) { // todo: create & check cor_id_last too?
cwrap_log_flush();
}
if (NULL == cwrap_log_file) { // todo: make thread safe
cwrap_log_file = fopen("$output_file_stem.cwrap.out","w+");
cwrap_log_fd = fileno(cwrap_log_file);
cwrap_log_time = time;
}
double time_elapsed = time - cwrap_log_time;
void * cwrap_log_sp = &format;
cwrap_log_sp_start = cwrap_log_sp_start ? cwrap_log_sp_start : cwrap_log_sp;
if (no_append) {
// if (cwrap_log_line_pos[cor_id] < CWRAP_LOG_LINE_MAX) { cwrap_log_line_pos[cor_id] += snprintf(&cwrap_log_line[cor_id][cwrap_log_line_pos[cor_id]], CWRAP_LOG_LINE_MAX - cwrap_log_line_pos[cor_id], "%%u " , cwrap_log_lines ); }
if (cwrap_log_line_pos[cor_id] < CWRAP_LOG_LINE_MAX) { cwrap_log_line_pos[cor_id] += snprintf(&cwrap_log_line[cor_id][cwrap_log_line_pos[cor_id]], CWRAP_LOG_LINE_MAX - cwrap_log_line_pos[cor_id], "T%%05ld " , syscall(SYS_gettid) ); } // linux; cat /proc/sys/kernel/pid_max
if (cwrap_log_line_pos[cor_id] < CWRAP_LOG_LINE_MAX) { cwrap_log_line_pos[cor_id] += snprintf(&cwrap_log_line[cor_id][cwrap_log_line_pos[cor_id]], CWRAP_LOG_LINE_MAX - cwrap_log_line_pos[cor_id], "C%%d " , cor_id ); }
if (cwrap_log_line_pos[cor_id] < CWRAP_LOG_LINE_MAX) { cwrap_log_line_pos[cor_id] += snprintf(&cwrap_log_line[cor_id][cwrap_log_line_pos[cor_id]], CWRAP_LOG_LINE_MAX - cwrap_log_line_pos[cor_id], "SP%%+6ld ", cwrap_log_sp - cwrap_log_sp_start ); }
if (cwrap_log_line_pos[cor_id] < CWRAP_LOG_LINE_MAX) { cwrap_log_line_pos[cor_id] += snprintf(&cwrap_log_line[cor_id][cwrap_log_line_pos[cor_id]], CWRAP_LOG_LINE_MAX - cwrap_log_line_pos[cor_id], "%%fs " , time_elapsed ); }
if (cwrap_log_line_pos[cor_id] < CWRAP_LOG_LINE_MAX) { cwrap_log_line_pos[cor_id] += snprintf(&cwrap_log_line[cor_id][cwrap_log_line_pos[cor_id]], CWRAP_LOG_LINE_MAX - cwrap_log_line_pos[cor_id], "%%.*s" , 2 * cwrap_log_indent[cor_id], &cwrap_log_spaces[0]); } // note: %% because of Perl
}
if (cwrap_log_line_pos[cor_id] < CWRAP_LOG_LINE_MAX) { cwrap_log_line_pos[cor_id] += vsnprintf(&cwrap_log_line[cor_id][cwrap_log_line_pos[cor_id]], CWRAP_LOG_LINE_MAX - cwrap_log_line_pos[cor_id], format , argument_list ); }
cwrap_log_lines ++;
cwrap_log_indent[cor_id] += indent_direction;
}
void cwrap_log_push(int indent_direction, int no_append, const char * format, ...)
{
va_list argument_list;
va_start(argument_list, format);
cwrap_log_push_v(indent_direction, no_append, format, argument_list);
va_end(argument_list);
}
void cwrap_log_pop(int indent_direction, int no_append, const char * format, ...)
{
va_list argument_list;
va_start(argument_list, format);
cwrap_log_push_v(indent_direction, no_append, format, argument_list);
cwrap_log_flush();
va_end(argument_list);
}
void cwrap_log(const char * format, ...)
{
va_list argument_list;
va_start(argument_list, format);
cwrap_log_push_v(0 /* no indent_direction */, 1 /* no append */, format, argument_list);
cwrap_log_flush();
va_end(argument_list);
}
void cwrap_log_hint(const char * format, ...)
{
va_list argument_list;
va_start(argument_list, format);
if (cwrap_log_line_hint_pos[cor_id] < CWRAP_LOG_LINE_MAX) { cwrap_log_line_hint_pos[cor_id] += vsnprintf(&cwrap_log_line_hint[cor_id][cwrap_log_line_hint_pos[cor_id]], CWRAP_LOG_LINE_MAX - cwrap_log_line_hint_pos[cor_id], format, argument_list); }
va_end(argument_list);
}
char * cwrap_log_hint_get(void)
{
char * hint = cwrap_log_line_hint_pos[cor_id] ? &cwrap_log_line_hint[cor_id][0] : "";
cwrap_log_line_hint_pos[cor_id] = 0;
return hint;
}
EOL
close $fd_c_file;
my $compile_cwrap = sprintf qq[ gcc -O2 -fPIC -c -o %s.cwrap.o %s.cwrap.c >> %s 2>&1], $output_file_stem, $output_file_stem, $output_file_log;
cwrap_log qq[step 1: ->c : running:%s], $compile_cwrap;
system $compile_cwrap;
$c2s =~ s~(\s+)([^\.\s]+\.o\s+)~$1$output_file_stem.cwrap.o $2~; # insert brand new .o file containing logging code etc
cwrap_log qq[step 2: o->exe: running:%s], $c2s;
}
system $c2s;
if ($? == -1) {
print "failed to execute: $!\n";
}
elsif ($? & 127) {
printf "child died with signal %d, %s coredump\n", ($? & 127), ($? & 128) ? 'with' : 'without';
}
else {
my $error_code = $? >> 8;
if ( $error_code > 0 ) {
system 'cat ' . $output_file_log;
$sub_end_hint = sprintf qq[; ERROR: failed with error code: %u], $error_code;
exit ( $error_code );
}
}
# router.cwrap.1.aux:
# ...
# /* router.c:102:NF */ extern crypto_pk_t *get_onion_key (void); /* () */
# ...
# /* router.c:1461:NF */ extern extrainfo_t *router_get_my_extrainfo (void); /* () */
# ...
# /* router.c:1796:NF */ extern void check_descriptor_bandwidth_changed (time_t now); /* (now) time_t now; */
# /* router.c:1820:NF */ static void log_addr_has_changed (int severity, const tor_addr_t *prev, const tor_addr_t *cur, const char *source); /* (severity, prev, cur, source) int severity; const tor_addr_t *prev; const tor_addr_t *cur; const char *source; */
# /* router.c:1848:NF */ extern void check_descriptor_ipaddress_changed (time_t now); /* (now) time_t now; */
# /* router.c:1882:NF */ extern void router_new_address_suggestion (const char *suggestion, const dir_connection_t *d_conn); /* (suggestion, d_conn) const char *suggestion; const dir_connection_t *d_conn; */
# /* router.c:1942:NF */ static int router_guess_address_from_dir_headers (uint32_t *guess); /* (guess) uint32_t *guess; */
# /* router.c:1956:NF */ extern void get_platform_str (char *platform, size_t len); /* (platform, len) char *platform; size_t len; */
# ...
# /* router.c:2503:NF */ extern const char *format_node_description (char *buf, const char *id_digest, int is_named, const char *nickname, const tor_addr_t *addr, uint32_t addr32h); /* (buf, id_digest, is_named, nickname, addr, addr32h) char *buf; const char *id_digest; int is_named; const char *nickname; const tor_addr_t *addr; uint32_t addr32h; */
# ...
# const char *shadow_format_node_description (char *buf, const char *id_digest, int is_named, const char *nickname, const tor_addr_t *addr, uint32_t addr32h) {
# printf("format_node_description()\n");
# const char * result = format_node_description(buf, id_digest, is_named, nickname, addr, addr32h);
# printf("return\n");
# return result;
# }
# .globl shadow_router_dump_router_to_string
# .section .data.rel,"aw",@progbits
# .align 8
# .type shadow_router_dump_router_to_string, @object
# .size shadow_router_dump_router_to_string, 8
# shadow_router_dump_router_to_string:
# .quad router_dump_router_to_string
# 8164 4251 E8000000 call _tor_malloc@PLT
# 8164 00
#
# 8168 4259 E8000000 call get_server_identity_key
# 8168 00
#
# 8174 426c E8000000 call shadow_router_dump_router_to_string <-- this
# 8174 00
#
# 8174 426c FF142500 call *shadow_router_dump_router_to_string <-- or this!
# 8174 000000
#
# 12689 .globl shadow_router_dump_router_to_string
# 12690 .section .data.rel,"aw",@progbits
# 12691 .align 8
# 12692 .type shadow_router_dump_router_to_string, @object
# 12693 .size shadow_router_dump_router_to_string, 8
# 12694 shadow_router_dump_router_to_string:
# 12695 0000 00000000 .quad other_router_dump_router_to_string
# 12695 00000000
#
# 37303 cfd6 00000000 .quad shadow_router_dump_router_to_string
# 37303 00000000
if ( $output_file_ext eq '.o' ) {
my $functions;
my $function_by_name;
{
my $aux_file = $input_file_stem . '.cwrap.1.aux';
open ( my $fd_aux_file, "<", $aux_file ) || die qq[];
sysread ( $fd_aux_file, my $aux_info, 9_999_999 );
close $fd_aux_file;
cwrap_log qq[step 2: c->c : read bytes of aux file: %u < %s], length $aux_info, $aux_file;
open ( my $fd_c_file, "<", $input_file ) || die qq[ERROR: cannot open: ] . $input_file;
sysread ( $fd_c_file, my $c_file, 9_999_999 );
close $fd_c_file;
cwrap_log qq[step 2: c->c : read bytes of c file: %u < %s], length $c_file, $input_file;
while ($c_file =~ s~\/\*(\s*cwrap_log.*?)\*\/~ $1 ~gs) {} # uncomment lines that look like: /* cwrap_log("foo") */ <-- todo: come up with a better regex :-)
while ($c_file =~ s~\/\/(\s*cwrap_log.*)~ $1~gm) {} # uncomment lines that look like: // cwrap_log("foo") <-- todo: come up with a better regex :-)
my $c2_file = $input_file_stem . '.cwrap.2.c';
open ( my $fd_c_file, ">", $c2_file ) || die qq[];
syswrite ( $fd_c_file, $c_file, length $c_file );
printf $fd_c_file qq[\n];
# printf $fd_c_file <<EOL;
#//extern int cwrap_log_get_lines(void);
#//extern void cwrap_log(int indent_direction, int no_append, const char * format, ...);
#//extern void cwrap_log_push(int indent_direction, int no_append, const char * format, ...);
#
#EOL
my $static_funcion_count = 0;
# /* /usr/include/x86_64-linux-gnu/sys/stat.h:493:NF */ extern int mknodat (int __fd, const char *__path, __mode_t __mode, __dev_t __dev); /* (__fd, __path, __mode, __dev) int __fd; const char *__path; __mode_t __mode; __dev_t __dev; */
# static int router_guess_address_from_dir_headers (uint32_t *); <no comment!!!> <-- special case!
while ( $aux_info =~ m~/\* ([^:]+:\d+:..) \*/ ([^\n\r]+)~gis ) { # /* ../../src/common/address.h:110:NF */ static sa_family_t tor_addr_family (const tor_addr_t *a); /* (a) const tor_addr_t *a; */
my ( $pre_line, $line_rest ) = ( $1, $2 ); # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
my ( $scope, $prototype, $comment ) = $line_rest =~ m~^(extern|static) ([^;]+);(.*)$~; # ^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ <-- comment optional & only for 'NF' $decl_type
my ( $return, $function, $params ) = $prototype =~ m~^(.*?)\s*([a-z0-9_]+)\s*\((.*)\)$~i; # ^^^^^^^^^^^ ^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^
my ( $vars ) = $comment =~ m~\(([^\)]+)\)~; # ^
my ( $file, $decl_type ) = $pre_line =~ m~^([^:]+):\d+:(..)~; # ^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^
#debug cwrap_log qq[- %15s %30s %-50s %s // ($vars)\n], $scope, $return, $function, $params;
#debug cwrap_log qq[- debug: fixme: $pre_line $line_rest\n] if ( $file =~ m~/stat.h~ );
next if ( $return =~ m~^$~ ); # ignore indirect functions for now e.g. /* libarchive/archive_pack_dev.c:325:NF */ extern pack_t (*pack_find (const char *name)); /* (name) const char *name; */
next if ( $file =~ m~^/~ ); # ignore system files for now, e.g. /* /usr/include/event2/util.h:288:NC */ extern int evutil_socketpair (int, int, int, int *);
next if ( $params =~ m~\.\.\.~ ); # todo: how to wrap va_list functions?
next if ( $params =~ m~\b__va_list_tag\b~ ); # todo: how to wrap __va_list_tag functions?
next if ( $function =~ m~^(memset|memmove|strlcpy|strlcat|evdns_shutdown|TO_EDGE_CONN)$~ ); # hack for Tor; TO_EDGE_CONN() is inline in or.h
# }
#
# while ( $aux_info =~ m~/\* ($o_file.c:\d+:.F) \*/ ([^\n\r]+)~gis ) { # /* router.c:1461:NF */ extern void routerstatus_get_verbose_nickname (char *buf, const routerstatus_t *router); /* (buf, router) char *buf; const routerstatus_t *router; */
# my ( $pre_line, $line_rest ) = ( $1, $2 ); # ^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# my ( $scope, $prototype, $comment ) = $line_rest =~ m~^(extern|static) ([^;]+);(.*)$~; # ^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# my ( $return, $function, $params ) = $prototype =~ m~^(.*?)\s*([a-z0-9_]+)\s*\((.*)\)$~i; # ^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# my ( $vars ) = $comment =~ m~\(([^\)]+)\)~; # ^^^^^^^^^^^
# my ( $file ) = $pre_line =~ m~^([^:]+):~; # ^^^^^^^^
if (( $file !~ m~^$input_file$~ ) # e.g. router.c
&& ( $decl_type =~ m~.F~ )) {
#debug cwrap_log qq[step 2: c->c : skipping aux line: %s %s // e.g. inline function defined in header file], $pre_line, $line_rest;
next;
}
my $function_static_postfix = $scope eq q[static] ? sprintf qq[_static_in_%s], $file : q[]; # static functions
$function_static_postfix =~ s~[^a-z0-9_]~_~gi;
$functions->{"$function$function_static_postfix"}{name} = $function;
$functions->{"$function$function_static_postfix"}{static_postfix} = $function_static_postfix;
$functions->{"$function$function_static_postfix"}{file} = $file;
$function_by_name->{$function}{mock} = "$function$function_static_postfix";
$function_by_name->{$function}{static_postfix} = $function_static_postfix;
$function_by_name->{$function}{file} = $file;
if (( $file !~ m~^$input_file$~ ) # e.g. router.c
|| ( $decl_type !~ m~.F~ )) {
#debug cwrap_log qq[step 2: c->c : skipping aux line: %s %s], $pre_line, $line_rest;
next; # only process further if we need to create the wrapper functions
}
printf $fd_c_file qq[\n];
printf $fd_c_file qq[extern unsigned long long cwrap_count_call_%s%s;\n], $function, $function_static_postfix;
printf $fd_c_file qq[extern %s (* cwrap_indirect_orig_%s%s) (%s);\n], $return, $function, $function_static_postfix, $params;
printf $fd_c_file qq[extern %s (* cwrap_indirect_wrap_%s%s) (%s);\n], $return, $function, $function_static_postfix, $params;
printf $fd_c_file qq[extern %s (* cwrap_indirect_mock_%s%s) (%s);\n], $return, $function, $function_static_postfix, $params;
my $not_void_return_assign = sprintf qq[%s cwrap_result =], $return if ( $return !~ m~\bvoid\b~ || $return =~ m~\*~ );
my $not_void_return_return = qq[return cwrap_result;] if ( $return !~ m~\bvoid\b~ || $return =~ m~\*~ );
my $function_exit;
if ( $return =~ m~\bvoid\b~ ) {
$function_exit = <<EOL;
if (0 == (cwrap_log_get_lines() - lines)) { cwrap_log_pop(-1, 0 /* append */, "} // " "#%%llu%%s", cwrap_count_call_$function$function_static_postfix, cwrap_log_hint_get()); }
else { cwrap_log_pop(-1, 1 /* no append */, "} // $function() #%%llu%%s", cwrap_count_call_$function$function_static_postfix, cwrap_log_hint_get()); }
EOL
}
elsif ( $return =~ m~\bint\b~ ) {
$function_exit = <<EOL;
if (0 == (cwrap_log_get_lines() - lines)) { cwrap_log_pop(-1, 0 /* append */, "} = %%'d // " "#%%llu%%s", cwrap_result, cwrap_count_call_$function$function_static_postfix, cwrap_log_hint_get()); }
else { cwrap_log_pop(-1, 1 /* no append */, "} = %%'d // $function() #%%llu%%s", cwrap_result, cwrap_count_call_$function$function_static_postfix, cwrap_log_hint_get()); }
EOL
}
else {
$function_exit = <<EOL;
if (0 == (cwrap_log_get_lines() - lines)) { cwrap_log_pop(-1, 0 /* append */, "} = ? // " "#%%llu%%s", cwrap_count_call_$function$function_static_postfix, cwrap_log_hint_get()); }
else { cwrap_log_pop(-1, 1 /* no append */, "} = ? // $function() #%%llu%%s", cwrap_count_call_$function$function_static_postfix, cwrap_log_hint_get()); }
EOL
}
printf $fd_c_file <<EOL;
static $return cwrap_wrap_$function$function_static_postfix ($params)
{
cwrap_count_call_$function$function_static_postfix ++;
// todo: add per binary/file/function verbosity checking
// todo: show function parameters for simple types, e.g. foo=123, bar=7
if (cwrap_count_call_$function$function_static_postfix <= 10000) cwrap_log_push(1, 1, "+ $function() {");
int lines = cwrap_log_get_lines();
$not_void_return_assign (* cwrap_indirect_orig_$function$function_static_postfix)($vars); // call wrapped function
if (cwrap_count_call_$function$function_static_postfix <= 10000) {
$function_exit
}
// todo: show return value for simple types
$not_void_return_return
}
EOL
#fixme: consider: __attribute__((used))
printf $fd_c_file qq[%s (* cwrap_indirect_orig_%s%s) (%s) = %s;\n] , $return, $function, $function_static_postfix, $params, $function ;
printf $fd_c_file qq[%s (* cwrap_indirect_wrap_%s%s) (%s) = cwrap_wrap_%s%s;\n], $return, $function, $function_static_postfix, $params, $function, $function_static_postfix;
printf $fd_c_file qq[%s (* cwrap_indirect_mock_%s%s) (%s) = cwrap_wrap_%s%s;\n], $return, $function, $function_static_postfix, $params, $function, $function_static_postfix; # point to wrapped function
printf $fd_c_file qq[unsigned long long cwrap_count_call_%s%s = 0;\n], $function, $function_static_postfix;
$static_funcion_count ++;
}
$sub_end_hint = sprintf qq[; %4u functions], $static_funcion_count;
close $fd_c_file;
cwrap_log qq[step 2: c->c : wrote bytes to c file: %u > %s], -s $c2_file, $c2_file;
cwrap_log qq[step 3: c->s : running: %s], $c2s2;
system $c2s2;
my $warnings = `egrep -c ": warning: " $output_file_log`;
$sub_end_hint .= sprintf ", %4u warnings", $warnings;
if ( $deps_file_stem ) {
my $po_file = $deps_file_stem . '.Tpo';
open ( my $fd_po_file, "<", $po_file ) || die qq[];
sysread ( $fd_po_file, my $po_file_bytes, 9_999_999 );
close $fd_po_file;
cwrap_log qq[step 4: ->Tpo: read bytes of Tpo file: %u < %s], length $po_file_bytes, $po_file;
$po_file_bytes =~ s~$input_file_stem.cwrap.2.c~$input_file_stem.c~gs; # change e.g. 'util.o: util.cwrap.2.c ../../orconfig.h /usr/include/fcntl.h \' to 'util.o: util.c ../../orconfig.h /usr/include/fcntl.h \'
$po_file_bytes =~ s~\.cwrap.h\:~~gs;
$po_file_bytes =~ s~\.cwrap.h\s*~~gs;
open ( my $fd_po_file, ">", $po_file ) || die qq[];
syswrite ( $fd_po_file, $po_file_bytes, length $po_file_bytes );
close $fd_po_file;
cwrap_log qq[step 4: ->Tpo: wrote bytes to Tpo file: %u > %s], length $po_file_bytes, $po_file;
}
else {
cwrap_log qq[step 4: ->Tpo: skipping .Tpo];
}
}
{
my $s3_file = $input_file_stem . '.cwrap.3.s';
open ( my $sfile, "<", $s3_file ) || die qq[ERROR: open failed: $!];
my @assembler_lines = <$sfile>;
close $sfile;
cwrap_log qq[step 5: s->s : read bytes of s file: %u bytes / %u lines < %s], -s $s3_file, scalar @assembler_lines, $s3_file;
my $call_lines;
my $call_line_replacements;
my $functions_ignored;
my $unique_functions_replaced;
# e.g. "call archive_write_new@PLT"
# e.g. "movq main_0_coroutine@GOTPCREL(%rip),"
# e.g. "leaq client_read_proxy(%rip)," from C code like this "filter->read = client_read_proxy;"
foreach ( grep ( m~(call|jmp|leaq|movq)[ \t]+([^ \t\@\(\r\n]+)(..rip|\@PLT|)~, @assembler_lines ) ) {
m~(call|jmp|leaq|movq)[ \t]+([^ \t\@\(\r\n]+)(..rip|\@PLT|)~;
my ( $op_type, $function ) = ( $1, $2 ); # e.g. leaq, client_read_proxy
next if ( $function =~ m~^cwrap_log~ );
next if ( $function =~ m~^\*cwrap_indirect_~ );
#next if ( $function =~ m~^__~ ); # e.g. __stack_chk_fail() or __fprintf_chk() or __errno_location()
next if ( $function =~ m~^\*~ ); # e.g. *40(%r14)
next if ( $function =~ m~\.part\.\d+$~ ); # e.g. tor_tls_get_by_ssl.part.6
next if ( $function =~ m~^[\.0-9\-]~ ); # e.g. .L828 or 144 or -1
next if ( $function =~ m~\.~ ); # e.g. av.7393
if ( exists $function_by_name->{$function} ) {
#debug printf $fd_output_file_log qq[%s %fs cwrap: function: replace: %-50s in %s], scalar localtime, Time::HiRes::time - $start_time, $function, $_;
my $function_mock = $function_by_name->{$function}{mock};
my $function_static_postfix = $function_by_name->{$function}{static_postfix};
my $function_file = $function_by_name->{$function}{file};
# s~(call[ \t]+)($function)(\@PLT|)$~$1 *cwrap_indirect_mock_$2$function_static_postfix$3(\%rip)~;
if (s~(call[ \t]+)($function)(\@PLT)$~$1 *cwrap_indirect_mock_$2$function_static_postfix$3(\%rip)~) {}
elsif (s~(call[ \t]+)($function)$~$1 *cwrap_indirect_mock_$2$function_static_postfix\@PLT(\%rip)~) {} # https://stackoverflow.com/questions/48369771/relocation-r-x86-64-pc32-against-symbol-when-calling-function-from-inline-assemb
elsif (s~(jmp[ \t]+)($function)(\@PLT)$~$1 *cwrap_indirect_mock_$2$function_static_postfix$3(\%rip)~) {}
elsif (s~(jmp[ \t]+)($function)$~$1 *cwrap_indirect_mock_$2$function_static_postfix\@PLT(\%rip)~) {}
elsif (s~(leaq[ \t]+)($function)~$1 cwrap_wrap_$2$function_static_postfix~) {} # todo: make this use the cwrap_indirect_mock_ pointer
elsif (s~(movq[ \t]+)($function)\@GOTPCREL~$1 cwrap_indirect_mock_$2$function_static_postfix\@PLT~) {}
if ( not exists $unique_functions_replaced->{$function} ) {
#debug cwrap_log qq[step 5: s->s : replaced new unique function: %s // %s], $function, $function_by_name->{$function}{static_postfix};
}
$unique_functions_replaced->{$function} ++;
$call_line_replacements ++;
}
else {
$functions_ignored->{$function} ++;
my $assembler_line = $_;
$assembler_line =~ s~\s+$~~g;
cwrap_log qq[step 5: s->s : function: ignore : %-50s in %s], $function, $assembler_line;
}
$call_lines ++;
}
foreach ( sort keys %{ $functions_ignored } ) {
cwrap_log qq[step 5: s->s : unique function ignored: %3u times %s], $functions_ignored->{$_}, $_;
}
cwrap_log qq[step 5: s->s : found %u call lines, replaced %u calls with %u unique functions, %u unique functions ignored], $call_lines, $call_line_replacements, scalar keys %{ $unique_functions_replaced }, scalar keys %{ $functions_ignored };
my $s4_file = $input_file_stem . '.cwrap.4.s';
open ( my $sfile, ">", $s4_file ) || die qq[];
foreach (@assembler_lines) {
print $sfile $_;
}
close $sfile;
cwrap_log qq[step 5: s->s : wrote bytes to s file: %u > %s], -s $s4_file, $s4_file;
}
cwrap_log qq[step 6: s->o : running: %s], $s2o;
system $s2o;
}
Raw
doit.sh
make CC="perl cwrap.pl gcc"
rm -f ./.libs/libarchive_test_1a.a
rm -f ./.libs/libarchive_test_1b.a
rm -f ./.libs/libarchive_test_2a.a
rm -f ./.libs/libarchive_test_2b.a
perl cwrap.pl gcc -DFROM_STREAM -Ilibarchive -O2 -fPIC -c -g -o example3.o example3.c
perl cwrap.pl gcc -DFROM_STREAM -Ilibarchive -fPIC -o example3 example3.o libarchive/*.o -lz -lexpat -lcrypto
strace -c ./example3 -r test_1.tar.gz test_2.tar.gz
egrep "\+ archive_read_open1" --after-context=300 example3.cwrap.out
chmod +r ./.libs/libarchive_test_1a.a ; sha256sum ./.libs/libarchive_test_1a.a
chmod +r ./.libs/libarchive_test_1b.a ; sha256sum ./.libs/libarchive_test_1b.a
chmod +r ./.libs/libarchive_test_2a.a ; sha256sum ./.libs/libarchive_test_2a.a
chmod +r ./.libs/libarchive_test_2b.a ; sha256sum ./.libs/libarchive_test_2b.a
Raw
doit4.sh
set -e
rm -f ./.libs/libarchive_test_1a.a
rm -f ./.libs/libarchive_test_1b.a
rm -f ./.libs/libarchive_test_2a.a
rm -f ./.libs/libarchive_test_2b.a
make CC="perl cwrap.pl gcc"
perl cwrap.pl gcc -DFROM_STREAM -O0 -fPIC -c -g -o cor.o cor.c
perl cwrap.pl gcc -DFROM_STREAM -Ilibarchive -O2 -fPIC -c -g -o example4.o example4.c
perl cwrap.pl gcc -DFROM_STREAM -Ilibarchive -fPIC -o example4 example4.o cor.o libarchive/*.o -lz -lexpat -lcrypto
strace -c ./example4 -r test_1.tar.gz test_2.tar.gz
#egrep "\+ archive_read_open1" --after-context=300 example4.cwrap.out
tail -50 example4.cwrap.out
chmod +r ./.libs/libarchive_test_1a.a ; sha256sum ./.libs/libarchive_test_1a.a
chmod +r ./.libs/libarchive_test_1b.a ; sha256sum ./.libs/libarchive_test_1b.a
chmod +r ./.libs/libarchive_test_2a.a ; sha256sum ./.libs/libarchive_test_2a.a
chmod +r ./.libs/libarchive_test_2b.a ; sha256sum ./.libs/libarchive_test_2b.a
Raw
example4.c
#include <string.h>
#include <stdlib.h>
#include <locale.h>
#include <unistd.h>
#include <fcntl.h>
#include <archive.h>
#include <archive_entry.h>
#include "cor.h"
#define CAST(TYPE, PTR) ((TYPE)(uintptr_t)(PTR))
#define BUFF_ARCHIVE_LEN (2 * 1024 * 1024)
#define ID_MAX (2)
int id_buff_archive_used[ID_MAX];
char id_buff_archive[ID_MAX][BUFF_ARCHIVE_LEN];
char * id_chunk_addr[ID_MAX];
int id_chunk_left[ID_MAX];
int id_chunk_size[ID_MAX];
int id_chunk_num [ID_MAX];
int id_chunk_todo[ID_MAX];
struct archive * id_archive[ID_MAX];
struct archive_entry * id_entry[ID_MAX] = {NULL};
la_ssize_t libarchive_read_callback(struct archive* a, void* client_data, const void** chunk)
{
cor_switch(cor_id, 0); // immediately give control back to coroutine 0 and let it decide which libarchive_read_callback() get to complete
int id = CAST(int, client_data);
*chunk = id_chunk_addr[id];
la_ssize_t bytes_available = id_chunk_left[id] < id_chunk_size[id] ? id_chunk_left[id] : id_chunk_size[id];
if ((id_chunk_num[id] < 3) && (id_chunk_left[id] > 123)) { // special business logic to cause archive_read_open() to call this callback multiple times
bytes_available = 7;
}
id_chunk_left[id] -= bytes_available;
id_chunk_addr[id] += bytes_available;
id_chunk_num [id] ++;
printf("%d=id - libarchive_read_callback() {} = %'ld=bytes_available\n", id, bytes_available);
cwrap_log_hint("; bytes_available=%'ld", bytes_available);
return bytes_available;
}
int id_max;
char ** filenames;
void main_n_coroutine(void) {
for (;;) {
int id = cor_id - 1;
printf("%d=id - main_n_coroutine(%d) {} // ready to unpack!\n", id, cor_id);
cor_switch(cor_id, 0); // immediately give control back to coroutine 0 AKA pre-launch
printf("%d=id - for archive file %d: read off disk, and archive_read_open()\n", id, id);
printf("%d=id - attempting to open: %s\n", id, filenames[id]);
int fd = open(filenames[id], O_RDONLY);
id_buff_archive_used[id] = read(fd, &id_buff_archive[id][0], sizeof(id_buff_archive[id]));
printf("%d=id - read %'d bytes into id_buff_archive[%d]\n", id, id_buff_archive_used[id], id);
close(fd);
printf("%d=id - archive_read_new() {}\n", id);
id_archive[id] = archive_read_new();
archive_read_support_format_all(id_archive[id]);
archive_read_support_filter_all(id_archive[id]);
#ifdef FROM_STREAM
id_chunk_addr[id] = &id_buff_archive[id][0];
id_chunk_left[id] = id_buff_archive_used[id];
id_chunk_size[id] = 256 * 1024;
id_chunk_num [id] = 0;
id_chunk_todo[id] = 1;
printf("%d=id - archive_read_open() {\n", id);
int res = archive_read_open(id_archive[id], CAST(void *, id), NULL, (archive_read_callback*)libarchive_read_callback, NULL);
printf("%d=id } = %s\n", id,
res == ARCHIVE_OK ? "ARCHIVE_OK" :
res == ARCHIVE_EOF ? "ARCHIVE_EOF" : "ERROR: UNKNOWN");
#else
std::cout << "calling archive_read_open_memory.." << std::endl;
int res = archive_read_open_memory(id_archive[id], &id_buff_archive[id][0], id_buff_archive_used[id]);
#endif
int flags = 0;
flags = ARCHIVE_EXTRACT_TIME;
flags |= ARCHIVE_EXTRACT_PERM;
flags |= ARCHIVE_EXTRACT_ACL;
flags |= ARCHIVE_EXTRACT_FFLAGS;
printf("%d=id - for archive file %d: archive_read_next_header() and archive_read_extract()\n", id, id);
do {
int res = archive_read_next_header(id_archive[id], &(id_entry[id]));
printf("%d=id - archive_read_next_header() {} = %s\n", id,
res == ARCHIVE_OK ? "ARCHIVE_OK" :
res == ARCHIVE_EOF ? "ARCHIVE_EOF" : "ERROR: UNKNOWN");
if (ARCHIVE_EOF == res) {
id_chunk_todo[id] = 0;
}
else {
printf("%d=id - archive_entry_pathname(entry) {} = %s // file to extract\n", id, archive_entry_pathname(id_entry[id]));
printf("%d=id - archive_read_extract() {\n", id);
res = archive_read_extract(id_archive[id], id_entry[id], flags);
printf("%d=id } = %s\n", id,
res == ARCHIVE_OK ? "ARCHIVE_OK" :
res == ARCHIVE_EOF ? "ARCHIVE_EOF" : "ERROR: UNKNOWN");
}
} while (id_chunk_todo[id]);
printf("%d=id - for archive file %d: archive_read_close() and archive_read_free()\n", id, id);
printf("%d=id - archive_read_close() {}\n", id);
archive_read_close(id_archive[id]);
printf("%d=id - archive_read_free() {}\n", id);
archive_read_free(id_archive[id]);
} /* for (;;) */
} /* main_n_coroutine() */
void main_0_coroutine(void) {
printf(" - main_0_coroutine() {}\n");
for (int id = 0; id < id_max; id ++) {
int cor_id_new = 1 + id;
cor_switch(0, cor_id_new);
}
int id_chunk_todo_still;
do {
id_chunk_todo_still = 0;
for (int id = 0; id < id_max; id ++) {
if (id_chunk_todo[id]) {
id_chunk_todo_still ++;
int cor_id_new = 1 + id;
cor_switch(0, cor_id_new);
}
}
} while (id_chunk_todo_still);
} /* main_0_coroutine() */
int main(int argc, char** argv) {
setlocale(LC_NUMERIC, "");
if(argc < 3) {
printf("%s {-r | -w} file[s]\n", argv[0]);
return 1;
}
char * mode = *++argv; // "-r" or "-w"
filenames = ++argv;
id_max = argc - 2;
// archive entry/file variables
struct stat st;
char buff_entry[2 * 1024 * 1024];
int fd;
if(0 == strcmp("-r", mode) /* read mode */) {
cor_launch_all(
id_max , /* coroutines to launch */
10000 , /* coroutine stack size */
&main_n_coroutine, "main_n_coroutine", /* coroutine main() */
&main_0_coroutine, "main_0_coroutine"); /* where execution continues */
// cwrap_log("- coroutines done!");
}
else {
int id = 0;
id_archive[id] = archive_write_new(); // initializes a new archive
archive_write_add_filter_gzip(id_archive[id]); // gzips incoming data
archive_write_set_format_pax_restricted(id_archive[id]); // only use extensions when necessary
archive_write_set_bytes_per_block(id_archive[id], 8192);
archive_write_open_filename(id_archive[id], "test.tar.gz");
for (int id = 0; id < id_max; id ++) {
printf("%d=id - attempting to open: %s\n", id, filenames[id]);
printf("- compressing: %s\n", filenames[id]);
stat(filenames[id], &st);
if(id_entry[id] == NULL) {
printf("- archive_entry_new() {}\n");
id_entry[id] = archive_entry_new();
}
else {
printf("- archive_entry_clear() {}\n");
archive_entry_clear(id_entry[id]);
}
// the following attributes are mandatory
archive_entry_set_pathname(id_entry[id], filenames[id]);
archive_entry_set_size(id_entry[id], st.st_size);
archive_entry_set_filetype(id_entry[id], st.st_mode);
archive_write_header(id_archive[id], id_entry[id]);
fd = open(filenames[id], O_RDONLY);
int len = read(fd, buff_entry, sizeof(buff_entry));
printf("- read() = %d\n", len);
while ( len > 0 ) {
archive_write_data(id_archive[id], buff_entry, len);
len = read(fd, buff_entry, sizeof(buff_entry));
printf("- read() = %d\n", len);
}
close(fd);
}
// done looping over files
archive_entry_free(id_entry[id]);
archive_write_close(id_archive[id]);
archive_write_free(id_archive[id]);
}
return 0;
} /* main() */
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment