Current lib/debug.rb is no longer being maintained and it is not appropriate for current Ruby features. Koichi Sasada started writing new lib/debug.rb (hereinafter referred to as the new debugger) to replace it(For more information, see this article). However, it was hard to write test for it because the test framework have to test the new debugger without interrupting debugger process. That's why we worked on developping the test framework for the new debugger as the part of GSoC 2021 project.
- Implementing the test framework for the new debugger
The main project goal was to implement test framework for the new debugger. However, we were able to accomplish the goal before first evaluation. Therefore, we re-set our advanced goal as follows.
-
Making the test framework faster with multithreaded programming
Execution speed of the test framework in the new debugger is slow. As of August 29, 2021, it took over six minutes to finish the test with Github Actions. To make it faster, multithreaded programming is an effective way.
-
Adding Chrome debugging support in the new debugger
It's sometimes hard for users to see long backtrace informations in the terminal. To solve the problem, Chrome Devtools frontend is one of solution.
In short, what I've done is as shown below. I met the main goal and one extra goal. Although there were many challenges, I was able to overcome with the help of great mentors.
- Implementing the test framework for the new debugger
- Making the test framework faster with multithreaded programming
- Adding Chrome debugging support in the new debugger
The following figure is the structure about the test framework. The parent process run the child process(pseudo terminal) by PTY module. The parent process takes the inputs of the child process and tests them.
This is generated by Typora
- Create the test framework in the single process
- Create the test framework in the multi processes
- Display line numbers in test programs
- Modify the way to get readable IO
- [Fix bug] Modify assert_line_text method to be able to make correct assertions
- Make the test framework logs easier to read
- Modify the test framework for testing a remote debugger
- Improve the test framework - remove test_console.rb
- Improve assert methods with redifining existing assert methods
- Improve assert methods without refining existing assert methods
- Improve the algorithm in create message method
- Fix issue #97
- Rename MessageCreationController
- fix issue #106
- fix issue #108
- Support multiple assertions after one command
- Fix issue #119
- fix the part of #118
- improve the regexp pattern in assert_line_text method
- Improve some small parts in the test framework
- Improve debug logs -add remote debuggee backlog
- remove manual_debug_code method
- Check if there are any errors in @last_backlog
- Add inspect method for clearer failure messages
- Make sure to enable
check_line_numonly when Github Workflow. - Add assert_finish method to output clear failure messages when the test is not finished correctly
- Test generator
Test generator helps new debugger's contributors to create tests. By running the new debugger and entering debug commands, tests will be created on them(For detailed instructions, see this link).
The structure of test generator is very similar to it of the test framework.
- The test generator runs the child process by PTY module and the debugger program is executed in the child process.
- The test generator gets standard inputs from an user, and enters them into debugger program(child process). It also generates the method from user's inputs,
- The test generator generates assertions from child process's standard output/error. It also outputs standard output/error in the terminal.
- Finally, the file is generated.
- Create the test generator
- Fix issue #141
- Improve some points in the test generator - Make default method name unique name, Escape @last_backlog when it is outputted as regexp, and so on.
- Improve some points in the test generator - Make sure to match only "q!" or "quit!" as a command, if pattern is one line, the parameter should not use array in the test generator
- Make sure to match ask patterns in the test_generator
- Improve the test generator -Make sure to capitalize the first letter of class name in the test generator, Make sure to remove the string "test" from class name when creating a file name
- Improve some logics in the test generator
- Fix issue #185
- Fix issue #187
- The test generator should add "Test" if the string "test" is not included in method name
- Fix issue #194
- Modify the test generator so that the line numbers are displayed correctly in the generated file
- Fix issue #235
- Update the guide for test builder to match the current format
- Writing some test cases
- Add some test cases for breakpoints to line
- Add a test case for issue #89
- Add a test case for help command
- Enable color display in tests
- Add some test cases for Color module
- Solve the
warning: redefining constant Struct::SESSION - Add a test case for delete command
- Add remote: false in assertion_helper test
- Add some test cases for break command
- Add some test cases for quit command
- Add some test cases for kill command
Multithreaded programming allows us to test concurrently as follows. If a test fails, the current thread will do nonlocal exits and other threads will be killed.
Before
After
These are generated by Typora
To add Chrome debugging support in the new debugger, I needed to understand Chrome Devtools Protocol. Hence, I created a prototype to comprehend it. Thanks to the great help of the chrome-remote-interface team, I was able to create it in node.js.
https://github.com/ono-max/prototype-chrome-debug
While writing the codes, I encountered some bugs. The following pull requests are patches for fixing them.
- Fix the bug for catch command
- Uncomment the necessary part
- Fix typo
- Fix the appearance of "Stop by~"
- Fix the appearance of "Stop by~": 2nd version
-
Implementing the test framework for the new debugger
All current debug commands have a sufficient level of test coverage and the test framework has no issue so far. The test generator is helpful for writing tests. However it has a problem that the
programmethod can't be added if it already exists. -
Making the test framework faster with multithreaded programming
The following table shows the mediums of measured program execution times for the new debugger's tests. Multithreaded testing of the new debugger are run with
$ rake test. On the other hand, single-threaded testing of it are run with$ RUBY_DEBUG_TEST_DISABLE_THREADS=1 rake test. All measurements were taken 10 times(For actual datas, see this link) and run times were measured by using the time that test-unit outputs after all tests are completed.Comparing the medians, multithreaded programming is about 2.39 faster than single-threaded one.
Table2: The mediums obtained based on Table1(Unit: seconds)
Medium Multiple threads 56.32 Single thread 134.40 The following list is the development information used for the measurements.
- MacBook Pro (13-inch, 2019, Two Thunderbolt 3 ports)
- Processor 1.4 GHz Quad-Core Intel Core i5
- Memory 8 GB 2133 MHz LPDDR3
-
Adding Chrome debugging support in the new debugger
There are three steps to achieve this goal
- Creating a prototype in node.js
- Creating a prototype in Ruby
- Adding Chrome debuging support in the new debugger
I only completed the first step. One reason that I'm not able to accomplish is that there are few example. It took a lot of trial and error.
The new debugger should have been a fascinating project through my project.
-
Implementing the test framework for the new debugger
New debugger's contributors can create new features without breaking existing ones. They also can write tests easily with the test generator.
-
Making the test framework faster with multithreaded programming
New debugger's contributors can experiment at shorter intervals. They can wait for the test to finish with less stress.
-
Implementing the test framework for the new debugger
I plan to develop the test generator to solve the problem that the
programmethod can't be added if it already exists. I also maintains the test framework. -
Adding Chrome debugging support in the new debugger
I create a prototype in Ruby. After I'm done building it, I will create a pull request to add Chrome debugging support.
I am greatful to my mentor Koichi Sasada for guidance and support. Your advice always motivated me! Thank you teaching me a lot of things!
Special thanks to Yusuke Endoh, Kenta Murata, Kazuhiro Nishiyama, Stan Lo, Andrea Cardaci, Geoffrey Litt and all awesome people in Ruby community for making time for me.