Working with SCAP is daunting. I'm in the "Sho" stage of "Sho Ha Re". SCAP is running, but only because I am following specific directions. There are hundreds of selected controls for SSG and STIG using SCAP. The basic runs only passes about half of the tests and there are many tests not even selected.
Breaking the STIG down would be helpful. For example, there are only 17 "Severity: High" tests. Wouldn't it make sense to have a test file that tests only for those 17?
What I'm trying to do is to create a simpler version of a STIG, a STIG that only tests a single control, or only the 17 "high" severity tests. I could of course manually pull out these tests. And I may do that. But a smarter approach would be to programatically build a small subset from the published source material. That way, I'm extracting the code.
I'm committed to acceptance test driven development. So I want the extracted controls. I need to write code to extract the controls. I need a way of testing my extract code. So I need a testing framework. Let's research testing frameworks.
Decide on a python testing framework. I need criteria.
- simplest to start. I like shallow learning curves.
- great if it provides for integrated tests
- easy reporting could be useful
- runs from CLI and can be part of an automation chain
- output can be in machine readable format
"Of the three projects, it looks like nose might well become the standard, with py.test having a smaller but loyal community and zope.testing remaining popular only for projects built atop the Zope framework. But all are actively maintained, and each has some unique features" [2]
unittest is built into python and maybe easiest to get started.
"I think of pytest as the run-anything, no boilerplate, no required api, use-this-unless-you-have-a-reason-not-to test framework. This is really where testing gets fun." [2]
"The doctest test framework is a python module that comes prepackaged with Python." [2]
"Robot Framework is a generic test automation framework for acceptance testing and acceptance test-driven development (ATDD). It has easy-to-use tabular test data syntax and utilizes the keyword-driven testing approach. Its testing capabilities can be extended by test libraries implemented either with Python or Java, and users can create new keywords from existing ones using the same syntax that is used for creating test cases." [5]
Robotframework seems to be very powerful general testing framework, including libraries for running operating system tests and taking screenshots. [7]
"Buildbot is a continuous integration system designed to automate the build/test cycle. By automatically rebuilding and testing the tree each time something has changed, build problems are pinpointed quickly, before other developers are inconvenienced by the failure." [8]
"Software Freedom Conservancy is a not-for-profit organization that helps promote, improve, develop, and defend Free, Libre, and Open Source Software (FLOSS) projects. Conservancy provides a non-profit home and infrastructure for FLOSS projects. This allows FLOSS developers to focus on what they do best — writing and improving FLOSS for the general public — while Conservancy takes care of the projects' needs that do not relate directly to software development and documentation." [9]
"Robot framework also gives very nice test reports, and the ability to organize your tests with a very simple but robust tagging system. Robot has a test runner that can output in XML, but also in any format you want using its listener interface (for example, you can stream test results to a remote monitoring tool, save results as JSON, etc)" [10]
"I'd suggest to use a DirectoryUpload step to get the logs from the slave to the public_html dir of the master, possibly into a builder/step dependent subdir, http://djmitche.github.com/buildbot/docs/0.7.10/#Transferring-Files. And then just addURL. You might be able to do that by just subclassing DirectoryUpload and to self.step_status.addURL in finished." [11]
I like the idea of trying robotframework because of it's general purpose approach. It appears that it may be easy to learn and robot framework can be applied to whole stack testing.
It may also make sense to start simply with unittest and graduate to pytest of nose after having a basic understanding.
- [1] http://www.libcrack.so/2012/01/09/a-brief-analysis-of-python-testing-software/
- [2] http://pythontesting.net/start-here/
- [3] http://www.ibm.com/developerworks/aix/library/au-python_test/
- [4] http://docs.python.org/2/library/unittest.html
- [5] https://code.google.com/p/robotframework/
- [6] http://robotframework.org/
- [7] http://robotframework.org/#test-libraries
- [8] http://trac.buildbot.net/
- [9] http://sfconservancy.org/overview/
- [10] https://groups.google.com/forum/#!topic/robotframework-users/nK8TkN8QHvs
- [11] http://comments.gmane.org/gmane.comp.python.buildbot.devel/4221
I presume by "Sho Ha Re" you mean what seems to be more commonly transliterated as "Shu Ha Ri"? Took me a bit to find that one, even though it seemed famililar.