Skip to content

Instantly share code, notes, and snippets.

@pdxjohnny

pdxjohnny/0_docs_parsers_updated.rst

Last active June 18, 2024 04:31
Show Gist options
  • Save pdxjohnny/deb3384d1ab7abfc49221ef4089b110d to your computer and use it in GitHub Desktop.
Save pdxjohnny/deb3384d1ab7abfc49221ef4089b110d to your computer and use it in GitHub Desktop.
Download ZIP
Can you please write me a file called docs/parsers.md which documents the functionality added in this patch. Please return your response markdown as an attachment? In the documentation please explain in detail including setting up a new package and entrypoint and class referenced through setup.cfg to and entry_points.txt via file:. Use the added…
Raw
0_docs_parsers_updated.rst

Parsers

Overview

This document details the functionality added by the latest patch to the parsers in the project. The patch introduces and tests various parsers for different programming languages and ensures they correctly identify and handle specific file types.

The addition of these parsers enhances the capability of the project to handle a variety of file types associated with different programming languages. The comprehensive test ensures that these parsers are correctly mapped and function as expected.

New Parsers

The following parsers have been added to the project:

  • DartParser
  • GoParser
  • JavaParser
  • JavascriptParser
  • PerlParser
  • PhpParser
  • PythonParser
  • PythonRequirementsParser
  • RParser
  • RubyParser
  • RustParser
  • SwiftParser
  • BanditParser

Test Implementation

A new test class TestParsers has been introduced to verify that the expected file types are correctly mapped to their respective parsers. The test ensures that the actual valid files match the expected valid files.

Test Method

  • test_parser_match_filenames_results_in_correct_valid_files: This test compares the EXPECTED_VALID_FILES dictionary with the actual_valid_files dictionary imported from cve_bin_tool.parsers.parse. If there is any discrepancy between the two, the test will fail, indicating that the loaded file types do not match the expected registered file types.

Usage

To utilize these parsers, ensure that your project includes the following imports:

from cve_bin_tool.parsers.dart import DartParser
from cve_bin_tool.parsers.go import GoParser
from cve_bin_tool.parsers.java import JavaParser
from cve_bin_tool.parsers.javascript import JavascriptParser
from cve_bin_tool.parsers.perl import PerlParser
from cve_bin_tool.parsers.php import PhpParser
from cve_bin_tool.parsers.python import PythonParser, PythonRequirementsParser
from cve_bin_tool.parsers.r import RParser
from cve_bin_tool.parsers.ruby import RubyParser
from cve_bin_tool.parsers.rust import RustParser
from cve_bin_tool.parsers.swift import SwiftParser
from cve_bin_tool.parsers.bandit import BanditParser

Setting Up a New Package and Entry Point

To implement a new parser plugin, such as a Bandit parser, follow these steps:

1. Create the Parser Class

First, create the parser class. This class should be located in the appropriate directory within your project. For example, you might place it in cve_bin_tool/parsers/bandit.py.

.. includefile:: cve_bin_tool/parsers/static_analysis_bandit.py

2. Set Up setup.py

Next, configure the setup.py file boilerplate.

# setup.py

import sys
import site
import setuptools

# See https://github.com/pypa/pip/issues/7953
site.ENABLE_USER_SITE = "--user" in sys.argv[1:]

setuptools.setup(use_scm_version=True)

3. Set Up setup.cfg

Next, configure the setup.cfg file to include your new parser as an entry point. This allows the parser to be dynamically discovered and used by the project.

# setup.cfg

[metadata]
name = cve-bin-tool-parser-static-analysis-bandit
version = 1.0.0
description = CVE Binary Tool Parser Plugin: Static Analysis: Bandit

[options]
packages = find:
entry_points = file: entry_points.txt
setup_requires =
    setuptools_scm[toml]>=3.4.3

4. Create entry_points.txt

You may also need to configure an entry_points.txt file if your project uses it to manage entry points.

# entry_points.txt

[cve_bin_tool.parsers]
bandit = cve_bin_tool.parsers.bandit:BanditParser

5. Install your plugin

You need to activate your virtualenv before installing if you set one up.

$ python -m pip install -e .

6. Run cve-bin-tool and see your plugin's findings

In this example we implemented the BanditParser which is a static analysis tool for Python files. We'll test that it loads by scanning a .py file.

$ cve-bin-tool --log debug setup.py
Raw
docs_parsers_updated.md

Parsers Documentation

Overview

This document details the functionality added by the latest patch to the parsers in the project. The patch introduces and tests various parsers for different programming languages and ensures they correctly identify and handle specific file types.

New Parsers

The following parsers have been added to the project:

  • DartParser
  • GoParser
  • JavaParser
  • JavascriptParser
  • PerlParser
  • PhpParser
  • PythonParser
  • PythonRequirementsParser
  • RParser
  • RubyParser
  • RustParser
  • SwiftParser
  • BanditParser

Test Implementation

A new test class TestParsers has been introduced to verify that the expected file types are correctly mapped to their respective parsers. The test ensures that the actual valid files match the expected valid files.

Test Method

  • test_parser_match_filenames_results_in_correct_valid_files: This test compares the EXPECTED_VALID_FILES dictionary with the actual_valid_files dictionary imported from cve_bin_tool.parsers.parse. If there is any discrepancy between the two, the test will fail, indicating that the loaded file types do not match the expected registered file types.

Usage

To utilize these parsers, ensure that your project includes the following imports:

from cve_bin_tool.parsers.dart import DartParser
from cve_bin_tool.parsers.go import GoParser
from cve_bin_tool.parsers.java import JavaParser
from cve_bin_tool.parsers.javascript import JavascriptParser
from cve_bin_tool.parsers.perl import PerlParser
from cve_bin_tool.parsers.php import PhpParser
from cve_bin_tool.parsers.python import PythonParser, PythonRequirementsParser
from cve_bin_tool.parsers.r import RParser
from cve_bin_tool.parsers.ruby import RubyParser
from cve_bin_tool.parsers.rust import RustParser
from cve_bin_tool.parsers.swift import SwiftParser
from cve_bin_tool.parsers.bandit import BanditParser

Setting Up a New Package and Entry Point

To implement a new parser plugin, such as a Bandit parser, follow these steps:

1. Create the Parser Class

First, create the parser class. This class should be located in the appropriate directory within your project. For example, you might place it in cve_bin_tool/parsers/bandit.py.

# cve_bin_tool/parsers/bandit.py

class BanditParser:
    def __init__(self):
        # Initialize the parser
        pass

    def parse(self, file_path):
        # Implement the parsing logic
        # For example, read the file and extract relevant information
        with open(file_path, 'r') as file:
            data = file.read()
            # Process data here
            return data  # Return processed data or specific findings

Explanation

  • class BanditParser:: This line defines the BanditParser class.
  • def __init__(self):: This is the constructor method for the class, used to initialize any attributes or perform any setup required.
  • def parse(self, file_path):: This method takes a file path as an argument and contains the logic to read and process the file.
    • with open(file_path, 'r') as file:: Opens the file specified by file_path in read mode.
    • data = file.read(): Reads the content of the file into the data variable.
    • return data: Returns the read data or processes it further to extract specific information.

2. Set Up setup.cfg

Next, configure the setup.cfg file to include your new parser as an entry point. This allows the parser to be dynamically discovered and used by the project.

# setup.cfg

[metadata]
name = cve-bin-tool
version = 1.0.0
description = CVE Binary Tool

[options]
packages = find:

[options.entry_points]
cve_bin_tool.parsers =
    bandit = cve_bin_tool.parsers.bandit:BanditParser

3. Create entry_points.txt

You may also need to configure an entry_points.txt file if your project uses it to manage entry points.

# entry_points.txt

[cve_bin_tool.parsers]
bandit = cve_bin_tool.parsers.bandit:BanditParser

Conclusion

The addition of these parsers enhances the capability of the project to handle a variety of file types associated with different programming languages. The comprehensive test ensures that these parsers are correctly mapped and function as expected.

For any issues or further enhancements, please refer to the contribution guidelines in the project's repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment