Cdaprod/recon_data_to_aws_dynamodb.ipynb

## recon_data_to_aws_dynamodb.ipynb
{"cells":[{"metadata":{},"id":"7e7cffa9","cell_type":"markdown","source":"<a href=\"https://github.com/Cdaprod\">\n    <img src=\"https://github.githubassets.com/images/modules/logos_page/GitHub-Mark.png\" width=\"20\" align=\"left\" style=\"margin-right:10px\"> My name is David Cannan, follow me on GitHub!\n</a>\n\n---"},{"metadata":{},"id":"4ea2f8b0","cell_type":"markdown","source":"# Ingesting Reconnaissance Data into AWS DynamoDB\n\nHello there! If you've been working with reconnaissance tools and are looking for a way to organize and store your data efficiently, you're in the right place. In this notebook, I'll guide you through the process of ingesting your recon data into Amazon Web Service's DynamoDB. Not only is this a tutorial, but you can also reuse this notebook for multiple recon data sets. Let's get started!"},{"metadata":{},"id":"de447f92","cell_type":"markdown","source":"## Prerequisites\n1. **AWS Account**: You should have an AWS account and necessary permissions to create and manage DynamoDB tables.\n2. **boto3**: This is AWS's SDK for Python. We'll use it to interact with DynamoDB.\n\nYou can install it using pip:"},{"metadata":{"trusted":false},"id":"45a4811b","cell_type":"code","source":"!pip install boto3","execution_count":null,"outputs":[]},{"metadata":{},"id":"a5a21ae2","cell_type":"markdown","source":"## Setting Up Variables\n\nBefore we begin, let's set up the necessary variables. You can either source these from your environment (for security reasons) or input them directly.\n\n**Note**: Always be cautious when handling AWS credentials. Avoid hardcoding them directly."},{"metadata":{"trusted":false},"id":"1fbd5eca","cell_type":"code","source":"import os\n\n# AWS Credentials\nAWS_ACCESS_KEY = os.environ.get('AWS_ACCESS_KEY', 'YOUR_ACCESS_KEY')\nAWS_SECRET_KEY = os.environ.get('AWS_SECRET_KEY', 'YOUR_SECRET_KEY')\nAWS_REGION = os.environ.get('AWS_REGION', 'us-west-1')\n\n# Recon Data Directory\nRECON_DIRECTORY_PATH = os.environ.get('RECON_DIRECTORY_PATH', '/path/to/your/tool/output')\n\n# Target Company Name\nTARGET_COMPANY_NAME = os.environ.get('TARGET_COMPANY_NAME', 'example.com')\n\n## Parsing the Recon Data\n\nOur first step is to parse the data from our recon tool's output. The provided directory structure contains multiple files that we'll read and prepare for ingestion into DynamoDB.","execution_count":null,"outputs":[]},{"metadata":{},"id":"b99a0371","cell_type":"markdown","source":"## Parsing the Recon Data\n\nOur first step is to parse the data from our recon tool's output. The provided directory structure contains multiple files that we'll read and prepare for ingestion into DynamoDB."},{"metadata":{"trusted":false},"id":"dc382051","cell_type":"code","source":"def parse_file(file_path):\n    with open(file_path, 'r') as f:\n        return [line.strip() for line in f.readlines()]","execution_count":null,"outputs":[]},{"metadata":{},"id":"e1687924","cell_type":"markdown","source":"## Ingesting Data into DynamoDB\n\nNow that we have our data ready, let's store it in DynamoDB. We'll first initialize our DynamoDB resource using our AWS credentials, and then we'll ingest our parsed data."},{"metadata":{"trusted":true},"cell_type":"code","source":"import boto3\n\ndef ingest_into_dynamodb(directory_path, target_company):\n    # Initialize the DynamoDB resource\n    dynamodb = boto3.resource('dynamodb', region_name=AWS_REGION, aws_access_key_id=AWS_ACCESS_KEY, aws_secret_access_key=AWS_SECRET_KEY)\n    table = dynamodb.Table('recon_data')\n    \n    # Walk through the directory and ingest data\n    for root, dirs, files in os.walk(directory_path):\n        for file in files:\n            if file.endswith('.txt'):  # Only consider .txt files\n                file_path = os.path.join(root, file)\n                recon_type = os.path.basename(root)  # The directory name is the recon type\n                values = parse_file(file_path)\n                \n                for value in values:\n                    item = {\n                        'company_name': target_company,\n                        'data_type': recon_type,  # Using directory name as recon type\n                        'file_name': file,\n                        'value': value\n                    }\n                    table.put_item(Item=item)\n\n# Execute the ingestion\ningest_into_dynamodb(RECON_DIRECTORY_PATH, TARGET_COMPANY_NAME)\n","execution_count":null,"outputs":[]},{"metadata":{},"id":"d7e298da","cell_type":"markdown","source":"## Additional AWS Services\n\nNow that our data is in DynamoDB, there are several AWS services we can leverage:\n\n1. **Amazon Athena**: Query the data in DynamoDB using SQL.\n2. **AWS Lambda**: Automate and trigger specific actions based on the data.\n3. **Amazon QuickSight**: Visualize and analyze the data.\n\nFeel free to explore these services and integrate them into this notebook as per your needs."},{"metadata":{},"id":"1c195554","cell_type":"markdown","source":"That's it for this tutorial! You now have a reusable notebook to ingest recon data into DynamoDB. Remember to replace placeholders with actual values or source them from environment variables as needed. Once you're ready, you can expand upon this notebook, integrate additional AWS services, and adapt it to your specific needs."}],"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.10.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":5}