Blog - Microsoft Sentinel Custom Log Ingestion

CustomLogIngestion.md

Microsoft Sentinel - Custom Log Ingestion

This post will describe how to ingest custom logfiles in to a Log Analytics workspace using GitHub Actions!

Why yet another method

Since I started with Microsoft Sentinel in the days it was still called Azure Sentinel I didn'found it difficult to send my own logging custom logging data to the attached Log Analytics workspace. Although the Microsoft Sentinel GitHub page provides a good description on how to export data from a workspace, it doens't describe anything on uploading that same data to another workspace.

Somewhere in 2019 Microsoft docs was updated including a PowerShell script to upload data to a log analytics workspace. Although this solution works and is still oftenly used in many Microsoft Sentinel Data Connectors for 3 party connectors, it still requires some coding skills.

The thing is with this old script that is not written very modular, and unfortunatly is still copy/pasted in new data connectors and solutions causing a lot of legacy code duplication.

Recently Microsoft made the Azure Monitor HTTP Data Collector API is in public preview.

Whoehoe! you might think, but unfortunatly the script hasn't been changed and is still the same as already described in 2019! In this new era I would at least expect that:

• The workspaceId and workspaceKey would be passed as securestring instead of clear text
• support for logfiles is bigger that the allowed limit of 25MB

To overcome this and make the script more readable, I've created an updated version that is available on my own GitHub

GitHub Actions rules the world!

Recently I started to play around with the SuperPowers of GitHub Actions GitHub actions are little building blocks that can be used in a yml pipeline, and are actually little wrappers around code.

The GitHub Actions uses code packages in Docker containers, which run on the GitHub servers and which, in turn, are compatible with any programming language.

Adding the Action

1. Open your GitHub repository and select Actions in the top ribbon menu Once the action page has openen select setup a workflow for yourself

2. add the following code block to your workflow

name: CustomLogs
on:
  pull_request:
    paths:
      - samples/**

jobs:
  custom-logs:
    name: Custom Logs
    runs-on: ubuntu-latest
    steps:
      - name: Check out repository code
        uses: actions/checkout@v3
      - name: Microsoft Sentinel Custom Logs
        uses: SecureHats/custom-logs@v0.2.0
        with:
          filesPath: samples
          workspaceId: ${{ secrets.WORKSPACEID }}
          workspaceKey: ${{ secrets.WORKSPACEKEY }}

The GitHub Action uses some parameters to control to which workspace the logs files needs to be send. In the table below you can find a description of these parameters.

4. As you might have noticed 2 secrets are used in this action, so these needs to be created. These secrets can be created by following this documentation link.

Note: The secrets are case sensitive

Testing the Action

After everything has been set up correctly we can start testing out GitHub Action! Depending on how the workflow in step #2 has been configured, the Action is triggered on a push to specific folder. In this case we configured the samples folder as the trigger folder on line 5 and line 17 as the input folder for the action.

So the only this we now have to do is drop a file in the samples folder. As soon as the file has been pushed to the folder, the GitHub Action starts to run as show in the image

Once the file has been succesfully processed it will take between 5 and 15 minutes before the data is visible in the Log Analytics workspace.

Happy coding!