Create a gist now

Instantly share code, notes, and snippets.

@marpaia /README.md Secret
Last active May 11, 2018

What would you like to do?
Pack import tool

Query Pack Import Tool

To run the tool, download the import.go file somewhere locally. It can then be executed via go run:

$ go run ./import.go -help

Usage of /var/folders/wp/6fkmvjf11gv18tdprv4g2mk40000gn/T/go-build234469651/command-line-arguments/_obj/exe/import:
  -hostname string
    	Kolide server hostname (default "https://localhost:8080")
  -pack_dir string
    	Directory of packs
  -token string
    	Kolide authentication token
exit status 2

Usage

The import.go script accepts three command-line flags:

hostname

This is the hostname of your Kolide server. This should be in the format https://foobar.xyz:1234. Note that https:// must be prepended to the hostname and there should be no trailing slashes.

pack_dir

This is the local directory where all of your packs are located. Absolute paths are preferred.

token

This is a valid authentication token, so that the tool can communicate with the Kolide API. This isn't super elegant, but to get this token:

  • go to Kolide
  • open the web inspector
  • select the "XHR" tab
  • refresh the page if you don't see any requests in the sidebar
  • select a request from the sidebar
  • under the headers tab, look for "Request Headers"
  • copy JUST the token (NOT the "Bearer " text)

use the force

Sample output

To see what this script should output on a successful execution, observe the following output:

$ go run import.go -pack_dir ~/Desktop -token "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzZXNzaW9uX2tleSI6InQxWEdqU1JvU0N3NWNubExIcWduSSszeDNOWmFDMjNXalRzMm5kVU44VnVRb3NUem5MNzI5OWhObFBsREFuK09JenBCcWdNaWY1TGtRRUV3Q3lVdHNBPT0ifQ.XBxgopLiuwUSQX_NnHmnX4A0oVDVFk9tgmwuyAC7IGQ"
2017/03/07 11:28:47 .DS_Store is not a query pack... skipping.
2017/03/07 11:28:47 .localized is not a query pack... skipping.
2017/03/07 11:28:47 Created pack centos.conf (30)
2017/03/07 11:28:47 Created query listening_ports (276)
2017/03/07 11:28:47 Added query listening_ports to pack centos.conf (35)
2017/03/07 11:28:47 Created query disk_encryption (277)
2017/03/07 11:28:47 Added query disk_encryption to pack centos.conf (36)
2017/03/07 11:28:47 Created query schedule (278)
2017/03/07 11:28:47 Added query schedule to pack centos.conf (37)
2017/03/07 11:28:47 Created query events (279)
2017/03/07 11:28:47 Added query events to pack centos.conf (38)
2017/03/07 11:28:47 Created query sudoers (280)
2017/03/07 11:28:47 Added query sudoers to pack centos.conf (39)
2017/03/07 11:28:47 Created query etc_hosts (281)
2017/03/07 11:28:47 Added query etc_hosts to pack centos.conf (40)
2017/03/07 11:28:47 Created query kernel_modules (282)
2017/03/07 11:28:47 Added query kernel_modules to pack centos.conf (41)
2017/03/07 11:28:47 Created query ramdisk (283)
2017/03/07 11:28:47 Added query ramdisk to pack centos.conf (42)
2017/03/07 11:28:47 Created query authorized_keys (284)
2017/03/07 11:28:47 Added query authorized_keys to pack centos.conf (43)
2017/03/07 11:28:47 Created query mounts (285)
2017/03/07 11:28:47 Added query mounts to pack centos.conf (44)
2017/03/07 11:28:47 Created query os_version (286)
2017/03/07 11:28:47 Added query os_version to pack centos.conf (45)
2017/03/07 11:28:47 Created query logged_in_users (287)
2017/03/07 11:28:47 Added query logged_in_users to pack centos.conf (46)
2017/03/07 11:28:47 Created query iptables (288)
2017/03/07 11:28:47 Added query iptables to pack centos.conf (47)
2017/03/07 11:28:47 Created query root_directory (289)
2017/03/07 11:28:47 Added query root_directory to pack centos.conf (48)
2017/03/07 11:28:47 Created query private_ssh_key (290)
2017/03/07 11:28:47 Added query private_ssh_key to pack centos.conf (49)
2017/03/07 11:28:47 Created query acpi_tables (291)
2017/03/07 11:28:47 Added query acpi_tables to pack centos.conf (50)
2017/03/07 11:28:47 Created query crontab (292)
2017/03/07 11:28:47 Added query crontab to pack centos.conf (51)
2017/03/07 11:28:47 Created query open_sockets (293)
2017/03/07 11:28:47 Added query open_sockets to pack centos.conf (52)
2017/03/07 11:28:47 Created query ip_forwarding (294)
2017/03/07 11:28:47 Added query ip_forwarding to pack centos.conf (53)
2017/03/07 11:28:47 Created query suid_bin (295)
2017/03/07 11:28:47 Added query suid_bin to pack centos.conf (54)
2017/03/07 11:28:47 Created query arp_cache (296)
2017/03/07 11:28:47 Added query arp_cache to pack centos.conf (55)
2017/03/07 11:28:47 Created query osquery_info (297)
2017/03/07 11:28:47 Added query osquery_info to pack centos.conf (56)
2017/03/07 11:28:47 Created query rpm_packages (298)
2017/03/07 11:28:47 Added query rpm_packages to pack centos.conf (57)
2017/03/07 11:28:47 Created query kernel_info (299)
2017/03/07 11:28:47 Added query kernel_info to pack centos.conf (58)
2017/03/07 11:28:47 Created query usb_devices (300)
2017/03/07 11:28:47 Added query usb_devices to pack centos.conf (59)
2017/03/07 11:28:47 Created query last (301)
2017/03/07 11:28:47 Added query last to pack centos.conf (60)
2017/03/07 11:28:47 import.go is not a query pack... skipping.
2017/03/07 11:28:47 import_backup.go is not a query pack... skipping.

This was developed with the following as the objective packs file:

{
  "queries": {
    "acpi_tables": {
      "query": "select * from acpi_tables;",
      "interval": 86400,
      "description": "General reporting and heuristics monitoring."
    },
    "kernel_info": {
      "query": "select * from kernel_info;",
      "interval": 7200,
      "description": "Report the booted kernel, potential arguments, and the device."
    },
    "usb_devices": {
      "query": "select * from usb_devices;",
      "interval": 7200,
      "description": "Report an inventory of USB devices. Attaches and detaches will show up in hardware_events."
    },
    "crontab": {
      "query" : "select * from crontab join hash using (path);",
      "interval" : "3600",
      "version" : "1.4.5",
      "description" : "Retrieves all the jobs scheduled in crontab in the target system.",
      "value" : "Identify malware that uses this persistence mechanism to launch at a given interval"
    },
    "etc_hosts": {
      "query" : "select * from etc_hosts;",
      "interval" : "86400",
      "version" : "1.4.5",
      "description" : "Retrieves all the entries in the target system /etc/hosts file.",
      "value" : "Identify network communications that are being redirected. Example: identify if security logging has been disabled"
    },
    "kernel_modules": {
      "query" : "select * from kernel_modules;",
      "interval" : "3600",
      "platform" : "linux",
      "version" : "1.4.5",
      "description" : "Retrieves all the information for the current kernel modules in the target Linux system.",
      "value" : "Identify malware that has a kernel module component."
    },
    "last": {
      "query" : "select * from last;",
      "interval" : "3600",
      "version" : "1.4.5",
      "description" : "Retrieves the list of the latest logins with PID, username and timestamp.",
      "value" : "Useful for intrusion detection and incident response. Verify assumptions of what accounts should be accessing what systems and identify machines accessed during a compromise."
    },
    "open_sockets": {
      "query" : "select distinct pid, family, protocol, local_address, local_port, remote_address, remote_port, path from process_open_sockets where path <> '' or remote_address <> '';",
      "interval" : "86400",
      "version" : "1.4.5",
      "description" : "Retrieves all the open sockets per process in the target system.",
      "value" : "Identify malware via connections to known bad IP addresses as well as odd local or remote port bindings"
    },
    "logged_in_users": {
      "query" : "select liu.*, p.name, p.cmdline, p.cwd, p.root from logged_in_users liu, processes p where liu.pid = p.pid;",
      "interval" : "3600",
      "version" : "1.4.5",
      "description" : "Retrieves the list of all the currently logged in users in the target system.",
      "value" : "Useful for intrusion detection and incident response. Verify assumptions of what accounts should be accessing what systems and identify machines accessed during a compromise."
    },
    "ip_forwarding": {
      "query" : "select * from system_controls where oid = '4.30.41.1' or oid = '4.2.0.1';",
      "interval" : "3600",
      "version" : "1.4.5",
      "description" : "Retrieves the current status of IP/IPv6 forwarding.",
      "value" : "Identify if a machine is being used as relay."
    },
    "mounts": {
      "query" : "select * from mounts;",
      "interval" : "3600",
      "version" : "1.4.5",
      "description" : "Retrieves the current list of mounted drives in the target system.",
      "value" : "Scope for lateral movement. Potential exfiltration locations. Potential dormant backdoors."
    },
    "ramdisk": {
      "query" : "select * from block_devices where type = 'Virtual Interface';",
      "interval" : "3600",
      "version" : "1.4.5",
      "description" : "Retrieves all the ramdisk currently mounted in the target system.",
      "value" : "Identify if an attacker is using temporary, memory storage to avoid touching disk for anti-forensics purposes"
    },
    "listening_ports": {
      "query" : "select name, path, listening_ports.* from processes join listening_ports using (pid);",
      "interval" : "3600",
      "version" : "1.4.5",
      "description" : "Retrieves all the listening ports in the target system.",
      "value" : "Detect if a listening port iis not mapped to a known process. Find backdoors."
    },
    "suid_bin": {
      "query" : "select suid_bin.*, md5, sha1, sha256 from hash join suid_bin using (path);",
      "interval" : "3600",
      "version" : "1.4.5",
      "description" : "Retrieves all the files in the target system that are setuid enabled.",
      "value" : "Detect backdoor binaries (attacker may drop a copy of /bin/sh). Find potential elevation points / vulnerabilities in the standard build."
    },
    "arp_cache": {
      "query" : "select * from arp_cache;",
      "interval" : "3600",
      "version" : "1.4.5",
      "description" : "Retrieves the ARP cache values in the target system.",
      "value" : "Determine if MITM in progress."
    },
    "disk_encryption": {
      "query" : "select * from disk_encryption;",
      "interval" : "86400",
      "version" : "1.4.5",
      "description" : "Retrieves the current disk encryption status for the target system.",
      "value" : "Identifies a system potentially vulnerable to disk cloning."
    },
    "iptables": {
      "query" : "select * from iptables;",
      "interval" : "3600",
      "platform" : "linux",
      "version" : "1.4.5",
      "description" : "Retrieves the current filters and chains per filter in the target system.",
      "value" : "Verify firewall settings are as restrictive as you need. Identify unwanted firewall holes made by malware or humans"
    },
    "osquery_info": {
      "query" : "select * from time, osquery_info;",
      "interval" : "86400",
      "version" : "1.4.5",
      "description" : "Retrieves the current version of the running osquery in the target system and where the configuration was loaded from.",
      "value" : "Identify if your infrastructure is running the correct osquery version and which hosts may have drifted"
    },
    "os_version": {
      "query" : "select * from os_version;",
      "interval" : "86400",
      "version" : "1.4.5",
      "description" : "Retrieves information from the Operative System where osquery is currently running.",
      "value" : "Identify out of date operating systems or version drift across your infrastructure"
    },
    "rpm_packages": {
      "query" : "select * from rpm_packages;",
      "interval" : "86400",
      "platform" : "redhat,centos",
      "version" : "1.4.5",
      "description" : "Retrieves all the installed RPM packages in the target Linux system.",
      "value" : "General security posture."
    },
    "schedule": {
      "query": "select name, interval, executions, output_size, wall_time, (user_time/executions) as avg_user_time, (system_time/executions) as avg_system_time, average_memory, last_executed from osquery_schedule;",
      "interval": 7200,
      "removed": false,
      "version": "1.6.0",
      "description": "Report performance for every query within packs and the general schedule."
    },
    "events": {
      "query": "select name, publisher, type, subscriptions, events, active from osquery_events;",
      "interval": 86400,
      "removed": false,
      "description": "Report event publisher health and track event counters."
    },
    "sudoers": {
      "query": "select file.*, md5, sha1, sha256 from hash join file using (path) where path = '/etc/sudoers';",
      "interval": 86400,
      "removed": false,
      "description": "Rules for running commands as other users via sudo"
    },
    "authorized_keys": {
      "query": "select authorized_keys.* from users join authorized_keys using (uid);",
      "interval": 86400,
      "removed": false,
      "description": "A line-delimited authorized_keys table"
    },
    "root_directory": {
      "query": "select file.* from users join file using (uid) where path = '/' or path = '/root';",
      "interval": 86400,
      "removed": false,
      "description": "Information on the permissions of the root directory"
    },
    "private_ssh_key": {
      "query": "select file.* from users join file using (uid) where path like '/home/%%/.ssh/id_rsa' or path like '/home/%%/.ssh/id_dsa' or path like '/root/.ssh/id_rsa' or path like '/root/.ssh/id_dsa' or path like '/data/home/%%/.ssh/id_rsa' or path like '/data/home/%%/.ssh/id_dsa';",
      "interval": 86400,
      "removed": false,
      "description": "Ensure that users do not have private ssh keys on servers"
    }
  }
}
package main
import (
"bytes"
"crypto/tls"
"encoding/json"
"errors"
"flag"
"fmt"
"io/ioutil"
"log"
"net/http"
"os"
"strconv"
"strings"
)
var (
token = flag.String("token", "", "Kolide authentication token")
hostName = flag.String("hostname", "https://localhost:8080", "Kolide server hostname")
packDir = flag.String("pack_dir", "", "Directory of packs")
)
var httpClient = &http.Client{
Transport: &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
},
}
type serverError struct {
Message string `json:"message"`
Errors []struct {
Name string `json:"name"`
Reason string `json:"reason"`
} `json:"errors"`
}
func createPack(name, description string) (uint, error) {
type createPackRequest struct {
Name string `json:"name"`
Description string `json:"description"`
}
type createPackResponse struct {
Pack struct {
ID uint `json:"id"`
} `json:"pack"`
serverError
}
body := createPackRequest{
Name: name,
Description: description,
}
b, err := json.Marshal(body)
if err != nil {
return 0, err
}
request, err := http.NewRequest(
"POST",
*hostName+"/api/v1/kolide/packs",
bytes.NewBuffer(b),
)
if err != nil {
return 0, errors.New("Error creating request object: " + err.Error())
}
request.Header.Add("Authorization", fmt.Sprintf("Bearer %s", *token))
response, err := httpClient.Do(request)
if err != nil {
return 0, errors.New("Error making request:" + err.Error())
}
defer response.Body.Close()
var responseBody createPackResponse
err = json.NewDecoder(response.Body).Decode(&responseBody)
if err != nil {
return 0, errors.New("Error decoding HTTP response body")
}
if len(responseBody.Errors) != 0 {
errs := []string{}
for _, e := range responseBody.Errors {
errs = append(errs, e.Reason)
}
return 0, errors.New(strings.Join(errs, ";"))
}
return responseBody.Pack.ID, nil
}
func createQuery(name, query, description string) (uint, error) {
type createQueryRequest struct {
Name string `json:"name"`
Query string `json:"query"`
Description string `json:"description"`
}
type createQueryResponse struct {
Query struct {
ID uint `json:"id"`
} `json:"query"`
serverError
}
body := createQueryRequest{
Name: name,
Query: query,
Description: description,
}
b, err := json.Marshal(body)
if err != nil {
return 0, err
}
request, err := http.NewRequest(
"POST",
*hostName+"/api/v1/kolide/queries",
bytes.NewBuffer(b),
)
if err != nil {
return 0, errors.New("Error creating request object: " + err.Error())
}
request.Header.Add("Authorization", fmt.Sprintf("Bearer %s", *token))
response, err := httpClient.Do(request)
if err != nil {
return 0, errors.New("Error making request:" + err.Error())
}
defer response.Body.Close()
var responseBody createQueryResponse
err = json.NewDecoder(response.Body).Decode(&responseBody)
if err != nil {
return 0, errors.New("Error decoding HTTP response body")
}
if len(responseBody.Errors) != 0 {
errs := []string{}
for _, e := range responseBody.Errors {
errs = append(errs, e.Reason)
}
return 0, errors.New(strings.Join(errs, ";"))
}
return responseBody.Query.ID, nil
}
func addQueryToPack(packID, queryID uint, interval uint64, snapshot, removed bool) (uint, error) {
type addQueryToPackRequest struct {
PackID uint `json:"pack_id"`
QueryID uint `json:"query_id"`
Interval uint64 `json:"interval"`
Snapshot bool `json:"snapshot"`
Removed bool `json:"removed"`
}
type addQueryToPackResponse struct {
Scheduled struct {
ID uint `json:"id"`
} `json:"scheduled"`
serverError
}
body := addQueryToPackRequest{
PackID: packID,
QueryID: queryID,
Interval: interval,
Snapshot: snapshot,
Removed: removed,
}
b, err := json.Marshal(body)
if err != nil {
return 0, err
}
request, err := http.NewRequest(
"POST",
*hostName+"/api/v1/kolide/schedule",
bytes.NewBuffer(b),
)
if err != nil {
return 0, errors.New("Error creating request object: " + err.Error())
}
request.Header.Add("Authorization", fmt.Sprintf("Bearer %s", *token))
response, err := httpClient.Do(request)
if err != nil {
return 0, errors.New("Error making request:" + err.Error())
}
defer response.Body.Close()
var responseBody addQueryToPackResponse
err = json.NewDecoder(response.Body).Decode(&responseBody)
if err != nil {
return 0, errors.New("Error decoding HTTP response body")
}
if len(responseBody.Errors) != 0 {
errs := []string{}
for _, e := range responseBody.Errors {
errs = append(errs, e.Reason)
}
return 0, errors.New(strings.Join(errs, ";"))
}
return responseBody.Scheduled.ID, nil
}
type queryStanza struct {
Query string `json:"query"`
Interval interface{} `json:"interval"`
Description string `json:"description"`
Version string `json:"version"`
Value string `json:"value"`
Snapshot *bool `json:"snapshot"`
Removed *bool `json:"removed"`
}
type packFile struct {
Queries map[string]queryStanza `json:"queries"`
}
func init() {
flag.Parse()
if packDir == nil || *packDir == "" {
*packDir = "."
}
}
func main() {
files, err := ioutil.ReadDir(*packDir)
if err != nil {
log.Fatalln("Could not list files:", err)
}
for _, file := range files {
content, err := os.Open(file.Name())
if err != nil {
log.Fatalf("Could not open file at path %s: %s", file, err)
}
defer content.Close()
var pack packFile
jsonParser := json.NewDecoder(content)
if err = jsonParser.Decode(&pack); err != nil {
log.Printf("%s is not a query pack... skipping.", file.Name())
continue
}
packID, err := createPack(file.Name(), "")
if err != nil {
log.Fatalf("Error creating pack %s: %s", file.Name(), err)
}
log.Printf("Created pack %s (%d)", file.Name(), packID)
for name, query := range pack.Queries {
interval, err := convertToUint64(query.Interval)
if err != nil {
log.Fatalln(err)
}
queryID, err := createQuery(name, query.Query, query.Description)
if err != nil {
log.Fatalf("Error creating query %s: %s", name, err)
}
log.Printf("Created query %s (%d)", name, queryID)
removed := true
if query.Removed != nil {
removed = *query.Removed
}
snapshot := false
if query.Snapshot != nil {
snapshot = *query.Snapshot
}
scheduledID, err := addQueryToPack(packID, queryID, interval, snapshot, removed)
if err != nil {
log.Fatalf("Error scheduling query %s: %s", name, err)
}
log.Printf("Added query %s to pack %s (%d)", name, file.Name(), scheduledID)
}
}
}
func convertToUint64(input interface{}) (uint64, error) {
switch i := input.(type) {
case int, float64, uint, uint64:
return uint64(i.(float64)), nil
case string:
value, err := strconv.ParseUint(i, 10, 64)
if err != nil {
return 0, errors.New("Error converting string to uint: " + err.Error())
}
return value, nil
default:
return 0, errors.New("Got an unacceptable type for interval")
}
}
@groob

This comment has been minimized.

Show comment Hide comment
@groob

groob Jan 17, 2018

	for _, file := range files {
		content, err := os.Open(file.Name())

there's a bug here which assumes you need to open the file from the same folder. file.Name() is only the name of the file, not the path.

groob commented Jan 17, 2018

	for _, file := range files {
		content, err := os.Open(file.Name())

there's a bug here which assumes you need to open the file from the same folder. file.Name() is only the name of the file, not the path.

@audibleblink

This comment has been minimized.

Show comment Hide comment
@audibleblink

audibleblink Mar 7, 2018

related to @groob's comment. I had to cd into the pack dir first

cd path/to/packs
go run /path/to/import.go -hostname ... -secret ... -packs_dir $PWD

related to @groob's comment. I had to cd into the pack dir first

cd path/to/packs
go run /path/to/import.go -hostname ... -secret ... -packs_dir $PWD
@benbasscom

This comment has been minimized.

Show comment Hide comment
@benbasscom

benbasscom Apr 25, 2018

I am running into an issue where if a query already exists, it errors out and stops the import. Is there a way to have it add the existing name to the pack and then continue on?

I am running into an issue where if a query already exists, it errors out and stops the import. Is there a way to have it add the existing name to the pack and then continue on?

@offsecn00b

This comment has been minimized.

Show comment Hide comment
@offsecn00b

offsecn00b May 11, 2018

@benbasscom - I'm not sure if anyone is still looking at this code but I forked it and patched it so that it will catch the http.statuscode 409 and then do a lookup of the query ID by name and return that query ID to the script so that the existing query can be added to the new pack. I've never coded GO before so no warranties but you can try if it if you like.

https://github.com/offsecn00b/OSqueryPackImporter/blob/master/import.go

@benbasscom - I'm not sure if anyone is still looking at this code but I forked it and patched it so that it will catch the http.statuscode 409 and then do a lookup of the query ID by name and return that query ID to the script so that the existing query can be added to the new pack. I've never coded GO before so no warranties but you can try if it if you like.

https://github.com/offsecn00b/OSqueryPackImporter/blob/master/import.go

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment