Shell snippet. Takes output from bro2json.py and creates an array of JSON objects grouped by uid and id fields

gistfile1.sh

# This more or less just shows the format of the log data on input.

zcat bro2jsondata.json.gz| sed  's/id\./id_/g' | ./json -g | jq '[ map(select(.uid and .uid != "-" )) | group_by(.uid) | .[0] ]'
[
  [
    {
      "type": "conn",
      "id_orig_h": "10.10.10.17",
      "resp_ip_bytes": "0",
      "ts": "1384833674.270877",
      "resp_bytes": "0",
      "id_resp_h": "192.168.1.254",
      "conn_state": "S0",
      "local_orig": "F",
      "id_orig_p": "42923",
      "orig_ip_bytes": "152",
      "resp_pkts": "0",
      "orig_bytes": "96",
      "missed_bytes": "0",
      "duration": "0.000001",
      "uid": "000iFCzEelf",
      "proto": "udp",
      "history": "D",
      "id_resp_p": "53",
      "service": "dns",
      "orig_pkts": "2",
      "tunnel_parents": "(empty)"
    },
    {
      "ts": "1384833674.270877",
      "uid": "000iFCzEelf",
      "qclass_name": "C_INTERNET",
      "id_resp_h": "192.168.1.254",
      "trans_id": "35726",
      "RA": "F",
      "rcode": "-",
      "TC": "F",
      "id_orig_h": "10.10.10.17",
      "AA": "F",
      "rcode_name": "-",
      "Z": "0",
      "id_orig_p": "42923",
      "rejected": "F",
      "TTLs": "-",
      "qtype": "1",
      "qtype_name": "A",
      "query": "www.example.com",
      "type": "dns",
      "qclass": "1",
      "answers": "-",
      "proto": "udp",
      "id_resp_p": "53",
      "RD": "T"
    }
  ]
]
 
# Uncompress the JSON file, correct the id fieldnames, make a proper array, then use JQ wizardry.
# Note, this 'sed' and './json' business are fixable in original bro2json.py script.

 zcat bro2jsondata.json.gz | sed  's/id\./id_/g' | ./json -g | jq '[ map(select(.uid and .uid != "-" )) | group_by(.uid) | .[0] | { uid: (.[] | select(.type == "conn") | .uid), timestamp: (.[] | select(.type == "conn") | .ts | tonumber | . * 1000000), id: {orig_h: (.[] | select(.type == "conn") | .id_orig_h), orig_p: (.[] | select(.type == "conn") | .id_orig_p), resp_h: (.[] | select(.type == "conn") | .id_resp_h), resp_p: (.[] | select(.type == "conn") | .id_resp_p) }, logs: [(.[] | del(.uid) | del(.id_resp_p) | del(.id_resp_h) | del(.id_orig_p)|del(.id_orig_h))] }]'
[
  {
    "logs": [
      {
        "type": "conn",
        "resp_ip_bytes": "0",
        "ts": "1384833674.270877",
        "resp_bytes": "0",
        "conn_state": "S0",
        "local_orig": "F",
        "orig_ip_bytes": "152",
        "resp_pkts": "0",
        "orig_bytes": "96",
        "missed_bytes": "0",
        "duration": "0.000001",
        "proto": "udp",
        "history": "D",
        "service": "dns",
        "orig_pkts": "2",
        "tunnel_parents": "(empty)"
      },
      {
        "ts": "1384833674.270877",
        "qclass_name": "C_INTERNET",
        "trans_id": "35726",
        "RA": "F",
        "rcode": "-",
        "TC": "F",
        "AA": "F",
        "rcode_name": "-",
        "Z": "0",
        "rejected": "F",
        "TTLs": "-",
        "qtype": "1",
        "qtype_name": "A",
        "query": "www.example.com",
        "type": "dns",
        "qclass": "1",
        "answers": "-",
        "proto": "udp",
        "RD": "T"
      }
    ],
    "id": {
      "resp_p": "53",
      "resp_h": "192.168.1.254",
      "orig_p": "42923",
      "orig_h": "10.10.10.17"
    },
    "timestamp": 1384833674270877,
    "uid": "000iFCzEelf"
  }
]

This is a work in progress, and really just a means to save my work.

I've managed to group the logs into an array, pulled out the "id" object, and the timestamp (according to conn log) and uid. I've also removed these items from the sublogs, with the exception of the timestamp, since I think it's plausible that you may have multiple entries in other logs over the course of a given connection, thus unique times.