macOS-Wireshark-luagcrypt.md

AES decryption in a Lua plugin of Wireshark on Mac

Confirmed environment

Item	version
OS	macOS Catalina
Wireshark	3.6.2
Lua for building luagcrypt	5.2.4
luagcrypt	https://github.com/Lekensteyn/luagcrypt 5763e94b751e6ff0dcf20e328c299d49eb8fcbce
libgcrypt	stable 1.10.0 (bottled)

Installing Wireshark

1. Download dmg file from https://www.wireshark.org/#download
2. Open the dmg file and follow the instruction.

Installing libgcrypt

$ brew install libgcrypt

Installing Lua for building luagcrypt

$ anyenv install luaenv
$ (exec $SHELL -l)
$ luaenv install 5.2.4

Lua 5.2.X is the version Wireshark 3.6.2 uses.

Installing luagcrypt

1. Download source files.

$ git clone https://github.com/Lekensteyn/luagcrypt.git

2. Customize Makefile.

$ cd luagcrypt
$ vi Makefile

--- a/Makefile
+++ b/Makefile
@@ -1,9 +1,9 @@
-LUA_VER     = 5.2
+LUA_VER     =
 LUA         = lua$(LUA_VER)
-LUA_DIR     = /usr/local
-LUA_INCDIR  = $(LUA_DIR)/include/lua$(LUA_VER)
+LUA_DIR     = ${HOME}/.anyenv/envs/luaenv/versions/5.2.4
+LUA_INCDIR  = $(LUA_DIR)/include/

-LUA_LIBDIR  = $(LUA_DIR)/lib/lua/$(LUA_VER)
+LUA_LIBDIR  = $(LUA_DIR)/lib/

 CFLAGS      = -Wall -Wextra -Werror=implicit-function-declaration
 CFLAGS     += -O2 -g -I$(LUA_INCDIR)
@@ -31,7 +31,7 @@ clean:
        $(RM) luagcrypt.so luagcrypt.gcda luagcrypt.gcno luagcrypt.o luagcrypt.c.gcov

 install: luagcrypt.so
-       install -Dm755 luagcrypt.so $(DESTDIR)$(LUA_LIBDIR)/luagcrypt.so
+       install -m 755 luagcrypt.so $(DESTDIR)$(LUA_LIBDIR)/luagcrypt.so

 checkcoverage:
        $(MAKE) -s clean

3. Build luagcrypt

$ make

4. Install luagcrypt library

Please tell me if you know better way to do it.

$ mkdir /usr/local/lib/lua 
$ cd /usr/local/lib/lua
$ ln -s $HOME/.anyenv/envs/luaenv/versions/5.2.4//lib/lua 5.2
$ make install

Example

You need to put the script file into $HOME/.config/wireshark/plugins/

-- [[
    MIT License
    Copyright 2022 yasuyuki.kozakai@gmail.com
]]

local gcrypt = require("luagcrypt")

function decrypt(key, iv, data)
    local cipher = gcrypt.Cipher(gcrypt.CIPHER_AES256, gcrypt.CIPHER_MODE_CBC)
    cipher:setkey(key)
    cipher:setiv(iv)
    return cipher:decrypt(data)
end

foo = Proto("foo","Foo Protocol")
foo.fields.iv  = ProtoField.bytes("foo.iv","IV", base.SPACE)
foo.fields.len = ProtoField.uint16("foo.len","Len")
foo.fields.enc_data = ProtoField.bytes("foo.enc","Encrypted", base.SPACE)
foo.fields.dec_data = ProtoField.bytes("foo.dec","Decrypted", base.SPACE)

function foo.dissector(buffer, pinfo, tree)
    pinfo.cols.protocol = "Foo"

    local key = '\x00\x01\x02\x03\x04\x05\x06\x07\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f' ..
                '\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f'

    local subtree = tree:add(foo, buffer(), "Foo Protocol")
    local iv = buffer(0, 16)
    local len = buffer(16, 2):uint()
    local encrypted = buffer(18, len)
    local plaintext = decrypt(key, iv:raw(), encrypted:raw())
    local decrypted = ByteArray.new(plaintext, true):tvb("Decrypted data")

    subtree:add(foo.fields.iv, iv)
    subtree:add(foo.fields.len, len)
    subtree:add(foo.fields.enc_data, encrypted)
    subtree:add(foo.fields.dec_data, decrypted())
end

udp_table = DissectorTable.get("udp.port")
udp_table:add(50000, foo)