Skip to content

Instantly share code, notes, and snippets.

@brabect1
Last active June 15, 2024 14:33
Show Gist options
  • Save brabect1/78bf524336f1e1af8bff37f6682c86fa to your computer and use it in GitHub Desktop.
Save brabect1/78bf524336f1e1af8bff37f6682c86fa to your computer and use it in GitHub Desktop.
Using Tcl Huddle serialization format. #tcl #huddle

Tcl Huddle

Tcllib's ::huddle is a native serialization format and is part of its YAML package. ::huddle's function is the same as of JSON or YAML; that is to augment commond data types along with the data. When JSON or YAML get parsed into an interpretter, individual data get represented by its native type; that is, string becomes a string, list becomes a list, map becomes a map/hash/dictionary, etc.

In Tcl, the problem is that all these types are mostly interchangeble (which is normally considered a feature) and cannot be distinguished. That is, by de-serializing a JSON or YAML to Tcl's native representation would lose the type information that is critical for serializing the structure again.

... TODO: example

... TODO: alternatives ton, Alternative JSON, likely more ... huddle provides type customization

Using Huddle

Surprisingly there are not many examples besides those in Tcllib's manual.

... TODO more examples

Creating data types is as simple as calling huddle <typename> <value>. This comes from how the huddle::addType works, which creates an ensemble command <typename> under the huddle namespace. Hence creating huddle objects looks like:

# Creating scalar data types
set hstring [huddle string "foo bar"]
set hnum [huddle number 1.23]
set hbool [huddle boolean true]
set hnull [huddle null]

# creating empty list
set hlist [huddle list]

# creating empty dict
set hdict [huddle create]
#set hdict [huddle dict]; # ERROR, `dict` is not an ensemble command

Creating data types using huddle compile:

# Creating various scalar data types
set hstring [huddle compile string "foo bar"]
set hnum [huddle compile number 1.23]
set hbool [huddle compile boolean true]
set hnull [huddle compile null ""]

# Note: For the `boolean` type, one can use any of the usual boolean
# representations. E.g. true can be expressed as `true`, `True`, `yes`,
# `1`, etc.

# Creating lists
set hstring_list [huddle compile list {foo bar}]
set hstring_list [huddle comile {list string} {foo bar}]
set hnum_list [huddle compile {list number} {1 2 3}]
set hbool_list [huddle compile {list boolean} {0 0 1 0}]

# Creating dictionaries
# Note: The dictionary type specification (i.e. the 2nd argument to
# `huddle compile <spec> <val>`) represents a map of key-type pairs,
# where key is a glob pattern for keys. Normally one uses `*` for
# dictionary where all values have the same type. The `<spec>` then
# looks like `{dict <key1> <type1> ... <keyN> <typeN>}`.
set hstring_dict [huddle compile {dict * string} {a foo b bar}]
set hnum_dict [huddle compile {dict * number} {a 1 b 2 c 3}]
set hbool_dict [huddle compile {dict * boolean} {a 0 b true c NO}]

# Creating more complex structures
set hstruct [huddle compile {list list string} {{foo bar} {a b c}}]; # list of string lists
set hstruct [huddle compile {list list number} {{1 2} {1.1 1.2 1.3}}]; # list of number lists
set hstruct [huddle compile {dict * {list number}} {a {1 2} b {1.1 1.2 1.3}}]; # dict of number lists
set hstruct [huddle compile {dict b number c {list boolean} * string} {a "foo bar" b 1 c {0 0 1 0} d "hello world"}];

Buidling a huddle structure bottom up:

# create individual leaf types
set hstring ...
set hnum ...
set hbool ...

# create a list object
set hlist {HUDDLE {L {}}}; # manually create an empty list (may use `huddle compile` or `huddle list`)
huddle append hlist $hstring
huddle append hlist $hnum
...
huddle set hlist 1 $hstring; # overwrite the item at index 1
...
${huddle::types(callback:L)} llength $hlist; # gets the length of `hlist`

# create a dict object
set hdict [huddle create]; # create an empty dict
huddle append hdict a $hstring b $hnum
huddle append hdict c $hlist
...
huddle append hdict a $hnum; # ovewrites key `a`
huddle set hdict a $hlist;   # another way to overwrite key `a`
...

Custom Huddle Types

The following example shows how to define a custom path type. For simplicity, our path is merely another string. For other examples refer to huddle.test testsuite.

Warning

::huddle v.0.3 has a bug in jsondump and the following code and the serialization would fail there. For details see tcllib's bug report https://core.tcl-lang.org/tcllib/tktview?name=d0e1cf6be1 This has been fixed in v.0.4.

package require huddle;

namespace eval ::myspc {
    namespace export *;

    namespace eval path {
        variable settings;

        set settings {
            publicMethods {path}
            tag p
            isContainer no
        };

        proc path {src} {
            return [::huddle::wrap [list p $src]];
        }

        proc equal {p1 p2} {
            return [expr {$p1 eq $p2}];
        }

        proc jsondump {data {offset "  "} {newline "\n"} {begin ""}} {
            # JSON permits only oneline string
            set data [string map {
                    \n \\n
                    \t \\t
                    \r \\r
                    \b \\b
                    \f \\f
                    \\ \\\\
                    \" \\\"
                    / \\/
                } $data
            ];
            return "\"$data\""
        }
    }

}

# create a simple map with one string and one path
set h {HUDDLE {D {str {s foo} pth {p bar}}}};
puts "\nh=$h";
huddle addType ::myspc::path
puts "jsondump(h)=[huddle jsondump $h]";
# Copyright 2022 Tomas Brabec
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import pathlib
import json
import argparse
import sys
def json2huddle(data):
if data is None:
return None
elif isinstance(data,str):
return "s {" + data + "}"
elif isinstance(data,(int,float)):
return "n " + str(data)
elif isinstance(data,bool):
return "b " + str(data).lower()
elif isinstance(data,list):
l = []
for i in data:
s = json2huddle(i)
if s is None: l.append("null")
else: l.append("{" + s + "}")
return "L {" + " ".join(l) + "}"
elif isinstance(data,dict):
l = []
for k, v in data.items():
l.append(k)
s = json2huddle(v)
if s is None: l.append("null")
else: l.append("{" + s + "}")
return "D {" + " ".join(l) + "}"
return "? {}"
# Instantiate the parser
parser = argparse.ArgumentParser(description='Optional app description')
parser.add_argument('files', metavar='file', type=pathlib.Path, nargs='*',
help='JSON files to validate')
opts = parser.parse_args()
if not opts.files:
s = None
data = json.load(sys.stdin)
s = json2huddle(data)
if s is None: s = "HUDDLE null"
else: s = "HUDDLE (" + s + ")"
print(s)
else:
for f in opts.files:
p = pathlib.Path( f )
if p.is_file():
t = p.with_suffix('.huddle')
print( str(t) )
s = None
with open(f, 'r') as myfile:
data=json.load(myfile)
#print(json.dumps(data, indent=2))
#print("----")
s = json2huddle(data)
if s is None: s = "HUDDLE null"
else: s = "HUDDLE {" + s + "}"
print(s)
if not s is None and not t.is_file():
with open(t, 'w') as myfile:
myfile.write(s)
# Copyright 2023 Tomas Brabec
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
package require json::write;
proc isdict {dict} {
expr { [catch { dict info $dict } ] ? 0 : 1 }
}
# modified version from https://wiki.tcl-lang.org/page/Tcllib+JSON
proc dict2json {dict} {
return [json::write object {*}[dict map {k v} ${dict} {
if {[isdict $v]} {
set v [dict2json $v];
} else {
if {[llength ${v}] > 1} {
set l {};
foreach r ${v} {
lappend l [json::write string ${r}];
}
set v [json::write array {*}${l}];
} else {
set v [::json::write string $v];
}
}
}]];
}
source huddle.tcl
parray ::huddle::types
puts "----"
foreach {k v} [namespace ensemble configure ::huddle -map] {
puts "## $k -> $v";
}
puts "------"
::huddle::Type_string create
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment