nqp::(un)pack proposal

unposal.md

my @items := nqp::unpack( 'U2 (xxxC)[W]', $binary_string );

I, FROGGS, solemnly propose the implementation of pack() and unpack in nqp.

For one, these are useful for Perl 5 and Perl 6, and the other reason is that they should be pretty fast. If a backend then can even provide a faster implementation (e.g. in C), then it would be easy to #ifdef it in.

The signatures I propose are:

my $str  := nqp::pack( "CCC", [65, 66, 67] );
my @list := nqp::unpack( "CCC", $str );

You might wonder how this will cope with Buf, Str, utf8 and what not. Since pack/unpack work on byte-level, and one must be able to extract a single byte of a utf8-byte-sequence, it gets a string of octets.

class utf8
    multi method unpack(utf8:D: $template) {
        nqp::unpack( $template, nqp::decode(self, 'ascii') );
    }
}

Implementaion details

I would implement it exactly like nqp::sprintf, because unpack supports group-repeatition and relative positioning inside groups.

What needs to be discussed is how diretives like I should be handled. I'd like to see some sort of dynamic variable, a hash, which can be set to specify how many bytes a native int should have when packing it into a string. Same technique would be useful for sprintf's platform specific directives.

<FROGGS> TimToady: how do you think about pack/unpack/sprintf? should they be as Perl5-ish as possible? (without loosing sanity)
<FROGGS> TimToady: or do you have something else in mind?
<TimToady> FROGGS: eventually I'd like to treat pack/unpack templates as a sort of assembly language, and compile a struct definition down to it for serializing, and then we'd have a more standard way to define eqv
<TimToady> so anything that's not inconsistent with that is fine
<FROGGS> hmmm, I fear I can't imagine what you meant by the first line
<TimToady> other than that, the whole point of pack/unpack is sort of serial violation the type system, so I'm not sure how you'd define "insane" there :)
<TimToady> but I'm okay with supporting most of what P5 does
<FROGGS> by (in)sanity I mean the way we factor the support of int sizes
<TimToady> well, int sizes are a storage type, so unpack doesn't have to return anything other than Int
<FROGGS> I guess a fixed width perl directive/letter would be fine
<FROGGS> yeah, but when you pack something you need to know how many bytes it should take
<TimToady> and that's where it'd be nice to be able to turn a class repr into a pack format
<TimToady> pack formats and struct defs are roughly isomorphic on this level
<TimToady> so it'd be nice not to have to duplicate that info
<TimToady> I think of the pack format as the "assembly language" end of it
[...] http://irclog.perlgeek.de/perl6/2013-09-06#i_7551485

TimToady: it took me a while, but is that what you think?

class Foo is repr('CStruct') {
    has uint8 $.foo;
    has   Str $.bar;
    has int32 @.baz;
}

say Foo.new( :foo(65), :bar<wesome!>, :baz([13, 10])).pack

OUTPUT: Awesome!␀␀␀␍␀␀␀␤

For unpack we could/should take sized arrays into account, so has uint8[10] $.x would replace C10.
P5 also support relative positioning to groups etc, how do we handle that?
Or better: How we handle groups at all? Like has Dog $.norbert?

http://www.catb.org/esr/structure-packing/