Skip to content

Instantly share code, notes, and snippets.

@raiph

raiph/.md

Last active May 25, 2021
Embed
What would you like to do?
Raku's "core"

Then mathematical neatness became a goal and led to pruning some features from the core of the language. ... Another way to show that LISP was neater than Turing machines was to write a universal LISP function and show that it is briefer and more comprehensible than the description of a universal Turing machine.

— John McCarthy, History of Lisp

TL;DR Like Lisps, Raku is built on top of a tiny general purpose programming core. Unlike Lisps, Raku's core models a universal object atop a turing machine. This article drills down to that core: a metamodel ("model of a model") of a model of computation ("how units of computations, memories, and communications are organized").

Why I wrote this article

This article began with u/faiface's reddit "I'm impressed with Raku"1. One sentence in particular stood out for me:

I still generally prefer languages with a small, orthogonal core

I took this to mean they thought Raku did not have a small, orthogonal core. So I wrote a personal response to them that has morphed into this article.

I had fun writing the following; I hope you enjoy the journey and destination.

The journey begins

I still generally prefer languages with a small, orthogonal core

My original version of this article began:

Me too from many standpoints including: initial attraction and comfort zone; instinctual sense of formal aesthetics; deep debugging of a thorny problem; modifying a language; working on its compiler; and discussing a language's bowels with folk pointing out they prefer a small core. :)

But I didn't elaborate on any of those notions. Attraction, comfort, and aesthetics are about subjective experience. Debugging and working on a compiler are more tractable but I skipped those topics too.2

Instead, this article focuses on drilling down through Raku's layers to its core:

  • Unwrapping Rakudo Star Stripping away trimmings such as the standard library.

  • What is the Raku language? Narrowing focus to Raku's fixed syntax and semantics. And discovering there aren't any.

  • What is Raku's "single semantic model"? If Raku has no fixed syntax or semantics, what does it have?

  • What is nqp? Rakudo, the Raku compiler, is built atop a subset of Raku named nqp. But a subset of nothing remains nothing...

  • What's the core of MoarVM? MoarVM is short for "Metamodel on a runtime VM". What's that about?

  • What does KnowHOW know how to do? Describing the core.

  • So, does Raku have "a small, orthogonal core"? I say both yes and no. What do you think?

Feel free to start your journey wherever you prefer -- but as your tour guide I recommend we start at the start.

Unwrapping Rakudo Star

As with any programming language, there are cores within cores within cores until you hit electrons. Let's start by finding Raku's equivalent of a standard library so we can distinguish features bootstrapped from the base language from that underlying base.

The package named "Rakudo Star" is the standard "batteries included" distribution offered to newbies. It's a big set of libraries (and docs, and tools, etc.), not just a standard library.

A small part of Rakudo Star is the Rakudo compiler package. This includes the standard library but also more than that. For example, the Rakudo compiler package includes the Test module. But you have to explicitly import it (by writing use Test;) to use its features. Many PLs require such explicit importing even for standard library functions, but Raku makes things easier than that.

A canonical example of a feature that is available without explicitly importing anything is print, which displays a string:

print 42; # 42

We can ask a function to display what source file it's declared in by calling the .file method on the function's symbol table entry, which for a subroutine is its name prefixed with &:

print &print.file ; # SETTING::src/core.c/io_operators.pm6

So the code that implements print is stored in a folder named core.c. That sure sounds core-ish. But Raku bootstraps as a usable programming language -- turing complete but also well beyond that -- without print.

What we have uncovered is Raku's equivalent of a standard library (or rather a prelude).3


So, if we put the standard library aside, and focus on what's left, we've arrived at the Raku language proper, right?

What is the Raku language?

Quick apparent tangent. Raku lets devs add operators. Here's a factorial operator:

sub postfix:<!> (Int \n where * > 0) { n == 1 ?? 1 !! n * (n-1)! }
              ^                                                ^

Note how this postfix operator is added to the language at the time the compiler parses the > (indicated with the first ^) so it's already recognized and working by the time the compiler encounters it in code (indicated by the second ^).

The following line yields an error at compile-time when placed before the above sub declaration. But when placed afterwards, the code compiles (and therefore runs) correctly:4

print 5! # 120

While not all of Raku is modifiable using such simple code, every aspect -- syntax and semantics -- is modifiable. Unlike most PLs, Raku was built with the notion of a systematically mutable language as its bedrock.


Also unlike most PLs, Raku is actually a braid of sub-languages, plural, rather than just a singular language.

Devs can replace or alter the syntax or semantics of any of its sub-languages, add new ones -- several Raku modules do exactly that -- or even remove sub-languages. Indeed devs can remove all of the Raku sub-languages if they want to go to extremes, yielding a language with zero surface syntax so that no program will compile, not even a blank one.

Where's the core in a Ship of Theseus that can vanish?


Quoting Larry from his 2001 Apocalypse #1:

First, Raku will support multiple syntaxes that map onto a single semantic model. Second, that single semantic model will in turn map to multiple platforms.

Raku's core is its "single semantic model".

What is Raku's "single semantic model"?

I asked the question "What is the “semantic model” introduced in Apocalypse #1?" on StackOverflow as part of writing this article.5

The answer by jnthn, the lead dev of the Rakudo compiler toolchain covered a range of options, all of which are relevant to our quest for Raku's true core, but here I'll quote just his last one:

We could see RakuAST as an alternative syntax for Raku expressed as an object graph. Given it will also be the representation the compiler frontend uses for Raku code, we can also see it as a kind of syntax-independent gateway to the Raku semantic model.

So, while jnthn has written "RakuAST will be found at the very heart of Rakudo", it will still only be a "syntax-independent gateway" to what we seek.

Next, we drill down to what both RakuAST and the existing "old" AST layer (QAST) target: nqp.

What is nqp?

nqp is:

  • An inner doll within Raku's "Matryoshkas of Theseus".6

  • A programming language focused on constructing and compiling programming languages.7

In one sense nqp's semantics are determined by nqp and the NQP VM that serves both Raku and nqp. But NQP is just an abstract VM. And it's written in nqp. Furthermore, nqp is written in nqp too. So neither nqp nor NQP can really help us out.

To continue or journey to the center of Raku, we note that running Rakudo, or nqp, means running it with a selected backend, and it's that backend that's actually running code on the underlying platform. So we'll look at that.

nqp/NQP/Rakudo experimentally just about support JVM and JS backends, but we're going to look at the production backend: MoarVM8.

What's the core of MoarVM?

At the start I wrote:

Like Lisps, Raku is built on top of a tiny general purpose programming core. Unlike Lisps, Raku's core models a universal object atop a turing machine. This article drills down to that core: a metamodel ("model of a model") of a model of computation ("how units of computations, memories, and communications are organized").

Raku's metamodel is known as 6model. It is built atop existing platform object systems for the JVM and JS backends, but is implemented directly on the metal in carefully written C89 code in MoarVM.

At the core of this metamodel is a single data structure (a C struct in MoarVM) with associated code. This structure is a self-describing singleton "thing" called a KnowHOW. On MoarVM it's about 30 lines of C code.

What does KnowHOW know how to do?

Let's momentarily zoom out to the 30,000 foot view and then, with all the setup done so far in this nearly finished article, rapidly drill back down.

We can zoom out and then back in with just three lines of Raku code:

say Mu.HOW.^name ;         # "Perl6::Metamodel::ClassHOW"  30,000 foot view -- Raku
say Mu.HOW.HOW.^name ;     # "NQPClassHOW"                 Virtual VM view -- nqp/NQP
say Mu.HOW.HOW.HOW.^name ; # "KnowHOW"                     Low-level backend -- MoarVM
  • Mu.HOW is the "Higher Order Workings" object for Raku's Mu type object. Perl6::Metamodel::ClassHOW is the name of the nqp class of the nqp object instance Mu.HOW that knows how to make a Raku class work. For example it implements, in nqp, calling a method on an instance of a Raku class.

  • Mu.HOW.HOW is another nqp object that knows how the Mu.HOW object works. For example it implements, in nqp, calling a method on an instance of an nqp class.

  • Mu.HOW.HOW.HOW is the single data structure at the core of 6model. On MoarVM this is the C struct plus code that knows how the 6model C struct in MoarVM works. For example it implements, in C, calling a function associated with a 6model data structure (aka a "method", though whether "objects" and "methods" exist at this bootstrap stage is like asking which came first, the chicken or the egg).

Have we arrived at the core?

In one respect, no. Because this is just C code that targets underlying hardware.

But for the purposes of this article, we have the following satisfying result:

say Mu.HOW.HOW.HOW.HOW.^name ;           # KnowHOW
say Mu.HOW.HOW.HOW.HOW.HOW.^name ;       # KnowHOW
say Mu.HOW.HOW.HOW.HOW.HOW.HOW.^name ;   # KnowHOW
...

That is to say, calling .HOW on Mu.HOW.HOW.HOW returns its invocant, i.e. itself, i.e. this inner KnowHOW knows how it itself works. More concretely, it includes a function written in C that implements calling a function on all 6model objects, which by definition includes calling a function on itself. It also includes a slot declaring its type, and another declaring its type's constructor's kind, and both these slots point to itself.

So this ultimate KnowHOW -- the abstract conceptual singleton heart of 6model -- is Raku's true core, and there's a tiny concrete implementation of it in each backend.

So, does Raku have "a small, orthogonal core"?

The singleton KnowHOW "thing" clearly qualifies as "small" (a handful of lines of C code in MoarVM, somewhat more in the JVM and JS backends). The entirety of Raku is built up from (copies of) that one object.

But is it "orthogonal"?

Quoting Wikipedia's first sentence in the Computer Science section of its Orthogonality page:

Orthogonality in programming language design is the ability to use various language features in arbitrary combinations with consistent results.

Because Raku's tiny "universal object" core keeps direct access to its attributes entirely private it can serve as an Actor. This is why OO::Actors is just 30 or so LoC.

And given that an actor is a "universal primitive of concurrent computation", and concurrent computation is more general than sequential computation, having an actor-capable core means Raku supports "the ability to use various language features in arbitrary combinations with consistent results".

That said, Raku isn't ideological, so it's not an Actor only PL.

So, is it "orthogonal"?

I'll let you decide. :)

Footnotes

1 The 13th highest ever scoring post in /r/programminglanguages at the time it was posted.

2 One thing I've now elided from my original version of this article, but I want to keep here in this footnote for posterity is the comment "Which is more your type of poison, Kernel or Frank?"

3 The .c in core.c stands for Raku Christmas. Rakud Diwali, the second official major version of Raku was released in November 2018, and there's a corresponding core.d folder. The modules in core.d are lexically concatenated with the modules stored in core.c to form the lexical SETTING corresponding to Rakudo Diwali. cf Haskell's Prelude.

4 To confirm this for yourself, click on this tio.run link to see the above code failing at compile-time. Note how the say 'program starts to run'; line does not execute despite being the first line. Then cut the say 5! and paste it as the last line. Now the code works.

5 Only my 3rd ever SO question -- unlike my 333+ raku answers. In case it wasn't already obvious, I 💓 Raku. :)

6 nqp is a subset of Raku. It has the same braided architecture as Raku, but drops some of the sub-languages that Raku has. While its grammar (parsing) sub-language is a large chunk of Raku's (which inherits from it), nqp's other sub-languages are much smaller. nqp's equivalent of Raku's standard library is also tiny in comparison to Raku's.

7 For those interested in technical arcana, nqp is a bootstrapping, self-hosted, meta-compiler, a cross between a modern day retelling of META II, and a metaphorical ML for constructing and compiling languages with both static and dynamic features.

8 An Erlang/Elixir/BEAM enthusiast wrote a fairly popular brief intro to MoarVM. They described it as "a fantastic piece of technology".

@samebchase

This comment has been minimized.

Copy link

@samebchase samebchase commented Jul 15, 2020

I /love/ your SO answers. Have learnt a lot from them, thanks for all your efforts.

@uzluisf

This comment has been minimized.

Copy link

@uzluisf uzluisf commented Feb 3, 2021

Unfortunately I cannot say I understood everything but that's on me and I def need to reread it a few more times. However this is a highly informative write-up and I really liked the onion-layered view you gave of the language we know as Raku.

Yes, I've taken you on a long and perhaps silly journey. Hopefully you enjoyed reading it as much as I enjoyed writing it. :)

This is anything but silly. It suffices to say I always enjoy reading your write-ups about Raku and programming language design at large I come across on SO, Reddit, Github/Gitlab issues/gist, etc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment