There are many ideas of what we could change or where to go next, but those are often poorly thought out. The sheer volume of what we don't know about the existing system, good or bad, is so large we're working in the dark. This is a blindspot induced by the current state of the code base.
What we do know, with great certainty, is that we need tests that not only test things but also:
- teach the reader
- are a useful reasoning tools
- remember decisions of the past
- help uncover lost ideas
- shine a light on dark corners and issues
Going through the exercise of improving tests will address the blindspot and is going to teach all of us where to go next and which ideas are more likely to get us there.
tl;dr: write tests that describe language features in great detail, along with commentary for language implementators and users -- managing all this as code
A language specification defines what's allowed, disallowed, undefined, guaranteed, not guaranteed, and known failure modes. More than a written document, our language specification is executable that, to the maximum extent possible, performs automated conformance testing of candidate compilers. Along with test code codifying the specification, technical commentary describes to the reader the detailed aspects of the language. The scope of the specification is the language and potentially any compiler interfaces, but exclusive of the standard library unless necessary. The specification is structured such that concepts are introduced as late as possible, leading to a natural layering of language facilities that build upon the previous. This layering allows for incremental learning for author or consumer and natural path of progress for any compiler implementation as a software engineering project.
The above is a highlevel description of the end goal insofar as strictly technical artifact and its properties. The specification is a tool within a continuously improving development cycle that reinforces quality, clear communication, development speed, and bold experimentation.
The biggest benefit will be upon conclusion of the project, but as it progresses the intended use of the specification is as follows:
- document the language both as it is and make early decisions of what to no longer support due to issues elucidated through attempts at codification and testing
- identify bugs, and if relatively straightforward, resolve them as the appropriate behaviour is specified
- using the logical progression of concepts in the language, layering them upon each other in a progressive fashion, we will highlight the architecture of language features and spot potential simplifications of the language itself
- provide an entry point for those wishing to contribute by way of clarifying the specification
- and much more
After the language specification is complete, the specification will be the undeniable source of truth and at the center of many or our processes:
- reduce the burden of contribution and maintenance - allow compiler developers to freely refactor and make bold changes with a strong assurance that the specification will be automatically checked
- ongoing simplicity and quantifiable debate - using the specification and seeing the impact of changes we can quantify the merit and simplicity of various proposals
- less RFCs and other heavy processes and more code - proposal of new features must work their way through the specification itself, allow the proposer, their collaborators, and others to see the impact not simply in some giant RFC, but executable code
- regression tests - as part of our CI we'll hae a continuous regression tests suite
- clear communication of changes - specifications can be diffed, older ones run against newer compilers and vice versa; tagging, reviews, and all the other tools many are familiar with are easily leveraged
- teaching and education* - contributors and programmers alike can easily and gradually learn the language itself by way of the specification; not to the exclusion of our teaching materials of course
What are some examples to follow?
See the following:
- Code: https://github.com/nim-works/nimskull/tree/devel/tests/lang - implement language specification in form of proper tests
- PR: Specification tests PR1
- PR: Specification tests PR2
- PR: Specification tests PR3