Literals, such as booleans, integers, floats, and character strings are actually vectors of length 1 (a.k.a., scalars).
booleans:
true or false
integers (can have arbitrary _ as separators):
_1_, 45_, 1_000_000
floats
2.34, 3.14, -6.23e-32
Integer literals default to i32.
Float literals default to f64.
Using the i8, i16, i32, i64, and i128 suffixes switches to 8-bit, 16-bit, 32-bit, 64-bit, and 128-bit signed values.
There are also unsigned variants: u8, u16, u32, u64, and u128.
There are two special integer values: isize and usize that represent the native integer size on the platform.
In-memory collections will be indexed using usize and their counts will be usize.
Using the f32 and f64 suffixes specify 32-bit and 64-bit.
The char type is a 32-bit value representing a Unicode character.
The null literal represent an empty, immutable collection, of any type.
The as keyword converts a vector of one type to another (i.e., performs a cast):
42i32 as i64
Casts must always be explicit.
Character strings are surrounded by double quotes ("):
"Hello, world!"
Characters can be any valid UTF-8 unicode character sequence.
Escape sequences (e.g., \t, \n, \u0000) within r strings are ignored.
The r prefix can be used create multiline strings. The string ends when a corresponding number of quote symbols (") appear at the end.
r"Hello,
World
!!!
"
or
r"""Why so many?"""
or simply
r"Hello"
Character strings prefixed with an f are interpolated:
let name = "World"
f"Hello, ${name}"
The r and f prefixes can be combined.
Tuples are ordered, fixed-length types where each member can be a different type:
("Hello", 123)
A tuple is comparable (sortable) if all its elements are comparable.
Tuples are compared element-wise (a.k.a., lexicographically):
("a", 123) == ("a", 123)
("a", 123) < ("b", 123)
("a", 123) < ("a", 234)
Tuples can only be compared to other tuples with the same number and type of elements, where all elements are comparable.
The elements of a tuple can be accessed using .0, .1, etc. (0-based).
Entities are key/value pairs, where each member can be a different type.
Key names must be unique within entities:
{
a: 123,
b: "Hello",
c: false
}
Entity types can be declared:
let Thing = type {
a: i32,
b: String,
c: bool
};
Entities can be nested to represent complex entity graphs. Some stores support complex types - others do not.
A source can be read from. A source defines whether it can be read from one time (a stream) or multiple times.
All sources are associated with a record type.
An empty source can be constructed with the null keyword.
# Null can be any type, so we must declare applications is Application
let applications: Application = null;
All variables are 1-dimensional vectors, a.k.a., scalars. These represent in-memory sources:
let application = Dog { name: "Ralph" };
An appender can have new records written to it. Mutable, in-memory collections are appendable.
Editable, remote stores are also appendable. Appenders support "insert" operations.
Finally, stores represent in-memory or remote collections that can be modified in-place.
Stores support operations such as update and delete. Some stores are appenders, as well.
It's possible to have read-only stores, which are just exposed as sources.
It's possible to have a write-only store, where it can only be updated, appended to, or deleted from, but not read.
Example: Appending to an in-memory appender:
let things: Thing = mut []; # an empty mutable collection
select Thing { a: 1, b: "Hello", c: true } into things; # append 1
select Thing { a: 2, b: "Goodbye", c: false } into things; # append 2
from things as t
select t;
This can be more succinctly said as:
let thing1 = Thing { a: 1, b: "Hello", c: true };
let thing2 = Thing { a: 2, b: "Goodbye", c: false };
let things = [thing1, thing2];
from things as t
select t;
Example: select - get all applications (Applications is a data source)
type Application = {
id: i64,
name: String,
installDate: LocalDateTime,
publisher: String,
runCount: i32
};
# ...
from app_source as a
select a;
The distinct operation removes duplicates from the results.
Example: distinct - get unique applications:
from app_source a
distinct a
select a;
This is short-hand for a grouping operation:
from app_source as a
group by a as u
limit 1
aggregate u;
This operation is only valid if the records are comparable.
Example: distinct on - get unique installation dates:
from app_source as a
distinct on a.installDate
select a.installDate;
This is short-hand for a grouping operation:
from app_source as a
group by a.installDate as g
limit 1
aggregate g.installDate;
An order by clause can be used to control which record is retrieved when a limit is applied.
Specific fields can be returned:
from app_source as a
select { a.name, a.publisher, a.installDate };
A subset of fields can be given an alias -- treat it like an anonymous entity.
In the example below, b could then be filtered, mapped, etc.:
from app_source as a
select { a.name, a.publisher, a.installDate } as b
where b.publisher == "microsoft"
select b;
Selected fields can be aliased:
from app_source as a
select {
appName: a.name,
publisherName: a.publisher,
installationDate: a.installDate
};
We can filter out records in an entity set using where.
The where keyword can appear multiple times, anywhere in the query.
When two wheres appear next to each other, it's the same as and.
All operators have an equivalent prefixed by - (all custom operators start with -):
Comparisons:
==, <>, !=, <, <=, >, >=,
-eq, -ne, -lt, -le, -gt, -ge,
-ieq, -ine,
-contains, -icontains, -match, -imatch
Boolean logic:
&&, ||, !,
and, or, not
-and, -or, -not
Arithmetic:
+, -, *, /, // (integer-truncating division), ** (power), % (modulus)
-plus, -minus, -times, -div, -int-div, -pow, -mod
Bitwise:
<<, >>, >>>, ~, &, |,
-shl, -shr, -ushr, -compl, -bit-and, -bit-or
Value assertions:
is, is not
-is, -is-not
The is/is not operators are for checking for null and the special floating point value(s) NaN.
select null == null # null, not true
select null != null # null, not false
select NaN == NaN # false, not true
select NaN != NaN # true, not false
Example: select with filter - get applications published by Microsoft:
from app_source as a
where a.publisher == "microsoft"
select a;
The next two queries are equivalent:
from app_source as a
where a.publisher == "microsoft" and a.installDate > now()
select a;
from app_source as a
where a.publisher == "microsoft"
where a.installDate > now()
select a
Example: logical or:
from app_source as a
where a.publisher == "microsoft" or a.name == "Adobe Acrobat Reader"
select a;
Entities can be grouped together by specifying the fields to group by.
Example: Select with group by and "having":
# Get the number of applications per publisher,
# where there are more than 10 applications per publisher
# We must explicitly name the Count property since the name cannot be derived
from app_source as a
group by a.publisher as g
aggregate { g.publisher, count: count(g) } as pg
where pg.count >= 10
select pg.publisher, count: pg.count;
Example: group by multiple fields:
# Get the number of applications installed on a given day per publisher,
# where there are more than 10.
from app_source as a
group by a.publisher, a.installDate as g
aggregate { g.publisher, g.installDate, count: count(g) } as pg
where pg.count >= 10
select pg.publisher, pg.installDate, pg.count;
The aggregate operation is similar to select, except it works on the entire collection, not a single record. In the previous example, if select had been used instead, the call to count would have always returned 1 since a single Application would have been passed in. The aggregate call causes the entire group (publisher/install date group) to get passed to count instead. Notice that g.publisher and g.installDate are accessible in both situations.
Example: select with order by
from app_source as a
order by a.name
select a;
Example: compound order by
from app_source as a
order by a.publisher, a.name, a.installDate desc
select a;
You can also use the asc keyword to sort ascending (default). The placement of nulls can be controlled using nulls first or nulls last.
Example: paging - order by required
from app_source as a
order by a.name
select a
offset 20 # 0-based
limit 10;
Records can be added to an Appender. An Appender can be an in-memory store or wired up to insert into a remote store.
In the example below, assume app_appender is configured to point to an in-memory store with an Application record type.
Following any query with into [appender] will perform the equivalent of an INSERT.
Example: value/insert
# All primitives and objects are singleton collections (vectors of 1 dimension)
let singleton = Application { name: "Microsoft Word", publisher: "Microsoft", installDate: Today() };
from singleton as a
select a
into app_appender;
Example: insert/select - returns inserted record with all updated values (primary keys, etc.), if available.
from singleton as a
select a
into app_appender as a # pretend this returns the Application with a primary key, created by date, modified by date, etc.
select a
Records can be updated in a store. A store can be an in-memory data structure or remote.
Each store is associated with a record type (e.g., Application).
Example: select/update
# Fix bad installation dates by setting future dates to now.
from app_store as a
let now = now()
where a.installDate > now
update a { installDate: now };
Example: update/select - returns updated record with all updated values (defaults, etc.)
from app_store as a
let now = now()
where a.installDate > now
update a { installDate: now } # assume this returns back last modified date, etc.
select a;
Example: update based on values from the current record
from app_store as a
where a.name == "Microsoft Word"
update a { runCount: a.runCount + 1 }; # increment the number of times the application has ran
Example: select/delete
from app_store as a
where a.publisher == "microsoft"
delete a;
Example: Collections are concatenated using the append operation
let msApps =
from apps as a
where a.publisher == "microsoft"
select a;
let nsApps =
from apps as b
where b.publisher == "nullsoft"
select b;
from msApps append nsApps as c
select c;
Example: Collections are set unioned using the union operation
let msApps =
from apps as a
where a.publisher == "microsoft"
select a;
let nsApps =
from apps as b
where b.publisher == "nullsoft"
select b
from msApps union nsApps as c
select c
Except: The set difference can be computed using the except operation
let msApps =
from apps as a
where a.publisher == "microsoft"
select a;
let nsApps =
from apps as b
where b.publisher == "nullsoft"
select b;
from msApps except nsApps as c
select c
Example: Set symmetric difference isn't directly supported. It is equivalent to the following:
let msApps =
from apps as a
where a.publisher == "microsoft"
select a;
let recentApps =
from apps as b
where b.installDate >= Today()
select b;
from (msApps except recentApps) union (recentApps except msApps) as c
select c;
For forward-only collections, this operation may not be possible.
Example: Set intersection is performed using the intersect operation
let msApps =
from apps as a
where a.publisher == "microsoft"
select a;
let nsApps =
from apps as b
where b.publisher == "nullsoft"
select b;
from msApps intersect nsApps as c
select c;
For left, right and full joins, the entity on the left or right can be null.
When a collection may possibly be null, its elements must be accessed using the ?. operator.
However, within a join's on clause, the ?. is not necessary because it is handled implicitly.
Example: The join or inner join operation matches two collections by a condition
from apps as a
where a.publisher = "microsoft"
join appDependencies as ad on a.applicationId == ad.applicationId # many-to-many table
join dependencies as d on ad.dependencyId == d.dependencyId
select { appName: a.Name, depName: d.Name };
Example: The left join operation matches two collections, even if there's no matching record on the right
from apps as a
where a.publisher = "microsoft"
left join appDependencies as ad on a.applicationId == ad.applicationId # many-to-many table
left join dependencies as d on ad.dependencyId == d.dependencyId
select { appName: a.name, depName: d?.name }; # d can be null, so ?. is needed to traverse
Example: The on clause can contain multiple conditions
from customers as c
left join entityNotes as en on c.customerId == en.entityId and "Customer" == en.entityType
select { customerId: c.customerId, note: en?.content } # en could be null, so ?. is needed to traverse
Example: The right join operation matches two collections, even if there's no matching record on the left
from apps as a
where a.publisher == "microsoft"
right join appDependencies as ad on d.dependencyId == ad.dependencyId # many-to-many table
right join dependencies as d on ad.applicationId == a.applicationId
select { appName: a?.name, depName: d.name }
Example: The full outer join operation includes all records on the left of right, matching them if possible
from apps as a
where a.publisher == "microsoft"
full join appDependencies as ad on d.dependencyId == ad.dependencyId # many-to-many table
full join dependencies as d on ad.applicationId == a.applicationId
select { appName: a?.name, depName: d?.name }
Example: The zip operations combines corresponding elements from two collections, based on order of retrieval. If one of the collections is smaller than the other, only elements up to the smaller size are returned.
from apps as a
where a.publisher == "microsoft"
zip 0.. as i # 0 up to infinity
select { appName: a.name, index: i }
Reusable chunks of code can be placed within functions.
let greet = fn(name: String): String {
select f"Hello, {name}"
};
The last expression, if not followed by a ;, is treated as the result.
The return type (e.g., : String) is only required if there's an ambiguity.
Example: Recursive functions/queries
# Return types must be explicit for recursive functions.
let ManagedEmployee = type {
employeeId: i32,
name: String,
level: i32,
managerId: i32?, # The ? indicates it's optional
managerName: String?
};
let findEmployee = fn(employees: Employee, id: i32, level: i32 = 0): ManagedEmployee {
from employees as e
where e.employeeId == id
let managers = findEmployee(employees, e.managerId, level + 1)
left join managers as m on e.managerId == m.employeeId
select
{
e.employeeId,
e.name,
level,
managerId: m?.employeeId, # or e.managerId
managerName: m?.name
}
};
from findEmployee(employees, 123) as e
order by e.level
select { e.employeeId, e.name, e.managerId, e.managerName };
Example: Ranges can be used to create a collection of integers
# Produces the range between 1 and 9.
from 1..10 as v # same as [1, 2, 3, 4, 5, 6, 7, 8, 9]
select v;
Example: The ..= operator can be used to create an inclusive range
# Produces the range between 1 and 10.
from 1..=10 as v # same as [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
select v;
Example: A step can be provided
# Produces 2, 4, 6, 8, 10
from 2..=10..2 as v # same as [2, 4, 6, 8, 10]
select v;
The reduce operation can convert a collection into a scalar value, or another collection. Given a function called ifelse, here is a simple implementation of max:
from values as v # imagine values is a collection of numbers
select reduce(v, fn(x, y) { ifelse(x <= y, x, y) })
Another version of the reduce operation takes a starting value (a.k.a., seed):
from values as v
select reduce(v, 0, fn(x, y) { ifelse(x <= y, x, y) })
Example: exists/some/any
from apps as a
where a.publisher == "microsoft"
aggregate any(msApps);
Example: count
from apps as a
where a.publisher == "microsoft"
aggregate count(a);
Example: first
from apps as a
where a.publisher == "microsoft"
aggregate first(msApps); # returns empty if no such element
Example: The ?? operator can be used to return a default if no elements are present
from apps as a
where a.publisher == "microsoft"
aggregate first(a.name) ?? "";
Example: last
from apps as a
where a.publisher == "microsoft"
aggregate last(a); # returns empty if no such element
Example: The ?? operator can be used to return a default if no elements are present
from apps as a
where a.publisher == "microsoft"
aggregate last(msAppNames) ?? "";
Concatenating a string involves calling append on two strings.
select "Hello" append "World"; # produces "HelloWorld"
This is because a string is a collection of characters.
To create a collection of strings, the strings must be wrapped in collections:
let greeting = ["Hello"] append ["World"]; # produces ["Hello", "World"]
from greeting as g
select g;
This is the same as just saying:
from ["Hello", "World"] as s
select s;
Example: Collections can be created using the [] syntax:
let numbers = [1, 2, 3, 4, 5];
By default, collections are immutable. Splice operations can be used to create new collections: Example: Replacing an element
let numbers = numbers[..2] append [8] append numbers[3..];
Notice we are shadowing the original numbers. In some situations, the compiler might detect the original numbers is no longer in scope and replace this operation with an in-place mutation.
This can also be done using an update statement, if the collection is mutable:
let numbers = mut 1..5; # the mut keyword tells the compiler this collection can be modified
from number as n
where n == 3
update n { [n]: 8 }; # [n] tells the compiler the index of the value being replaced
Values can be deleted using the slice syntax or a delete statement
let numbers = 1..5;
let numbers = [..2] append [3..];
or
let numbers = mut 1..5;
from numbers as n
where n == 3
delete n;
Traits provide information during compilation to enable features on collections, or control how particular behaviors are executed.
Primitive- a primitive type (e.g.,i32,u64,char,f64,bool, etc.).Composite- a type that contains properties (or an emptytype).Nested[T] : Composite- indicates that a type has one or more nestedCompositetypes.Comparable[T]- indicates that a type can be compared or sorted.Iterable[T]- indicates that a collection can be read, sequentally, at least one time.Finite[T]- indicates whether a collection has a fixed number of values.Infinite[T]- indicates that a collection is potentially infinite.Appendable[T]- indicates that a collection can be appended to.Updatable[T]- indicates that values in a collection can be updated.Prunable[T]- indicates that values in a collection can be deleted.Grouping[T]: Iterable[T]- a group of records from agroup byclause.Mapping[K, V]- an collection of key/value pairs. The keys must be the same type. The values must be the same type.Ordinal[T]: Mapping[usize, T]- a collection that can be indexed by offset.
Custom traits can also be created:
let Fruit = type {
shape: String,
color: String
};
Traits support a concept called "structural typing". Types can declare that they extend a trait explicitly. Traits can also extend other traits. Any object with the same properties specified by a trait are implicitly instances of that trait, even if their types don't explicitly extend that trait.
# Apple is a "type of" Fruit with an extra "name" property
let Apple = type Fruit {
name: String;
};
# Each apple instance must provide all properties
let apple = Apple {
shape: "round",
color: "red",
name: "Empire"
};
# Since only shape and color are selected, fruits is still Fruit, but no longer Apple.
let fruits =
from apple as a
select { a.shape, a.color };
The object selected (i.e., { a.shape, a.color }) is an anonymous type. The fruits variable could have been initialized as let fruits: Fruit = ... and the compiler could have enforced that the selected object had the necessary properties. When defining functions, an explicit return type must be provided if selecting an anonymous type is the final expression.
An object can combine traits using the + operator.
Example: Defining a function that only works with an indexed collection of primitives:
let firstPrimitive = fn(values: Primitive + Indexable): Primitive {
first(values)
};
# Legal: firstPrimitive([1, 2, 3]) # i32 is primitive
# Illegal: firstPrimitive(["Hello", "World"]) # String is not primitive!
Within a type, a value can be marked optional using the ? suffix after the type. Both primitives and composite types must be marked optional if their values can ever be missing.
When a value is null, it means it is a vector (or collection) with 0 elements. The following code is a no-op, but is perfectly legal:
let applications: Application = null;
from applications as a
select a;
This is similar to the following:
let application = Application { name: "Word", publisher: "Microsoft" };
from application as a
where 1 != 1 # never true - filters out all applications
select a;
The null keyword is equivalent to [] and has the same restrictions. However, [] is reserved to represent in-memory collections and can be prefixed by mut to make it editable, whereas null cannot.
A null must always be accounted for. When accessing a nullable composite's properties, they must be accessed using ?.. This is common when performing left, right, and full joins.
A null can be subtituted with another value using ??.
let Horse = type {
name: String? # The question mark says null is a legal value
age: i32 # The age is not optional
};
let horses: Horse? = [ # The question mark here says the collection can contain null horses
{ name: "Ed", age: 12 },
{ name: null age: 32 },
null
];
from horses as h # h is Horse?
let name = h?.name # name is String?
let age = h?.age # age is i32? because h is optional
where count(name ?? "") != 0
select h;
Once a value is checked for null, ?. and ?? are no longer necessary within the scope of an expression:
from horses as h # h is Horse? here
where h is not null # after this condition, h is Horse
let name = h.name # name is still String?
let age = h.age # age is i32 because h cannot be null (compare to above)
where name is not null # after this condition, name is String, but h.name is still String?
where count(name) != 0 # count here is getting number of characters in name
select h;
A window is a moving group of rows, with an offset in relation to the current row. For example, window rows -2..0 creates a window including the previous two rows and the current row, where windows rows 0.. includes the current row as well as all subsequent rows. Similar to group by, this splits a collection into multiple collections and must be aggregated to get back to a single collection. Following is an example of a cumulative sum (summing the current item with all previous items):
from values as v
window rows ..0 as w
aggregate { value: v, sum: sum(w) }
Of note, v is still available during and after a window, unlike a group by, which shadows any preceeding aliased collection. Within the scope of a window operation, a variable named row is available which is the index of the current row, with 0 being the current row and all other indexes being relative.
The keyword rows can be replaced with range.