1
0
dust/README.md

223 lines
8.9 KiB
Markdown
Raw Permalink Normal View History

2023-08-22 15:40:50 +00:00
# Dust
2023-10-05 19:27:08 +00:00
Dust is a data-oriented programming language and interactive shell. Dust can be used as a replacement for a traditional command line shell, as a scripting language and as a data format. Dust is expression-based, has first-class functions, lexical scope and lightweight syntax. Dust's grammar is formally defined in code and its minimalism is in large part due to its tree sitter parser, which is lightning-fast, accurate and thoroughly tested.
2023-08-22 15:40:50 +00:00
A basic dust program:
```dust
2023-10-05 19:27:08 +00:00
output { "Hello world!" }
2023-08-22 15:40:50 +00:00
```
2023-08-23 04:38:51 +00:00
Dust can do two (or more) things at the same time with effortless concurrency:
2023-08-22 15:40:50 +00:00
```dust
2023-10-05 15:39:19 +00:00
run <
function { output 'will this one finish first?' }
function { output 'or will this one?' }
>
2023-08-22 15:40:50 +00:00
```
2023-08-23 04:38:51 +00:00
Dust can do amazing things with data. To load CSV data, isolate a column and render it as a line plot in a GUI window:
2023-08-22 15:40:50 +00:00
```dust
2023-09-11 19:23:13 +00:00
read_file("examples/assets/faithful.csv")
2023-10-05 15:39:19 +00:00
-> from_csv
-> rows
2023-10-05 16:04:02 +00:00
-> transform <{item.1}>
2023-10-05 15:39:19 +00:00
-> plot
2023-08-22 15:40:50 +00:00
```
2023-10-05 19:27:08 +00:00
Dust is also a minimal, obvious data format. It is easier to write than JSON and easier to read than TOML and YAML. However, because it is a programming language, it is able to self-reference, perform calculations or load external data.
```dust
foo = "bar"
numbers = [1 2 3 4]
truths = {
dust = "the best thing ever"
favorite_number = numbers.3
another_number = numbers.0 + numbers.1
}
old_faithful_data = read_file { "faithful.csv" }
```
2023-08-23 21:28:19 +00:00
<!--toc:start-->
- [Dust](#dust)
- [Features](#features)
- [Usage](#usage)
- [Installation](#installation)
2023-10-05 16:04:02 +00:00
- [Implementation](#implementation)
2023-08-28 22:05:49 +00:00
- [Contributing](#contributing)
2023-08-23 21:28:19 +00:00
- [The Dust Programming Language](#the-dust-programming-language)
2023-10-05 16:04:02 +00:00
- [Declaring Variables](#declaring-variables)
2023-10-05 15:39:19 +00:00
- [Integers and Floats](#integers-and-floats)
2023-08-23 21:28:19 +00:00
- [Lists](#lists)
- [Maps](#maps)
- [Tables](#tables)
- [Functions](#functions)
2023-10-05 15:39:19 +00:00
- [Empty Values](#empty-values)
2023-08-23 21:28:19 +00:00
<!--toc:end-->
2023-08-22 15:40:50 +00:00
## Features
2023-10-05 15:39:19 +00:00
- Simplicity: Dust is designed to be easy to learn and powerful to use, without compromising either.
- Speed: Dust is built on [Tree Sitter] and [Rust] to prioritize performance and correctness.
- Data format: Dust is data-oriented, so first and foremost it makes a great language for defining data.
2023-08-23 04:38:51 +00:00
- Pipelines: Like a pipe in bash, dust features the yield `->` operator.
- Format conversion: Effortlessly convert between dust and formats like JSON, CSV and TOML.
- Structured data: Dust can represent data with more than just strings. Lists, maps and tables are easy to make and manage.
2023-08-22 15:40:50 +00:00
## Usage
Dust is an experimental project under active development. At this stage, features come and go and the API is always changing. It should not be considered for serious use yet.
2023-08-28 22:05:49 +00:00
To get help with the shell you can use the "help" tool.
```dust
2023-10-05 15:39:19 +00:00
help # Returns a table will all tool info.
2023-10-05 19:27:08 +00:00
help {"random"} # Returns a table with info on tools in the specified group.
2023-08-28 22:05:49 +00:00
# The above is simply a shorthand for this:
2023-10-05 19:27:08 +00:00
help -> where { input, function <tool> { tool == "random" } }
2023-08-28 22:05:49 +00:00
```
2023-08-22 15:40:50 +00:00
## Installation
2023-08-28 14:18:55 +00:00
You must have the default rust toolchain installed and up-to-date. Install [rustup] if it is not already installed. Run `cargo install dust-lang` then run `dust` to start the interactive shell. Use `dust --help` to see the full command line options.
To build from source, clone the repository and run `cargo run` to start the shell. To see other command line options, use `cargo run -- --help`.
2023-08-22 15:40:50 +00:00
2023-10-05 16:04:02 +00:00
## Implementation
Dust is formally defined as a Tree Sitter grammar in the tree-sitter-dust module. Tree sitter generates a parser, written in C, from a set of rules defined in JavaScript. Dust itself is a rust binary that calls the C parser using FFI. Dust does not use Javascript at runtime.
Tree Sitter generates a concrete syntax tree, which the Rust code maps to an abstract syntax tree by traversing each node once. Tree sitter is fast enough to be updated on every keystroke which is perfect for a data-oriented language like Dust because it allows only the relevant sections to be re-evaluated and the result displayed instantly.
2023-08-23 21:28:19 +00:00
## Contributing
2023-10-05 16:04:02 +00:00
Please submit any thoughts or suggestions for this project. For instructions on the internal API, see the library documentation. Implementation tests are written in dust and are run by a corresponding rust test so dust tests will be run when `cargo test` is called.
2023-08-23 21:28:19 +00:00
2023-08-23 04:38:51 +00:00
## The Dust Programming Language
2023-10-05 16:04:02 +00:00
It should not take long for a new user to learn the language, especially with the assistance of the shell. If your editor supports tree sitter, you can use [tree-sitter-dust] for syntax highlighting and completion support. Aside from this guide, the best way to learn dust is to read the examples and tests to get a better idea of what dust can do.
2023-08-22 15:40:50 +00:00
2023-10-05 16:04:02 +00:00
### Declaring Variables
2023-08-22 15:40:50 +00:00
2023-10-05 19:27:08 +00:00
Variables have two parts: a key and a value. The key is always a string. The value can be any of the following data types:
2023-08-22 15:40:50 +00:00
- string
- integer
- floating point value
- boolean
- list
- map
- table
- function
Here are some examples of variables in dust.
2023-08-23 04:38:51 +00:00
```dust
2023-10-05 15:39:19 +00:00
string = "The answer is 42."
integer = 42
float = 42.42
2023-10-05 19:27:08 +00:00
list = [1 2 string integer float] # Commas are optional when writing lists.
2023-10-05 16:04:02 +00:00
map = {
key = `value`
}
2023-08-22 15:40:50 +00:00
```
2023-10-05 16:04:02 +00:00
Note that strings can be wrapped with any kind of quote: single, double or backticks. Numbers are always integers by default. And commas are optional in lists.
2023-10-05 15:39:19 +00:00
### Integers and Floats
2023-08-22 15:40:50 +00:00
2023-10-05 15:39:19 +00:00
Integer and floating point values are dust's numeric types. Any whole number (i.e. without a decimal) is an integer. Floats are declared by adding a single decimal to or number. If you divide integers or do any kind of math with a float, you will create a float value.
2023-08-22 15:40:50 +00:00
### Lists
2023-10-05 15:39:19 +00:00
Lists are sequential collections. They can be built by grouping values with square brackets. Commas are optional. Values can be indexed by their position to access their contents. Their contents can be indexed using dot notation with an integer. Dust lists are zero-indexed.
2023-08-22 15:40:50 +00:00
2023-08-23 04:38:51 +00:00
```dust
2023-10-05 15:39:19 +00:00
list = [true 41 "Ok"]
2023-08-22 15:40:50 +00:00
2023-10-05 19:27:08 +00:00
assert_equal { list.0 true }
2023-08-29 02:07:20 +00:00
2023-10-05 15:39:19 +00:00
the_answer = list.1 + 1
2023-08-29 02:07:20 +00:00
2023-10-05 19:27:08 +00:00
assert_equal { the_answer, 42 }
2023-08-22 15:40:50 +00:00
```
### Maps
2023-10-05 15:39:19 +00:00
Maps are flexible collections with arbitrary key-value pairs, similar to JSON objects. Under the hood, all of dust's runtime variables are stored in a map, so, as with variables, the key is always a string. A map is created with a pair of curly braces and its entries and just variables declared inside those braces. Map contents can be accessed using dot notation and a value's key.
2023-08-22 15:40:50 +00:00
2023-08-23 04:38:51 +00:00
```dust
2023-10-05 15:39:19 +00:00
reminder = {
message = "Buy milk"
2023-10-05 19:27:08 +00:00
tags = ["groceries", "home"]
2023-10-05 15:39:19 +00:00
}
2023-08-22 15:40:50 +00:00
2023-10-05 19:27:08 +00:00
output { reminder.message }
2023-08-22 15:40:50 +00:00
```
### Tables
2023-10-05 19:27:08 +00:00
Tables are strict collections, each row must have a value for each column. If a value is "missing" it should be set to an appropriate value for that type. For example, a string can be empty and a number can be set to zero. Dust table declarations consist of a list of column names, which are identifiers enclosed in pointed braces. The column names are followed by a pair of curly braces filled with list values. Each list will become a row in the new table.
2023-08-22 15:40:50 +00:00
2023-08-23 04:38:51 +00:00
```dust
2023-10-05 19:27:08 +00:00
animals = table <name species age> {
2023-10-05 15:39:19 +00:00
["rover" "cat" 14]
["spot" "snake" 9]
["bob" "giraffe" 2]
}
2023-08-22 15:40:50 +00:00
```
2023-08-23 04:38:51 +00:00
Querying a table is similar to SQL.
2023-10-05 15:39:19 +00:00
2023-08-23 04:38:51 +00:00
```dust
2023-10-05 15:39:19 +00:00
names = select name from animals
youngins = select species from animals where age <= 10
2023-08-23 04:38:51 +00:00
```
2023-08-22 15:40:50 +00:00
2023-10-05 19:27:08 +00:00
The keywords `table` and `insert` make sure that all of the memory used to hold the rows is allocated at once, so it is good practice to group your rows together instead of using a call for each row.
2023-08-23 04:38:51 +00:00
```dust
2023-10-05 19:27:08 +00:00
insert into animals {
2023-10-05 15:39:19 +00:00
["eliza" "ostrich" 4]
["pat" "white rhino" 7]
["jim" "walrus" 9]
2023-10-05 19:27:08 +00:00
}
2023-08-22 15:40:50 +00:00
2023-10-05 19:27:08 +00:00
assert_equal { count { animals }, 6 };
2023-08-22 15:40:50 +00:00
```
### Functions
2023-10-05 19:27:08 +00:00
Functions are first-class values in dust, so they are assigned to variables like any other value. The function body is wrapped in single parentheses. To create a function, use the "function" keyword. The function's arguments are identifiers inside of a set of pointed braces and the function body is enclosed in curly braces. To call a fuction, invoke its variable name and use a set of curly braces to pass arguments (or leave them empty to pass nothing). You don't need commas when listing arguments and you don't need to add whitespace inside the function body but doing so may make your code easier to read. Use your best judgement, the parser will disambiguate any valid syntax.
2023-08-22 15:40:50 +00:00
2023-08-23 04:38:51 +00:00
```dust
2023-10-05 15:39:19 +00:00
say_hi = function <> {
2023-10-05 19:27:08 +00:00
output {"hi"}
2023-10-05 15:39:19 +00:00
}
2023-08-22 15:40:50 +00:00
2023-10-05 15:39:19 +00:00
add_one = function <number> {
number + 1
}
2023-08-22 15:40:50 +00:00
2023-10-05 19:27:08 +00:00
say_hi {}
assert_equal { add_one{3}, 4 }
2023-08-22 15:40:50 +00:00
```
2023-10-05 15:39:19 +00:00
This function simply passes the input to the shell's standard output.
2023-08-22 15:40:50 +00:00
2023-08-23 04:38:51 +00:00
```dust
2023-10-05 15:39:19 +00:00
print = function <input> {
2023-10-05 19:27:08 +00:00
output { input }
2023-10-05 15:39:19 +00:00
}
2023-08-22 15:40:50 +00:00
```
2023-10-05 15:39:19 +00:00
### Empty Values
2023-08-22 15:40:50 +00:00
2023-10-05 15:39:19 +00:00
Dust does not have a null type. Instead, it uses the "empty" type to represent a lack of any other value. There is no syntax to create this value: it is only used by the interpreter. Note that Dust does have the NaN value, which is a floating point value that must exist in order for floats to work as intended. Integers will never be NaN and no value will ever be null or undefined.
2023-08-22 15:40:50 +00:00
[dnf]: https://dnf.readthedocs.io/en/latest/index.html
[evalexpr]: https://github.com/ISibboI/evalexpr
2023-08-28 14:18:55 +00:00
[rustup]: https://rustup.rs