On this page:
3.1 Create a New Definition File
3.2 The Usual Stuff
3.3 Declare the Version
3.3.1 Editions
3.3.2 Revisions
3.3.2.1 Revision Numbers
3.3.2.2 Revision Names
3.3.3 Optional Reading:   What About Semantic Versioning?
3.4 Declare Supported Racket Versions
3.5 Declare Supported Operating Systems
3.6 Package Inputs
3.6.1 Everything is an Input
3.6.2 Integrity Information
3.6.2.1 Creating an Integrity Expression
3.6.3 Authenticating Inputs
3.6.4 Abstract and Concrete Package Inputs
3.7 Package Outputs
3.7.1 Monadic Types
3.7.2 Adding a Second Output
3.7.3 Outputs Can Create Duplicate Data
3.8 User-defined Metadata
3.9 The Finished Definition
8.0

3 Defining Packages

Xiden builds software using package definitions. In this section we will write a package definition that, when installed, extracts one of two available archives.

We will cover why we can feel confident in the build due to the integrity checking and authentication steps that happen before we use the archives.

If you are already familiar with how package definitions work and just want an example to copy, then skip to The Finished Definition.

3.1 Create a New Definition File

Create a new blank Racket module. The name of that file is up to you, but I’ll use definition.rkt. We will use the xiden language.

"definition.rkt"

#lang xiden

3.2 The Usual Stuff

Every dependency management tool has you write the same things, so let’s start there. Since I’m hosting some files for this guide on my site, I’ll use related information.

You can define the name of your package, your identity as a provider, a short description of your package, tags, and so on. Everything in this example should not need much explanation.

"definition.rkt"

#lang xiden
 
(name "my-first-package")
(provider "sagegerard.com")
(description "Fun playtime in a tutorial")
(tags "fun" "tutorial" "example")
(url "https://sagegerard.com")

The provider definition is less obvious. A provider is not necessarily the author of the package, but rather a name of the party responsible for distribution. In this case, they are the same party.

There’s no restriction on how you name a provider, but a domain name is useful as a verifiable identifier when data is split across different networks.

3.3 Declare the Version

Next, let’s declare the version of our package.

"definition.rkt"

#lang xiden
 
(name "my-first-package")
(provider "sagegerard.com")
(description "Fun playtime in a tutorial")
(tags "fun" "tutorial" "example")
(url "https://sagegerard.com")
 
(edition "default")
(revision-number 0)
(revision-names "alpha" "2020-10-01")

But wait, nothing there looks like a version number. Xiden versions package definitions using editions and revisions, not major or minor version numbers. This means users can find software like they would a book. When defining a package, you may specify an edition, a revision number, and any revision names to act as aliases for that number. The revision names are freeform, but should relate to meaningful, unique stages in your project’s life.

3.3.1 Editions

An edition is a name for a design or target audience. Think of it as a semantic alternative to a major version number. When you wish to adapt your software to a different audience without disrupting existing users, change the edition.

Editions address technical and social concerns. They allow you to divide up your user base without changing the (branded) name of your software or the general value it offers. This may be preferable to forcing one implementation to accomodate everyone. You can use this as an excuse to refactor, rewrite, or delete swaths of code that are irrelevant to an audience. It also helps one avoid nonsensical numerical schemes that try to track breaking changes across all possible audiences.

Define editions sparingly and thoughtfully to keep maintenance costs manageable.

3.3.2 Revisions

A revision is a numbered implementation of an edition. Given an edition, a user can select a package definition using a revision number or a revision name.

3.3.2.1 Revision Numbers

A revision number is an exact non-negative integer.

If you are releasing a new package definition with the same edition, then increment the revision number. If you are starting a new edition, then the revision number must start at 0 as it does in the examples on this page.

3.3.2.2 Revision Names

A revision name is an alias for a revision number. A revision name can be any string that contains at least one non-digit.

A package definition may include a list of at least zero revision names for the revision number.

A revision name should be unique within an edition. If a user searches for a package definition using a revision name, and that name refers to more than one revision number in an edition, then the provider is responsible for correcting the ambiguity.

3.3.3 Optional Reading: What About Semantic Versioning?

Semantic Versioning (“SemVer”) allows you to infer what happened to software depending on which of three version numbers changed. Any assumptions about the change hold so long as the relevant package author follows SemVer rules closely and voluntarily. Since SemVer depends on ideal human compliance like every other scheme, I don’t think it contributes much in the end.

In my mind, a version is a form of identification. Jeff Atwood used the term “dog tag,” since we often ask a user what version they are using when our program fails out in the field. This came before the simple advice to just use a date or timestamp for each version.

This makes sense, but neither scheme helps users with discovery. No one knows if version 3.49.111 or Oct-10-2017 is a good fit for them without additional context. But even non-technical users would understand that they probably want a specific revision of a teacher’s edition. I don’t see why software should be any harder to evaluate than a book. If I later extended this idea to a versioning scheme that not only identified software, but captured the nature of all related changes, then I might tell you after consulting a patent attorney.

We can do without the bureaucracy and social pressure of rules like those defined in SemVer. The scheme defined on this page offers no pretense that you can look at two versions and know if the most recent one is backwards-compatible. All this scheme does is give you room to divide up your userbase into audiences and your product into designs. Then, you can chronicle releases in terms of your relationship with each audience. This can be messy, but that’s life. At least with this scheme you can decide what changes to make in terms of the people affected by those changes. That’s as good as it gets.

If you still prefer Semantic Versioning after reading all that, then you can always define an edition that uses semantic version strings as revision names and use a plugin that resolves Semantic Version queries. See Plugins for more information.

3.4 Declare Supported Racket Versions

Next, we can decide what versions of Racket we want to support if our package includes a Racket program. For that, we use racket-versions. When you run a package, Xiden will check if the running version of Racket is an element of the set defined by racket-versions. If it isn’t, that halts use of a package.

This example defines software that can run from Racket v6.0 to Racket v7.7.0.5. Each list of two versions is an inclusive interval, so support includes the shown versions.

(racket-versions ("6.0" "7.7.0.5"))

Gaps in versions are not expected due to Racket’s commitment to backward compatibility, but you can express them in the event one Racket version does not interact well with your release.

You can also declare version support as unbounded on one side of an interval using "*". This definition of racket-versions matches every version of Racket except those strictly between "7.2" and "7.4".

(racket-versions ("*" "7.2") ("7.4" "*"))

If you have particular behavior that depends on exact Racket versions, then you may call out those individual versions. This example adds such a version that would otherwise be excluded.

(racket-versions ("*" "7.2") ("7.4" "*") "7.0.1.2")

We likely want to support as many Racket versions as we can while staying backward compatible. Let’s just define support for v5.0 and up.

"definition.rkt"

#lang xiden
 
(name "my-first-package")
(provider "sagegerard.com")
(description "Fun playtime in a tutorial")
(tags "fun" "tutorial" "example")
(url "https://sagegerard.com")
 
(edition "default")
(revision-number 0)
(revision-names "alpha")
 
(racket-versions ("5.0" "*"))

3.5 Declare Supported Operating Systems

Racket is cross-platform, but your package might not be. Maybe you need PowerShell. I won’t judge. Limiting OS support allows you to make reasonable assumptions about available binaries (e.g. GNU coreutils), and for offering tailored experiences to users.

You can declare the operating systems you support by writing os-support with a list of acceptable values of (system-type 'os).

This example is a statement of support for UNIX-like systems, Windows, and MacOSX.

(os-support unix windows macosx)

By default, Xiden assumes each package definition will work on every operating system. While this means we don’t have to include the above line in the final version of our package definition, I’ll put it in just to be explicit.

3.6 Package Inputs

Now for the interesting stuff. In Xiden, a package definition is a program. An actively running version of that program is a package. Like any other running program, packages use inputs to function.

A package input is a deferred request for exact bytes. I’ll define one for now.

"definition.rkt"

#lang xiden
 
(name "my-first-package")
(provider "sagegerard.com")
(description "Fun playtime in a tutorial")
(tags "fun" "tutorial" "example")
(url "https://sagegerard.com")
 
(edition "default")
(revision-number 0)
(revision-names "alpha")
 
(racket-versions ("5.0" "*"))
(os-support unix windows macosx)
 
(input "default.tgz"
       (sources "https://sagegerard.com/xiden-tutorial/default.tgz"))

This input defines an archive of source code we’ll need to build our project. The archive contains a throwaway Racket module and Scribble document. "default.tgz" is the name of the file that we use locally in our build. The sources list tells Xiden where it can find the actual bytes for that file. I’m only using one source here, but you can add other sources in case one isn’t available.

3.6.1 Everything is an Input

A package input can be any file, not just Racket packages or code. You can use a Python distribution as an input, or a critical security patch. This means that you can use Xiden to build any software.

While we won’t cover it here, another benefit of package inputs is that you can substitute them. If a build is taking too long because it compiles a huge project from source, you can adjust the definition to use pre-built binaries instead.

3.6.2 Integrity Information

We named a file that we want, but how do we know we got the right file? For that, we need to declare integrity information with our input.

(input "default.tgz"
       (sources "https://sagegerard.com/xiden-tutorial/default.tgz")
       (integrity 'sha384 (hex "299e3eb744725387e0355937727cf6e3c938eda2355cda82d58596fd535188fa624217f52f8c6e7d5ee7cb1d458a7f75")))

Integrity information tells Xiden if it got the exact bytes the input requires. If it did not, then the build will fail. This is a good thing because it helps make builds reproducible. An input might only be available during a build, or may persist after a build for run-time use. More on that later.

If you are not familiar with integrity checking, just know that there are functions to take a file and turn it into a fixed-length byte string called a digest. If two files produce the same digest, then we can assume the files are the same. That is, unless the function itself has a collision, where two different files produce the same digest. This is a sign to use a different function!

The function we’re using in this case is SHA-384, which we represent here as sha384. Since it’s hard to type the exact bytes of a digest, we can give Xiden the expected digest as a string we copy and paste from elsewhere. Here we tell Xiden that the SHA-384 digest of "default.tgz" comes from a hex string. That tells Xiden how to translate the digest as a string back to bytes for comparison.

3.6.2.1 Creating an Integrity Expression

There are a couple of ways you can generate an integrity expression. Before you try, download the file at the link shown in the integrity expression in the previous snippet. Our goal is to create the same expression we saw in the code.

Let’s start by making an integrity expression by hand. If you followed Setup, then you should have OpenSSL installed on your system. You can check the digest for yourself by running this command:

$ openssl dgst -sha384 default.tgz

SHA384(default.tgz)= 299e3eb744725387e...

There’s a tricky part here. Yes, the digests match our code, but that doesn’t mean you should paste it right into new definitions. Typically you want to write a definition using a trusted copy of a file.

We’ll assume trust in this file because it is not used to run code on your system. From here you can write an integrity expression by hand. Just paste it in this example where you see DIGEST.

(integrity 'sha384 (hex DIGEST))

Knowing how to produce your own digest is a valuable skill if you want to verify data that arrived on your system. But an external tool might use a different encoding and algorithm than what Xiden supports. If you want the entire integrity expression with options supported by Xiden, then use the xiden mkint command. This example does the same thing, except you’ll get an entire integrity expression as output.

$ xiden mkint sha384 hex default.tgz

(integrity 'sha384 (hex "299e3eb744725387e...

Alternatively, you can tell Xiden to read from standard input. Just use a dash in place of the file.

$ cat default.tgz | xiden mkint sha384 hex -

(integrity 'sha384 (hex "299e3eb744725387e...

Assuming you trust the input, you only need to copy and paste the expression into your definition. If you want programmatic control over integrity expressions, then use the xiden/integrity module.

3.6.3 Authenticating Inputs

It’s one thing to get the bytes we wanted, but did they come from someone we trust? Thankfully it is possible for people to sign their work so that we can check.

You may declare a signature with an input. A signature expression includes a source of a public key used to verify the signature, and a source for the signature itself. The public key, when stored as a file called public.pem must work when used in openssl pkeyutl -pubin -inkey public.pem ....

Signatures are applied per-input. This example fetches both a public key and a signature from the same host that provides an artifact. Here I use a public key that is only used for this tutorial.

(input "default.tgz"
       (sources "https://sagegerard.com/xiden-tutorial/default.tgz")
       (integrity 'sha384 (hex "299e3eb744725387e0355937727cf6e3c938eda2355cda82d58596fd535188fa624217f52f8c6e7d5ee7cb1d458a7f75"))
       (signature "https://sagegerard.com/xiden-tutorial/public.pem"
                  "https://sagegerard.com/xiden-tutorial/default.tgz.sign"))

The signature form accepts a string that locates a public key, and a string that locates a signature, in that order. xiden uses the signature and public key to confirm that the raw bytes of the digest specified in the integrity information was signed with a corresponding private key.

While Xiden can fetch public keys from the Internet for you, it will refuse to process any input where you do not affirm your trust in the corresponding public key. Vetting public keys is out of scope for this guide. Just know that if you do not trust the public key, then a signature verified by that key won’t offer you any value. See Trusting Public Keys and Executables to learn how to affirm trust for individual public keys.

3.6.4 Abstract and Concrete Package Inputs

An abstract package input (or just “abstract input”) is a package input with only a name.

(input "server.rkt")

Such inputs cannot be used to fetch exact bytes, so we won’t use them in our final definition. They are still important to discuss, because you’ll eventually need them.

Abstract inputs are useful as placeholders in package definitions that require the end user to define inputs. By contrast, we’ve been writing concrete package inputs, which are package inputs that include least one source of bytes. A concrete package input does not have to include integrity information or a signature.

3.7 Package Outputs

A package output is a named deliverable from the package, such as documentation, libraries, or tests.

Every package definition should define a default output, because if a user does not request a particular output from a package, then Xiden will use the output named "default". If you do not define a default output, then Xiden will tell the user about the outputs available in the definition.

Recall in the last section that we defined inputs named "default.tgz". We want to fetch and extract that archive.

(output "default"
        archive-input := (input-ref "default.tgz")
        archive-path := (resolve-input archive-input)
        (extract archive-path)
        (release-input archive-input))

When building, current-directory is bound to a unique directory, such that two packages only conflict if evidence shows those packages will produce identical output. You can assume the directory is empty, and yours to populate.

Notice that we manually free the archive using release-input. This is because when you reference an input, Xiden lazily writes it to disk and issues a symbolic link to the file with the given name. Xiden cannot predict what inputs to keep around, so it leaves that to you. We don’t need our archive once the contents are on disk, so we delete the link using release-input.

We do not delete the actual file because if something goes wrong with a package, you might not want to fetch every input again. If there are no incoming links for a file in Xiden, then it is eligible for garbage collection in a separate process.

3.7.1 Monadic Types

We now have two endpoints to our program, and some processing in between. So what’s the := for in the package output? If you are familiar with Haskell and monads, just know that outputs use a notation similar to do and skip this section.

:= kind of like let, but it isn’t exactly the same because this abbreviated program does not work.

(output "default"
        archive-input := (input-ref "default.tgz")
        (extract (resolve-input archive-input))
        (release-input archive-input))

:= does bind a value to an identifier, but it also discovers the value to bind from special context called monads. There are many tutorials that explain monads poorly, and this would likely be one of them. So we’ll just focus on an abbreviated introduction that keeps us moving.

Here are two functions.

(define (add5 v) (+ v 5))
(define (sub2 v) (- v 2))

You can compose them.

(sub2 (add5 4))

Happy days.

One day someone changes the functions so that they return information to store in a program log.

(define (add5 v) (values (+ v 5) (format "Adding 5 to ~s" v)))
(define (sub2 v) (values (- v 2) (format "Subtracting 2 from ~s" v)))

This is preferable than using displayln and the like in functional programs because this way, calling the function has no side-effects. Everything returned from each function is expressed purely in terms of arguments.

Problem is, (sub2 (add5 4)) no longer works. Monads are just the stuff that makes function composition work again without changing the definition of add5 and sub2. That means something is sitting around that knows the first value from this kind of function is a number, and the second value is a string. It knows that composing two such functions means doing something like this:

(define (compose-extra f g)
  (define-values (fv fs) (f v))
  (define-values (gv gs) (g fv))
  (values gv (string-append fs "\n" gs)))

This brings us back to our package output. These instructions work because := understands how to find the value you want from all the extra stuff.

archive-path := (resolve-input "default.tgz")
(extract archive-path)

But (extract (resolve-input archive-input)) doesn’t work because you passed the value you want plus the extra stuff to extract.

In other words, the value returned from (resolve-input archive-input) is not the same as the value bound to archive-path when using :=.

How you deal with the values depends on the type, which is normal. You’ll pick up on the different types as you go.

3.7.2 Adding a Second Output

Assume "default.tgz" has everything a user would need. Some users might only want the libraries, not the documentation. Storage is cheap, but hundreds of dependencies add up.

We can define a new output for our budget-conscious users:

(output "default"
        archive-path := (resolve-input "default.tgz")
        (extract archive-path)
        (release-input archive-path))
 
(output "minimal"
        archive-path := (resolve-input "minimal.tgz")
        (extract archive-path)
        (release-input archive-path))

This version works, but we don’t want to repeat ourselves every time we want a new output. To reduce repetition, we can write a procedure. But if we want to use the same notation and logic as outputs, we’ll need to return an mdo form.

(define (unpack input-name)
  (mdo archive-input := (input-ref input-name)
       archive-path := (resolve-input archive-input)
       (extract archive-path)
       (release-input archive-input)))
 
(output "default" (unpack "default.tgz"))
(output "minimal" (unpack "minimal.tgz"))

By adding an output, we changed what the user can request from a package. Since the build extracts an archive with the same name as the output, we’ll need a new package input to go with this output.

(input "default.tgz"
       (sources "https://sagegerard.com/xiden-tutorial/default.tgz")
       (integrity 'sha384 (hex "299e3eb744725387e0355937727cf6e3c938eda2355cda82d58596fd535188fa624217f52f8c6e7d5ee7cb1d458a7f75"))
       (signature "https://sagegerard.com/xiden-tutorial/public.pem"
                  "https://sagegerard.com/xiden-tutorial/default.tgz.sign"))
 
(input "minimal.tgz"
       (sources "https://sagegerard.com/xiden-tutorial/minimal.tgz")
       (integrity 'sha384 (hex "6cc38a7e2513fa9abd2ac079e9c8efbab9385458275c927e77527a189ed9ac393d734a4cf306787425bf722a5ac025c6"))
       (signature "https://sagegerard.com/xiden-tutorial/public.pem"
                  "https://sagegerard.com/xiden-tutorial/minimal.tgz.sign"))

Now when our users choose to build "minimal" output, they will only ever download and extract the "minimal.tgz" archive.

3.7.3 Outputs Can Create Duplicate Data

Xiden assumes that different outputs produce different content, which makes one kind of human error possible.

Assume we add an output to act as an alias of another.

(output "default" (unpack "default.tgz"))
(output "full" (unpack "default.tgz"))
(output "minimal" (unpack "minimal.tgz"))

This works, but if a user requests the "full" output and then the "default" output, then the same archive would be extracted twice into different directories. This pollutes the disk with redundant data, which is probably not what you want.

Different package outputs are expected to produce different things. If you believe that two outputs are equivalent, then combine them into one output. If multiple outputs end up creating too much duplicate data, then you might want to consider defining the common data in another package definition.

3.8 User-defined Metadata

A lot of what we’ve added to our code counts as metadata, such as url, tags, and description. All of the entries we’ve defined so far are metadata that Xiden readily recognizes due to their widespread use.

If you want to store other information in your definition, then you can use the metadatum form.

(metadatum support-email "support@example.com")

A metadatum works like define, in that you can bind one identifier to a value. When someone uses require or one of its variants on a package definition, they can inspect an expanded metadata binding to see all user-defined metadata.

> (module anon xiden/pkgdef2 (metadatum support-email "support@example.com"))

> (require 'anon)

> metadata

'#hasheq((support-email . "support@example.com"))

Metadata can only be literal strings, and are not meant for use in program logic.

3.9 The Finished Definition

Here is the file we’ve authored. To recap, it defines a build that simply extracts an archive depending on the requested output. We’ll discuss this definition further in Command Line Interface.

"definition.rkt"

#lang xiden
 
(name "my-first-package")
(provider "example.com")
(description "Fun playtime in a tutorial")
(tags "fun" "tutorial" "example")
(url "https://sagegerard.com")
 
(edition "default")
(revision-number 0)
(revision-names "alpha")
 
; -----------------------------------------------
; Platform support
(racket-versions ("5.0" "*"))
(os-support unix windows macosx)
 
; -----------------------------------------------
; User Metadata
(metadatum support-email "support@example.com")
 
 
; -----------------------------------------------
; Inputs
(input "default.tgz"
       (sources "https://sagegerard.com/xiden-tutorial/default.tgz")
       (integrity 'sha384 (hex "299e3eb744725387e0355937727cf6e3c938eda2355cda82d58596fd535188fa624217f52f8c6e7d5ee7cb1d458a7f75"))
       (signature "https://sagegerard.com/xiden-tutorial/public.pem"
                  "https://sagegerard.com/xiden-tutorial/default.tgz.sign"))
 
(input "minimal.tgz"
       (sources "https://sagegerard.com/xiden-tutorial/minimal.tgz")
       (integrity 'sha384 (hex "6cc38a7e2513fa9abd2ac079e9c8efbab9385458275c927e77527a189ed9ac393d734a4cf306787425bf722a5ac025c6"))
       (signature "https://sagegerard.com/xiden-tutorial/public.pem"
                  "https://sagegerard.com/xiden-tutorial/minimal.tgz.sign"))
 
; -----------------------------------------------
; Outputs
 
(define (unpack input-name)
  (mdo archive-input := (input-ref input-name)
       archive-path  := (resolve-input archive-input)
       (extract archive-path)
       (release-input archive-input)))
 
(output "default" (unpack "default.tgz"))
(output "minimal" (unpack "minimal.tgz"))