3 Pipeline Macro Library🔗ℹ

3.1 shell/pipeline-macro stability🔗ℹ

Unstable features are flagged in the documentation. There are few of them.

The base set of pipeline operators is likely to change, and some of the names I want to review before a stable release.

Some pipeline options (&in, etc) are currently backed by syntax parameters, which is wrong. It causes things to be... not as lexical as they are supposed to be. That will be changed.

3.2 shell/pipeline-macro guide🔗ℹ

This module is a macro DSL wrapper around the Mixed Unix-style and Racket Object Pipelines library. It is designed for running pipelines of external processes (which pass each other ports) and Racket functions (which pass each other objects). It does this with a very flat syntax and user-definable pipeline operators, which provide a lot of convenient sugar for making pipelines shorter. It is particularly tailored for use in a line-based syntax, like that of the Rash language.

Here are some quick examples:

;; Pipe the output of ls to grep.

(run-pipeline =unix-pipe= ls -l =unix-pipe= grep foo)

;; To save on space, let's assume % is bound to =unix-pipe=

(run-pipeline % ls -l % grep foo)

We can also pipeline objects. Object pipelines are full of functions instead of process specifications.

;; This will return 3

(run-pipeline =object-pipe= list 0 1 2 =object-pipe= length)

;; To save on space, let's assume %> is bound to =object-pipe=

(run-pipeline %> list 0 1 2 %> length)

We can mix the two:

;; Capitalized ls output.  =object-pipe=

;; automatically converts ports to strings.

(run-pipeline % ls -l %> string-upcase)

I am really running out of steam for documenting right now... TODO - write a good guide.

3.3 shell/pipeline-macro reference🔗ℹ

3.3.1 Running Pipelines🔗ℹ

The pipeline flags affect the options passed to shell/mixed-pipeline/run-mixed-pipeline and are documented separately.

The pipeline-member-specs are transformed according to the pipeline operators given. If the first non-flag argument to run-pipeline is not a pipeline operator, then a default is put in its place as determined by a lexical default operator. The default is determined by the lexical context of the first form among the arguments (after the run-pipeline identifier itself). The full names of pipeline operators are conventionally identifiers surrounded with = signs.

At the time of writing I’m not really sure what to write here, so have an example:

(run-pipeline =object-pipe= list 1 2 3

=map= + 1 current-pipeline-argument

=map= + 1)

3.3.2 Pipeline Flags🔗ℹ

&<, &>, &>>, and &>! each take a file name and cause (respectively) input redirection from the given file, output redirection to the given file erroring if the file exists, output redirection appending to the given file, and output redirection truncating the given file. &in, &out, and &err take an argument suitable to be passed to #:in, #:out, and #:err of shell/mixed-pipeline/run-mixed-pipeline.

&bg and &pipeline-ret toggle #:bg and #:return-pipeline-object, and &strict, &permissive, and &lazy set the #:strictness argument. Not stable: All of the file redirection flags may change how they do things like dollar, glob, and tilde expansion.

3.3.3 Pipeline Operators🔗ℹ

The core module only provides a few simple pipeline operators. There are many more in the demo/ directory in the source repository. Most of them are hastily written experiments, but some good ones should eventually be standardized.

Options all take an argument, must precede any arguments, and are as follows:

#:err - Takes an expression to produce an error redirection value suitable for unix-pipeline-member-spec.

#:success - Takes an expression suitable for the #:success argument of unix-pipeline-member-spec.

TODO - env modification

Arguments are evaluated left-to-right, and the first argument (that does not produce an empty list) is checked to be a valid executable path. If it is not a valid executable path, an exception is raised before evaluating the other arguments. (Note that an executable could be removed between a successful check and an attempt to spawn a subprocess, so it could still error after evaluating the other arguments.)

The operator quotes other arguments, and then passes through to =basic-unix-pipe=. But it takes some extra optional arguments:

#:as - This is sugar for adding on an object pipeline member afterward that parses the output somehow. This should be given either #f (no transformation), a port reading function (eg. port->string), or one of a pre-set list of symbols: 'string, 'trim, 'lines, or 'words.

#:e> - Accepts a file name (as an identifier), redirects the error stream to that file. Produces an error if the file exists.

#:e>! - Accepts a file name (as an identifier), redirects the error stream to that file. Truncates the file if it exists.

#:e>> - Accepts a file name (as an identifier), redirects the error stream to that file. Appends to the file if it exists.

BUT: if the first argument is a pipeline alias defined with define-pipeline-alias or define-simple-pipeline-alias, then the operator from that alias is swapped in instead, skipping everything else that this operator would normally do.

(run-pipeline =unix-pipe= echo $HOME/*.rkt)

(define-simple-pipeline-alias d 'ls '--color=auto)

(define dfdir 'dotfiles)

(run-pipeline =unix-pipe= d $HOME/$dfdir)

At runtime, the first argument is evaluated and tested to see if it exists as an executable on the path. If it’s not, an error is raised before the other arguments are evaluated. (This is an optimization for error messages – the executable could still be removed between evaluating other arguments and executing the program, in which case the error message is worse.)

Usually \| is used instead.

Note that the backslash is required in the normal racket reader because | is normally treated specially. In the Rash reader, you can get this by typing just |.

Unstable. The name will probably change.

As with other object pipes, when used as a pipeline starter it generates a lambda with no arguments, and as a pipeline joint it generates a lambda with one argument, current-pipeline-argument.

Unstable. The name will probably change.

To discover whether current-pipeline-argument is used, each argument is local-expanded. So f-arg must evaluate to function, and NOT be the name of a macro to be expanded with the other args.

Note: If =basic-object-pipe= follows a subprocess form, it should close the received port. Operators that handle ports automatically such as \|>> close the port automatically, but \|> allows things like lazy reading into a stream, so it can not apply an automatic policy that will always work effectively.

Usually \|> is used instead.

Note that the backslash is required in the normal racket reader because | is normally treated specially. In the Rash reader, you can get this by typing just |>.

Unstable. The name will probably change.

The same caveat applies that f-arg must NOT be a macro you expect to expand with the other arguments – it must evaluate to a function.

Usually \|>> is used instead.

Note that the backslash is required in the normal racket reader because | is normally treated specially. In the Rash reader, you can get this by typing just |>>.

Unstable in that it will not be a syntax parameter in the future, but something with better semantics for what it’s supposed to do.

I’ve written various other pipeline operators that are a little more exciting and that are currently in the demo directory of the repository. I’ll eventually polish them up and put them somewhere stable. They include things like unix pipes that automatically glob things, unix pipes that have lexically scoped alias resolution, =filter=, =map=, =for/stream=,=for/list/unix-arg=,=for/list/unix-input=...

3.3.4 Defining Pipeline Operators🔗ℹ

If you use the #:operator keyword, the same transformer is used in start and joint positions.

If a transformer function is not specified for one of the options, a default implementation (that generates an error) is used.

The transformer will receive a syntax object corresponding to (name-of-pipe argument ...), so it will likely want to ignore its first argument like most macros do. But simetimes it may be useful to recur.

If a pipeline operator is used outside of run-pipeline, it raises a syntax error.

The operator must desugar to code which produces a pipeline-member-spec.

Example uses are in the demo directory in the repository.

Unstable: Currently for operators to desugar to another operator, the new operator must be at the top-level of the returned syntax or it will raise an error (IE that a pipeline operator was used out of context). This should be fixed later.

transformer must be a syntax transformer function, and must return a syntax object that starts with a pipeline operator.

;; Unix `find` has to take the directory first, but we want

;; to always add the -type f flag at the end.

(define-pipeline-alias find-f

(syntax-parser

[(_ arg ...) #'(=unix-pipe= find arg ... -type f)]))

;; these are equivalent

(run-pipeline =unix-pipe= find-f ".")

(run-pipeline =unix-pipe= find "." -type f)

Unstable: I might re-think this slightly... if it changes it will be similar and existing aliases will probably still work, but I’m not 100% sure so I want to hedge.

(define-simple-pipeline-alias ls 'ls '--color=auto)

;; these are equivalent

(run-pipeline =unix-pipe= d -l $HOME)

(run-pipeline =unix-pipe= 'ls '--color=auto -l $HOME)

Note that arguments are passed through as-is to =unix-pipe=, so if you use the name of another alias it will cause alias replacement a second time unless you quote the command name or put it in a string.

Unstable: The name might change or something. I just want to hedge the possibility that I want to re-look at aliases.

It’s used to get the current-pipeline-argument detected and inserted automatically if missing.

Unstable – this is not a great API and I don’t want the burden of maintaining it.

3.3.5 Inspecting Pipelines🔗ℹ

See Inspecting Mixed Pipelines.

3.4 Demo stuff reference🔗ℹ

These things are documented in the Rash documentation, but I’m adding these definitions to not have broken links...