On this page:
3.1 shell/  pipeline-macro stability
3.2 shell/  pipeline-macro guide
3.3 shell/  pipeline-macro reference
3.3.1 Running Pipelines
run-pipeline
3.3.2 Pipeline Flags
&bg
&pipeline-ret
&in
&<
&out
&>
&>!
&>>
&err
&strict
&permissive
&lazy
&lazy-timeout
with-pipeline-config
splicing-with-pipeline-config
3.3.3 Pipeline Operators
=composite-pipe=
=basic-unix-pipe=
=unix-pipe=
\|
=basic-object-pipe/  expression=
=basic-object-pipe/  form=
=basic-object-pipe=
\|>
=object-pipe/  expression=
=object-pipe/  form=
=object-pipe=
\|>>
default-pipeline-starter
3.3.4 Defining Pipeline Operators
define-pipeline-operator
define-pipeline-alias
define-simple-pipeline-alias
current-pipeline-argument
_
expand-pipeline-arguments
3.3.5 Inspecting Pipelines
3.4 Demo stuff reference
=map=
=filter=
7.1

3 Pipeline Macro Library

 (require shell/pipeline-macro) package: shell-pipeline

3.1 shell/pipeline-macro stability

Unstable features are flagged in the documentation. There are few of them.

The base set of pipeline operators is likely to change, and some of the names I want to review before a stable release.

Some pipeline options (&in, etc) are currently backed by syntax parameters, which is wrong. It causes things to be... not as lexical as they are supposed to be. That will be changed.

3.2 shell/pipeline-macro guide

This module is a macro DSL wrapper around the Mixed Unix-style and Racket Object Pipelines library. It is designed for running pipelines of external processes (which pass each other ports) and Racket functions (which pass each other objects). It does this with a very flat syntax and user-definable pipeline operators, which provide a lot of convenient sugar for making pipelines shorter. It is particularly tailored for use in a line-based syntax, like that of the Rash language.

Here are some quick examples:

;; Pipe the output of ls to grep.
(run-pipeline =unix-pipe= ls -l =unix-pipe= grep foo)
 
;; To save on space, let's assume % is bound to =unix-pipe=
(run-pipeline % ls -l % grep foo)

We can also pipeline objects. Object pipelines are full of functions instead of process specifications.

;; This will return 2
(run-pipeline =object-pipe= list 1 2 3 =object-pipe= second)
 
;; To save on space, let's assume %> is bound to =object-pipe=
(run-pipeline %> list 1 2 3 %> second)

We can mix the two:

;; Capitalized ls output.  =object-pipe= automatically converts ports to strings.
(run-pipeline % ls -l %> string-upcase)

I am really running out of steam for documenting right now... TODO - write a good guide.

3.3 shell/pipeline-macro reference

3.3.1 Running Pipelines

syntax

(run-pipeline pipeline-flag ...
              pipeline-member-spec ...
              pipeline-flag ...)
 
pipeline-member-spec = pipe-operator pipe-operator-arg ...
     
pipeline-flag = &bg
  | &pipeline-ret
  | &in file-expression
  | &< file-name
  | &out file-expression
  | &> file-name
  | &>> file-name
  | &>! file-name
  | &err file-expression
  | &strict
  | &permissive
  | &lazy
  | &lazy-timeout timeout-expression
Run a pipeline. This is a macro wrapper for shell/mixed-pipeline/run-mixed-pipeline that uses pipeline operator macros to specify the pipeline to be run. So you should read the docs about that as well.

The pipeline flags affect the options passed to shell/mixed-pipeline/run-mixed-pipeline and are documented separately.

The pipeline-member-specs are transformed according to the pipeline operators given. If the first non-flag argument to run-pipeline is not a pipeline operator, then a default is put in its place as determined by a lexical default operator. The default is determined by the lexical context of the first form among the arguments (after the run-pipeline identifier itself). The full names of pipeline operators are conventionally identifiers surrounded with = signs.

At the time of writing I’m not really sure what to write here, so have an example:

(run-pipeline =object-pipe= list 1 2 3
              =map= + 1 current-pipeline-argument
              =map= + 1)

This returns

(list 3 4 5)

. Notice that current-pipeline-argument is placed automatically for the second =map= operator – it could have been left off of the other one as well, but I wanted to show what identifier is sneakily added.

If we instead run
(run-pipeline =object-pipe= list 1 2 3
              =map= + 1 current-pipeline-argument
              =map= + 1
              &bg)
we will get a pipeline object back. Conceptually it is still running when it is returned, though in this case it’s likely finished by the time we can inspect it. We can use pipeline?, pipeline-success?, pipeline-return, etc on it.

3.3.2 Pipeline Flags

pipeline-flag

(&bg)

pipeline-flag

(&pipeline-ret)

pipeline-flag

(&in port-expression)

pipeline-flag

(&< file-name)

pipeline-flag

(&out port/reader-expression)

pipeline-flag

(&> file-name)

pipeline-flag

(&>! file-name)

pipeline-flag

(&>> file-name)

pipeline-flag

(&err port-expression)

pipeline-flag

(&strict)

pipeline-flag

(&permissive)

pipeline-flag

(&lazy)

pipeline-flag

(&lazy-timeout)

These identifiers are all errors if used outside of run-pipeline. They are essentially used in place of #:keywords to not conflict with pipeline operators that take keywords.

&<, &>, &>>, and &>! each take a file name and cause (respectively) input redirection from the given file, output redirection to the given file erroring if the file exists, output redirection appending to the given file, and output redirection truncating the given file. &in, &out, and &err take an argument suitable to be passed to #:in, #:out, and #:err of shell/mixed-pipeline/run-mixed-pipeline.

&bg and &pipeline-ret toggle #:bg and #:return-pipeline-object, and &strict, &permissive, and &lazy set the #:strictness argument. Not stable: All of the file redirection flags may change how they do things like dollar, glob, and tilde expansion.

syntax

(with-pipeline-config option ... body ...)

 
option = #:in in-expr
  | #:out out-expr
  | #:err err-expr
  | #:starter default-pipeline-starter
Parameterizes the body forms to use default-pipeline-starter when no starter is explicitly given, and to use in-expr, out-expr, and err-expr as default redirections. Note that the expressions given for redirection are evaluated for every pipeline run in the body.

syntax

(splicing-with-pipeline-config option ... body ...)

 
option = #:in in-expr
  | #:out out-expr
  | #:err err-expr
  | #:starter default-pipeline-starter
Like with-pipeline-config, except that it splices into definition contexts.

3.3.3 Pipeline Operators

The core module only provides a few simple pipeline operators. There are many more in the demo/ directory in the source repository. Most of them are hastily written experiments, but some good ones should eventually be standardized.

pipeline-operator

(=composite-pipe= (pipe-op arg ...) ...+)

Produces a composite pipeline member spec made from the pipeline operators given. This is really more for use when defining new pipeline operators than for use in a pipeline itself.

pipeline-operator

(=basic-unix-pipe= options ... args ...+)

Produces a unix-pipeline-member-spec with the given arguments as the process/function argument list. The arguments are not quoted. Any argument that produces a list will be spliced into the argument list.

Options all take an argument, must precede any arguments, and are as follows:

#:err - Takes an expression to produce an error redirection value suitable for unix-pipeline-member-spec.

#:success - Takes an expression suitable for the #:success argument of unix-pipeline-member-spec.

TODO - env modification

Arguments are evaluated left-to-right, and the first argument (that does not produce an empty list) is checked to be a valid executable path. If it is not a valid executable path, an exception is raised before evaluating the other arguments. (Note that an executable could be removed between a successful check and an attempt to spawn a subprocess, so it could still error after evaluating the other arguments.)

pipeline-operator

(=unix-pipe= arg ...+)

This is the pipe that does more or less what you expect. It does tilde expansion (~ -> $HOME). It does globbing. When you have $identifiers-with-dollar-signs they are expanded into variable references. When $DOLLAR_IDENTIFIERS_ARE_CAPITALIZED they are expanded to environment variable lookups.

The operator quotes other arguments, and then passes through to =basic-unix-pipe=. But it takes some extra optional arguments:

#:as - This is sugar for adding on an object pipeline member afterward that parses the output somehow. This should be given either #f (no transformation), a port reading function (eg. port->string), or one of a pre-set list of symbols: 'string, 'trim, 'lines, or 'words.

#:e> - Accepts a file name (as an identifier), redirects the error stream to that file. Produces an error if the file exists.

#:e>! - Accepts a file name (as an identifier), redirects the error stream to that file. Truncates the file if it exists.

#:e>> - Accepts a file name (as an identifier), redirects the error stream to that file. Appends to the file if it exists.

BUT: if the first argument is a pipeline alias defined with define-pipeline-alias or define-simple-pipeline-alias, then the operator from that alias is swapped in instead, skipping everything else that this operator would normally do.

(run-pipeline =unix-pipe= echo $HOME/*.rkt)
(define-simple-pipeline-alias d 'ls '--color=auto)
(define dfdir 'dotfiles)
(run-pipeline =unix-pipe= d $HOME/$dfdir)

At runtime, the first argument is evaluated and tested to see if it exists as an executable on the path. If it’s not, an error is raised before the other arguments are evaluated. (This is an optimization for error messages – the executable could still be removed between evaluating other arguments and executing the program, in which case the error message is worse.)

Usually \| is used instead.

Unstable: The way dollar, tilde, and glob expansion is done by default may change (though it will still be consistent with examples in the GPCE paper). The #:as flag may go away (it’s redundant with just piping to a reader function).

pipeline-operator

(\| arg ...+)

Alias for =unix-pipe=.

Note that the backslash is required in the normal racket reader because | is normally treated specially. In the Rash reader, you can get this by typing just |.

pipeline-operator

(=basic-object-pipe/expression= e)

The simplest object pipe. e is simply the body of a lambda. When used as a pipeline starter, the lambda accepts no arguments. Otherwise it is a single-argument function, and current-pipeline-argument is used to refer to its argument.

Unstable. The name will probably change.

pipeline-operator

(=basic-object-pipe/form= arg ...+)

Creates an object pipe where (arg ...) is the body of a function.

As with other object pipes, when used as a pipeline starter it generates a lambda with no arguments, and as a pipeline joint it generates a lambda with one argument, current-pipeline-argument.

Unstable. The name will probably change.

pipeline-operator

(=basic-object-pipe= f-arg arg ...)

Like =basic-object-pipe/form=, except that when not used as a pipeline starter, if the current-pipeline-argument is not used within the arguments, it is appended as the last argument.

To discover whether current-pipeline-argument is used, each argument is local-expanded. So f-arg must evaluate to function, and NOT be the name of a macro to be expanded with the other args.

Note: If =basic-object-pipe= follows a subprocess form, it should close the received port. Operators that handle ports automatically such as \|>> close the port automatically, but \|> allows things like lazy reading into a stream, so it can not apply an automatic policy that will always work effectively.

Usually \|> is used instead.

Unstable. The name will probably change and/or this name may be given to a pipe of different meaning. But the \|> alias will NOT have its meaning changed (IE it is stable).

pipeline-operator

(\|> arg ...+)

Note that the backslash is required in the normal racket reader because | is normally treated specially. In the Rash reader, you can get this by typing just |>.

Unstable. The name will probably change.

pipeline-operator

(=object-pipe/expression= arg ...+)

Like =basic-object-pipe/expression=, but when it receives a port as an argument, it converts it to a string.

Unstable. The name will probably change.

pipeline-operator

(=object-pipe/form= arg ...+)

Like =basic-object-pipe/form=, but when it receives a port as an argument, it converts it to a string.

Unstable. The name will probably change.

pipeline-operator

(=object-pipe= f-arg arg ...)

Like =basic-object-pipe=, but when it receives a port as an argument, it converts it to a string.

The same caveat applies that f-arg must NOT be a macro you expect to expand with the other arguments – it must evaluate to a function.

Usually \|>> is used instead.

Unstable. The name will probably change. But note that the \|>> alias is stable.

pipeline-operator

(\|>> arg ...+)

Alias for =object-pipe=.

Note that the backslash is required in the normal racket reader because | is normally treated specially. In the Rash reader, you can get this by typing just |>>.

pipeline-operator

default-pipeline-starter

Syntax parameter determining which pipeline operator is inserted when a run-pipeline form doesn’t explicitly start with one.

Unstable in that it will not be a syntax parameter in the future, but something with better semantics for what it’s supposed to do.

I’ve written various other pipeline operators that are a little more exciting and that are currently in the demo directory of the repository. I’ll eventually polish them up and put them somewhere stable. They include things like unix pipes that automatically glob things, unix pipes that have lexically scoped alias resolution, =filter=, =map=, =for/stream=,=for/list/unix-arg=,=for/list/unix-input=...

3.3.4 Defining Pipeline Operators

syntax

(define-pipeline-operator name start-or-joint ...)

 
start-or-joint = #:start transformer
  | #:joint transformer
  | #:operator transformer
Define a pipeline operator. Pipeline operators can act differently when they are in the starting position of a pipeline or later (joint position). Specifically, when an operator creates an object-pipeline-member-spec, it needs to have a function that accepts 0 arguments when in the start position and 1 argument in others.

If you use the #:operator keyword, the same transformer is used in start and joint positions.

If a transformer function is not specified for one of the options, a default implementation (that generates an error) is used.

The transformer will receive a syntax object corresponding to (name-of-pipe argument ...), so it will likely want to ignore its first argument like most macros do. But simetimes it may be useful to recur.

If a pipeline operator is used outside of run-pipeline, it raises a syntax error.

The operator must desugar to code which produces a pipeline-member-spec.

Example uses are in the demo directory in the repository.

Unstable: Currently for operators to desugar to another operator, the new operator must be at the top-level of the returned syntax or it will raise an error (IE that a pipeline operator was used out of context). This should be fixed later.

syntax

(define-pipeline-alias name transformer)

Defines an alias macro recognized by =unix-pipe= and maybe others.

transformer must be a syntax transformer function, and must return a syntax object that starts with a pipeline operator.

;; Unix `find` has to take the directory first, but we want
;; to always add the -type f flag at the end.
(define-pipeline-alias find-f
  (syntax-parser
    [(_ arg ...) #'(=unix-pipe= find arg ... -type f)]))
 
;; these are equivalent
(run-pipeline =unix-pipe= find-f ".")
(run-pipeline =unix-pipe= find "." -type f)

Unstable: I might re-think this slightly... if it changes it will be similar and existing aliases will probably still work, but I’m not 100% sure so I want to hedge.

syntax

(define-simple-pipeline-alias name cmd arg ...)

Simple sugar for define-pipeline-alias. It defines an alias transformer that uses =unix-pipe=.

(define-simple-pipeline-alias ls 'ls '--color=auto)
;; these are equivalent
(run-pipeline =unix-pipe= d -l $HOME)
(run-pipeline =unix-pipe= 'ls '--color=auto -l $HOME)

Note that arguments are passed through as-is to =unix-pipe=, so if you use the name of another alias it will cause alias replacement a second time unless you quote the command name or put it in a string.

Unstable: The name might change or something. I just want to hedge the possibility that I want to re-look at aliases.

The name of the implicit argument for object pipes. The default is an error, and pipe operators that accept it must set it up using expand-pipeline-arguments or syntax-parameterize.

Usually _ is used instead.

syntax

_

Alias for current-pipeline-argument. It’s an underscore, if you’re having trouble telling which of the many horizontal line characters it is since it’s all alone right there in that bar.

TODO - document this.

It’s used to get the current-pipeline-argument detected and inserted automatically if missing.

Unstable – this is not a great API and I don’t want the burden of maintaining it.

3.3.5 Inspecting Pipelines

See Inspecting Mixed Pipelines.

3.4 Demo stuff reference

These things are documented in the Rash documentation, but I’m adding these definitions to not have broken links...

You can get these with

(require rash/demo/setup)

pipeline-operator

(=map= arg ...)

Unstable in that it will probably be moved into the main rash module, but I think it will stay the same now.

pipeline-operator

(=filter= arg ...)

Unstable in that it will probably be moved into the main rash module, but I think it will stay the same now.