On this page:
1.1 shell/  pipeline stability
1.2 shell/  pipeline guide
1.3 shell/  pipeline reference
run-pipeline/  out
pipeline-status/  list
prop:  alias-func
and/  success
or/  success

1 Basic Unix-style Pipelines

 (require shell/pipeline) package: shell-pipeline

1.1 shell/pipeline stability

This library is not entirely stable.

Forthcoming features include features such as process redirection (similar to <() and >() in Bash).

Some specific things that may change are the names of keyword arguments to run-pipeline, and the type of arguments and exact semantics of the redirection options for pipelines. Also, the extra run-pipline/ functions.

1.2 shell/pipeline guide

This library makes unix-style pipelines of external programs and racket functions easy. You can write things as simply as (run-pipeline '(cat /etc/passwd) '(grep root) '(cut -d : -f 1)), which will print "root\n" to stdout (on unix systems) and will return a pipeline object. To get the output as a string, use run-pipeline/out the same way. You can also put racket functions in the pipeline. If you have a racket implementation of grep called my-grep, you can do (run-pipeline '(cat /etc/passwd) `(,my-grep root) '(cut -d : -f 1)) to get the same results. So you can write all sorts of filter functions in Racket rather than using shell commands.

Symbols in pipelines are turned into strings before they are passed in as arguments to subprocesses. Arguments to racket functions are not transformed in any way.

This library DOES work on MS Windows, and if it can’t find a program it retries the name with a .exe at the end. But Microsoft doesn’t seem to believe in having useful shell utilities, or in putting program executables on the PATH, or adding program locations to the PATH. So it will probably still be more useful on Unix than on Windows.

1.3 shell/pipeline reference


(run-pipeline member    
  [#:in in    
  #:out out    
  #:err err    
  #:strictness strictness    
  #:lazy-timeout lazy-timeout    
  #:background? bg?])  any/c
  member : (or/c list? pipeline-member-spec?)
  in : (or/c input-port? false/c) = (current-input-port)
  out : 
(or/c port? false/c path-string-symbol?
     (list/c path-string-symbol? (or/c 'append 'truncate 'error)))
   = (current-output-port)
  err : 
(or/c port? false/c path-string-symbol?
     (list/c path-string-symbol? (or/c 'append 'truncate 'error)))
   = (current-error-port)
  strictness : (or/c 'strict 'lazy 'permissive) = 'lazy
  lazy-timeout : real? = 1
  bg? : any/c = #f
Run a pipeline. Each member should be either a pipeline-member-spec or a list, where the first of the list is the command and the rest are arguments. The command may be a symbol, string, path, or function. If it is a string or path, it will spawn a subprocess. If it is a function, it will use that function in a thread. If the command is an alias-func, it is called with its arguments before the pipeline is started to receive a new command/argument list which replaces it.

A pipeline-member-spec, in addition to the command/argument list, has an error-port specification. All lists given will be turned into pipeline-member-specs using the err specification.

Each member of the pipeline will have its current-output-port connected to the current-input-port of the next member. The first and last members use in and out, respectively, to communicate with the outside world.

All ports specified (in, out, err) may be either a port, the symbol 'null, #f, or a path/string/symbol. The error port may be 'stdout, in which case the output port will be used. If #f is given, then a port will be returned in the pipeline struct returned (similar to subprocess). If 'null is given a null output port or empty string port is used. If a path/string/symbol is given, then a file at that path is opened.

Output and error ports may be a list of a path/string/symbol and any of 'error, 'append, or 'truncate to specify what should happen if the file already exists.

Beware that just as with subprocess, if you pass #f to get an input, output, or error port out of a pipeline, the resulting port may be a file-stream-port, and you will need to be sure to close it. Otherwise all file-stream-port handling in the pipeline and for file redirection is done automatically.

strictness determines how success is reported. If strictness is 'strict, then the pipeline is successful when all members are successful, and if there are errors the first member to have an error is reported. If strictness is 'lazy, success is similar, but treats any members that were killed as successful. If strictness is 'permissive, then errors are ignored except for the last pipeline member, which is what bash and most other shell languages do.

Also, if strictness is 'lazy or 'permissive, then when a pipeline member finishes, pipeline members before it may be killed. In permissive mode they may be killed immediately, and in lazy mode they have lazy-timeout seconds to finish before they are killed. This process killing happens to not wait for long (potentially infinitely so) processes in the middle of a pipeline when only a small part of their output is used. For instance, piping the output of a large file (or cat-ing an infinite pseudo-file) to the "head" command. This mirrors what bash and other shells do.

If background? is false, then run-pipeline uses pipeline-wait to wait until it finishes, then returns the status with pipeline-status. If background? is not false, then run-pipeline returns a pipeline object.


(run-pipeline/out member    
  [#:in in    
  #:status-and? status-and?])  any/c
  member : (or/c list? pipeline-member-spec?)
  in : (or/c input-port? false/c path-string-symbol?)
   = (open-input-string "")
  status-and? : any/c = #f
Like run-pipeline, but string-ports are used as the input, output, and error ports. It does not return until the pipeline finishes, and returns the output string. If the pipeline has an unsuccessful status, an exception is raised (with the contents of the error port).


(pipeline-member-spec? pmspec)  boolean?

  pmspec : any/c
Is it a pipeline-member-spec?


(pipeline-member-spec argl 
  [#:err err 
  #:success success-pred]) 
  argl : any/c
  err : 
(or/c port? false/c path-string-symbol?
      (list/c path-string-symbol? (or/c 'append 'truncate 'error)))
   = hidden-default-value
  success-pred : (or/c false/c procedure? (listof any/c))
   = hidden-default-value
Make a pipeline-member-spec. argl is the command/argument list. The first value in the list is the command, and should either be a path-string-symbol? to a command or a function. err is the error port specification for it to use. success-pred is a predicate that will be applied to the return value of a pipeline member to determine its success. Subprocesses will be considered successful if they return 0 when success-pred is #f, or if success-pred is a list, if they return 0 or a member of that list, or if success-pred is a function, it will be successful if (success-pred return-value) returns a non-#f value. Function pipeline members are always considered unsuccessful if they throw an error. Otherwise, they are successful if success-pred is #f, if their return is a member of success-pred when it is a list, or when (success-pred return-value) is true when it is a function.

err and success-pred default to values that can be overridden by the defaults set by the pipeline-running functions. But in the end they default to current-error-port and #f.


(pipeline? p)  boolean?

  p : any/c
Is it a pipeline object? Note that this is not the same as shell/mixed-pipeline/pipeline?.

Also, pipelines are synchronizable.


(pipeline-port-to p)  (or/c false/c output-port?)

  p : pipeline?
Get initial input port (if one was provided initially, this will be false)


(pipeline-port-from p)  (or/c false/c input-port?)

  p : pipeline?
Get final output port (if one was provided initially, this will be false)


(pipeline-err-ports p)  (listof (or/c false/c input-port?))

  p : pipeline?
Get list of error ports for the pipeline (each one that was provided will be false)


(pipeline-wait p)  void?

  p : pipeline?
Wait for the pipeline to finish. If the pipeline strictness is permissive, then a pipeline is finished when the ending member of the pipeline is finished. Pipelines are also synchronizable events that are ready for synchronization when the pipeline is finished.


(pipeline-kill p)  void?

  p : pipeline?
Kill a running pipeline.


(pipeline-running? p)  boolean?

  p : pipeline?
Is the pipeline currently running?


(pipeline-success? p)  any/c

  p : pipeline?
Waits for the pipeline to terminate (according to pipeline-wait). Returns #t if the pipeline was considered successful, else #f. If the strictness argument is ’strict or ’lazy, then all members must succeed. If it is ’permissive then only the last one must succeed. A pipeline member is considered successful if it was a subprocess and returned 0, or if it was a thread and raised no uncaught exceptions. If strictness is lazy then pipeline members that were killed are also considered successful.


(pipeline-status p)  any/c

  p : pipeline?
Waits for the pipeline to terminate (according to pipeline-wait). Returns the status of the pipeline. If the strictness was ’strict or ’lazy, then the status will be the status of the first unsuccessful member or the status of the last member. If the strictness was ’permissive, then the status will be the status of the last member. The status of any member is its return code for a process, the return of or exception thrown by the function of a thread member, or ’killed if it was killed.


(pipeline-status/list p)  (listof any/c)

  p : pipeline?
A list of the exit statuses of all the pipeline members.


(shellify func)  procedure?

  func : procedure?
Convenience function for putting Racket functions into pipelines.

Takes a procedure which takes a string as its first argument and returns a string. Returns a procedure which will turn its current-input-port into a string and pass it to the original procedure as its first argument. It then displays the output string of the function to its current-output-port.

Struct property for alias-funcs. The property should be a function that takes the structure as an argument and produces a function that takes the argument list and produces a new one.

This may be renamed to be less confusing with shell/pipeline-macro/define-pipeline-alias.


(struct alias-func (func)
    #:extra-constructor-name make-alias-func)
  func : procedure?
Wrapper struct with prop:alias-func. An alias function must return a non-empty list suitable for a pipeline-member-spec.

If you want an alias in Rash, this is probably not what you want. See shell/pipeline-macro/define-pipeline-alias.

This will likely be renamed to be less confusing.

;; A simple case -- have an alias that sets initial arguments.
(define ls-alias (alias-func (λ args (list* 'ls '--color=auto args))))
;; Slightly more complicated: `find` requires that its path argument go before
;; its modifier flags.
(define find-files-alias (alias-func (λ args `(find ,@args -type f))))


(path-string-symbol? p)  boolean?

  p : any/c
Like path-string?, except it also includes symbols that would be valid paths.


(and/success e ...)

Like and, but only treats pipeline objects as truthy if they pass pipeline-success?.


(or/success e ...)

Like or, but only treats pipeline objects as truthy if they pass pipeline-success?.