5 Distributed programming with CapTP
(require goblins/ocapn/captp) | package: goblins |
CAUTION: CapTP support is still in early days. Please do not rely yet on this protocol being stable.
CapTP is a system for enabling distributed programming in a mutually suspicious network. It originated in the E programming language but has since appeared in Cap’N Proto, Waterken, Agoric’s SwingSet, and now Goblins.
5.1 What CapTP gives you
Invocation of remote objects resembles the same as with local objects. You can use <- and on which means that any most programs that were even originally designed mostly for local-only computing naturally scale out to a networked environment.
All this is done while mostly hiding the abstraction of the networked protocol from the user.
Object capability security is upheld. A remote machine cannot make use of any capability that has not been handed to it.
Live object references between CapTP endpoints are incredibly cheap, merely represented as integers on each side. This keeps message sizes small and efficient.
CapTP provides (acyclic) distributed garbage collection. Remote machines can indicate when they no longer need an object and the machine locally containing that reference can reclaim it if appropriate.
Promise pipelining means that messages can be sent to the resolution of a promise before that resolution actually occurs. Over the network this is represented as a pipeline of messages. This can reduce round trips significantly, which is a big win. To re-quote Mark S. Miller:
"Machines grow faster and memories grow larger. But the speed of light is constant and New York is not getting any closer to Tokyo."
CapTP is written independently of network transport abstractions. It can be run over local unix domain sockets, an OpenSSL connection, Tor Onion Services, or something custom. CapTP operates under the assumption of secure pairwise channels; the layer which provides the pairwise channels is called "MachineTP" and can be written in a variety of ways. A simple MachineTP abstraction is provided, but presently still requires some manual wiring. (In the future, even simpler abstractions will be provided which cover most user needs.)
All of these combine into an efficient and powerful protocol that, most powerfully of all, the author of Goblins programs mostly need not think too much about. This circumvents the usual years and years of careful message structure coordination and standardization for many kinds of distributed programs, which can simply be written as normal Goblins programs rather than bespoke protocols.
5.2 CapTP Limitations
Goblins’ CapTP provides acyclic distributed garbage collection. Cycles between servers are not automatically recognized. Full cycle-collecting distributed garbage collection has been written, but requires special cooperation from the garbage collector that we don’t have access to in Racket (or in most languages).
Goblins’ CapTP does not do anything about memory usage or resource management on its own. (Features for this will come as Spritely sub-projects in the future.)
While CapTP in theory routes capabilities to specific remote objects, since the network is mutually suspicious we can’t assume that the remote end isn’t conspiring some way to hand those capabilities to other objects that we don’t expect. The right way to think about this from an object capability perspective is that a remote misbehaving machine is equivalent to a misbehaving object with the surface area of the entire machine.
TEMPORARY: The API is heavily in flux.
5.3 CapTP usage example
There’s still work to be done in terms of making Goblins’ CapTP "easier" to use and set up. In the meanwhile, let’s give an example showing off Goblins’ Tor Onion Services support.
5.3.1 Launch a Tor daemon for Goblins
In order for this to work, Goblins needs a Tor daemon to talk to... Goblins can set up and manage the onion services itself once it has that. So far it seems like running a single Goblins tor process "as your user" is the easiest option, and let various Goblins processes connect to that.
Here’s an example config file template... replace <user> with your actual username:
DataDirectory /home/<user>/.cache/goblins/tor/data/ |
SocksPort unix:/home/<user>/.cache/goblins/tor/tor-socks-sock RelaxDirModeCheck |
ControlSocket unix:/home/<user>/.cache/goblins/tor/tor-control-sock RelaxDirModeCheck |
Log notice file /home/<user>/.cache/goblins/tor/tor-log.txt |
Put in ~/.config/goblins/tor-config.txt or something.
You’ll also want to make the relevant directories:
mkdir -p ~/.cache/goblins/tor/data |
Okay, now you can launch the Tor daemon:
tor -f ~/.config/goblins/tor-config.txt |
In the future we should probably make a raco launcher that does the above for users so they don’t need to read this section. :P
5.3.2 Two Goblins programs chatting over CapTP
Here’s our server, save as captp-alice.rkt:
#lang racket (require goblins goblins/actor-lib/bootstrap goblins/ocapn goblins/ocapn/netlayer/onion net/url) ;; Machine to talk to the network on (define-vat-run machine-run (make-vat)) (define mycapn (machine-run (define-values (onion-netlayer _onion-private-key _onion-service-id) (new-onion-netlayer)) (spawn-mycapn onion-netlayer))) ;; Get the command line argument for Alice's address (define alice-sref (command-line #:args (alice-sref-str) (string->ocapn-sturdyref alice-sref-str))) ;; Set up a vat for "alice" (define-vat-run a-run (make-vat)) (a-run (define alice-vow (<- mycapn 'enliven alice-sref)) (on (<- alice-vow "Alice") (lambda (heard-back) (displayln (format "Alice heard back: ~a" heard-back)) ;; we're done (semaphore-post finished)))) ;; A semaphore for us to signal once we've heard back from Alice (define finished (make-semaphore)) (sync finished)
Traditionally Alice talks to Bob, you see. So now we need Bob.
Now save the following client code as captp-bob.rkt:
#lang racket (require goblins goblins/actor-lib/bootstrap goblins/ocapn goblins/ocapn/netlayer/onion net/url) ;; Bob is going to be a "greeter" of sorts, so let's define what ;; that is (define (^greeter _bcom my-name) ; constructor args (lambda (your-name) ; invocation args (format "Hello ~a, my name is ~a!" your-name my-name))) ;; Now let's make a vat for bob, and run bob in it (define-vat-run b-run (make-vat)) (define bob (b-run (spawn ^greeter "Bob"))) ;; First, let's set up our "machine" for the network (define-vat-run machine-run (make-vat)) (machine-run (define-values (onion-netlayer _onion-private-key _onion-service-id) (new-onion-netlayer)) (define mycapn (spawn-mycapn onion-netlayer)) ;; Now we need a sturdyref for Bob... let's print it out so that ;; we know how to connect (define bob-sturdyref ($ mycapn 'register bob 'onion)) ;; And let's print out to the command line (displayln (format "Bob's sturdyref: ~a" (url->string (ocapn-sturdyref->url bob-sturdyref))))) ;; And we need some way to keep things open despite the other vats ;; humming along in their own threads... this kluge does it (sync (make-semaphore))
Okay, let’s try it out. Open up two terminals, and in the first one run:
racket captp-bob.rkt |
It’ll spit out a captp "sturdyref" URI. Copy it and in a new terminal paste it as the argument to captp-alice.rkt:
racket captp-alice.rkt STURDYREF-GOES-HERE |
You should see:
Bob heard back: Hello Bob, my name is Alice! |
If that worked, your connection succeeded! Horray!
5.4 CapTP API
procedure
(spawn-mycapn netlayer ...) → local-refr?
netlayer : local-refr?
Returns an object representing the machine with various methods that Christine really ought to document.
Sorry, these docs are pretty sparse; we hope to make them better. Hopefully the above example helps get you started.
5.5 Netlayers
5.5.1 Tor Onion Services
(require goblins/ocapn/netlayer/onion) | package: goblins |
procedure
(new-onion-netlayer [ #:tor-control-path tor-control-path #:tor-socks-path tor-socks-path #:tor-ocapn-socks-dir tor-ocapn-socks-dir])
→
local-refr? string? string?
tor-control-path : (or/c path? string?) = default-tor-control-path tor-socks-path : (or/c path? string?) = default-tor-socks-path
tor-ocapn-socks-dir : (or/c path? string?) = default-tor-ocapn-socks-dir
Returns three values to its continuation: the set up netlayer, the private key (formatted in a way that Tor recognizes) for this onion service should you wish to restore it with restore-onion-netlayer, and the "service id" of this instance (ie the public key, or rather the left side of the ".onion" in an onion address).
procedure
(restore-onion-netlayer [ #:tor-control-path tor-control-path #:tor-socks-path tor-socks-path #:tor-ocapn-socks-dir tor-ocapn-socks-dir])
→
local-refr? string? string?
tor-control-path : (or/c path? string?) = default-tor-control-path tor-socks-path : (or/c path? string?) = default-tor-socks-path
tor-ocapn-socks-dir : (or/c path? string?) = default-tor-ocapn-socks-dir
5.5.2 Fake Intarwebs
(require goblins/ocapn/netlayer/fake-intarwebs) | |
package: goblins |
Sometimes you want to just test whether or not your code works over captp without actually starting up a captp connection.
The "fake intarwebs" module is pretty... basic. But the idea is that there’s a "network" (like a fake version of the internet) which has a set of names mapped to locations which route to incoming connections. Each abstract machine sets up its own captp process with corresponding netlayer.
No attempt is made to actually process-separate the systems. (Though this too could be done, it could be nice to compose this module with Racket’s places... if you try this, let us know!) However, the objects do communicate over the captp system, just entirely locally and in-process.
Look at the unit tests at the bottom of goblins/ocapn/netlayer/fake-intarwebs.rkt for an example of usage.