orchestra
| (require orchestra) | package: orchestra |
Library to build distributed real time applications like multi-player games and collaborative programs.
It uses a client-server architecture in which the server stores a state that is made available to clients by exchanging messages over websocket connections.
NOTE: The library is ready to use. This documentation is in an advanced state but still incomplete.
1 Examples of use
Orchestra comes with an app containing two built-in examples of use.
Once you have installed orchestra, you can serve the app with
raco orchestra-examples |
By default it listens on port 8000, so you can visit http://localhost:8000 with your browser. If you would like to use a different port use the -p option
raco orchestra-examples -p 9000 |
The main page shows a menu that shows a menu with links to two examples: Drag and Timer.
In the main page of both examples you have a menu with the list of existing worlds of the app. You can add more world by clicking on the button ’Add links pair’.
A world is an independent instance of the same app. In the two examples each world has two components. Controller and Controlled in the drag example and Manager and Observer in the timer example.
To follow the example you need to visit the two links of the same world with two different browser windows or with two different devices. The examples server accepts connections from different devices to allow this to work.
The drag example consists of a box inside an enclosing area. The controller side allows you to drag the box (either with a mouse or touch). The controlled box cannot be directly moved by the user. Instead it follows the movements of the controller.
The timer example has a 10 seconds countdown that is synchronized between the manager and the observer. The manager has a button that allows to start, pause and restart the countdown while the observer simply follows the manager.
2 Introduction and Design
The steps to build a distributed application from scratch (this is, starting with websocket connections) can be divided in two: the logic and the wiring.
The logic involves defining the state and a system of messages between the server and clients along with handlers to process them.
The wiring consists in giving a structure to isolated websocket connections to allow for the logic to be implemented.
The library takes care of the wiring and let’s you focus on the logic.
To be more specific, let us look at the core of orchestra’s functioning: the world.
A world in orchestra is a description of all the elements that go in an app along with how they interact with each other. The term also denotes a specific instance of that world.
Say for example you are building a website for people to play chess. In this case you can define the world to be a single game of chess, which involves two players and potential observers.
To define a world you need a state and roles.
The state can be any value or data structure. When you define a state you also provide a list of operations that can be applied to it. The state is stored in the server.
The roles define the behaviour of clients. Each client connection is assigned a role and there can be more than one client with the same role.
In the chess website example, the state would be the position of the pieces, and whose turn it is. There would also be three roles, white pieces player, black pieces player and the observer. Note that each role has a different behaviour (which motivates the definition of roles this way) the white role can only move the white pieces when it is their turn, the black role can only move the black pieces when it is their turn and the observer role can only watch the game without moving any pieces.
Once you have a world definition, you need to define the messages that can be exchanged between the clients and server. You also need to write functions that process these messages and execute actions based on them. We refer to these functions as handlers.
In the chess example, each time a player moves a piece, it sends a message to the server with the move and the server updates the state accordingly and sends messages to the other roles to inform them of the change of state.
Each time you define a new world, a galaxy is also created. The galaxy contains information about the definition of the world and all it’s instances.
With these definitions, we can be more precise about the wiring and the logic.
The state
The roles
The messages
The handlers.
The wiring makes an app work once these definitions are provided.
3 Tutorial
3.1 Drag Example
The code of the drag example can be found in the examples/drag sub-folder.
The drag example has two components the controller and the controlled.
As the controller is dragged, the controlled follows its movements. The app works properly even if the size of the moving area is different on each side.
The controller detects dragging mode
The controller tracks the move of the pointer (mouse or touch)
The controller sends movement information to the server
The server received the movement information and
3.1.1 The Logic
Let us go over how the 4 elements of the logic are defined this app.
The app works even if the size of the enclosing area on the controller and controlled side are different. For that reason the saved coordinates are normalized with respect to the width and height of the enclosing area.
As an example suppose the width and height of the enclosing area are 100 and 200 pixels respectively. The position where the upper left corner of the box is 10 pixels right from the left edge and 40 pixels down from the upper edge is represented by (0.1,0.2) since 0.1=10/100 and 0.2=40/200.
Three operations are defined on the state. A get operation that simply retrieves the value of the state. A set operations that sets the value of the state. A move operation that updates the position provided how much to move on each coordinate.
role-info : Message sent from the server to the clients when a connection is established. It contains the role of the corresponding client either "controller" or "controlled".
init : Message sent from the server to the clients when a connection is established. It contains the position (the value of the state).
move : Message sent from the controller to the server and also from the server to the controlled. It contains the deltas (the change in position) for the horizontal and vertical coordinates. The deltas are normalized with respect to the width and height of the enclosing area.
The handlers
The handlers are the functions that take actions when a message is received.
When a client receives a role-info message, it’s handler depend on the role. If it is the controlled it ignores it and waits for other message. On the other hand the controller activates dragging for it’s box.
When a client receives the init message, it sets the position of the box to the one provided in the message.
When the server receives a move message, it updates the state (the saved normalized position) with the coordinates that correspond to the move and it resends the same message to controlled. when controlled receives the message it moves it’s box according to the provided deltas.
3.1.2 Client side
The JavaScript code implementing the client functionality is the Drag class in the file examples/drag/static/drag.js.
Both the controller and the controlled instantiate the Drag class. The first argument is a CSS selector of a single HTML element which is the element that can be moved using the class. The second argument is optional, it is the name of a CSS class to add to the element while it is being dragged. The third argument, also optional, says whether or not there should be a websocket connection and by default it is false.
For the example we need to focus on three methods, set_up_controller, set_position and move.
set_up_controller makes the element draggable using the mouse or touch. The controller calls this method when it receives the role-info message.
set_position Moves the box to a specific position. It takes as argument a vector with the normalized coordinates of the box in the enclosing area. Both the controller and controller call this method when they receive the init message.
move Moves the box according to a delta vector. The first argument is the delta vector, this is it’s two elements say how much to move the x and y coordinates according to the current position. The second argument is boolean and it says where the delta vector contains normalized deltas or not. When they are not normalized the vector is taken as an exact number of pixels, when they are normalized they are proportions with respect to the width and height of the enclosing elements. By default it is true. The controlled calls this method when it receives a move message.
The drag menu comes with three links that help understand how the client side works. They are next to the ’Add links pair’ button and they are
Client intro 1: Shows the effects of set_up_controller. It has a draggable element inside an enclosing area and a button to activate and deactivate dragging.
Client intro 2: Shows the functioning of the move method. It has widgets that allow you to change the size and direction of the change in each coordinate.
- Client intro 3: Has a version of the app without websockets. It shows two draggable elements with enclosing areas of different sizes. The bigger one is labeled as controller and the smaller one as controlled. The steps followed in this page are
The controller detects dragging is happening and saves the position of the pointer (mouse or touch) when it starts.
- While dragging is on, the controller takes every mouse movement and
Computes the difference of each coordinate with the saved pointer position (the deltas).
Computes the normalized deltas (divides the changes in x and y by the width and height, respectively, of the enclosing element).
Calls the move method with the normalized deltas.
Updates the saved position of the pointer.
Calls the move method of the controlled with the normalized deltas.
When the controller detects dragging stops, it stops tracking the movement of the pointer.
In the example app everything works exactly the same than in Client intro 3, except step 2-e where instead of calling move on the controlled element the controller sends a move message to the server.
3.1.3 Server Side
The file examples/drag/example_drag.rkt in the orchestra package contains the server side code of the drag example.
The server side of an orchestra application requires the following steps
Define the state(s)
Define the world
Define the handlers (of messages)
Define the app websocket dispatcher
Integrate the websocket dispatcher with the rest of your application
The orchestra package helps you with steps 1 to 4 and for step 5 you use functions of racket’s web server. The Web Applications in Racket documentation shows how to use the serve/servlet function.
Orchestra helps you to create a dispatcher (in step 4) that takes care of managing the websocket connections of your application. dispatchers are not compatible with serve/servlet. You need to use the lower-level function serve.
The web server documentation about dispatchers is very condensed and hard to follow at first. A very good and friendlier introduction is offered in the blog post The Missing Guide to Racket’s Web Server.
Defining the state
A state consists of a value or values that can be modified or queried by actions defined by the user. We refer to such values as state value. In the context of this library, different worlds have different states. When defining a state it is necessary to manage states for different worlds. For this purpose, each state should also have what we call the state-reference which should manage different states indexed by world ids (which are integers).
A state-reference can be anything, but natural objects for this purpose are hash tables or vectors, which allow to index different values.
To define a state you use the macro define-state.
As mentioned in The Logic the state is a vector with the normalized coordinates of the dragabble element.
(require orchestra)
3.2 Timer Example
3.2.1 Logic
The timer class takes an HTML element where the timer is displayed. Then the number of seconds for the timer, then a wsync server date object and finally the number of milliseconds to update the timer.
When the websocket connects, it receives a message with the time and the role.
There are two roles the manager and the observer. The manager has buttons to start, pause, resume and restart the timer. It also has an input to change the time. By default the time is 5 minutes but the manager can change it.
The client connects and receives the remaining time and started time. When the started time is null it means the timer hasn’t started or it is in pause. When it is not null, it is the timestamps with the moment at which the timer was started.
Based on that the client should set the time and start it or have it paused.
Another message with the role information (manager or observer) is sent. The observer side does nothing while the manager shows the buttons play/pause.
When play is pressed, a play message is sent to the server. The server sends the play message with the remaining time and the server timestamp.
When pause is pressed, a pause message is sent to the server, it computes the remaining time and it sends it to the manager and observer.
Each message has the form {action : action-name ... } followed by additional parameters needed for the provided action.
- role-info : has an additional key role containing either "manager" or "observer".
server -> manager : The client adds the buttons start, pause and restart.
server -> observer : The client does nothing.
- remaining-time : has an additional argument started which can either be 'null or a positive number representing the latest timestamp in milliseconds that the timer was started.
server -> manager : Sets and shows the received remaining time. If started is not 'null the timer runs assuming it had been started at the provided time. Otherwise it is paused.
server -> observer : Same than manager.
- start : Depending on the sender, it might have no additional argument or one additional argument started
manager -> server : it is sent with no additional argument. The server sets the state started time to the moment in which the message is received.
server -> manager : it is sent along with started. Resumes the timer with the provided started timestamp.
server -> observer : Same than the previous one.
- pause : depending on the sender, it might have no additional argument or one additional argument remaining_time
manager -> server : it is sent with no additional argument. The server updates the remaining time according to the time the message is received and send a pause message with the time remaining to both the manager and the observer.
server -> manager : Pauses the timer and sets it to the received remaining time.
server -> observer : Same than the previous one.
- restart : No additional argument.
manager -> server : Sets the remaining time to the original time and started to 'null. Re-sends the message to the manager and observer,
server -> manager : Sets the remaining time to the full original time.
server -> observer : Same than the previous one.
4 Reference
Orchestra is an interface for building applications that require synchronization among different clients. An example of application are multiplayer online games.
The most granular element in orchestra is the occurrence. Each occurrence has a unique websocket endpoint that identifies it.
Each occurrence has a role. The user defines the different roles and how they behave when the shared state changes. It is also possible to provide a maximum number of occurrences for a given role.
The set of all roles and occurrences is called a world. All worlds of the same type are grouped inside a galaxy.
Each galaxy has a base URL. And each occurrence has a unique websocket endpoint formed by the base URL plus a unique code.
4.1 Structs
| (require orchestra/structs) | package: orchestra |
symbol’s are used to represent role’s and occurrence’s.
struct
state-refs : (hash/c state? state-reference/c) roles : (hash/c symbol? integer?) code-bytes : integer? relayer : ws-relayer? codes : (hash/c integer? (hash/c symbol? (or/c string? (vector/c string?))))
roles is a hash? whose keys are roles and their value is the maximum number of such role that can exist in a world.
code-bytes is the number of bytes to use when generating the unique random codes. Codes are strings twice as long of code-bytes where each character is either a digit or letter between a and f.
relayer is a ws-relayer? that manages the connections of the different worlds in the galaxy.
codes is a hash table whose keys are world ids and their corresponding value is another hash whose keys are roles. The value for each role depend on whether the role can have more than one instance or not. If the role can only have one instance the value is the unique code in the world for that role. If it can have multiple instances, the value is a vector whose values are the unique codes.
struct
(struct exn:orchestra exn ())
struct
4.2 World
| (require orchestra/world) | package: orchestra |
syntax
(define-world name #:state (state ...) #:roles (role ...) #:code-bytes N maybe-db-executor maybe-json-messages)
role =
| role-name | (role-name max-role-occurrences) maybe-db-executor = #:db-executor db-executor maybe-json-messages = #:json-messages json-messages
state : state?
role : identifier?
max-role-occurrences : positive-integer?
N : positive-integer?
db-executor : (or/c db-executor/c 'memory)
json-messages : boolean?
If db-executor is not provided it takes the value 'memory, which creates an in-memory SQLite database.
When json-messages is #t (the default value) then all receiving messages are assumed to be JSON strings and messages that are send are automatically transformed to JSON strings before being sent.
N is the number of bytes to use to generate random strings for the unique URLs. The actual length of the strings is twice the value of N.
name-galaxy is a galaxy struct for worlds of type name.
name:db-executor is a database executor. The world used for storing the worlds parameters and the endpoints for the different roles.
name:let/db syntactic sugar for name:db-executor. (name:let/db conn body ...+) is equivalent to (name:db-executor (lambda (conn) body ...+))
name-index a schema witn a single column next-id and a single row. It’s value is the next number to be used as the world id. It is incremented each time a new world is created.
- name-endpoint is a schema containing the endpoints information in the galaxy. It has the columns
endpoint is a string with the unique code for the given observation.
role-in-schema is a string with the name of the role.
role-index is an integer corresponding to the index of the role (relevant when a role can have more than one instance).
world-id is an integer with the world id.
name:add-world Adds a state with the next available index to all the state references. It returns the id of the new world. It takes two optional keyword arguments #:add-endpoints? (default #t) and #:load? (default #t). When the first one is true, all endpoints are created along with the new world. When the second one is true, endpoints are also loaded.
name:remove-world Takes a world id as argument and erases that index from all the states in the world. It also erases all the entries with the provided world id in the endpoints table and the websocket relayer of the galaxy.
name:lookup-max-role-id Takes two arguments a role (a symbol) and a world id (an integer) and returns the highest existing role id for that combination of values in the endpoints table. If no such combination of role and world exists it returns #f.
name:lookup-world-id Takes a unique code as argument (a string) and returns the world id (an integer) associated with it. If there unique code does not exist in the endpoints table it returns #f.
name:add-endpoint Takes two arguments, a role (a symbol) and a world id (an integer). It generates a random unique code and adds it to the endpoints table for the provided role and unique world id. The length of the random code is twice the number of bytes declared when defining the world. Optionally it takes the keyword argument #:code which allows you to provide a unique code as a string.
For each state the function name:state-do is created. For each of them the first argument is an action name (a symbol?) followed by any arguments necessary for the corresponding action.
name:existing-ids is a thunk that returns a list with all the world id’s present in the endpoints database
name:endpoints-to-codes-hash is a function that takes a world id and adds all the endpoints in the database with that world id and adds them to the codes hash.
name:code->endpoint-entity is a function that takes a unique code and returns an entity from the endpoints table with that unique code. If the unique code is not present in the table it returns #f.
name:load-endpoint is a function that takes a unique code and adds entries for it to the relayer and codes hash. If no such endpoint exists the exception exn:orchestra:invalid-code is raised.
name:unload-endpoint is a function that takes a unique code and removes it from the relayer and codes hash. If no such endpoint exists the exception exn:orchestra:invalid-code is raised.
name:load-world Takes a world id as argument and loads all the elements in the endpoints table with that world id.
name:unload-world Takes a world id as argument and unloads all elements in the endpoints table with that world id.
Defines the parameters name:current-id, name:current-role, name:current-role-index and name:current-code. They are used by the message function.
name:message Takes two arguments, a role and a message. It sends the message to the websocket connection of the provided role in the world given by the parameter name:current-id. If there is more than one instance of the provided role, the message is sent to all of them. To send the message to a specific instance of that role you can additionally provide the keyword argument #:index with the integer of the role index to which you want to send it.
name:create-dispatcher is a macro that defines a dispatcher for websocket connections. The first argument must be the keyword #:url-match followed by a regular expression to match a URL paths. Then optionally followed by #:subprotocol-list to which you should provide a list of protocols (as symbols not string). If not provided the subprotocol list is taken to be (#f). There can be more than one subprotocol section for each #:url-match. For each subprotocol list, you must provide a list of keyword arguments #:role-handler and optionally #:role-onopen where role is the name of the role. All roles must be included. So the list of role handlers are applied for connection that has the provided prefix and the provided subprotocols. If at some point you want the handler to stop receiving messages and close the connection you can use the function handler-exit provided by orchestra/utils.
4.3 Database
| (require orchestra/database) | package: orchestra |
value
db-executor/c : contract? = (-> (-> connection? any) any)
procedure
(create-db-executor connect-proc #:max-connections max-connections #:max-idle-connections max-idle-connections) → db-executor/c connect-proc : (-> connection?) max-connections : positive-integer? max-idle-connections : positive-integer?
It creates a pool of connections using connect-proc, it passes max-connections and max-idle-connections to connection-pool.
The return value is a function that takes another function as argument. The argument function should take a connection? as argument and do something with it.
#lang racket/base (require orchestra/database db) (define dbfile "/tmp/test.db") (define dbexecutor (create-db-executor (lambda () (sqlite3-connect #:database dbfile #:mode 'create )))) (dbexecutor (λ (db) (query-exec db "CREATE TABLE example(id INTEGER)")))
4.4 State
| (require orchestra/state) | package: orchestra |
value
value
syntax
(define-state name #:new new-proc #:add add-proc #:remove remove-proc #:actions (hash action-name action-proc ... ...))
new-proc : (-> state-reference/c)
add-proc : (-> state-reference/c integer? any)
remove-proc : (-> state-reference/c integer? any)
action-name : symbol?
action-proc : state-action/c
add-proc should be a function that takes a state-reference, an integer and adds a new state value to the provided index. If the index already exists it doesn’t do anything.
remove-proc should take a state reference and an integer, and removes the index and its value.
action-name is a symbol representing some action and action-proc a procedure? that performs the desired action. action-proc should take a state reference and integer as the first two arguments followed by any other parameters, if any, necessary to perform the action. The only restriction on the additional arguments is that they cannot be keyword arguments.
Actions are used to query or modify state values.
Example:
#lang racket/base (define INIT-VALUE 0) (define (hvalue-new) (make-hash)) (define (hvalue-add a-hash n) (unless (hash-has-key? a-hash n) (hash-set! a-hash n INIT-VALUE))) (define (hvalue-remove a-hash n) (when (hash-has-key? a-hash n) (hash-remove! a-hash n))) (define (hvalue-get a-hash n) (hash-ref a-hash n)) (define (hvalue-set! a-hash n new-val) (hash-set! a-hash n new-val)) (define-state 'hval #:new hvalue-new #:add hvalue-add #:remove hvalue-remove #:actions (hash 'get hvalue-get 'set hvalue-set!))
On the code above the state reference is a hash table whose keys are the world indices.
procedure
(state-new name) → state-reference/c
name : symbol?
procedure
name : symbol? ref : state-reference/c n : integer?
procedure
(state-remove name ref n) → any
name : symbol? ref : state-reference/c n : integer?
procedure
(state-action name action) → state-action/c
name : symbol? action : symbol?
4.5 Relayer
| (require orchestra/ws-relay) | package: orchestra |
A websocket relayer allows to send messages to websocket connections corresponding to different endpoints.
struct
(struct ws-relayer (relay-hash cleanup-thread))
relay-hash : (and/c hash? hash-equal? (not/c immutable?) hash-strong?) cleanup-thread : (or/c #f thread?)
cleanup-thread is either #f or a thread that periodically loops over all connection managers in the values of relay-hash and gets rid of the finished connections in each of them.
procedure
(make-ws-relayer #:cleanup-frequency cleanup-frequency)
→ ws-relayer? cleanup-frequency : 300
procedure
(add-cmanager! a-relayer code [ #:single-connection? single-connection?]) → void? a-relayer : ws-relayer? code : string? single-connection? : boolean? = #f
procedure
(remove-cmanager! a-relayer code) → void?
a-relayer : ws-relayer? code : string?
procedure
a-relayer : ws-relayer? code : string? message : (or/c string? bytes? input-port?)
procedure
(ws-relay-keys a-relayer) → (listof string?)
a-relayer : ws-relayer?
4.6 Websocket Manager
| (require orchestra/ws-manager) | package: orchestra |
A websocket connection manager allows you to group together different websocket connections and to send messages to all elements of the group. It also provides a mechanism to get rid of finished connections in the group. The main use in orchestra is to manage connections from the same endpoint.
struct
(struct cmanager ( connections-hash single-connection? cleanup-thread)) connections-hash : (and/c hash? hash-equal? (not/c immutable?) hash-strong?) single-connection? : boolean? cleanup-thread : (or/c #f thread?)
single-connection? is #t when the role accepts a single connection and #f otherwise.
cleanup-thread is either #f or a thread that runs periodically and gets rid of finished connections in connections-hash.
You are unlikely to need to create a cmanager struct from scratch. For that use
procedure
(make-conn-manager [ #:single-connection? single-connection? #:cleanup-frequency cleanup-frequency]) → cmanager single-connection? : boolean? = #f cleanup-frequency : (or/c #f positive-integer?) = #f
procedure
(send-message cmgr message) → void?
cmgr : cmanager? message : (or/c string? bytes? input-port?)
procedure
(ncons cmgr) → nonnegative-integer?
cmgr : cmanager?
procedure
(add-connection! cmgr c) → void?
cmgr : cmanager? c : ws-conn?
procedure
(clean-connections! cmgr) → any/c
cmgr : cmanager?
4.7 Utils
| (require orchestra/utils) | package: orchestra |
procedure
(random-code nbytes) → string?
nbytes : nonnegative-integer?
procedure
(ws-request? req) → boolean?
req : request?