deta: Functional Database Mapping

@#(struct:element #f (Bogdan Popa < #(struct:element #(struct:style #f (#(struct:target-url mailto:bogdan@defn.io))) (bogdan@defn.io)) >))

This library automatically maps database tables to Racket structs and lets you perform CRUD operations and arbitrary queries on them. Sort of like an ORM.

The API is currently stable, but small things may change before 1.0. Watch the GitHub repository if you want to stay on top of potential changes.

1 Principles

The main principle behind this library is "explicitness without tedium." By that I mean that it should be clear to someone who is reading code that uses this library how that code maps to database operations, while making the common cases of mapping data between database tables and Racket structs straightforward and simple.

1.1 Non-goals

• Support for every SQL dialect. Despite the fact that both SQLite and PostgreSQL are currently supported by this library, the focus is on PostgreSQL and SQLite just serves as a check to ensure that nothing is too PostgreSQL specific.

• Being a general purpose SQL DSL. For some queries you may have to resort to raw SQL or the sql library. deta is very much an "80% solution."

• Being externally-extensible. The SQL AST as well as all of the dialect code is considered private and any new dialects (such as MySQL) will have to be added to the library itself.

If you’re down with that, then, by all means, carry on and read the tutorial!

2 Compared to ...

2.1 Racquel

Racquel takes a more classic approach to database mapping by being a real ORM. It is based on the class system, with entities backed by mutable objects and with support for associations via lazy loading. Deta’s approach is the opposite of this by focusing on working with immutable structs, avoiding associations and any sort of lazy behavior altogether.

2.2 sql

The sql library is great at statically generating SQL queries. The problem is that the generated queries are not composable at runtime. You have to write macros to compose them and I’ve found that that gets tedious quickly.

On top of giving you composable queries – as you can hopefully see from the tutorial –, deta also automatically maps CRUD operations to structs, which is out of scope for sql.

3 Tutorial

deta builds upon the db library. You will use deta to generate your mappings and create queries, but the db library will be doing the actual work of talking to the database and handling transactions.

Let’s start by creating a database connection in the usual way:

> (require db)

>

> (define conn     (postgresql-connect #:database "deta"                         #:user     "deta"                         #:password "deta"))

Next, let’s define a schema for books:

> (require deta)

>

> (define-schema book     ([id id/f #:primary-key #:auto-increment]      [title string/f #:contract non-empty-string? #:wrapper string-titlecase]      [author string/f #:contract non-empty-string?]      [published-on date/f]))

The above generates a struct named book with fields for the table’s id, title, author and published-on columns, an associated “smart constructor” called make-book and functional setter and updater functions for each field.

> (require gregor)

>

> (define a-book     (make-book #:title "To Kill a Mockingbird"                #:author "Harper Lee"                #:published-on (date 1960 7 11)))

>

> (book-id a-book)

#<sql-null>

> (book-title a-book)

"To Kill A Mockingbird"

> (book-title    (update-book-title a-book (lambda (t)                                (string-append t "?"))))

"To Kill A Mockingbird?"

>

; schema entities are immutable so the above did not change a-book

> (book-title a-book)

"To Kill A Mockingbird"

We can use the schema to issue DDL commands to the database and create the table:

> (create-table! conn 'book)

While the DDL functionality is convenient for the purposes of this tutorial, in real world projects you should probably use something like north to manage your database table schemas.

Now that we have a table, we can insert the book that we created into the database:

> (define saved-book     (insert-one! conn a-book))

>

> (book-id saved-book)

1

Let’s insert a few more books:

> (void    (insert! conn (make-book #:title "1984"                             #:author "George Orwell"                             #:published-on (date 1949 6 8))                  (make-book #:title "The Lord of the Rings"                             #:author "J.R.R. Tolkien"                             #:published-on (date 1954 7 29))                  (make-book #:title "The Catcher in the Rye"                             #:author "J.D. Salinger"                             #:published-on (date 1949 7 16))))

And now let’s query for all of the books published before 1955:

> (require threading)

>

> (for/list ([b (in-entities conn (~> (from book #:as b)                                       (where (< b.published-on (date "1955-01-01")))                                       (order-by ([b.published-on #:desc]))))])     (book-title b))

'("The Lord Of The Rings" "The Catcher In The Rye" "1984")

Sweet! Here’s the query we just ran:

> (displayln    (~> (from book #:as b)        (where (< b.published-on (date "1955-01-01")))        (order-by ([b.published-on #:desc]))))

#<query: SELECT b.id, b.title, b.author, b.published_on FROM books AS b WHERE b.published_on < (DATE '1955-01-01') ORDER BY b.published_on DESC>

What about dynamic parameters, you may ask? Let’s wrap the above query in a function:

> (define (books-before year)     (~> (from book #:as b)         (where (< b.published-on ,(sql-date year 1 1)))         (order-by ([b.published-on #:desc]))))

>

> (for/list ([b (in-entities conn (books-before 1950))])     (book-title b))

'("The Catcher In The Rye" "1984")

>

> (for/list ([b (in-entities conn (books-before 1955))])     (book-title b))

'("The Lord Of The Rings" "The Catcher In The Rye" "1984")

Any time the query combinators encounter an unquote, that value gets replaced with a placeholder node in the query AST and, when the query is eventually executed, the value is bound to its prepared statement. This makes it safe and easy to parameterize your queries without having to worry about SQL injection attacks.

Oftentimes, you’ll want to query data from the DB that doesn’t match your schema. For example, let’s say we want to grab the number of books published by year from our database. To do that, we can declare a virtual schema – one whose entities can’t be persisted – and project our queries onto that schema.

> (define-schema book-stats     #:virtual     ([year date/f]      [books integer/f]))

>

> (define books-published-by-year     (~> (from book #:as b)         (select (as                   (cast (date_trunc "year" b.published-on) date)                   year)                 (count b.title))         (group-by year)         (order-by ([year]))         (project-onto book-stats-schema)))

>

> (for ([s (in-entities conn books-published-by-year)])     (displayln (format "year: ~a books: ~a"                        (book-stats-year s)                        (book-stats-books s))))

year: #<date 1949-01-01> books: 2 year: #<date 1954-01-01> books: 1 year: #<date 1960-01-01> books: 1

If we hadn’t wrapped our query with project-onto, then the data would’ve been returned as values which we could destructure inside the for loop, exactly like in-query from db.

It’s time to wind things down so let’s delete all the books published before 1950:

> (query-exec conn (delete (books-before 1950)))

Re-run the last query to make sure it worked:

> (for ([s (in-entities conn books-published-by-year)])     (displayln (format "year: ~a books: ~a"                        (book-stats-year s)                        (book-stats-books s))))

year: #<date 1954-01-01> books: 1 year: #<date 1960-01-01> books: 1

That’s all there is to it. You now know the basics of deta. Thanks for following along! If you want to learn more be sure to check out the reference documentation below.

4 Reference

4.1 Query

4.1.1 DDL

4.1.2 Entity CRUD

Raises a user error if any of the entities are based on virtual schemas.

Equivalent to:

(match (insert! conn e)

[(list e) e]

[_ #f])

#:batch-size controls how many rows to fetch from the databaase at a time. It is analogous to in-query’s #:fetch argument.

If there are no results then #f is returned.

Raises a user error if any of the entities don’t have a primary key field.

Equivalent to:

(match (update! conn e)

[(list e) e]

[_ #f])

Raises a user error if any of the entities don’t have a primary key field.

Equivalent to:

(match (delete! conn e)

[(list e) e]

[_ #f])

4.1.3 Query Expressions

q-expr		=		(and q-expr ...+)
		|		(array q-expr ...)
		|		(as q-expr id)
		|		(case [q-expr q-expr] ...+)
		|		(case [q-expr q-expr] ...+       [else q-expr])
		|		(fragment expr)
		|		(not q-expr)
		|		(or q-expr ...+)
		|		(list q-expr ...)
		|		(subquery expr)
		|		'(q-expr ...)
		|		,expr
		|		,@expr
		|		ident
		|		boolean
		|		string
		|		number
		|		app
app		=		(q-expr q-expr ...)
ident		=		symbol

The grammar for query expressions.

Tuples are created using the (list 1 2 3) or '(1 2 3) syntax:

> (select _ (in 1 '(1 2 3 4)))

(query "SELECT 1 IN (1, 2, 3, 4)")

> (select _ (in 1 (list 1 2 3 4)))

(query "SELECT 1 IN (1, 2, 3, 4)")

Dynamic lists can be turned into tuples using the ,@(list 1 2 3) syntax:

> (let ([xs (list 1 2 3)])     (select _ (in 1 ,@(map add1 xs))))

(query "SELECT 1 IN (2, 3, 4)")

Note that the splicing syntax splices scalar lists directly into the query so you must be careful when using it in conjunction with untrusted user input.

Arrays are created using the (array 1 2 3) syntax:

> (select _ (array-concat (array 1 2) (array 3 4)))

(query "SELECT ARRAY[1, 2] || ARRAY[3, 4]")

Various operators have built-in support and generate queries predictably. Operator names are always lower-case so (in a b) is valid, while (IN a b) is not. If you find an operator that you need doesn’t produce the query you expect, then open an issue on GitHub and I’ll fix it as soon as I can.

Within a query expression, the following identifiers are treated specially by the base (i.e. PostgreSQL) dialect. These are inherited by other dialects, but using them with those dialects may result in invalid queries.

4.1.4 Query Combinators

An error is raised if q is anything other than a SELECT query.

> (delete (from "users" #:as u))

(query "DELETE FROM users AS u")

> (define-schema user     ([id id/f #:primary-key #:auto-increment]      [username string/f]))

>

> (from "users" #:as u)

(query "SELECT * FROM users AS u")

>

> (from user #:as u)

(query "SELECT u.id, u.username FROM users AS u")

>

> (let ([tbl "users"])     (from ,tbl #:as u))

(query "SELECT * FROM users AS u")

>

> (~> (from (subquery (from user #:as u)) #:as out)       (select (count out.*)))

(query  "SELECT COUNT(out.*) FROM (SELECT u.id, u.username FROM users AS u) AS out")

Adds a GROUP BY clause to query. If query already has one, then the new columns are appended to the existing clause.

> (~> (from "books" #:as b)       (select b.year-published (count *))       (group-by b.year-published))

(query  "SELECT b.year_published, COUNT(*) FROM books AS b GROUP BY b.year_published")

> (~> (from "posts" #:as p)       (join "post_images" #:as pi #:on (= p.id pi.post-id))       (join #:left "comments" #:as c #:on (= p.id c.post-id))       (select p.* c.*))

(query  "SELECT p.*, c.* FROM posts AS p JOIN post_images AS pi ON p.id = pi.post_id LEFT JOIN comments AS c ON p.id = c.post_id")

The first form raises a syntax error if n is not an exact positive integer or 0.

> (~> (from "users" #:as u)       (offset 20)       (limit 10))

(query "SELECT * FROM users AS u LIMIT 10 OFFSET 20")

The first form raises a syntax error if n is not an exact positive integer or 0.

> (~> (from "users" #:as u)       (offset 10))

(query "SELECT * FROM users AS u OFFSET 10")

> (~> (from "users" #:as u)       (order-by ([u.last-login #:desc]                  [u.username])))

(query "SELECT * FROM users AS u ORDER BY u.last_login DESC, u.username")

> (define direction 'desc)

> (~> (from "users" #:as u)       (order-by ([u.last-login ,direction])))

(query "SELECT * FROM users AS u ORDER BY u.last_login DESC")

> (~> (from "artworks" #:as a)       (order-by ([a.year #:desc #:nulls-last])))

(query "SELECT * FROM artworks AS a ORDER BY a.year DESC NULLS LAST")

> (define-schema book-stats     #:virtual     ([year-published integer/f]      [books integer/f]))

> (~> (from "books" #:as b)       (select b.year-published (count *))       (group-by b.year-published)       (order-by ([b.year-published #:desc]))       (project-onto book-stats-schema))

(query  "SELECT b.year_published, COUNT(*) FROM books AS b GROUP BY b.year_published ORDER BY b.year_published DESC")

Omitting the select does not produce the expected query in this case since the books table has different fields from a book-stats value:

>

> (~> (from "books" #:as b)       (group-by b.year-published)       (order-by ([b.year-published #:desc]))       (project-onto book-stats-schema))

(query  "SELECT * FROM books AS b GROUP BY b.year_published ORDER BY b.year_published DESC")