pyffi - Use Python from Racket🔗ℹ

4.1 Python 3.1x🔗ℹ

In order to use pyffi you need Python 3.10 or newer.

The official distribution of Python is here: https://www.python.org/downloads/

If you prefer to use an alternative source of distribution (read your favorite package manager), that’s fine too - as long as it includes both the standard interpreter as well as the shared library libpython.

4.2 One-time Configuration🔗ℹ

The last installation step consists of telling pyffi where your shared library libpython is placed.

The easiest way of doing this is to run the script configure-pyffi.

• Open a terminal.

• Check that python3 or python invokes your Python interpreter.
  

• Run: raco pyffi configure

If the command python3 is not in your path, then you can write this instead:

• raco pyffi configure <path-to-your-python-command>

This will find the location of your shared library, print it, and, store it in your Racket preferences under the key pyffi:libdir.

You are now ready to use pyffi.

4.3 Optional: NumPy for scientific computing🔗ℹ

The Python package NumPy has the slogan:

The fundamental package for scientific computing with Python.

If you want to use pyffi/numpy then you need to install NumPy in your Python environment first.

There are many ways of installing NumPy, but the simplest is to use the following in a terminal:

• python3 -m pip install numpy

5.1 Atomic values: numbers, Booleans and None🔗ℹ

Atomic Python values (numbers, Booleans and None) are automatically converted to their corresponding Racket values.

> (require pyffi)

> (initialize)

> (post-initialize)

> (run "12")

12

> (run "34.")

34.0

> (run "5+6j")

5.0+6.0i

> (run "False")

#f

> (run "True")

#t

> (list (run "None"))

'(#<void>)

5.2 Compound Values: strings, tuples, lists, and, dictionaries🔗ℹ

Compound (non-atomic) Python values such as strings, tuples, lists and dicts are not converted to Racket values. Instead they are wrapped in a struct named obj. Due to a custom printer handler these wrapped values print nicely.

> (run "'Hello World'")

(obj "str" : 'Hello World')

> (run "(1,2,3)")

(obj "tuple" : (1, 2, 3))

> (run "[1,2,3]")

(obj "list" : [1, 2, 3])

> (run "{'a': 1, 'b': 2}")

(obj "dict" : {'a': 1, 'b': 2})

The values display nicely too:

> (displayln (run "'Hello World'"))

Hello World

> (displayln (run "(1,2,3)"))

(1, 2, 3)

> (displayln (run "[1,2,3]"))

[1, 2, 3]

> (displayln (run "{'a': 1, 'b': 2}"))

{'a': 1, 'b': 2}

Printing and displaying a Python object use the __repr__ and __str__ methods of the object respectively.

The idea is that Racket gains four new data types: pystring, pytuple, pylist and pydict.

To convert a compound value use pystring->string, pytuple->vector, pylist->list or pydict->hash.

> (pystring->string (run "'Hello World'"))

"Hello World"

> (pytuple->vector (run "(1,2,3)"))

'#(1 2 3)

> (pylist->list (run "[1,2,3]"))

'(1 2 3)

> (pydict->hash (run "{'a': 1, 'b': 2}"))

'#hash(("a" . 1) ("b" . 2))

Similarly, you can convert Racket values to Python ones.

> (string->pystring "Hello World")

(obj "str" : 'Hello World')

> (vector->pytuple #(1 2 3))

(obj "tuple" : (1, 2, 3))

> (list->pylist '(1 2 3))

(obj "list" : [1, 2, 3])

> (hash->pydict (hash "a" 1 "b" 2))

(obj "dict" : {'b': 2, 'a': 1})

It’s important to note that creating Python values using string->pystring, vector->pytuple, list->pylist and hash->pydict is much more efficient than using run. The overhead of run is due to the parsing and compiling of its input string. In contrast string->pystring and friends use the C API to create the Python values directly.

The data types also have constructors:

> (pystring #\H #\e #\l #\l #\o)

(obj "str" : 'Hello')

> (pytuple 1 2 3)

(obj "tuple" : (1, 2, 3))

> (pylist 1 2 3)

(obj "list" : [1, 2, 3])

> (pydict "a" 1 "b" 2)

(obj "dict" : {'a': 1, 'b': 2})

The new types pystring, pytuple, pylist and pydict can be used with for.

> (for/list ([x (in-pystring (string->pystring "Hello"))]) x)

'(#\H #\e #\l #\l #\o)

> (for/list ([x (in-pytuple (vector->pytuple #(1 2 3)))]) x)

'(1 2 3)

> (for/list ([x (in-pylist (list->pylist '(1 2 3)))]) x)

'(1 2 3)

> (for/list ([(k v) (in-pydict (hash->pydict (hash "a" 1 "b" 2)))]) (list k v))

'(((obj "str" : 'b') 2) ((obj "str" : 'a') 1))

5.3 Builtin functions and modules🔗ℹ

> (run* "x = 1+2")

Here the statement x = 1+2 is parsed, compiled and executed. The result of the expression 1+2 is stored in the global variable x.

But due to the overhead of run it is better to make a direct variable reference.

> main.x

3

Here main is the name we have given to the module used for the global namespace of the Python interpreter. The dotted identifier main.x thus references the variable x in the global namespace.

Since Python modules are first class values, we can see their printed representations:

> main

(obj "module" : <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)>)

> builtins

(obj "module" : <module 'builtins' (built-in)>)

> (builtins.abs -7)

7

> (builtins.list "Hello")

(obj "list" : ['H', 'e', 'l', 'l', 'o'])

> (builtins.range 2 5)

(obj "range" : range(2, 5))

> (builtins.list (builtins.range 2 5))

(obj "list" : [2, 3, 4])

If you find the name builtins too long, then you can give it a new, shorter name.

> (define b builtins)

> (b.abs -7)

7

If you access the abs functions directly, you get a callable object:

> b.abs

(obj callable "builtin_function_or_method" : <built-in function abs>)

A callable object can be used just like a normal Racket function:

> (map b.abs '(1 -2 3 -4))

'(1 2 3 4)

To use functions from the Python standard library, you need to import it before you can use it. The standard library sys provide a lot of system information. Let’s use it to find the version of the Python interpreter.

> (import sys)

> sys.version_info

(obj "version_info" : sys.version_info(major=3, minor=14, micro=1, releaselevel='final', serial=0))

The list of modules in The Python Standard Library is long, so let’s just try one more.

We want to print a text calendar for the current month.

The expression (calendar.TextCalendar) instantiates a TextCalendar object. The syntax (object .method arg ...) is used to invoke the method formatmonth with the arguments 2022 and 7 (for July).

The documentation for formatmonth shows its signature:

formatmonth(theyear, themonth, w=0, l=0)

The two first arguments theyear and themonth are positional arguments and the two last arguments w and l are keyword arguments both has 0 has as default value.

The keyword argument w specifies the width of the date columns. We can get full names of the week days with a width of 9.

> (displayln ((calendar.TextCalendar) .formatmonth 2022 7 #:w 9))

July 2022   Monday   Tuesday  Wednesday  Thursday   Friday   Saturday   Sunday                                              1         2         3      4         5         6         7         8         9        10     11        12        13        14        15        16        17     18        19        20        21        22        23        24     25        26        27        28        29        30        31

5.4 Objects, Callable objects, Functions, Methods and Properties🔗ℹ

All values in Python are represented as objects. This differs from Racket, where most data types (e.g. numbers and strings) aren’t objects.

In the data model used by Python, all objects have an identity, a type and a value. In practice the identity of a Python object is represented by its address in memory [the CPython implementation never moves objects].

To represent a Python object in Racket it is wrapped in an obj struct. The structure contains the type name as a string and a pointer to the object. The wrapper sets the structure property gen:custom-write to display, write and print the wrapped objects nicely.

The result of repr() is used to write an object.
The result of str() is used to display an object.

> (define s (string->pystring "foo"))

> (repr s)

"'foo'"

> (writeln s)

(obj "str" : 'foo')

> (str s)

"foo"

> (displayln s)

foo

Functions and in general callable objects support the well-known syntax f(a,b,c). Such callable objects are wrapped in an callable-obj struct, which has obj as a super type. The callable-obj use the struct property prop:procedure to make the wrapper applicable.

> (run* "def f(x): return x+1")

> (define f main.f)

> f

(obj callable "function" : <function f at 0x120cbb060>)

> (f 41)

42

Function calls with keywords work as expected. A Python keyword is simply prefixed with #: to turn it into a Racket keyword, as this example shows:

> (run* "def hello(name, title='Mr'): return 'Hello ' + title + ' ' + name")

> (displayln (main.hello "Foo"))

Hello Mr Foo

> (displayln (main.hello #:title "Mrs" "Bar"))

Hello Mrs Bar

In order to illustrate methods, let’s look at the Calendar class in the calendar module.

> (import calendar)

> calendar.Calendar

(obj callable "type" : <class 'calendar.Calendar'>)

Calling the class gives us an instance object. We pass 0 to make Monday the first week day.

> (calendar.Calendar #:firstweekday 0)

(obj "Calendar" : <calendar.Calendar object at 0x120e4dbe0>)

One of the methods of a calendar object is monthdatescalendar.

> (define cal (calendar.Calendar #:firstweekday 0))

> cal.monthdatescalendar

(obj callable "method" : <bound method Calendar.monthdatescalendar of <calendar.Calendar object at 0x120f00e10>>)

The syntax obj.method gives us a bound method, which we can call. Bound methods are wrapped in method-obj to make them applicable.

However, we can also invoke the monthdatescalendar method directly with the help of the syntax (obj .method argument ...).

> (pyfirst (pyfirst (cal .monthdatescalendar year month)))

(obj "date" : datetime.date(2022, 8, 29))

Method invocations can be chained. That is, if a method call returns an object, we can invoke a method on it. The fist element of a list can be retrieved by the pop method, so we can replace the two calls to pyfirst with two invocations of .pop.

> (cal .monthdatescalendar year month  .pop 0 .pop 0)

(obj "date" : datetime.date(2022, 8, 29))

Besides methods an object can have properties (attributes). The syntax is obj.attribute. Most Python objects carry a little documentation in the oddly named __doc__ attribute.

> (displayln cal.__doc__)

Base calendar class. This class doesn't do any formatting. It simply provides data to subclasses.

5.5 Exceptions🔗ℹ

An exception on the Python side is converted to an exception on the Racket side. The exception will be printed with a full traceback.

> (run "1/0")

run: Python exception occurred;

ZeroDivisionError: division by zero

File "<string>", line 1, in <module>

6.1 The Big Picture🔗ℹ

In order to use the functions in pyffi you need to start a Python interpreter. The call (initialize) does just that. A standard approach would be to make the call in your "main" module of your program.

That the Python interpreter isn’t available until the main module has been instantiated leads to a few complications. The problem is the interpreter instance is not available when the modules required by "main" is instantiated.

As an example: The module pyffi/numpy contains bindings for NumPy. If the main module looks like this:

#lang racket

(require pyffi pyffi/numpy)

(initialize)

Then pyffi/numpy is instantiated before the interpreter is started. This means pyffi/numpy can’t inspect the Python module numpy to get the function signatures it needs.

To solve this problem pyffi/numpy registers a number of initialization thunks to be run after the interpreter has started. The function post-initialize runs these initialization thunks.

To sum up, the typical main module for a program that uses pyffi starts:

#lang racket

(require pyffi pyffi/numpy)

(initialize)

(post-initialize)

6.2 Reference🔗ℹ

The precise steps taken are:

• The locations of libpython and the folder containing the standard library is fetched from the preferences (keys 'pyffi:libdir and 'pyffi:data).

• Calls Py_Initialize which starts a Python interpreter. Initializes the table of of loaded modules and creates the modules builtins, __main__ and sys.

• Imports __main__, builtins, operator, traceback and inspect.

• Creates instances of True, False, and, None.

8.1 Python Lists - pylist🔗ℹ

Despite the name a Python list is not a singly linked list, but an array. Objects with the type "list" will be called pylist to tell them apart from standard Racket lists.

The operations pylist, list->pylist and vector->pylist can be used to construct pylists from Racket values.

In in for-loops, use in-pylist to iterate through the elements.

> (pylist 1 2 3)

(obj "list" : [1, 2, 3])

> (pylist? (pylist 1 2 3))

#t

> (pylist? (pylist))

#t

> (pylist? '(1 2 3))

#f

> (pylist 1 2 3 4)

(obj "list" : [1, 2, 3, 4])

> (pylist)

(obj "list" : [])

> (pylist (pylist 1 2 3 4) 5 (pylist 6 7))

(obj "list" : [[1, 2, 3, 4], 5, [6, 7]])

This function takes constant time.

> (pylist-ref (pylist "a" "b" "c" "d") 1)

"b"

> (pyfirst (pylist "a" "b" "c" "d"))

"a"

> (pysecond (pylist "a" "b" "c" "d"))

"b"

This function takes constant time.

> (define xs (pylist 0 1 2 3 4))

> (pylist-set! xs 1 #t)

> xs

(obj "list" : [0, True, 2, 3, 4])

> (list->pylist '(1 2 3 4))

(obj "list" : [1, 2, 3, 4])

> (list->pylist '(1 "foo" #(3 4)))

(obj "list" : [1, 'foo', (3, 4)])

> (list->pylist '(1 (2 3) #(4 (5 6))))

(obj "list" : [1, [2, 3], (4, [5, 6])])

> (vector->pylist '#(1 2 3 4))

(obj "list" : [1, 2, 3, 4])

> (vector->pylist '#(1 "foo" #(3 4)))

(obj "list" : [1, 'foo', (3, 4)])

> (vector->pylist '#(1 (2 3) #(4 (5 6))))

(obj "list" : [1, [2, 3], (4, [5, 6])])

> (pylist-length (pylist 1 2 3))

3

This function takes time proportional to the size of xs.

> (pylist->list (pylist 1 2 3 #t #f "a"))

'(1 2 3 #t #f (obj "str" : 'a'))

This function takes time proportional to the size of xs.

> (pylist->vector (pylist 1 2 3 #t #f "a"))

'#(1 2 3 #t #f (obj "str" : 'a'))

This function takes time proportional to the size of xs.

> (pylist->pytuple (pylist 1 2 3 #t #f "a"))

(obj "tuple" : (1, 2, 3, True, False, 'a'))

> (define xs (pylist 0 1 2 3))

> (for/list ([x (in-pylist xs)])     x)

'(0 1 2 3)

Worst case this function takes time proportional to the size of xs.

> (define xs (pylist 0 1 2 3))

> (pylist-insert! xs 2 #t)

> xs

(obj "list" : [0, 1, True, 2, 3])

> (define xs (pylist 10 11 12 13))

> (pylist-length xs)

4

> (pylist-append-item! xs 14)

> xs

(obj "list" : [10, 11, 12, 13, 14])

> (pylist-length xs)

5

> (define xs (pylist 1 2 3 4))

> (pylist-reverse! xs)

> xs

(obj "list" : [4, 3, 2, 1])

> (define xs (pylist 3 2 4 1))

> (pylist-sort! xs)

> xs

(obj "list" : [1, 2, 3, 4])

> (define ys (pylist 3 #t 2 4 #f #f 1))

> (pylist-sort! ys)

> ys

(obj "list" : [False, False, True, 1, 2, 3, 4])

In Python notation: list[low:high].

> (define xs (pylist 1 2 3 #t #f "a"))

> (pylist-get-slice xs 1 3)

(obj "list" : [2, 3])

8.2 Python Tuples - pytuple🔗ℹ

Python tuples correspond to immutable Racket vectors.

Even though there is no datastructure in Racket called "tuple", Python tuples will have the name "pytuple" to match the names of pylist and pydict.

The operations pytuple, list->pytuple and vector->pytuple can be used to construct pytuples from Racket values.

In in for-loops, use in-pytuple to iterate through the elements.

> (pytuple 1 2 3)

(obj "tuple" : (1, 2, 3))

> (pytuple? (pytuple 1 2 3))

#t

> (pytuple? (pytuple))

#t

> (pytuple? '(1 2 3))

#f

> (pytuple 1 2 3 4)

(obj "tuple" : (1, 2, 3, 4))

> (pytuple)

(obj "tuple" : ())

> (pytuple (pytuple 1 2 3 4) 5 (pytuple 6 7))

(obj "tuple" : ((1, 2, 3, 4), 5, (6, 7)))

This function takes constant time.

> (pytuple-ref (pytuple "a" "b" "c" "d") 1)

"b"

> (list->pytuple '(1 2 3 4))

(obj "tuple" : (1, 2, 3, 4))

> (list->pytuple '(1 "foo" #(3 4)))

(obj "tuple" : (1, 'foo', (3, 4)))

> (list->pytuple '(1 (2 3) #(4 (5 6))))

(obj "tuple" : (1, [2, 3], (4, [5, 6])))

> (vector->pytuple '#(1 2 3 4))

(obj "tuple" : (1, 2, 3, 4))

> (vector->pytuple '#(1 "foo" #(3 4)))

(obj "tuple" : (1, 'foo', (3, 4)))

> (vector->pytuple '#(1 (2 3) #(4 (5 6))))

(obj "tuple" : (1, [2, 3], (4, [5, 6])))

> (pytuple-length (pytuple 1 2 3))

3

This function takes time proportional to the size of xs.

> (pytuple->list (pytuple 1 2 3 #t #f "a"))

'(1 2 3 #t #f (obj "str" : 'a'))

This function takes time proportional to the size of xs.

> (pytuple->vector (pytuple 1 2 3 #t #f "a"))

'#(1 2 3 #t #f (obj "str" : 'a'))

This function takes time proportional to the size of xs.

> (define xs (pytuple->immutable-vector (pytuple 1 2 3 #t #f "a")))

> xs

'#(1 2 3 #t #f (obj "str" : 'a'))

> (immutable? xs)

#t

This function takes time proportional to the size of xs.

> (pytuple->pylist (pytuple 1 2 3 #t #f "a"))

(obj "list" : [1, 2, 3, True, False, 'a'])

> (define xs (pytuple 0 1 2 3))

> (for/list ([x (in-pytuple xs)])     x)

'(0 1 2 3)

In Python notation: list[low:high].

> (define xs (pytuple 1 2 3 #t #f "a"))

> (pytuple-get-slice xs 1 3)

(obj "tuple" : (2, 3))

8.3 Python Dictionaries - pydict🔗ℹ

Dictionaries in Python are associative arrays indexed by keys. Given a key one can lookup a value. Think of dictionaries as sets of key/value pairs.

Any immutable value can be used as a key, so strings, numbers and tuples can always be used as keys.

The corresponding data structure in Racket is the hash table.

Use the operations pydict->hash and hash->pydict to convert back and forth between pydicts and hash tables.

> (pydict? (hash->pydict (hash "a" 1  "b" 2)))

#t

> (hash->pydict (hash "a" 1  "b" 2))

(obj "dict" : {'b': 2, 'a': 1})

> (hash->pydict (hash 1 "x"  2 "y"))

(obj "dict" : {1: 'x', 2: 'y'})

> (hash->pydict (hash #(1 2) "tuple used as key"))

(obj "dict" : {(1, 2): 'tuple used as key'})

The function convert-key is used to convert the keys from Python values to Racket ones.
The function convert-value is used to convert the values.

The default value for the key conversion is pr/key.
The default value for the value conversion is pr.

> (pydict->hash (hash->pydict (hash "a" 1  "b" 2)))

'#hash(("a" . 1) ("b" . 2))

> (pydict->hash (hash->pydict (hash 1 "x"  2 "y")))

'#hash((1 . (obj "str" : 'x')) (2 . (obj "str" : 'y')))

> (pydict->hash (hash->pydict (hash #(1 2) "tuple used as key")))

'#hash((#(1 2) . (obj "str" : 'tuple used as key')))

The key to val mappings are added to the table in the order that they appear in the argument list, so later mappings can hide earlier mappings if the keys are equal.

The default value for the key and value conversion is rp.

> (pydict "a" 1  "b" 2)

(obj "dict" : {'a': 1, 'b': 2})

> (pydict 1 "x"  2 "y")

(obj "dict" : {1: 'x', 2: 'y'})

> (pydict #(1 2) "tuple used as key")

(obj "dict" : {(1, 2): 'tuple used as key'})

If failure-result is a procedure, it is called (through a tail call) with no arguments to produce the result.

Otherwise, failure-result is returned as the result.

> (define d (pydict "a" 1  "b" 2))

> d

(obj "dict" : {'a': 1, 'b': 2})

> (pydict-ref d "a")

1

> (pydict-ref d "b")

2

> (pydict-ref d "c")

pydict-ref: no value found for key

key: "c"

> (define d (pydict "a" 1))

> (pydict-set! d "a" 11)

> (pydict-set! d "b" 22)

> d

(obj "dict" : {'a': 11, 'b': 22})

> (define d (pydict "a" 1 "b" 2))

> (pydict-remove! d "b")

> (pydict-remove! d "c")

> d

(obj "dict" : {'a': 1})

> (define d (pydict "a" 1  "b" 2))

> d

(obj "dict" : {'a': 1, 'b': 2})

> (pydict-clear! d)

> d

(obj "dict" : {})

> (define d (pydict "a" 1  "b" 2))

> (pydict-contains? d "a")

#t

> (pydict-contains? d "x")

#f

> (define d1 (pydict "a" 1  "b" 2))

> (define d2 (pydict-copy d1))

> (list d1 d2)

'((obj "dict" : {'a': 1, 'b': 2}) (obj "dict" : {'a': 1, 'b': 2}))

> (pydict-set! d1 "a" 11)

> (list d1 d2)

'((obj "dict" : {'a': 11, 'b': 2}) (obj "dict" : {'a': 1, 'b': 2}))

> (define d (pydict "a" 1  "b" 2))

> (pydict-keys d)

(obj "list" : ['a', 'b'])

> (define d (pydict "a" 1  "b" 2 "c" 1))

> (pydict-values d)

(obj "list" : [1, 2, 1])

> (define d (pydict "a" 1  "b" 2 "c" 1))

> (pydict-count d)

3

If a key k is present in both pydicts and is mapped to values v1 and v2 in d1 and d2 respectively, then k is mapped to v2 if override? is true, and mapped to v1 otherwise.

> (define d1 (pydict "a" 1  "b" 2))

> (define d2 (pydict "b" 22 "c" 33))

> (pydict-merge! d1 d2)

> d1

(obj "dict" : {'a': 1, 'b': 22, 'c': 33})

> (define d1 (pydict "a" 1  "b" 2))

> (define d2 (pydict "b" 22 "c" 33))

> (pydict-merge! d1 d2 #f)

> d1

(obj "dict" : {'a': 1, 'b': 2, 'c': 33})

> (define d (pydict "a" 1 "b" 2))

> (for/list ([(key value) (in-pydict d)])     (list key value))

'(((obj "str" : 'a') 1) ((obj "str" : 'b') 2))

8.4 Python Strings - pystring🔗ℹ

Strings in Python are represented as objects with type str. Python strings are immutable sequences of Unicode code points.

There is no character type in Python, so various Python libraries use strings of length 1 to represent characters.

> (string->pystring "foo")

(obj "str" : 'foo')

> (pystring? (string->pystring "foo"))

#t

> (pystring #\f #\o #\o)

(obj "str" : 'foo')