Artifact: On the Cost of Type-Tag Soundness
This package supplements our paper On the Cost of Type-Tag Soundness. It contains code to process our datasets and generate the submitted paper.
Local copy: http://www.ccs.neu.edu/home/types/resources/pdf/gm-pepm-2018.pdf
Source code: https://github.com/nuprl/retic_performance
Note: the code in this package assumes that certain relative paths exist. For example, it assumes the Karst datasets are in the folder "../data/karst" relative to the source for the gm-pepm-2018 package.
1 Using the Datasets
The gm-pepm-2018 package includes command-line tools for interacting with a dataset.
These tools are defined in modules under the "script/" folder and are declared in the "info.rkt" file for gm-pepm-2018.
rp-info : print static information about a given benchmark
rp-plot : render a plot, similar to plots in the paper
rp-perf : print summary information about a benchmark’s performance
rp-python : analyze Python and Reticulated programs
rp-sample : choose random configurations to sample
Example: to generate an overhead plot for Espionage, run:
raco rp-plot Espionage
For more information about a command CMD, run:
raco CMD --help
2 Reference
#lang gm-pepm-2018 | package: gm-pepm-2018 |
Other modules in the collection process the Reticulated benchmarks and the datasets. The modules are documented in the sections below.
syntax
syntax
syntax
syntax
syntax
syntax
syntax
syntax
syntax
syntax
syntax
syntax
syntax
syntax
syntax
syntax
syntax
syntax
syntax
syntax
syntax
syntax
procedure
(generate-bibliography) → part?
procedure
title : string? author : pre-content? url : element? lib : (listof pre-content?) description : pre-content?
procedure
(exact-runtime-category cat-name names make-description) → element? cat-name : string? names : (listof string?) make-description : (-> string? pre-content?)
procedure
(percent-slower-than-typed name) → integer?
name : string?
procedure
(definition term description ...) → paragraph?
term : string? description : string?
procedure
(lib-desc name description ...) → pre-content?
name : string? description : string?
procedure
(deliverable D) → element?
D : (or/c pre-content? real?)
procedure
(approximation r s pct) → element?
r : (or/c pre-content? real?) s : (or/c pre-content? real?) pct : (or/c pre-content? real?)
procedure
(id x) → pre-content?
x : any/c
procedure
(Integer->word n) → string?
n : integer?
procedure
(pythonexternal filename) → element?
filename : string?
procedure
(pythoninline code ...) → element?
code : string?
procedure
(Section-ref tag) → element?
tag : string?
procedure
(section-ref tag) → element?
tag : string?
2.1 Static Benchmark Information
a set of fully-typed Python modules;
other Python modules and data files.
The "other modules" may be typed or untyped; the point is that all configurations use identical copies of these modules.
procedure
(benchmark-info? v) → boolean?
v : any/c
procedure
(benchmark->name bm) → string?
bm : benchmark-info?
procedure
bm : benchmark-info?
procedure
(benchmark->src bm) → path-string?
bm : benchmark-info?
procedure
(benchmark->num-modules bm) → natural?
bm : benchmark-info?
procedure
(benchmark->num-configurations bm) → natural?
bm : benchmark-info?
procedure
bm : benchmark-info?
procedure
(benchmark->sloc bm) → natural?
bm : benchmark-info?
procedure
(benchmark->karst-data bm) → (or/c #f path-string?)
bm : benchmark-info?
procedure
(benchmark->sample-data bm) → (or/c #f path-string?)
bm : benchmark-info?
procedure
bm : benchmark-info?
procedure
bm : benchmark-info?
procedure
bm : benchmark-info?
The idea with the "arbitrary" list is to return data that suffices to build an "obviously wrong" plot. Now that the paper is finished, we could error if the data does not exist.
procedure
(all-benchmarks) → (listof benchmark-info?)
procedure
(configuration->natural bm cfg) → natural?
bm : benchmark-info? cfg : configuration?
procedure
bm : benchmark-info? n : natural?
procedure
(configuration<? cfg0 cfg1) → boolean?
cfg0 : configuration? cfg1 : configuration?
procedure
bm : benchmark-info?
2.2 Python Data
The Python script "gm-pepm-2018/script/explode_module.py" extracts information about the structure of a Python module (e.g., number of classes).
(require gm-pepm-2018/script/python) | package: gm-pepm-2018 |
struct
(struct python-info (name module?))
name : symbol? module? : (listof module-info?)
struct
(struct module-info (name function?))
name : symbol? function? : (listof function-info?)
struct
(struct class-info (name field* method*))
name : symbol? field* : (listof field-info?) method* : (listof function-info?)
struct
(struct function-info (name dom* cod))
name : symbol? dom* : (listof field-info?) cod : (or/c string? #f)
struct
(struct field-info (name type))
name : symbol? type : string?
procedure
(python-sloc ps) → natural?
ps : python-path?
procedure
py : python-info?
procedure
(python-info->num-modules py) → natural?
py : python-info?
procedure
py : python-info?
procedure
(python-info->num-functions py) → natural?
py : python-info?
procedure
py : python-info?
procedure
(python-info->num-classes py) → natural?
py : python-info?
procedure
py : python-info?
procedure
(python-info->num-methods py) → natural?
py : python-info?
procedure
py : python-info?
procedure
(python-info->num-parameters py) → natural?
py : python-info?
procedure
py : python-info?
procedure
(python-info->num-returns py) → natural?
py : python-info?
procedure
(python-info->field* py) → (listof field-info?)
py : python-info?
procedure
(python-info->num-fields py) → natural?
py : python-info?
procedure
py : python-info?
procedure
(python-info->num-types py) → natural?
py : python-info?
2.3 Reticulated Performance Data
procedure
(performance-info? v) → boolean?
v : any/c
procedure
(performance-info->name pi) → symbol?
pi : performance-info?
procedure
(benchmark->performance-info bm) → performance-info?
bm : benchmark-info?
procedure
(python-runtime pi) → real?
pi : performance-info?
procedure
(untyped-runtime pi) → real?
pi : performance-info?
procedure
(typed-runtime pi) → real?
pi : performance-info?
procedure
(num-configurations pi) → natural?
pi : performance-info?
procedure
(num-types pi) → natural?
pi : performance-info?
procedure
(overhead pi r) → real?
pi : performance-info? r : real?
procedure
(min-overhead pi) → real?
pi : performance-info?
procedure
(max-overhead pi) → real?
pi : performance-info?
procedure
(mean-overhead pi) → real?
pi : performance-info?
procedure
(deliverable D) → (-> performance-info? natural?)
D : real?
procedure
(typed/python-ratio pi) → real?
pi : performance-info?
procedure
(typed/retic-ratio pi) → real?
pi : performance-info?
procedure
(untyped/python-ratio pi) → real?
pi : performance-info?
procedure
D : real? pi : performance-info?
procedure
(count-configurations pi good?) → natural?
pi : performance-info? good? : (-> real? boolean?)
procedure
pi : performance-info? good? : (-> real? boolean?)
procedure
(performance-info-has-karst-data? pi) → boolean?
pi : performance-info?
procedure
→ (cons/c natural? (listof path-string?)) pi : performance-info?
procedure
(performance-info%sample pi ps) → performance-info?
pi : performance-info? ps : path-string?
procedure
(unzip-karst-data ps) → (or/c #f path-string?)
ps : path-string?
procedure
(performance-info-src pi) → path-string?
pi : performance-info?
procedure
input : (or/c path-string? performance-info?) f : (-> A configuration? natural? (listof real?) A) init : A
2.4 Generating Plots
(require gm-pepm-2018/script/plot) | package: gm-pepm-2018 |
procedure
(overhead-plot pi) → pict?
pi : performance-info?
procedure
(exact-runtime-plot pi) → pict?
pi : performance-info?
procedure
(validate-samples-plot pi) → pict?
pi : performance-info?
procedure
(samples-plot pi) → pict?
pi : performance-info?
parameter
(*LEGEND-VSPACE*) → natural?
(*LEGEND-VSPACE* n) → void? n : natural?
= 10
parameter
(*LEGEND-HSPACE*) → natural?
(*LEGEND-HSPACE* n) → void? n : natural?
= 20
parameter
(*OVERHEAD-PLOT-WIDTH*) → natural?
(*OVERHEAD-PLOT-WIDTH* n) → void? n : natural?
= 600
parameter
(*OVERHEAD-PLOT-HEIGHT*) → natural?
(*OVERHEAD-PLOT-HEIGHT* n) → void? n : natural?
= 300
parameter
(*OVERHEAD-FONT-FACE*) → string?
(*OVERHEAD-FONT-FACE* f) → void? f : string?
= "bold"
parameter
(*OVERHEAD-FONT-SCALE*) → (>=/c 0)
(*OVERHEAD-FONT-SCALE* n) → void? n : (>=/c 0)
= 0.03
parameter
(*OVERHEAD-LABEL?*) → boolean?
(*OVERHEAD-LABEL?* lbl?) → void? lbl? : boolean?
= #f
parameter
(*OVERHEAD-LINE-COLOR* c) → void? c : plot-color/c
= 3
parameter
(*OVERHEAD-LINE-STYLE* s) → void? s : plot-pen-style/c
= 'solid
parameter
(*OVERHEAD-LINE-WIDTH*) → (>=/c 0)
(*OVERHEAD-LINE-WIDTH* lw) → void? lw : (>=/c 0)
= 1
parameter
(*OVERHEAD-MAX*) → natural?
(*OVERHEAD-MAX* om) → void? om : natural?
= 10
parameter
(*OVERHEAD-SHOW-RATIO* sr) → void? sr : (or/c boolean? symbol?)
= #t
parameter
(*OVERHEAD-SAMPLES*) → natural?
(*OVERHEAD-SAMPLES* os) → void? os : natural?
= 20
parameter
(*FONT-SIZE*) → natural?
(*FONT-SIZE* fs) → void? fs : natural?
= 10
parameter
(*CACHE-SIZE*) → natural?
(*CACHE-SIZE* cs) → void? cs : natural?
= (expt 2 16)
parameter
(*POINT-SIZE*) → natural?
(*POINT-SIZE* ps) → void? ps : natural?
= 3
parameter
(*POINT-ALPHA*) → (>=/c 0)
(*POINT-ALPHA* a) → void? a : (>=/c 0)
= 0.4
parameter
(*CONFIGURATION-X-JITTER*) → real?
(*CONFIGURATION-X-JITTER* j) → void? j : real?
= 0.4
parameter
(*OVERHEAD-FREEZE-BODY*) → boolean?
(*OVERHEAD-FREEZE-BODY* f?) → void? f? : boolean?
= #f
parameter
(*CONFIDENCE-LEVEL*) → (between/c 0 100)
(*CONFIDENCE-LEVEL* c) → void? c : (between/c 0 100)
= 95
parameter
(*INTERVAL-ALPHA*) → (>=/c 0)
(*INTERVAL-ALPHA* r) → void? r : (>=/c 0)
= 1
parameter
(*RATIO-DOT-SYM* s) → void? s : point-sym/c
= 'plus
parameter
(*RATIO-DOT-SIZE*) → natural?
(*RATIO-DOT-SIZE* n) → void? n : natural?
= 8
parameter
(*RATIO-DOT-COLOR*) → string?
(*RATIO-DOT-COLOR* c) → void? c : string?
= "firebrick"
parameter
(*TYPED/PYTHON-RATIO-XTICK?*) → boolean?
(*TYPED/PYTHON-RATIO-XTICK?* t?) → void? t? : boolean?
= #f
2.5 Rendering Figures
(require gm-pepm-2018/script/render) | package: gm-pepm-2018 |
procedure
(render-overhead-plot* bm*) → pict?
bm* : (listof benchmark-info?)
procedure
(render-exact-runtime-plot* bm*) → pict?
bm* : (listof benchmark-info?)
procedure
(render-static-information bm*) → table?
bm* : (listof benchmark-info?)
procedure
(render-ratios-table bm*) → table?
bm* : (listof benchmark-info?)
procedure
(render-samples-plot* bm*) → pict?
bm* : (listof benchmark-info?)
procedure
(render-validate-samples-plot* bm*) → pict?
bm* : (listof benchmark-info?)
parameter
(*PLOT-HEIGHT* h) → void? h : exact-integer?
parameter
(*SINGLE-COLUMN?* single?) → void? single? : boolean?
procedure
(get-ratios-table bm*) → list?
bm* : (listof benchmark-info?)
procedure
(ratios-table-row rt) → list?
rt : list?
procedure
(ratios-row-retic/python rr) → string?
rr : list?
procedure
(ratios-row-typed/retic rr) → string?
rr : list?
procedure
(ratios-row-typed/python rr) → string?
rr : list?
2.6 System Calls
(require gm-pepm-2018/script/system) | package: gm-pepm-2018 |
procedure
(shell program args) → string?
program : path-string? args : (or/c path-string? (listof path-string?))
procedure
(md5sum ps) → string?
ps : path-string?
2.7 Utility Functions
(require gm-pepm-2018/script/util) | package: gm-pepm-2018 |
procedure
(path-string->string ps) → string?
ps : path-string?
This function is surprisingly useful in untyped code, and extremely useful in typed code.
procedure
(ensure-directory d) → void?
d : path-string?
procedure
(file-remove-extension ps) → path-string?
ps : path-string?
procedure
(save-pict ps p) → boolean?
ps : path-string? p : pict?
procedure
(natural->bitstring n #:pad k) → string?
n : natural? k : natural?
procedure
(bitstring->natural s) → natural?
s : string?
procedure
(integer->word n [#:title? title?]) → string?
n : integer? title? : any/c = #f