AWS S3 Synchronization
1 S3 Synchronization API
2 S3 gzip Support
3 S3 Web Page Support
4 S3 Web Page Configuration
5 S3 Routing Rules

AWS S3 Synchronization

Synchronize an S3 bucket and a filesystem directory using

  raco s3-sync src dest

where either src or dest should start with s3:// to identify a bucket and item name or prefix, while the other is a path in the local filesystem to a file or directory. Naturally, a / within a bucket item’s name corresponds to a directory separator in the local filesystem, and a trailing / syntactically indicates a prefix (as opposed to a complete item name).A bucket item is ignored if its name ends in /. A bucket can contain an item whose name plus / is a prefix for other bucket items, in which case attempting to synchronize both from the bucket will produce an error, since the name cannot be used for both a file and a directory.

For example, to upload the content of src-dir with a prefix dest-path within bucket, use

  raco s3-sync src-dir s3://bucket/dest-path

To download the items with prefix src-path within bucket to dest-dir, use

  raco s3-sync s3://bucket/src-path dest-dir

If src refers to a directory or prefix (either syntactically or as determined by consulting the filesystem or bucket), dest cannot refer to a file or bucket item. If dest refers to directory or prefix while src refers to a file or item, the src file or item name is implicitly added to the dest directory or prefix.

The following options (supply them after s3-sync and before src) are supported:

1 S3 Synchronization API

 (require s3-sync) package: s3-sync

The s3-sync library uses aws/s3, so use ensure-have-keys, s3-host, and s3-region before calling s3-sync.


  [#:upload? upload? 
  #:jobs jobs 
  #:shallow? shallow? 
  #:check-metadata? check-metadata? 
  #:dry-run? dry-run? 
  #:delete? delete? 
  #:include include-rx 
  #:exclude exclude-rx 
  #:make-call-with-input-file make-call-with-file-stream 
  #:get-content-type get-content-type 
  #:get-content-encoding get-content-encoding 
  #:acl acl 
  #:reduced-redundancy? reduced-redundancy? 
  #:upload-metadata upload-metadata 
  #:upload-metadata-mapping upload-metadata-mapping 
  #:link-mode link-mode 
  #:log log-info 
  #:error raise-error]) 
  local-path : path-string?
  s3-bucket : string?
  s3-path : (or/c #f string?)
  upload? : any/c = #t
  jobs : inexact-positive-integer? = 1
  shallow? : any/c = #f
  check-metadata? : any/c = #f
  dry-run? : any/c = #f
  delete? : any/c = #f
  include-rx : (or/c #f regexp?) = #f
  exclude-rx : (or/c #f regexp?) = #f
  make-call-with-file-stream : 
(or/c #f (string?
          . -> .
          (or/c #f
                 (input-port? . -> .  any)
                 . -> . any))))
   = #f
  get-content-type : (string? path? . -> . (or/c string? #f))
   = #f
  get-content-encoding : (string? path? . -> . (or/c string? #f))
   = #f
  acl : (or/c #f string?) = #f
  reduced-redundancy? : any/c = #f
  upload-metadata : 
(and/c (hash/c symbol? string?)
 = #hash()
  upload-metadata-mapping : 
 (string? . -> . (and/c (hash/c symbol? string?)
 (and/c (hash/c string?
                (and/c (hash/c symbol? string?)
   = #hash()
  link-mode : (or/c 'error 'follow 'redirect 'redirects 'ignore)
   = 'error
  log-info : (string . -> . void?) = log-s3-sync-info
  raise-error : (symbol? string? any/c ... . -> . any) = error
Synchronizes the content of local-path and s3-path within s3-bucket, where s3-path can be #f to indicate an upload to the bucket with no prefix path. If upload? is true, s3-bucket at s3-path is changed to have the content of local-path, otherwise local-path is changed to have the content of s3-bucket at s3-path.

Typically, local-path refers to a directory and s3-path refers to a prefix for bucket item names. If local-path refers to a file and upload? is true, then a single file is synchronized to s3-bucket at s3-path. In that case, if s3-path ends with a / or it is already used as a prefix for bucket items, then the file name of local-path is added to s3-path to form the uploaded item’s name; otherwise, s3-path names the uploaded item. If upload? is #f and s3-path is an item name (and not a prefix on other item names), then a single bucket item is downloaded to local-path; if local-path refers to a directory, then the portion of s3-path after the last / is used as the downloaded file name.

If shallow? is true, then in download mode, bucket items are downloaded only when they correspond to directories that exist already in local-path (which is useful when local-path refers to a directory). In both download and upload modes, a true value of shallow? causes the state of s3-bucket to be queried in a directory-like way, exploring only relevant directories; that exploration can be faster than querying the full content of s3-bucket if it contains many more nested items (with the prefix s3-path) than files within local-path.

If check-metadata? is true, then in upload mode, bucket items are checked to ensure that the current metadata matches the metadata that would be uploaded, and the bucket item’s metadata is adjust if not.

If dry-run? is true, then actions needed for synchronization are reported via log, but no uploads, downloads, deletions, or redirection-rule updates are performed.

If jobs is more than 1, then downloads and uploads proceed in background threads.

If delete? is true, then destination items that have no corresponding item at the source are deleted.

If include-rx is not #f, then it is matched against bucket paths (including s3-path in the path). Only items that match the regexp are considered for synchronization. If exclude-rx is not #f, then any item whose path matches is not considered for synchronization (even if it also matches a provided include-rx).

If make-call-with-file-stream is not #f, it is called to get a function that acts like call-with-input-file to get the content of a file for upload or for hashing. The arguments to make-call-with-file-stream are the S3 name and the local file path. If make-call-with-file-stream or its result is #f, then call-with-input-file is used. See also make-gzip-handlers.

If get-content-type is not #f, it is called to get the Content-Type field for each file on upload. The arguments to get-content-type are the S3 name and the local file path. If get-content-type or its result is #f, then a default value is used based on the file extension (e.g., "text/css" for a "css" file).

The get-content-encoding argument is like get-content-type, but for the Content-Encoding field. If no encoding is provided for an item, a Content-Encoding field is omitted on upload. Note that the Content-Encoding field of an item can affect the way that it is downloaded from a bucket; for example, a bucket item whose encoding is "gzip" will be uncompressed on download, even though the item’s hash (which is used to avoid unnecessary downloads) is based on the encoded content.

If acl is not #f, then it use as the S3 access control list on upload. For example, supply "public-read" to make items public for reading. More specifically, if acl is not #f, then 'x-amz-acl is set to acl in upload-metadata (if it is not set already).

If reduced-redundancy? is true, then items are uploaded to S3 with reduced-redundancy storage (which costs less, so it is suitable for files that are backed up elsewhere). More specifically, if reduced-redundancy is true, then 'x-amz-storage-class is set to "REDUCED_REDUNDANCY" in upload-metadata (if it is not set already).

The upload-metadata hash table provides metadata to include with any file upload (and only to files that are otherwise determined to need uploading). The upload-metadata-mapping provides a mapping from bucket item names to metadata that adds and overrides metadata for the specific item.

The link-mode argument determines the treatment of soft links in local-path:

The log-info and raise-error arguments determine how progress is logged and errors are reported. The default log-info function logs the given string at the 'info level to a logger whose name is 's3-sync.

Changed in version 1.2 of package s3-sync: Added 'redirects mode.
Changed in version 1.3: Added the upload-metadata argument.
Changed in version 1.4: Added support for a single file as local-path and a bucket item name as s3-path.
Changed in version 1.5: Added the check-metadata? argument.
Changed in version 1.6: Added the upload-metadata-mapping argument.
Changed in version 1.7: Changed upload-metadata-mapping to allow a procedure.

2 S3 gzip Support

 (require s3-sync/gzip) package: s3-sync


(make-gzip-handlers pattern 
  [#:min-size min-size]) 
(string? path? . -> . (or/c #f
                            (path? (input-port? . -> .  any)
                             . -> . any)))
(string? path? . -> . (or/c string? #f))
  pattern : (or/c regexp? string? bytes?)
  min-size : exact-nonnegative-integer? = 0
Returns values that are suitable as the #:make-call-with-input-file and #:get-content-encoding arguments to s3-sync to compress items whose name within the bucket matches pattern and whose local file size is at least min-size bytes.

3 S3 Web Page Support

 (require s3-sync/web) package: s3-sync

Added in version 1.3 of package s3-sync.


(s3-web-sync ...)  void?

Accepts the same arguments as s3-sync, but adapts the defaults to be suitable for web-page uploads:

4 S3 Web Page Configuration

 (require s3-sync/web-config) package: s3-sync

Added in version 1.3 of package s3-sync.


web-acl : string?

The default access control list for web content, currently "public-read".

The default storage mode for web content, currently #t.

Default metadata for web content, currently (hash 'Cache-Control "max-age=0, no-cache").


(web-upload-metadata-mapping item)

(and/c (hash/c symbol? string?)
  item : string?
Item-specific metadata for web content, currently produces (hash 'Cache-Control "max-age=31536000, public") for a item that ends in ".css", ".js", ".png", ".jpg", ".jpeg", ".gif", ".svg", ".ico", or ".woff".

Added in version 1.7 of package s3-sync.
Changed in version 1.8: Added ".woff".

Default regexp for paths to be gzipped, currently #rx"[.](html|css|js|svg)$".

Default minimum size for files to be gzipped, currently #rx"[.](html|css|js)$".

5 S3 Routing Rules

 (require s3-sync/routing-rule) package: s3-sync

Added in version 1.9 of package s3-sync.


(add-routing-rules bucket 
  [#:preserve-existing? preserve-existing?] 
  #:log-info log-info-proc 
  #:error error-proc) 
  bucket : string?
  rules : (listof routing-rule?)
  preserve-existing? : any/c = #t
  log-info-proc : (lambda (s) (log-info s))
  error-proc : error
Configures the web-site routing rules at bucket to include each of the routing rules in rules. Unless preserve-existing? is false, existing routing rules are preserved except as overridden by rules.


(redirect-prefix-routing-rule #:old-prefix prefix 
  [#:new-prefix new-prefix 
  #:new-host new-host 
  #:redirect-code redirect-code]) 
  prefix : string?
  new-prefix : (or/c string? #f) = #f
  new-host : (or/c string? #f) = #f
  redirect-code : (or/c string? #f) = #f
Creates a routing rule that redirects an access with a prefix matching prefix so that the prefix is replaced by new-prefix, the access is redirected to new-host, or both. At least one of new-prefix or new-host must be non-#f.

Changed in version 1.10 of package s3-sync: Added #:redirect-code argument.


(routing-rule? v)  boolean?

  v : any/c
Returns #t if v is a routing rule as created by redirect-prefix-routing-rule, #f otherwise.