4 Search (SXPath)

The W3C "XPath" standard describes a standardized way to perform searches in XML documents. For instance, the XPath string "/A/B/C" (thank you, Wikipedia) describes a search for a C element whose parent is a B element whose parent is an A element that is the root of the document.

The sxpath function performs a similar search over SXML data, using either the standard XPath strings or a list of Racket values.

> (require sxml/sxpath)

> ((sxpath "/A/B/C")    '(*TOP* (A (B (C)))))

'((C))

> ((sxpath '(A B C))    '(*TOP* (A (B (C)))))

'((C))

> ((sxpath "//p[contains(@class, 'blue')]/text()")    '(*TOP*      (body       (p (@ (class "blue")) "P1")       (p (@ (class "blue green")) "P2")       (p (@ (class "red")) "P3"))))

'("P1" "P2")

(This documentation desperately needs more examples.)

If we use Neil Van Dyke’s html parser, we might parse this into the following sxml document:

> (define example-doc     '(*TOP*       (aaa "\n" "  "         (bbb "\n" "     "           (ccc) "\n" "     "           (www " www content " (xxx) (www "\n" "     " (zzz) "\n" "  ")))         "\n" "  "         (xxx "\n" "     " (ddd " content in ccc \n" "     ") "\n" "  ")         "\n")))

To extract the xxx’s inside the aaa from the example document:

> ((sxpath '(aaa xxx)) example-doc)

'((xxx "\n" "     " (ddd " content in ccc \n" "     ") "\n" "  "))

To extract all cells from an HTML table:

> (define table     `(*TOP*       (table        (tr (td "a") (td "b"))        (tr (td "c") (td "d")))))

> ((sxpath '(table tr td)) table)

'((td "a") (td "b") (td "c") (td "d"))

To extract all cells anywhere in a document:

> (define table     `(*TOP*       (div        (p (table            (tr (td "a") (td "b"))            (tr (td "c") (td "d"))))        (table         (tr (td "e"))))))

> ((sxpath '(// td)) table)

'((td "a") (td "b") (td "c") (td "d") (td "e"))

One result may be nested in another one:

> (define doc     `(*TOP*       (div        (p (div "3")           (div (div "4"))))))

> ((sxpath '(// div)) doc)

'((div (p (div "3") (div (div "4")))) (div "3") (div (div "4")) (div "4"))

There’s also a string-based syntax, txpath. As shown in the grammar above, sxpath assumes that any strings in the path are expressed using the txpath syntax.

So, for instance, the prior example could be rewritten using a string:

> (define doc     `(*TOP*       (div        (p (div "3")           (div (div "4"))))))

> ((sxpath "//div") doc)

'((div (p (div "3") (div (div "4")))) (div "3") (div (div "4")) (div "4"))

More generally, lists in the s-expression syntax correspond to string concatenation in the txpath syntax.

So, to find all italics that appear at top level within a paragraph:

> (define doc     `(*TOP*       (div        (p (i "3")           (froogy (i "4"))))))

> ((sxpath "//p/i") doc)

'((i "3"))

Handling of namespaces in sxpath is a bit surprising. In particular, it appears to me that sxpath’s model is that namespaces must appear fully expanded in the matched source. For instance:

> ((sxpath "//ns:p" `((ns . "http://example.com")))    '(*TOP* (html (http://example.com:body                   (http://example.com:p "first para")                   (http://example.com:p                    "second para containing"                    (http://example.com:p "third para") "inside it")))))

'((http://example.com:p "first para")   (http://example.com:p    "second para containing"    (http://example.com:p "third para")    "inside it")   (http://example.com:p "third para"))

But the corresponding example where the source document contains a namespace shortcut does not match in the same way. That is:

> ((sxpath "//ns:p" `((ns . "http://example.com")))    '(*TOP* (@ (*NAMESPACES* (ns "http://example.com")))            (html (ns:body (ns:p "first para")                           (ns:p "second para containing"                                 (ns:p "third para") "inside it")))))

'()

It produces the empty list. Instead, you must pretend that the shortcut is actually the namespace. Thus:

> ((sxpath "//ns:p" `((ns . "ns")))    '(*TOP* (@ (*NAMESPACES* (ns "http://example.com")))            (html (ns:body (ns:p "first para")                           (ns:p "second para containing"                                 (ns:p "third para") "inside it")))))

'((ns:p "first para")   (ns:p "second para containing" (ns:p "third para") "inside it")   (ns:p "third para"))

Ah well.

4.1 Trailing Matches

There’s probably a better name for this feature than "trailing matches", please let me know what it is (or better yet, make a pull request).

It may sometimes be the case that you’re looking for abc nodes that contain def nodes. You could try

(sxpath '(// abc def))

... but that would give you the inner def nodes, not the abc parent nodes that contain the def nodes. If you pore over the expansion above, you will discover that you can do this, using ... well, essentially, using a pair of nested lists:

((sxpath '(// (abc (def))))

'(x (x (abc (x)

(def "1")))

(abc (x) "2")

(abc (def) "3")))

4.2 Custom Predicates

If you take a look at the expansion above, you’ll see that you can directly embed procedures (essentially a "3D syntax" approach) into sxpath terms, to provide more customizable matching. However, you’ll be hard-pressed to find out exactly what the interface to those procedures looks like. I’ve done some experimentation and some light code-reading, and it appears to me that such procedures are called with two arguments; a list of nodes, and another list ... of ... variable bindings? I realize that it’s uncommon for documentation to be so wishy-washy, but I’m erring on the side of getting the information out there.

Anyhow, suppose (just for instance) that you wanted to be able to use sxpath to find divs with a particular classname (e.g. zzz), but that—like CSS—you wanted to treat the value of the "class" property as being a space-separated list of class names, so what you’re really looking for is a node whose class property contains "zzz" as part of a space-separated list, for instance "yyy zzz aaa".

Can you do this? Yes you can! Here’s some example code:

;; an sxpath matcher for "class" nodes with the given classname

;; as part of its space-separated list

(define ((class-str-includes? classname) nodes vars-thingy)

(ormap (node-with-class-str classname) nodes))

;; is this a node named "class" with the given classname as

;; part of its space-separated list?

(define ((node-with-class-str classname) node)

(match node

[(list 'class (? string? classnames-str))

(member classname

(map string-trim (regexp-split #px" " classnames-str)))]))

((sxpath `(// abc (z (@ class ,(class-str-includes? "bar")))))

'(*TOP* (x (x (x (abc (y (x))))

(abc (z (@ (class "bar tar")) (x) "blorg"))

(abc (z (@ (class "car bar")) (x) "blorg3"))

(abc (z (@ (class "baz car")) (x) "blorg2"))))))

There are several clunky things about this solution; it’s clunky that I repeat "class" in two different places, but abstracting over this would probably naturally soak up the @ as well, and I wasn’t sure I wanted to do that. Also, I was surprised to see that the class node was actually passed to the matcher, rather than its content. But sure, okay. Also, the name vars-thingy should strongly suggest that I’m not entirely sure what’s going on here.

As before: pull requests welcome!

Deprecated; use sxpath instead.

procedure (sxml:filter pred) → sxml-converter

pred : sxml-converter-as-predicate

procedure (sxml:complement pred) → sxml-converter-as-predicate

pred : sxml-converter-as-predicate

procedure (node-join selector) → sxml-converter

selector : sxml-converter

procedure (node-reduce converter) → sxml-converter

converter : sxml-converter

procedure (node-or converter) → sxml-converter

converter : sxml-converter

procedure (node-closure converter) → sxml-converter

converter : sxml-converter

The following procedures depend explicitly on the root node.

Here’s an example:

> (((sxml:preceding (ntype?? 'www)) example-doc) ((sxpath `(aaa xxx)) example-doc))

'((www "\n" "     " (zzz) "\n" "  ")   (www " www content " (xxx) (www "\n" "     " (zzz) "\n" "  ")))

> (define doc '(*TOP* (div (p "foo") (p "bar")                            (img "baz") (p "quux"))))

> (((sxml:preceding-sibling (ntype?? 'p)) doc) ((sxpath '(// img)) doc))

'((p "bar") (p "foo"))