XQuery and XPath subscripting

Today I took a long time, together with folks from the eXist-DB Community (thanks Luca Guariento and line0 for the brainstorming), to figure out why a function was receiving a node sequence instead of a single node.

The question was why using this code was not working:

for $entry in functx:distinct-nodes($entries)
    let $term := $entry//tei:orth[1]
    let $lcterm := fn:lower-case($term/text())

And fn:lower-case was complaining for receiving a sequence of nodes, instead of a single one. Our main dig was looking into the structure of the orth tag, and their contents. But we were looking into the wrong side of the road. It was interesting to learn some more details on XQuery, but it did not help on understanding the problem.

Some time later, I tried this one, and it looked like it was working:

for $entry in functx:distinct-nodes($entries)
    let $term := ($entry//tei:orth)[1]
    let $lcterm := fn:lower-case($term/text())

But it was not clear why. How the parenthesis are relevant? I tried a couple of small documents, and the usage of the parenthesis or not was not making any difference.

It took a while until I get to this small example:

xquery version "3.1";

let $x :=
<r>
    <foo>
       <tag>a</tag>
       <bar>
         <tag>b</tag>
       </bar>
    </foo>
    <foo>
        <tag>c</tag>
        <tag>d</tag>
    </foo>
</r>

for $foo in $x//foo
let $ans1 := $foo//tag[1]
let $ans2 := ($foo//tag)[1]
return <ans><a> {$ans1} </a><b> {$ans2} </b></ans> 

The result from this query looks like this:

<ans>
    <a>
        <tag>a</tag>
        <tag>b</tag>
    </a>
    <b>
        <tag>a</tag>
    </b>
</ans>
<ans>
    <a>
        <tag>c</tag>
    </a>
    <b>
        <tag>c</tag>
    </b>
</ans>

Can you notice the difference?

So, basically

$foo//tag[1]  — first tag from any child

($foo//tag)[1] — first tag from the result set

Hope this helps someone.

XQuery: Creating Document with Processing Instruction

Today I stumped on how to create a XML document, storing it in a XQuery variable, that is composed from both a processing instruction and an XML body.

While this might seem easy at first, creating the document as once:

let $xml := <?xml-stylesheet href="common.css"?><foo>bar</foo>

The truth is that this does not work. Nevertheless, there is a set of ways to construct documents on XQuery. One of them, that looks really powerful, is the computed constructors. While this recommendation does not have many examples, this other page from Altova might be useful to visualize how to use them.

After banging with the head for some time, with this solution:

let $pi = <?xml-stylesheet href="common.css"?>
let $xml = <foo>bar</foo>
return document {($pi,$xml)}

This were still not working. Or, in other words, they were working, but eXide, the Web IDE for eXist-DB, was misbehaving. Instead of presenting the full document, it was only showing the processing instruction. But when trying this block of code on the function I was preparing, it worked like a charm.

Thanks to Joe Wicentowski for pointing me in the computed constructors documentation.

Experimenting eXist-DB on Docker

I’ve been using eXist-DB for some time for the project of the Dicion├írio da Academia de Ci├¬ncias de Lisboa, that is being revived from the PDF into TEI, so that a new digital version can be soon released.

Recently I needed to update the server where eXist-DB was running, and decided to use a dockerized version of it. Although that can make things a little slower (not really sure), it makes things easier to replicate, and now I can have, easily, the dictionary database running on my laptop or in the server, using the same code.

I am using the default latest version of eXist-DB docker image. The only difference is that, because my XQuery code uses FunctX functions, I needed to import that module. Thus, my Dockerfile is composed by:

FROM existdb/existdb:latest

ADD http://exist-db.org/exist/apps/public-repo/public/functx-1.0.1.xar /exist/autodeploy

I have the data and application on a GIT repository, as exported by the eXist-DB backup tool. Thus, I decided to create a simple script to import the data, instead o creating the docker image already with that data. Therefore, my docker-compose.yml file is composed by:

version: '3.3'
services:
  exist:
    build: ./dacl/docker
    container_name: exist
    ports:
        - 8080:8080
        - 8443:8443
    volumes:
        - ./data:/exist/data
        - ./config:/exist/config
        - ./dacl:/import
        - ./outdir:/export

The relevant parts:

  • The path to the folder including the Dockerfile (dacl/docker)
  • Ports 8080 and 8443 are used by eXist-DB, and I am just forwarding them to the host
  • Created four volumes: data stores the binary database data, config stores the configuration files, and import and export volumes are used to import data, and export data for backup.

For importing all data into the database I am using a shell script. The first five lines import some collections. The last two execute two auxiliary scripts, the first to re-index application data, and the second to create the proper groups, users and assign passwords.

First, the import.sh script is:

#!/usr/bin/env bash

docker-compose exec -T exist java org.exist.start.Main backup -u admin -r /import/db/academia/__contents__.xml
docker-compose exec -T exist java org.exist.start.Main backup -u admin -r /import/db/apps/academia/__contents__.xml

docker-compose exec -T exist java org.exist.start.Main backup -u admin -r /import/db/academia-2001/__contents__.xml
docker-compose exec -T exist java org.exist.start.Main backup -u admin -r /import/db/apps/academia-2001/__contents__.xml

docker-compose exec -T exist java org.exist.start.Main backup -u admin -r /import/db/schemas/__contents__.xml

docker-compose exec -T exist java org.exist.start.Main client -u admin -F /import/xq/repair.xq
docker-compose exec -T exist java org.exist.start.Main client -u admin -F /import/xq/users.xq

Note that the XQuery scripts are being held in the same folder that is mounted in the import volume. Otherwise, you will not be able to access it from inside the container.

The repair XQuery script holds this code:

import module namespace repair="http://exist-db.org/xquery/repo/repair"
at "resource:org/exist/xquery/modules/expathrepo/repair.xql";
repair:clean-all(),
repair:repair()

And finally, the users Xquery script has the following code:

sm:passwd('admin','admin-password),
sm:create-group('dacl'),
sm:create-account('ana','ana-password','dacl'),
sm:chgrp(xs:anyURI('/db/academia'), 'dacl'),
sm:chgrp(xs:anyURI('/db/academia-2001'), 'dacl'),
sm:chgrp(xs:anyURI('/db/apps/academia-2001'), 'dacl'),
sm:chgrp(xs:anyURI('/db/apps/academia'), 'dacl'),
sm:chmod(xs:anyURI('/db/academia-2001'), 'rwxrwx---'),
sm:chmod(xs:anyURI('/db/academia'), 'rwxrwx---'),
sm:chmod(xs:anyURI('/db/apps/academia-2001'), 'rwxrwx---'),
sm:chmod(xs:anyURI('/db/apps/academia'), 'rwxrwx---')

Also, in case it gets useful, this is my backup.sh script

docker-compose exec -T exist java org.exist.start.Main backup -u admin -p admin.entrada -b /db/academia -d /export
docker-compose exec -T exist java org.exist.start.Main backup -u admin -p admin.entrada -b /db/academia-2001 -d /export
docker-compose exec -T exist java org.exist.start.Main backup -u admin -p admin.entrada -b /db/apps/academia -d /export
docker-compose exec -T exist java org.exist.start.Main backup -u admin -p admin.entrada -b /db/apps/academia-2001 -d /export
docker-compose exec -T exist java org.exist.start.Main backup -u admin -p admin.entrada -b /db/schemas -d /export

rsync -aASPvz --delete-after outdir/db/ dacl/db/

DATE=`date +%Y%m%d`
cd dacl && git commit -a -m "Backup $DATE" && git push origin v5

Of course this is not rocket science, and this approach might have a lot of problems, but in the other hand, it might get handy to someone.

eXist-db: Installing XQuery functx module

existdbUsing the blog as a notepad, I will start posting here some notes on things I discover and are not very clear in the documentation (or I just did not find it at first).

In the last times I have been hacking in eXist-db, and writing XQuery. I noticed a website with a lot of interesting functions with the functx prefix. The eXist-db website help reference returns hits for this module. But it is not installed by default (or at least, it can happen on not being installed by default).

To install it, just run this XQuery command:

repo:install-and-deploy("http://www.functx.com", "1.0",
"http://exist-db.org/exist/apps/public-repo/public/functx-1.0.xar")

After installation the functx module can be loaded with
import module namespace functx = "http://www.functx.com" at "/db/system/repo/functx-1.0/functx/functx.xql";

Also, for reference, I found the list of eXist packages here.