Roaster: an Open API Router for eXist

We’re happy to announce that the request routing library, which was originally developed for TEI Publisher 7, has been released as a separate package with extended functionality. The library, called roaster, is generic and can be used for any eXist-based project. It implements the Open API 3.0 standard to support well-documented, versioned and formally specified APIs.

Background

In previous versions of TEI Publisher, clients (i.e. your web browser) would communicate with the server by directly calling a variety of XQuery scripts. The server-side API, if you can even call it one, was thus scattered over many different files. Finding your way through the code, figuring out what parameters are expected or how the response should look like was rather difficult. Overwriting the default behaviour – e.g. to replace the generated table of contents – required substantial coding skills. The scripts also changed between TEI Publisher versions. So, if you were working on a standalone edition generated from Publisher, updates could be tricky.

With TEI Publisher 7, the entire server-side API can be viewed on a single documentation page. It clearly describes the URL paths you can use, as well as any parameter you can pass in and the type it should conform to. One can also see the different possible responses and what kind of content they would return.

The new API

Looking at the first route of the documents section in the API (see screenshot below), it is easy to construct a URL which returns the source XML: the path template to use is /api/document/{id} and {id} should contain the path to a document – relative to the data root of TEI Publisher.

Documents API screenshot

So to retrieve the TEI/XML for Graves’ letter, located in the file path test/graves6.xml, we can use the following URL:

https://teipublisher.com/exist/apps/tei-publisher/api/document/test%2Fgraves6.xml

Note that the / in the path needs to be URL encoded with %2F. This is a requirement of the Open API specification.

If instead of the TEI/XML we would like to see the letter rendered to HTML, we can use the third route in the list and simply add /html to the end of the URL:

https://teipublisher.com/exist/apps/tei-publisher/api/document/test%2Fgraves6.xml/html

or if we prefer a PDF:

https://teipublisher.com/exist/apps/tei-publisher/api/document/test%2Fgraves6.xml/pdf

For sure, as an ordinary user, you don’t need to know any of this: using the web interface of TEI Publisher, the web components on the page take care of constructing and calling above URLs for you. But if you are a developer, having a well-defined API is a game changer. Just imagine that you want to support your co-workers with a script which allows them to preview a local TEI document as HTML on the fly: sending the content of the document with an HTTP POST request to /api/preview is all you need! Our Visual Studio Code plugin does it like this.

You can use any script or programming language you like, say bash, python, perl – you name it. And because Open API is a widely used standard, there are plenty of tools for documentation, testing or code generation. The API documentation page in TEI Publisher is generated by such a tool (Swagger UI).

Implementation

roaster is essentially an implementation of the Open API standard in pure XQuery. It reads the formal API specification (in JSON format) and determines for each HTTP request coming in, which route to take. It will also check if the parameters, headers or request bodies passed in comply to the rules given in the definition of the route. An error will be generated if the request is not in compliance with the definition, e.g. because a required parameter is missing or has a wrong type. It can also fill in default values for parameters, enforce correct content types for the response etc.

From a developer perspective, this means you don’t have to worry about parameters. Your handler function will receive a single parameter, containing all the necessary information about the request and you can safely assume that it complies with the spec you provided.

If you are interested in the details, please refer to the README. For a TEI Publisher-related example, check out the FAQ article, which describes how to replace the default table of contents with a custom one.

Roaster 1.0.0 can be installed into your local eXist via the package manager in the dashboard. TEI Publisher 7 shipped with a slightly older version, 0.5.1., but you can run both versions side by side.

TEI Publisher 7: an example of collaborative development

2020-12-18, Wolfgang Meier

We’re happy to announce the final release of TEI Publisher 7. This is the second release coordinated by e-editiones and another important step to improve interoperability, sustainability and long-term maintenance of editions created with TEI Publisher. While release 6 came with a major redesign of the client side parts of TEI Publisher, version 7 focusses on the server, featuring a clean and extensible API based on the Open API standard.

The plan for TEI Publisher 7 (and 6 before) was born during a meeting in Basel in November last year, when some of the Swiss edition projects using TEI Publisher came together to discuss how the software could be developed into an even more sustainable platform. Many of them became founding members of e-editiones a few months later.

Fortunately the Swiss Nationale Infrastruktur für Editionen – Infrastructure nationale pour les éditions agreed to support the outlined ideas, funding a substantial part of the work. Other member projects contributed time and money, and served as testbeds for the new versions.

TEI Publisher 7 is thus more than just another technological milestone – it is the product of a cooperative development model in which users, researchers, institutions and developers work hand in hand towards a shared goal.

We’re looking forward to continue this success story next year and would again like to invite everybody to become part of it. Support the common cause by becoming a member, convince your institutions to take part, and contribute in one of the many possible ways.

Read the full release notes on the TEI Publisher website.

New Visual Studio Code extension for TEI Publisher

Enhancing TEI documents with semantic markup, i.e. encode places, people or terms, can be a time consuming and tedious task. For the Karl Barth edition project, we have thus created a new extension for Visual Studio Code (vscode) and published it as Open Source.

Where it all started: a Word plugin

Within the Karl Barth edition, texts to be published in new volumes are encoded in Microsoft Word and transformed to TEI using TEI Publisher’s powerful docx2TEI conversion feature based on the TEI processing model.

To simplify semantic encoding, eXist Solutions created a Word plugin for the project, which allows editors to easily encode entities occurring within the text: select a string representing an entity, click a button and the plugin will run a query against the database of the Karl Barth Gesamtausgabe, showing potential matches for each possible category.

With another click, the selected entity is inserted into the Word document, including a reference to the entity in the database. This proved to be a huge timesaver: what took minutes before – switching to a different window, doing a search and copying the ID – is now done in seconds. A screenshot of the Word plugin is shown below:

MS Word screenshot

The plugin also allows editors to get a preview of how the current text would look like in TEI and how it would be rendered to HTML by TEI Publisher’s ODD-based transformations. The preview feature sends the underlying Word XML to a TEI Publisher instance, which transforms it first to TEI, and then – if requested – to HTML. Editors can thus see the text as it would later appear on the website.

Porting it to Visual Studio Code

However, the Word-workflow only covers the new volumes: we still have 54 already published volumes in TEI, which need to be extended. Parts of this is done automatically (via entity detection), but there’s still a lot of manual editing involved. Thus the question arised if the Word plugin could be ported to an XML editor.

For sure, oXygen would have been the obvious choice, but since the Word Plugin was already written in javascript/typescript, we decided to give Visual Studio Code a try. It turned out to be rather straightforward and the first usable version was done within one evening. Thrilled by the success, we invested two more days to make the plugin more generic, in particular, add connectors for other authority databases. Beyond the Karl Barth Gesamtausgabe, we now have connectors for the German GND, metagrid, Geonames and Google places.

Enity lookup screencast

The TEI snippet to be inserted for each of the entity types can be configured in the settings. For example, you could replace <persName> with <name type="person"> if you like.

Previews

The HTML preview feature has been ported as well: pressing a keyboard shortcut or clicking the toolbar icon will present you with a list of ODDs available for transformations, and then open a new editor panel showing the transformation result in HTML. All you need is a TEI Publisher 7 instance which has a matching ODD installed. The plugin uses the general API introduced with TEI Publisher 7, which will also be exposed automatically by any application generated from it.

HTML preview screencast

Snippets

Finally, it is rather straightforward to create reusable snippets in vscode. To start with, we implemented the one shortcut we use most in oXygen and miss everywhere else: wrap the current selection with an element (the Scholarly XML extension described below also has this, but it doesn’t hurt to have it twice in case you’re not using that).

And since we have to encode a lot of foreign texts, we also added one for <foreign>. Doing more would be easy: maybe you have some suggestions? Just write them into our github issues.

Combine with other extensions

If you combine this with other extensions available on the vscode marketplace, you get rather decent XML editing support – not as sophisticated as with oXygen, but good enough for some serious work.

For schema-based validation and context sensitive suggestions one can choose between two extensions: Scholary XML is lightweight and provides nice suggestions with documentation. It only understands relax-ng though and does not support catalogs, which means you need to reference the schema via a processing instruction in every TEI file. The second option, XML Language Support by Red Hat, supports XML schema and catalogs, which you can use to associate the TEI namespace with a schema once and for all. On the downside, it uses more system resources as it starts a background Java process.

To set up a catalog, just create a file catalog.xml somewhere with e.g. the following content and change the vscode configuration property xml.catalogs to point to it:

<catalog xmlns="urn:oasis:names:tc:entity:xmlns:xml:catalog">
  <uri
      name="http://www.tei-c.org/ns/1.0"
      uri="https://tei-c.org/Vault/P5/current/xml/tei/custom/schema/xsd/tei_all.xsd" />
</catalog>

Finally, if you use TEI Publisher as a basis for your own editions, you definitely want to install the Language Server and Client for XQuery/eXistdb. It supports working on XQuery files, syncing your local changes with the database, installing XAR files and more.

Get it

The extension is available from Microsoft’s Visual Studio Marketplace. To install it in your own Visual Studio Code, just open the Extensions tab and search for tei-publisher-vscode.

All source code is available on the e-editiones github and we would like to invite others to further extend it, e.g. by adding connectors to other authority APIs. Connectors are simple javascript/typescript classes with just two methods. See the existing connectors for inspiration.

If you have questions, suggestions or would like to discuss further extensions: you’ll find us in the e-editiones slack rooms.

2 Comments on New Visual Studio Code extension for TEI Publisher