Project context

Discussion of project context here.

Relation of ATMO transcription, text, and sentence vocabularies

All ATMO documents begin life as transcriptions of manuscripts, using the ATMO TEI customization for transcriptions described in ATMO technical report 2015-03. Such transcriptions use the TEI sourceDoc element and have a structure which mirrors the physical organization of a manuscript, with writing surfaces, zones of writing, and lines.

Documents to be translated and annotated are then transduced into the TEI customizations described here, which use the TEI text element and have structures based on the logical organization of the text, with sections, paragraphs, verse lines and line groups, and sentences. There are two structures here, not one, because sentences generally nest within paragraphs, but do not necessarily nest neatly within (or neatly contain) verse structures. So (oversimplifying slightly and leaving some details for later) we will define both a purely textual view, in which sentences sometimes cross structural boundaries, and a sentence view, in which textual structures sometimes cross sentence boundaries.

In order to support translation back and forth among the three forms without loss of information, each form needs to include the same information, covering the physical, the textual, and the sentential structure of the source, each conceived of as a hierarchical structure of nesting constituents. How we represent these multiple structures in XML is described below.

Concurrent hierarchies

We analyse our documents as having three distinct structures, each representable as a tree of nesting elements. One describes the physical structure of pages, regions on pages, lines in regions, and characters within lines. One describes what is sometimes called the logical structure of texts or collections of texts, paragraphs, blocks of verse, and phrase-level elements occurring within paragraphs and verses. A third divides the text into sentences or sentence-like units, each sentence consisting of words.

For brevity we will refer to these as the P, L, and S structures.

Each structure forms a tree, so each is individually suitable for representation using XML. But the nodes in the three trees don't nest within each other: we have three trees over essentially the same frontier of leaves, not one tree with elements of three different kinds.

The data structure is essentially that defined by the SGML specification with its optional feature CONCUR. (See for further discussion of CONCUR.)

Trojan-horse markup

Since CONCUR has not been widely implemented for XML (see and ), we use a different representation of our concurrent structures, adapting a markup idiom introduced in by Steven J. DeRose () called Trojan-Horse markup.

In our XML representations of documents, we choose one of the structures to represent in the conventional way using XML elements (one XML element for each node in the tree structure. The structure so represented we call the dominant tree. The other trees are also represented, by Trojan-Horse (TH) elements; we call them recessive trees.

It will be seen that we have our choice of three different views of the document, with different dominant trees: P, L, and S. The three views are equivalent in the information they record, and we generate the desired view on demand. (To reduce confusion, we normally treat the S view as the master copy and generate the other views on demand when needed. But some maintenance tasks are best performed in views P or L, and some presentations of documents on the Web are best based on views P and L.)

Examples needed.

Validating multiple concurrent structures

Each distinct tree structure in the document can readily be validated using any document-oriented schema language (DTDs, Relax NG, XSD, ...). If there are constraints on the interrelation of the trees (if an element E in tree X cannot begin within element F of tree Y, for example), then validation of those constraints requires either something like rabbit/duck grammars () or a predicate-based schema language (like assertions in XSD or like Schematron) and some indeterminate amount of ingenuity in formulating predicates to check the constraints.

The ATMO project does not currently impose any inter-tree constraints; the trees are treated as completely independent of each other.

It is simplest to define each tree structure independently, and validate it independently of the others: a pre-processor can strip all markup related to the recessive views before invoking a validator.

But for editing documents in conventional XML editors, a schema that describes just the dominant tree in the current document view does not suffice: we need a schema which also describes and accepts the Trojan-Horse elements needed for the recessive views. Because there are non inter-tree constraints, the TH-elements for recessive views can occur at any point in any element of the dominant view.

The schemas defined here

In the light of the preceding discussion, it will be clear that we need several schemas, of different kinds: one base schema for each different view (P, L, S), and one augmented schema, which adds declarations for TH-elements for the recessive views and modifies the content models of the base schema to make them allow TH-elements at any location.

The base schema for view P is defined in ATMO technical report 2015-03; the augmented schema for P will be added there soon.

This document defines first the base schemas and then the augmented schemas for views L and S.

These views share many elements, and in fact they have non-nesting elements only (as far as is now known) for verse texts, since sentences do not nest within the verse structures.

Some complications also arise for the augmented schemas from the fact that all three views share the same header, and may share other elements. Conceptually, the universe of all elements should be the same for all schemas; it is the union of four sets of elements:

H: teiHeader and all of its possible descendants (this should be the same in all ATMO schemas), also TEI.

P: sourceDoc and all of its possible descendants in the base grammar for the page-oriented view

T: text and all of its possible descendants in the text-oriented view

S: text and all of its possible descendants in the sentence-oriented view

Some elements must be declared as conventional content elements. Some elements must be declared as Trojan-Horse elements. Some elements must be declared as both; within the header they will appear as conventional elements, and within the text or source-document they will appear as Trojan-Horse elements. The rules are:

Any element type in H must be able to appear as a conventional content element.

Any element type in the set associated with the dominant view must be able to appear as a conventional content element, and will in fact never appear as a Trojan-Horse element.

Any element type not in the set for the dominant view but in the set for a recessive view must be able to appear as a Trojan-Horse element.

It follows that in the augmented grammars, elements must be declared as follows:

In grammar PTS (augmented grammar for P):

Elements in P are declared as content elements, with the addition of Trojan Horse attributes (e.g. th:doc, which indicates the document type of the element; strictly speaking this is redundant but it simplifies processing).

For elements which are members of both P &intersect; H the Trojan-Horse attributes are optional.

For elements in P but not in H, the Trojan-Horse attributes are required.

Elements in either recessive set (T, S) which are neither in P nor in H are declared as Trojan-Horse elements: required to be empty, and required to carry appropriate Trojan-Horse attributes.

Elements in either recessive set (T, S) which are in H but not P are declared as a choice between content elements (with no Trojan Horse attributes) or empty Trojan Horse elements (with required Trojan-Horse attributes).

Elements in H not already described are declared as content elements, with no Trojan-Horse attributes.

Or in summary form:

P &union; (H \ (T &union; S)): content-element declaration.

(T &union; S) \ (P &union; H): Trojan-Horse declaration.

(H &union; T &union; S) \ P: both content-element and Trojan-Horse declarations.

And similarly (mutatis mutandis) for the other two augmented grammars.

A Venn diagram may be helpful here. Here D is the set of elements available in the text area of the dominant view, R the union of the elements in the text areas in the two recessive views, and H the header, as above. The areas within the Venn diagram are labeled C if those elements get conventional content-element declarations, TH if they get Trojan-Horse declarations, and C+TH if they should be able to take either form.

A Venn diagram showing the intersections of the various sets of elements and how they are to be declared.

Open questions

Some design decisions remain open.

Does Oxygen deal acceptably well with Perso-Arabic script in attribute values? If so, we can have m, etc., be empty elements carrying all information in attributes. If not, all or some of those attributes need to be brought out into sub-elements.

How should the synchronization of base text and extensive commentary (as in Prov. 450) be managed? (Do we have examples like that among our transcripts?)

How should text movement between views be managed? Are there in fact any headings added in the margin? If so, view P will want them in one place and views T and S in another.

The overall structure of the schema

The schemas for view T and view S are both essentially a list of instructions to include particular TEI modules or declare new elements. They share a large common core, but differ in some details.

Excluded from this list are the modules certainty, corpus, dictionaries, drama, iso-fs, nets, spoken, tagdocs, and textcrit.

For the moment, we include all the elements of the manuscript description module, even though they are used only in the TEI header. As a task for the future, we should remove any elements not needed in the records for our manuscripts, so that there is less clutter in menus of possible elements.

The header module

We include the header module:

From the header module we exclude declarations for several elements we expect never to use:

The correspAction, correspContext, and correspDesc elements all relate to collections of letters.

The prefixDef, listPrefixDef, namespace, and xenoData elements all relate to technical capabilities of TEI markup which we are not using at the moment.

The typeNote element describes information relevant to the interpretation of the rend attribute; it is irrelevant to manuscripts.

It may perhaps be more useful to identify the elements in the TEI header module which are included, rather than just those which are excluded. These fall into several classes. Those we expect to need most frequently are the standard parts of a TEI header and elements we know we will need: The teiHeader encloses the header. The fileDesc, authority, availability, cRefPattern, distributor, edition, editionStmt, extent, funder, idno, licence, notesStmt, principal, publicationStmt, refsDecl, refState, seriesStmt, sponsor, and titleStmt elements are all standard elements for the bibliographic description of the TEI document itself. The sourceDesc element will contain the description of the source manuscript in the form of an msDesc element. handNote is used to describe the hands of a manuscript. The encodingDesc, projectDesc, and samplingDecl elements provide basic information about the preparation of the digital facsimile. The rendition element can be used to document special values of the rend attribute; the styleDefDecl element can be used to identify a formal styling language (like CSS) used in the definition of rendering styles.

The refsDecl, cRefPattern, and refState elements relate to machinery for canonical references; we expect to provide canonical numberings for sentences in annotated texts and expect to use these to document them.

The revisionDesc, change, and listChange elements are used to record the history of the TEI document. appInfo and application allow the encoding description to include explicit records of automatic processing of the document by software; we will use these elements to record the translation of the manuscript descriptions from the Master DTD to TEI P5 and the preparation of the scanned images in JPEG form from the TIFFs supplied by LUB.

Some elements we do not expect to need very often, but include in the schema in case we need them, on the theory that having unused elements in the schema will cause fewer headaches during the project than lacking an element we need. The biblFull element may be needed for bibliographic data within the annotations in the header. The profileDesc, calendar, calendarDesc, catRef, classCode, creation, keywords, langUsage, language, and textClass elements describe the text(s) contained in the manuscript, in general terms. We will not use them in the initial description of any manuscript, but if we enhance the metadata for any manuscript, these elements are likely to be useful. In the encodingDesc element, the catDesc, category, classDecl, geoDecl, tagUsage, tagsDecl, and taxonomy elements may be useful for enriched manuscript descriptions; they won't be part of the initial representation. The editorialDecl, correction, hyphenation, interpretation, normalization, punctuation, quotation, segmentation, and stdVals elements are all relevant for transcriptions, though not for digital facsimiles. TODO: include them!

Some elements should perhaps be suppressed; further thought is needed. abstract scriptNote

The text-structure module

The basic text-structure module defines elements used in structuring the text, such as div and related elements; it also includes title pages and material that often appears there, as well as the essential TEI element.

We are thus omitting: argument, div1, div2, div3, div4, div5, div6, div7, docEdition, docImprint, floatingText, imprimatur, and salute.

The core module

From the core module, we include a number of elements for use in transcription and annotation of the text. First, specialized elements important for generic text structure: head, l, lg, list, item, note, p, sp, speaker.

Verse (lg and l) is included in view T.

Then, phrase-level elements for signaling properties of the inscription and/or the transcription: add corr gap del hi unclear

A number of phrase-level elements signal specific semantic or linguistic properties of a passage: date distinct emph foreign gloss mentioned name num ptr q ref rs term title

And finally, we also include: index milestone

We include the following elements not for use within the text but because they are used in existing headers: author pubPlace resp respStmt

In the version of the TEI used as the basis for this design, this means we are omitting the following elements from the core module: abbr addrLine address analytic bibl biblScope biblStruct binaryObject cb choice cit citedRange desc divGen editor email expan gb graphic headItem headLabel imprint label lb listBibl measure measureGrp media meeting monogr orig pb postBox postCode ptr publisher quote ref reg relatedItem said series sic soCalled stage street teiCorpus textLang time

The names and dates module

We include the module for names and dates because it includes elements for country and city (settlement), which we need. We exclude most of the rest of it.

The elements country, settlement, and orgName are included primarily because they are used in the headers (in the manuscript descriptions).

The primary sources module

We include the module for primary sources in order to get the elements elements damage, fw, and space. We may wish to exclude other elements here, but we keep them for now.

Miscellaneous modules

A few modules are included for the sake of a relatively small number of elements from each module.

We include the gaiji module to allow the encoding of non-Unicode characters, if and when we encounter them.

We include the figures and tables module because documentation of non-Unicode glyphs requires figure, and there is tabular material in some of our manuscripts. (There may also be drawings which should be included via figure.)

We include the module for linking in order to get the elements seg and ab The other elements in the module we exclude, at least for now.

The caesura element from the verse module is included in all views. (It's empty and cannot conflict with any hierarchy.)

Project requirements for linguistic annotation

One of the principal aims of the project is to provide detailed linguistic annotation, word by word and morpheme by morpheme, of selected texts.

Note, however, that resource limitations make it impossible to annotate every document, and the fact that we use human taggers means that even documents which are being annotated will need to be available (and valid) before the annotation is complete. So the schemas defined here need to work for documents that do not have such annotation, as well as for documents that do have it, and for documents which are partially annotated.

The problem we need to solve in this part of this document is to construct an XML representation of markup for the internal structure of words which satisfies as well and simply as possible the following requirements and desiderata. (See also ATMO Technical Report 2017-01 for a discussion of the annotation from a purely linguistic point of view.)

The markup should identify segments of type word, affix, and clitic.

To simplify the overall markup, any parts of words not marked as affixes or clitics will also be marked up as segments. (In theory, segments which are neither words nor affixes might be left untagged; in practice, it's simpler if the they are tagged.)

For simplicity of exposition, each of the smallest segments identified will be called a morph in this discussion. For affixes and clitics, the term morpheme would be appropriate enough, but morpheme and even morph may be a stretch for word stems which themselves are composed of several morphemes; this can happen when an annotation aims to record the inflectional morphology of the text, but not the derivational morphology. The patience of the reader is requested for this terminological awkwardness. The generic segment may be used instead even though it is not in general restricted to the smallest (atomic) units identified by segmentation.

The examples below typically use the following elements with the meanings indicated:

w marks segments identified as words.

m marks segments identified as morphs, typically (but not necessarily) individual morphemes; the use of m does not imply that the morph so tagged is not an affix or clitic.

clitic marks clitics.

aff marks affixes.

mm marks segments for which the tagger specifically wishes to avoid any implication that the segment is monomorphemic.

(Note: At this point, any exceptions to these patterns in the examples below are errors which should be corrected.)

The markup should make it possible to reconstruct the appropriate conventional written form for the materials. Ideally the process should be easy or at least straightforward.

In particular, it should be possible to ensure that white space is supplied where appropriate and only where appropriate.

In effect, this means that it is desirable to ensure that the orthographic words of the conventional written form of the text can be reconstructed. By orthographic word is meant a token delimited by white space or punctuation; the concept does not apply in all writing systems. Often (i.e. in many or most linguistic analyses, of languages written in writing systems which use white space to delimit words) orthographic words and the words which appears as units in linguistic analysis are the same; one of the problems addressed below is how to deal with cases where they are not the same.

It is not the task of this discussion to specify how to segment text into words or words into morphs, but how to represent a segmentation given by the linguistic analysis.

This description thus has no opinion on whether English man consists of one morph or two, or whether French dix-sept seventeen, Uyghur tört besh four or five, or Uyghur on besh fifteen are respectively one word or two. In all three cases, the problem we face is to ensure that both of the obvious analyses can be represented.

The markup should represent the most common cases simply.

The markup should support validation of the structural relations among words, affixes, and clitics.

Together with the preceding desideratum, this means in effect that in-line markup is likely preferable to stand-off markup.

The markup should have as clean a translation into vanilla TEI as can conveniently be managed.

It is not guaranteed that all of these can be satisfied fully at the same time; tradeoffs may be necessary.

Sentences

Since linguistic annotation proceeds sentence by sentence, in the S view all tagged words should be gathered together into sentences or sentence-like units. Paragraphs, for example, will not directly contain character data, or even atmo:lit / atmo:lat pairs of the kind used in the initial transcriptions. (It should be explained that lit is short for literal or literatim; the atmo:lit element contains a literatim transcription of material in the manuscript. lat is short for Latin and contains an information-preserving transliteration of the mterial into the Latin alphabet.) Instead, paragraphs should be segmented into sentences. See below () for the pattern modifications necessary for this.

Markup for sentences is (in this version of these schemas, at least) specific to the S grammar.

Open question: Should sentence markup also be part of the T grammar? Sentences do normally nest within paragraphs, etc., but we do not expect nesting of sentences and verse, so including sentences in the T grammar would involve making them sometimes normal matryoshka elements, and sometimes Trojan-Horse elements.

Tentative answer: no; sentences will not be part of the T grammar. Rationale: easier to understand and design the schemas, easier to understand and work with the markup in any single view.

Words and morphemes

Most word types in any dictionary, and most word tokens in any text, will for our purposes have substantially similar structures. We describe that basic structure first, and postpone until later any discussion of complications (see ). An attempt has been made to make the considerations general, but it is the needs of the ATMO project which are central. The XML encodings are inspired by the TEI usage of the BNC and the XML version of the Brown corpus.

Most of the English examples are constructed; some are drawn from the Brown corpus. They sometimes place POS information and other annotation on the word (to echo the usage of Brown) and sometimes on parts of the word (to be more similar to the Uyghur examples here). Uyghur examples are from data collected by the Uyghur Light Verbs (UyLVs) project; I've used examples from published texts where possible, but it has not always been possible. The linguistic analyses are those in the UyLVs data, unless otherwise noted.

In the base case:

Every word (viewed as a linguistic object) is written as one blank- or punctuation-delimited sequence of characters (= one orthographic word).

Every orthographic word represents (spells? writes?) one linguistic word.

Every linguistic word is composed of a sequence of one or more morphs.

Every orthographic word is composed of a sequence of one or more morphs.

For now we ignore the distinctions among types of morphs. A morph is a morph; we don't attempt to distinguish sub-classes of morphs.

Examples for the base case: English characters ]]> Uyghur atasighe to [the] father: ]]>

Here and in other examples,

reg = regularized form (for Uyghur, this will be the conventional spelling of the segment in Uyghur Latin script (ULY), or something approximating it.

ipa = transcription (fairly broad) into the International Phonetic Alphabet.

pos = part-of-speech tag (like most part-of-speech tag sets, the one used here includes some morphological and other information that goes beyond the eight parts of speech identified by ancient grammarians of Latin). For English examples, the tags used are from the Brown corpus, or made up when necessary. For Uyghur examples, the tags used are from the Uyghur Light Verbs project.

ilg = interlinear gloss: for lexical morphs, a rough translation into English; for grammatical morphs a tag (often the same as the POS tag) describing its grammatical function.

In what follows, many the examples omit for brevity several attributes or elements planned for ATMO:

lit literal transcription of original script

lat Latin transliteration of lit

ums underlying morphemic segment (an abstract representation of a morpheme or morph which has different surface appearances based on context)

For the base case, we might define words and morphemes something like this. As the following section shows, these definitions are too simple. But they provide a pattern we can extend to handle the complications presented by the data.

word contains a sequence of morphemes constituting one linguistic word and one orthographic word. morpheme represents a single morpheme or segment of a word; attributes indicate the spelling, part of speech, and other properties. Literatim transcription from ms xs:string Transliteration into Latin alphabet xs:string Regularized Latin spelling xs:string IPA approximation xs:string Underlying morphological segment xs:string Part of speech tag for this morph xs:string Interlinear gloss xs:string

More complicated word/morpheme structures

Elements for words and morphs

The elements for words and sub-word segments are common across schemas.

When annotation is present, then normally every word in the text will be marked as a word, its inflectional morphemes will be identified, and each segment will be given a regularized spelling, a part of speech code, and an interlinear gloss.

Tier-by-tier annotation and the atmo:phr element

As an alternative to the markup shown in the preceding sections using w, m, etc., we also provide a level-by-level, tier-by-tier, or horizontal form for annotation.

In this encoding, each level of representation or annotation is encoded using an XML element (atmo:lit, atmo:lat, atmo:pos, etc.). Within a level, words and punctuation tokens are delimited by whitespace, morphemes by hyphens. Hash marks (#) are used to represent orthographic white space within segments the annotator chooses to regard as single linguistic words. If sentences are tagged and fully annotated, a paragraph in the horizontal form will have a structure like this: ... ... ... ... ... ... ... ... ... ... ... ]]> When sentences are not tagged (e.g. in the view T), the paragraph will consist of atmo:phr and other phrase-level elements: ... ... ... ... ... ... ]]>

As can be seen, the atmo:phr element is essentially a wrapper (or substitute) for #PCDATA, in manuscript text.

Encloses parallel versions of character data in a transcription. literatim text Contains a literal (unregularized, uncorrected) transcription of some portion of the exemplar. Latin text Contains a 1:1 transliteration of the text in the sibling atmo:lit element.

No attempt is made in this version to constrain the character set of this element. It would be a good idea.

regularized and segmented text Contains a version of the text in the sibling atmo:lat element in which spelling has been regularized and selected morpheme boundaries have been indicated within words by inserting hyphens or (for clitics) equals signs.

No attempt is made in this version to constrain the character set of this element, or the pattern of whitespace and segment boundaries. It would probably be a good idea to do so.

International Phonetic Alphabet transcription Contains a rendering of the text of the sibling elements in the International Phonetic Alphabet (IPA), to indicate the (conjectured) pronunciation. The text is segmented as in the atmo:seg sibling.

No attempt is made in this version to constrain the character set of this element. It would probably be a good idea to do so.

underlying morphemic segment Contains a version of the text in the sibling atmo:seg element in which each segment is replaced by a standardized spelling representing the postulated underlying morpheme. This provides a standard written representation which reverses the phonological effects of context on the morpheme and allows phonetically different realizations of the same morpheme to be readily found. part-of-speech tagging Contains a transcription of the text of the sibling atmo:seg element in which each segment is replaced by a part-of-speech tag.

No attempt is made in this version to constrain the values to tags actually defined in the project's POS tag set. It would be a good idea to do so.

interlinear gloss Contains a transcription of the text of the sibling atmo:seg element in which each segment with a lexical meaning is replaced by an English-language gloss and each segment with a grammatical meaning is replaced by its POS tag or a similar grammatical indicator.

A short example of annotation

An example may make the similarities and differences between the representation of a document using atmo:w and atmo:phr elements.

The tier-by-tier horizontal representation is easier to build by hand, so it is not infrequently the initial annotated representation of documents. In horizontal form, the second and third full sentences on p. 7 of manuscript Prov. 351 might be represented as follows. Some long lines have been broken for this display. سودانی دفع swdany dfʾ swda-ny dfʾ N-ACC N melancholy-ACC cure بولسون دیسه ترنجه بین ذعفرانینی اوج کون bwlswn dysh trnǰh byn z_ʾfranyny awǰ kwn bwl-swn dy-sh trnǰhbyn z_ʾfrany-ny awǰ kwn LVN-3VOL Vt-COND N N-ACC NU N LVN-3VOL say-COND turnajabin saffron-ACC three day ناشته دا ایچسه سودانی دفع قیلور našth da ayčsh swdany dfʾ qylwr našth-da ayč-sh swda-ny dfʾ qyl-wr-0 N-LOC Vt-COND N-ACC N LVN-IPFV.DIR-3 breakfast-LOC drink-COND melancholy-ACC cure LVN-IPFV.DIR-3 To heal melancholy, if one drinks turnajabin with saffron at breakfast for three days, (his/her) melancholy heals. اجغ نمرسه لاردین aǰġ nmrsh lardyn aǰġ nmrsh-lar-dyn AJ PN.INDEF-PL-ABL spicy thing-PL-ABL البته حزر قیلماق کراک albth ḥzr qylmaq krak albth ḥzr qyl-maq krak AV N LVN-GER XAJ of.course abstain LVN-GER XAJ It is necessary to abstain from spicy (اجغ) foods. ]]>

In the word-by-word form, the same material will look like this: To heal melancholy, if one drinks turnajabin with saffron at breakfast for three days, (his/her) melancholy heals. It is necessary to abstain from spicy (اجغ) foods. ]]>

Again long lines have been broken for display; in an XML editor, with each atmo:m element on a single line, the representation may be slightly easier to read. The example shows lit and lat attributes on w; these are strictly speaking redundant and could be reconstructed from the content of the element, but they provide a convenient check on the results of any automated segmentation process. Redundant attributes reg, pos and ilg may also be used. The schema will be written to allow them, and some software may populate and use them.

Open issue: We do not currently have a story about when horizontal markup is used and when vertical markup is used. Segmentation of texts into paragraphs and sentences in an XML editor will be easier with vertical markup; manual annotation without a specialized user interface such as an XForm will be easier with horizontal markup.

It would be desirable to have a story.

It would also be desirable to have stylesheets to translate back and forth between horizontal and vertical formats.