Can a Team Tag Consistently? Experiences on the Orlando Project Terry Butler Arts Technologies for Learning Centre University of Alberta Terry.Butler@UAlberta.ca Sue Fisher Susan Hockey Greg Coulombe Patricia Clements Susan Brown Isobel Grundy Kathryn Carter Kathryn Harvey Jeanne Wood 1999 University of Virginia Charlottesville, VA ACH/ALLC 1999 editor encoder Sara A. Schmidt Introduction Given that a team of thirty-five humanities researchers cannot possibly use structural and interpretive SGML tags consistently over a period of three years, what issues face a major SGML research project wanting to impose adequate amounts of consistency to their tagged data? This is the key question currently facing the Orlando Project as we take stock and embark on a process of tag cleanup work. Extent of the Project's Tagging The Orlando Project is in the 4th year of its 6 year tenure as a Major Collaborative Research Initiative funded by the Social Sciences and Humanities Research Council of Canada and the Universities of Alberta and Guelph. Our aim is to research, write, and tag in SGML an integrated history of women's writing in the British Isles. We are not tagging pre-existing texts; rather we are creating our own literary history through conducting primary research that we then filter through SGML tagging. To do this we have created three unique yet interdependent SGML document types (DTDs): one that permits the description of biography, one that takes into account all the factors that contribute to a writing career, and one that provides an architecture for describing chronological events. Our DTDs are modeled structurally on the TEI but each contains many interpretive tags that allow us to foreground our research practices and label the intellectual content of our material. For example, the biography DTD has tags for birth, family, education, and political affiliations; writing documents use tags for such text-specific information as genre, intertextuality, literary awards, and relations with publishers; events documents contain chronological events that have such information as organization names and places tagged. Currently there are 252 unique tags in our DTDs and 114 unique attributes. These tags are used extensively across over 2,200 documents in our system. As of April 1999, the total number of elements in use in all project documents was over 640,000. For example, in biography documents alone the <quote> tag is used 2,135 times and the <name> tag 8,103 times. There are 51,125 uses of <date> in events documents. With element numbers of this scale, it became clear to us that we simply couldn't clean up our tags on an element by element basis. This paper will address the issues that we have faced when trying to achieve tagging consistency on the project. It will also report on a pilot study on tag consistency work that we have undertaken in the Fall and Winter of 1998-1999. Is Consistency Possible? We realize that achieving complete consistency with a diverse team of taggers using a complex tagset is impossible. Surveys of similar work done in the fields of indexing [Leonard, 1977], tagging linguistic data [Garside, 1993], and hypertext linking [Ellis, 1994] show that only a modest degree of consistency can be achieved by a team, even with ample training and a more focused and smaller tagset than we possess. Our own process of document analysis when we developed our DTDs lead us to the same conclusion: it is immensely difficult for a group of people to come to a common understanding of exactly what a particular tag means and what its use in the research document should be. However, there is a level of consistency to which we aspire, and it is predicated upon the needs of the scholarly community who will be our end users. Therefore, some tag cleanup will be necessary to ensure adequate search and retrieval, chronological sorting, and consistent presentation of material to our research communities. Current and Past Efforts at Achieving Tagging Consistency Although we have only begun tag cleanup work in earnest in the Fall of 1998, we have had practices in place from the early stages of the project that have helped us work towards consistent tagging. Our Graduate Research Assistants (GRAs) receive three full-time days worth of training each September, after which they begin tagging and have their work reviewed by experienced team members. This training and checking is complemented by a comprehensive online glossary of our SGML tags and tagging practices. When a tagger cannot find an answer in the documentation or is faced with a new tagging/research problem, she can pose a question of our student e-mail discussion list. Here Postdoctoral Fellows answer most questions that come up (referring the more difficult cases to an e-mail list of Co-investigators and Postdoctoral fellows) and revise and augment the documentation when necessary. Our practices for ensuring tag consistency have proven enormously useful, but they cannot in themselves moderate the work of 20 researches using such a complex tag set. Tags sometimes get used without the tagger realizing that an issue has been discussed or appears in the documentation. At over 500 pages, the documentation is a testament to the fact that a complex research project deals with issues that can't be tracked by all taggers at all times. Tag cleanup pilot In the Fall of 1998, we began a tag cleanup pilot study to investigate the problem of inconsistent tagging and to draw conclusions about the degree of consistency which will be achievable. During the pilot, GRAs were each given a group of representative tags to look at in detail across all project documents. The following is a list of some of the strategies they used to uncover inconsistency problems in the data: Comparing the use of a tag to how its use has been documented and recommending a change in practice or documentation where necessary. Investigating the use of 'odd' sub-elements. For example, there is an instance in the system where a word has been tagged as both <genre> and <characterName>. In all likelihood, this is a tagging mistake. Finding missing attribute values such as, for example, the titletype attribute on <title>. Investigating "odd" uses of attributes. Finding improperly filled in attributes. For example, the value attribute on <date> has some inconsistencies in terms of how standard dates should be expressed. Investigating the over-use of a single tag in a document. In their reports, the GRA's noted which problems could be fixed with batch search and replace capabilities and which would need to be manually fixed. The pilot helped us realize that the tags most in need of fixing were those that acted as index/linking tags (name, orgname), those that governed presentation of our material (title, quote), and those that needed to be consistent for automated sorting/alphabetizing (date, name, orgname, title). Computer Tools In the spring of 1999, we turned our attention to developing a set of protocols for fixing these tags. We defined and ran a set of batch changes that would address many of the consistency problems outlined in the tag cleanup reports. Our programmer has also created a set of tools over the last year that would ease the process of manual cleanup. A Web front-end to the SGREP program with canned search queries allows team members to see just how a tag or attribute has been used (or misused) across all project documents. A "System-wide Document Statistics Report" provides statistical information on the use of tags and their attributes and reports documents where tag use has varied widely from normal usage. An finally, a "Tag Cleanup Reporting Tool" alphabetizes any given tag according to its standard or reg attribute across any combination of project documents. These tools will be demonstrated and their merits discussed in the during session. Conclusions The insights gleaned from our pilot study have helped us better quantify and assign the work that we are currently undertaking. By setting priorities for fixing the problems, we hope that we have defined a level of consistency that we can achieve and that we can learn to be content with. We also hope that we can now avoid having qualified researchers spending valuable time fixing problems that can be more quickly and accurately fixed by computer processes. We hope that doing this work will make our data rich in its breadth and depth of research and tagging and will make our end product meaningful and accessible to our user community. References Lawrence E. Leonard Inter-Indexer Consistency Studies, 1954-1975: A Review of the Literature and Summary of the Study Results Univ of Illinois Graduate School of Library Science, Occasional Papers 131 Univ of Illinois Graduate School of Library Science December 1977 David Ellis Jonathan Furner-Hines Peter Willett On the Creation of Hypertext Links in Full-Text Documents: Measurement of Inter-Linker Consistency Journal of Documentation 50 2 67-98 June 1994 Roger Garside The Large-scale Production of Syntactically Analysed Corpora Literary & Linguistic Computing 8 1 39-46 1993