The Reliability of Human Disambiguation in Text Markup Kevin J. Keen Department of Epidemiology and Biostatistics Case Western Reserve University kjkeen@darwin.cwru.edu Paul A. Fortier Department of French, Spanish, and Italian University of Manitoba fortier@cc.umanitoba.ca 1999 University of Virginia Charlottesville, VA ACH/ALLC 1999 editor encoder Sara A. Schmidt Semantic Fields Studying semantic fields or literary themes in texts confronts researchers with a paradox. A computer string search will produce a list of the frequencies of words potentially related to the semantic field. But polysemy implies an un-measurable difference between the potential allusion and the real allusion. The semantic field of solitude is important from sociological and psychological perspectives as an indication of imperfect adaptation to one's milieu. It is also a frequently occurring literary theme. The only practical response is disambiguation by human informants. But the reliability of such a process is a concern. Singh (1986) and Fokkema (1988) discuss the matter in a theoretical and speculative mode without systematic empirical evidence. Nothing bearing directly on the topic can be found in recent issues of Computers and the Humanities or Literary and Linguistic Computing. The correctness of a given choice by an individual informant or rater is generally unknowable. Usually, such a choice is a matter of degree and judgment within the cultural context of a language. The consistency of markup from one informant to another is a likely reasonable touchstone for assessing the reliability of the disambiguation process. Data Nine 20th century novels were chosen for this analysis: Bernanos, Journal d'un curé de campagne, Camus, L'Étranger and La Chute, Céline, Voyage au bout de la nuit, Gide, L'Immoraliste and La Porte étroite, Mauriac, Le Noeud de Vipères, Proust, La Fugitive, and Sartre, La Nausée. French thesauri identify seventy words (in the sense of lemmata) related to the concept of solitude. These strings were used to search the texts for words potentially related to solitude in the ARTFL database. Words centered in 60 characters of context were given to six informants with minimal instructions: mark the words, which from a reading of the context, do not evoke the concept of human solitude and go back to the ARTFL database for more context in doubtful cases. Table 1 summarizes the number of allusions to human solitude. Note that the range of the number of counts of allusions to solitude varies widely according to the text examined. To assess the impact of providing a greater amount of context to informants, 300 characters of context from the original text were obtained using the same set of strings as for 60 characters of context. Identical instructions were given to six informants--two of whom participated in the first project. Table 2 summarizes the number of allusions to human solitude for a context of 300 characters. When a larger context is provided, the range of variability in the results increases. On the other hand, the consistency of the raters is slightly improved. Analysis The sample intraclass correlation coefficient, denoted as formula ICC(3,1) by Shrout and Fleiss (1979), has been chosen as the measure of agreement among the informants. For dichotomous responses, an intraclass correlation coefficient of one-quarter for a sample consisting of two informants is to be interpreted to reveal that two informants were in agreement for twenty-five per cent of the observations after correcting for the possibility that any agreements of the two informants were entirely due to chance The measure of reliability of a sum of responses from a random sample of a set of a fixed number of raters drawn from a population of raters is known as the Spearman-Brown prophecy after Spearman (1910) and Brown (1910). Cronbach's alpha, due to Cronbach (1951), is an estimate of the Spearman-Brown prophecy. Both the Spearman-Brown prophecy and Cronbach's alpha are increasing functions of the intraclass correlation--parameter and statistic, respectively--and each asymptotically equal 100% in the limit as the number of raters increases without bound despite the consideration that adding more raters only increases the chances that everybody will not agree. Statements made regarding a population based on a sample must be qualified by a probability clause. The standard choice for a probability clause is such that the statement be correct in the long run, if the statistical procedure were to be done unboundedly many times, 95% of the time--or nineteen times out of twenty. So a measure of the degree of difficulty associated with a specific text in determining whether there is an allusion to solitude is given by the minimum number of informants to achieve a value of 95% for Cronbach's alpha nineteen times out of twenty. Results Hypothesis tests revealed statistically significant differences regarding reliability among the different texts and among the two character-string lengths. Further exploratory analysis ruled out a particular informant, a particular part of speech, or a particular class of frequency of use as being influential. The 95% confidence intervals for the intraclass correlation coefficient are given in Table 3 for both 60 character and 300 character groups surrounding each type. Note that for each novel the confidence interval for rater reliability is shifted to the left with the 300 character groups compared to the 60 character groups. With more context, the estimate of reliability becomes lower. Note the lack of overlap for the 95% confidence intervals for the two sizes of character groups for each of the novels by Bernanos, Céline, and Proust. It is conjectured that as the sample size surrounding each type increases, there is more opportunity for subjectivity based on personal opinion. Table 4 presents the number of informants required to achieve 95% for Cronbach's alpha nineteen times out of twenty for 60 and 300 character contexts. For each novel, as the size of context increases, the number of informants required to achieve 95% for Cronbach's alpha increases. Moreover, the size of a jury to decide whether a word alludes to human solitude with 300 characters of context ought to be no fewer than 45 based on the novels surveyed. With respect to literary analysis, note that the novels of Camus have a greater spread in jury size than those of Gide. Conclusion It appears that the use of informants for studying semantic fields, or literary themes, is justifiable from a statistical perspective. However, the large numbers of informants or jury members appears prohibitive. Paradoxically, reliability appears to decrease as the number of characters of content increases. Further studies are needed to determine whether increases in reliability can be obtained by changing the focus from a word to a sentence or longer passage. These results suggest a high degree of subjectivity when a single individual scores the semantic content of literary data. The meaning of individual words in context is a matter of opinion, and cannot be taken as definitive until a high degree of consensus among a large number of raters or informants is achieved. Table 1Scores for Human Solitude with 60 Characters of Context Texts raw p1 p2 s1 s2 s3 s4 BJC 260 68 89 51 44 100 91 CET 74 22 19 13 11 23 26 CCH 115 32 41 26 31 36 41 CVN 488 89 133 78 41 127 153 GIM 89 28 42 36 18 47 48 GPE 108 26 41 36 26 37 54 MNV 159 42 70 47 28 92 71 PFU 304 49 80 57 44 82 76 SNA 233 111 113 77 86 95 122 Table 2.Scores for Human Solitude with 300 Characters of Context Texts raw p3 p4 p5 s5 s6 s7 BJC 260 98 44 154 45 73 51 CET 74 23 11 48 6 14 10 CCH 115 37 34 61 23 30 26 CVN 488 153 64 199 53 85 60 GIM 89 42 29 49 13 25 22 GPE 108 36 29 49 13 25 22 MNV 159 80 32 113 28 41 43 PFU 304 68 53 126 49 69 44 SNA 233 114 96 151 67 70 79 Table 3.95% confidence interval: intraclass correlation coefficient Texts 60 Characters 300 characters Lower Upper Lower Upper BJC 0.56 0.66 0.43 0.54 CET 0.48 0.68 0.29 0.51 CCH 0.68 0.79 0.54 0.69 CVN 0.51 0.59 0.42 0.50 GIM 0.40 0.59 0.28 0.48 GPE 0.43 0.60 0.33 0.50 MNV 0.47 0.60 0.33 0.50 PFU 0.63 0.71 0.49 0.59 SNA 0.59 0.69 0.48 0.59 Table 4.Number of informants required for a value of 95% or more for Cronbach's alpha nineteen times out of twenty. Texts 60 Characters 300 Characters BJC 15 25 CET 19 43 CCH 9 16 CVN 18 26 GIM 28 45 GPE 25 37 MNV 21 25 PFU 12 19 SNA 14 21 Explanation of Symbols used in the Tables Texts: BJC: Bernanos, Journal d'un curé de campagne CET: Camus, L'Étranger CCH: Camus, La Chute CVN: Céline, Voyage au bout de la nuit GIM: Gide, L'Immoraliste GPE: Gide, La Porte étroite MNV: Mauriac, Le Noeud de Vipères PFU: Proust, La Fugitive SNA: Sartre, La Nausée. Informants: raw: raw solitude data as downloaded from the ARTFL database s(1-7): French Literature Graduate Student p(1-5): French Literature Professor References W. Brown Some experimental results in the correlation of mental abilities British Journal of Psychology 3 296-322 1910 Georges Bernanos Journal d'un curé de campagne Oeuvres Romanesques Bibliothèque de la Pléiade Albert Béguin Paris Gallimard 1935 1961 Albert Camus L'Étranger Théâtre, Récits, Nouvelles Bibliothèque de la Pléiade Roger Quilliot Paris Gallimard 1945 1962 Albert Camus La Chute Théâtre, Récits, Nouvelles Bibliothèque de la Pléiade Roger Quilliot Paris Gallimard 1956 1962 L.-F. Céline Voyage au bout de la nuit Romans Bibliothèque de la Pléiade H. Mondor Paris Gallimard 1932 1962 J. Cohen A coefficient of agreement for nominal scales Educational and Psychological Measurement 20 37-46 1960 L. J. Cronbach Coefficient alpha and the internal structure of tests Psychometrika 16 297-334 1951 Douwe Fokkema On the reliability of literary studies Poetics Today 9 529-543 1988 André Gide L'Immoraliste Romans, Récits, Soties, Oeuvres lyriques Bibliothèque de la Pléiade Y. Davet J.-J. Thierry Paris Gallimard 1902 1958 André Gide Porte étroite Romans, Récits, Soties, Oeuvres lyriques Bibliothèque de la Pléiade Y. Davet J.-J. Thierry Paris Gallimard 1909 1958 François Mauriac Le Noeud de Vipères Paris Grasset 1932 Marcel Proust La Fugitive À la recherche du temps perdu Bibliothèque de la Pleiade Pierre Clarac André Ferré 3 vol. Pars Gallimard 1925 1954 Jean-Paul Satre La Nausée Oeuvres Romanesques Bibliothèque de la Pléiade Michel Contat Michel Rybalka Paris Gallimard 1938 1981 P. E. Shrout J. L. Fleiss Intraclass correlations: uses in assessing rater reliability Psychological Bulletin 86 420-428 1979 Gurbhagat Singh The identity of the literary text: Problems and reliability of reader response: An inquiry into theoretical positions The Literary Criterion 21 4 49-57 1986 C. Spearman Correlation calculated with faulty data British Journal of Psychology 3 271-295 1910