In all comparative research, the validity of findings depends on the equivalence of the materials being compared (Palicki et al., 2023; Rinke et al., 2022). In a text as data project, this means that a researcher must first select equivalent document sources. While not feasible for all research projects, given that some studies rely on “found data” (Salganik, 2017, pp. 82-83) with unknown generative processes, materials need to be sufficiently equivalent in relation to the research question at hand.³ This guarantees that detected differences are not mere byproducts of subtle differences in the selection of documents. During the process of selecting document sources it is vital to ensure the equivalence of the source sampling procedure across cases, and make note of any remaining discrepancies (e.g., when specific corpora, for which no suitable equivalent exists in other cases, are included for substantive reasons) so that they can be addressed later in the analysis.

When equivalent document sources have been identified, in many cases, corpus creation furthermore requires the selection of equivalent document samples. This can involve narrowing inclusion criteria to specific relevant document populations (usually, using keywords) or creating stratified samples that capture key dimensions of variability in the population under study. Whenever the cases under study also involve different languages, researchers need to additionally ensure that their retrieval strategies yield equivalent document samples (e.g., translated keywords often find or miss different subsets of relevant debates, owing to divergent connotations and language-specific expressions).⁴ To achieve this goal, researchers need to reflect on potential sources of sampling bias and empirically evaluate whether sample equivalence has been achieved.

Suppose we wanted to replicate Muddiman and Stroud’s (2017) study of news user commentary in the New York Times (NYT) across multiple languages. In such a scenario, a first consideration concerns resource bias and retrieval bias, as recommended by (Grimmer et al., 2022).

Resource bias arises when the availability of documents differs systematically across languages and cases. For their study, Muddiman & Stroud (2017) were able to access all comments posted in response to NYT coverage since the inauguration of its commentary feature, including also comments that were subsequently moderated or deleted. Besides the obvious challenges of determining which media outlet maps well onto the NYT, other media outlets may have opened commentary sections later, selectively, or failed to keep long-term records, and very few, if any, will grant comparable access to moderated content. In some countries, media primarily rely on social media platforms to enable user commentary, subjecting the posting, moderation, and archiving of content to external platform policies and norms. In a similar vein, analyses of X (formerly Twitter) content need to account for the platform’s very different roles and adoption rates across contexts, considering whether comparable political discourse might occur on various social networks instead. Media outlets, platforms, and other public forums can differ between countries concerning their policies for moderating or surpressing uncivil content, adding systematic distortions.

Retrieval bias occurs when the sampling strategy for identifying relevant documents works differently between cases in ways that introduce bias into subsequent measurements. For example, a search string for identifying swearwords may have high precision and recall in one language, but perform considerably worse when translated into another language (e.g., Russian, profanity is often rooted in sexual language; in German, it tends to use fecal metaphors; and in the U.S., profanity is frequently abbreviated [“bs”] or blanked [“f***”]).⁵ In most cases, it cannot be assumed that identical sampling strategies, applied across cases and languages, will reliably generate equivalent samples.

When evaluating equivalence among language-specific documents, reliance on domain expertise or involvement of case experts becomes essential (Palicki et al., 2023; Rinke et al., 2022). Often, collecting equivalent samples across languages requires comparative reasoning. For instance, in cases where text sources are media outlets, a comparison of market shares within each case can guide equivalent source selection (Rössler, 2012). Suppose document samples are retrieved with search strings in different languages for different cases. In that case, these retrieved documents should be compared with human-annotated data to assess recall and precision, and misclassifications should be inspected (Mahl et al., 2022). A good example of such careful practice is provided by Rauh & Wilde (2018), who offer a transparent account of how they evaluated and ensured cross-lingual equivalence in their selection of EU legislative debates.

After selecting the corpus for each language, researchers need to ensure that the produced units and textual representations are equivalent across cases. In most comparative research, this may require actions such as harmonising unitization or stripping uninformative, idiosyncratic content across cases. When dealing with multilingual text, linguistic perspectives also come into play. Factors like how languages segment sentences into words, compound word formation, function word affixation, and morphological variations demand attention. Researchers are encouraged to reflect on and empirically test the equivalence of the data inputs across languages and cases. Importantly, the input data should be selected with the intended analytical model in mind, as different approaches (e.g., dictionary methods, topic models, or transformers) may place different demands on the structure and quality of input data.

Data input validation first concerns unitization, i.e., the determination of what level of text relevant meanings are expressed and to what units of text they pertain. For multilingual analyses, this is especially important as differences in the meaning contained within each unit of text may confound equivalent measurement. Cultural, stylistic, and linguistic differences affect unitization, such as in the case of languages that prefer long (e.g., German) or short (e.g., English) sentences, or in Chinese speech, which does not segment words in writing, resulting in substantively different meanings of paragraphs. To ensure comparability, researchers may need to adapt the unit of analysis per language—for instance, combining several English sentences to match a dense German one, or using word segmentation in Chinese. Justifying these choices with linguistic and contextual insight is key to valid cross-language analysis.

In CTA, unitization does not end here, however; rather, texts need to be further broken down into so-called “tokens” - typically derived from words - whose distribution over different documents presents the primary source of information for many computational algorithms. However, tokenization is far from language-neutral. When deciding on the appropriate set of tokenisation steps, researchers need to consider in what form relevant meanings are encoded in the text. To ensure that equivalent information is retained in multiple languages and cases, it may be required to use different preprocessing strategies for different languages and cases so that the obtained textual representations are comparable and adequate for the intended analysis. For instance, many languages create numerous distinct tokens for the same word, depending on its case, gender, etc., which appear as independent from one another to a computer unless harmonised. In English, this may be addressed by stemming (as used by Muddiman & Stroud, 2017). However, in languages that do not inflect words solely at the end, more sophisticated forms of lemmatization are required. For example, Hebrew has pre-, in-, and suffixes: “ An error in the conversion from LaTeX to XML has occurred here. lSn’w”, “ An error in the conversion from LaTeX to XML has occurred here. Sn’w”, “ An error in the conversion from LaTeX to XML has occurred here. Swn’”, “ An error in the conversion from LaTeX to XML has occurred here. tSn’” all reduce to the same root meaning “hate,” requiring more advanced processing to treat them as equivalent.

Similarly, information expressed by conjunctions, prepositions, and other tokens typically discarded as “stopwords” in English (e.g., Muddiman & Stroud, 2017) is often conveyed through morphological variation, word order, or other mechanisms in other languages. For example, the English phrase “son of a bitch” risks losing key relational meaning if “of” is removed as a stopword. In contrast, the equivalent Arabic expression “ An error in the conversion from LaTeX to XML has occurred here. ibn al-‘AhiraT¿” retains its structure and meaning through morphological encoding, illustrating how English-centric preprocessing can inadvertently distort multilingual semantic comparisons. These linguistic properties may confound a comparative study of uncivil language in unforeseen ways.

Additionally, in certain languages, incivility might manifest through the use of explicit swear words, while in others, it might be expressed through specific multi-word phrases. In Dutch, the intensifying use of disease-related terms such as kankerlijer (cancer sufferer) or tyfushond (typhus dog) creates compound insults that carry strong emotional weight, a pattern less common in English. Furthermore, the target of uncivil language is more easily observable in languages that hardwire grammatical function in particular word inflections (e.g., Turkish encodes the accusative case). Similarly, when one is interested in the extent to which uncivil language is gendered, it helps to consider how each of the languages under study distinguishes genders (if at all).

To determine input data validity, some extent of linguistic familiarity – and to the extent that pragmatic equivalence is required, cultural familiarity – is necessary. Researchers are generally advised to consult with native speakers and case experts when preprocessing unfamiliar languages and carefully validate outputs from standard pre-processing tools (such as those in NLTK, Bird et al., 2009), which may not generalize across languages. Tasks such as token reduction⁶. and word sense disambiguation⁷ must be adapted with care, and multilingual preprocessing should be approached not as a standardized pipeline, but as a language-sensitive design challenge.

Ideally, researchers can offer evidence that input data alignment is established. If an analysis relies on preprocessing, a way to test the validity of the input data is to check if the meaning that is studied can still be easily recognised from the preprocessed text. At the same time, all variations that are lost should be immaterial to the pursued research question. One strategy that should help with detecting validity issues and biases in multilingual analysis at the level of preprocessing is to translate a few texts before preprocessing them and subject them to the respective preprocessing stages. The key test of cross-lingual validity is that the representations obtained after preprocessing map well onto one another, such that how relevant information is expressed (e.g., by single tokens, by token collocation patterns, or by sequentially arranged tokens) is similar between languages and cases. In studies where translation is part of the preprocessing stage, such as multilingual CTA projects that bring corpora into a shared representational space (e.g., English language text), back-translation is an effective validation technique. By iteratively applying machine translation tools, this method assesses the semantic stability of translated texts by testing whether repeated cycles of translation yield consistent and interpretable results (Chew et al., 2025). As one example of a study that presents extensive input validation steps, Segev (2019) discusses and argues for the selection of certain word types and the exclusion of others to increase the cross-linguistic and cross-national validity of the measurement instrument.

In addition to striving for equivalent textual representations, the researcher must also ensure the equivalence of the intended modeling strategy across languages and cases. In any use of CTA, the primary consideration governing validity is whether the algorithm indeed recognises the meaning intended to be measured. In multilingual projects, researchers must confirm that the algorithms are equally effective in all languages and cases. In practical terms, validation efforts concern model development, model selection, and model calibration.

For their recognition of incivility, Muddiman & Stroud (2017) relied on a carefully validated dictionary of one-word tokens. Transferring the same design to different languages, however, numerous sources of bias arise from linguistic and context-related factors that may compromise the effectiveness of the same modeling strategy. A deeper understanding of the algorithmic design is a prerequisite for understanding how modeling assumptions and linguistic features interact across languages.

Most rule-based algorithms (e.g., dictionaries or search strings) offer considerable researcher control and thus generally facilitate reflection upon modeling choices across languages. Enabled to recognize that terms included in Muddiman and Stroud’s (2017) incivility dictionary are polysemic, context-dependent, or require multi-word expressions in other languages, researchers can relatively easily add disambiguation criteria or shift toward a multi-word enabled strategy, while adjusting keywords for case-specific contexts as needed. For example, in Quebecois French, liturgical terms like tabarnak and câlice function as strong profanities and would likely be included in a dictionary of incivility for that context. In contrast, the same terms would appear semantically benign in continental French and thus be excluded. This contrast underscores how regional cultural variation affects which expressions register as uncivil and must be accounted for in multilingual modeling.

Matters are more complicated for unsupervised, supervised algorithms, or more recent textual representations such as embeddings, where association measures are typically hard-coded into the algorithm. Researchers must assess whether document-level word co-occurrences (common in classic machine learning) suffice as representations of the expressed meaning, or whether more restrictive measures need to be obtained (e.g., by designing specific features that focus estimation on particular patterns in the data). Moreover, unsupervised algorithms tend to make strict assumptions about the generative structure of a text. For instance, many clustering algorithms assume that every token belongs to exactly one cluster, while factor analytic or topic modeling algorithms permit that the same token may contribute to more than one cluster. Especially in languages with many homonyms (i.e., words that have the same spelling) or words that appear as part of different multi-word expressions, some clustering solutions are inappropriate for certain analyses, as the same token cannot be constrained to one role exactly ⁸. Languages that form compound words (that is, two or more words joined together to create a single, distinct word with a unified meaning) may pose issues as key terms in the analysis are too infrequent to constitute recognised patterns. For instance, if we were to search for patterns in the topical structure of uncivil comments, the algorithm has much better chances to recognize welfare and cheat (both words common also in other contexts) as a pattern than Sozialbetrüger in German, a relatively rare term unique to the topic). Especially for supervised algorithms, providing substantive justifications for the choice of specific algorithms or hyperparameters often presents a major challenge, as very little is known how exactly modelling language in a hierarchical fashion (e.g., decision tree algorithms), in a vector space (e.g., SVM), or a neural network might affect classification. While it is evident (and plausible) that such choices matter, virtually no theory is available for justifying specific decisions.

Such considerations are not only relevant for models that rely on word co-occurrences (such as bag-of-words models), but also for models that represent words as embeddings. For example, word embeddings represent each word as a vector of real numbers that is based on the premise that words that appear a lot near each other in a set of documents should have similar vector representations (for an overview, see Rodriguez & Spirling, 2022). While early embeddings were primarily developed for English and faced challenges in other languages due to data scarcity and linguistic differences, recent advances have produced embeddings that cover dozens of languages (e.g., Polyglot) as well as multilingual embedding systems (Licht, 2023; Vries, 2021), with promising cross-lingual alignment (Chew et al., 2025). Nevertheless, theoretical concerns persist regarding how grammatical word order and syntactic structures—such as the Subject-Verb-Object order in English versus Subject-Object-Verb or other variations in other languages—may influence the semantic quality and comparability of embeddings. Understanding these effects remains an important open question, especially when embeddings are used for measurement across languages.

Returning to our ongoing example of detecting uncivil language, different languages may have different swearing cultures, which will have implications for either approach. For instance, in some languages, name-calling relies on well-known slurs or offensive adjectives that frequently co-occur and are easy to detect. In others, incivility may take more subtle or culturally specific forms. In Dutch, for instance, calling someone a pannenkoek (pancake) is mildly offensive. It is roughly akin to calling someone an idiot, even though the literal meaning is benign. In some online subcultures, such as incel communities, incivility involves the creative coining of novel terms (neologisms) or the reappropriation of innocuous words (e.g., foid, roastie, or skype used derogatorily), which can be difficult to detect without cultural or contextual knowledge. In East Asian languages like Korean, supervised machine learning may, to a significant extent, simply rely on honorific forms to decide that no incivility can be present. In contrast, in most Western languages, such a form is unavailable or degraded, and thus unavailable for classification. When measuring uncivil language, such cultural differences should be explicitly acknowledged in the codebook to be used for annotating training documents for multilingual supervised classification or in the interpretation of topics in a multilingual topic model.

The easiest way to ascertain process validity is to subject a few texts to the algorithm’s transformation and determine whether the represented information (e.g., associations) reflects meaningful features of the processed text, in the case of multilingual analysis, whether it reflects the same meaningful features in equivalent ways. In general, for multilingual analysis, it is desirable to determine that equivalent modeling choices indeed emphasise and discount equivalent information. If researchers create their own set of manually annotated training documents that inform the model, testing intercoder-reliability across languages and cases for these training materials serves as another process validation strategy. Calculating metrics like Machine Translation Accuracy (MTA) can serve as an effective validation strategy for comparing the multilingual outputs of various models. For an implementation that assesses the consistency of top words per topic across languages via MTA, refer to Lind et al. (2022). Validation strategies encompass further activities like visualising how inputs relate to outputs and identifying features that are most influential in the model’s decision-making process (see, for example, the LIME technique as implemented in Ho & Chan (2023)). When relying on generative LLMs like ChatGPT, researchers can, for example, interrogate these models to explain their reasoning for why they annotate text in one way or another, and they can do so in multiple languages (see e.g., Kuzman et al., 2023). As a positive example, Maurer & Diehl (2020) offer important details on how they validated their French and English dictionaries against crowd-sourced benchmarks and by further employing a qualitative pre-analysis of individual keywords.

The fourth validation stage concerns the quality of the obtained measures and their equivalence across languages and across cases concerning criterion validity (the obtained measures correlate with a trusted third variable) and face validity (the results appear plausible for each case) (Krippendorff, 2004). For comparative multilingual applications, output validation needs to be considered for each included language and case.

Our discussion on output validation focuses on the type and quality of a third variable that can serve as a trusted benchmark for case and language comparisons. Researchers are advised to assess available third variables for either the same construct (for testing convergent validity) or distinctly different constructs (for testing divergent validity). When selecting or developing a human-coded benchmark as a third variable, as was the case in Muddiman and Stroud’s (2017) study, its quality can be assessed by inspecting the quality of the codebooks, human coder training, and intercoder reliability tests. Codebook definitions should apply to all languages and cases (i.e., etic constructs), while rules and examples may need to be adjusted to different contexts and settings (e.g., racist incivility tends to focus on different ethnic groups in different countries). As with monolingual projects, it is advantageous to train all involved coders in joint sessions and to clarify issues or adjust the codebook collaboratively (Rössler, 2012). Intercoder reliability tests should cover reliability across languages/cases as well as within languages/cases (Peter & Lauf, 2002). If all coders are skilled in one language, they can code the same material to establish intercoder reliability (Hopmann et al., 2016). Care should be taken to ensure that the manually coded material is representative of all languages and cases. To achieve this, a random subset per language and case can be translated into a common language. To further assess intercoder reliability within language/case, at least two coders code the original language material per case. If a measured meaning is far more dominant, or largely absent, in one compared case, careful interpretation of performance scores is crucial; high accuracy does not necessarily imply valid discrimination (e.g., if incivility appears in only 3% of cases, not coding it achieves 97% accuracy). Employing chance- and category frequency-corrected metrics might be necessary for informative measurement validity. For example, when examining uncivil language, do we see a higher recall or precision in some languages than others? And what features are predictive of observed differences?

Applying the convergent validation strategies involves the selection of a high-quality gold standard as described above and the calculation of recall and precision scores for each language and case in the dataset. For example, Lind et al. (2021) compare the results from a supervised classification approach with a benchmark coded by native speakers. Temporão et al. (2018) assess the convergent and predictive validity of their computationally obtained ideological scaling measures of social media users. Another key strategy for output validation, then, is to examine misclassifications carefully (Ho & Chan, 2023). Systematic issues in precision are detected by inspecting positively classified but irrelevant documents. Recall issues can be examined by drawing a random sample of texts for manual classification and examining which texts are missed by the model. When a proper gold standard exists, examining false positives and negatives can uncover possible biases and direct enhanced performance of the employed tools. Last but not least, it is of course advisable to critically check the results manually with case experts for their plausibility.