Analysis of word use in the reading literacy and science literacy parts of PISA 2018: Comparison of the Icelandic translation and the original English version

Abstract

Icelandic learners’ performance in the reading and science literacy parts of PISA has declined from 2000 to 2015, and the drop in mean scores is one of the most dramatic among participating countries. The percentage of Icelandic participants in the highest proficiency levels has fallen, and the percentage in the lowest levels has risen.

PISA tests are written in two parallel source versions, English and French, and then translated into other languages. OECD publishes guidelines (2016) for translators in which it is stated that translators should avoid simplifying or complicating the vocabulary and the syntax. Due to the direct relationship between word understanding and text comprehension (Laufer & Ravenhorst-Kalovski, 2010), it is of high importance that translated words be carefully chosen. If there is a higher number of difficult words in one language than another, the readers may have more difficulty in applying the requested reading strategy. Such bias may affect the validity of the measurement. Studies have demonstrated a strong relationship between the extent to which words are known by individuals and word frequency (Baayen, Wurm & Aycock, 2007; Balota, Yap & Cortese, 2006; Gardner, Rothkopf, Lapan & Lafferty, 1987; Meunier & Segui, 1999; Oldfield & Wingfield, 1965).

In the PISA 2018 translation and adaptation guidelines (OECD, 2016, p. 11) it is stated that „longer words tend to be less frequent, more technical and/or more abstract than short words“. Nonetheless there is no requirement that translators refer to word frequency lists as an effort to match the frequency of translated words with the words in the original version.

The purpose of this research was to compare the alignment of word frequency in Icelandic translated texts and the original English versions of PISA 2018.

Two text parts were randomly selected from the reading literacy section and two from the natural science section of PISA 2018. Information about the frequency of Icelandic words was obtained from a frequency list based on the Icelandic Gigaword Corpus (Steinþór Steingrímsson, Sigrún Helgadóttir, Eiríkur Rögnvaldsson, Starkaður Barkarson & Jón Guðnason, 2018; Stofnun Árna Magnússonar í íslenskum fræðum, 2017). The software VocabProfile (Cobb, n.d.) was used for the English words, based on two corpuses: the New General Service List and the New Academic Word List. The words were grouped into frequency bands, the most common 1000 words in each band for the most frequent 4000 words, and less frequent words together in one band. When translated Icelandic words did not fall into the same frequency band as the corresponding English words, appropriate Icelandic synonyms were looked for, and the length of the synonyms was compared.

Results of the study indicate that the share of words in the highest frequency band was lower in the Icelandic translated texts than in the English versions, and the share of Icelandic words in the lowest frequency band was higher, in all four analysed texts. Additionally, the English words were more evenly distributed between the five frequency bands than the Icelandic words. Furthermore, among the words that did not belong to the same frequency band in Icelandic as in English, the proportion of Icelandic words of lower frequency was higher. If the Icelandic translators had made use of synonyms in the same or adjacent frequency band to the English corresponding words, a better equilibrium between the languages could have been obtained. More than 30% of more frequent Icelandic words were longer than their less frequent synonyms, which suggests that for Icelandic words it is not a reliable rule that longer words tend to be less frequent than shorter words.

Our findings are an indication that the PISA translation guidelines should include a requirement that translators make use of word frequency lists when choosing words for their translations, so as to make the comparison between countries more valid. If word knowledge is more challenging in one country than another, the impact of the different proficiency factors to be measured is not the same, which may affect the validity of the study. The findings should contribute to the interpretation of PISA results in reading and science literacy for 2018, at least when comparing participants who took the tests in Icelandic and English.