SMC Monthly Report: September 2024
EventsSMC WebsiteSMC new website launched in Bengaluru. Riya Sabu volunteered to initiate the new website,
This post is originally published here
My work, Quantitative Analysis of the Morphological Complexity of Malayalam Language is accepted for presentation at the 23rd International Conference on Text, Speech and Dialogue to be held from September 8-10, 2020. The conference proceedings is published by Springer and you can read the paper here. I will share the presentation slides and video after the conference.
This blog-post is a bit detailed discussion of the following aspects, analysed in the paper:
By measuring the morphological complexity and comparing it against the same measurements for Dravidian languages and European languages, I show that Malayalam is morphologically more complex than them.
Malayalam is a language with productive morphology. That means Malayalam words undergo inflections, derivations and compounding, producing an infinite vocabulary. For example the noun ആന (elephant) can undergo inflections by attaching suffixes to it to form words like ആനകള് (elephants), ആനയുടെ (of elephant), ആനയ്ക്ക് (for elephant) etc. A lot of words can be derived from a single word പാടുക (sing) as പാട്ട്(song), പാട്ടുകാരി(female singer), പാട്ടുകാര്(singers), പാടി(sung), പാടിയിട്ടുണ്ട് (has sung), പാടിക്കൊണ്ടിരിക്കുന്നു(is singing), പാടാം (shall sing) etc. Multiple words combining to form new words can be seen in ആനക്കുട്ടി (baby elephant), മണല്ത്തരി (sand grains) etc. These compound words can further undergo inflections and derivations. This is the reason behind the complexity (richness) of Malayalam morphology.
A native Malayalam speaker produces sufficiently complex words during natural speech with super ease. ഞാനിപ്പൊക്കൊണ്ടുവരാം is a 'compound word', meaning I shall bring it soon. But you will never find the word ഞാനിപ്പൊക്കൊണ്ടുവരാം in a Malayalam dictionary, because it is practically impossible to list all possible compounding, inflections and derivations of Malayalam words. So a dictionary will have only entries for the root words. Having said this, it must be intuitively clear that Malayalam is a morphologically rich language.
But this is not a feature unique to Malayalam. There are many languages in the world that follow this pattern of productive morphology. Other Dravidian languages like Tamil and Kannada, European languages of Turkish, Finnish, Dutch etc. belong to this category by varying extends.
So Malayalam is morphologically complex. You can go on making new words to express yourselves in Malayalam. But is there a quantitative measure to indicate how much is the complexity? How do you know if its morphological complexity is comparatively higher than another language, say Finnish?
Well, there are multiple definitions for morphological complexity, but no unanimously accepted measures.
One definition by Patrick Juola follows an information theoretic approach based on Kolmogrov Complexity measure. (Read more: Measuring linguistic complexity: The morphological tier, Patrick Juola). Another approach by Bane separates word stems, affixes and signatures and defines it in terms of Description Length(DL). The number of possible inflection points, the number of inflectional categories, number of morpheme types are all indicators of morphological complexity. (Read more: Quantifying and Measuring Morphological Complexity, Max Bane). These methods involve detailed linguistic know-how, requires annotated text corpus with morpheme tags, human expert judgements or availability of an automatic tagger for the language to be analysed.
Alternatively there are corpus linguistic analysis techniques to measure morphological complexity and these techniques are proved to give results comparable to those based on typological information. (Read More: A Comparison Between Morphological Complexity Measures: Typological Data vs. Language Corpora, Christian Bentz)
The corpus linguistic parameters being analysed here are Types and Tokens.
Types and Tokens are related to word counts in a text corpus. Any element in the set of all distinct words in a text corpus is called a type and every occurrence of type in a corpus is called token.
For example there are 7 types and 9 tokens in the following sentence.
To be or not to be is the question
On the other hand, its translation in Malayalam has 4 types and 4 tokens.
വേണോ വേണ്ടയോ എന്നതാണ് ചോദ്യം
For a morphologically complex language more meanings get embedded into every wordform and diverse wordforms appear in a corpus which are seldom repeated.
For the sample in English above,the ratio between types and tokens is given by:
TTR = Type Count/ Token Count = 7/9 = 0.77
But for the Malayalam sample,
TTR = Type Count/ Token Count = 4/4 = 1
A morphologically complex language with inflectional and agglutinative nature will have relatively less number of text tokens in a parallel corpora, but will have more number of unique types. (Read More: Comparing morphological complexity of Spanish, Otomi and Nahuatl, Ximena Gutierrez-Vasques)
It is important to note that the TTR of a text sample is affected by its total length; obviously, the longer the text goes on, the more likely it is that the next word will be one that has already occurred.
For a language with simple morphology, the number of unique words(types) reaches a saturation as the total number words(tokens) under consideration increases. It means, even if you parse more and more on a text corpus, it will be difficult for you to find a word that has not appeared before. But for a language like Malayalam with productive morphology, the number of types does not actually reach a saturation level easily.
Type-Token Growth Rate is an indicator of how does the cumulative number of unique types vary with the cumulative number of total tokens parsed.
In general higher the value of TTR, more will be the morphological complexity. (Read More: A Comparison Between Morphological Complexity Measures: Typological Data vs. Language Corpora, Christian Bentz).
The TTR depends largely on the length of text analysed. So another way to analyse the relative number of types and tokens are in terms of TTR on a window of specified text size. Then that window is moved over the entire span of text. The resulting values of TTR is averaged across all window positions to obtain the Moving Average Type Token ratio(MATTR).
We will now analyse and compare the following indicators of morphological complexity measurements on a large Malayalam text corpus.
To apply these measures and analyse these values for Malayalam, the Malayalam Wikipedia article dump, as available from SMC Malayalam corpus is used.
The corpus is cleaned up for text normalization, removal of punctuation, other language script etc. using the following script
Once this is done, there are more than 8 million tokens in the corpora. The number of unique types in the first 1000 tokens were 805. The first 2000 tokens
contained 1487 unique types. Computing this cumulatively for the 8 million tokens resulted in the following table. The interval of cumulative counts in the successive rows is not fixed. It is small initially (1000) and slowly rises(1000000).
Tokens | Types |
---|---|
1000 | 805 |
2000 | 1487 |
3000 | 2175 |
4000 | 2777 |
5000 | 3432. |
6000 | 4040 |
7000 | 4649 |
8000 | 5321 |
9000 | 5892 |
10000 | 6479 |
100000 | 44487 |
200000 | 75530 |
300000 | 103794 |
400000 | 128660 |
500000 | 153265 |
600000 | 176817 |
700000 | 198450 |
800000 | 219060 |
900000 | 241250 |
1000000 | 261124 |
2000000 | 452906 |
3000000 | 615515 |
4000000 | 769028 |
5000000 | 912205 |
6000000 | 1035903 |
7000000 | 1151922 |
8140110 | 1284652 |
Table 1: Cumulative count of the number of tokens and the unique types in them. Tabulated from the text corpus of Malayalam Wikipedia articles extracted on 1st January, 2019 available from SMC Malayalam corpus
!
Figure 1: Type Token Growth Rate Analysis for Malayalam, plotted from Table 1, using the data from the text corpus of Malayalam Wikipedia articles extracted on 1st January, 2019 available from SMC Malayalam corpus
The Figure 1 on the Type Token Growth Rate indicates a steep increase initially. As the token count reaches 8 million, the unique type count is around 1.2 million.
This result is comparable with that of the Type-Token Growth rate reported in the paper, Statistical Analyses of Telugu Text Corpora, G.BharadwajaKumar et. al.
Figure 2: Type Token Growth Rate Analysis for different languages, reused from the paper Statistical Analyses of Telugu Text Corpora, G.BharadwajaKumar et. al.
The Type-Token Growth Rate for Malayalam is higher than Finnish, as can be seen in the Figure 3 below. The number of unique types for Finnish is ~0.7 million for 8 million tokens, while that for Malayalam it is 1.2 million types per 8 million tokens.
It may also be noted that the unique type counts get saturated for English, and the graph is quite a straight line, as the token count increases.
Figure 3: Type Token Growth Rate Analysis for different languages, reused from the paper Analysis of Morph-Based Speech Recognition and the Modeling of Out-of-Vocabulary Words Across Languages, Mathias Creutz et. al.
The ratio of the number of distinct words in a text corpus (Types) to the total word count (Tokens) is called Type Token Ratio (TTR). But it largely depends on the total length of the text being analysed. Kimmo Kettunen reports that, other methods of morphological complexity measurements involving linguistic analysis and correlates with the TTR based analysis of morphological complexity, in the study Can Type-Token Ratio be Used to
Show Morphological Complexity of Languages?
From the Malayalam Wikipedia dump from SMC text corpus, the following table of TTR is obtained.
Number of Tokens | Number of Types | Type Token Ratio (TTR) |
---|---|---|
1000 | 805 | 0.81 |
2000 | 1487 | 0.74 |
3000 | 2175 | 0.73 |
4000 | 2777 | 0.69 |
5000 | 3432 | 0.69 |
6000 | 4040 | 0.67 |
7000 | 4649 | 0.66 |
8000 | 5321 | 0.67 |
9000 | 5892 | 0.65 |
10000 | 6479 | 0.65 |
100000 | 44487 | 0.44 |
200000 | 75530 | 0.38 |
300000 | 103794 | 0.35 |
400000 | 128660 | 0.32 |
500000 | 153265 | 0.31 |
600000 | 176817 | 0.29 |
700000 | 198450 | 0.28 |
800000 | 219060 | 0.27 |
900000 | 241250 | 0.27 |
1000000 | 261124 | 0.26 |
2000000 | 452906 | 0.23 |
3000000 | 615515 | 0.21 |
4000000 | 769028 | 0.19 |
5000000 | 912205 | 0.18 |
6000000 | 1035903 | 0.17 |
7000000 | 1151922 | 0.16 |
8140110 | 1284652 | 0.16 |
Table 2: Cumulative count of the number of tokens and TTR for that token count. Tabulated from the text corpus of Malayalam Wikipedia articles extracted on 1st January, 2019 available from SMC Malayalam corpus, , containing about 8 million words.
Upon plotting the TTR details from the Table 2, results in the following Figure 4.
Figure 4: Type Token Ratio, plotted from Table 2. It uses the data from the text corpus of Malayalam Wikipedia articles extracted on 1st January, 2019 available from SMC Malayalam corpus, containing about 8 million words.
It shows the the TTR continuously decreases with increase in the number of tokens and takes a value at around 0.16, when the token count is 8 million. When the Token count is 2 million Malayalam words, the TTR is around 0.23. This is close to the value (~0.26 for a 2 million token count) reported in Statistical Analyses of Telugu Text Corpora, G. Bharadwaja Kumar et al.
The value of TTR obtained for Malayalam with that of the reported TTR values in other languages is shown below. The corpus size of languages under study were different. So the TTR of Malayalam is computed for the same text size for which data was available for the language under comparison. The corpus size in millions is indicated against each comparison graph pair.
Figure 5: Comparison of Malayalam TTR with that of DoE-CIIL corpus as reported by Bharadwaja Kumar et al.
A similar comparison for languages from the text of European Union constitution is shown below. The data available from Kettune's study is used to plot this graph. The corpus size in millions is indicated against each comparison graph pair.
Figure 6: Comparison of Malayalam TTR with that of European Union constitution corpus as reported by Kimmo Kettunen.
Malayalam clearly shows more morphological complexity than the European languages, Finnish, Estonian, Czech, Slovak, English and Spanish in terms of TTR values. Values of TTR obtained for Malayalam when compared with other Indian languages Marathi, Hindi, Tamil, Kannada and Telugu indicate a higher level of morphological complexity for Malayalam.
As seen above, TTR depends largely on the text size. So Michael A. Covington suggests using Moving-Average Type–Token Ratio(MATTR) as a measure of lexical complexity in his paper Cutting the Gordian Knot: The Moving Average Type-Token Ratio(MATTR).
For comparison across different experiments, he suggests calculating the TTR over a smoothly moving window of size 500. The code snippet used to extract the TTR over each window position is given below:
Token index | Number of Types | Type Token Ratio |
---|---|---|
1-500 | 425 | 0.85 |
2-501 | 426 | 0.852 |
3-502 | 426 | 0.852 |
4-503 | 427 | 0.854 |
5-504 | 427 | 0.854 |
6-505 | 427 | 0.854 |
7-506 | 428 | 0.856 |
8-507 | 429 | 0.858 |
9-508 | 429 | 0.858 |
10-509 | 430 | 0.86 |
11-510 | 430 | 0.86 |
12-511 | 430 | 0.86 |
.. | .. | .. |
871-1370 | 400 | 0.8 |
872-1371 | 399 | 0.798 |
873-1372 | 399 | 0.798 |
874-1373 | 399 | 0.798 |
.. | .. | .. |
1000-1500 | 396 | 0.792 |
MA-TTR | 0.834 |
Table 3: Moving Average Type Token Ratio tabulated over smoothly moving windows of token length 500, for the first 1000 window positions. Tabulated from the text corpus of Malayalam Wikipedia articles extracted on 1st January, 2019 available from SMC Malayalam corpus, containing about 8 million words.
{{< figure src="/img/malayalam-complexity-analysis/tsd1030d.png">}}
Figure 7: Type Token Ratio, plotted from Table 3. MATTR of different sections of the corpus is also indicated.
Taking the average over 1000 window positions result in an MATTR value of 0.834. The MATTR calculations of European languages by Kettunen on European Union constitution, indicates that this value is quite high. The highest reported value is for Finnish, which is 0.60. He has also noted the dependency of these values on the nature of the corpus. A more informal corpus will have more unique types in general.
Computing MATTR with 0.1 million tokens of SMC corpus resulted in a value 0.806 for Malayalam. Kettunen has reported MATTR values on European Union constitution corpus with each language having a token count slightly above 0.1 million. A comparative graph of the MATTR values reported by Kettunen with the values obtained for Malayalam is plotted in Figure 7. It clearly indicates a higher degree of morphological complexity for Malayalam in terms of MATTR on a formal text corpus. An equivalent comparison with other Indian languages could not be done due to non availability of reported studies.
Figure 7: MATTR comparison of Malayalam with other European languages as reported by Kettunen