This blog post is the second post in the series "Clustering Text Documents". In the previous blog post (Clustering Text Documents: TF-IDF Weighting), we represented a given set of documents as a vector of tf-idf weights and in this blog post we'll calculate the cosine of angles between those document…
This blog post is the first post in the series "Clustering Text Documents". In this blog post, we'll mathematically define the TF-IDF algorithm along with an example and its python implementation. TF-IDF is a popular method used in text mining and information retrieval to evaluate the importance of…
One of the challenges faced by statistical part-of-speech taggers is the presence of words in test datasets that do not exist in the training dataset. Such words are called unknown words. In this blog post we'll look into different ways to tag unknown words. A statistical part-of-speech tagger…
Part-of-Speech tagging is a common sequence-tagging problem in natural language processing. It is the process of assigning a single word class label to each token in the input sentence. For example, for input: इराक सीमाबाट सेना हटाइने।, the output of the tagger is इराक-कुवेत/NN सीमा/NN बाट/II सेना/NN हटाइने/NN ।/YF.…
Source: Devanagari (Unicode block) Unicode is a standard for representing characters in different languages using four digit hexadecimal number called code points. Each character is associated with a unique code point. In python, these code points are represented as \uXXXX, where \u indicates Unicode and XXXX is the four…
Text analysis applications require frequent pattern matching and searching. For this reason, regular expressions play an important role in text analysis. Regular expressions are special sequence of characters that are useful for searching in texts. They can be used to split and modify given texts. Some common text processing…
Nepali, being a highly inflectional and derivational language, a single word can represent various grammatical forms and meanings. For example a verb root लेख्(lekh) can show different forms such as: लेख्छु(lekh-chu), लेख्छस्(lekh-chas), लेखछेस्(lekh-ches), लेख्छ(lekh-cha), लेखी(lekh-i), लेख्यो(lekh-yo), लेखे(lekh-e). Stemming is the process of reducing inflectional(or sometimes derivational) forms of words to…
Like nominal inflections, verbal inflections in Nepali occur due to suffixation. Verbs in Nepali are highly inflected. Inflectional suffixes operate on verb roots to represent different grammatical categories of verb. In this blog post, we'll discuss verbal inflections in Nepali. Different grammatical categories encoded by verbal inflections(suffixes) in…
Figure 1: Word Formation in Nepali The process of word formations(शब्दनिर्माण / शब्दरचना) in Nepali can be categorized into: inflection(रुपायन) and derivation(व्युत्पादन). Types of word formation process in Nepali are given in Figure 1. In this blog post, we'll discuss nominal(noun, pronoun and adjective) inflections in Nepali. …
The increasing amount of information being shared over the web makes it a huge source of data/information. To extract this data for analysis you need web scrapping. It is a popular technique to get data from web page in whatever format suitable for your analysis. Also, make sure that you…
