NLP - Preprocessing of Text

As a data scientist, we may use NLP for sentiment analysis (classifying words to have positive or negative connotation) or to make predictions in classification models, among other things. Typically, whether we’re given the data or have to scrape it, the text will be in its natural human format of sentences, paragraphs, tweets, etc. From there, before we can dig into analyzing, we will have to do some cleaning to break the text down into a format the computer can easily understand.

These steps are needed for transferring text from human language to machine-readable format for further processing. We will also discuss text preprocessing tools.After a text is obtained, we start with text normalization. Text normalization includes:
 
- Tokenization
- Remove stop words
- Remove sparse terms and particular words
- Stemming


Tokenization: It is the process of breaking up the original raw text into component pieces or otherwise known as tokens.
 
Remove stop words:

“Stop words” are the most common words in a language like “the”, “a”, “on”, “is”, “all”. These words do not carry important meaning and are usually removed from texts. It is possible to remove stop words using Natural Language Toolkit (NLTK), a suite of libraries and programs for symbolic and statistical natural language processing.

A scikit-learn tool also provides a stop words list:

from sklearn.feature_extraction.stop_words import ENGLISH_STOP_WORDS

It’s also possible to use spaCy, a free open-source library:

from spacy.lang.en.stop_words import STOP_WORDS


Remove sparse terms and particular words:

In some cases, it’s necessary to remove sparse terms or particular words from texts. This task can be done using stop words removal techniques considering that any group of words can be chosen as the stop words.

Stemming:

One of the most common and effective stemming tools is Porter's algorithm developed by Martin Porter
in 1980. The algorithm employs five phases of word reduction, each with its own set of mapping rules.
  • In the first phase simple suffixed mapping rules are defined as:
       From a given set of stemming rules only one rule is applied, based on the longest suffix S1. Thus,    caresses reduces to caress but not to cares.
  • More sophisticated phrases that consider the length and complexity of the word before applying a rule. For example:
 
Spacy didn't have stemming because it figures lemmatization is a much more effective
way of actually reducing these words.

Lemmatization:

In contrast to stemming, lemmatization looks beyond word reduction, and considers a language's full vocabulary to apply a morphological analysis to words. The lemma of 'was' is 'be'  and the lemma of 'mice' is 'mouse'.

Lemmatization is typically seen as much more informative than simple stemming which is why the Spacy library has opted to only have lemmatization available instead of simple stemming.

Lemmatization looks at surrounding text to determine a given word's part of speech, it does not categorize phrases.
 
For more understanding, let try some code given on my github link:

Comments

Post a Comment

Popular posts from this blog

Importance of Activation Functions

Image
The purpose of activation functions is to introduce non-linearities into the network. Non-linear function allows to approximate arbitrarily complex functions that make neural network extremely powerful. For example, when a trained network with weights W and the network has only two inputs- x 1  ,  x 2  and we pass it through a non-linearity. Before applying non-linearity, If we feed in with a new input x 1 = -1 and x 2 = 2, the idea can be generalized a little bit more if we compute the line, we get minus -6. When we apply a sigmoid non-linearity, it collapses between 0 and 1. Sigmoid function results anything greater than 1 as above 0.5 and anything less than 1 as below 0.5. The reason why we use non-linear activation function is when deal with network thousands or millions of parameters and dimensional spaces then visualizing these type of plots becomes extremely difficult.
Read more

Introduction to Deep Learning

Image
Deep learning is an incredibly powerful tool   of Machine Learning that is based on learning data representations instead of task-specific algorithms. Deep Learning uses networks where data transforms through a number of layers before producing the output. Traditional machine learning algorithms typically try to define as the set of rules or features in the data and these are usually hand-engineered that’s why it tends to be brittle in practice. For example, if you want to perform facial detection, the first thing you have to do is classify or recognize mouth, eyes, ears and all in the image, if you find everything then you can say there is a face in the image. To recognize each thing you have to define a set of features. The key idea of deep learning is that you will need to learn the features just from raw data, you just have to take a bunch of images of faces and then the deep learning algorithm is going to develop some hierarchical representation of first dete...
Read more

The idea of Neural Network

Image
The basic idea behind a neural network is to simulate lots of densely interconnected brain cells inside a computer so you can get it to learn things, recognize patterns, and make decisions in a human-like way.   The amazing thing about a neural network is that you don't have to program it to learn explicitly: it learns all by itself, just like a brain! It's important to note that neural networks are (generally) software simulations: they're made by programming very ordinary computers, working in a very traditional fashion with their ordinary transistors and serially connected logic gates, to behave as though they're built from billions of highly interconnected brain cells working in parallel. No-one has yet attempted to build a computer by wiring up transistors in a densely parallel structure exactly like the human brain. Building Dense Layer from scratch: Single Layer Neural Network: A single hidden layer is fed into a single output layer. The stat...
Read more