Importance of Activation Functions

The purpose of activation functions is to introduce non-linearities into the network. Non-linear function allows to approximate arbitrarily complex functions that make neural network extremely powerful.
For example, when a trained network with weights W and the network has only two inputs- x1 ,  x2  and we pass it through a non-linearity.
Before applying non-linearity, If we feed in with a new input x1= -1 and x2= 2, the idea can be generalized a little bit more if we compute the line, we get minus -6.
When we apply a sigmoid non-linearity, it collapses between 0 and 1. Sigmoid function results anything greater than 1 as above 0.5 and anything less than 1 as below 0.5.

The reason why we use non-linear activation function is when deal with network thousands or millions of parameters and dimensional spaces then visualizing these type of plots becomes extremely difficult.

Comments

Post a Comment

Popular posts from this blog

Introduction to Deep Learning

Image
Deep learning is an incredibly powerful tool   of Machine Learning that is based on learning data representations instead of task-specific algorithms. Deep Learning uses networks where data transforms through a number of layers before producing the output. Traditional machine learning algorithms typically try to define as the set of rules or features in the data and these are usually hand-engineered that’s why it tends to be brittle in practice. For example, if you want to perform facial detection, the first thing you have to do is classify or recognize mouth, eyes, ears and all in the image, if you find everything then you can say there is a face in the image. To recognize each thing you have to define a set of features. The key idea of deep learning is that you will need to learn the features just from raw data, you just have to take a bunch of images of faces and then the deep learning algorithm is going to develop some hierarchical representation of first dete...
Read more

The idea of Neural Network

Image
The basic idea behind a neural network is to simulate lots of densely interconnected brain cells inside a computer so you can get it to learn things, recognize patterns, and make decisions in a human-like way.   The amazing thing about a neural network is that you don't have to program it to learn explicitly: it learns all by itself, just like a brain! It's important to note that neural networks are (generally) software simulations: they're made by programming very ordinary computers, working in a very traditional fashion with their ordinary transistors and serially connected logic gates, to behave as though they're built from billions of highly interconnected brain cells working in parallel. No-one has yet attempted to build a computer by wiring up transistors in a densely parallel structure exactly like the human brain. Building Dense Layer from scratch: Single Layer Neural Network: A single hidden layer is fed into a single output layer. The stat...
Read more