Neural Network – Back Propagation – Tutorial in Python

Introduction:

Neural Network is a concept which tries to mimic human brain to solve computational issues. Neural network consists of basic units called Neurons. Each neuron has some functions and they together with other neurons solve complex issues. Neural network as a concept has been around since 1960’s (maybe before that too). But it lost its importance in 1990’s. Due to recent advancements in computational power and some algorithmic breakthrough’s, it has started to play an important role for many complex machine learning problems.

Brief overview:

Neuron : Each Neuron can have one or more inputs and one or more outputs. Each connection between other neurons with some weights associated.

Feed forward : The input is processed through one  or many layers of neurons to get the output.

Back Propagation : The Error which is the difference between the predicted output and the original target output, is propagated from the output to the input in a reverse manner, so that the weights of the neurons adjust itself to the error.

Python Code:

Feed Forward

Back Propagation

Full code :

Learning materials :

Important Concepts of Machine Learning

In the following days, I am going to write some tutorials and discuss some major concepts of machine learning.

Starting from

  1. SVM — (The most famous Support Vector Machines which was widely used/acknowledged till recently)
  2. Random Forest — (Widely used algorithm in kaggle competitions recently. Also used in Microsoft Kinect)
  3. Neural Networks, Back Propagation — (Important to understand the concept of Deep learning)
  4. RBM — (Building blocks of Deep Learning)
  5. Auto Decoders (Used in Google net)

There are many software packages available for SVM, Random Forest and are fairly simple to use. But the difficult part in using those supervised algorithms (SVM, Random Forest) is selected good features( Feature Engineering). We need to choose which features (characteristics) works better in predicting the output and also pre-process(mean, scale) it. Even-though through random forest  we can find which features are important and which ones are not, this is not enough. A major breakthrough in this area has happened in Deep Learning Neural Networks where the system itself finds the features that are useful and uses it for learning. This deep learning is playing a major role in today’s speech processing, image processing technology.

Alarm – One Click

Normal Alarm clock takes upto 3 to 5 clicks to create an alarm. So I thought of a simple way to set/unset alarm in 1 click. It took me 3 hours to develop this simple app which is also compatible with Android 5.0 – lollipop. We can have 3 different settings – All week, Only weekday, weekend.

one click activity screen

Alarm – One click

AlarmClock API : One limitation that I had with Native Alarm Clock api of Android is there is no way to cancel an alarm. So we have to use AlarmManager instead of AlarmClock to set/unset an alarm.

Github link for the code

Android Speech Processing API Review

Recently I got a chance to work with Android Speech Processing API. Our team developed an app called Meeting1 which will take meeting minutes.

If the app is turned on,

  1. The app listens to the meeting and converts the speech into text
  2. Then the text is sent to the server where it is consolidated into minutes at the end of the meeting
  3. In a multi-user mode, Many devices are used in parallel to process the speech and the server will eliminate duplicate context while finally giving out the meeting minutes to the orgainzer
  4. Meeting minutes can be shared by the organizer afterwards by email or salesforce technologies (This app was developed for the DreamForce Hackathon competition)

Android Speech Processing API :

I would say that Android Speech Processing API is at early stage. Below is my review of the api

  • Speed of speech : It did a poor job of converting speech under normal speed. But even if you slow your speech by 20%, it improved the efficiency a lot
  • Context : Again if the context is about ‘YOU’ or a ‘LOCATION’ or any other things that you can use in google now like weather, sports, tv shows. It did a better job. But it did a terrible job if the context is changed to a particular topic like ‘Android’, ‘America’, ‘Washing Machine’ or even for reading excerpts from any book
  • Length of Speech : It can determine a a maximum of 12 – 18 words in a single attempt. In our case, we have to repeatedly call the api in a loop to make continuous speech processing
  • Noisy Background : As expected noisy background led to a poor job
  • Accent : Surprisingly, android does better for different accents. (Google has done a great job of training it for a wide variety of accent)

Given all this limitations which will/can be improved, the speech processing api can be used as a COMMAND to operate your app like to get Driving directions, find the best book, filtering or sorting of results. (Since this field is rapidly improving, you can expect major progress sooner)

To counter the limitations, we have implemented the following techniques in our app.

  • Continuous loop of the api to get more text converted at a go
  • Context : Allow the user to correct things and learn from it locally in the server. Use multiple devices for simulatenous conversions of speech (I am not sure if this would really make a difference in the long run but it did for us)

Finally, if you want to take a look at the code of our app, you can use https://github.com/anooshm/meeting1_android

Python for Machine Learning and Data Engineering

Read this if you are new to Machine learning.

In this post, I want to summarize the useful tutorials to learn machine learning using python. Python provides a comprehensive set of tools for ML. iPython Notebook is one of the best tools that I have come across in recent times.

Tools/Packages:

  • Pandas — Importing data and for preprocessing
  • Numpy — Fast, Efficient Matrix manipulations
  • Sklearn — Machine learning toolkit.
  • iPython — Interactive environment to code.

These 2 videos will help you kickstart on iPython and Sklearn

Exploring Machine Learning with Scikit-learn

Diving Deeper into Machine Learning with Scikit-learn

If you are new to python, I would recommend the python course on Udacity

In the meantime, Kaggle also provides good tutorial set to try out python

http://blog.kaggle.com/2013/01/17/getting-started-with-pandas-predicting-sat-scores-for-new-york-city-schools/

http://blog.kaggle.com/2012/07/02/up-and-running-with-python-my-first-kaggle-entry/

If you find any other useful tutorials, please add it to the comment

Beginner’s useful resource for Machine Learning / Data Science

I am new to Machine Learning / Data Science. As I searched for more resources to learn ML – the concept, tools, I found these interesting resources.

Complete Machine learning overview : Coursera Machine Learning by Andrew Ng. This is the best place to start learning it from the scratch

Complete Data Science : Data Science from Harvard .

As the name states, the difference between the 2 courses is that the first course is more oriented towards different algorithms for learning from the data, the second course is more oriented towards data collection, data pre-processing and understanding the data.

Tools:

There are a variety of tools/languages that can be used. I have tried Octave, R, Weka, Python, Matlab, Julia. R is one of the widely used in the industry each has its own advantages/disadvantages. Finally I chose to use python since I am more comfortable to python language. (I have tried Octave, R, Weka, Python).

http://www.experfy.com/blog/python-data-science/

http://programmers.stackexchange.com/questions/181342/r-vs-python-for-data-analysis

https://www.kaggle.com/forums/t/5243/pros-and-cons-of-r-vs-python-sci-kit-learn

Kaggle :

Kaggle is an active community for machine learning. you get lot of data sets with which you learn as well as compete with other kagglers. Start trying out the competitions that are posted in there