(Based on explanation by Dr. Jacob Høxbroe Jeppesen)
Machine learning is the field of research and applications that has branched off from artificial intelligence in 1980s. Now, machine learning researchers try to develop engineering techniques that can help applications learn from existing data to perform better.
Since, machine learning is performing approximation, we can call it a higher dimensional regression technique based on statistics.
The goal of machine learning is to generate approximator functions from a given input and output data set. The approximator functions can be of the following types.
- principal component analysis (PCA)
- neural networks
- convolutional neural networks (CNN)
- fuzzy logic
- no longer popular in the field of machine learning.
The PCA tries to create features from input data set. The CNN can work with the labeled (known inputs and outputs) data sets to create approximator functions.
In the CNN, the approximator functions are the weights of cross-edges in a stage of the neural network. During the training phase, the weights are adjusted (called back propagation of error) by matching the generated outputs with the actual outputs.
Autoencoder is a popular kind of CNN. Autoencoder has both the encoder and decoder in it. The encoder receives input and generates the latent features as output. The decoder receives the latent features as input and generates approximated output. In the training phase, both the parts of the autoencoder (encoder + decoder) are trained together. In the usage phase, the encoder and decoder can be kept separately. This separation can be a great benefit to remote communication because only encoded latent features need to be transmitted to the remote decoder.
Each stage of a multi-stage convolutional neural network consists of 3x3 dimensional neural network. Outputs from neurons of 1-dimension are connected to all the inputs of the second dimension and so on. Each connection from one neuron to another has a constant weight. Each neuron acts as a summation of all the inputs it receives. As a part of training stage, the weights of the network are adjusted to produce outputs that closely match the input.
The engineering heuristics in a typical neural network:
- dimensions, typically 3x3 (arrived at based on practical experience)
- number of stages
- neurons (simple summers)
- weights of connections (real numbers)
There has also been some research in the kinds of neurons we can use. Aarhus University researchers are working on different kinds of neurons and their broad conclusion is that it is possible to reduce the size of CNN by using a more complex neuron.
There have been some research on the way to set the weights of connections in a neural network. The possible choices are:
- random (best worst case scenario)
- copy from another trained neural network used in the same domain
- copy from another trained neural network used in adjacent domain
Researchers constantly look at the output of an autoencoder to look for performance gains. One of the ways of inducing performance gains is to use expert judgement to remove the redundant data from the input through past experience or by looking at the generated output of the autoencoder. Another way is to back propagate the useful errors and discard the useless errors. That way, the training of the autoencoder happens for approximated features of interest.
Tensorflow is the most popular implementation platform for the CNNs. From a software engineering standpoint, Tensorflow is a distributed numerical computing platform that can work well with the GPUs.
As a side note, Google is the number one AI company in the world and they are already number one. Google wants to provide machine learning services over API. Google also has a very sophisticated Deep Mind project which tries to mimic the human brain. The interesting thing is that all the knowlege and training of a neural network stays in the company and it is a pure service. The knowlege is only going grow as a competitive advantage unless the models / trained network is given out to open source.