Deep Neural networks have shown superior performance for learning complex abstract representations of data. These representations often correlate with our own understanding of the world. For example, a convolutional neural network learns about very high level concepts like cars, animals etc. It is even more interesting to note that it does it in a simple to complex heirarchical order. So the first layer learns simpler concepts such as edges, and then these edges are used as features to learn shapes, and so on.
It must be noted that this immense power of Deep Neural Networks can be extended beyond images. Deep learning is currently the state of the art in almost all tasks in natural language processing tasks such as sentiment analysis.
We propose to use a Deep Neural Network for learning complex user behaviour and interconnections by posing it as a collaborative filtering task. In this task, a large network is made to predict how likely a user is to score on a particular question, given his past history. Once our objective is defined, we can easily use some optimisation technique such as gradient descent or a combination of gradient descent and genetic algorithms.
If feature engineering is used, it will most likely introduce biases on the features that were chosen. This has been the state of the art since the past several years. We propose to remove this process entirely.
Instead of giving the inputs as features to our models, we give the raw data directly (after doing basic data whitening techniques to normalise the data). It is up to the model to come up with whatever features that are useful to get a good representation of the user behaviour.
The main objective of the network is to capture that patterns in the behaviour of users that describe their characteristics most vividly, while also keeping a relatively low dimensional space (representation vectors). Once we have these representation vectors , it is a trivial task to find the interconnected relations and patterns that exist in the data.
We tried this approach on 3 kinds of data to validate our theory. It must be noted that all of these approaches were under a semi supervised setting.
- Image data -- A very deep neural network based demonising autoencoder was used ( 60 layer encoder and 60 layer decoder). The model was able to describe extremely useful concepts like images of people playing sports, people dancing, etc.
- Word data -- A non negative sampling method was used to train the model on word data from the english dictionary. The model was able to describe concepts like technology (e.g. words like 'intel' and 'microsoft' were marked similar, but 'cat' and 'banana' were marked dissimilar)
- Behaviour data -- A deep learning based matrix factorisation technique was used to capture useful patterns in movies and user ratings.
In the previous sections, the examples were illustrated using user behaviour data. However, there is no restriction on the source of the behaviour.
The same algorithm can be applied to understand a company's hiring criteria. Once we get the representation vectors of the company, and since we already have Users' behaviour data, it is a simple task to find which company is most likely to hire a user. This is extremely useful for companies as they can reduce the number of testing rounds and thus save a lot of logistical workload.
Conversely, the same technique can be applied to find which user is most likely to be hired by a company. In this case, he can get a detailed analysis several months in advance of placement.
Colleges , hiring companies and users form an interconnected chain. Our representation vectors can be used in this 3 way interconnection.
Since we gather data from several colleges, we can get the general performance of the entire college in comparison to other colleges. This is useful for the college as it can analyse the strengths and weaknesses of its students as compared to other colleges.
It is also useful for hiring companies to understand in general how the students of the college performs and wether or not its worth hiring candidates from the college.
As discussed in the previous section, we can get details on how likely users are to get hired based on their performance and the hiring criteria of a company.
As a growing startup, it is very challenging to get colleges on board with us to gather their test data (we have a strict policy on being transparent about what data we collect).
Because we choose to remove human biases and go for a completely data driven approach, it is not possible for us to give a college instant results, and thus they are hesitant to share their data.
We request the government to help explain the advantages of a data driven approach and convince colleges to be willing to let their students take our exams.
A written recommendation is required as it will help alleviate the reluctancy.