Today's Progress: Uploaded the project to GitHub.
Thoughts: Image importing and prepararion was already done, as well as the transfer learning using 'Imagenet' on a ResNet50. Data Augmentation as also done. Added the model top and it seems to work but the results are bad. There is no improvment on the accuracy nor decrease in the loss. Having trouble uploading .json model file after saving it.
Link to work: Leukocyte classification
Today's Progress: I've solved the problem with the json file uploading. I'm trying to implement the model.evaluate or model.predict().
Thoughts: Did't manage to solve the test problem. Evaluate and predict functions ask for different input formats than those I'm getting from the test data. Still have to discover why the model is not training properly.
Today's Progress: No progress... I tried for almost two hours to solve the problem with the prediction and evaluation but I can't get it to work.
Thoughts: (In Keras) I'm using an ImageDataGenerator, then flow_from_directory, then model_predict_generator. I think I'll try to rework the project without the generator type functions....
Today's Progress: Still no progress...
Thoughts: Tomorrow I will search for another coding option... There is something about the output of the pre_trained network and the input of the model top that I'm not gettin right.
Today's Progress: I found this. I messed with it for a couple of hours.
Thoughts: The code seems to work with my data set. But I guess it is better to try it for some time and see if it really is doing what is supposed to do. I will add code to save the weights, training and validation arrays, and model. I still need to get the plots, a confusion matrix and get some test examples working.
Today's Progress: Got the plots working. Changed the code so I could see the labels atribuition using a build function with class_indices. Got the test part working with print of probabilities, results, precision and confusion matrix.
Thoughts: All seems to be working well. I need to find a way to save the history to a .csv or other file type to be able to generate a comparison plot of different types of architectures. I have to learn how to do the plot. Next step is to try the network with a data augmented dataset to see if it overfits.
Today's Progress: Tried VGG16, InceptionV3, ResNet50 and DenseNet121 with the small dataset.
Thoughts: Results look fine for most architectures. Tomorrow I will try running with the bigger datasets and see how it goes.
Today's Progress: Tried ResNet50 with the big dataset.
Thoughts: Only completed the ResNet50 implementation. Results were quite bad. There is some error with the NN that I couldn't solve.
Today's Progress: Tried VGG16, with big dataset.
Thoughts: The results weren't good, but the NN seems to be working properly.
Today's Progress: Tried InceptionV3 and DenseNet121 with the big dataset.
Thoughts: InceptionV3 is not working. I have to debug to see what is going wrong. I've yet to solve the ResNet50 problem. Actual results are: "Big" dataset (train=1000, validation=1600, test=2020) ResNet50 = 0.0 ... InceptionV3 = Error VGG16 = 0.76 DenseNet121 = 0.88
Small dataset (original images) (train=190, validation=43, test=31) Resnet50 = 0.0 ... InceptionV3 = 0.71 VGG16 = 0.55 DenseNet121 = 0.77
DenseNet121 seems to be going somewhere. I might try to fine tune it to see if it gets better. Before that I want to try DenseNet169, DenseNet201, Xception, and InceptionResNetV2.
Today's Progress: Learned a bit about Linux CLI to connect to my institute using VPN to run my ipynbs on their new 2 1080 gpu board.
Thoughts: First time connecting remotely (maybe second, if I count that bad trading bot experience...). Fairly new to this anyway. Never worked with Linux so I checked some begginer tutorials on youtube and learned some basic commands. I also got some youtube playlists on a notepad so I can progress by doing some courses on machine and deep learning, mainly held by siraj and sentdex.
Today's Progress: Eliminated the repetition of files with similar names (testing different architectures) by coding a way to test the different architectures and datasets in a single .ipynb.
Thoughts: As I have to use diffent input sizes, and get different output arrays, there's the need to define all that in a function in orther to reduce the amount of variables that need to be changed. Initially got the error "'int' object is not subscriptable" , but it was only the variable order definition when calling a model selection function.
Today's Progress: (~2h) Retried VGG16, ResNet50 and InceptionV3. So no real progress...
Thoughts: I tested the architectures on the new machine. I haven't used floydhub for a couple of days. At this point, after working for 3 or 4 months on this project I fell that I need to do something different. The last week didn't bring anything new. I feel a bit stuck in this 'try the same thing on different places' but I'm not really making the NNs work better. At this point I don't feel like I'm progressing. Although I seem to be a bit better at finding solutions for my problems... A small bit... The next step should be the selection of the architecture with best results with the leukocyte identification. But before that I still want to try 2 or 3 other architectures.
Today's Progress: (2h) Tried Xception, DenseNet169, Denset201, coded a way to save the test results to a .txt.
Thoughts: Tried all the architectures that I'll be comparing for my work. The .ipynb is working well apparently. I need to learn how to make plots that allow me to compare the accuracies and losses of the architectures, and another to visaulize wich one is getting the best performance with the default settings. From then I think I'll choose the best one, check if it is working perfectly and fine tune it to make my report.
Today's Progress: (2h) Generated the plots comparing 8 architectures
Thoughts: Although this isn't ML code, it is part of getting familiarized with Python. It has been hard to find time and will to keep working lately, but I force myself to sit in front of the compute after 10 pm to practice something every day, and without really notecing it 2 hours go by... This time I got this train accuracies and train losses. The test results were: DenseNet121 - 87.98% DenseNet169 - 82.57% DenseNet201 - 87.97% InceptionV3 - 82.82% ResNet50 - 25.25% (?) VGG16 - 77.87% VGG19 - 77.57% Xception - 85.30% There is some problem with ResNet50 that I haven't been able to solve...
Today's Progress:(2h) Started to work on the original file to test the architectures that I've used until now but without transfer learning
Thoughts: I thought I would be able to change a little bit the ipynb that I adapted to use the transfer learning on leukocyte identification but it seems it doesn't work. I got some problems with the "fit_generator".
Today's Progress:(1h) Started new file to test networks without transfer learning
Thoughts: I think I have to use a different approach to work without transfer learning. So from the errors I'm getting I think I need to get the data and labels separated from the beggining. Didn't finish as I have very little time today.
Today's Progress:(1.5h) Followed some machine learning videos by Sentdex on linear Regression.
Thoughts:I've been experiencing some problems with my laptop. There's something wrong with the battery and power adaptor, so I spent more than half an hour trying to get it working... When it started I tried to run the file without transfer learning and it was just painfully slow. In the mean while I watched some videos related with ML on Sentdex's great youtube channel.
Today's Progress:(1h)No real progress.
Thoughts: Trying to make test results on the trained VGG16 (without transfer learning) but without success until this point. Also, it has been difficult to get a little time to concentrate on ML.
Today's Progress:(1h) Started a new ipynb run some tests without transfer learning.
Thoughts: I was forced to restart because I can't adapt the code I used with transfer learning to work without t.l.. I am training the NN with VGG16. Have some problems to run the tests.
Today's Progress: Tested a lower value in learning rate(1h)
Thoughts: My teacher advised me to use a lower value of lr. I'm still looking for the best architecture to use with or without tranfer learning. I need to end this work but I would really like to get into something new. I seem to doing the same things over and over...
Today's Progress:(1h) Got the ipynb code working.
Thoughts: ... but the results are really bad. Now I'll see if there is some problem with the code and try different learning rates, and I might try Adam as an optimizer since I've been using RMSprop.
Today's Progress:(1h) Another restart.
Thoughts: Created another file from start... Not seing results....
Today's Progress:(1h) The code works on floyd and locally.
Thoughts:... but not remotely in my schools machine.
Today's Progress:(3h) Trained all the architectures for 100 epochs.
Thoughts:The results show overfitting for all the NNs. The datasets are small so training for 100 epochs is unjustified. The best results are aroud 85% accuracy with transfer learning. Still have to run all the architectures without transfer learning
Today's Progress:(1h) Started to learn pytorch.
Thoughts:It was really difficult to find one hour to practice today. I decided to take a break from my project and learn a bit of pytorch as I felt I was not learning anything lately. So I'll be revisiting the basics of ML and hopefully learn how to transfer what I did in keras to pytorch. Today I got how to code a linear model and the model loss.
Today's Progress:(1h) Gradient descent and Backpropagation in Pytorch. Data and Variable in Pytorch
Thoughts: Writting the code by hand while I try to understand it?Taking notes and analysing code.
Today's Progress:(1.5h) Forward pass and Pytorch rhythm
Thoughts: Working 9-5, playing some gigs at night (4 times this week) and having a 2 year old are making me feel a bit tired. Hopefully I'll keep on the challenge trouhg august, and surely september will be better.
Today's Progress:(1h) Logistic regression
Thoughts: Logistic regression pytorch example. "Wid and Deep" NNs. Diabetes classification example
Today's Progress:(1h) Data Loader in pytorch. Softmax
Thoughts:Data loader. Custom dataloader. MNIST dataset Loading. Softmax classifier
Today's Progress:(1h) Cross Entropy Loss. Convolutional Neural Networks
Thoughts:Basic and more advanced implemententions in pytorch. Analizing code....
Today's Progress:(2h) Inception Module. Recurrent Neural Networks
Thoughts:Copied some pytorch code while tried to understand it.
Today's Progress:(1h) Under the hood of RNN, LSTM and GRU. Name classification NN
Thoughts: Copied the code while tried to understand it.
Today's Progress:(2h) Full implementation of RNN
Thoughts:Copied the code while tried to understand it.
Today's Progress:(1.5h) Pytorch basics on pytorch.org
Thoughts: Spent the last week with the pytorch tutorials from this youtube channel. Copying the code by hand and analysing it. As I didn't find more tutorials (or maybe I should have searched longer) I headed to pytorch.org to study a bit more, while I copy some examples to jupyter notebook. Maybe I should start another project instead of studying the documentation but at this point this is what I seem to be able to do...
Today's Progress:(1h) Deep Learning with pytorch.org
Thoughts:I'm "coding along" with pytorch docs. Going through all of it. Don't know if it is a good idea. But as I've been terribly busy this month, I'm struggling to find a more creative aproach to learn ML. So I'll keep going trough the code for some more days until I have more time. I plan to go back to keras to complete my leukocyte identification project, that I intend to complete until the end of september, max...
Today's Progress:(1h) "Learning Pytorch with examples" on pytorch.org
Thoughts:"Coding along" and analysing the code.
Today's Progress:(2h) "Transfer Learning" on pytorch.org
Thoughts:"Coding along" and analysing the code.
Today's Progress:(45m) "Transfer Learning" on pytorch.org
Thoughts: This was the only free time I managed to find to practice a bit.
Today's Progress:(1h) "Data Loading and Processing" on pytorch.org
Thoughts: Messing around with the sugested ways to preprocess data. Going through a script that identifices parts of faces, while it shows how to resize and crop images.
Today's Progress:(1h) "Data Loading and Processing" on pytorch.org
Thoughts: Finished this part of the tutorial. Learned a bit about transforms, compose transforms and about torchvision in general.
Today's Progress:(2h) "Deep Learning for NLP with Pytorch" on pytorch.org
Thoughts: Another introduction to pytorch on the docs... Computation Graphs and Automatic Differentiation. Deep Learning building blocks: affine maps, non-linearities and objectives. softmax and probabilities
Today's Progress:(2h) "Word Embeddings: Encoding Lexical semantics". "LSTMs" on pytorch.org
Thoughts:Word embedding in pytorch. Computing word embeddings: Continuous Bag-of-Words.
Today's Progress:(2h) "Classifying Names with a Character-Level RNN" on pytorch.org
Thoughts Going through the basic RNN code.
Today's Progress:(1h) "Generating Names with a Character-Level RNN"
Thoughts: Going through the code but I'm having some trouble in understanding all of the concepts behind the code.
Today's Progress:(1.5h) " "Generating Names with a Character-Level RNN" on pytorch.org
Thoughts Continuing with pytorch.org. I'll code along until I end the intermediate part. Then, back to keras and leukocyte identification.
Today's Progress:(1.5h) "Translation with a Sequence to Sequence Network and Attention" on pytorch.org
Thoughts I understood the basic seq2seq concept, but the use of some type conversion functions is acomplex.
Today's Progress:(2h) "Translation with a Sequence to Sequence Network and Attention" on pytorch.org
Thoughts Finished the topic
Today's Progress:(1.5h) "Reinforcement Learning (DQN) Tutorial" on pytorch.org
Thoughts A lot of complex stuff on the pre-processing and input extraction parts.
Today's Progress:(1.5h) "Writing distributed Applications" on pytorch.org
Thoughts Labeled as Intermediate, for me this topic falls more on the advanced side, probobably beacause os the lack of CS background on my part.
Today's Progress:(1h) "Spacial Transformer Networks tutorial" on pytorch.org
Thoughts Half way through the challenge!! Although I didn't manage to make the register every day, I did code everyday for at least one hour (except for one day when I could only find 45 minutes).
Today's Progress:(1h) "Neural Transfer" on pytorch.org
Thoughts I have a lot to learn to be able to fully understand this advanced tutorials, but I'll keep going and go throught all of them and maybe later get back to more basic stuff to fill in some holes.
Today's Progress:(1h) "Creating extensions using numpy and scipy" on pytorch.org
Thoughts Complex...
Today's Progress:(1h) "Transfering a model form Pytorch to Caffe2 and Mobile using ONNX"
Thoughts Very complex...
Today's Progress:(1h) "Transfering a model form Pytorch to Caffe2 and Mobile using ONNX"
Thoughts Ended the tutorial but didn't get a lot of it. I'm not confortable with android.
Today's Progress:(1h) Deep Learning basics with Python, TensorFlow and Keras
Thoughts Reviewing some basics. Basic CNN and own data loading.
Today's Progress:(1h) Deep Learning with Python, TensorFlow, and Keras tutorial
Thoughts As I was without the computer this day the only thing I got to do was watch the new sentdex playlist, for about one hour.
Today's Progress:(1h) "Custom C++ and CUDA Extensions", on pytorch.org
Thoughts Finished the tutorials on pytorch.org. Some of the intermediate and advanced stuff is way above my current habilities, but I still got to understand pytorch a bit better. Hopefully I'll get to work on it at a decent level.
Today's Progress:(2h) Deep Neural Networks with PyTorch - Stefan Otte talk.
Thoughts Went through half the talk and transcribed the code. Learned a bit more about tensor capabilities and debugging.
Today's Progress:(1h) Random stuff...
Thoughts Just went to random DL stuff. Spent an hour in from of the computer but wasn't that productive. Kind of back to work depression...
Today's Progress:(4h) Reviewed the code of my work with leukocyte identification. Added to the dataset.
Thoughts I spent some time reviewing the code form thesis project related with leukocyte identification, but most of the time was spent searching for images on google and segmenting them to feed the NN. I got each of the 4 categories (lymphocyte, monocyte, eosinophil and Neutrophil) around 110-120 different images. Then generated more than 2500 for each category in the training folder. I will be working with 200 validation and 200 test images to see if the results are better than before.
Today's Progress:(4.5h) Deep Neural Networks with PyTorch (Stefan Otte talk). Leukocyte identification.
Thoughts Spent a lot of time reviewing the scripts that I ran about a month ago. There are two bugs that I haven't been able to solve. I also coded along Stefan Otte youtube talk.
Today's Progress:(5h) Study architectures and "history" of the Imagenet contest winners until 2015. Experimenting with keras and pytorch.
Thoughts Spent some timd writing about vgg, inception, resnet, lenet, fznet. Tried to debug my file for training NN from scratch, without success. Experimented a bit with pytorch trying to translate what I had done in keras.
Today's Progress:(5h) Continuing writing for my thesis. Reading paper about NNs. Experimenting with keras.
Thoughts Discover that there is something 'wrong' with the way the machine (that I'm running remotely) is running the .ipynb's. I get errors that I don't get on my laptop. It should be something related with the GPUs. I'm asking for some advice on that. I continued to write about NNs for my thesis...
Today's Progress: (2h) Writing about NN. Reading papers.
Thoughts 'Ended' the writting about ILSVRC competitors (until 2015). Tried to solve the GPU issue with my scrips without success.
Today's Progress: (2h) Batch normalization an data augmentation
Thoughts I read some papers on batch normalization and data augmentation and got some information for my thesis.
Today's Progress: (2h) "Automate the boring stuff with Python", Fine-tunning
Thoughts Spent 5h working on my thesis, but the DL related work was about 2 hours with some reading on fine-tunning and learning a bit more python.
Today's Progress: (2h) Started Move 37 course, "Automate the boring stuff with python"
Thoughts Started the "Move 37" course (Bellman Equation part.) and continued with the python course.
Today's Progress: (1.5h) Writting about datasets. "Automate the boring stuff with python"
Thoughts Though I've been working my thesis, I feel that I'm not dedicating as much time to ML as I should. On this day I spent some time getting better with python with "Automate the boring stuff..." youbube course (which is very nice by the way). I was also conditioned by the headache given by texniccenter... latex can be a pretty painful experience..
Today's Progress: (1h) Solved the problem with the remote machine.
Thoughts Me and my teacher got to solve (mostly him :)...) the problem with the remote machine. We tried some different batch sizes, chanded some parameters on the optimizer.
Today's Progress:(2.5h) Results to .csv. "Automate the boring stuff with python".
Thoughts Got to export the results of the NO_transferLearning .ipynb to .csv, and print the result (witch are bad by the way). practiced some more python with the "Automate the boring stuff..."
Today's Progress: (1h) "Automate the boring stuff with python". Experimented with optimizers.
Thoughts
Today's Progress: (1h) "Automate the boring stuff with python". Experimented with optimizers.
Thoughts
Today's Progress: (1h) "Automate the boring stuff with python".
Thoughts Worked a lot on my regular job. Came home late. Managed to practice python for one hour. I fell exausted ...
Today's Progress: (3h) Messing around with NNs.
Thoughts I am choosing 3h as a value, but I spent probably a lot more messing around with NNs. Debugging, testing different epochs numbers, different learning rates. Solved an issue with pytorch, although I worked most of the day with keras.
Today's Progress: (2h) Messing around with the dataset.
Thoughts I woudn't call this coding :D... But it is an important part of the process. I spent almost two hour going through every image in dataset. I deleted a bunch that seemed mislabeled, and will be doing the data augmentation tomorrow. Let's see if the results improve.
Today's Progress: (1h) Reviewed theory.
Thoughts Due to the lack of time, the only thing I managed to do was read my note books with code transcriptions from udemy and youtube courses.
Today's Progress: (1h) Messing around with the dataset (pt.n).
Thoughts Worked for some around 3h on some NN related notions on my thesis. Rediscovered some some stuff from Sentdex. Tremendously usefull.
Today's Progress: (3h) Adaped a custom network from a Sentdex example.
Thoughts Adapted a small network to work with the 4 categories on my leukocyte classification. Also tried Greyscale images instead of color. The results seem promising with this small network. I'll be trying with the pretrained layers later
Today's Progress: (1h) Messing around with the leukocyte ipynbs.
Thoughts No real progress.
Today's Progress: (1h) "Automate the boring stuff with python".
Thoughts One hour of python practice.
Today's Progress: (1h) Segmenting the leukocyte dataset using Paint3D. "Automate the boring stuff...."
Thoughts Spent 2h segmenting the dataset images with Paint3D wich allows to keep only the cell body. How I wish I knew how to automate this. Practiced python and worked on theory stuff for my thesis. Also got to lose some time and patience with LaTex...
Today's Progress: (1h) Segmenting the leukocyte dataset using Paint3D. "Automate the boring stuff...."
Thoughts Same as the day before...
Today's Progress: (1 h) Segmenting the leukocyte dataset using Paint3D. "Automate the boring stuff...."
Thoughts ... and editing the .tex thesis file.
Today's Progress: (1.5h) Segmenting the leukocyte dataset using Paint3D. "Automate the boring stuff...."
Thoughts Python practice.
Today's Progress: (1.5h) "Automate the boring stuff...."
Thoughts Worked on python and messed around with the jupyter notebooks
Today's Progress: (2h) Segmenting the leukocyte dataset using Paint3D. "Automate the boring stuff...."
Thoughts Learn some more cool stuff on python. I tried to run my notebooks using x2go but something is not working.
Today's Progress: (1h) "Automate the boring stuff....". Running the ipynbs to check if all is going as it should.
Thoughts Well... not mucth to think of...
Today's Progress: (3h) "Automate the boring stuff....", messing around with ipynbs.
Thoughts Spent the day messing around with the ipynbs and with python.
Today's Progress: (1.5h) "Automate the boring stuff....". Rerunning the NNs for 100 epochs.
Thoughts After segmenting all the images I am running the files again, and hopefully this will be the last time and I will be able to end the project and move on. I've been doing the "Automate the boring stuff" youtube videos, wich I find tremendously usefull, being selftaught.
Today's Progress: (1h) "Automate the boring stuff...."
Thoughts While I train the models I try to learn more about phthon.
Today's Progress: (2h) Callbacks
Thoughts The results I have been achieving show high accuracy and validation accuracy, and low losses overall, but the predictions made by the model are bad. Most of them are wrong, and I don't seem to be able to find a solution at this point. I also tried to implement callbacks (CSVLogger, ModelCheckpoint and EarlyStopping) but there is something about this that forces an error when ploting the model history.
Today's Progress: (1h) Trying some ideas with python and testing the NNs
Thoughts Again nothing that can identify as quantifiable progress, but in terms of the project I have to finish by october 31st I manage to twick some things on the process that make possible to obtain better results.
Today's Progress: (2h) Trying some ideas with python and testing the NNs
Thoughts Same as the last day but for a bit longer.
Today's Progress: (1.5h) Tested NNs. Messed a bit with keras.
Thoughts I was tired and without energy but ended up experimenting for almost 2 hours.
Today's Progress: (3h) Trained NNs and generated plots
Thoughts Spent most of my day working on my thesis. Running some ipynbs. Changing some small things related to plotting. Tested some plot ideas.
Today's Progress:(2h) Keras and python practice.
Thoughts I have been running the NNs, working on the theoretical side of my thesis and practicing python.
Today's Progress: (1h) Running NNs. "Automate the boring stuff with python".
Thoughts I have been running the NNs over my leukocyte dataset. I believe I'm finally getting some good results. Probably because of a second huge dose of preprocessing using Paint3D that allows to "eliminate" the background (all pixels to 255) in the images. while I train the models (without transfer learning) I practice python following the "automate the boring stuff..." YouTube video. Great tutorials.
Today's Progress: (2h) Working on my thesis ipynbs
Thoughts Same as before.
Today's Progress: (2h) Working on my thesis ipynbs
Thoughts Same as before.
Today's Progress: (2h) Working on my thesis ipynbs
Thoughts It is over :) The hardest part for most of the time was actually being motivated to update this gist. There were 3 or 4 days where I almost didn't get time to practice. One day I only did 45 minutes, but overall I got well over an hour each day, probably closer to two hours. Having a job and a family, with a son under three years old makes thing a little bit more complicated, but I'm happy that I finished the challenge and that it help me with my thesis. Also, I believe that I already gained the habit of programming (or learning to program) on a regular basis, so the good part of this is not feeling guilty if I decide to skip a day because I want to work on more theoretical stuff, or even to take a day off.