Project Abstract | Milestones | Pull Requests & Issues | Updates | Statistics | Links
The Poverty Probability Index (PPI) is a poverty measurement tool for organizations and businesses with a mission to serve the poor. A PPI survey consists of 10 questions about a household’s characteristics using which asset ownership is scored to compute the likelihood that the household is living below the poverty line.
Leveraging Cloud Vision, a field officer would simply have to take a series of photos with their smartphone camera inside and outside of the house and then the Cloud Vision would be able to deduce based on the images the response to the 10 questions by detecting whether the objects mentioned in the survey are present in the household or not.
This year the project aims to do the following:
- Increase the accuracy of the models for object detection
- Train more models using the Google Cloud Vision platform to recognize more objects
- Improve the current augmentation procedure to include more techniques and hence build a better dataset
- Integrate the AutoML API with the android client
- Implement auto-filling of the survey based on the results received from the AutoML API when it detects objects in the images uploaded
- Enhance the UI/UX of the android client
Mentors: @Nayan, @rahulg963
There were a few changes in the project plans once we actually got to working on it. The main change was that instead of using Google Cloud Vision and the AutoML API, we instead decided to go ahead with the MLKit SDK released by Google which is completely offline and doesn't not require any internet connection.
The cons of using Cloud Vision were:
❌ Google Cloud Vision is paid and would cost more than a 1000$ per month just to keep the model deployed
❌ The initial training phase would also cost quite a bit of money
❌ There would be some latency in getting results back since data has to be uploaded, analysed and returned. The speed of this process would entirely depend on the connection speed of the user
The pros of going ahead with MLKit:
✔️ No payment needed at all
✔️ No latency as network connection isn’t needed
✔️ No cost in training the model or re-training it
✔️ Models can be trained using transfer learning with TensorFlow and then be converted into tflite models
Given these factors, we dropped the plan of using Google Cloud Vision and AutoML and instead decided to use MLKit.
1) Data Augmentation
The first step of the project was to improve the data augmentation techniques. Since the it is difficult to find a dataset containing photos of the items needed in a household setting, it is important to increase the size of the dataset as much as possible by using augmentation techniques.
I created a new python script which takes in a directory of images as the input and augments all data and stores the output in another directory. I also created a Jupyter Notebook which can be used to visualize all the augmentation techinques on a single image without running the python script.
The augmentation techniques I added were:
- Cropping
- Histogram Equalization
- Translation and Padding
- A variety of blurring techniques
- Saturation and Hue
- Salt and pepper noise
- Adaptive Gaussian thresholding
- Affine transformation
- Parallel Light Technique
- Spot Light Technique
These techniques can be viewed here: https://github.com/openMF/ppi-vision/blob/master/DataCollection/ImageAugmentation.ipynb, done by #115.
The data augmentation script can be found here: https://github.com/openMF/ppi-vision/blob/master/DataCollection/GenerateImages.py, done by #113.
2) Image Labelling using MLKit and Autofilling the survey
The next step was to do the main image labelling part on an android device for which I have used MLKit. A user select multiple photos from their device, or click photos directly using a camera and use MLKit to detect all objects within the photo taken. A gif of this can be seen below:
For people intersted in knowing how object classification(the basis of image labelling works, I have also prepared a sample Jupyter Notebook that demonstrates the process over here: https://github.com/openMF/ppi-vision/blob/master/ImageClassification/ImageClassificationExample.ipynb.
The work for this part is split amongst the PRs #148, #151 and #153.
Once the objects within an image/images are detected, I used the data obtained to provide a user with suggestions as to which survery option would be most suitable. The system can detect whether a house has katcha or pakka outer walls, whether there is a television, a refridgerator, VCR, the number of telephones, etc. A screenshot of how this looks can be seen below:
 |
 |
 |
|---|
3) Improving the UI/UX of the android app
The last major part of the project was to improve the UI/UX of the application. Since the app only needs to have 2 major screens, I shifted the layout from a navigation drawer based layout to a bottom navigation bar. Besides, I improved the display of the home search fragment, the user and clinet profiles, the image analysis section and the question display fragment.
 |
 |
 |
|---|---|---|
| Login Screen | Bottom Navigation + Enhanced user profile page | Bottom Navigation + Enhanced Search Fragment |
 |
 |
|
| Client Profile Page | Image Analysis Activity | Questions Fragment |
4) Documentation
Earlier, there was hardly any documentation for the project. During my GSoC work, I also worked on documenting the application. I have done the following work on documenting the project:
- I created the project wiki: https://github.com/openMF/ppi-vision/wiki
- Created a proper readme to describe the project so anyone can see what the project is about: https://github.com/openMF/ppi-vision/blob/master/README.md
- A detailed explanation of the project on the Mifos Confluence: https://mifosforge.jira.com/wiki/spaces/MIFOSX/pages/977305715/Computer+Vision+Based+PPI+Tool
 |
 |
|---|---|
 |
 |
- Adding a Code of Conduct to the project: https://github.com/openMF/ppi-vision/blob/master/CODE_OF_CONDUCT.md
- Adding contribution guidelines to help new contributors get started with the project easily: https://github.com/openMF/ppi-vision/blob/master/CONTRIBUTING.md
- Add issue templates to the project for bug fixes, feature requests and chores: https://github.com/openMF/ppi-vision/tree/master/.github/ISSUE_TEMPLATE
 |
 |
|---|---|
| Options that show up on creating an issue | Bug Report Template |
 |
 |
| Chore Template | Feature Request Template |
- Added a pull request template so contributors can ensure they followed some guidelines and reducing the chance of a build failure: https://github.com/openMF/ppi-vision/blob/master/.github/PULL_REQUEST_TEMPLATE.md
 |
 |
|---|---|
| PR Template in markdown | PR Template when filled after contributor satisfies the conditions |
5) Continuous Integration(CI)
There was no CI in the project earlier. I added CI using GitHub actions for both the python scripts as well as the android app. Now if any contributor sends a pull request, the changes can be built using GitHub actions and if any code is broken, the reviewers can see it right there.
 |
 |
 |
|---|---|---|
| Android Workflow | Python Workflow | GitHub Actions Results |
 |
 |
 |
| Python CI Results | Android CI Results | Examples of a workflow failing on a PR |
6) Other miscellaneous work done
- Helped review merge all existing prs in the project as well as fix some minor bugs.
- Refactored the login of the application using Retrofit and RXJava such that it doesn't have a dependency on non-maintained libraries like khttp.
- Added the feature to save the credentails once a user logs in and also a feature to log out. Earlier no credentials were saved and a user had to log in everytime the app was opened.
- Reformat code properly using the spotless plugin.
- Follow proper code practices by using theindexing operator, property-access syntax, etc where ever possible which had not been done before.
- Added an about section to the application so users can see the developers, the GitHub repo, license and social media pages of Mifos.
7) Link to APK
- Link to APK: https://drive.google.com/file/d/1sZkow9zy7Hs6Bpc2GCsrpO6smVVQ6xH_/view
- Link to live preview on appetize.io: https://appetize.io/app/9m1cruam36xd6k205p7qf4b3d4?device=nexus5&scale=75&orientation=portrait&osVersion=8.1
Commits/PRs Made
- #196: Show dialog to user when logging out
feat - #194: Change profile icon
fix - #192: Change color of options menu
fix - #190: Improve code quality
chore - #188: Rename variables to follow the general convention
chore - #187: Make variables and methods private where ever needed
chore - #186: Add ripple effect to buttons
feat - #185: Add action bar to computer vision activity
fix - #183: Show user a message if analyse images is pressed without any images selected
fix - #182: Fix crash in clicking on a client
fix - #181: Update README.md
chore - #179: Add issue annd PR templates
chore - #177: Add CONTRIBUTING.md
chore - #175: Add a code of conduct to the project
chore - #173: Add about screen
feat - #172: Codestyle fix using spotless
chore - #171: Improve UI of computer vision activity
feat - #168: Update UI of question fragment
feat - #166: Enhance UI of user and client profile sections
feat - #165: Enhance UI of home page
feat - #162: Shift to a bottom navigation layout
feat - #157: Add feature to autofill the survey
feat - #153: Remove google services dependencies
chore - #151: Enhance object detection system
feat - #148: Add MLKit Object Detection
feat - #145: Add toolbar for ComputerVision activity
fix - #137: Add an example image classification notebook
feat - #135: Clean up app and remove unused resources
chore - #133: Add option to logout
feat - #131: Refactor login
feat - #125: Add hindi translation
feat - #117: CI for python scripts
feat - 89ad68e: Add CI using GitHub Actions
feat - #115: Enhance the augmentation notebook
feat - #113: Create python script for augmentation
feat - #110: Fix the needed lint issues
chore - #107: Add a new dependency for http requests
feat - #105: Fix method of obtaining the authKey
fix - #103: Update app dependencies and make other needed changes
chore - #101: Migrate app dependencies to androidX
chore - #99: Update gradle and kotlin version
chore - #98: Update API endpoint
chore - #97: Add parallel and spot light augmentation techniques
feat - #96: Add salt and pepper noise, adaptive Gaussian thresholding and affine transformation
feat - #95: Add saturation and hue techniques to augment images
feat - #94: Fix unchecked nullability
chore - #93: Fix idiomatic errors
chore - #92: Disable search button when search query is empty
fix - #91: Changed responsiveness of Login button based on emptiness of text fields
fix - #90: Added augmentation techniques for blurring
feat - #88: Enhance UI of client profile page
feat - #87: Enhance UI of search activity
feat - #86: Set tool bar title for different activities
feat - #85: Status bar color
fix - #81: Enhance user profile page
feat - #80: Add translation and padding for augmentation
feat - #79: Add option to share the app easily
feat - #76: Clicking on profile opens profile page
feat - #75: Keyboard hidden when nav drawer is opened
fix - #72: Improve UI of search activity
feat - #71: Added internet connectivity check in the login screen
feat - #70: Added check for internet connectivity in search button
feat - #66: Keyboard goes down on pressing search if query is suitable
fix - #65: Add histogram equalization techiques for augmentation
feat - #64: Deleted activity_user_profile2
chore - #31: Add bluring techniques for image augmentation
feat - #28: Add cropping technique for image augmentation
feat - #26: Fix typo in password length toast
fix - #24: Make username a single-line text field, press next on keyboard to directly enter password field
fix - #22: Add toggle for password visibility
feat
Issues
- #195: Display a dialog to user when logging out
feat - #193: Change profile icon in bottom nav bar
fix - #191: Color of options in action bar
fix - #189: Improve code quality
chore - #184: Add an action bar in computer vision activity
fix - #180: Update README.md
chore - #178: Add issue and PR templates to the project
chore - #176: Add a contributions.md to the project
chore - #174: Add a code of conduct for the project
chore - #170: Improve UI of computer vision activity
feat - #169: Enhance UI of the question fragment
feat - #167: Revamp UI of user and client profile fragments
feat - #164: Enhance UI of home page
feat - #163: Shift the app to bottom navigation layout
feat - #152: Remove dependency on google services
chore - #150: Enhance the object detection system
feat - #147: Auto-fill survey
feat - #146: Add MLKit for object detection
feat - #144: Toolbar in NewPPISurveyActivity not visible
fix - #136: Add an example notebook for image classification
feat - #134: Remove unused resources from the app
chore - #132: Add an option for logging out
feat - #130: Refactor the login of the application
feat - #124: Add hindi translation
feat - #116: Setup CI for the python script
feat - #114: Modify the image augmentation notebook
feat - #112: Create a python script for doing image augmentation
feat - #109: Fix lint issues in the application
chore - #108: Add GitHub Actions workflow for CI
feat - #106: Replace the current khttp dependency
feat - #104: Fix the method of obtaining the authKey
fix - #102: Update app dependencies
chore - #100: Migrate app dependencies to androidX
chore - #84: Improve UI of the search page
feat - #83: Set title for activities
feat - #82: Set a color for the app status bar
fix - #69: Improve search option edit text
feat - #63: Add an option to share the app
feat - #54: Add a notification of no internet if internet connection is missing when user presses search
feat - #53: Keyboard should go down when the search button is pressed
fix - #52: Codestyle fixing
chore - #51: About Screen for the app
feat - #50: Fix naming convention for variables
chore - #49: Make variables private
chore - #47: No check for internet connection in login screen
feat - #45: Keyboard does not go down when the nav drawer is opened
fix - #44: Clicking on the users photo in nav drawer should also open the profile page
feat - #43: Add translation and padding augmentation techniques
feat - #38: Enhance UI of user profile page
feat - #36: Enhance UI of client profile page
feat - #35: Delete unused activity_user_profile2.xml
chore - #34: Add histogram equalization techniques for image augmentation
feat - #30: Apply blurring techniques for augmentation
feat - #27: Enhance the cropping technique for data augmentation
feat - #25: Typo in toast message
fix - #23: Multi-line username field
fix
Tags:
- feature :
feat- bug fix :
fix- chore related to maintainance, documentation, etc -
chore
- Meeting 1: Notes
- Blog Post: GSoC'20: Período de Unión Comunitaria
- Meeting 2: Notes
- Meeting 3: Notes
- Meeting 4: Notes
- Meeting 5: Notes
- Meeting 6: Notes
- Meeting 7: Notes
- Blog Post: GSoC'20: Período de codificación uno
- Meeting 8: Notes
- Meeting 9: Notes
- Blog Post: GSoC'20: Período de codificación dos
- Blog Post: GSoC'20: Período de codificación tres
- Blog Post: GSoC'20: Resumen del trabajo realizado
- Total PRs sent: 69
- Total issues opened: 57
- Total commits: 78
- Total lines of code added: 15,269
- Total lines of code deleted: 6,775
It was a great experince working with the Mifos Initiative for GSoC'20. Thanks to Ed Cable and Rahul Goel for giving me this wonderful opporutnity and to Nayan Ambali for mentoring me. I look forward to continue being a part of the Mifos community and continue contributing to Vision PPI as well as other projects along with helping new contributors find their way around the community.