Final documentation of the work done in RoboComp "DNN’s for precise manipulation of household objects" project through GSoC'20.

GSoC-2020-Final-Report.md

Google Summer of Code 2020 - RoboComp - Mohamed Shawky

Student's Name	Mohamed Shawky
Github Profile	DarkGeekMS
Organization	RoboComp
Project Title	DNN’s for precise manipulation of household objects
Mentors	Pablo Bustos, Marco
Primary Repository	robocomp/grasping

Brief Description

This is the final documentation of the work done in DNN’s for precise manipulation of household objects project through GSoC'20. In this project, I, Mohamed Shawky, worked with RoboComp and my mentor, Pablo Bustos, to develop fast and robust pose estimation and grasping pipeline using the power of RoboComp framework and the recent advances in Deep Learning. I used two recent DNN architectures, which are Segmentation-driven 6D Object Pose Estimation and PVN3D to build pose estimation component that estimates objects poses from RGBD images. Also, I was able to adapt the previous work with Kinova Gen3 arm to PyRep API for fast and robust path planning and grasping system. Finally, I worked on the integration of pose estimation and grasping into CORTEX architecture (based on Deep State Representation (DSR) and implemented using Real-time Pub/Sub (RTPS) and Conflict-free Replicated Data Type (CRDT)).

Completed Work

Here is a summary of all the work done through each period of the project.

Community Bonding Period

• Communicated with my mentor, Pablo Bustos, to discuss the previous work on the grasping problem and form detailed project workflow.

• Familiarized myself more with RoboComp framework and 6D object pose estimation literature.

• Started working on Segmentation-driven 6D Object Pose Estimation network implementation based on open-source inference code.

First Coding Period

• Completed a full implementation of Segmentation-driven 6D Object Pose Estimation network (found here).

• Trained the network on YCB-Videos dataset, which is one of the most diverse open-source datasets.

• Tested the trained network on YCB-Videos dataset and data collected from CoppeliaSim environments.

• Implemented a Python data collector (found here) using PyRep API, Open3D and CoppeliaSim to collect more data for augmentation.

• Retrained the network on augmented dataset.

Second Coding Period

• Improved the implementation of Segmentation-driven 6D Object Pose Estimation network by using focal loss for semantic segmentation and fixing some training issues.

• Retrained the network on the final settings.

• Investigated further improvements of pose estimation pipeline, which led me to PVN3D network.

• Studied PVN3D network and added better inference code to its open-source implementation (found here).

• Completed two pose estimation components. One for RGB inference using Segmentation-driven 6D Object Pose Estimation network (found here) and another for RGBD inference using PVN3D network (found here).

• Adapted the previous work with Kinova Gen3 arm to PyRep API, where I added embedded Lua scripts to the arm's model and called them, remotely, through PyRep API. Thus, we have a fast and robust system for path planning and grasping.

• Implemented viriatoGraspingPyrep component, which integrates and tests the whole pose estimation and grasping pipeline (found here).

Third Coding Period

• Created the first complete grasping demo of Kinova Gen3 arm using DNN-estimated poses (found here).

• Studied the workflow of the new DSR architecture through installation, testing and code review.

• Discussed the process of pose estimation and grasping integration into DSR with my mentor, Pablo Bustos.

• Merged both RGB and RGBD pose estimation components into one component, named objectPoseEstimation, for DSR integration (found here).

• Implemented object detection component that uses RetinaNet to improve object tracking in DSR (found here).

• Implemented graspDSR C++ agent (found here), which is an interface between objectPoseEstimation component and the shared graph (a.k.a. G). Also, it plans the dummy targets for the robot's arm to reach the required object.

• Tested the validity of the arm's embedded Lua scripts using DSR.

• Helped debugging and improving some features of DSR Core API.

• Wrote a detailed documentation about DSR integration and its problems (found here).

• Improved visualization and memory allocation in pose estimation components.

• Revisited all the previous work and improved some documentation.

Results and Demonstrations

Figure(1): Visualizations of the estimated poses by Segmentation-driven 6D Object Pose Estimation on YCB-Videos dataset.

Figure(2): Visualizations of the estimated poses by objectPoseEstimation component in CoppeliaSim environment.

Figure(3): Demo of path planning using the DNN-estimated poses and the embedded Lua scripts.

Figure(4): Complete Demo of the arm grasping and manipulation using DNN-estimated poses without DSR.

Figure(5): Test demo of the embedded Lua scripts in DSR using the simulator poses.

Contributions Summary

Repo	Commits	PRs	Issues
robocomp/grasping	link	-	-
robocomp/dsr-graph	link	link	link
robocomp/robocomp	link	link	link
robocomp/DNN-Services	link	-	-
robocomp/web	link	link	-

List of Posts

1. Introduction
2. Complete Work Pipeline
3. Component Structure and Pose Estimation DNN Improvements
4. Final Pose Estimation Settings
5. Pipeline Integration and First Grasping Demo
6. Final Grasping Results and Integration with DSR
7. Work Done with DSR Integration
8. Final Project Submission

Future Work

The work of this project has been completed successfully with full documentation and a decent set of demos. However, there is still some work to be done with DSR framework to be able to completely use the implemented pose estimation and grasping pipeline through DSR. This work can be summarized as follows :

• Solve the related open issues on robocomp/robocomp repo.

• Update viriatoDSR to pass the arm target poses to viriatoPyrep from G.

• Add grasping methods to viriatoPyrep, similar to viriatoGraspingPyrep.

• Test the pose estimation and grasping pipeline with tracking and social navigation, once done.

Acknowledgement

I would like to thank my mentor, Pablo Bustos, for providing me with excellent guidance through the project. Also, I would like to thank RoboComp developers, especially Esteban Martinena Guerrero, for their help with DSR integration process.