MINOS - simulator of the agent , navigating in indoor environment. Based on WebGL (Graphic library on javascript), server <-> client framework with configuration files.
Links to original code and my fork with several update and recent code (gist maybe not up-to-date).
Interactive play video.
Use SUNCG/Matterport3D (Mp3D) as predefined datasets. Use special partial datasets for MINOS simulator (ask authors of the datasets).
- download SUNCG via link, you get from authors' email (suncg.v2.zip, ~9 Gb)
- download Mp3D via .py script from author's email (~ 5 Gb)
- you download "minos" task (still without object segmentation)
- you can download any other task or bunch of data (look at "download_mp.py" script). For ex.: "semantic_voxel_label_data", "semantic_voxel_label_models", "region_classification_data", "region_classification_models", "region_segmentations" (data only: .ply and .json files). A bit later about it.
- RGB, RGBA, Grayscale
- Depth
- Segmentation with predefined labels (SUNCG, Matterport3D - only in full dataset)
- Forces
- Normals from meshes
- Measurements (velocity, accelaration)
- Retexturing (SUNCG)
- Partialy edit FOV (vertical only: added in enhancements)
- Empty room/ clutter concrete objects
- Task parameters: goal specification, reward function for each step, success objective
Sum up, search for data configuration in
- config/sensors.yml: preconfigured "sensors" to use, with its name, in various tasks (via configuration files). Change various parameters, add your own camera, depth sensor etc.
- config/envs/.py: preconfigured configuration files for tasks, added in MINOS paper. State usage of concrete sensors (observations), task-related parameters (look at Features) + scene sampling, train/test/val split
- Other configurations and parameters documented (briefly) in API.md, terminal arguments for execution shown in config/sim_config.py & config/sim_args.py.
Made my own minos_test_mp3d.py and minos_test_suncg.py to specify 3d scenes for generating RGBA, depth and objectId (semantic) data. (See below)
Use installation guide from minosworld/minos github
- Download data for minos (SUNCG/Matterport3D) via links from authors (or ask me). Place them in one folder for convinience.
- Download nvm and install node. IssueSolution: use node > 9.4.0. Check via
node -v
, install vianvm install node 9.4.0 nvm alias default 9.4.0
- Inside minos/server folder install server part of the framework
npm install
.
IssueSolution: can be problems with pkg-config, x11, xi and xext packages (no packages installed). Install them via sudo apt-get install pkg-config xserver-xorg-dev libxext-dev libxi-dev
IssueSolution: with little modification of main code, you can start server manually, without the start of client part (see below)
IssueSolution: can be problems with socketio_client (wrong version from ubuntu repositories). Use this code
sudo apt-get uninstall socketio_client
pip install https://github.com/msavva/socketIO-client-2/zipball/master
This repository is commented in file .../minos_python_repackage/minos/lib/Simulator.py
and should be updated by your own.
node [--max-old-space-size=4096 {default: 4096}] server/server.js [--port 4444 {default: 4899}]
But, one server can work only with one client simultaneously!
4. Via virtualenv or throughout your system install packages via pip3 install -r requirements.txt
. This will install openai lib gym as well and make access to minos lib
5. Use export NODE_BASE_URL=path/to/data_parent_dir
(or use another variants, see minosworld/minos github)
6. Use it! or install more (gym_minos openai gym environment, minos_unreal Reinforcement learning implementation of algorithm for navigation)
- Parial FOV editing (only vertical FOV, which change horizontal FOV)
- Memory leak when not closing simulator's server (done, but not tested in my fork) No problem now!//3. Error with non-square resolution (current issue on original github) Evade this room (filter parameter)//4. Problem with rendering of several scenes (make an issue on github)
- No object segmentation data preloaded by mp3d_task (loaded by myself, current issue on github)
1.1 solution: Install NVIDIA drivers and CUDA without opengl files see here
1.2 solution, Easier: prepend LD_LIBRARY_PATH with path to mesa libraries (preinstall them as libgl1-mesa, mesa-utils)
2.1 solution: nvidia-xconfig - have to try (will update):)
2.2 solution: use nvidia/opengl or even cudagl lib - tried, but I had problems. Good luck to you and let me know about SUCCESS!
Use pygame_client for
python3 -m tools.pygame_client [--key1 --key2 pararm]
- manual agent control in simulator via keyboard
Example:
python3 -m tools.pygame_client --depth -s normal -s objectId -s objectType -s map --navmap --width 128 --height 128
(Not recommended) For separate execution of server and client part (1 client for 1 simulator only):
python3 -m tools.pygame_client [--depth] [-s normal] [-s objectType] [--width 244] [--height 244] [--save_video test] [--port 4444 {default: 4899}]
(need modify files, a little)
- --depth display depth
- -s something display sensor:
- normal - colored normals' map;
- objectId - colored labels of objects, by instance number;
- objectType - colored ..., by type
- map - show topview grayscale map
- --navmap - show shortestpath for agent to specified goal on topview map
- --width, --height - display and observations resolution!!! (display only rgba, no matter what observations you look for)
- --agent_config: name of config file for agent (without ".yml"), predefined in config folder (agent_gridworld - discrete, agent_continous - continous, etc.), you can write yours. Include physical params (weight, size, ...) and movement params (speed, rotation speed, ...)
- --env_config: name of config task-related environment for agent (without ".py"). Predefined in config/envs folder, you can write yours. Include task-specification ("pointgoal", "roomgoal" or "objectgoal"), scene specification, i.e. path to table of scenes and train/test/val split of states (in config), observations specification by name of the sensor, etc.
- Others you can find in sim_config.py file with default values!!!
You need to modify code by yourself to gain observations from data and save them as images or video (or just use my rewrited demo.py script, see below). First of all, use env_config and agent_config to state necessary parameters, like observation types, resolution for environment.
There are plenty scripts useful for RL tasks, like generating RL datasets (like "scene<->initial_place" configurations, to use for benchmarking by other algorithms).
OpenAI gym API is used as prebuild python package gym/gym_minos, which is then used as 'indoor-env' environemnt in your local copy of gym library.
We can use this API for generating control logic and automatical generation of output sensor data.
(if installing separately): via pip install -e .
inside gym folder. (better use origin/python-repackage branch, rather then origin/master)
Then look inside gym/demo.py file. I rewrote code for generating sensor output from different scenes and locations, as 4 views of 90 degree camera rotations. Use only env_config and agent_config to execute gym session:
python3 demo.py --env_config objectgoal_suncg_mf --agent_config agent_gridworld --width 128 --height 128
Change width
and height
for another resolution of images (only square, by now). Displayed window will be 4 times bigger (predefined in pygame module).
My implementation uses own minos_test_suncg.py and minos_test_mp3d.py environmental configs, plus agent_gridworld config. Action space in gym environment 'indoor-v0' consists of 3 actions: turnLeft, turnRight, forward (parametrized with available_controls in config file, see RoomSimulator.py).
def run_gym(sim_args):
env = gym.make('indoor-v0')
env.configure(sim_args)
try:
print('Running MINOS gym example')
for i_episode in range(100):
print('Starting episode %d' % i_episode)
observation = env.reset()
done = False
num_steps = 0
name = sim_args['env_config'].split('_')[-1]
while not done:
img_prefix = 'pics/'
actions_str =[None] + ['turnRight']*3
for action_i, action_str in enumerate(actions_str):
env.render(mode='human')
if acntion_str is not None:
observation, reward, done, info = env.step(actions_dict[action_str])
img = observation['observation']['sensors']['color']['data']
img = img.reshape((img.shape[1], img.shape[0], img.shape[2]))
scipy.misc.toimage(img, cmin=0, cmax=255).save(name + '_' +img_prefix + str(i_episode) + '_' + str(action_i) + 'img' + '.png')
depth = observation['observation']['sensors']['depth']['data']
depth = depth.reshape((depth.shape[1], depth.shape[0], depth.shape[2]))
depth *= (255.0 / depth.max()) # naive rescaling for visualization
depth = depth.astype(np.uint8)
scipy.misc.toimage(depth, cmin=0, cmax=255).save(name + '_' +img_prefix + str(i_episode) + '_' + str(action_i) +'depth' + '.png')
if 'suncg'==name:
objectId = observation['observation']['sensors']['objectId']['data']
scipy.misc.toimage(objectId, cmin=0, cmax=255).save(name + '_' +img_prefix + str(i_episode)+ '_' + str(action_i) + 'objectId' + '.png')
num_steps += 1
done = True
except Exception as e:
print(traceback.format_exc())
env._close()
Addtional options
- Use predefined episodes (in original repository) in parameter 'states_file', with concrete train/val/test split. Specify type of dataset in 'episode_schedule' ('train'/'val'/'test'). Specify filters for room types, Ids and characteristics with 'scene', 'scene_filter' and 'episode_filter'. See other params in config/sim_config.py (i.e. in gist config). More in issue and API.md.
- Use another resolution, with 'resolution' parameter,
width
andheight
, and 'resize: True' for sensors yaml file. - Use objectId information for SUNCG, and wait in my fork or (original code)[] for Matterport3D objectID access via observations.
Video SUNCG images generation and Matterport3D images generation. Only RGB display.
Links to original code and my fork as MS thesis repository
Additional API include class Environment, which aggregate deepmind lab maze, gym and indoor_environment for MINOS simulator. IndoorEnvironment class is a wrapper for RoomSimulator class in minos library (lib/RoomSimulator.py).
This package imports sim_config and RoomSimulator from minos from minos.lib.RoomSimulator import RoomSimulator; from minos.config import sim_config
, as well as add Tensorflow Unreal model to provide code for training, display, evaluation and vizualisation of NN weights (original repository is a fork of A3C UNREAL model for DeepMind-lab maze)
See main usage in README.md
- main.py - training of A3C model with 4 simultaneously trained simulators
- display.py - display training process with score, value function and policy
- evaluate.py - evaluate success rate on test scenes. See example in gist
Video of slightly trained algorithm
Included in server.js with bundle of STK module.
See SUNCG/Matterport3D.
Some examples:
- Voxel semantic segmentation: Matterport3D example model in Lua, adapted from ScanNet & SUNCG SCCNet for semantic scene completion
- Normal prediction with Mp3D + SUNCG finetuning. [code][full pdf for Mp3D with other tasks]
- Other tasks: Matterport3D tasks and toolbox, SUNCG toolbox + SSTK as aggregation of Mp3D, SUNCG, ScanNet and ShapeNet models adapted code (with Dockerfile).
Pros | Cons |
---|---|
Multisensor output | No point-cloud output |
Augmentation functionality (retexturing, sensors parameters etc.) | -//- |
support SUNCG & Matterport3D (few simulators have code for this) | -//- |
Open Source, realtime support, JS + Python | Issues, bugs, enhancements DIY |
Mostly for RL tasks | -//- |
Hi, your brief introduction is really helpful, but did you wrote a path finding algorithm (such as A*, BFS, DFS,...) to automatically move the agent in the world ?