MotionHint: Self-Supervised Monocular Visual Odometry with Motion Constraints

This is the official PyTorch implementation of "MotionHint: Self-Supervised Monocular Visual Odometry with Motion Constraints".

MotionHint: Self-Supervised Monocular Visual Odometry with Motion Constraints

Cong Wang, Yu-Ping Wang and Dinesh Manocha

ICRA 2022 (arXiv pdf)

This code is for non-commercial use; please see the license file for terms.

If you find our work useful in your research please consider citing our paper:

@inproceedings{DBLP:conf/icra/WangWM22,
  author    = {Cong Wang and
               Yu{-}Ping Wang and
               Dinesh Manocha},
  title     = {MotionHint: Self-Supervised Monocular Visual Odometry with Motion
               Constraints},
  booktitle = {2022 International Conference on Robotics and Automation, {ICRA} 2022,
               Philadelphia, PA, USA, May 23-27, 2022},
  pages     = {1265--1272},
  publisher = {{IEEE}},
  year      = {2022}
}

⚙️ Setup

First, assuming you are using a fresh Anaconda distribution, you can set up the environment with:

conda env create -f environment.yaml

This command will install all packages used during training and testing.

Dataset

You can download the KITTI Odometry dataset from here. You need to download the color odometry dataset (~65GB) and the ground truth poses (~4MB), and organize them as:

data_path
│
└───poses
│   │   00.txt
│   │   01.txt
|   |   ...
|   |   21.txt
│   
└───sequences
    │   
    └───00
    |   |   calib.txt
    |   |   times.txt
    |   └───image_2
    |   |   |   000000.png
    |   |   |   000001.png
    |   |   |   ...
    |   |      
    |   └───image_3
    |       |   000000.png
    |       |   000001.png
    |       |   ...
    |   
    └───01
    |   |   calib.txt
    |   |   times.txt
    |   └───image_2
    |   |   |   000000.png
    |   |   |   000001.png
    |   |   |   ...
    |   |      
    |   └───image_3
    |       |   000000.png
    |       |   000001.png
    |       |   ...
    |
    └───02
    |
    ...

Training

To train our MotionHint models, you need to first download the pretrained model of Monodepth2, and replace 'PRETRAINED_MONODEPTH2' in all training scripts with your own path.

Using the shell scripts in ./scripts, you can train the self-supervised visual odometry with our MotionHint.

There are totally three setups in our paper,

If you want to train the network using 'Ground Truth' setup, you should run:

./scripts/train_gt_setup.sh 0     # 0 infers using the gpu 0.

If you want to train the network using 'Paired Poses' setup, you should run:

./scripts/train_pair_setup.sh 0     # 0 infers using the gpu 0.

If you want to train the network using 'Unpaired Poses' setup, you should run:

./scripts/train_unpair_setup.sh 0     # 0 infers using the gpu 0.

Before running all scripts, you need to change the values of 'DATA_PATH' and 'LOG_PATH' in the scripts. The models will be saved in LOG_PATH/models, and the latest model will be used for evaluation.

You can also change some parameters in scripts and options.py to do some ablation study.

Evaluation

We directly employ the KITTI Odometry Evaluation Toolbox to evaluate our models. For convenience, we have integrated the toolbox into our code, and written a script to do the evaluation.

If you want to evlaute a MotionHint model, you can run:

./scripts/eval_model.sh YOUR_MODEL_PATH EVALUATION_NAME

The evaluation results will be shown in the terminal, and saved in ./evaluations/result/EVALUATION_NAME.

Results and Pre-trained Models

Model name	Seq 09 translation error(%)	Seq 09 rotation error(°/100m)	Seq 09 ATE(m)	Seq 10 translation error(%)	Seq 10 rotation error(°/100m)	Seq 10 ATE(m)
MonoDepth2	15.079	3.335	69.019	12.102	4.927	19.371
MotionHint(Ground Truth)	13.502	2.998	62.337	10.377	4.453	17.541
MotionHint(Paired Pose)	14.071	3.099	64.704	10.976	4.495	17.752
MotionHint(Unpaired Pose)	9.761	2.226	46.036	8.679	3.334	13.282

All models above are trained for 20 epochs, and then the lastest version is taken as the final model.