Skip to content

Commit

Permalink
code refactor for OpenSTL V0.2.0
Browse files Browse the repository at this point in the history
  • Loading branch information
Lupin1998 committed Apr 18, 2023
1 parent aeeaf0a commit b678838
Show file tree
Hide file tree
Showing 153 changed files with 2,049 additions and 963 deletions.
5 changes: 5 additions & 0 deletions .coveragerc
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
[report]
exclude_lines =
@abstractmethod
@abc.abstractmethod
raise NotImplementedError
77 changes: 37 additions & 40 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
# SimVP: Towards Simple yet Powerful Spatiotemporal Predictive learning
# OpenSTL: Open-source Toolbox for SpatioTemporal Predictive Learning

<p align="left">
<a href="https://arxiv.org/abs/2211.12509" alt="arXiv">
<img src="https://img.shields.io/badge/arXiv-2211.12509-b31b1b.svg?style=flat" /></a>
<a href="https://github.com/Westlake-AI/MogaNet/blob/main/LICENSE" alt="license">
<a href="https://github.com/chengtan9907/OpenSTL/blob/master/LICENSE" alt="license">
<img src="https://img.shields.io/badge/license-Apache--2.0-%23002FA7" /></a>
<a href="https://simvpv2.readthedocs.io/en/latest/" alt="docs">
<img src="https://readthedocs.org/projects/simvpv2/badge/?version=latest" /></a>
<a href="https://simvpv2.readthedocs.io/en/latest/" alt="docs">
<a href="https://github.com/chengtan9907/OpenSTL/issues" alt="docs">
<img src="https://img.shields.io/github/issues-raw/chengtan9907/SimVPv2?color=%23FF9600" /></a>
<a href="https://github.com/chengtan9907/SimVPv2/issues" alt="resolution">
<a href="https://github.com/chengtan9907/OpenSTL/issues" alt="resolution">
<img src="https://img.shields.io/badge/issue%20resolution-1%20d-%23B7A800" /></a>
</p>

Expand All @@ -25,47 +25,35 @@ This repository is an open-source project for video prediction benchmarks, which

## Introduction

This is the journal version of our previous conference work ([SimVP: Simpler yet Better Video Prediction](https://arxiv.org/abs/2206.05099), In CVPR 2022).

It is worth noticing that the hidden Translator $h$ in SimVP can be replaced by any [MetaFormer](https://arxiv.org/abs/2111.11418) block (satisfying the macro design of `token mixing` and `channel mixing`).
This is the journal version of our previous conference work ([SimVP: Simpler yet Better Video Prediction](https://arxiv.org/abs/2206.05099), In CVPR 2022). It is worth noticing that the hidden Translator $h$ in SimVP can be replaced by any [MetaFormer](https://arxiv.org/abs/2111.11418) block (satisfying the macro design of `token mixing` and `channel mixing`).
<p align="center">
<img width="75%" src="https://user-images.githubusercontent.com/44519745/219116763-bb4ab408-7f25-47d3-b93e-79834c7b065e.png"> <br>
</p>

The performance of SimVPs on the Moving MNIST dataset. For the training time, the less the better. For the inference efficiency (frames per second), the more the better.
<p align="center">
<img width="70%" src="https://user-images.githubusercontent.com/44519745/219116933-a34f5133-b38f-4657-bea4-c49361696fd2.png"> <br>
</p>

Quantitative results of different methods on the Moving MNIST dataset ($10 \rightarrow 10$ frames).
<p align="center">
<img width="75%" src="https://user-images.githubusercontent.com/44519745/219117248-90825af9-28e8-42c6-8fec-858ef7f6dd92.png"> <br>
</p>

<p align="right">(<a href="#top">back to top</a>)</p>

## Overview

* `simvp/api` contains an experiment runner.
* `simvp/core` contains core training plugins and metrics.
* `simvp/datasets` contains datasets and dataloaders.
* `simvp/methods/` contains training methods for various video prediction methods.
* `simvp/models/` contains the main network architectures of various video prediction methods.
* `simvp/modules/` contains network modules and layers.
* `tools/non_dist_train.py` is the executable python file with possible arguments for training, validating, and testing pipelines.
* `openstl/api` contains an experiment runner.
* `openstl/core` contains core training plugins and metrics.
* `openstl/datasets` contains datasets and dataloaders.
* `openstl/methods/` contains training methods for various video prediction methods.
* `openstl/models/` contains the main network architectures of various video prediction methods.
* `openstl/modules/` contains network modules and layers.
* `tools/train.py` and `tools/train.py` are the executable python file with possible arguments for training, validating, and testing pipelines.

## News and Updates

[2023-02-18] `SimVP` v0.1.0 is released. Benchmark results and config files are updated for MMNIST, TaxiBJ, and WeatherBench datasets.
[2023-04-19] `OpenSTL` v0.2.0 is released.

## Installation

This project has provided an environment setting file of conda, users can easily reproduce the environment by the following commands:
```shell
git clone https://github.com/chengtan9907/SimVPv2
cd SimVPv2
git clone https://github.com/chengtan9907/OpenSTL
cd OpenSTL
conda env create -f environment.yml
conda activate SimVP
conda activate OpenSTL
python setup.py develop
```

Expand All @@ -76,21 +64,22 @@ python setup.py develop
* fvcore
* numpy
* hickle
* scikit-image=0.16.2
* scikit-image
* scikit-learn
* torch
* timm
* tqdm
* xarray
</details>

Please refer to [install.md](docs/en/install.md) for more detailed instructions.

## Getting Started

Please see [get_started.md](docs/en/get_started.md) for the basic usage. Here is an example of single GPU training SimVP+gSTA on Moving MNIST dataset.
Please see [get_started.md](docs/en/get_started.md) for the basic usage. Here is an example of single GPU non-dist training SimVP+gSTA on Moving MNIST dataset.
```shell
bash tools/prepare_data/download_mmnist.sh
python tools/non_dist_train.py -d mmnist --lr 1e-3 -c ./configs/mmnist/simvp/SimVP_gSTA.py --ex_name mmnist_simvp_gsta
python tools/train.py -d mmnist --lr 1e-3 -c ./configs/mmnist/simvp/SimVP_gSTA.py --ex_name mmnist_simvp_gsta
```

<p align="right">(<a href="#top">back to top</a>)</p>
Expand Down Expand Up @@ -140,11 +129,11 @@ We support various video prediction methods and will provide benchmarks on vario
<details open>
<summary>Currently supported datasets</summary>

- [x] [KTH Action](https://ieeexplore.ieee.org/document/1334462) (ICPR'2004) [[download](https://www.csc.kth.se/cvap/actions/)] [[config](https://github.com/chengtan9907/SimVPv2/configs/kth)]
- [x] [KittiCaltech Pedestrian](https://dl.acm.org/doi/10.1177/0278364913491297) (IJRR'2013) [[download](https://figshare.com/articles/dataset/KITTI_hkl_files/7985684)] [[config](https://github.com/chengtan9907/SimVPv2/configs/kitticaltech)]
- [x] [Moving MNIST](http://arxiv.org/abs/1502.04681) (ICML'2015) [[download](http://www.cs.toronto.edu/~nitish/unsupervised_video/)] [[config](https://github.com/chengtan9907/SimVPv2/configs/mmnist)]
- [x] [TaxiBJ](https://arxiv.org/abs/1610.00081) (AAAI'2017) [[download](https://github.com/TolicWang/DeepST/tree/master/data/TaxiBJ)] [[config](https://github.com/chengtan9907/SimVPv2/configs/taxibj)]
- [x] [WeatherBench](https://arxiv.org/abs/2002.00469) (ArXiv'2020) [[download](https://github.com/pangeo-data/WeatherBench)] [[config](https://github.com/chengtan9907/SimVPv2/configs/weather)]
- [x] [KTH Action](https://ieeexplore.ieee.org/document/1334462) (ICPR'2004) [[download](https://www.csc.kth.se/cvap/actions/)] [[config](configs/kth)]
- [x] [KittiCaltech Pedestrian](https://dl.acm.org/doi/10.1177/0278364913491297) (IJRR'2013) [[download](https://figshare.com/articles/dataset/KITTI_hkl_files/7985684)] [[config](configs/kitticaltech)]
- [x] [Moving MNIST](http://arxiv.org/abs/1502.04681) (ICML'2015) [[download](http://www.cs.toronto.edu/~nitish/unsupervised_video/)] [[config](configs/mmnist)]
- [x] [TaxiBJ](https://arxiv.org/abs/1610.00081) (AAAI'2017) [[download](https://github.com/TolicWang/DeepST/tree/master/data/TaxiBJ)] [[config](configs/taxibj)]
- [x] [WeatherBench](https://arxiv.org/abs/2002.00469) (ArXiv'2020) [[download](https://github.com/pangeo-data/WeatherBench)] [[config](configs/weather)]

</details>

Expand All @@ -160,23 +149,31 @@ SimVPv2 is an open-source project for video prediction methods created by resear

## Citation

If you are interested in our repository and our paper, please cite the following paper:
If you are interested in our repository or our paper, please cite the following paper:

```
@article{tan2022simvp,
title={SimVP: Towards Simple yet Powerful Spatiotemporal Predictive Learning},
author={Tan, Cheng and Gao, Zhangyang and Li, Stan Z},
author={Tan, Cheng and Gao, Zhangyang and Li, Siyuan and Li, Stan Z},
journal={arXiv preprint arXiv:2211.12509},
year={2022}
}
```
```
@misc{tan2023openstl,
title={OpenSTL: Open-source Toolbox for SpatioTemporal Predictive Learning},
author={Tan, Cheng and Li, Siyuan and Gao, Zhangyang and Li, Stan Z},
howpublished = {\url{https://github.com/chengtan9907/OpenSTL}},
year={2023}
}
```

## Contribution and Contact

For adding new features, looking for helps, or reporting bugs associated with SimVPv2, please open a [GitHub issue](https://github.com/chengtan9907/SimVPv2/issues) and [pull request](https://github.com/chengtan9907/SimVPv2/pulls) with the tag "help wanted" or "enhancement". Feel free to contact us through email if you have any questions. Enjoy!
For adding new features, looking for helps, or reporting bugs associated with `OpenSTL`, please open a [GitHub issue](https://github.com/chengtan9907/OpenSTL/issues) and [pull request](https://github.com/chengtan9907/OpenSTL/pulls) with the tag "help wanted" or "enhancement". Feel free to contact us through email if you have any questions. Enjoy!

- Cheng Tan (tancheng@westlake.edu.cn), Westlake University & Zhejiang University
- Siyuan Li (lisiyuan@westlake.edu.cn), Westlake University & Zhejiang University
- Cheng Tan (tancheng@westlake.edu.cn), Westlake University & Zhejiang University
- Zhangyang Gao (gaozhangyang@westlake.edu.cn), Westlake University & Zhejiang University

<p align="right">(<a href="#top">back to top</a>)</p>
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
15 changes: 0 additions & 15 deletions configs/weather/simvp_t2m_5_625/SimVP_ConvNeXt.py

This file was deleted.

15 changes: 0 additions & 15 deletions configs/weather/simvp_t2m_5_625/SimVP_HorNet.py

This file was deleted.

15 changes: 0 additions & 15 deletions configs/weather/simvp_t2m_5_625/SimVP_MLPMixer.py

This file was deleted.

15 changes: 0 additions & 15 deletions configs/weather/simvp_t2m_5_625/SimVP_MogaNet.py

This file was deleted.

15 changes: 0 additions & 15 deletions configs/weather/simvp_t2m_5_625/SimVP_Poolformer.py

This file was deleted.

15 changes: 0 additions & 15 deletions configs/weather/simvp_t2m_5_625/SimVP_VAN.py

This file was deleted.

15 changes: 0 additions & 15 deletions configs/weather/simvp_t2m_5_625/SimVP_ViT.py

This file was deleted.

14 changes: 0 additions & 14 deletions configs/weather/simvp_t2m_5_625/SimVP_gSTA.py

This file was deleted.

15 changes: 0 additions & 15 deletions configs/weather/simvp_tcc_5_625/SimVP_IncepU.py

This file was deleted.

15 changes: 0 additions & 15 deletions configs/weather/simvp_tcc_5_625/SimVP_Swin.py

This file was deleted.

15 changes: 0 additions & 15 deletions configs/weather/simvp_tcc_5_625/SimVP_Uniformer.py

This file was deleted.

15 changes: 0 additions & 15 deletions configs/weather/simvp_uv10_5_625/SimVP_ConvMixer.py

This file was deleted.

File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
19 changes: 19 additions & 0 deletions configs/weather/tcc_5_625/ConvLSTM.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
method = 'ConvLSTM'
# reverse scheduled sampling
reverse_scheduled_sampling = 0
r_sampling_step_1 = 25000
r_sampling_step_2 = 50000
r_exp_alpha = 5000
# scheduled sampling
scheduled_sampling = 1
sampling_stop_iter = 50000
sampling_start_value = 1.0
sampling_changing_rate = 0.00002
# model
num_hidden = '128,128,128,128'
filter_size = 5
stride = 1
patch_size = 2
layer_norm = 0
# training
lr = 1e-4
Loading

0 comments on commit b678838

Please sign in to comment.