psgtr_r50_epoch_60.pth file is not accessable

[EvilFreelancer]

Hi, thanks for your amazing project! It has a lot of potential.

I've tried to follow your guide and stuck on this step:

> You should make directory 'checkpoints' in moai/sgg and upload checkpoint of Scene Graph Generation after downloading it, where its checkpoint filename should be 'psgtr_r50_epoch_60.pth'

[Checkpoint Link](https://entuedu-my.sharepoint.com/personal/jingkang001_e_ntu_edu_sg/_layouts/15/onedrive.aspx?id=%2Fpersonal%2Fjingkang001%5Fe%5Fntu%5Fedu%5Fsg%2FDocuments%2Fopenpsg%2Fwork%5Fdirs%2Fpsgtr%5Fr50&ga=1) is from [Panoptic SGG](https://github.com/Jingkang50/OpenPSG).

I can't download psgtr_r50_epoch_60.pth file because it seems hidden, I've tried to get access via my personal account, but seems it is not possible.

Can you open access please?

[ByungKwanLee]

Some error of link accessibility occurs! Have updated README.md!

[EvilFreelancer]

Hi, thanks a lot! I've downloaded file, placed it in checkpoints folder, but now I have this error:

(venv) [pasha-pc] ~/Documents/Repository/gpt/MoAI $ python demo.py 
/home/pasha/Documents/Repository/gpt/MoAI/venv/lib/python3.11/site-packages/torch/_utils.py:776: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  return self.fget.__get__(instance, owner)()
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:02<00:00,  1.26s/it]
[2024-03-14 10:00:02,808] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/pasha/Documents/Repository/gpt/MoAI/venv/lib/python3.11/site-packages/mmdet/models/task_modules/builder.py:39: UserWarning: ``build_assigner`` would be deprecated soon, please use ``mmdet.registry.TASK_UTILS.build()`` 
  warnings.warn('``build_assigner`` would be deprecated soon, please use '
/home/pasha/Documents/Repository/gpt/MoAI/venv/lib/python3.11/site-packages/mmdet/models/task_modules/builder.py:32: UserWarning: ``build_match_cost`` would be deprecated soon, please use ``mmdet.registry.TASK_UTILS.build()`` 
  warnings.warn('``build_match_cost`` would be deprecated soon, please use '
/home/pasha/Documents/Repository/gpt/MoAI/venv/lib/python3.11/site-packages/mmdet/models/task_modules/builder.py:46: UserWarning: ``build_sampler`` would be deprecated soon, please use ``mmdet.registry.TASK_UTILS.build()`` 
  warnings.warn('``build_sampler`` would be deprecated soon, please use '
Loads checkpoint by local backend from path: moai/sgg/checkpoints/psgtr_r50_epoch_60.pth
Traceback (most recent call last):
  File "/home/pasha/Documents/Repository/gpt/MoAI/demo.py", line 20, in <module>
    = prepare_moai(moai_path='BK-Lee/MoAI-7B', bits=4, grad_ckpt=False, lora=False, dtype='fp16')
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pasha/Documents/Repository/gpt/MoAI/moai/load_moai.py", line 86, in prepare_moai
    sgg_model = init_detector(cfg, 'moai/sgg/checkpoints/psgtr_r50_epoch_60.pth', palette="none").cuda()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pasha/Documents/Repository/gpt/MoAI/venv/lib/python3.11/site-packages/mmdet/apis/inference.py", line 102, in init_detector
    metainfo = DATASETS.build(test_dataset_cfg).metainfo
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pasha/Documents/Repository/gpt/MoAI/venv/lib/python3.11/site-packages/mmengine/registry/registry.py", line 570, in build
    return self.build_func(cfg, *args, **kwargs, registry=self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pasha/Documents/Repository/gpt/MoAI/venv/lib/python3.11/site-packages/mmengine/registry/build_functions.py", line 72, in build_from_cfg
    raise KeyError(
KeyError: '`cfg` or `default_args` must contain the key "type", but got {\'test_mode\': True, \'filter_cfg\': {\'filter_empty_gt\': True, \'min_size\': 32}, \'pipeline\': [{\'type\': \'LoadImageFromFile\'}, {\'type\': \'Resize\', \'scale\': (1333, 800), \'keep_ratio\': True}, {\'type\': \'Normalize\', \'mean\': [123.675, 116.28, 103.53], \'std\': [58.395, 57.12, 57.375], \'to_rgb\': True}, {\'type\': \'Pad\', \'size_divisor\': 32}, {\'type\': \'PackDetInputs\', \'meta_keys\': (\'img_id\', \'img_path\', \'ori_shape\', \'img_shape\', \'scale_factor\', \'img\')}], \'lazy_init\': True}\nNone'

[ByungKwanLee]

Can you upload your package files: mmdet/apis/inference.py and mmcv/transforms/processing.py

I will compare my file with your files

This error is due to loading panoptic SGG model

[EvilFreelancer]

Here is the link to gist with mmdet/apis/inference.py and mmcv/transforms/processing.py:

https://gist.github.com/EvilFreelancer/51d65199081b45ed45ef60b21719b088

---

I have installed CUDA 11.7 and python 3.9 virtual environment (instead of conda), here is the steps to reproduce it:

python3.9 -m venv venv
. venv/bin/activate
export CC=gcc-11 CXX=g++-11
pip install torch==2.0.1+cu117 torchvision==0.15.2+cu117 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu117
pip install natten==0.15.1+torch200cu118 -f https://shi-labs.com/natten/wheels
pip install -r assets/requirements/requirements.txt
pip install -r assets/requirements/requirements_custom.txt
pip install wheel
pip install flash-attn --no-build-isolation

After launching demo.py I see following errors:

KeyError: '`cfg` or `default_args` must contain the key "type", but got {\'test_mode\': True, \'filter_cfg\': {\'filter_empty_gt\': True, \'min_size\': 32}, \'pipeline\': [{\'type\': \'LoadImageFromFile\'}, {\'type\': \'Resize\', \'scale\': (1333, 800), \'keep_ratio\': True}, {\'type\': \'Normalize\', \'mean\': [123.675, 116.28, 103.53], \'std\': [58.395, 57.12, 57.375], \'to_rgb\': True}, {\'type\': \'Pad\', \'size_divisor\': 32}, {\'type\': \'PackDetInputs\', \'meta_keys\': (\'img_id\', \'img_path\', \'ori_shape\', \'img_shape\', \'scale_factor\', \'img\')}], \'lazy_init\': True}\nNone'

[ByungKwanLee]

I checked your files did not change from my instruction in README

Please carefully follow the procedure of README because panoptic SGG has compatible issue to general torch and transformers library

[EYcab]

@ByungKwanLee ,I have followed your instructions like EvilFreelancer and ran into the same bug.Please let us know if you have a solution.

[ByungKwanLee]

Could you show your error report?

[EYcab]

[EYcab]

The above is based entirely on the instructions provided in README and the psgtr part runs into errors @ByungKwanLee.

[EYcab]

this is based on CUDA 11.3, torch==1.10.0+cu113 and mmcv-full==1.4.3 mmdet==2.20,0 via openmmlab wheel installation like what it says in https://github.com/Jingkang50/OpenPSG?tab=readme-ov-file#get-started and everything else under your instruction.

Yet,this error is quite consistent with @EvilFreelancer experienced and what I had yesterday after executing entirely on your instructions in README.

[davideuler]

I've got the same problem of : KeyError: 'cfg or default_args must contain the key "type".

Dependencies installed:

pip install natten==0.15.1+torch200cu118 -f https://shi-labs.com/natten/wheels

pip install -r assets/requirements/requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install -r assets/requirements/requirements_custom.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install flash-attn --no-build-isolation -i https://pypi.tuna.tsinghua.edu.cn/simple

$ pip freeze |grep torch

natten==0.15.1+torch200cu118
torch==2.0.1+cu118
torchaudio==2.0.2+cu118
torchvision==0.15.2+cu118

[pupba]

@ByungKwanLee
@EYcab와 같은 에러가 발생합니다.
혹시 아직 bug가 안고쳐졌을 까요?
제 의존성은 다음과 같습니다.

torch==2.0.1+cu118
torchaudio==2.0.2+cu118
torchvision==0.15.2+cu118
natten==0.15.1+torch200cu118

제가 알아낸 것은 mmengine/registry/build_functions.py의

if 'type' not in cfg:
        # bug
        if default_args is None or 'type' not in default_args:
            raise KeyError(
                '`cfg` or `default_args` must contain the key "type", '
                f'but got {cfg}\n{default_args}')

부분에서 다음 사진과 같은 에러가 발생합니다.

[ByungKwanLee]

It may be the problem of third-party library issue. Would recommend reading the readme carefully to solve the issue of library