[Bug Report] Using set_world_poses_from_view with camera sensors

[elle-miller]

Hi there,

There is potentially a bug with setting the camera pose via set_world_poses_from_view, otherwise I am using it incorrectly.

This is my config & instantiation for the TiledCamera. I am using a personal DirectRLEnv.

# in the cfg
tiled_camera: TiledCameraCfg = TiledCameraCfg(
    prim_path="/World/envs/env_.*/Camera",
    offset=TiledCameraCfg.OffsetCfg(pos=(0.85, -1.10, 1.6), rot=(1,0,1,1), convention="world"),
    data_types=["rgb"],
    spawn=sim_utils.PinholeCameraCfg(
        focal_length=24.0, focus_distance=400.0, horizontal_aperture=20.955, clipping_range=(0.01, 2.5)
    ),
    width=80,
    height=80, debug_vis=True
)
eye = [0.9, 0.9, 0.6]
target = [0.3, 0, 0.2]

# ... and then in _setup_scene() 
self._tiled_camera = TiledCamera(self.cfg.tiled_camera)
eyes = torch.tensor(self.cfg.eye, dtype=torch.float, device=self.device).repeat((self.num_envs, 1)) + self.scene.env_origins
targets = torch.tensor(self.cfg.target, dtype=torch.float, device=self.device).repeat((self.num_envs, 1)) + self.scene.env_origins
self._tiled_camera.set_world_poses_from_view(eyes=eyes, targets=targets)
self.scene.sensors["tiled_camera"] = self._tiled_camera

This setup leads to this error:

File "/home/emil/code/external/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/envs/direct_rl_env.py", line 118, in __init__
    self._setup_scene()
  File "/home/emil/code/external/IsaacLab/IsaacLabExtension/exts/multimodal_gym/multimodal_gym/tasks/franka/lift.py", line 339, in _setup_scene
    self._tiled_camera.set_world_poses_from_view(eyes=eyes, targets=targets)
  File "/home/emil/code/external/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/sensors/camera/camera.py", line 339, in set_world_poses_from_view
    env_ids = self._ALL_INDICES
AttributeError: 'TiledCamera' object has no attribute '_ALL_INDICES'

My question is, am I using set_world_poses_from_view in the wrong place? If so, where should I call it from? I can get around this issue by putting the set_world_poses_from_view call in a later function call like _configure_gym_env_spaces(). However, this creates problems for me because the first timestep of collected image observations are from the TiledCameraCfg pose, instead of the updated eyes/targets one.

I want to use set_world_poses_from_view because it is more intuitive than setting the pos and quat in TiledCameraCfg.

System Info

• Isaac Sim Version: 4.1
• OS: Ubuntu 20.04
• Latest Isaac Lab release

Checklist

• I have checked that there is no similar issue in the repo (required)
• I have checked that the issue is not in running Isaac Sim itself and is related to the repo

Acceptance Criteria

• I can set the TiledCamera pose using eyes/targets instead of pos/quat, and this view renders correctly from timestep 0

[Mayankm96]

Seems like an implementation issue on our side. The variable _ALL_INDICES is not being defined. @pascal-roth can you take a look, please? The fix should be simple.

[pascal-roth]

Hi,

This is not an implementation error. The variable _ALL_INDICIES is available once the TiledCameras has been initialized. As the function set_world_poses_from_view directly updates the camera's view, it is impossible to use this function unless the camera has been initialized, i.e., when also _ALL_INDICIES is available.

As a fix, you can set initialize the scene and then use the set_world_poses_from_view before executing the first simulation step. This should result in your desired behavior.

Will close this issue now, as there is no implementation bug.