Skip to content

Home

Jump to bottom
David Cardinal edited this page Nov 7, 2023 · 97 revisions

ISET3d development is now entirely based on PBRT Version 4, including GPU support

ISET3d uses computer graphics path tracing to render physically-realistic complex scenes -- with labels -- that accurately represent natural scene properties (illumination, surface reflectance, multiple bounces). ISET3d is integrated with ISETCam and ISETBio.

A goal of the ISET software collection is to produce code that makes the ideas clear. Computational efficiency is valuable, but a secondary concern. Please let us know if you have a clearer way to implement an idea, or please propose a clarifying comment in the code or wiki.

We expect ISET3d and ISETCam will be toolboxes (libraries) on a user's Matlab path. Projects will be in a separate directory or GitHub repository.

NOTE We are working actively on the 'dev' branch. If you have an issue getting something to run using the master branch, it is worth doing a checkout of the dev branch to try it.

Workflow

The typical workflow is to edit a recipe, and then invoke a series of functions - write, render, show. We invoked the series of functions so often that we now commonly use the function piWRS(recipe) to call the series of functions.

The ISET3d function piRead reads a PBRT scene file, and parses the contents into a Matlab class we call a recipe. The user can adjust the recipe properties using recipe.set() to specify how the scene will be rendered. This includes specifying the spatial resolution, number of rays per pixel, the multi-element optics, material properties, and the positions of objects.

When the user is finished editing the recipe, we use piWrite to create an updated version of the PBRT files. By default, these are written out into the iset3d/local folder.

The piRender function calls a docker image to render the PBRT files. Rendering options include generating (a) the radiance or irradiance, (b) the depth map, and (c) pixel-by-pixel labels for the material or object. Typically, we show the rendered data in an ISETCam (or ISETBio) window invoking sceneWindow() or a lens is specified, oiWindow().

ISETCam/Bio integration

ISET3d works with both ISETCam and ISETBio. Specifically, this means that it manages to render with pinhole, a realistic lens model (we call omni) from ISETCam, and a human eye model based on sceneEye in ISETBio.

When the recipe specifies pinhole optics, piRender returns the spectral radiance of a scene. If the recipe specifies a lens, the pRender output is the spectral irradiance at the surface of the sensor (aka film or retina). The radiance and irradiance formats are those used by ISETBio and ISETCam (scene and optical image).

PBRT integration

ISET3d ray traces using code based on Matt Pharr's PBRT V4 (Physical based ray tracing). Our modifications to PBRT are encapsulated in a Docker image that can be launched as a container. The Docker image can run on some modern GPUs. To do so requires an image generated for a specific Nvidia GPU architecture (e.g. Pascal, Turing, Volta, or Ampere). See the tab at the right for more instructions on how to run the system on a CPU or GPU.

Function naming Convention

We use the prefix pi for functions in this repository. The two letters refer to PBRT and ISET. (The original name of this repository was pbrt2iset). This prefix acts as an informal version of a namespace.

Tutorials

Ongoing activities

  • December 2020 - We are using PBRT Version 3 as the default throughout ISET3d. We no longer support the V2 legacy functions.
  • April 2021 - We reorganized the representation of graphics assets (objects and transformations) into a tree.
  • October 2021 - Led by Zhenyi Liu, we are moving from PBRT V3 to PBRT V4. We have a working version of that implementation; our current research still uses V3.
  • December, 2021 - We are working on remote rendering support using Nvidia GPUs and PBRT V4. Currently this work in progress is in the dockerDev branch of ISET3d-v4
  • January, 2022 - Our default has been moved to PBRT V4. It can now be run locally or remotely, and on either a CPU or a Compute Capability 6 or better Nvidia GPU -- IF you have a Docker image built for that architecture.
  • July, 2022 - We renamed the old iset3d ISET3d-V3 and ISET3d-v4 became ISET3d. Maybe we should have left it as ISET3d-V4.

ISET3d development is led by Brian Wandell's Vistalab group at Stanford University and supported by contributors from other research institutions and industry.

Clone this wiki locally