Home
Beginning January 15, 2022, our lab has stopped development of ISET3d and moved to ISET3d-V4
We will be developing only the newer (V4) repository. We plan to move the relevant portions of these wiki pages over and renaming this repository as ISET3d-V3.
The ISET3d repository is an imaging toolbox that contains code allowing Matlab users to render PBRT files into ISETCam (or ISETBio) scenes and optical images. As an example here is a synthetic image rendered using ISET3d:
We are developing this toolbox to enable vision scientists and engineers to work with quantitative graphics methods, starting by specifying three-dimensional scenes with physical units. Further, because these scenes are synthetic they can be labeled precisely and accurately, and thus be used for machine learning applications.
ISET3d provides the ability to create physically-realistic complex scenes -- with labels -- that accurately represent natural scene properties (illumination, surface reflectance, multiple bounces) which we hope will advance science and engineering.
ISET3d is fully integrated with ISETCam and ISETBio. A goal of the whole ISET software collection, just as with PBRT, is for the code to make the ideas clear. Computational efficiency is valuable, but a secondary concern.
[NOTE: Please let us know if you have a clearer way to implement an idea, or please propose a clarifying comment in the code or wiki. Double thanks if you can speed up a calculation and still keep it clear!]
We assume that ISET3d will be used as a toolbox (library) after being placed on a user's Matlab path. Users should keep their projects in a separate directory, and if you use git, into a separate repository.
While ISET3d relies on pbrt for rendering, more broadly it assumes that you have docker running and uses pre-built docker images that encapsulate pbrt. For ISET3d-v3, those images are fairly simple, but for v4 they contain a number of other important libraries.
The ISET3d function piRead reads a PBRT scene file, and parses the contents into a Matlab class we call a recipe. The user can adjust the recipe properties using recipe.set() to specify how the scene will be rendered. This includes specifying the spatial resolution, number of rays per pixel, the multi-element optics, the material properties, and the positions of objects. Recipe data is then written out to a PBRT scene file using piWrite. Finally, piRender calls a docker image -- currently vistalab/pbrt-v3-spectral -- to render the recipe. Rendering options include generating (a) the radiance or irradiance, (b) the depth map, and (c) pixel-by-pixel labels for the material or object.
When the recipe specifies pinhole optics, piRender returns the spectral radiance of a scene. If the recipe specifies a lens, the pRender output is the spectral irradiance at the surface of the sensor (aka film). The radiance and irradiance formats are those used by ISETBio and ISETCam (scene and optical image).
- t_piIntro.mlx - Illustrates the basics of ISET3d.
We use the prefix pi for functions in this repository. The two letters refer to PBRT and ISET. (The original name of this repository was pbrt2iset). This prefix acts as an informal version of a namespace.
- December 2020 - We are using PBRT Version 3 as the default throughout ISET3d. We no longer support the V2 legacy functions.
- April 2021 - We reorganized the representation of graphics assets (objects and transformations) into a tree.
- October 2021 - Led by Zhenyi Liu, we are moving from PBRT V3 to PBRT V4. We have a working version of that implementation; our current research still uses V3.
- December, 2021 - We are working on remote rendering support using Nvidia GPUs and PBRT V4. Currently this work in progress is in the dockerDev branch of ISET3d-v4