DReyeVR

Welcome to DReyeVR, a VR driving simulator for behavioural and interactions research.

Submission Video Demonstration (YouTube)

This project extends the Carla 0.9.13 [GitHub] build that adds virtual reality integration, a first-person maneuverable ego-vehicle, eye tracking support, and several immersion enhancements.

Highlights

Ego Vehicle

Fully drivable virtual reality (VR) ego-vehicle with SteamVR integration (see EgoVehicle.h)

• SteamVR HMD head tracking (orientation & position)
  • We have tested with the following devices:

    Device	VR Supported	Eye tracking
    HTC Vive Pro Eye	YES	YES (SRanipal 1.3.1.1)
    Oculus/Meta Quest 2	YES	NO

• Vehicle controls
  • Generic keyboard WASD + mouse
  • Support for Logitech Steering wheel with this open source LogitechWheelPlugin
    • Includes force-feedback with the steering wheel.
    • We used a Logitech G923 Racing Wheel & Pedals
      • Full list of supported devices can be found here though we can't guarantee out-of-box functionality without testing.
• Realistic (and parameterizable) rear & side view mirrors (WARNING: very performance intensive)
• Vehicle dashboard
  • Speedometer
  • Gear indicator
  • Turn signals
• Dynamic steering wheel
  • Adjustable parameters, responsive to steering input
• "Ego-centric" audio
  • Responsive engine revving (throttle-based)
  • Turn signal clicks
  • Gear switching
  • Collisions
• Fully compatible with the existing Carla PythonAPI and ScenarioRunner
  • Minor modifications were made. See Usage.md documentation.
• Fully compatible with the Carla Recorder and Replayer
  • Including HMD pose/orientation & sensor reenactment
• Ability to handoff/takeover control to/from Carla's AI wheeled vehicle controller

Ego Sensor

Carla-compatible ego-vehicle sensor (see EgoSensor.h)

• Eye tracking support through SRanipal Vive eye tracking
  • Note this requires use of the HTC Vive Pro Eye VR headset
  • Eye tracker data includes:
    • Timing information (based off headset, world, and eye-tracker)
    • 3D Eye gaze ray (left, right, & combined)
    • 2D Pupil position (left & right)
    • Pupil diameter (left & right)
    • Eye Openness (left & right)
    • See DReyeVRData.h:EyeTracker for the complete list
  • Eye reticle plotting in real time
• Real-time user inputs (throttle, steering, brake, turn signals, etc.)
• Gaze focus info (3D world hit point, hit actor)
• Image (screenshot) frame capture based on the camera. WARNING: very performance intensive.
• Fully compatible with the LibCarla data serialization for streaming to a PythonAPI client (see LibCarla/Sensor)
  • We have also tested and verified support for (rospy) ROS integration our sensor data streams

Other additions:

• Custom DReyeVR config file for one-time runtime params. See DReyeVRConfig.ini
  • Especially useful to change params without recompiling everything.
  • Uses standard c++ io management to read the file with minimal performance impact. See DReyeVRUtils.h.
• World ambient audio
  • Birdsong, wind, smoke, etc. (See Docs/Sounds.md)
• Non-ego-centric audio (Engine revving from non-ego vehicles)
• Synchronized Replay with per-frame frame capture for post-hoc analysis
• Recorder/replayer media functions
  • Added in-game keyboard commands Play/Pause/Forward/Backward/etc.
• Static in-environment directional signs for natural navigation (See Docs/Signs.md)

Install

See Docs/Install.md to either:

• Install and build DReyeVR on top of a working Carla (0.9.13) build.
• Use our Carla fork with DReyeVR pre-installed.

OS compatibility

OS	VR	Eye tracking	Audio	Keyboard+Mouse	Racing wheel
Windows	Y	Y	Y	Y	Y
Linux	Y	N	Y	Y	N

• While Windows (10) is recommended for optimized VR support, all our work translates to Linux systems except for the eye tracking and hardware integration which have Windows-only dependencies.
  • Unfortunately the eye-tracking firmware is proprietary & does not work on Linux
    • This is (currently) only supported on Windows because of some proprietary dependencies between HTC SRanipal SDK and Tobii's SDK. Those interested in the Linux discussion for HTC's Vive Pro Eye Tracking can follow the subject here (Vive), here (Vive), and here (Tobii).
  • Additionally, the LogitechWheelPlugin we use only has Windows support currently. Though it should be possible to use the G923 on Linux as per the Arch Wiki.

Documentation & Guides

• See Docs/Usage.md to learn how to use several key DReyeVR features
• See Docs/SetupVR.md to learn how to quickly and minimally set up VR with Carla
• See Docs/Sounds.md to see how we added custom sounds and how you can add your own custom sounds
• See Docs/Model.md to see how we added a responsive steering wheel to the vehicle mesh
• See Docs/Signs.md to add custom in-world directional signs and dynamically spawn them into the world at runtime
• See Docs/LODs.md to learn how we tune the Level-Of-Detail modes for vehicles for a more enjoyable VR experience

Citation

If you use our work, please cite the corresponding paper:

@misc{silvera2022dreyevr,
      title={DReyeVR: Democratizing Virtual Reality Driving Simulation for Behavioural & Interaction Research}, 
      author={Gustavo Silvera and Abhijat Biswas and Henny Admoni},
      year={2022},
      eprint={2201.01931},
      archivePrefix={arXiv},
      primaryClass={cs.HC}
}

Acknowledgements

• This project builds upon and extends the CARLA 0.9.13 simulator
• This repo includes some code from CARLA: Computer Vision Center (CVC) at the Universitat Autonoma de Barcelona (UAB) & Intel Corporation.
• This repo includes some code from Hewlett-Packard Development Company, LP. See nvidia.ph. This is a modified diagnostic tool used during development.

Licenses

• Custom DReyeVR code is distributed under the MIT License.
• Unreal Engine 4 follows its own license terms.
• Code used from other sources that is prefixed with a Copyright header belongs to those individuals/organizations.
• CARLA specific licenses (and dependencies) are described on their GitHub