A framework for photorealistic hardware-in-the-loop agile flight simulation using Unity3D and ROS.

FlightGoggles was created by the AgileDrones group at the MIT FAST Lab and will be published in the proceedings of ICRA 2018.

For optimal performance, a GPU with >=1.7GB of VRAM is required. In our experience, a 2GB Nvidia Quadro M1000M mobile workstation GPU is capable of rendering ~30FPS of RGBD images. A desktop 12GB Nvidia Titan Xp is capable of rendering ~120FPS of RGB images and up to 180FPS of grayscale images, depending on the complexity of the scene and render resolution.

GPUs with lower than 1.7GB of VRAM are usable, but not officially supported.

• Download the appropriate FlightGoggles simulation environment binary from the releases page.
• Extract the binary and mark the file as executable (chmod +x <flightGogglesBinary>)

# Install required libraries
sudo apt install libzmqpp-dev libeigen3-dev libopencv-dev
# Clone client bindings
git clone --recursive https://github.com/AgileDrones/FlightGogglesClientBindings.git
# Setup and build
cd FlightGogglesClientBindings
mkdir build
cd build
cmake ../ && make

# Run FlightGoggles Client
cd FlightGogglesClientBindings/build/bin/
./GeneralClient
# Run FlightGoggles Simulation Environment
./<EXTRACTED_FLIGHTGOGGLES_BINARY>

The FlightGoggles simulation environment will prompt the user for the IP address of the client. This can be entered in manually and will be cached for future rendering sessions.

However, if you'd like to define the client IP through CLI, the -client-ip flag can be used (see example below).

./<EXTRACTED_FLIGHTGOGGLES_BINARY> -client-ip "192.168.0.10"

After running either of these commands, you should see 2 OpenCV windows pop up with live feeds of RGBD data. Additionally, the FlightGoggles binary will open a debug render window with concatenated RGBD data.

RGB	Depth

Butterfly_World	Hazelwood_Loft_Full_Day	NYC_Subway

Hazelwood_Loft_Full_Night	Museum_Day	Museum_Day_Small

FlightGoggles is able to load external scenes from over 40 different filetypes including OBJ, DAE, and FBX. Externally loaded scenes must have lighting embedded in their scene files to render correctly. To load an external scene that already has lighting, set the following settings in the FlightGoggles client:

flightGoggles.state.sceneIsInternal = false;
flightGoggles.state.sceneFilename = "/PATH/TO/SCENE/FILE";

If you find this work useful for your research, please cite:

@inproceedings{sayremccord2018visual,
  title={Visual-inertial navigation algorithm development using photorealistic camera simulation in the loop},
  author={Sayre-McCord, Thomas and
  Guerra, Winter and
  Antonini, Amado and
  Arneberg, Jasper and
  Brown, Austin and
  Cavalheiro, Guilherme and
  Fang, Yajun and
  Gorodetsky, Alex and
  McCoy, Dave and
  Quilter, Sebastian and
  Riether, Fabian and
  Tal, Ezra and
  Terzioglu, Yunus and
  Carlone, Luca and
  Karaman, Sertac},
  booktitle={2018 IEEE International Conference on Robotics and Automation (ICRA)},
  year={2018}
}