Skip to the content.

license badge   unity 2020.3.20f1   perception 0.9.0-preview.2

📣 PSP-HDRI+ accepted at ICML 2022

❇️ PSP-HDRI+: A Synthetic Dataset Generator for Pre-Training of Human-Centric Computer Vision Models

PSP-HDRI+ Paper     Poster     PSP-HDRI+ Demo Video

Salehe Erfanian Ebadi, Saurav Dhakad, Sanjay Vishwakarma, Chunpu Wang, You-Cyuan Jhang,
Maciek Chociej, Adam Crespi, Alex Thaman, Sujoy Ganguly
Unity Technologies

Citation

@inproceedings{ebadi2022psphdri,
      title={PSP-HDRI+: A Synthetic Dataset Generator for Pre-Training of Human-Centric Computer Vision Models},
      author={Salehe Erfanian Ebadi and Saurav Dhakad and Sanjay Vishwakarma and Chunpu Wang and You-Cyuan Jhang and 
      Maciek Chociej and Adam Crespi and Alex Thaman and Sujoy Ganguly},
      booktitle={First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022},
      year={2022},
}

❇️ PeopleSansPeople v1.0

Paper     macOS and Linux Binaries     Demo Video

Salehe Erfanian Ebadi, You-Cyuan Jhang, Alex Zook, Saurav Dhakad,
Adam Crespi, Pete Parisi, Steven Borkman, Jonathan Hogins, Sujoy Ganguly
Unity Technologies

Summary

Abstract (click to expand) In recent years, person detection and human pose estimation have made great strides, helped by large-scale labeled datasets. However, these datasets had no guarantees or analysis of human activities, poses, or context diversity. Additionally, privacy, legal, safety, and ethical concerns may limit the ability to collect more human data. An emerging alternative to real-world data that alleviates some of these issues is synthetic data. However, creation of synthetic data generators is incredibly challenging and prevents researchers from exploring their usefulness. Therefore, we release a human-centric synthetic data generator PeopleSansPeople which contains simulation-ready 3D human assets, a parameterized lighting and camera system, and generates 2D and 3D bounding box, instance and semantic segmentation, and COCO pose labels. Using PeopleSansPeople, we performed benchmark synthetic data training using a Detectron2 Keypont R-CNN variant. We found that pre-training a network using synthetic data and fine-tuning on target real-world data (few-shot transfer to limited subsets of COCO-person train) resulted in a keypoint AP of 60.37 ± 0.48 (COCO test-dev2017) outperforming models trained with the same real data alone (keypoint AP of 55.80) and pre-trained with ImageNet (keypoint AP of 57.50). This freely-available data generator should enable a wide range of research into the emerging field of simulation to real transfer learning in the critical area of human-centric computer vision.

Synthetic Data Generator

PeopleSansPeople executable binaries come with:

A comparison between our benchmark generated data with PeopleSansPeople and the COCO person dataset.

#train#validation#instances (train)#instances w/ kpts (train)
COCO64,1152,693262,465149,813
PeopleSansPeople490,00010,000>3,070,000>2,900,000

Generated Data and Labels

PeopleSansPeople produces the following types of labels in COCO format: 2D bounding box, human keypoints, semantic and instance segmentation masks. In addition PeopleSansPeople generates 3D bounding boxes which are provided in Unity’s Perception format.

           
Generated image and corresponding labels: 2D bounding box, human keypoints, semantic and instance segmentation masks in COCO format. 3D bounding box annotations are provided separately in Unity Perception format.

Benchmark Results

Here we show a comparison of gains obtained from pre-training on our synthetic data and fune-tuning on COCO person class over training from scratch and pre-training with ImageNet. For each dataset size we show the results of the best performing model.

bbox AP (COCO person val2017)
size of real datascratchw/ ImageNetw/ PeopleSansPeopleΔ / scratchΔ / ImageNet
64113.8227.6142.58+28.76+14.97
641137.8242.5349.04+11.22+6.51
3205752.1552.7555.04+2.89+2.29
6411556.7356.0957.44+0.71+1.35
keypoint AP (COCO person val2017)
size of real datascratchw/ ImageNetw/ PeopleSansPeopleΔ / scratchΔ / ImageNet
6417.4723.5146.40+38.93+22.89
641139.4845.9955.21+15.73+9.22
3205758.6860.2863.38+4.70+3.10
6411565.1265.1066.83+1.71+1.73
keypoint AP (COCO test-dev2017)
size of real datascratchw/ ImageNetw/ PeopleSansPeopleΔ / scratchΔ / ImageNet
6416.4021.9044.43+38.03+22.53
641137.3044.2052.70+15.40+8.50
3205755.8057.5060.37+4.57+2.87
6411562.0062.4063.47+1.47+1.07

Simulated Clothing Appearance Diversity



Top row: our 3D human assets from RenderPeople with their original clothing textures.
Bottom row: using our Shader Graph randomizers we are able to swap out clothing texture albedos, masks, and normals, yielding very diverse-looking textures on the clothing, without needing to swap out the clothing items themselves.

Additional examples







Additional images generated with PeopleSansPeople. Notice the high diversity of lighting, camera perspectives, scene background and occluders, as well as human poses, their proximity to each other and the camera, and the clothing texture variations. Our domain randomization is done here with naïvely-chosen ranges with uniform distributions. It is possible to drastically change the look and the structure of the scenes by varying the randomizer parameters.

Citation

@article{ebadi2021peoplesanspeople,
      title={PeopleSansPeople: A Synthetic Data Generator for Human-Centric Computer Vision}, 
      author={Salehe Erfanian Ebadi and You-Cyuan Jhang and Alex Zook and Saurav Dhakad and 
      Adam Crespi and Pete Parisi and Steven Borkman and Jonathan Hogins and Sujoy Ganguly},
      year={2021},
      eprint={2112.09290},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Source code

Unity Environment Template here

macOS and Linux binaries here

PeopleSansPeople in the press

License

PeopleSansPeople is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.