Creation of synthetic training data

Synthetic training data
© Fraunhofer IPM
Fig. 1: Digital 3D model of a road scene
Synthetic training data for ANNs
© Fraunhofer IPM
Fig. 2: The 3D model is used to create a photo-realistic rendering which simulates a real camera image.
Synthetic training data for ANNs
© Fraunhofer IPM
Fig. 3: Based on the same data, a segmentation is automatically generated, where every pixel of the camera image is assigned to an object class.

Using ANNs in a wide range of applications

Artificial neural networks (ANNs) are capable of the fully automated interpretation of measurement data. This makes them a central component in applying innovative process chains in 3D measurement technology. Currently, ANNs tend to be prepared for an interpretation task using manually-generated training data. This time-consuming and costly process is often the limiting factor in the use of such networks. Fraunhofer IPM develops software which automatically creates synthetic training data from three-dimensional models. This method for generating the input data to train the ANN is much more cost-effective. In turn, there is less financial risk associated with piloting the application of an AI-based data interpretation, making the automated interpretation of measurement data affordable.

From 3D model to simulated measurement data: training networks in virtual space

Synthetic training data is based on a three-dimensional dataset which creates a virtual representation of the surroundings, e.g. a typical city scene (Fig. 1). This specially constructed digital 3D model world can be used to create a photo-realistic depiction (Fig. 2) which simulates a real camera image and therefore the parameters of existent measuring devices. In addition, a fully-automated segmentation of the 3D model can be generated, in which an object class is assigned to each pixel of the camera image (Fig. 3). As a scene is constructed from objects which have previously been individually defined and are therefore already known, the segmentation is also already included in the data. This eliminates the need for the manual assignment which had been performed previously. 3D models allow for the production of measurement data, for example camera images, as well as segmented images – the training data – with 100% accuracy, while the manual process is always prone to error. Synthetically producing training data is therefore more efficient and produces more accurate results than manually annotating data. In addition, the training data can be generated from different, arbitrary perspectives within the space at hand. Thus, diverse applications using different imaging perspectives – for instance from the air, the road or the sidewalk – can be realized using just one dataset.

A deep understanding of the process chain is key

When rendering the scenes, we draw on our comprehensive expertise in the fields of modeling and measurement technology. In addition to the 3D geometry, the material properties of the surfaces, the lighting conditions and – when outside – the weather conditions also play a role, as well as other properties depending on the application. These can also include dynamic properties, such as the movement of objects in the 3D scene. To ensure a reliable and efficient interpretation of the data, the measurement system used in the later application must also be considered. Ultimately, the 3D models and all available additional information are used to create realistic virtual measurement data for different measuring devices.

Measurement data created with mobile mapping systems primarily exist as 3D point clouds, 2D camera images or 360° panoramic images. Laser scanners and cameras are used to collect the measurement data. The task of modeling is then to reconstruct the device-specific characteristics: How does the measurement system interact with the surrounding environment? To this end, it is necessary to algorithmically reproduce the interaction of the light pulses (sunlight, ambient light, artificial lighting, laser beam) with each surface material as well as the processes in each measuring device (scanner, camera) in sufficient depth.

To develop and apply ANN training software as efficiently as possible requires a careful balance between the most physically exact reproduction and an acceptable approximation of a given phenomenon based on heuristics. This requires extensive knowledge of each imaging process and the state of the art as regards the algorithmic reproduction of such processes. Fraunhofer IPM has a library of algorithms which make it possible to create a wide range of 3D scenes with different properties from a modular system of generic 3D model components through parameterization.