Fraunhofer Institute for Physical Measurement Techniques IPM
Fraunhofer Institute for Physical Measurement Techniques IPM

Automated interpretation of point clouds

Using AI-based methods from the field of deep learning, we ensure a deep semantic understanding of mobile mapping data, such as point clouds and camera images.

Semantic data analysis

Artificial Intelligence for the automated analysis of 2D and 3D data

Monitoring the condition of traffic routes, buildings, or agricultural land creates enormous quantities of image data and very large 3D point clouds. Currently, such mobile mapping data is still predominantly analyzed manually in a time-consuming and expensive process. That is why Fraunhofer IPM relies on automated analysis using AI. In AI-based semantic segmentation of camera images or point clouds, each pixel or 3D point is assigned to a specific object class. This semantic understanding allows the measurement results to be automatically converted into reduced, vectorized data formats, such as 3D models or GIS-compatible 2D maps, in a subsequent step.

 

 

Semantic segmentation of 2D images and 3D point clouds
© Fraunhofer IPM
Example for the semantic segmentation of object and surface classes of a street scene based on 2D RGB images and 3D point clouds

The Supervised Deep Learning method

We primarily use AI methods from the field of supervised deep learning for our application-specific, high-precision data analysis solutions. In this approach, an algorithm learns to recognize objects based on manually labeled training data. Artificial neural networks (ANNs) are trained using appropriate datasets to accomplish this. An accurate ANN is essential for automated data analysis. ANNs learn the corresponding output patterns for specific input patterns. Based on this "experience" new, unknown input data can be analyzed in real time. ANNs have proven very robust against variations in characteristic colors, edges, or shapes. Data sources for automated object recognition can include 2D camera data, 3D scan data (point clouds), or fused data.

In principle, Supervised Deep Learning can be applied to any type of object or scenario. Such objects can include parked cars, damage to concrete bridges, fastening bolts on railroad tracks, and door and window openings in buildings, for example. The approach is superior to traditional geometric recognition algorithms for 2D image data or 3D point clouds because it readily adapts to slight variations in measurement data, such as colors, lighting, and object shapes.

Optimization and real-time AI

In terms of complex data analysis problems, AI approaches often demonstrate significantly more power than traditional algorithms. However, AI-based solutions require considerably more computing power, often in the form of dedicated accelerators, such as graphics cards. To make the use of AI as cost-effective as possible, we optimize the AI and the entire software chain in accordance with the available hardware.

Depending on the application, this may also involve optimizing for less powerful edge hardware or embedded platforms. In such cases, AI-based interpretation of measurement data can happen directly within the measurement system in real time. This yields immense systemic advantages by eliminating the effort required for data handling, storage, and post-processing. The result is available immediately after the measurement.

 

ANN training and generating training datasets

Expand all Close all

Depending on the application, the input data for training an ANN can be either 3D point clouds from laser scanners, data from photogrammetric or stereo camera systems, or even simple 2D images.

2D image data

Using camera images as ANN training data offers several advantages. Thanks to their very high resolution, they allow for the detection of very fine structures, such as different road surfaces or cracks. While the process of manually annotating image data is time-consuming, it is less complex than annotating 3D point clouds. Additionally, the artificial neural networks we use place fewer demands on the hardware. However, one drawback is that a very large amount of training data is required, especially for complex scenarios with many different object classes. To obtain georeferenced data, the image information must also be localized in 3D space as a separate step using classical geometric operations. This places higher demands on both the calibration and complexity of the measurement systems that provide the data.

3D point cloud data

Point clouds are a demanding data source for AI training, requiring more storage capacity and computing power. Furthermore, annotating point clouds is time-consuming. However, modern point cloud AI architectures can achieve remarkable accuracy with relatively little training data. Also, the overall complexity of the software decreases, and various sources of error, such as projection errors due to inaccurate calibration, are eliminated.

Careful selection of training data and thorough preparation are key to an effective AI solution for automated semantic data analysis. We use proprietary tools and workflows to ensure high-quality training data and thus reliable data analysis results. A specially trained team handles the labeling of smaller datasets. Additionally, our data analysis experts create annotation guidelines that service providers use to annotate larger datasets and perform subsequent quality control.

Synthetic training data and alternative approaches to reducing training data

Creating training data is often the biggest cost driver for AI-based data analysis. We develop methods to specifically reduce the volume of training data. We use sophisticated software libraries to statistically analyze existing training data, remodel it using freely available 3D models, and vary it algorithmically. Then, the entire measurement system and process are simulated in a realistic manner. This enables us to generate additional synthetic yet realistic training data. This synthetic data considerably increases the volume of training data, thereby enhancing the accuracy of the analysis.

In addition, we are investigating how AI methods from the fields of semi-supervised and unsupervised learning can be used to further improve the quality of results. Due to their methodological nature, these approaches rely exclusively on the measurement data. The entire annotation process is eliminated.

To automate the analysis of measurement data, a neural network must be trained as efficiently as possible. This involves optimizing numerous hyperparameters. We have high-performance servers at our disposal for training the ANN that enable the parallelization of computational processes and the processing of very large neural networks.

The KNN must be integrated into an overall software system. It also needs to be supplemented and monitored using classical methods to detect potential AI errors. We cover all software platforms and delivery models and build custom software products.

Ultimately, the accuracy of the overall process determines the quality of automated data analysis. Therefore, we make a point of conducting an end-to-end evaluation of the results based on representative data, as well as a scientifically sound comparison of the analysis results with the actual facts.