The project VIS-TWIN aims to develop AI-based strategies for automated point cloud to 3D model transformation for digital twins with minimal user interaction requirements. Relying on advanced AI technologies for object segmentation and 3D modeling, the process of 3D modeling will be tailored for fast modeling while data collection and annotation effort is minimized. By integrating the project core technological developments into an available crowd-native, real-time rendering platform (partner 3dverse), two use cases will be addressed, both strongly benefiting from a fast point cloud to digital twin transformation. By connecting the DS Automotion in-house software to the 3dverse platform, the planning of digital driving courses will be optimized. Virtual safety trainings can be performed in real-scene environments within the pool3 simulation software, enabled by fast virtualization with low-cost consumer devices and connection to the 3dverse platform. To achieve these aims, the project will address the following objectives:

• Development of AI-based object segmentation methods to detect catalogued objects in low- and high-quality point cloud data.

• Creating a workflow transforming point cloud data into accurate 3D models based on the developed object segmentation methods, existing mesh-generation approaches and minimal user interaction.

• Advancing the 3dverse platform to allow fast and accurate 3D model generation relying on the developed workflow.

• Enabling an improved commissioning process (DS Automotion) to significantly reduce the time and cost required to plan digital driving courses for automated guided vehicles (AGVs) and autonomous mobile robots (AMRs).

• Enable immersive training simulations using real-customer data by creating highly realistic and customer-specific training environments for vehicle simulations (Pool3).

These objectives will be achieved by be relying on the following strategies:

• Advanced AI technology: to achieve high-performance object segmentation as the basis for 3D modeling, the project will employ 1. self-supervised learning to pre-train models on large, non-annotated datasets, mitigating the effects of smaller training dataset sizes and sensor variability and 2. 2D foundation model knowledge distillation to avoid overfitting and to increase domain generalization, while uncertainty prediction will improve label aggregation strategies.

• Handling Data Quality and Limited Datasets: advanced preprocessing techniques, such as diffusion- and transformer-based methods will be used to complete and recover missing data from point clouds. Synthetic data and data augmentation techniques will be applied to enhance the diversity and robustness of the training dataset, ensuring the platform can handle varying data quality effectively.

• Integration with existing Systems: The VIS-TWIN developments will be integrated in the 3dverse platform, which will then be connected to existing software tools. DS Automotion will connect their existing commissioning and VDA5050 master control tools to the 3dverse platform, while Pool3 will connect their vehicle simulation tool.

• Scalability and Adaptability: The developed methods will be designed to be scalable and adaptable to new customer data. By mapping detected, class-agnostic object instance representations to a catalog of possible objects, the need for retraining is minimized and only the object catalog needs to be extended to integrate new objects.