A scalable pipeline to create synthetic datasets from functional–structural plant models for deep learning

Abstract

Abstract In plant science, it is an established method to obtain structural parameters of crops using image analysis. In recent years, deep learning techniques have improved the underlying processes significantly. However, since data acquisition is time and resource consuming, reliable training data are currently limited. To overcome this bottleneck, synthetic data are a promising option for not only enabling a higher order of correctness by offering more training data but also for validation of results. However, the creation of synthetic data is complex and requires extensive knowledge in Computer Graphics, Visualization and High-Performance Computing. We address this by introducing Synavis, a framework that allows users to train networks on real-time generated data. We created a pipeline that integrates realistic plant structures, simulated by the functional–structural plant model framework CPlantBox, into the game engine Unreal Engine. For this purpose, we needed to extend CPlantBox by introducing a new leaf geometrization that results in realistic leafs. All parameterized geometries of the plant are directly provided by the plant model. In the Unreal Engine, it is possible to alter the environment. WebRTC enables the streaming of the final image composition, which, in turn, can then be directly used to train deep neural networks to increase parameter robustness, for further plant trait detection and validation of original parameters. We enable user-friendly ready-to-use pipelines, providing virtual plant experiment and field visualizations, a python-binding library to access synthetic data and a ready-to-run example to train models.