The Multimodal Data Analysis with Multi‑Task Deep Learning (MDLMA) project, carried out as part of a consortium led by the University of Lübeck’s Institute for Medical Informatics (IMI) and Institute for Medical Engineering (IMT), focused on developing a unified, domain‑ and modality‑invariant framework for image reconstruction, enhancement, segmentation and registration of heterogeneous biomedical data. Over a three‑year period funded by the German Research Foundation (DFG) under grant number 031L0202B, the consortium produced a series of software tools and algorithms that were released as open‑source code and presented at international conferences.

In the image‑improvement work package (WP2), convolutional neural networks were trained to suppress metal artefacts in computed tomography (CT) scans. A partial‑convolution architecture was shown to outperform standard convolutional layers, yielding cleaner reconstructions on simulated X‑Cat phantom data, although the overall quality still lagged behind conventional interpolation methods. The same network was adapted for direct deep‑learning‑based CT reconstruction (T2.2), a task postponed to a later phase due to consistency issues between raw and reconstructed data. The resulting artefact‑reduction pipeline was integrated into the Maxwell computing cluster at DESY and demonstrated improved signal‑to‑noise ratios in both ex‑vivo and in‑vivo laboratory CT datasets. The work was published in the 2021 Medical Imaging with Deep Learning conference, where the authors reported a conditional generative adversarial network that achieved a 15 % reduction in mean squared error compared with baseline reconstructions.

Segmentation (WP3) and registration (WP4) modules leveraged shared feature representations learned across modalities. The segmentation network achieved a Dice coefficient of 0.87 on a multi‑modal dataset comprising SR µCT, small‑angle X‑ray scattering (SAXS), histology, in‑vivo magnetic resonance imaging (MRI) and CT, and successfully transferred to external datasets without retraining. Registration models incorporated semantic cues from segmentation outputs and learned modality‑invariant embeddings, attaining sub‑millimetre alignment accuracy between ex‑vivo µCT and histology slices. A temporal registration tool was also developed to fuse longitudinal CT and MRI series, enabling consistent multi‑time‑point analysis.

The multi‑task learning work package (WP5) unified these components into a single platform. By jointly optimizing reconstruction, segmentation and registration objectives, the system achieved a 12 % improvement in segmentation Dice scores and a 9 % reduction in registration residuals compared with single‑task baselines. The platform was deployed on the Maxwell cluster, where it processed previously unseen modalities such as SAXS and lab‑CT with minimal performance loss, demonstrating the framework’s generality.

WP6 focused on packaging the entire pipeline into a user‑friendly framework. The resulting software suite, comprising the partial‑convolution artefact‑reduction tool, the GAN‑based quality assessment module, the multi‑modal segmentation and registration networks, and the multi‑task orchestration layer, was released under permissive open‑source licenses on GitLab and GitHub. No patents were filed, but the consortium identified potential pathways for commercialization, including licensing to industry partners and offering training services.

Collaboration across the consortium was structured around six work packages, with IMT contributing 3 person‑months to WP1, 17 person‑months to WP2, 2 person‑months to WP3, 1 person‑month to WP4, 4 person‑months to WP5, and 2 person‑months to WP6; IMI contributed 2 person‑months to WP1, 2 person‑months to WP2, 3 person‑months to WP3, 10 person‑months to WP4, 12 person‑months to WP5, and 2 person‑months to WP6. The project’s deliverables included a common data format, a unified database interface, and a suite of evaluation metrics. The consortium also collaborated with external partners Hereon and DESY to provide computational resources and to facilitate the deployment of the platform in a high‑performance computing environment. The project’s outcomes lay a solid foundation for future research in multimodal medical imaging and open avenues for industrial application of the developed multi‑task deep‑learning framework.