SEARCH · PubMed Health

Results for “multimodal learning”

Explore indexed PubMed citations for clinical trials, systematic reviews and public health research. Read source abstracts and follow each citation to its original PubMed record.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2Linked to original sources

PubMed2026

Multimodal artificial intelligence and machine learning in oncology: from data integration to precision cancer care.

Cancer remains a major global health burden, with approximately 20 million new cases and 9.7 million cancer-related deaths reported globally in 2022. While advances in radiological imaging, molecular profiling, and clinical data have enhanced the interpretation of disease progression, the availability of multiple such modalities still does not meet the needs of a large patient population. This narrative review focuses on the role of multimodal artificial intelligence and machine learning in bridging the gap in interpreting heterogeneous modalities to improve risk prediction, prognostic assessment, and treatment decision-making in precision oncology. Multimodal frameworks such as Pathomic Fusion illustrate how complementary histopathological and genomic information can be integrated for cancer diagnosis and prognostic modeling. Multimodal models have demonstrated potential in virtual biopsy, cancer screening, prognostic prediction, radiotherapy planning, intraoperative guidance, and clinical-trial design using digital twins and synthetic control arms. The major limitations of incorporating multimodal artificial intelligence and machine learning in oncology include data heterogeneity, demographic or institutional biases, and reproducibility challenges that hinder translation. Accordingly, appropriate data-governance strategies, fairness audits, and privacy-preserving approaches such as federated learning should be considered where appropriate. Future progress will depend on the development of standardized benchmarking datasets, robust external validation, seamless integration with electronic health records and picture archiving and communication systems, and the implementation of explainable, secure, and clinically validated multimodal artificial intelligence frameworks that support precision oncology in routine clinical practice.

deep learning↗

PubMed2006

Multimodal visualization interface for data management, self-learning and data presentation.

A multimodal visualization software, called the Data Manager (DM), has been developed to increase interdisciplinary communication around the topic of visualization and modeling of various aspects of the human anatomy. Numerous tools used in Radiology are integrated in the interface that runs on standard personal computers. The available tools, combined to hierarchical data management and custom layouts, allow analyzing of medical imaging data using advanced features outside radiological premises (for example, for patient review, conference presentation or tutorial preparation). The system is free, and based on an open-source software development architecture, and therefore updates of the system for custom applications are possible.

Anatomy↗

PubMed2025

Foundation model based multimodal transformer framework for survival analysis in HER2 stratified breast cancer.

Objective. To improve survival prediction for HER2-positive breast cancer by integrating histopathological, molecular, and clinical data using a multimodal transformer framework.Approach. We propose a multimodal transformer framework for breast cancer survival prediction using HER2 stratified (SurvMBC), a foundation model-enhanced architecture that fuses three data modalities: whole-slide images, clinical narratives, and molecular features. Tumor microenvironment features are extracted using a pathology language and image pre-training (PLIP), clinical narratives are processed with BioBERT, and miRNA expression plus DNA methylation data are embedded using Gen2Vec. These representations are integrated through a cross-modal transformer with attention mechanisms for survival prediction.Main results. The model was evaluated on 1,095 HER2-positive breast cancer patients from The Cancer Genome Atlas. SurvMBC achieved a concordance index (C-index) of 0.857 (95% CI: 0.834, 0.880), a low integrated Brier score, and a strong inverse negative binomial log-likelihood. Risk stratification based on model outputs significantly separated high- and low-risk groups (log-rankp< 0.01) and showed strong associations with tumor stage, grade, and hormone receptor status (allp< 0.05).Significance. SurvMBC demonstrates the effectiveness of multimodal fusion in addressing tumor heterogeneity and improving prognostic accuracy. The attention-based integration enables context-aware learning of survival-relevant features across modalities, supporting individualized risk stratification and risk-adaptive treatment planning for HER2 stratified breast cancer patients.

Breast Neoplasms↗

PubMed2026

Artificial intelligence (AI) uses in stereotactic radiosurgery (SRS): diagnosis with brain metastasis (BM) - A systematic review.

BACKGROUND: Brain metastases (BM) are the most common intracranial tumors in adults, and stereotactic radiosurgery (SRS) has become a mainstay of management. However, several diagnostic challenges persist in the SRS pathway, particularly the differentiation of radiation necrosis (RN) from true tumor progression, which conventional MRI and even advanced imaging techniques often cannot reliably resolve. Recent advances in artificial intelligence (AI) offer the potential to address these diagnostic limitations. This systematic review synthesizes current literature on AI applications for MRI-based diagnostic decision support in BM patients undergoing SRS, with a focus on radiomics and deep learning tools for distinguishing RN from progression, classifying molecular and histologic subtypes, and predicting treatment response. METHODS: A systematic review was performed in accordance with PRISMA guidelines. PubMed, Web of Science, and Scopus were searched using a targeted query combining terms related to AI, brain metastasis, diagnosis or imaging, and SRS. After screening 483 records and applying strict inclusion and exclusion criteria, 18 studies published between 2015 and 2025 were included. Data were extracted on study design, cohort characteristics, imaging modality, AI methodology, validation strategy, and reported diagnostic performance. RESULTS: Among the 18 included studies, AI models demonstrated strong performance across diagnostic tasks in the BM-SRS pathway. The differentiation of RN from true tumor progression was the most extensively studied application, addressed by 14 of 18 studies, with reported AUCs ranging from 0.71 to 0.94. Support vector machines, random-forest ensembles, convolutional neural networks, and transformer-based multimodal architectures were widely used. The literature evolved from single-sequence radiomic classifiers in 2018 to multimodal deep learning frameworks fusing imaging with clinical and genomic data in 2025. Contrast-enhanced T1-weighted MRI was the dominant imaging input, and texture-based radiomic features (GLCM, GLSZM, GLDM, and wavelet-derived features) were the most consistently predictive. The highest-performing models reached AUCs of 0.85-0.91 through multimodal integration of imaging with clinical and genomic features, and consistently outperformed expert neuroradiologist read on matched cases. Remaining studies addressed longitudinal segmentation-based detection of local failure and adverse radiation effects, BRAF mutation status in melanoma BM, early Gamma Knife treatment response, and primary tumor histology classification, with more variable performance. CONCLUSION: AI models, particularly those integrating MRI-derived radiomic features with clinical and genomic data, show high accuracy in supporting diagnostic decisions for BM patients treated with SRS. The post-SRS differentiation of radiation necrosis from true tumor progression has reached the greatest level of maturity and is closest to clinical translation, with potential to reduce unnecessary biopsies, personalize surveillance intervals, and rationalize treatment-pathway decisions. Other diagnostic applications, including molecular subtyping and primary tumor histology classification, remain exploratory and require further multicenter validation. Integration of AI tools into multidisciplinary tumor-board workflows, combined with prospective validation and standardized reporting, will be essential to realize the full clinical benefits of AI in SRS for brain metastases.

Humans↗

PubMed2026

A multi-modal survival prediction framework with group-based batch training and structural consistency alignment.

OBJECTIVE: Integrating whole-slide images (WSIs) with transcriptomic profiles is pivotal for enhancing cancer survival prediction. However, the intrinsic gigapixel resolution and variable sequence lengths of WSIs create a fundamental trade-off between training efficiency and the preservation of data heterogeneity in existing frameworks. Furthermore, substantial statistical and structural discrepancies between histological and genomic modalities often impede effective cross-modal alignment and fusion, thereby limiting prognostic accuracy. METHODS: We propose PRISM, an efficient multi-modal learning framework for integrating WSIs with transcriptomic profiles. To reconcile training efficiency with full data heterogeneity, PRISM first stochastically partitions variable-length WSI sequences into a main subset and a complementary residual subset, both of which are packed into fixed-length groups for batch training. The main subset is processed in the main branch, utilizing isolation masking to maintain intra-group sequence independence. Simultaneously, the residual subset is consolidated into "hyperslides" within a residual branch that leverages tailored supervision, effectively capturing inter-slide correlations. Furthermore, PRISM integrates an Informative Token Aggregation (ITA) module to reduce redundancy in WSIs and employs Cross-batch Structural Consistency Alignment (CBSCA) mechanism to enhance inter-modal structural connectivity. Finally, efficient cross-modal feature interaction is achieved through a Low-rank Bilinear Gated Fusion (LBGF) module. Code is available at https://github.com/Alisa2080/PRISM. RESULTS: Compared with existing methods, PRISM achieves the best overall C-index across five TCGA cohorts. On the larger TCGA-BRCA dataset, PRISM requires only 6 hours of training time, substantially reducing computational cost relative to strong multimodal baselines. Furthermore, comprehensive evaluations demonstrate that PRISM achieves the best overall IBS ranking and favorable time-dependent AUC performance at 1, 3, and 5 years, thereby delivering a more favorable trade-off between prognostic performance and computational efficiency. CONCLUSION: PRISM provides a favorable balance between predictive performance, calibration quality, and computational efficiency, highlighting its potential for practical deployment in multimodal survival modeling for computational pathology.

Humans↗

PubMed2025

Study Protocol for HeartMagic: A Prospective Observational Cohort Characterizing Subtypes of Heart Failure With Preserved Ejection Fraction.

BACKGROUND: Heart failure (HF) is a life-threatening syndrome with significant morbidity and mortality. Although evidence-based drug treatments have effectively reduced morbidity and mortality in HF with reduced ejection fraction (EF), few therapies have been demonstrated to improve outcomes in HF with preserved EF. This may be caused by the existence of several HF with preserved EF subtypes that each need different treatments. There is therefore an unmet need for a comprehensive approach to subtype patients with HF with preserved EF. This protocol details the approach employed in the HeartMagic (Heart Failure Studied With a Machine Learning, Genomics, and Imaging Combination) study to address this gap. METHODS: This prospective multicenter observational cohort study will include 500 consecutive patients with HF with preserved EF at 2 Swiss university hospitals, along with 50 age-matched patients with HF with reduced EF and 50 healthy controls. In addition to routine clinical workup, participants undergo genomic, transcriptomic, and metabolomic analyses, and the anatomy, composition, and function of the heart are quantified by comprehensive echocardiography and magnetic resonance imaging. Quantitative magnetic resonance imaging is also applied to characterize the kidney. The primary outcome is a composite of 1-year cardiovascular mortality or rehospitalization. Machine learning-based multimodal clustering will be employed to identify distinct HF with preserved EF subtypes. Statistical analysis will include group comparisons, survival analysis, and integrative multimodal clustering combining clinical, imaging, ECG, genomic, transcriptomic, and metabolomic data to identify and validate HF with preserved EF subtypes. CONCLUSIONS: The integration of comprehensive magnetic resonance imaging with extensive genomic and metabolomic profiling in this study will result in an unprecedented panoramic view of HF with preserved EF and help distinguish functional subgroups, which may provide a basis for personalized therapies.

Aged↗

PubMed2004

A multidisciplinary approach to teach responses to weapons of mass destruction and terrorism using combined simulation modalities.

STUDY OBJECTIVE: To reinforce concepts presented in the lectures; understand the complexity and speed of casualty and information generation during a Weapons of Mass Destruction and Terrorism (WMD/T) event; experience the novelty of combined weapons' effects; recognize the time course of the various chemical, biological, and radiation agents; and make challenging decisions with incomplete and conflicting information. SETTINGS: Two environments simulated simultaneously: one a major trauma center emergency room (ER) with two patient simulators and several human actors; the other an Emergency Operations Command Center (EOC). TARGET AUDIENCE: Students for this course included: clinicians, scientists, military and intelligence officers, lawyers, administrators, and logistic personnel whose jobs involve planning and executing emergency response plans to WMD/T. SIMULATION SCRIPT: A WMD/T attack in Washington, D.C., has occurred. Clinical students performed in their real life roles in the simulated ER, while nonclinical students did the same in the simulated EOC. Six ER casualties with combined WMD/T injuries were presented and treated over 40 minutes. In the EOC, each person was given his or her role title with identification tag. The EOC scenario took cues from the action in the ER via two television (TV) news feeds and telephone calls from other Emergency Operations Assets. PERFORMANCE EXPECTATIONS: Students were expected to actively engage in their roles. Student performances were self-evaluated during the debriefing. DEBRIEFING: The two groups were reunited and debriefed utilizing disaster crisis resource management tools. ASSESSMENT OF EFFECTIVENESS: Students answered an 18-point questionnaire to help evaluate the usefulness and acceptance of multimodality patient simulation. LESSONS LEARNED: Large-scale multimodality patient simulation can be used to train both clinicians and nonclinicians for future events of WMD/T. Students accepted the simulation experience and thought that scenario was appropriately realistic, complex, and overwhelming. Difficulties include the extensive man-hours involved in designing and presenting the live simulations. EOC-only sessions could be staged with only a few video cassette recorders, TVs, telephones, and callers.

Disasters↗

PubMed1992

Systemic treatment for locally advanced breast cancer: what we still need to learn after a decade of multimodality clinical trials.

Multimodality therapy of locally advanced breast cancer with initial chemo-(hormono)-therapy followed by locoregional treatment has become increasingly popular during the past decade. A paucity of large randomised clinical trials leaves the following unanswered questions: does systemic treatment impact on long-term control of distant metastases? What is the best treatment sequence? The most effective drug combination? The optimum treatment duration? Future prospects in the treatment of locally advanced breast cancer include the use of haematopoietic growth factors to increase the dose-intensity of neoadjuvant chemotherapy, the investigation of autologous bone marrow transplantation with high dose chemotherapy on a larger scale, the development of new approaches designed at interrupting the "autocrine loop" of breast cancer local growth factors and the introduction of diphosphonates in the adjuvant systemic therapy.

Antineoplastic Agents↗

PubMed2024

Protocol to perform integrative analysis of high-dimensional single-cell multimodal data using an interpretable deep learning technique.

The advent of single-cell multi-omics sequencing technology makes it possible for researchers to leverage multiple modalities for individual cells. Here, we present a protocol to perform integrative analysis of high-dimensional single-cell multimodal data using an interpretable deep learning technique called moETM. We describe steps for data preprocessing, multi-omics integration, inclusion of prior pathway knowledge, and cross-omics imputation. As a demonstration, we used the single-cell multi-omics data collected from bone marrow mononuclear cells (GSE194122) as in our original study. For complete details on the use and execution of this protocol, please refer to Zhou et al.1.

Deep Learning↗

PubMed2005

Ensembling local learners through multimodal perturbation.

Ensemble learning algorithms train multiple component learners and then combine their predictions. In order to generate a strong ensemble, the component learners should be with high accuracy as well as high diversity. A popularly used scheme in generating accurate but diverse component learners is to perturb the training data with resampling methods, such as the bootstrap sampling used in bagging. However, such a scheme is not very effective on local learners such as nearest-neighbor classifiers because a slight change in training data can hardly result in local learners with big differences. In this paper, a new ensemble algorithm named Filtered Attribute Subspace based Bagging with Injected Randomness (FASBIR) is proposed for building ensembles of local learners, which utilizes multimodal perturbation to help generate accurate but diverse component learners. In detail, FASBIR employs the perturbation on the training data with bootstrap sampling, the perturbation on the input attributes with attribute filtering and attribute subspace selection, and the perturbation on the learning parameters with randomly configured distance metrics. A large empirical study shows that FASBIR is effective in building ensembles of nearest-neighbor classifiers, whose performance is better than that of many other ensemble algorithms.

Algorithms↗

PubMed2026

Multimodal alignment improves generalizability of genomic biomarker prediction in computational pathology.

Computational pathology models that use digitized histopathology whole-slide images have the potential to become a cost-effective and scalable alternative to molecular assays for the prediction of genomic biomarkers, a key task in precision oncology. However, as new genomic biomarkers are discovered or quantified, large, labeled datasets must be prospectively collected to train new models. To address this challenge, we developed multimodal alignment for biomarker learning and generalization (MARBLE), a multimodal contrastive pretraining strategy that integrates structured biomarker knowledge into representation learning of histopathology images. MARBLE aligns histopathology-derived representations with representations of genomic biomarkers generated by a large language model (LLM) and a protein language model (PLM). This biologically informed alignment enables data-efficient generalization to novel, out-of-distribution biomarkers. Using the MSK-IMPACT cohort of over 40,000 patients across multiple biomarker panel versions, we design experiments grounded in real-world data to demonstrate the value of our proposed approach.

CP: computational biology↗

PubMed1986

Distributional expectations and the induction of category structure.

Previous research on how categories are learned from observation of exemplars has largely ignored the possible role of prior expectations concerning how exemplars will be distributed. The experiments reported here explored this issue by presenting subjects with category-learning tasks in which the distributions of exemplars defining the categories were varied. In Experiments 1 and 2 the distributional form of a category was found to affect speed of learning. Learning was faster when a category's distribution was normal than when it was multimodal. Also, subjects in the early stages of learning a multimodal category responded as if it were unimodal. These results suggested that subjects enter category-learning tasks with expectations of unimodal, possibly normal, distributions of exemplars. Experiments 3 and 4 attempted to manipulate subjects' prior expectations by varying the distribution of exemplars in the first of two consecutive category-learning tasks. Learning a multimodal category was influenced by the shape of a previously learned distribution and was facilitated when the earlier distribution was either multimodal or skewed, rather than normal. These results are interpreted as support for a dual-process model of category learning that incorporates the effects of prior expectations concerning exemplar distributions.

Adolescent↗

PubMed2002

Temporal bone dissection simulation--an update.

We report on our continued development of a virtual simulation for temporal bone dissection that provides stereoscopic display, haptic feedback, and aural simulation into a straightforward, comprehensive learning environment. The multimodal interface provides a seamless simulation for non-deterministic drilling and cutting of bone in the surgical context, as well as an intuitive interface for the intelligent tutor for learning regional anatomy. We present novel methodologies for integrating multimodal and multiresolution data sets, including extension to functional and structural segmentation. We will present our initial efforts to validate this environment. Through continued iterations, it is our hope that the system will provide a valuable tool for training future otologic surgeons as well as an environment for the quantitative evaluation of surgical skill.

Aged↗

PubMed2026

Deep learning-based cross-attention fusion of multimodal MRI for survival prediction and risk stratification in IDH-wildtype glioblastoma: a multicenter study.

BACKGROUND: Glioblastoma (GBM) exhibits profound molecular and spatial heterogeneity, complicating prognostic evaluations. While multiparametric MRI provides crucial multidimensional biological information, conventional end-to-end deep learning integration strategies, such as early or late fusion, often fail to capture complex nonlinear cross-modal interactions. We aimed to systematically evaluate a cross-attention fusion (CAF) architecture for GBM survival prediction and quantify its incremental prognostic value relative to existing clinical tools. METHODS: In this multicenter retrospective study, 386 adults with IDH-wildtype, WHO grade 4 GBM were assembled from an institutional cohort (n = 226), the Chinese Glioma Genome Atlas (CGGA, n = 62), and The Cancer Genome Atlas (TCGA, n = 98). Using a unified 3D ResNet-18 backbone, we compared single-modality models, early fusion, late fusion, and CAF on preoperative T1-weighted, contrast-enhanced T1-weighted (T1CE), and T2-weighted MRI, and integrated the resulting deep learning risk score with routine clinical variables through multivariable Cox regression. Performance was assessed using Harrell's C-index, time-dependent AUC, and decision curve analysis. RESULTS: CAF showed numerically higher, more consistent C-index trends than early fusion, late fusion, and single-modality models (pooled C-index 0.629, 95% CI 0.594-0.664), although pairwise differences in time-dependent AUC were not statistically significant. Integrating clinical variables raised the pooled C-index to 0.691 (95% CI 0.660-0.721) in the treatment-era model, with comparable performance across the three cohorts (Local 0.688; CGGA 0.716; TCGA 0.689); a pre-treatment configuration excluding adjuvant therapy yielded a pooled C-index of 0.642. Under leave-one-cohort-out external validation, the combined model retained significant risk stratification in all held-out cohorts (C-index 0.63-0.71; all log-rank P < 0.01), albeit with attenuated discrimination. The deep learning risk score remained independent after multivariable adjustment (HR 1.41 per SD, 95% CI 1.26-1.57; P < 0.001). Kaplan-Meier analysis confirmed significant high- versus low-risk separation in all cohorts, and decision curve analysis showed greater net benefit than clinical-only and deep-learning-only models. CONCLUSION: The CAF-derived risk score offers prognostic information complementary to routine clinical variables, representing a promising noninvasive tool for individualized risk stratification when molecular profiling is incomplete or unavailable; these findings warrant prospective external validation before clinical use.

cross-attention fusion↗

PubMed2004

Synaptic organization in the adult honey bee brain is influenced by brood-temperature control during pupal development.

Recent studies have shown that the behavioral performance of adult honey bees is influenced by the temperature experienced during pupal development. Here we explore whether there are temperature-mediated effects on the brain. We raised pupae at different constant temperatures between 29 and 37 degrees C and performed neuroanatomical analyses of the adult brains. Analyses focused on sensory-input regions in the mushroom bodies, brain areas associated with higher-order processing such as learning and memory. Distinct synaptic complexes [microglomeruli (MG)] within the mushroom body calyces were visualized by using fluorophore-conjugated phalloidin and an antibody to synapsin. The numbers of MG were different in bees that had been raised at different temperatures, and these differences persisted after the first week of adult life. In the olfactory-input region (lip), MG numbers were highest in bees raised at the temperature normally maintained in brood cells (34.5 degrees C) and significantly decreased in bees raised at 1 degrees C below and above this norm. Interestingly, in the neighboring visual-input region (collar), MG numbers were less affected by temperature. We conclude that thermoregulatory control of brood rearing can generate area- and modality-specific effects on synaptic neuropils in the adult brain. We propose that resulting differences in the synaptic circuitry may affect neuronal plasticity and may underlie temperature-mediated effects on multimodal communication and learning.

Animals↗

PubMed2026

H&E to recurrence score: A step forward, but not yet a substitute for genomic testing.

Shamai and colleagues developed a multimodal deep-learning model that predicts Oncotype DX recurrence scores from routine H&E slides and clinicopathological variables in hormone receptor‑positive, HER2‑negative early breast cancer. Validated across the TAILORx trial and six external cohorts (over 5000 patients), the model achieved an AUC of 0.898 for identifying recurrence score ≥26 and recapitulated genomic assay patterns of chemotherapy benefit. Notably, 31% of clinically high-risk postmenopausal women were downgraded to low risk by AI, suggesting potential to reduce overtreatment. However, several limitations preclude immediate clinical substitution for genomic testing. First, intratumoural heterogeneity leads to discordant predictions with unclear management guidance. Second, the model's chemotherapy benefit estimates rely on TAILORx's age-based menopausal surrogates, which may not reflect real-world hormonal status or LHRH agonist use. Third, predictive value in node-positive disease remains untested in randomised datasets such as RxPONDER. Additionally, calibration uncertainty near risk thresholds and global scalability issues (including IHC requirements and digital pathology infrastructure) persist. While this represents a landmark step toward democratising precision oncology, the AI tool should currently serve as a complementary decision aid, with genomic testing remaining the gold standard for intermediate, borderline, or discordant cases.

Breast cancer↗

PubMed2005

Globally multimodal problem optimization via an estimation of distribution algorithm based on unsupervised learning of Bayesian networks.

Many optimization problems are what can be called globally multimodal, i.e., they present several global optima. Unfortunately, this is a major source of difficulties for most estimation of distribution algorithms, making their effectiveness and efficiency degrade, due to genetic drift. With the aim of overcoming these drawbacks for discrete globally multimodal problem optimization, this paper introduces and evaluates a new estimation of distribution algorithm based on unsupervised learning of Bayesian networks. We report the satisfactory results of our experiments with symmetrical binary optimization problems.

Algorithms↗

PubMedDate not supplied

Experiential influences on multimodal perception of emotion.

The impact of 2 types of learning experiences on children's perception of multimodal emotion cues was examined. Children (aged 7-12 years) were presented with conflicting facial and vocal emotions. The effects of familiarity were tested by varying whether emotions were presented by familiar or unfamiliar adults. The salience of particular emotional expressions was tested by contrasting the performance of physically abused and nonabused children. Children exhibited a preference for auditory expressions produced by their mothers but not by strangers. Additionally, abused children were biased to rely on auditory cues when their own abusive mother was expressing anger. These results are discussed in terms of the impact of both typical and atypical early experiences on the development of emotion perception.

Affect↗