PubMed Health⌕ Search

SEARCH · PubMed Health

Results for “multimodal modeling”

Explore indexed PubMed citations for clinical trials, systematic reviews and public health research. Read source abstracts and follow each citation to its original PubMed record.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3Linked to original sources
PubMed2001

Treatment of non-incarcerated sexually compulsive/addictive offenders in an integrated, multimodal, and psychodynamic group therapy model.

Group therapy is a widely accepted model of care for sex offenders. However, most group therapy takes place in prison utilizing a cognitive-behavioral, psychoeducational model of care. Given that over 250,000 sex offenders are in ambulatory treatment, newer models of care may need to be employed to address their specific needs. I present a naturalistic outpatient study involving my private practice of 109 consecutive sex offenders treated over a 10-year period. A multimodal model of care informed by psychodynamic theory is presented with detailed case material from one group session. Preliminary results of recidivism are presented.

Adult↗
PubMed2026

Multimodal Deep Learning and Foundation Models for Early Detection and Forecasting of Plant Diseases.

Plant diseases destroy 20-40% of global food production annually, posing a critical threat to food security for a projected population of 9.7 billion by 2050. Conventional diagnostic approaches relying on expert visual assessment are slow, costly, and unsuitable for modern agricultural scales. While deep convolutional neural networks demonstrated early promise, single-modality, image-centric systems consistently fail under real-world field conditions characterized by variable lighting, co-occurring infections, and cultivar diversity. This review synthesizes a decade of progress across four interconnected frontiers: the evolution of deep learning architectures for plant disease detection; the adaptation of foundation models including CLIP, SAM, and DINOv2 to agricultural contexts; the development of multimodal fusion frameworks integrating imagery, environmental, genomic, and hyperspectral data; and the transition from static disease diagnosis to descriptive comparison of reported metrics, which suggested that multimodal approaches frequently reported improved diagnostic performance relative to corresponding single-modality baselines, although direct cross-study comparison was limited by methodological heterogeneity. A systematic review following PRISMA guidelines identifies eligible comparative studies. Descriptive comparison of reported performance metrics across these studies indicated that multimodal approaches generally achieved higher accuracy and sensitivity than single-modality models, particularly for pre-symptomatic disease detection. Eight critical research gaps are identified, including the absence of a unified agricultural foundation model and limited climate-aware forecasting under non-stationary climate projections. A structured research agenda is proposed to accelerate translation from laboratory performance to globally equitable, field-deployable crop protection systems.

convolutional neural networks↗
PubMed2026

Development and Validation of a Multimodal Clinical, Pathologic, and Genomic Model for Breast Cancer Recurrence.

PURPOSE: To develop and validate a multimodal recurrence-risk model integrating histology, genomic testing, and clinical variables. METHODS: We developed AI-Path, a whole-slide image biomarker for recurrence prediction trained in CALGB 9344, and validated it in three independent cohorts: TAILORx, a multi-site Chicago cohort, and the MDX-BRCA cohort. We then integrated AI-Path with Oncotype DX Recurrence Score (RS), tumor size, and nodal status into a Cox model, PathClinRS, fit using 60% of cases from TAILORx, with the remaining 40% held out for validation. The primary end point was distant recurrence-free interval. Performance was assessed using Harrell's concordance index (C-index) and Kaplan-Meier analyses. RESULTS: A total of 12,418 patients were included. In TAILORx, AI-Path outperformed RS for distant recurrence (C-index, 0.682 vs 0.647; P = .038), driven by superior prediction of late recurrence (0.656 vs 0.567; P < .001). In node-negative disease, PathClinRS outperformed RSClin in the TAILORx fitting (0.72 vs 0.70; P = .016) and validation sets (0.74 vs 0.70; P = .004). In node-positive disease, PathClinRS outperformed RSClinN+ in Chicago (0.94 vs 0.74; P < .001) and MDX-BRCA (0.71 vs 0.66; P = .004) cohorts. Compared with NATALEE eligibility, PathClinRS identified nearly twice as many high-risk node-negative patients while maintaining a comparable 10-year distant recurrence risk (16.7% vs 16.6% per NATALEE eligibility in TAILORx fitting; 21.0% vs 19.4% in TAILORx validation). PathClinRS identified 68% of intermediate risk premenopausal patients as low-risk with no evidence of chemotherapy benefit, compared to only 36% identified as low risk by standard clinicopathologic criteria. CONCLUSION: Digital histopathology provides prognostic information complementary to genomic assays and has the potential to personalize therapy beyond existing clinicogenomic tools.

Journal Article↗
PubMed2003

Model of surgical procedures for multimodal image-guided neurosurgery.

OBJECTIVE: Improvement of the planning stage of image-guided surgery requires a better anticipation of the surgical procedure and its anatomical and functional environment. This anticipation should be provided by acquisition of multimodal medical images of the patient and by a better understanding of surgical procedures. In this paper, we propose improvements to the planning and performance of multimodal image-guided neurosurgery through the use of information models related to neurosurgical procedures. MATERIALS AND METHODS: A new generic model of surgical procedures is introduced in the context of multimodal image-guided craniotomies. The basic principle of the model is to break down the surgical procedure into a sequence of steps defining the surgical script. In the model, a step is defined by an action. The model assigns to each surgical step a list of image entities extracted from multimodal preoperative images (i.e., anatomical and/or functional images) which are relevant to the performance of that particular step. A semantic validation of the model was performed by instantiating the model entities for 29 surgical procedures. RESULTS: The resulting generic model is described by a UML class diagram and a textual description. The validation showed the relevance of the model, confirming the main underlying assumptions. It also provided some leads to improve the model. CONCLUSION: While further validation is needed, the initial benefits of this approach can already be outlined. It should add real value to the different levels of image-guided surgery, from preprocessing to planning, as well as during surgery. Models of surgical procedures can manage image data according to the surgical script, which should lead to better anticipation of surgery through the development of simulation tools. Furthermore, the models may improve the performance of surgery using microscope-based neuronavigation systems by making it possible to adapt both visualization and interaction features of multimodal preoperative images according to the model.

Humans↗
PubMed2026

Advancing cancer detection and treatment using longitudinal routine clinical data.

Cancer management remains fragmented across its continuum, from late-stage diagnosis and salvage therapies to non-personalized surveillance. Here, we present Oncoformer, a unified multimodal transformer model trained on the China Oncology Multimodal Prediction and Surveillance Study (COMPASS) cohort (3.67 million individuals, 17.7 million clinical visits) and validated on independent external cohorts, including the UK Biobank. Oncoformer integrates longitudinal electronic health records with chest X-ray imaging to address multiple clinical tasks: pan-cancer diagnosis (area under the receiver operating characteristic curve [AUROC] = 0.956), future cancer prediction up to 1 year before diagnosis (AUROC = 0.869), tumor stage inference (mean AUROC > 0.90), patient-specific treatment-response forecasting, and recurrence-free survival stratification across ten cancer types (all p < 0.01). Staging predictions were independently validated against postoperative pathological endpoints and shown to converge on core cancer genomic pathways. By translating routine clinical data into a dynamic view of cancer evolution, Oncoformer provides a framework for risk-informed cancer prediction and treatment stratification using routine clinical data.

Humans↗
PubMed2026

Breast Cancer Recurrence Status Assessment in 5 Years Using Multimodal Integrated Learning: A Feasibility Study.

Despite advances in breast cancer detection and treatment, recurrence after curative therapy continues to impact long-term survival and quality of life. Therefore, early identification of high-risk patients is crucial to guide personalized treatment and follow-up strategies. Although genomic assays provide valuable prognostic insights, their high cost and limited accessibility hinder widespread adoption in clinical practice. Recent machine learning or deep learning approaches leveraging clinical, imaging, or multimodal data have shown promise but do not reflect real-world clinical scenarios. This study proposes a deep learning-based multimodal framework for predicting 5-year breast cancer recurrence using routinely collected clinical data. The framework consists of three main components. First, we adopted automated tumor segmentation with MedSAM to extract the tumor region from ultrasound images. The radiomics features are extracted from those tumor regions. Second, report features are extracted using a Med-Contrastive Pre-trained Transformers (MedCPT)-based approach incorporating predefined, clinically informed queries. Third, a multimodal integration model jointly processes image, radiomics, clinical features, and report features through modality-specific branches. The image branch employs the Ultrasound Foundation Model (USFM) as the backbone, while structured tabular data is processed using the FT-Transformer architecture. The features of all branches are fused using a mixture-of-experts (MoE)-based classifier, and the entire model is trained using a progressive fusion training strategy. Experimental results confirm the feasibility of using ultrasound images with tumor mask integration for recurrence prediction and demonstrate the additive value of integrating multiple data modalities through the proposed multimodal integration model. The final model for recurrence prediction achieved an AUC of 0.7540, accuracy of 74.61%, sensitivity of 70.41%, and specificity of 76.44%. This feasibility study's findings underscore the potential of the proposed multimodal deep learning framework to provide accessible, accurate, and generalizable recurrence risk prediction using routinely available clinical data, potentially supporting more informed treatment decisions and personalized post-treatment monitoring in real-world clinical practice.

Breast cancer recurrence↗
PubMed2003

Short convalescence after vaginal prolapse surgery.

OBJECTIVE: Retrospectively to describe the recommended convalescence according to patients who had undergone vaginal prolapse surgery in 1996-98, and prospectively to describe the need for and limiting factors for convalescence after vaginal prolapse surgery in 1999-2000 at a Danish University Hospital. METHODS: The retrospective study included a validated, postal, questionnaire and review of patient files. In the prospective study, we followed consecutive women after vaginal surgery in a fast-track setting using a multimodal rehabilitation model with well-defined recommendations for the convalescence period. RESULTS: In the retrospective study, 188 women (79%) with a median age of 66 years (range, 30-88) answered the questionnaire. They had been recommended a convalescence of median 6 weeks (range, 1-12) for most activities. The subjective recurrence rate was 22% within median 6 months (range, 0-24) after first-time prolapse surgery. In the prospective study, there were 41 consecutive women with a median age of 69 years (range, 44-88). Convalescence was median <1 week for most non-strenuous activities, <2 weeks for light work and <4 weeks for sexual intercourse, sports and work with lifts exceeding 10 kg. Limiting factors were fatigue and pain. The 1-year subjective recurrence rate was 17%. CONCLUSION: Traditionally, recommended convalescence has been median 6 weeks after vaginal prolapse surgery. Convalescence has been shortened to 1-3 weeks with a multimodal rehabilitation model with revised, non-restrictive recommendations. Further studies are necessary to evaluate the impact of different convalescence recommendations on the recurrence of prolapse.

Activities of Daily Living↗
PubMed2003

Models of surgical procedures for multimodal image-guided neurosurgery.

Improvement of image guided surgery systems requires a better anticipation of the surgical procedure. This anticipation may be provided by a better understanding of surgical procedures and/or the use of information models related to neurosurgical procedures. We are introducing a generic model of surgical procedures in the context of multimodal image-guided craniotomies. The basic principle of the model is to break down the surgical procedure into a sequence of steps defining the surgical script. Each step is defined by an action; the model assigns to each surgical step a list of image entities extracted from multimodal preoperative images (anatomical and/or functional images) which are relevant to the performance of that particular step. The model has been built in two phases: creation and consolidation. Besides, a planning software prototype based on the generic model has been built. The resulting generic model is described by an UML class diagram and textual description. Some initial benefits of this approach can already be outlined: improvement of multimodal information management, enhancement of the preparation and the guidance of the surgical act.

Humans↗
PubMed2025

Multimodal deep learning for immunotherapy response prediction and biomarker discovery in non-small cell lung cancer.

OBJECTIVE: Immunotherapy has emerged as a promising treatment for advanced non-small cell lung cancer (NSCLC), but accurately predicting which patients will benefit from it remains a major clinical challenge. To address this, we aim to develop a novel multimodal method, DeepAFM, that integrates histopathology, genomic features, and clinical information to predict patient responses to anti-PD-(L)1 immunotherapy. MATERIALS AND METHODS: A total of 93 patients with advanced NSCLC were included in this study. Histopathological whole-slide images were processed using a self-supervised VQVAE2 for representation learning. PCA and K-means clustering were then applied for dimensionality reduction and feature grouping. Key regions of interest were visualized through permutation importance evaluation and color-coding techniques. The extracted histopathological features, along with genomic alterations and clinical variables, were integrated into the DeepAFM multimodal prediction model. RESULTS: The DeepAFM achieved a high predictive performance with an area under the curve (AUC) of 0.77 (95% confidence interval: 0.69-1.00). Attention-based heatmaps revealed that the model could identify critical pathological patterns, genomic mutations, and clinical indicators associated with patient responses to immunotherapy. DISCUSSION: The integration of multimodal data enabled the model to capture complex interactions among pathology, genomics, and clinical characteristics, enhancing the interpretability and predictive power of immunotherapy response prediction. The visualization techniques facilitated the identification of biologically meaningful features and potential biomarkers. CONCLUSION: This study demonstrates the effectiveness of the DeepAFM in predicting responses to immunotherapy in advanced NSCLC. The approach not only improves prediction accuracy but also provides valuable insights for personalized treatment strategies and biomarker discovery.

Humans↗
PubMed2002

Precise modelling of the eye for proton therapy of intra-ocular tumours.

A new method is described that allows precise modelling of organs at risk and target volume for radiation therapy of intra-ocular tumours. The aim is to optimize the dose distribution and thus to reduce normal tissue complication probability. A geometrical 3D model based on elliptic shapes was developed that can be used for multimodal model-based segmentation of 3D patient data. The tumour volume cannot be clearly identified in CT and MR data, whereas the tumour outline can be discriminated very precisely in fundus photographs. Therefore, a multimodal 2D fundus diagram was developed, which allows us to correlate and display simultaneously information extracted from the eye model, 3D data and the fundus photograph. Thus, the connection of fundus diagram and 3D data is well-defined and the 3D volume can be calculated directly from the tumour outline drawn onto the fundus photograph and the tumour height measured by ultrasound. The method allows the calculation of a precise 3D eye model of the patient, including the different structures of the eye as well as the tumour volume. The method was developed as part of the new 3D treatment planning system OCTOPUS for proton therapy of ocular tumours within a national research project together with the Hahn-Meitner-Institut Berlin.

Eye Neoplasms↗
PubMed2005

Quantum-statistical analysis of multimode far-infrared and submillimeter-wave astronomical interferometers.

We present a detailed quantum-statistical model of multimode far-infrared and submillimeter-wave astronomical interferometers. The scheme identifies explicitly the optical modes associated with each telescope and uses these to trace the quantum-statistical properties of the field from a source through the telescopes, through the beam combiners, and onto the detectors. The scheme can be used with any optical configuration, and elegant expressions result for the average rate at which photons are detected by the pixels of an imaging array, the mean-square fluctuations in the rates, and the correlations between the fluctuations in the rates of different pixels. Numerous extensions to the basic technique are possible.

Journal Article↗
PubMed2007

A multicompartment vascular model for inferring baseline and functional changes in cerebral oxygen metabolism and arterial dilation.

Functional hemodynamic responses are the composite results of underlying variations in cerebral oxygen consumption and the dilation of arterial vessels after neuronal activity. The development of biophysically based models of the cerebral vasculature allows the separation of the neuro-metabolic and neuro-vascular influences on measurable hemodynamic signals such as functional magnetic resonance imaging or optical imaging. We describe a multicompartment model of the vascular and oxygen transport dynamics associated with stimulus-driven neuronal activation. Our model offers several unique features compared with previous formulations such as the ability to estimate baseline blood flow, volume, and oxygen consumption from functional data. In addition, we introduce a capillary compliance model, arterial and venous oxygen permeability, and model the dynamics of extravascular tissue oxygenation. We apply this model to multimodal optical spectroscopic and laser speckle imaging of the rat somato-sensory cortex during nine conditions of whisker stimulation. By fitting the model using a psuedo-Bayesian framework to incorporate multimodal observations, we estimate baseline blood flow to be 94 (+/-15) mL/100 g min and baseline oxygen consumption to be 6.7 (+/-1.3) mL O(2)/100 g min. We calculate parametric, linear increases in arterial dilation (R(2)=0.96) and CMRO(2) (R(2)=0.87) responses over the nine conditions. Other parameters estimated by the model include vascular transit time and volume reserve, oxygen content, saturation, diffusivity rate constants, and partial pressure of oxygen in the vascular compartments and in the extravascular tissue. Finally, we compare this model to earlier work and find that the multicompartment model more accurately describes the observed oxygenation changes when compared with a single compartment version.

Arteries↗
PubMed2003

A virtual environment for investigating schizophrenic patients' characteristics: assessment of cognitive and navigation ability.

Patients with schizophrenia have thinking disorders such as delusions or hallucinations because they have a deficit in the ability to systematize and integrate information. Therefore, they cannot integrate or systematize visual, auditory, and tactile stimuli. The multimodal integration model of the brain can provide a theoretical background from which one can approach multimodal stimulus integration. In this study, we suggest a virtual reality system for the multi-modal assessment of cognitive ability of schizophrenia patients. The virtual reality system can provide multimodal stimuli, such as visual and auditory stimuli, to the patient and can evaluate the patient's multimodal integration and working memory integration abilities by making the patient interpret and react to multimodal stimuli, which must be remembered for a given period of time. The clinical study showed that the virtual reality program developed is comparable to those of the Wisconsin Card Sorting Test (WCST) and the Standard Progressive Matrices (SPM), and it provides some information related to the schizophrenic patients' behavior in 3D virtual environment.

Adult↗
PubMed2001

Large strain behaviour of brain tissue in shear: some experimental data and differential constitutive model.

In this paper, some experimental measurements of the behaviour of bovine brain tissue under large shear strains in vitro are reported, and a constitutive model which is consistent with the data is developed. It was determined that brain tissue is not strain-time separable, showing slower relaxation at higher strains, and that the stresses in shear are not linear with increasing shear strain. The new constitutive model is a differential model, including both an "elastic" term, of the Mooney type and a nonlinear viscoelastic term. The latter allows for the change in relaxation behaviour with strain, by modifying an upper convected multimode Maxwell model with a damping function. The model shows good agreement with the experimental shear results and could be used to describe other types of data.

Animals↗
PubMed2026

Integrating metagenomic next-generation sequencing into a multimodal diagnostic framework for spinal infection: enhancing etiological identification and clinical prediction.

BACKGROUND: Spinal infection (SI) remains diagnostically challenging because of heterogeneous etiologies, nonspecific clinical manifestations, and the limited sensitivity of conventional microbiological approaches, particularly following empirical antimicrobial exposure. Although metagenomic next-generation sequencing (mNGS) enables unbiased pathogen detection, its incremental clinical value beyond pathogen identification and its role within integrated diagnostic strategies remain incompletely established. METHODS: We retrospectively analyzed 208 consecutive patients with suspected SI between August 2022 and August 2025. Final diagnoses were established using a multidisciplinary-adjudicated composite reference standard incorporating clinical, radiological, microbiological, and histopathological evidence. The diagnostic performance of mNGS was compared with conventional culture and histopathology. Furthermore, multimodal predictive models integrating clinical variables and microbiological information were developed using L1-regularized logistic regression. RESULTS: In the comparative cohort, mNGS achieved a significantly higher diagnostic yield than culture (66.5% vs. 27.41%, P < 0.001). Among confirmed SI cases, mNGS demonstrated higher sensitivity than conventional culture (91.67% vs. 40.15%, P < 0.001). mNGS identified a substantially broader pathogen spectrum, ranging from fastidious organisms such as Mycobacterium tuberculosis and Brucella to rare pathogens including Talaromyces marneffei and Coxiella burnetii, and maintained robust sensitivity (98.2%) despite prior antibiotic exposure. While an integrated clinical model achieved an AUC of 0.916, mNGS as a standalone modality provided superior discriminative power (AUC = 0.889) compared to histopathology (AUC = 0.836), the Conventional Biomarker Model (AUC = 0.742), and culture (AUC = 0.693). CONCLUSIONS: mNGS is a high-yield diagnostic tool for spinal infection, particularly in culture-negative and antibiotic-pretreated scenarios. Integrating mNGS into a multimodal clinical framework facilitates etiological clarity and precision antimicrobial therapy.

Humans↗
PubMed2026

A multi-modal survival prediction framework with group-based batch training and structural consistency alignment.

OBJECTIVE: Integrating whole-slide images (WSIs) with transcriptomic profiles is pivotal for enhancing cancer survival prediction. However, the intrinsic gigapixel resolution and variable sequence lengths of WSIs create a fundamental trade-off between training efficiency and the preservation of data heterogeneity in existing frameworks. Furthermore, substantial statistical and structural discrepancies between histological and genomic modalities often impede effective cross-modal alignment and fusion, thereby limiting prognostic accuracy. METHODS: We propose PRISM, an efficient multi-modal learning framework for integrating WSIs with transcriptomic profiles. To reconcile training efficiency with full data heterogeneity, PRISM first stochastically partitions variable-length WSI sequences into a main subset and a complementary residual subset, both of which are packed into fixed-length groups for batch training. The main subset is processed in the main branch, utilizing isolation masking to maintain intra-group sequence independence. Simultaneously, the residual subset is consolidated into "hyperslides" within a residual branch that leverages tailored supervision, effectively capturing inter-slide correlations. Furthermore, PRISM integrates an Informative Token Aggregation (ITA) module to reduce redundancy in WSIs and employs Cross-batch Structural Consistency Alignment (CBSCA) mechanism to enhance inter-modal structural connectivity. Finally, efficient cross-modal feature interaction is achieved through a Low-rank Bilinear Gated Fusion (LBGF) module. Code is available at https://github.com/Alisa2080/PRISM. RESULTS: Compared with existing methods, PRISM achieves the best overall C-index across five TCGA cohorts. On the larger TCGA-BRCA dataset, PRISM requires only 6&#xa0;hours of training time, substantially reducing computational cost relative to strong multimodal baselines. Furthermore, comprehensive evaluations demonstrate that PRISM achieves the best overall IBS ranking and favorable time-dependent AUC performance at 1, 3, and 5&#xa0;years, thereby delivering a more favorable trade-off between prognostic performance and computational efficiency. CONCLUSION: PRISM provides a favorable balance between predictive performance, calibration quality, and computational efficiency, highlighting its potential for practical deployment in multimodal survival modeling for computational pathology.

Humans↗