PubMed HealthSearch

SEARCH · PubMed Health

Results for “Benchmark”

Explore indexed PubMed citations for clinical trials, systematic reviews and public health research. Read source abstracts and follow each citation to its original PubMed record.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 recordsLinked to original sources

A complete diploid human genome benchmark for personalized genomics.

Human genome resequencing typically involves mapping reads to a reference genome to call variants; however, this approach suffers from both technical and reference biases, leaving many duplicated and structurally polymorphic regions of the genome unmapped. Consequently, existing variant benchmarks, generated by the same methods, fail to assess these complex regions. To address this limitation, we present a telomere-to-telomere genome benchmark that achieves near-perfect accuracy (i.e. no detectable errors) across 99.4% of the complete, diploid HG002 genome. This benchmark adds 701.4 Mb of autosomal sequence and both sex chromosomes (216.8 Mb), totaling 15.3% of the genome that was absent from prior benchmarks. We also provide a diploid annotation of genes, transposable elements, segmental duplications, and satellite repeats, including 39,144 protein-coding genes across both haplotypes. To facilitate application of the benchmark, we developed tools for measuring the accuracy of sequencing reads, phased variant call sets, and genome assemblies against a diploid reference. Genome-wide analyses show that state-of-the-art de novo assembly methods resolve 2-7% more sequence and outperform variant calling accuracy by an order of magnitude, yielding just one error per 100 kb across 99.9% of the benchmark regions. Adoption of genome-based benchmarking is expected to accelerate the development of cost-effective methods for complete genome sequencing, expanding the reach of genomic medicine to the entire genome and enabling a new era of personalized genomics.

Journal Article

DARKIN: a zero-shot benchmark for phosphosite-dark kinase association using protein language models.

MOTIVATION: Protein language models (pLMs) have emerged as powerful tools for capturing the intricate information encoded in protein sequences, facilitating various downstream protein prediction tasks. With numerous pLMs available, there is a critical need for diverse benchmarks to systematically evaluate their performance across biologically relevant tasks. Here, we introduce DARKIN, a zero-shot classification benchmark designed to assign phosphosites to understudied kinases, termed dark kinases. Kinases, which catalyze phosphorylation, are central to cellular signaling pathways. While phosphoproteomics enables the large-scale identification of phosphosites, determining the cognate kinase responsible for the phosphorylation event remains an experimental challenge. RESULTS: In DARKIN, we prepared training, validation, and test folds that respect the zero-shot nature of this classification problem, incorporating stratification based on kinase groups and sequence similarity. We evaluated multiple pLMs using two zero-shot classifiers: a novel, training-free k-NN-based method, and a bilinear classifier. Our findings indicate that ESM, ProtT5-XL, and SaProt exhibit superior performance on this task. DARKIN provides a challenging benchmark for assessing pLM efficacy and fosters deeper exploration of under-characterized (dark) kinases by offering a biologically relevant test bed. AVAILABILITY AND IMPLEMENTATION: The DARKIN benchmark data and the scripts for generating additional splits are publicly available at: https://github.com/tastanlab/darkin.

Protein Kinases

Protein Language Model Decoys for Target Decoy Competition in Proteomics: Quality Assessment and Benchmarks.

Large-scale proteomics relies heavily on target-decoy competition for false discovery rate estimation in peptide identification, and the performance of this strategy depends strongly on the design of the decoy database. Classical generators such as reversal and shuffling remain widely used. Here, we introduce the first protein language model-based (PLM) decoy generation for peptide identification and benchmark it against classical strategies. We evaluate these approaches using three complementary quality-control layers: sequence-based separability, search-engine-agnostic spectral-space diagnostics, and end-to-end mass spectrometry benchmarks, including pipelines with rescoring. Across these analyses, PLM-based decoys are harder for sequence-only neural networks to distinguish than most classical generators, suggesting fewer obvious sequence-level artifacts. However, this signal is only weakly informative for search performance. Spectral diagnostics further show that short peptides occupy a particularly crowded target-decoy space and are therefore especially prone to local collisions across all generators. In full search pipelines, reverse decoys remain a strong baseline, and current PLM-based generators do not yet provide a clear overall advantage. We therefore view PLM-based decoys not as universal replacements for reverse decoys but as tunable tools for benchmarking, diagnostics, stress testing, and future adaptive decoy optimization, with increasing value as search models become more expressive.

Proteomics

Integrated computational and experimental benchmarking of Bacillus phage endolysins reveals the relationship between peptidoglycan-fragment recognition descriptors and antibacterial performance.

Protein-based antibacterials such as bacteriophage endolysins offer a targeted therapeutic strategy against Gram-positive pathogens. However, prioritizing the most effective candidates from the large sequence diversity available remains a significant challenge. Here we present a standardized computational-experimental benchmarking framework that evaluates seven phage-derived endolysin variants (E1, E2, E3, E7, E10, E12, and E15) identified from Bacillus genomes. We combined molecular docking and residue-level interaction mapping against muramyl dipeptide (MDP), a minimal conserved peptidoglycan motif, with 1000-ns molecular dynamics simulations, MM/PBSA binding free-energy estimation, and matched functional inhibition assays against Staphylococcus aureus and Micrococcus luteus. Computational analyses revealed generally favorable MDP recognition across variants, albeit with notable differences in contact patterns and complex stability profiles. Experimental screening identified E2 as the most potent antibacterial agent against both species, while E7 and E1 performed strongly in selected computational metrics. Integrated analysis showed only modest correlations between computational descriptors of fragment recognition/stability and observed antibacterial performance. This study establishes a practical comparative benchmarking platform for endolysin candidate prioritization, nominates E2 and E7 as promising candidates for further development, and highlights E1 as a potential structural scaffold for rational engineering, while explicitly demonstrating both the utility and the current limitations of using minimal peptidoglycan fragments as proxies for full cell-wall recognition in lysin benchmarking.

Endopeptidases

A unified benchmark of supervised and retrieval-based methods for viral genomic sequence classification.

The rapid growth of genomic sequencing demands fast, accurate, and scalable analysis methods. In viral genomic classification, expanding labeled reference collections can make supervised models costly to update and dependent on fixed label sets, motivating retrieval-based genomic classification as a simpler, more flexible alternative. We present a unified benchmark of supervised and retrieval-based methods for viral genomic sequence classification across three viral classification tasks: hepatitis C virus (HCV) genotyping, COVID-19 discrimination, and human papillomavirus (HPV) genotyping. We compare standard sequence encodings (one-hot, k-mers, FCGR) with dense embeddings (dna2vec, DNABERT). For each representation, we evaluate supervised classifiers (Random Forest, Decision Tree, XGBoost) and retrieval-based classification, where sequence vectors are indexed with FAISS and labels are assigned via similarity-weighted k-NN. Furthermore, we benchmark multiple FAISS index types (Flat, IVF, HNSW, IVFPQ, OPQ) to characterize accuracy-speed-memory trade-offs at scale. The results show that XGBoost and retrieval using Flat or IVF indexes achieve strong classification performance under different computational profiles. Compressed indexes such as IVFPQ and OPQ substantially reduce memory usage, although their accuracy loss depends on the dataset and representation. Overall, supervised XGBoost provides a favorable accuracy-size trade-off, while retrieval-based classification remains competitive and allows labeled reference sequences to be incorporated without retraining a global classifier. This benchmark provides practical guidance for selecting sequence representations, classifiers, and vector-search indexes under different accuracy, memory, and update requirements.

Genome, Viral

LAMBDA: a prophage detection benchmark for genomic language models.

Transformer-based genomic sequence models represent an emerging frontier in computational biology. Yet, their embeddings have not yet shown the same level of predictive power as natural and protein language models, highlighting a gap between current implementations and theoretical promise. Existing benchmarks for DNA language models primarily focus on classifying regulatory elements in eukaryotic genomes, leaving open the fundamental question of whether these models learn sequence-level features across whole genomes. We introduce LAMBDA, a benchmark designed to rigorously evaluate genome language model embeddings through phage-bacteria sequence discrimination across four categories of increasing complexity: probing tasks, fine-tuning assessments, diagnostic tests, and genome-wide prophage detection. Our comprehensive analysis of current genomic language models provides insight into the importance of training data selection relative to model size, the need for domain-specific training, and the capabilities and limitations of genomic language models for detecting prophage sequences. This benchmark represents a challenging genomic annotation task in the bacterial domain and addresses a key computational problem with direct relevance to microbiology and medicine.

Prophages

Benchmarking large language models for extracting biobank-derived insights into health and disease.

Biobank-scale datasets such as the UK Biobank have become foundational resources for advancing biomedical discovery. Yet the complexity and heterogeneity of these resources, spanning genomics, imaging, clinical records, and metadata, pose substantial barriers to access and interpretation. Large Language Models (LLMs) offer a promising avenue for making such datasets more navigable through natural language interfaces. However, the extent to which current general-purpose LLMs can retrieve and synthesize biobank-specific insights has not yet been systematically evaluated. In this study, we present a reproducible, multi-metric evaluation framework to benchmark the capabilities of leading LLMs. We evaluated six leading large language models: Gemini 3 Pro, Claude Opus 4.5, Claude Sonnet 4.5, GPT-5.2, Mistral Large 2, and DeepSeek V3, on four benchmark tasks designed to assess biobank-related knowledge retrieval. We evaluate model performance across six dimensions (semantic accuracy, factual correctness, domain knowledge, reasoning quality, response depth, and biobank specificity) and assessed output consistency using curated UK Biobank references and a robust random baseline. All models outperformed the baseline by 2&#xd7; to 3&#xd7;&#x2009;, with strong statistical separation (p&#x2009;<&#x2009;0.001), confirming meaningful biobank-specific knowledge retrieval. Gemini 3 Pro achieved the highest overall accuracy across tasks such as keyword synthesis, institution recognition, and topic inference, while Claude Sonnet 4.5 demonstrated the most uniform performance across evaluation dimensions. Our benchmark provides a rigorous framework for evaluating LLMs in biomedical settings. Using the UK Biobank as a real-world testbed, we highlight both the capabilities and limitations of current models, measuring their capacity to recall structured biomedical knowledge consistent with authoritative biobank metadata.

Large Language Models

Assessment of Gene Set Enrichment Analysis using curated RNA-seq-based benchmarks.

Pathway enrichment analysis is a ubiquitous computational biology method to interpret a list of genes (typically derived from the association of large-scale omics data with phenotypes of interest) in terms of higher-level, predefined gene sets that share biological function, chromosomal location, or other common features. Among many tools developed so far, Gene Set Enrichment Analysis (GSEA) stands out as one of the pioneering and most widely used methods. Although originally developed for microarray data, GSEA is nowadays extensively utilized for RNA-seq data analysis. Here, we quantitatively assessed the performance of a variety of GSEA modalities and provide guidance in the practical use of GSEA in RNA-seq experiments. We leveraged harmonized RNA-seq datasets available from The Cancer Genome Atlas (TCGA) in combination with large, curated pathway collections from the Molecular Signatures Database to obtain cancer-type-specific target pathway lists across multiple cancer types. We carried out a detailed analysis of GSEA performance using both gene-set and phenotype permutations combined with four different choices for the Kolmogorov-Smirnov enrichment statistic. Based on our benchmarks, we conclude that the classic/unweighted gene-set permutation approach offered comparable or better sensitivity-vs-specificity tradeoffs across cancer types compared with other, more complex and computationally intensive permutation methods. Finally, we analyzed other large cohorts for thyroid cancer and hepatocellular carcinoma. We utilized a new consensus metric, the Enrichment Evidence Score (EES), which showed a remarkable agreement between pathways identified in TCGA and those from other sources, despite differences in cancer etiology. This finding suggests an EES-based strategy to identify a core set of pathways that may be complemented by an expanded set of pathways for downstream exploratory analysis. This work fills the existing gap in current guidelines and benchmarks for the use of GSEA with RNA-seq data and provides a framework to enable detailed benchmarking of other RNA-seq-based pathway analysis tools.

Humans

LAMBDA: A Prophage Detection Benchmark for Genomic Language Models.

Transformer-based genomic sequence models represent an emerging frontier in computational biology. Yet, their embeddings have not yet shown the same level of predictive power as natural and protein language models, indicating a gap between current implementations and theoretical promise. Existing benchmarks for DNA language models primarily focus on classifying regulatory elements in eukaryotic genomes, leaving open the fundamental question of whether these models learn sequence-level features across whole genomes. We introduce LAMBDA, a benchmark designed to rigorously evaluate genome language model embeddings through phage-bacteria sequence discrimination across four categories of increasing complexity: probing tasks, fine-tuning assessments, diagnostic tests, and genome-wide prophage detection. Our comprehensive analysis of current genomic language models provides novel insights into the importance of training data quality relative to model size, the need for domain-specific training, and the application of genomic language models for detecting prophage sequences. This benchmark represents a challenging genomic annotation task in the bacterial domain and addresses a key computational problem with direct relevance to microbiology and medicine.

DNA language model

Comprehensive benchmarking of somatic structural variant detection at ultra-low allele fractions.

Postzygotic mosaicism gives rise to somatic structural variants (SVs) at ultra-low variant allele fractions (VAFs), which pose challenges for detection due to the high-coverage sequencing required and noise introduced by sequencing artifacts. Although somatic SV detection has been extensively studied in cancer, these studies are not directly applicable to the study of tissue mosaicism, as they rely on matched normals, target higher VAF ranges, and are enriched for different types of SVs. We present comprehensive benchmark data and best practices for non-cancer somatic SV detection. We created a synthetic mosaic sample by combining six HapMap individuals at varying proportions, generating allele fractions as low as 0.25%. This sample was sequenced to ~2,300x total coverage using Illumina, PacBio, and Nanopore technologies across multiple sequencing centers. A high-confidence benchmark SV set containing over 21,000 pseudo-somatic insertions and deletions &#x2265;50bp was derived from haplotype-resolved assemblies. We evaluated 12 SV discovery pipelines and identified caller-specific strengths and sequencing platform-specific shortcomings. We find that short read-based approaches show reduced recall for insertions and repeat-associated SVs, whereas long-read sequencing achieves high accuracy throughout the genome, increasing linearly with coverage. The best algorithm's sensitivity exceeded 80% for VAFs &#x2265;4% and 15% for VAFs of 0.5-1% with 60x coverage. The publicly available benchmarking data and comparative analysis of current methods provide a foundation for robust discovery of SV mosaicism in non-cancer tissues..

Journal Article

An open benchmark and language models for AI in aging biology.

Over the past two decades, human aging has been characterized across DNA methylation, transcriptomic, proteomic, and clinical modalities, yet no benchmark evaluates whether AI systems can interpret these heterogeneous data types in the context of aging biology. We introduce LongevityBench, an open suite of 17 tasks spanning five biodata domains, and use it to assess 18 frontier AI systems from six developer teams. Despite recent advances in AI, no single model dominates all tasks, with omics-based age prediction being the hardest task regardless of scale. To test whether these gaps can be closed without frontier-scale resources, we fine-tuned a family of five multitask Longevity-LLMs on domain-specific aging data. The compact (0.6B-9B parameters) Longevity-LLMs matched or exceeded far larger frontier systems on LongevityBench, showing that general-purpose language models can be adapted to structured-omics tasks. We publicly release the benchmark, models, and Longevity Claw, an agentic research interface for aging researchers.

Aging

Essence: A benchmarking-validated transformer framework for early diagnosis of Parkinson's disease using cerebrospinal fluid protein biomarkers.

Parkinson's disease (PD) is a progressive neurodegenerative disorder characterized by motor and non-motor symptoms. The lack of objective molecular biomarkers limits early diagnosis and personalized treatment. Here, we propose Essence, a benchmarking-validated framework integrating cerebrospinal fluid (CSF) proteomics with traditional and deep learning models to identify robust protein signatures for PD. Using data from two independent cohorts, 1266 high-confidence proteins are quantified, among which 178 exhibit differential abundance between PD and healthy controls (HC). Through systematic benchmarking of ten machine learning algorithms and four neural architectures, the Transformer model consistently outperforms alternatives across multiple feature selection strategies, achieving an area under the receiver operating characteristic curve (AUC) of 1.0000 with only 35 features. Functional analyses of the top-ranked 35 proteins reveal enrichment in neuroinflammatory, synaptic, and oxidative stress-related pathways. Importantly, spatial transcriptomic profiling based on the Allen Brain Atlas shows region-specific expression of these biomarkers in PD-relevant brain structures, including the striatum, subthalamic nucleus, hippocampus, and white matter tracts. This anatomical alignment supports the functional relevance of the identified markers and highlights their potential utility in early-stage diagnosis and mechanistic understanding of PD.

Benchmarking

Beyond benchmarking: an expert-guided consensus approach to spatially aware clustering.

Spatial omics technologies have revolutionized the study of tissue architecture and cellular heterogeneity by integrating molecular profiles with spatial localization. In spatially resolved transcriptomics, delineating higher-order anatomical structures is critical for understanding how cellular organization affects function. However, the reliability of current benchmarks of spatially aware clustering (SAC) methods is undermined by their narrow focus on Visium and brain tissue datasets and the incorrect interpretation of manual annotation as ground truth. Here we present SACCELERATOR, a community-driven, extensible framework that standardizes data formatting, method integration and metric evaluation, enabling rapid inclusion of new methods and datasets. Our analysis revealed substantial limitations in the generalizability and reproducibility of SAC methods and shows that anatomical labels commonly used as ground truths are often biased, error prone and unsuitable for benchmarking. Rather than ranking methods, we propose a consensus-guided workflow where descriptive spatial metrics highlight high-entropy regions of method disagreement, enabling targeted feedback for tissue experts. Applied to brain and cancer datasets, this approach uncovered biologically meaningful patterns overlooked by individual SAC methods and manual annotations, highlighting the need for iterative, expert-in-the-loop evaluation.

Benchmarking

Benchmarking large language models for genomic knowledge with GeneTuring.

Large language models (LLMs) show promise in biomedical research, but their effectiveness for genomic inquiry remains unclear. We developed GeneTuring, a benchmark consisting of 16 genomics tasks with 1,600 curated questions, and manually evaluated 48,000 answers from ten LLM configurations, including GPT-4o (via API, ChatGPT with web access, and a custom GPT setup), GPT-3.5, Claude 3.5, Gemini Advanced, GeneGPT (both slim and full), BioGPT, and BioMedLM. A custom GPT-4o configuration integrated with NCBI APIs, developed in this study as SeqSnap, achieved the best overall performance. GPT-4o with web access and GeneGPT demonstrated complementary strengths. Our findings highlight both the promise and current limitations of LLMs in genomics, and emphasize the value of combining LLMs with domain-specific tools for robust genomic intelligence. GeneTuring offers a key resource for benchmarking and improving LLMs in biomedical research.

Benchmark

Benchmarking Assembly-Free K-mer Methods for Species Identification in Complex Plant Groups: A Case Study in Populus.

Species identification in taxonomically complex plant groups is frequently limited by the inadequacy of organellar markers, whose phylogenetic signal is disrupted by cytonuclear discordance and chloroplast capture. Using the taxonomically complex genus Populus as a model, we evaluated an assembly-free k-mer workflow against a curated SNP reference benchmark. Whole-genome resequencing data from 235 Populus individuals were curated to a 202-individual, 34-species reference dataset in which all retained species are strictly monophyletic in a genome-wide SNP analysis. Independent maximum likelihood analyses further confirmed that the 31 non-hybrid backbone species each maintained high-support monophyly, while taxa of documented reticulate origin showed placement patterns consistent with their reticulate histories. ABBA-BABA D-statistics detected widespread residual allele sharing within the backbone, though the strongest signals did not correspond to the species pairs responsible for the few k-mer identification failures. Against this benchmark, complete plastomes showed limited resolution, recovering only 3.0% species monophyly and 71.1% nearest-neighbor assignment. The optimized k-mer workflow, operating directly on raw reads without assembly or alignment, recovered 91.2% species monophyly, 99.0% nearest-neighbor assignment, and 98.0% group-average assignment. K-mer length was the primary accuracy-controlling parameter, with k = 31 falling within a stable accuracy plateau. Distance-based metrics reached near-saturation at 0.2&#xd7; sequencing depth, indicating that low-coverage genome skimming can support scalable nuclear genome-based identification with standard computational resources. K-mer distance heatmaps also flagged unusual genomic affinities in hybrid-origin and outlier samples, providing a rapid screen for subsequent population genomic analyses. These results support assembly-free k-mer distances as an efficient tool for reference-based species identification and sample screening in complex plant groups, with residual limitations concentrated near recently diverged species boundaries. Model-based phylogenomic, coalescent, and network analyses remain necessary for resolving deeper species relationships and detailed introgression histories.

Populus

Developing and Benchmarking One Health Genomic Surveillance Tools for Influenza A Virus in Wastewater.

Influenza A viruses (IAV) remain a persistent One Health threat, and whole-genome sequencing from wastewater offers a promising surveillance tool. However, IAV is at low abundance in wastewater, making it difficult to sequence. We benchmarked four targeted enrichment methods suited for whole-genome sequencing including custom and off-the-shelf amplicon and probe-based methods. Our custom HA tiled-amplicon panel was sensitive, fast, and cost-effective, making it suitable for monitoring low-abundance seasonal variants of known subtypes. However, its reliance on conserved and intact primer-binding sites limited primer design to fewer subtypes. A previously published universal amplicon method targeted all IAV subtypes, but it performed poorly in wastewater due to its reliance on intact genome segments. Probe-capture methods were resilient to RNA degradation and mismatches, potentially enabling broader surveillance and detection of emerging strains. However, probes were costly, labor-intensive, and less sensitive than tiled-amplicon. When testing compatibility of sequencing methods with upstream virus concentration and extraction methods, ultrafiltration-based virus concentration outperformed large-volume direct extraction with all four sequencing methods. This set of benchmarking comparisons and custom panels provides needed information for the translation of IAV genomic sequencing into a routine component of wastewater surveillance.

Journal Article

Benchmarking of Reference-Based Tools for Strain-Level Resolution of Plant Microbiome.

Strain-level identification of each microbe is crucial for understanding its role in the host. Most of the existing tools have primarily been evaluated on human metagenomic datasets, whereas the plant microbiome exhibits greater diversity and complexity and thus poses a challenge in the strain-level resolution of individual microbes. In this study, we conducted a comprehensive benchmarking of available reference-based tools for strain-level resolution of the plant microbiome. We evaluated seven tools on various performance parameters, like computational requirements, F1-score and relative abundances using synthetic datasets comprising microbes known to have strong associations with plants as well as real plant microbiome datasets. Our results demonstrated a better performance of StrainScan on the synthetic data, achieving higher F1-score and more accurate relative abundance estimates as compared to other tools, but its performance declined gradually with increasing strain diversity. However, StrainGE and StrainScan exhibited competitive performance on real plant metagenome data. Overall, though StrainGE exhibited better performance, it was more computationally expensive. However, StrainScan performed better in detecting low-abundance strains. Our findings suggest the comparative suitability of the available tools for the strain-level analysis of plant metagenome data and highlight the need for the development of more efficient and accurate taxonomic classifiers capable of handling the complex plant metagenome data while maintaining computational efficiency.

Microbiota

Benchmarking the OptiSpray-&#x3bc;PAC Workflow against a Traditional Nanospray Capillary Interface for Multiplexed Quantitative Proteomics.

Nanoflow liquid chromatography coupled with tandem mass spectrometry (LC-MS/MS) underpins modern quantitative proteomics, yet the column-to-mass spectrometer interface remains an important yet often underappreciated determinant of analytical depth, sensitivity, and reproducibility. Here, we benchmark an integrated workflow comprising the newly developed OptiSpray ion source and a micropillar array column (&#x3bc;PAC) cartridge against a conventional Nanospray Flex Source with an Accucore resin-packed capillary column. We performed a TMTpro 18-plex experiment across nine human cell lines on a FAIMS Pro-equipped Orbitrap Exploris 480. Following basic-pH reversed-phase fractionation, 12 fractions were analyzed on both workflow configurations under matched chromatographic gradient and acquisition conditions. Across both configurations, we quantified >9000 protein groups with highly comparable quantitative reproducibility and principal component clustering. Direct comparison of protein abundance ratios across cell lines showed agreement (Pearson R2 &#x2248; 0.7-0.8) without systematic bias. These results were achieved without workflow-specific optimization of the OptiSpray-&#x3bc;PAC platform, enabling direct transfer of established acquisition methods. Despite differences in column architecture, both configurations delivered comparable proteome coverage and quantitative fidelity. These findings establish the OptiSpray-&#x3bc;PAC workflow as a standardized alternative to conventional capillary-based interfaces, offering simplified operation while preserving quantitative performance.

Humans