PubMed Health⌕ Search

SEARCH · PubMed Health

Results for “Performance benchmarking”

Explore indexed PubMed citations for clinical trials, systematic reviews and public health research. Read source abstracts and follow each citation to its original PubMed record.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 883 records · Page 49Linked to original sources
PubMed2025

HallmarkGraph: a cancer hallmark informed graph neural network for classifying hierarchical tumor subtypes.

MOTIVATION: Accurate tumor subtype diagnosis is crucial for precision oncology, yet current methodologies face significant challenges. These include balancing model accuracy with interpretability and the high costs of generating multi-omics data in clinical settings. Moreover, there is a lack of validated models capable of classifying hierarchical tumor subtypes across a comprehensive pan-cancer cohort. RESULTS: We present a graph neural network, HallmarkGraph, the first biologically informed model developed to classify hierarchical tumor subtypes in human cancer. Inspired by cancer hallmarks, the model's architecture integrates transcriptome profiles and gene regulatory interactions to perform multi-label classification. We evaluate the model on a comprehensive pan-cancer cohort comprising 11 476 samples from 26 primary cancers with 405 subtypes up to eight levels. The model demonstrates exceptional performance, achieving 5-fold cross-validation accuracy between 85% and 99% for tumor subtypes labeled with increasing details of genomic information. It also shows good generalizability on a validation dataset of 887 samples, assessed using three metrics that consider tumor subtypes at individual, combined, and sample levels. Benchmarking and ablation experiments show that hallmark-based embeddings slightly influence model performance, while the integrated multilayer perceptron plays a significant role in determining classifier accuracy. Additionally, we use the SHAP method to link cancer hallmarks with genes, identifying key features that influence model decisions. Our findings present a biologically informed machine learning framework capable of tracking tumor transcriptomic trajectories and distinguishing inter- and intra-tumor heterogeneity in pan-cancer. This approach holds promise for enhancing cancer diagnostics. AVAILABILITY AND IMPLEMENTATION: HallmarkGraph is accessible at https://github.com/laixn/HallmarkGraph.

Humans↗
PubMed2001

Performance indicators in water and sanitation for developing areas.

There is a move throughout the developed world to introduce benchmarking techniques to the management of water and sanitation systems. The indicators that are being considered for benchmarking purposes tend to cover a broad band of activities that are part of the scope of management of large organisations. Most of the lists that have been prepared do not really cater for the implementation and operation of small rural schemes. The greatest need is the implementation of sustainable schemes for this category of consumer. The implementation of a rural water and sanitation scheme involves the negotiation in the initial stages with the local community representatives of what the community wants and how it will be provided and operated. This may have to be preceded by a period of empowerment so that the negotiations can be conducted by informed people. At the conclusion of the negotiations a business plan can be prepared that should be implemented in association with the community. The project is at this stage mainly in the control of a professional engineer, responsible for the technical interpretation of the business plan and the management of the construction phase. Finally, after commissioning, a water service provider will be responsible for the operation, management and maintenance of the scheme on a sustainable basis. The key actions and activities are identified and performance criteria defined. As communication at the beginning of the project is a critical component, the success of this is confirmed by examination and discussion as to how much has in fact been communicated and understood correctly. Other performance indicators for the operational phase are derived from the conventional ones used in the developed world. Finally, it is important that the participants, do not see the measurement of performance as a threatening situation, but rather a stepping stone to improvement of not only performance, but also of subsequent reward.

Benchmarking↗
PubMed2006

Comparison of Affymetrix GeneChip expression measures.

MOTIVATION: In the Affymetrix GeneChip system, preprocessing occurs before one obtains expression level measurements. Because the number of competing preprocessing methods was large and growing we developed a benchmark to help users identify the best method for their application. A webtool was made available for developers to benchmark their procedures. At the time of writing over 50 methods had been submitted. RESULTS: We benchmarked 31 probe set algorithms using a U95A dataset of spike in controls. Using this dataset, we found that background correction, one of the main steps in preprocessing, has the largest effect on performance. In particular, background correction appears to improve accuracy but, in general, worsen precision. The benchmark results put this balance in perspective. Furthermore, we have improved some of the original benchmark metrics to provide more detailed information regarding precision and accuracy. A handful of methods stand out as providing the best balance using spike-in data with the older U95A array, although different experiments on more current arrays may benchmark differently. AVAILABILITY: The affycomp package, now version 1.5.2, continues to be available as part of the Bioconductor project (http://www.bioconductor.org). The webtool continues to be available at http://affycomp.biostat.jhsph.edu CONTACT: rafa@jhu.edu SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Algorithms↗
PubMed2005

Comparing outcomes of voice therapy: a benchmarking study using the therapy outcome measure.

The quality cycle requires clinicians to assess the outcomes of interventions. Benchmarking is an approach that has been advocated to compare current performance across different services to identify commonalities and significant differences. This article gives the results of a study of outcomes in speech and language therapy (SLT) using the therapy outcome measure (TOM) for patients with voice disorder (dysphonia) comparing outcomes of seven separate speech and language therapy services. The study aimed to identify the similarities and differences in outcomes of care provided by different services. Two hundred and forty patients with dysphonia (age range 3-87.5 years, average 51.9 years) were treated. The results indicated that although there was no significant difference in the profile of the severity of symptomology of patients referred to speech and language therapy in different geographical areas, there was a significant difference in the treatment outcomes across the services and in the stated reason for discharge from treatment. Nevertheless, most patients with dysphonia had a good outcome and this was associated with completion of the course of treatment. There were significant differences in the number of treatment contacts provided by the different services and in the duration (between admission and discharge) of treatment across the services. Benchmarking can provide useful information through use of routinely collected clinical data.

Adolescent↗
PubMedDate not supplied

Identifying best practice through benchmarking and outcome measurement.

Collecting and analyzing various types of data are essential to identifying areas for improvement. Data collection and analysis are routinely performed in hospitals and are even required by some regulatory agencies. Realization of the full benefits, which may be achieved through collection and analysis of data, should be actively pursued to prevent a meaningless exercise in paperwork. Internal historical comparison of data may be helpful but does not achieve the ultimate goal of identifying external benchmarks in order to determine best practice. External benchmarks provide a means of comparison with similar facilities, allowing the identification of processes needing improvement. The specialty of ophthalmology presents unique practice situations that are not comparable with other specialties, making it imperative to benchmark against other facilities where quick surgical case time, efficient surgical turnover times, low infection rates, and cost containment are essential and standard operations. Important data to benchmark include efficiency data, financial data, and quality or patient outcome data. After identifying facilities that excel in certain aspects of performance, it is necessary to analyze how their procedures help them achieve these favorable results. Careful data collection and analysis lead to improved practice and patient care.

Benchmarking↗
PubMed2005

Benchmarking approximate density functional theory. I. s/d excitation energies in 3d transition metal cations.

The performance of a number of different implementations of density functional theory (DFT) for predicting the s/d interconfigurational energies of the 3d transition metal cations is investigated. Systematic comparisons of computed results with experimental data indicate that gradient corrected correlation functionals, like the LYP GGA, efficiently correct the flaws of the LDA, but reveal shortcomings in the treatment of exchange by currently available GGAs. The admixture of exact exchange in hybrid functionals eventually leads to largely reduced errors. Several basis sets available for the 3d elements are tested in combination with the B3LYP functional. Finally, the influence of variations of the admixture of exact exchange is systematically tested. The results reveal that computed s/d excitation energies obtained for the individual ions depend in markedly different ways on the amount of exact exchange admixture and that there is no single optimal and transferable exchange parameter to create a hybrid functional that yields improved results for all ions alike. Several of the recently devised functionals perform as good as or slightly better than the B3LYP functional in the present study. But given the fact that the B3LYP functional has been identified as the most successful DFT method in an overwhelming number of systematic investigations in very many areas of chemical research, there is no persuasive motivation to recommend its replacement by one of the other functionals, as much less is known about their robustness.

Journal Article↗
PubMed2006

Benchmarking matching pursuit to find sleep spindles.

The aim of this study is to evaluate performance of Matching Pursuit (MP) algorithm against visual analysis for automatic sleep spindle (SS) detection in a sample of sleep stages 2-4 and REM pertaining to nine healthy young subjects. MP-SS voltage, frequency and duration characteristics were investigated for the amplitude threshold (AT) that maximized yield between test sensitivity and specificity. Parameter distribution curves were also built for correctly detected (true positive) and false-positive events. For sleep stage 2, MP reached 80.6% sensitivity and specificity for an AT value of 58.8. For all stages together, 81.2% sensitivity and specificity were reached for an AT value of 46.6. Specificity curves were adequate for all stages; sensitivity was lower for S3+4. Sigma frequency range activity with atypical characteristics was detected within REM sleep. Prevalence indexes obtained with MP were much higher than visual prevalence indexes for all stages; similar voltage, frequency and duration distribution curves were obtained for true positive and false positive events. For this sample of young male healthy subjects, the free-ware MP algorithm showed satisfactory performance for SS detection in sleep stage 2 as reported earlier, acceptable performance in sleep stages 3+4, although with lowered sensitivity, and sigma frequency range activity within REM sleep that needs better understanding. Within NREM sleep, correspondence between the MP automatic and the visual method was supported.

Adult↗
PubMed2002

Using comparison charts to assess performance measurement data.

BACKGROUND: In 1997 the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) announced the ORYX initiative, which integrates outcomes and other performance measurement data into the accreditation process. JCAHO uses control and comparison charts to identify performance trends and patterns that are provided to JCAHO surveyors in advance of a health care organization's (HCO's) survey. During the survey, the HCO is asked to explain its rationale for its selection of performance measures, how the ORYX data have been analyzed and used to improve performance, and the outcomes of these activities. CONSTRUCTING COMPARISON CHARTS: A comparison chart, a graphical summary of the comparison analysis, consists of actual (or observed) rates, expected rates, and expected ranges (upper and lower limits) for a given time frame. The expected range describes the degree of certainty that a given point is different from the average score (population). THE USE OF COMPARISON CHARTS: Comparison charts are primarily useful for telling an HCO whether one of its selected performance measures may be evidencing one of the three types of measurement outcomes: exemplary performance, average performance, or substandard performance (indicating an opportunity for improvement). The comparison charts compare an HCO's outcomes to those of its comparison group or to its risk-adjusted data. The charts provide guidance to an HCO about whether it should continue to monitor a process so as to maintain its current level of performance or whether it should try to improve its current performance.

Accreditation↗
PubMed2004

Informed consent and surgeons' performance.

This paper argues that the provision of effective informed consent by surgical patients requires the disclosure of material information about the comparative clinical performance of available surgeons. We develop a new ethical argument for the conclusion that comparative information about surgeons' performance--surgeons' report cards--should be provided to patients, a conclusion that has already been supported by legal and economic arguments. We consider some recent institutional and legal developments in this area, and we respond to some common objections to the use of report cards on the clinical performance of surgeons.

Benchmarking↗
PubMed2025

Benchmarking methods for measuring biosynthetic gene cluster similarity and determination of gene cluster families.

MOTIVATION: Natural products are often produced by a set of biosynthetic enzymes that are encoded by genes clustered together in the producer's genome, referred to as a biosynthetic gene cluster (BGC). The ability to compare and cluster BGCs is essential for several applications, including predicting which bacteria will make a known product and assessing the potential diversity of natural products produced by a set of bacteria. There are multiple methods for comparing and clustering BGCs based on their similarity, but there has been a lack of investigation into how strongly BGC similarity relates to product structural similarity and how these methods perform relative to each other. RESULTS: Using publicly available databases, we developed a benchmark dataset to assess how well different BGC similarity metrics correlate with the structural similarity of their products and how well these methods cluster BGCs. We found that all methods showed moderate correlation between BGC and structural similarity, with correlations improving for more similar BGCs and varying significantly by BGC biosynthetic class. Analysis of outliers revealed some outliers were due to mistakes or omissions in public datasets, while others represented deviation between BGC similarity and product structural similarity. All methods generally performed better on clustering metrics, with BiG-SCAPE performing the best after errors in the public datasets had been corrected. AVAILABILITY AND IMPLEMENTATION: Scripts and data required to reproduce the results are available at https://github.com/aswalker-lab/BGC-clustering-benchmark and processed similarity, clusters, and scaffolds are also available at https://huggingface.co/datasets/allie-walker/BGC-clustering-benchmark. Code is also available at Zenodo: 10.5281/zenodo.17373546.

Multigene Family↗
PubMed2005

The Cochrane Skin Group: promoting the best evidence.

BACKGROUND: The international Cochrane Skin group, established in 1997, organizes, writes and disseminates systematic reviews of therapeutic interventions in dermatology. OBJECTIVE: To introduce the Cochrane Skin Group and what it offers to cutaneous medicine and surgery providers. METHODS: Descriptive review of the structure and output of the Cochrane Collaboration Library and the Cochrane Skin Group. RESULTS: Systematic reviews of randomized controlled trials produced through the Cochrane Skin Group provide a benchmark for evidence summaries for informing clinical decisions in dermatology. CONCLUSION: The work performed by the Cochrane Skin Group is an important component for informing the evidence base for the clinical practice of cutaneous and surgical dermatology.

Benchmarking↗
PubMed2004

Evaluation of several lightweight stochastic context-free grammars for RNA secondary structure prediction.

BACKGROUND: RNA secondary structure prediction methods based on probabilistic modeling can be developed using stochastic context-free grammars (SCFGs). Such methods can readily combine different sources of information that can be expressed probabilistically, such as an evolutionary model of comparative RNA sequence analysis and a biophysical model of structure plausibility. However, the number of free parameters in an integrated model for consensus RNA structure prediction can become untenable if the underlying SCFG design is too complex. Thus a key question is, what small, simple SCFG designs perform best for RNA secondary structure prediction? RESULTS: Nine different small SCFGs were implemented to explore the tradeoffs between model complexity and prediction accuracy. Each model was tested for single sequence structure prediction accuracy on a benchmark set of RNA secondary structures. CONCLUSIONS: Four SCFG designs had prediction accuracies near the performance of current energy minimization programs. One of these designs, introduced by Knudsen and Hein in their PFOLD algorithm, has only 21 free parameters and is significantly simpler than the others.

Computational Biology↗
PubMed1996

Cost-effectiveness of clozapine monitoring after the first 6 months.

BACKGROUND: Clozapine is effective in treating patients with schizophrenia who do not respond to conventional neuroleptic drugs. The drug is unique in that it is available only with a US Food and Drug Administration-mandated system for weekly monitoring of patients' white blood cell counts. No study has been conducted to evaluate the cost-effectiveness of this mandatory monitoring system. METHODS: A benchmark case was established by utilizing cumulative incidence rates of agranulocytosis from a recent study with a large sample of clozapine-treated patients. We assumed a 20% mortality among patients with agranulocytosis, $30.61 in monitoring costs each week, and 14.4 years of remaining life expectancy after detection of agranulocytosis. Based on these bench-mark assumptions, cost-effectiveness ratios in dollars per quality-adjusted life-year were calculated for the first, second, and third 6-month periods during which a patient was receiving clozapine. Sensitivity analyses were performed with more conservative assumptions in 5 alternative scenarios. RESULTS: In the benchmark case, costs per quality-adjusted life-year gained were $61,694, $925,418, and $420,644 for the first, second, and third 6-month periods of clozapine treatment, respectively. In the alternative scenarios, these costs ranged from $7923 to $46,056 for the first 6-month period and from $54,025 to $690,850 for the second and third 6-month periods. CONCLUSIONS: While the costs of monitoring patients with schizophrenia in the first 6-month period of clozapine treatment seem to be justifiable, monitoring thereafter may not be cost-effective because of the very low incidence of agranulocytosis in the later periods.

Adult↗
PubMed2026

Targeting the F17-A Fimbrial gene: An efficient method for the quantitative detection of Escherichia coli F17.

Escherichia coli (E. coli) F17 is one of the leading bacterial causes of diarrhea in farm livestock, which cause huge economic losses and could also pose potential risks to public health. Generally, the monitoring the E. coli F17 is based on the polymerase chain reaction (PCR) and bacteria plate counting method, which were largely limited by the time-consuming nature and susceptibility to detection errors. Hence, there is an urgent need to develop a rapid and quantitative detection method for E. coli F17. In the present study, an E. coli F17 challenge experiment in ovine intestinal epithelial cells (IECs) was employed as an in vitro model. At different post-challenge time points (1 h, 2 h, and 3 h), two conventional methods (bacteria plate counting and microplate method) were conducted as benchmarks to estimate the number of E. coli F17 adhering to the IECs. Additionally, total genomic DNA was extracted and quantitative Real-time PCR (qPCR) was performed to detect the relative abundance of E. coli F17 fimbrial pilin (F17-A) and adhesion (F17-G) genes. Subsequently, statistical analyses, including Pearson's correlation coefficient (PCC) method and linear curve-fitting, were performed to evaluate the correlation between the abundance of F17-A/G genes and the results of the benchmark methods. The results showed that the relative abundances of both genes were highly correlated with the number of E. coli F17 that adhered to the IECs, among them, the F17-A gene showed a stronger correlation with the bacterial counts, exhibiting a correlation coefficient > 0.85. Furthermore, standard curves analyses further confirmed the out-performed quantitative performance of F17-A gene and a significantly stronger correlation with bacterial counts which exhibited an outstanding linear correlation (r = -0.9534, R2 = 0.9252) with amplification efficiency of 101.4%, The results of the present study indicate that targeting fimbrial genetic hallmarks via qPCR is an effective and promising method for E. coli F17 quantification, which could potentially contribute to epidemiological studies and pathogen monitoring in the livestock industry.

Detection↗
PubMed2026

scGPA: an LLM-assisted workflow for directional virtual gene perturbation analysis from single-cell transcriptomes.

BACKGROUND: Existing virtual perturbation methods can often infer directional changes by comparing predicted post-perturbation expression profiles with control cells. However, workflows that directly return direction-specific downstream candidate genes together with confidence scores, evidence support and interpretable summaries remain limited. We developed scGPA, an LLM-assisted workflow system for directional single-cell virtual gene perturbation analysis. METHODS: scGPA starts from raw single-cell RNA sequencing data and performs quality control, normalization, dimensionality reduction, clustering and cell-group selection. It then constructs cell-group-specific wild-type regulatory networks using repeated subsampling, principal component regression (PCR)/Ridge-based network inference and CP tensor denoising. Based on these networks, scGPA simulates dose-aware virtual knockdown of the target gene and applies signed perturbation propagation to estimate the magnitude and direction of downstream transcriptional responses. LLM assistance is used for marker-based cell-type annotation, evidence-guided candidate prioritization and user-facing biological summarization. RESULTS: We benchmarked scGPA across five public Perturb-seq datasets and compared its performance with GEARS, scGPT and a random baseline. The overall correct prediction rate of scGPA was 23.0%, exceeding those of GEARS (20.7%), scGPT (15.1%) and the random baseline (13.6%). These results indicate that scGPA achieved a higher correct prediction rate than the two comparator models and the random baseline. We subsequently evaluated scGPA using a public osteosarcoma single-cell dataset and performed qRT-PCR validation in 143B osteosarcoma cells. Among genes with significant experimental changes, scGPA achieved a directional concordance of 76.9%. When all tested downstream genes were counted, 37.0% were directionally correct, 51.9% showed no significant change and 11.1% changed in the opposite direction. CONCLUSIONS: scGPA provides a practical workflow system for predicting and prioritizing direction-specific downstream transcriptional responses after target-gene perturbation. By integrating single-cell regulatory network inference, signed virtual perturbation and LLM-assisted interpretation, scGPA supports target-gene function inference and downstream mechanistic investigation from single-cell transcriptomic data.

Single-Cell Gene Expression Analysis↗
PubMed2006

State of the art: refinement of multiple sequence alignments.

BACKGROUND: Accurate multiple sequence alignments of proteins are very important in computational biology today. Despite the numerous efforts made in this field, all alignment strategies have certain shortcomings resulting in alignments that are not always correct. Refinement of existing alignment can prove to be an intelligent choice considering the increasing importance of high quality alignments in large scale high-throughput analysis. RESULTS: We provide an extensive comparison of the performance of the alignment refinement algorithms. The accuracy and efficiency of the refinement programs are compared using the 3D structure-based alignments in the BAliBASE benchmark database as well as manually curated high quality alignments from Conserved Domain Database (CDD). CONCLUSION: Comparison of performance for refined alignments revealed that despite the absence of dramatic improvements, our refinement method, REFINER, which uses conserved regions as constraints performs better in improving the alignments generated by different alignment algorithms. In most cases REFINER produces a higher-scoring, modestly improved alignment that does not deteriorate the well-conserved regions of the original alignment.

Algorithms↗
PubMed2003

Quality monitoring for early hearing detection and intervention programs to optimize performance.

The purpose of this chapter is to review the principles of quality monitoring for newborn screening programs at both a hospital and a public health level. The focus is on quality assurance in Early Hearing Detection and Intervention (EHDI). The chapter addresses the need for a systems approach to quality, beginning with the birth screening, moving to follow-up re-screening and on to confirmatory evaluations and life-long intervention and management for the infants identified with permanent hearing loss. Benchmarks are identified to assist those involved in establishing programs or in improving program performance. As the use of electronic information systems in EHDI becomes widespread, the importance of strict computer-based logic rules to assure patient privacy is addressed.

Benchmarking↗