Is a Win-Win possible? Achieving pareto-optimal privacy-utility balance in fine-tuned genome language model embeddings against embedding reconstruction attacks.
MOTIVATION: Genomic data is among the most sensitive categories of personal information, and the growing adoption of language models for sequence analysis raises significant privacy concerns. Prior work demonstrated that embeddings from general-purpose language models adapted for genomic sequences leak substantial single-nucleotide information under reconstruction attacks, and that fine-tuning embeddings can reduce this vulnerability at certain positions. However, three critical questions remain unaddressed: (i) whether privacy-utility tradeoffs are inherent constraints or configuration-dependent phenomena; (ii) whether genomic-specialized models such as DNABERT-base and Nucleotide Transformer exhibit different vulnerabilities than adapted general-purpose models; and (iii) how to statistically validate whether observed privacy improvements represent meaningful gains. Addressing these gaps is essential for guiding model selection in privacy-sensitive genomic applications. RESULTS: We systematically evaluated 13 transformer architectures, 9 general-purpose and 4 genomic-specialized, under position-specific embedding reconstruction attacks. We assessed the vulnerabilities of both pre-trained and fine-tuned models to the single-nucleotide inference-reconstruction attack using our new metrics, including error-based privacy gain and Pareto dominance scores, and statistically validated the results via paired t-tests. XLNet-Large achieved the best observed privacy protection among all evaluated models (+19.5% mean privacy gain) while maintaining competitive prediction performance. General-purpose models outperformed genomic-specialized models in 56% of pairwise comparisons. Tokenization strategy, rather than domain specialization, emerged as the primary determinant of the privacy-utility balance. These findings provide evidence-based guidance for selecting models in privacy-sensitive short-window genomic applications. All privacy claims in this work are specific to position-wise embedding reconstruction attacks and do not extend to other privacy risks, such as membership inference or training data extraction, which may respond differently to fine-tuning. AVAILABILITY AND IMPLEMENTATION: The code is publicly available at https://github.com/AnonymousISCBConf/Win-Win-Privacy-Utility-Analysis.