Secure AI in Healthcare: Deploying On-Premise LLMs for Radiology with Data Privacy at the Core

The LLM Revolution in Healthcare: Balancing Innovation and Privacy

      Large Language Models (LLMs) are rapidly transforming various sectors, and healthcare, particularly radiology, stands to gain significantly. These powerful AI tools hold immense potential to streamline daily tasks, from optimizing protocol selection and documentation to enhancing report generation. Imagine LLMs summarizing complex clinical histories, structuring intricate reports, simplifying medical jargon for patient communication, or even drafting essential correspondence. In a field like radiology, where administrative burdens and reporting demands are constantly increasing, such applications could dramatically reduce repetitive documentation, improve overall efficiency, and combat clinician burnout. The rise of sophisticated open-weights LLMs also presents an opportunity for substantial cost savings, potentially offering open-source alternatives to expensive commercial software licenses.

      However, the integration of LLMs into clinical workflows faces a formidable barrier: stringent data protection and information security requirements. Most well-known proprietary LLMs, such as ChatGPT and Gemini, operate via cloud-based Application Programming Interfaces (APIs). This model necessitates that sensitive patient data, often categorized as Protected Health Information (PHI), leaves the hospital network. Such external data transfer raises serious concerns regarding confidentiality, data sovereignty, and compliance with critical regulations like HIPAA and GDPR. Consequently, many healthcare institutions globally prohibit the use of external LLM services with unanonymized clinical data. This challenge highlights the urgent need for secure, on-premise solutions that maintain data integrity within institutional control.

Building an Ironclad Infrastructure: The Isolation-First Approach

      To address the critical need for data security and regulatory compliance, a significant study from the University Hospital Bonn has designed and implemented a novel "isolation-first" architecture for deploying open-weights LLMs on-premise. This pioneering approach prioritizes stringent security measures to ensure that sensitive Protected Health Information (PHI) remains entirely within the hospital's control.

      The system relies on a containerized LLM inference stack, leveraging technologies like Docker Compose for modularity, reproducibility, and operational control. This setup facilitates strict network segmentation, effectively compartmentalizing the system's components to prevent unauthorized access. A core feature is host-enforced egress filtering, implemented through Linux iptables/ip6tables rules, which precisely dictates what outbound network traffic is permitted, preventing any accidental or malicious external connectivity. Active isolation monitoring continuously oversees the system for any unauthorized external communication attempts.

      The network topology employs a sophisticated split-horizon design with three distinct zones. An ingress demilitarized zone (DMZ) handles incoming traffic, allowing connections only from the secure hospital intranet. Core application services, including the OpenWebUI frontend for chat and user management and the vLLM inference backend for the DeepSeek-R1 model, reside within an internal Docker network that intentionally lacks a routed gateway, thus isolating them from external networks. An optional egress DMZ provides narrowly scoped "airlocks" for specific institutional integrations, such as Lightweight Directory Access Protocol (LDAP)-based authentication to the clinic's central user management systems, with all outbound connections explicitly pinned to private target IP addresses. This comprehensive defense-in-depth strategy ensures a highly secure environment for processing even the most sensitive clinical data.

Navigating Regulatory Hurdles: The Governance Pathway

      A crucial aspect of this study was not just the technical innovation but also the successful navigation of complex institutional governance. Deploying an AI system that processes unanonymized Protected Health Information (PHI) within a major university hospital requires meticulous adherence to regulatory frameworks and ethical guidelines. The research team meticulously defined and followed an institutional governance pathway to secure essential approvals.

      This pathway involved comprehensive consultations and explicit agreements from several key stakeholders: clinic management, compliance officers, data protection officers, and information security officers. Each entity played a vital role in reviewing the architecture, assessing potential risks, and ensuring full alignment with internal policies and external regulations like GDPR and HIPAA. Furthermore, the governance pathway mandated robust human oversight and thorough user training for the 22 participating radiologists and residents, ensuring responsible and informed interaction with the LLM system. The success in achieving these approvals underscores the viability of securely integrating advanced AI, proving that with an isolation-first design, the regulatory borders for handling sensitive data can be effectively overcome. This institutional commitment is a blueprint for other healthcare organizations worldwide.

Real-World Application: Pilot Evaluation in Radiology

      To gauge the practical utility and safety of the on-premise LLM, a prospective in-field pilot study was conducted over one week with 22 residents and radiologists. Participants were encouraged to use 10 predefined prompt templates whenever they found them beneficial in their daily work. The system served the open-weights DeepSeek-R1 model, a capable open-source language model, via the vLLM inference backend, with user interaction managed through an intuitive OpenWebUI frontend. The evaluation focused on clinical utility, system stability, and the frequency of critical errors in model outputs.

      The results were largely positive, with the system receiving high ratings for stability and user-friendliness throughout the pilot. Critically, no external connectivity events were monitored, empirically validating the effectiveness of the isolation-first architecture. The study revealed a clear distinction in clinical utility and risk based on task type. "Source text-anchored tasks," such as correcting or simplifying radiology reports, and generating recommendations based on radiology guidelines, received the highest utility ratings. These tasks leverage the LLM's ability to process and reformat existing, verified information. Conversely, "open-ended conclusion generation" based solely on findings demonstrated the highest frequency of critical errors, including clinically relevant hallucinations (generating false information) or omissions (missing crucial details). These findings highlight the importance of carefully defining the scope of LLM applications in clinical settings and emphasizing human oversight for high-stakes, generative tasks.

The Future of Secure Clinical AI: Practical & Reproducible

      The successful design, implementation, and prospective pilot evaluation of this isolation-first, on-premise LLM infrastructure marks a significant milestone for AI in healthcare. By demonstrating the feasibility of securely processing unanonymized Protected Health Information (PHI) within a highly regulated environment, the study paves the way for wider adoption of open-weights LLMs in clinical routines. This innovative architecture, now serving as the foundation for an official service at a German University Hospital with over 10,000 employees, proves that balancing cutting-edge AI with stringent data privacy is not only possible but practical.

      ARSA Technology, an experienced since 2018 AI & IoT solutions provider, understands the critical importance of secure, privacy-by-design systems for enterprise and government clients. While this specific architecture was developed by the University Hospital Bonn, it aligns perfectly with ARSA’s commitment to delivering robust and compliant AI deployments. For organizations seeking similar secure AI capabilities, ARSA offers custom AI solutions and edge AI systems like the ARSA AI Box Series, designed to ensure data remains on-premise and secure for various industries. The deployment package developed by the University Hospital Bonn has even been made publicly available at https://github.com/ukbonn/ukb-gpt, fostering transparency and reproducibility within the medical community. This collaborative spirit and focus on practical, secure deployments are crucial for advancing AI in sensitive sectors.

      Are you ready to explore how secure, on-premise AI solutions can transform your operations while upholding the highest standards of data privacy and regulatory compliance? Discover ARSA Technology’s enterprise-grade AI and IoT offerings by visiting our solutions pages or contact ARSA for a free consultation to discuss your specific needs.

      Source: Nowak, S., Laß, J.-F., Mesropyan, N., Salam, B., Piel, N., Bahaaeldin, M., Block, W., Sprinkart, A. M., Luetkens, J. A., Wulff, B., & Isaak, A. (2026). Secure On-Premise Deployment of Open-Weights Large Language Models in Radiology: An Isolation-First Architecture with Prospective Pilot Evaluation. arXiv preprint arXiv:2604.22768. https://arxiv.org/abs/2604.22768