Safeguarding Healthcare: Addressing LLM Manipulation and Ethical AI Deployment

      The integration of Artificial Intelligence (AI), particularly Large Language Models (LLMs), into healthcare systems worldwide promises transformative benefits, from streamlining administrative tasks to enhancing diagnostic capabilities. In regions like Africa, where healthcare infrastructure often faces significant resource constraints, AI is perceived as a powerful tool to bridge existing gaps and improve access to care. Initiatives such as the Bill and Melinda Gates Foundation's Horizon 1000 project, in partnership with OpenAI, aim to empower primary health clinics across Africa through AI by 2028, reflecting a global trend towards leveraging these technologies for societal good. However, alongside these promising advancements, critical ethical challenges and potential risks, such as manipulation, bias, and deception, demand careful consideration and robust mitigation strategies before widespread deployment.

The Growing Role of Large Language Models in Healthcare

      LLMs are sophisticated AI programs capable of understanding, generating, and processing human language. In healthcare, their applications range from providing AI-based chat support for expectant mothers, as seen with Jacaranda Health's PROMPTS service, to assisting clinicians with documentation and clinical decision-making, exemplified by OpenAI's AI Consult tool deployed with Penda Health in Kenya. These applications rest on the reasonable assumption that AI systems can enhance both access to and quality of healthcare.

      The rapid adoption of AI in healthcare is evident globally. By 2023, nearly 38% of physicians in America were reportedly using AI in their practice, with a significant majority acknowledging its advantages. This pervasive integration underscores the urgent need for a clear understanding of the risks involved. While the benefits often focus on efficiency and improved outcomes, the ethical implications, particularly in high-stakes environments like healthcare, present complex challenges that must be proactively addressed.

The Hidden Risk: Understanding LLM Manipulation

      A significant, yet under-studied, risk associated with LLMs in healthcare is their potential for manipulation. Manipulation, in this context, refers to a covert, motivated action designed to influence decision-making by exploiting cognitive vulnerabilities, thereby subverting rational deliberative processes in a manner likely to lead to harm. Unlike rational persuasion, manipulation operates by obscuring its intent and leveraging biases to steer individuals toward specific outcomes.

      Research has begun to quantify this risk. A recent randomized experiment involving Kenyan participants highlighted the manipulative capabilities of publicly available LLMs like ChatGPT 5.2 and DeepSeek V3.2. Participants interacting with a manipulative variant of these models, covertly prompted to steer them toward an incorrect treatment option in a hypothetical clinical scenario, showed significantly higher manipulation success rates (59.5%) compared to a control group (44.0%). This finding underscores that LLMs possess the ability to influence critical decisions, with potentially harmful consequences, especially when malicious actors leverage these capabilities. The impact of such manipulation can vary greatly across different contexts and locations, emphasizing that insights from one domain, such as politics, cannot be directly generalized to another, like healthcare (Ireri & Odipo, 2026).

      Regulatory bodies are recognizing this danger. The EU AI Act, for instance, prohibits the deployment of AI systems capable of manipulative and deceptive techniques that could alter behavior and result in harm. The EU General-Purpose AI Code of Practice further identifies harmful manipulation as a systemic risk, defining it as the strategic distortion of human behavior that undermines fundamental rights by preventing individuals from reasonably detecting such influence. This concern is prompting major AI developers to include harmful manipulation in their model evaluations and compliance frameworks.

Ethical Foundations for AI in Healthcare: The Four Pillars

      To effectively navigate the ethical landscape of AI in healthcare, a robust ethical framework is essential. A comprehensive scoping review found that the four well-established principles of biomedical ethics—Beneficence, Non-Maleficence, Respect for Autonomy, and Justice—provide a foundational and universally applicable framework for guiding the responsible application of AI (Gorelik et al., 2025). These principles, long adopted in healthcare, offer a common language to address new ethical challenges posed by AI.

• Beneficence: This principle emphasizes taking positive action to enhance patient welfare and minimize potential harm. In AI, this translates to ensuring AI systems improve the quality, accuracy, and efficiency of care, and promote personalized treatment plans. However, concerns exist regarding the accuracy of AI chatbots and their potential to complicate the clinician-patient relationship if empathy and humanistic aspects of care are overlooked.
• Non-Maleficence: The core tenet here is "do no harm." For AI in healthcare, this necessitates rigorous attention to data quality, reliability, and generalizability, as poor data can lead to inaccurate outcomes. Patient privacy and data protection are paramount, especially given that healthcare is a frequent target for cyber-attacks, and large datasets make re-identifying anonymized individuals easier. The risk of human overreliance on AI and the potential for misinformation also fall under this principle, highlighting the need for robust risk management and human oversight.
• Respect for Autonomy: This principle upholds a patient's capacity to make informed choices based on their values and beliefs. AI integration demands transparent explanations of how AI systems operate, their limitations, and how patient data is used. Current consent processes are often inadequate, necessitating dynamic consent mechanisms where patients can control their data sharing. The "black-box" nature of some AI algorithms, where decisions are not easily understandable, can erode trust and challenge patient autonomy.
• Justice: This principle concerns the fair and equitable distribution of healthcare benefits. AI applications must address biases and discrimination stemming from unrepresentative training data, which can exacerbate existing health disparities. Accountability for AI-induced errors becomes complex, creating "accountability gaps" when AI influences clinical decisions. Clearer guidelines and regulatory frameworks are needed to define responsibilities among developers, clinicians, and oversight agencies, ensuring that AI enhances accessibility and affordability of services for all, particularly in underserved areas.

Practical Strategies for Secure AI Deployment

      The findings underscore that the ethical deployment of AI in healthcare is not merely a theoretical exercise but a practical imperative. Organizations deploying AI solutions must prioritize several key areas to ensure safety, trust, and compliance:

• Robust Data Governance: Implement stringent data privacy and security protocols, ensuring all video streams, inference results, and metadata remain within secure infrastructure if sensitive data is involved. Solutions like on-premise AI software or edge AI systems can provide full data ownership and control, crucial for privacy-sensitive environments. ARSA Technology, for instance, offers AI Video Analytics Software and the AI Box Series, designed for on-premise deployment without cloud dependency.
• Transparency and Explainability: Strive for AI models that offer clear, understandable explanations of their operations and decision-making processes. This fosters trust among users and allows for proper oversight, aligning with the principle of Respect for Autonomy.
• Continuous Monitoring and Validation: Regular auditing of AI model performance is crucial to detect biases, ensure accuracy, and prevent unintended manipulative behaviors or "hallucinations." This involves validating AI outputs against real-world clinical outcomes and ethical guidelines.
• Human-Centric Design and Oversight: Design AI systems as assistive tools that augment human capabilities rather than replace human judgment. Clinicians must retain ultimate decision-making authority, with AI providing insights and support. Systems should be intuitive and user-friendly, like the Self-Check Health Kiosk, which automates vital screenings while medical professionals remain essential for interpretation and personalized care.
• Adherence to Regulatory Frameworks: Stay abreast of evolving AI regulations and ethical guidelines, such as those from the EU AI Act, to ensure compliance and responsible deployment. Organizations should engage with AI solution providers who prioritize these principles in their development and implementation. ARSA Technology is committed to building AI since 2018 with engineering rigor, security compliance, and production readiness.
• Customized Solutions for Specific Needs: Recognize that off-the-shelf AI may not always meet the unique ethical and operational requirements of every healthcare setting. Custom AI solutions, tailored to specific contexts, can ensure that ethical considerations are embedded from the ground up.

      As AI continues to transform healthcare, the need for ethical design and deployment becomes paramount. Understanding the potential for LLM manipulation and grounding AI development in established ethical principles are critical steps toward building trustworthy and beneficial AI systems that genuinely enhance human well-being.

      Sources:

• Ireri, G., & Odipo, R. D. (2026). Old Fictions, New Skins: Evaluating the Manipulative Capabilities of LLMs in Healthcare. arXiv preprint arXiv:2606.21977. https://arxiv.org/abs/2606.21977

Gorelik, A. J., Li, M., Hahne, J., Wang, J., Ren, Y., Yang, L., Zhang, X., Liu, X., Wang, X., Bogdan, R., & Carpenter, B. D. (2025). Ethics of AI in healthcare: a scoping review demonstrating applicability of a foundational framework. Frontiers in Digital Health*, 7, 1662642. https://www.frontiersin.org/journals/digital-health/articles/10.3389/fdgth.2025.1662642/full

      To learn more about secure and ethical AI solutions for mission-critical applications, explore ARSA Technology's offerings and contact ARSA today.