We’re pleased to share insights from Xinpeng Liu a PhD candidate in Computer Science and Law at the University of Galway, on discussions about AI’s role in healthcare from the latest AI4People Summit.
The session, titled AI in Healthcare, brought together experts to discuss key issues, including how to ensure the clinical effectiveness of AI, how to transform patient care with AI, how research facilitates AI innovation, and how to make innovation accessible globally.
The discussion featured a panel of experts, including Fausto Pedro García Márquez, Professor at Castilla-La Mancha University, Spain (UCLM); Ran Balicer, Public Health Professor at the Ben-Gurion University in Israel; Guillaume Bernard, Computer Research Engineer at LNE; Marco Lorenzi, tenured research scientist at the Inria Center of University Côte d’Azur; Alison Noble, Technikos Professor of Biomedical Engineering at the University of Oxford; and Kristian Vigenin, Member of the European Parliament and LIBE Committee member.
To watch this discussion please click here
Expert Perspectives on AI in Healthcare
The first part is speakers expressing their perspectives on AI in healthcare. The key ideas of speakers’ speech are as follows.
Guillaume Bernard, Computer Research Engineer at LNE: In the EU market, medical devices with embedded AI must comply with both the European Union Medical Devices Regulation (MDR) and the EU AI Act. Guillaume Bernard’s team is dedicated to helping medical device providers meet the compliance requirements of both MDR and the AI Act. Their approach consists of two main components. First, they look into the development process of the AI system to evaluate whether the system is sufficiently mature. For example, they assess whether the system meets transparency requirements, whether the training data is sufficiently comprehensive to cover all medical domains that the system is intended to support, and whether the samples represent different population groups—such as variations in age and gender. Second, they test the AI system itself. This is challenging because an AI system often functions as a black box. During the testing process, they examine the system’s input data and output results to understand its behavior. For example, they use evaluation datasets to assess the system’s performance, and then analyze which aspects meet the requirements of MDR and the AI Act and which aspects still fall short.
Marco Lorenzi, tenured research scientist at the Inria Center of University Côte d’Azur: As a research scientist, Marco has been confronted with the problem of gathering data from different sources, different hospitals. Marco has faced legal and ethical challenges. To address these issues, Marco has initiated an open-source framework for learning in health care. He participated in many consortia that involved many stakeholders and researchers from different disciplines. To address these challenges, work from multidisciplinary backgrounds is needed.
Alison Noble, Technikos Professor of Biomedical Engineering, University of Oxford: Addressing AI applications in healthcare requires interdisciplinary collaboration, and ethical considerations play a crucial role in these applications. As AI in the medical field is still in its early stages, we must learn how to use these technologies in the most appropriate and effective way. In addition, human oversight is essential in AI applications in health. When confronted with AI technologies, we must first reflect on why we are using AI at all. In the medical domain, the core of AI safety is patient safety. Ensuring the safety of patients during interactions with AI systems is paramount. Furthermore, although AI systems can empower patients by providing access to health information, they may also deliver misleading information, which can undermine trust in healthcare professionals and introduce medical risks. Finally, the global ethics of AI in healthcare is critically important. AI models are often trained on datasets derived from Western contexts, which may not be directly applicable to other regions. Ethical standards should therefore be grounded in a consensus among stakeholders from diverse backgrounds.
Kristian Vigenin, Member of the European Parliament, LIBE Member：AI technologies bring tremendous opportunities as well as significant risks. We must ensure that democratic societies shape AI, rather than allowing AI to shape society in ways that undermine fundamental rights and freedoms. The EU AI Act establishes a risk-based regulatory framework that reflects a strong commitment to fundamental rights, accountability, and democratic governance. Innovation must never come at the expense of human dignity. In the healthcare sector, AI offers important opportunities to improve diagnostics, enable personalized treatment, and enhance access to medical services. However, it also creates risks of discrimination, unequal access to technology, and increased surveillance. To ensure that doctors and patients can trust AI systems, we must build clear ethical foundations and robust regulatory safeguards. Governing AI requires global cooperation. Technical standards must be interoperable, and human rights principles must be recognized as universal. The oversight of AI systems must remain under human control, must be transparent, and must be understandable in its effects. We now have the opportunity to build a global ecosystem that both protects rights and encourages creativity—an endeavor that depends on political will, technological collaboration, and the courage to establish rules in a rapidly evolving technological landscape.
Ran Balicer, Public Health Professor at the Ben-Gurion University, IsraeI：Ran Balicer discussed use of AI in practice in a organization。In traditional healthcare technologies, once a technology is approved in one region, it can often become available for use in other regions as well. However, this approach does not apply to AI technologies. The fact that an AI system performs well in one region does not guarantee that it will perform well in another, because the dataset, data structure, and population characteristics may differ significantly. Therefore, we cannot assume that an AI system that works effectively in one institution will function equally well in another. The sustainability of healthcare systems is becoming increasingly important. With ageing populations and rising medical demands, healthcare structures must be transformed, and greater reliance on AI will be essential. Many healthcare organizations are attempting to adopt AI systems, but they struggle to determine whether these systems are truly responsible. Ran Balicer, working within an Israeli health services organization, developed a systematic checklist tool to evaluate whether an AI system meets the criteria for responsible AI. Many AI systems fail this evaluation. Every AI system may embed biases—even if we are unaware of them, they still exist. The key is to identify and address these biases so that the advantages of AI can outweigh the associated risks.
Discussion on Pre-Determined Questions
The second part of the working group session is to discuss pre-determined questions. The following questions have been discussed.
What essential components should a standalone healthcare framework for AI include to ensure clinical effectiveness?
The ideas of speakers to address question 1 are as follows.
Alison Noble: Alison recognizes the importance of checklists. It is not necessary for everyone to become an expert in every field; however, from the very beginning of AI deployment, it is essential to clearly identify which factors must be taken into account when assessing whether an AI system is responsible. Checklists and standards should be continuously updated over time, as increased experience and learning will help us become more comfortable with managing risks. It is also important to differentiate between AI use cases. AI is a platform technology with a wide range of applications. If each use case were governed by a completely separate framework, it would be difficult to comprehensively address ethical issues. Therefore, the development of solutions requires the joint participation of multiple stakeholders. Furthermore, successfully deploying AI in healthcare requires teams with the appropriate mix of skills. In addition to clinicians and patients, such teams must include data managers and project managers. Similar to previous industrial revolutions, AI is likely to give rise to new professions. AI systems can function effectively only when supported by an appropriate team structure. At present, the healthcare sector lacks a sufficient number of professionals with the necessary skills, and time is needed to build these new multidisciplinary teams.
Marco Lorenzi: Data itself must also be standardized, especially in the clinical domain. Data standardization includes the standardization of data formats, access to data, and interoperability between different tools and systems. While there has already been some experience with medical data standardization, this represents only a drop in the ocean. In many other areas of medicine, data standardization is still lacking. As such, data standardization constitutes one of the fundamental foundations for both the development and regulation of AI. In addition, hospital training is critically important. Hospitals are not only key stakeholders in AI systems but also major providers of training data. In some cases, medical data is proprietary, which makes data sharing difficult. Therefore, data standardization and hospital training must proceed in parallel. Moreover, hospitals face challenges beyond the shortage of professionals with the appropriate skills; they also suffer from limited resources. Hospitals already operate under significant pressure, and the additional burden of AI engineering risks overwhelming them. In France, public hospitals are managed in a centralized manner at the regional level, and this governance system is currently evolving. Beyond technical solutions, government investment is also essential to support the sustainable deployment of AI in healthcare.
Guillaume Bernard: Guillaume’s team has developed a Metrics Framework. The team is exploring how data provided by hospitals can be converted into multiple formats. They have also identified that healthcare institutions often possess large volumes of data, yet are uncertain whether this data can be used for medical AI systems. Addressing these issues is a central focus of the team’s ongoing work.
In what unexpected ways could AI transform patient care beyond improving efficiency and diagnostics?
The speakers have expressed the following ideas.
Marco Lorenzi: Many medical devices are now being equipped with AI capabilities, and these devices may become interconnected, enabling a multimodal view of patients. Data from different levels—such as biological images and clinical data—can be integrated to generate more personalized representations of patients.
Guillaume Bernard: AI can assist medical experts in healthcare and help reduce their workload. From the patient’s perspective, AI can support individuals in understanding their health needs, particularly in countries where access to medical resources is limited. However, it is also important to recognize that the use of AI systems in healthcare continues to face significant challenges. Many AI systems remain in the testing phase, and their behavior is not yet fully understood, which can raise red flag issues.
Alison Noble: AI, robotics, and simulation need to be discussed together. AI has already begun to be used in medical education, for example by enabling students to enter simulated training environments more quickly. Through pattern recognition, AI can identify disease features and treatment pathways beyond what is visible to the human eye. Whereas medical practice has traditionally been constrained by what clinicians can directly observe, AI allows us to rethink how patients are diagnosed and treated. AI is also transforming the doctor–patient relationship, with patients increasingly expected to take greater ownership of their own health conditions. At the same time, robust governance mechanisms are essential to ensure safety. In situations where outcomes remain uncertain, patients should not be encouraged to diagnose themselves, as this could place additional strain on healthcare systems.
How can research and development facilitate AI innovation in healthcare？
To address this question, the speakers have formed the following opinions.
Alison Noble: Companies tend to focus on how to generate revenue from AI tools, whereas healthcare systems are primarily concerned with health economics. Greater attention must therefore be paid to health economics, particularly to reducing costs and ensuring that healthcare remains affordable. Many healthcare systems operate without new or additional budgets, and as populations age, the introduction of new technologies places increasing pressure on public finances. As a result, a key question we must confront is whether, even if AI technologies prove effective, we are able to afford the costs of operating and maintaining them.
Marco Lorenzi: Pursuing technological innovation in a sustainable manner is highly challenging. This is particularly true when rapid development must be achieved while simultaneously meeting stringent regulatory and compliance requirements, as these competing demands significantly increase complexity. Other countries may operate under different priorities, enabling faster AI development, which in turn requires us to accelerate certain tasks in order to remain competitive. From an academic perspective, we are also confronted with the issue of talent loss. We must persuade researchers and professionals to stay, and to believe that it is possible to achieve domestically what they might otherwise seek to accomplish abroad. Identifying a “sweet spot” that balances regulatory compliance with rapid innovation is extremely difficult. Moreover, we are still lagging behind in terms of infrastructure, including insufficient computing capacity and limited ability to translate ideas into deployable products. Addressing these shortcomings requires substantial investment. Under conditions of fiscal constraint, building such a system is particularly challenging. If Europe is to remain competitive in the future, these issues must be taken seriously and addressed with urgency.
How can innovation be accessible globally？
Marco Lorenzi: Protecting intellectual property (IP) is highly challenging, financially demanding, and difficult to maintain over the long term. It is often hard to patent an idea with commercial potential. What is needed is a research-supportive ecosystem that can better protect research ideas and facilitate their transformation into commercial products. Marco referred to a case he had experienced in which a research idea was highly promising, yet there were insufficient resources to develop it into a long-term project. This highlights the need for support from well-resourced industry partners. IP itself is complex and costly, and in many cases, a company’s value is largely based on the amount of IP it holds.
Alison Noble: One reason for the complexity of these issues is that solutions developed by companies do not always generate returns, as their value is often tied to the data they own. Business models are typically built around resources that are unique and capable of creating a competitive advantage. However, changes in data governance have created significant difficulties for companies, and many existing business models struggle to adapt to the new environment. From a public governance perspective, stability is essential to enable companies to develop sustainable business models. Achieving this in the context of AI and healthcare is particularly challenging, as companies must invest in intellectual property, acquire data, and comply with regulatory requirements. At the same time, hospitals may lack sufficient financial resources to adopt new innovations. It is therefore important to avoid situations in which hospitals become dependent on companies that are no longer able to operate. These challenges underscore the need to find ways to reduce costs across the system.
Conclusion
At the end of the session, Fausto Pedro Garcia Marquez delivered the concluding remarks. According to him, artificial intelligence today is highly complex and is developing at a very rapid pace. The European Union, as well as many other countries, has adopted different definitions of AI, and different frameworks reflect divergent understandings of what AI is. In the coming years, AI is expected to play an important role in healthcare. Health-related issues are central to everyday life; however, healthcare itself is a highly complex domain, encompassing a wide range of applications, including medical robotics. As a result, there is a need to reach a consensus on what artificial intelligence actually means, and whether this meaning remains consistent across different countries and over time. This question is closely linked to ethics. Based on differing ethical approaches, policymakers may adopt different standards to protect society, technology, and stakeholders. At the same time, it is important to recognize that AI is also part of the commercial landscape. Many AI solutions are developed by companies whose primary objective is profitability. Therefore, stable standards are required in contractual arrangements and commercial practices. Policymakers must adopt a pluralistic perspective and actively engage with professionals, as those who develop AI solutions and those who apply them often hold different viewpoints. Overall, AI and healthcare constitute a highly complex, global issue, cutting across national borders, systems, and organizations. The discussion must remain focused on identifying the specific problems that AI is intended to address. There is no universally applicable solution; instead, a clear balance must be maintained between knowledge accumulation and problem-oriented approaches.
The following speakers further contributed to the conclusion.
Marco Lorenzi: This is indeed a highly complex issue. The environment is evolving on a daily basis, with new innovations emerging alongside regulatory requirements. In many cases, new problems only become apparent once systems are deployed in practice. Addressing these challenges requires interdisciplinary collaboration. Different disciplines hold different understandings of AI and healthcare, and successfully deploying systems necessitates acknowledging and engaging with these differences. These issues extend beyond purely technical concerns and also involve social and philosophical considerations. As such, there remains a substantial amount of work to be done, requiring broad and sustained interdisciplinary cooperation. In addition, greater awareness at the policy level is essential to ensure the sustainability and global competitiveness of AI systems.
Guillaume Bernard: From both the patient and healthcare professional perspectives, it is essential to understand what AI systems can do and what their limitations are. It is important to avoid both overconfidence in these systems and excessive caution that leads to their non-use. When AI systems are developed but not adopted in practice, this represents a waste of both time and financial resources. Despite the significant challenges associated with regulation, cost, and implementation, there remains strong optimism about the potential of AI systems. Realizing this potential will require substantial effort to ensure that AI is used effectively and responsibly.
Alison Noble: From a historical perspective, we are at a critical moment. Key stakeholders all hope that AI technologies will succeed in the healthcare sector. While we have not yet fully reached that point, collaboration among stakeholders can help identify the optimal application areas in which AI can deliver real value. At the same time, there are domains in which AI may not be appropriate, and it is important to identify these areas early and discontinue further exploration. Patient trust is also a crucial issue. Patients and the broader public must be actively engaged in this process. As healthcare increasingly extends into the community, AI applications are no longer confined to familiar and controlled environments such as hospitals. The conditions under which AI is used outside hospital settings differ significantly from those within them, and failing to recognize this distinction may give rise to challenges that exceed those associated with introducing AI in hospitals themselves.
