The promises and risks of AI in scientific research
The processing and analytical power of artificial intelligence (AI) has led researchers to readily adopt the technology in scientific studies. AI capabilities have also inspired optimism in scientists about the technology’s potential for more efficient and effective research. However, there are concerns that this optimism can lead researchers to ignore the shortcomings of AI, such as its tendency to hallucinate, produce plausible results that are later proven to be false.
Illusions of understanding
Lisa Messeri and M. J. Crockett closely examined the risks of using AI in scientific research in their article published in Nature. In this analysis, the authors categorised visions that can render researchers susceptible to illusions of understanding, where they believe that they grasp a concept more thoroughly than they do.
The authors found that individuals who rely on AI tools to overcome cognitive limitations, subjectivity and bias tend to overestimate their understanding of a subject. In turn, this creates scientific monocultures where certain methods and ideas dominate at the expense of alternative approaches, making scientific research less innovative and more prone to errors.
Scientists’ visions of AI
Messeri and Crockett identified four visions of AI-enabled research across various stages of scientific investigation. These visions include AI as Surrogates and Quants, where researchers believe AI tools can process data that are too large and complex for humans to collect and analyse effectively. In addition, as Oracles and Arbiters, scientists trust AI to search, summarise, analyse and evaluate existing literature and research findings more objectively than humans can achieve.
The authors further identified three categories of illusions of understanding that stem from these AI visions: illusions of explanatory depth; illusions of exploratory breadth; and illusions of objectivity. For example, a scientist using an AI Oracle to generate new hypotheses from existing research may experience an illusion of exploratory breadth. In this scenario, they might mistakenly believe that they are exploring all testable hypotheses, while in fact, they are only investigating a limited set of hypotheses that AI tools can test.
Not so fast
Messeri and Crockett focused only on the four visions of AI because they considered them as most relevant to the risks of developing illusions of understanding. However, other visions of AI might impact a researcher’s perception of their understanding. Future studies should examine other factors that might affect the understanding of a concept, such as level of expertise and stage of training.
Why does it matter?
The four visions framework of AI encourages researchers to think about why they want to use AI in their research, to identify where they may lack understanding. The study also highlights the importance of assessing the risks now, while AI tools are just beginning to be applied in scientific research. Addressing these risks will be more difficult once AI tools become well-established in the research process.
Take home messages
1. Enhanced productivity brought by AI tools does not guarantee improved scientific understanding or objectivity.
2. Scientists need to be clear as to why they want to use AI in their research so that they are aware of the risks of experiencing illusions of understanding.
3. Scientific research needs to retain the human element in which AI augments the research process while allowing for diverse standpoints to enhance scientific knowledge.
Guest author: Conor McQuaid, PhD
This article was written as part of a series of ‘journal club’ summaries for Scientific Writers Ltd and is based on the following publication.
Title: Artificial intelligence and illusions of understanding in scientific research
First Author: Messeri L, et al.
Journal: Nature
Date online: 6 March 2024
Other references
