Creating the First Open-Source AI Digital Scribe
Applying rigorous prompt engineering and open-source transparency to healthcare applications.
In a healthcare system burdened by administrative demands, providers are forced to rush through assessments, which increases the risk of incomplete documentation and affects clinical decision-making.
That’s why Morris Aguilar, Ph.D. – an academic researcher and a student of Prompt Engineering at The Multiverse School – is building an open-source, locally run digital scribe to help alleviate this problem.
✨ Advanced Prompt Engineering – Join Us in September
Hey – do you want to learn Advanced Prompt Engineering?
Starting in September, we’re re-offering courses like:
These courses are designed for developers and tech-savvy professionals who are looking to deepen their understanding of AI.
Consider joining us this fall 🍂
The Evolution and Importance of Digital Scribes in Healthcare
A digital scribe captures and transcribes the conversation between a doctor and a patient, converting it into structured medical notes.
The concept of digital scribes is not new. Early iterations of this technology date back to the advancement of voice recognition software in the 1990s and early 2000s. Companies like Dragon Medical pioneered voice-to-text solutions, enabling doctors to dictate their notes directly into EHRs.
It’s no wonder that we’ve been trying to solve this problem for over 30 years, because as Morris notes, "Physicians can spend around one-third of their time doing data entry.”
He goes on to note the importance of these notes, explaining that “... even though it's a very demanding and sometimes tedious task, it's actually a very important task, because not only is it used for diagnostic purposes, but also to communicate to other physicians what's going on with this patient, and for billing purposes like insurance. It's a legal document as well, so we have to make sure that it's a high quality note and it contains all the pertinent information.”
More recently, advancements in AI and natural language processing (NLP) have led to the development of more sophisticated digital scribes that can not only transcribe conversations, but also understand and organize medical information contextually.
The Critical Role of Prompt Engineering in Healthcare AI
So far, the adoption of digital scribes has been gradual, with many healthcare providers still relying on traditional human scribes or manual data entry due to concerns about accuracy, cost, and integration with existing systems.
The National Institute of Standards and Technology (NIST) emphasizes that ensuring the accuracy of AI systems in healthcare is vital, stating in their July 2024 Artificial Intelligence Risk Management Framework: “... a confabulated summary of patient information reports could cause doctors to make incorrect diagnoses and/or recommend the wrong treatments,” where ‘confabulated’ equates to a hallucination.
Large Language Models (LLMs) present a particular challenge due to their inherent variability. Outputs can differ significantly with each run, even when using the same input. This variability stems from the stochastic nature of these models, which rely on probability distributions to generate text.
Morris touches on this challenge, noting, "Depending on the problem you're trying to address, if you run an LLM with the same prompt more than once, the output will be different every single time." This variability underscores the importance of carefully crafted prompts and system-level instructions that guide the model towards consistent, reliable outputs.
Prompt engineering plays a critical role in managing these challenges. Research published in npj Digital Medicine shows that the structure and specificity of prompts can significantly influence model performance. This study found that prompts tailored to specific clinical contexts resulted in more consistent and reliable outputs from LLMs.
Additional techniques such as setting temperature parameters — a measure of the randomness in the model's output — and using consistent random seed states can help reduce variability, making the AI's behavior more predictable.
Morris explains, "Prompt engineering forces you to think through the problem you're trying to solve, and try to get a bit more granular."
Reliable outputs are especially crucial in healthcare, where the margin for error is minimal and the accuracy of AI-generated information directly impacts patient outcomes.
The Role of Open Source in Democratizing Medical Technology
One of the most compelling aspects of Morris's project is its foundation in open-source technology. Open-source software has been a cornerstone of innovation in AI, offering transparency, flexibility, and community-driven development. Early open-source models such as BLOOM aimed to democratize access to advanced AI, enabling researchers and developers to experiment, modify, and deploy these models in diverse environments.
And in 2024, the landscape of open-source LLMs has evolved significantly, with new models offering enhanced capabilities and greater accessibility. Notable among these are models like LLaMA 3, Falcon-180B, and GPT-NeoX, which have expanded the possibilities for developers and researchers alike.
Morris highlights the accessibility of these open-source models, stating, "Given that these are open-source models, any researcher can download the same version of the model that I use and run it on their own hardware."
This accessibility is not just about cost reduction — it's also about adaptability. Open-source models can be fine-tuned to fit specific needs, such as creating domain-specific language models for medical scribing, where understanding medical terminology and context is critical.
Moreover, the collaborative nature of open-source development accelerates innovation cycles. "It goes back to the whole open-source community thing where you can only get so much done on your own," Morris emphasizes. "But if there's an open-source community, they can chime in about any possible solution and pick up where you left off too."
This collaborative spirit is particularly important in the healthcare sector, where the stakes are high, and the ability to rapidly share and implement improvements can lead to better patient outcomes. By leveraging open-source models, Morris's project contributes to a broader movement toward accessible and equitable healthcare technology.
On Data Security and Privacy with Open Source LLMs
HIPAA sets strict guidelines for the protection of patient data, and compliance is mandatory for all healthcare entities.
Morris's project leverages localized AI solutions, reducing the need to transmit sensitive data over the Internet. This feature is a critical component of Morris’ project, since data security is only as strong as its weakest link. For recent illustration, just look to the recent ransomware disruptions of third party service providers, which resulted in millions of Americans facing delays in obtaining their prescriptions, and some paying significantly higher prices out-of-pocket due to insurance processing failures.
"Implementing AI in healthcare must prioritize data security and privacy, especially when dealing with sensitive patient information," Morris emphasizes.
Embracing the Future of AI in Healthcare
The integration of AI in healthcare holds immense potential for improving efficiency, reducing administrative burdens, and enhancing patient care. "I want to work on developing tools that augment physicians' abilities, not replace them." Morris says. “If we could free up this documentation time to perhaps spend more time building rapport with patients and gaining that trust with patients, that could possibly lead us to having patients adhere more to the treatment plan.”
The American Medical Association supports the ethical use of AI in healthcare, emphasizing the importance of maintaining patient trust and ensuring equitable access to AI technologies.
Despite the promising potential of AI, skepticism remains. Morris acknowledges and embraces this skepticism, believing it drives more rigorous testing and validation of AI models. "Healthy skepticism and critical evaluation of AI technologies are essential to ensure they are used appropriately and effectively," Morris states.
About Morris Aguilar, Ph.D.
This article is derived from an interview with Morris Aguilar, Ph.D. It has been thoughtfully crafted to present the key points and insights shared by Morris during the conversation.
Morris has been an academic researcher in the world of biomedical research and bioinformatics for almost a decade. His early fascination with medicine and science, as well as his affinity for hands-on learning, has made his journey into the tech space anything but conventional. He first had the chance to do research using CRISPR as a tool while he was an undergrad at the University of California, Santa Barbara (UCSB) in 2013. This research occurred around the height of CRISPR’s popularity, and he subsequently fell in love with the scientific research process.
After graduating in 2015, he spent two years working as a research technician at the Oakley Evolution Lab at UCSB, where he conducted molecular experiments and used bioinformatics tools to analyze DNA sequencing data. Through utilizing this specialized data analysis software, Morris began learning Python and finding ways to incorporate machine learning into his research. In 2017, he started his dual M.D./Ph.D. in Bioinformatics and Genomics via the Medical Scientist Training Program (MSTP) at thePenn State College of Medicine. He did his graduate research on human genomics and metabolism, using AI models from scikit-learn to aid in interpreting how sequencing and metabolic data are linked to complex diseases.
He is still finishing up his M.D., but received his Ph.D. in Bioinformatics and Genomics at the end of 2022, just as the current era of LLMs providing incredible utility began to get mainstream attention. He now uses the prompt engineering and AI toolchain knowledge from his time at The Multiverse School to enhance the speed of the iteration processes of his research.