Visual-Language Conversational AI for Neuro-Oncology Tumor Board

Foundation Model for Tumor Board Report Generation and Analysis

Asim Waqas

Department of Machine Learning, Moffitt Cancer Center & Research Institute,

Electrical Engineering Department, University of South Florida.

 

 

Aakash Tripathi

Department of Machine Learning, Moffitt Cancer Center & Research Institute,

Electrical Engineering Department, University of South Florida.

 

 

Ghulam Rasool

Departments of Machine Learning and Neuro-Oncology, Moffitt Cancer Center & Research Institute,

Electrical Engineering Department and Morsani College of Medicine, University of South Florida.

 

 

Code

(coming soon)

 

 

______________________________________________________________________________________________________

Overview

 

Welcome to our groundbreaking project aimed at revolutionizing the field of neuro-oncology with the power of foundation models. Our team is working on developing an advanced system capable of generating comprehensive and insightful tumor board reports. Tumor boards provide a collaborative and multidisciplinary approach to optimizing and individualizing cancer treatment. During a tumor board meeting, multidisciplinary specialists review a patient's medical records, imaging studies, pathology reports, and other relevant data with the aim of finding the best course of treatment based on the patient's needs and circumstances. Besides being a requirement for the National Cancer Institute (NCI) Cancer Center designation, tumor boards are known to improve patient management for various cancers. By leveraging the cutting-edge capabilities of foundation models, we aim to provide oncologists, neurosurgeons, and researchers with an invaluable tool that will enhance their decision-making process, streamline multidisciplinary collaborations, and ultimately improve patient outcomes. With the fusion of state-of-the-art artificial intelligence and medical expertise, our project strives to usher in a new era of personalized and data-driven care for individuals battling neuro-oncological conditions.

The development scheme of neuro-FMs, graphically compared with ChatGPT and Unified-IO models.

______________________________________________________________________________________________________

Method

 

As effective as tumor boards are, they face several operational challenges: (1) time constraints - all experts must prepare ahead and make themselves available simultaneously, (2) data management – various forms of clinical, radiographic, pathologic, medical records, and -omics data exist on different platforms and consequently can be challenging to share with the tumor board team simultaneously, and (3) reporting – there is no standardized/automated method to document the tumor board's findings, which can be used to identify suitable patients for ongoing clinical trials and communicate with patients.

We propose developing neuro-FM, a multimodal conversational AI that will process multi-modality data (radiological images, histopathology, and IHC data and images, information from EHR, and -omics data) and generate summary reports with annotated radiological and pathological images. Modern conversational AIs available in the commercial or research domain are based on LLMs and thus can process text only. GPT-4 was claimed to process images; however, such capability is not yet released. A summary of our solution is given below:

1. Classical and task-specific AI/ML methods for NLP and CV do not support building heterogeneous multimodal data applications for medical data processing to support conversational use in tumor boards.
2. The current state-of-the-art conversational AI models (e.g., ChatGPT and many others) are limited to text-only data at the input and output. Multimodal data processing is still not available in any conversational AI.
3. We propose developing neuro-FM to process multimodal medical data and generate rich text and image reports for near-oncology tumor board meetings. There are no other tools that can generate expert-facing medical reports consisting of text and images and discuss the findings in the reports.
4. Neuro-FM will show of feasibility of developing such multimodal conventional AI models for other areas of patient care where multi-disciplinary teams are involved in the process of decision-making.
5. The well-known hallucination problem of the generative AI will be addressed using an additional component in the design of the model that would post-process the output of the model and confirm key findings or numbers before presenting these to the user.

______________________________________________________________________________________________________

Conclusion

 

Neuro-FM output consists of a text report with annotated images for tumor board meetings given the available data (radiological studies, pathology images and reports, lab report, next-generation sequencing (NGS) data, etc.) of a patient as the input. Convergence of maximum data resolutions across varying time occurrences can accrue remarkable discoveries about the disease indiscernible in individual considerations. This project solution aims to converge the entire spectrum of the disease and understand the patient’s genetic, physiological, and psychosocial circumstances in a unified framework. 

______________________________________________________________________________________________________

Citation

@misc{neuro-FM-website,
title = {{Foundation Model for Neuro-Oncology Tumor Board}},
year = {2023},
author = {{Asim Waqas, Aakash Tripathi, Ghulam Rasool}},
note = {Available at: \url{https://lab.moffitt.org/rasool/neuro-fm/}}
}