Med-Gemini_A_Multimodal_Ecosystem_for_Medicine
1
2
3
4
5
6
7
8
9
10
11
12
Slide 1
NotebookLM to PPTXTurn NotebookLM Slides into Editable PPTXExport PPTX
Log In
NotebookLM
Integrating Diverse Clinical Data into a Unified Al Architecture
Med-Gemini: A Multimodal Ecosystem for Medicine
NotebookLM
Interactive Q&A
EHR Data
Single Output
Genomics Al
Single Input
Med-Gemini
Predictive Analytics
Comprehensive Report
Genomics Data
Single Output
Single Input
Pathology Al
Radiology Data
Single Output
Single Input
Radiology Al
The New Paradigm: Integrated Al
The Old Paradigm: Siloed Al
The evolution of medical Al is moving from a collection of narrow, single-task models to a new paradigm: a generalist, multimodal system that can reason across the full spectrum of patient data. Previous approaches focused on single input and output types, whereas new generative Al can address multimodal, multi-task challenges.
From Specialized Tools to an Integrated System
NotebookLM
Efficiency
Advanced Multimodal Reasoning
Gemini 1.5 Core
Optimized Performance Pathway
Multimodal Integration Pathway
Extended Context Pathway
Up to 1M Tokens
Long-Context Understanding
Med-Gemini is built upon Gemini 1.5, an architecture designed from the ground up for multimodal reasoning It inherits Gemini's core capabilities, making it the ideal foundation for a comprehensive medical Al.
The Foundation: A Native Multimodal Architecture
NotebookLM
Clinical Text
Polygenic Risk Prediction
Dermatology
3D Report Generation
Ophthalmology
2D Report Generation
Genomics
Histopathology
Visual Question Answering
Classification
Med-Gemini-Polygenic
Gemini
Med-Gemini-3D
Gemini
Med-Gemini-2D
3D Radiology
2D Radiology
The generalist Gemini model is fine-tuned on a massive medical dataset of over 7 million samples from 3.7 million images and cases. This process creates a family of specialized models, each optimized for a specific data structure while sharing the same underlying intelligence.
The Med-Gemini Ecosystem: Specializing the Foundation for Clinical Insight
NotebookLM
A groundbreaking proof of concept; further research is needed to meet expert quality
CTs were considered clinically acceptable."
of Al-generated reports for 3D head
53%
First-ever LMM-based 3D CT Report Generation.
3D Computed Tomography
Original report superior, Al missed key findings
Both lead to correct outcome, original more detalled
Both lead to correct outcome, with similar findings
57%
"equivalent or better" (IND1).
abnormal Al reports were judged
of
65%
of normal and
96%
On an out-of-distribution dataset,
abnormal Al reports were judged "equivalent or better" than original radiologist reports (MIMiC-CXR).
of
43%
of normal and
57%
Both lead to correct outcome, Al more detalled
Al superior, original report missed key findings
43%
MIMiC-CxR "Abnormal" Case Breakdown
2D Chest X-ray
Med-Gemini significantly advances the state-of-the-art in automated report generation for both 2D and 3D imaging, producing reports that are often considered equivalent or superior to the originals written by radiologists.
System Function: Automated Radiology Reporting
Visual Question Answering
Broad Classification Performance
NotebookLM
over the previous best-in-class model (ELIXR).
14-point absolute improvement
This represents a
(86.8%).
outperforming the Gemini Ultra baseline
90.7%
Med-Gemini-2D achieved a Macro-F1 score of
accuracy on VQA-Rad.
71.9%
: On the in-distribution MiMiC-CxR dataset,
CXR Classification
expert-evaluated
histopathology, ophthalmology, and dermatology.
Surpasses baselines across
What is the abnormality shown?
18 out of 20 tasks in
Beyond reporting, Med-Gemini-2D excels at diverse interpretation tasks, surpassing baselines in classification and setting new performance standards in challenging radiology Visual Question Answering (VQA) benchmarks.
System Function: Classification and Visual Q&A
NotebookLM
achieved higher AUCs than the linear PRS benchmark for all in-distribution health outcomes except one.
Med-Gemini-Polygenic
Key Result:
Med-Gemini-Polygenic
Standard Linear PRS Model
Stroke
Type 2 Diabetes
Coronary Artery Disease
0.75
0.7
0.72
0.65
AUC
0.81
0.85
0.80
0.8
AUC Performance for In-distribution Diseases
This novel encoding featurizes the entire genome for visual analysis.
Genomic Data (UK Biobank)
How It Works
Disease Risk Prediction
Polygenic Risk Scores Projected into 2D Image-like Input
scores (PRS) as image-like inputs, Med-Gemini-Polygenic can predict disease risk more accurately than standard linear methods and generalize to conditions for which it was never trained.
risk
complex polygenic
By representing
beyond pixels to genomics.
Med-Gemini extends its capabilities
System Function: Predicting Disease Risk from Genetic Data
NotebookLM
The patient's CT is unremarkable for acute issues. However, given the high polygenic risk for coronary artery disease, prioritizing a lipid panel, blood pressure management, and a discussion about lifestyle modifications would be advisable.
'Given the patient's CT findings and high polygenic risk for CAD, what follow-up should be prioritized?
Conversational Al Panel
High Risk
Coronary Artery Disease
Top 5% Risk
Stroke
Type 2 Diabetes
Polygenic Risk Profile
Genomic Risk Panel
IMPRESSION*
":Normal.
cardiopulmonary process.
:No acute
FINDINGS
Al Summary
L
Imaging Panel
Clinician's Dashboard
from a new CT scan to their underlying genetic risks—to provide a holistic, context-aware summary and decision support for clinicians.
A single, unified platform could analyze a patient's entire medical history—
The true power of the Med-Gemini ecosystem lies in i
integration.
A Vision for Comprehensive Clinical Assistance
NotebookLM
The Path Forward: From Benchmark to Bedside
While this research demonstrates significant potential, further work is essential to bridge the gap from benchmark performance to clinical deployment. This includes rigorous validation to ensure safety, reliability, and real-world utility.
Future Directions
The ecosystem demonstrates the first-ever LMM-based 3D CT report generation and a novel method for more accurate genomic risk prediction.
Med-Gemini sets a new standard in automated radiology reporting and VQA, validated through rigorous expert evaluation.
Fine-tuning a powerful foundation model creates a single, high- performing system for diverse medical tasks, from imaging to genomics.
Pioneering New Capabilities
New Performance Benchmarks
A Versatile Ecosystem
Engineering the Future of Generalist Medical Al
NotebookLM
2.2M
2.2M
PMC,MedVQA, etc.
Medical VQA
250K
250K
UK Biobank
~14K
~14K
EyePACS
Ophthalmology
~2K
~2K
PAD-UFES-20
Dermatology
210K
1.5M
Histopathology, PathVQA
Pathology
650K
650K
CT-US1
Radiology (3D)
360K
2.2M
MIMIC-CXR, CXR-US2, etc.
Radiology (2D)
No.Images
No.Examples
Dataset
Genomics
Modality
Med-Gemini was fine-tuned and instruction-tuned on a dataset of over 7 million samples from 3.7 million medical images and cases. The dataset spans diverse modalities and includes mostly free text paired with medical data, reducing the need for expensive expert labeling.
Appendix A: Training Data Overview
NotebookLM
Chart shows Med-Gemini's relative performance compared to State-of-the-Art (SoTA) or strong baselines.
) signify tasks evaluated by clinical experts.
Diamond markers
Classification
Genomic Disease Prediction
Report Generation
Question Answering
-20%
-10%
10%
20%
30%
40%
50%
Med-Gemini demonstrates promising or best-in-class performance across a comprehensive set of clinically relevant benchmarks, including 22 datasets across five different tasks and six distinct medical image modalities.
Appendix B: Relative Performance vs. Best-in-Class
0%
Relative ImprovementoverBest-in-Class(%)
Linear-probing performance on histopathology patch-classification tasks. While Med-Gemini outperforms Gemini Ultra on 7 of 9 in-distribution tasks, the specialized PathSSL model remains the top performer.
NotebookLM
TCGA study
Tissue type
Lung
PCam
CN
Med-Gemini
Gemini Ultra
0
20
40
60
AUC(%)
80
Out-of-Distribution Tasks
In-Distribution Tasks
PathSSL
Med-Gemini
Gemini Ultra
AugReg ImageNet
100
Med-Gemini-2D's image embeddings achieve performance comparable to a histopathology- specialized foundation model (PathSSL) while demonstrating strong results across multiple other clinical domains. It consistently outperforms the non-medically tuned Gemini Ultra baseline.
Appendix C: Histopathology Classification Performance