Video summary
Ndèye Maguette Mbaye presented her PhD research focused on leveraging multimodal data to enhance breast cancer treatment strategies and prognostic accuracy. Breast cancer is a highly heterogeneous disease characterized by distinct molecular subtypes, including Luminal A, Luminal B, and Triple Negative Breast Cancer (TNBC), each with varying prognoses and treatment targets. The study utilized two primary databases: one containing structured clinical and biological data from 150 patients, and another comprising free-text clinical reports. By integrating these diverse data sources, the project aimed to identify key predictive factors for relapse-free survival, addressing the challenges posed by imbalanced datasets where certain outcomes are less frequent than others.
The methodology involved a comprehensive pipeline that included data preprocessing, machine learning modeling, and interpretability analysis. For structured data, statistical features were extracted, while free-text reports were processed using Term Frequency-Inverse Document Frequency (TF-IDF) to capture the frequency and importance of medical terms and events. The study compared two main integration approaches: early integration, which concatenates all modalities before modeling, and late integration, which trains separate models for each modality and combines their predictions via majority vote or weighted averaging. Various algorithms, including Random Forest with resampling techniques to handle class imbalance and neural networks tuned with Keras, were employed to ensure robust performance across different data types.
A significant portion of the presentation was dedicated to interpreting model decisions using Local Interpretable Model-agnostic Explanations (LIME) and SHAP values. The researcher explored different global aggregation methods to synthesize local explanations, evaluating them based on their ability to maintain predictive performance when top features were removed. Results indicated that baseline Random Forest models often outperformed complex sampling methods, suggesting inherent robustness in the data. Furthermore, the analysis revealed common predictive features across text and structured data, such as specific doctor identifiers, though the team acknowledged potential biases associated with these variables and discussed strategies like using medical specialties instead of individual names to mitigate them.
In conclusion, the study demonstrated that both early and late integration methods yield comparable performance, with slight advantages depending on the specific dataset context. The researcher highlighted future perspectives involving more complex models like BERT adapted for electronic health records to better handle sequential patient histories. While current limitations include a lack of mammography images for deep learning applications, the work underscores the potential of multimodal machine learning in clinical settings. The session concluded with an interactive Q&A addressing topics such as intermediate integration methods, stack generalization, and the nuances of text cleaning and bias mitigation in medical AI models.
Read the full video transcript
good welcome back from the coffee break
for the finalization of today so magette
is an ESR with Chloe arsenkot who
unfortunately cannot be here due to
sickness but we are very happy to have
magette here and report on her work the
floor is yours
hi everyone I'm really happy to be here
to present a part of my PhD project so I
am maggot ESR 11 and I'm supervised by
Chloe I got As in Cut who's not here
today
um so I'm a PhD student at mint Polytech
in Paris and at the answer to query and
today I'm going to present uh I'm going
to talk about learning from multimodal
data to improve cancer treatment
so the presentation today will be
divided into four parts the first I will
talk about the context of the study then
I'll present the methods then the result
then uh at the end the conclusion and
perspectives
so as general context So today we're
going to talk about breast cancer
so breast cancer is the most common
cancer among women and it is
characterized by uh the fact that it's a
highly heterogeneous disease
so uh we can have different molecular
subtypes for breast cancer
the first one I will talk about is
called luminal a and here it's
characterized by the presence of hormone
receptors here estrogen and progesterone
in the absence of hir2 so these hormone
receptors controls the grow of the
growth of the cancer cells and for the
luminal a it constitute the most common
cancer in breast cancer cases and it has
a good pronoun stick because here we
have a Target we have a target for all
the treatments
and for the second subtypes it's called
uh it's uh the human epidermal gross
factor receptor two positive when it's
combined by the absence of the hormone
receptors it's called non-dominal and
when it's combined by the presence it's
called luminal B and here is 15 of the
breast cancer cases here also we have uh
quite good pronostic because of the
Advent of the hr2 targeted treatment and
the last one is called tnbc or triple
negative breast cancer here we don't
have the presence of the hormone
receptor or the her2
so uh it has a bad pronuncity actually
because we don't have a clear Target for
the treatment
now depending on those uh molecular
subtypes so we have different pronostic
but we also have different treatment
strategies so clinician will use
multi-disciplinary approach or a
combination of Therapies
and at the answer to query we usually
use the combination of the chemotherapy
and the surgery and depending on which
one come first we have two types of
chemotherapy we have the nail adjuvant
chemotherapy and the adjuvant
chemotherapy the neuro Advent
chemotherapy is done before the surgery
and here the aim is to reduce the trauma
size before the surgery in order to have
a non-invasive surgery and the adjuven
chemotherapy the aim is to kill the
remain cell after the surgery
so after those treatment how the
clinician can say that the treatment has
gone well we measure uh several
endpoints the first one is called
pathological complete response so it's
defined by this appearance of the
invasive cancer cell after the after
completion of the neoad-driven
chemotherapy and we also have two other
endpoints two of the survival endpoints
so the overall survival and the relaxed
free survival so which is the length of
time from the date of the diagnosis of
the or the first treatment that the
patient is still alive and for the
relapse free survival it's that's the
patient is still alive but with any any
sign of cancer or any sign of relapse
so in that context I uh we present the
pre-community mode project so which is a
project with uh different databases so
the first one is called database it
contained the information relative to
adjuvant chemotherapy and it has 15 150
patients and we also have the Alias
database which are basically all the
reports all the free text reports that
are written by the clinician so we
extract for this project the records
that belong to the same patient from the
database so we have here a mutual model
data set when we have structured data
with databasa and free text Data with
database Alias and the objectives of uh
pre-community mode will be to identify
and evaluate pronostic factors yeah
using a machine model data set so here I
will use the relapse survival status as
a endpoint for my prediction
so the methods that are used for it
so here I'm presenting uh the pipeline
so first uh analyze and extract the data
then do the pre-processing and then I
did a machine model learning and then do
the interpretation and evaluation of the
uh of the pronastic factors
so first let's talk about the data from
the two database
so for the database I have structured
data so I have over there clinical
informations uh we have 162 feature uh
for this uh for the clinical database
and we have also biological measurement
which are these two more markers and the
human markers and these uh these
features are sequential sequential
features uh and I also have for the
ilios database free text reports and
each patient have six to around 600
report uh
in his folder
so for the pre-processing of the data
set so for destructed data the clinical
information I use the
feature with the more value
and for the biological measurement I
extract statistical features uh for the
mean the variance the max Etc and other
features such as the alert because we
have the normal range of the biological
measurement and for the free text
reports I use for the prediction the TF
IDF of the diagrams so the TF IDF will
measure the frequency of the background
and the importance of that background
within the the corpus
and I was also able to extract another
modality called frequency of events
because we have the name of the event in
each report of patients so I was able to
say the to have the occurrence of unique
events for each patient
uh now for the methods part I'm doing
which model learning so integration
methods uh are important for the imagery
model learning and the first integration
methods I'm going to present is called
early integration so for this methods we
compute we perform we concatenate all
the modalities into one big input and
then compute the machine learning
methods and then do the prediction and
the interpretability
and for the late integration here we uh
we do the machine learning for each
modality
and then the prediction will be a
majority vote uh between the three uh
the three machine learning models
so the first uh models that I use uh was
random Forest so here you can see that I
have a highly imbalanced data set so for
that I use a random Forest as the
Baseline and I perform sampling methods
with the random Forest still so the
first one is balance random for us when
here
um it uh during the construction of the
random Forest it's down sample uh the
majority class in order to balance the
Boost the bootstrap sample and for the
smooth and the random first here the
smooth will over sample the minority
class it will create synthetic data set
that look like the minority class and I
also uh aggregate the balance random for
us and that's not random forest and I
did the prediction on the test set in
order to have the accuracy and then
identify by the predictive feature
another model that I used also was the
neural network
so here I
use the same train and validation set in
order to compare afterwards and also I
did the hyper parameters tuning with
Keras tuner and training with balance
batches in order to tackle the imbalance
data set
so another part that is important in
this project is the interpretability so
for the random Forest we will use the
the building random Forest
interpretation algorithm and for the
feed forward feed forward neural network
I will use shop and lime so uh so lime
it's uh similar to shop so it would do a
local interpretation uh it will perturb
like a uh instance uh and learn an
interpretable uh model locally around
the prediction as an interpretation
so these two uh methods so these two
methods are performing local
interpretation except maybe sharp but we
will take into account here the only the
local interpretation and then I also did
the global aggregation of those local
explanation
so uh here to perform the global
aggregation the first method of
aggregation uh it's called The Lime
aggregation it was uh uh it was uh given
by the people that uh
that uh wrote The Lime uh the lime
algorithm so here the first method will
be the square root of the sum of uh the
attribution
of the feature J of an old instance I
and uh so here with the first method
when it's about text it can be uh biased
because when we have like when we have a
word that occur uh many times we will
have a larger a larger uh aggregation uh
score so that's why I use also the
second method of aggregation because
here I will do the aggregation uh
methods on Text data so here the second
methods I use the line aggregation
average uh over the occurrence of the
word here
and the third method of global
aggregation here it's called average
attribute is the final at it's the mean
attribution with the N instances and
that's the third one it's the one that
is used for shop for example for example
and I will evaluate all those global
aggregation all those global
explanations sorry uh by using something
called The aopc Curve so the aopc will
uh we plot how the score uh is going to
change when we remove the suppose a top
feature that are given by the game by
the global aggregation
and here I'm going to show the result of
the model I'm split I'm showing here the
mean score uh so here for the early
integration uh we see that we have
stimulus AUC score but that the Baseline
which means the random Forest without
any sampling method outperform all the
others
and for the late integration here for
the structured data uh the same thing
happened with the Baseline which is the
random Forest without any sampling
methods
and the same happened also with
frequency of even and Text data and and
Text data as well so for the late
integration when we will take the
majority vote between the the the three
models the structure the text and the
frequency data
uh so here I perform weight uh weighted
late integration when the contribution
of the model will be proportional to
them to the to the score of those models
and we end up with a score that is uh
quite great or
0.79 and when we compare the early
integration to the late integration it
is similar in terms of F1 score but here
the early integration has a higher IEC
curve
now for the interpretation here I'm
showing the most important feature that
are from the random Forest building
algorithm
and here I'm showing the result of the
global aggregation here the first one
using the lime importance the average
importance and they're the average of
attributes so I perform it for the lime
and for the shop and here it's maybe not
clear but I'm showing uh the 10 most uh
diagram
that the all the gear all the
aggregation method has shown to be the
most important one and I compare also a
shop to a line to see if I can see uh
see uh backgrounds that are in common
and we can see that there are different
background that we can see both in shop
and in Lime such as a certain doctor's
name and uh other uh
other
other diagram that are really
reliable from a medical point of view
and
here to evaluate the the global
aggregation uh I'm plotting uh if we
remove the top feature predicted by a
wish shop we see that the average
importance increase the performance
so here
the all the the the the the one with the
dot is the Baseline when we remove a
random feature so here we can see like
the the two above line and average
attributes has shown a good result only
the average importance has uh increased
the the performance
so as a conclusion
and perspective
so the first first perspective here that
I am currently working on is to try a
more complex model in order to improve
the performance so I'm currently working
on uh Birds which is uh adapting the
electronical health record for
adapting bird for electrical health
records so here we can see that we can
have sequence of event when we take the
electronical health record which will
correspond to the history of the patient
so the token here would be the event
that happened in in a day for a patient
so uh
that's the perspective for now and as
conclusion uh here random forest for
random Forest resampling methods didn't
show uh Improvement regardless score so
the random Forest is quite robust for
the data that I have and the different
integration methods used are similar in
terms of scores I also have a sharp and
lime that show
similarities regarding the predictive
features
so for text reports and also the global
aggregation model methods use work in
general according to aopc so maybe we
should uh I should try other
interpretation methods or using Birds I
would have other uh other information
such as attention score for the events
so I will be able to compare what I will
have with attention attention score to
what I had for the global aggregation
methods so
that was it for me today I would like to
thank my supervisor Chloe and some
colleagues here all the Serbia members
or my lab members and all the ITN
members thank you for attention and if
you have questions I will be happy to
take them
[Applause]
thank you Market
for this talk
and now we have a time for questions
janisk first
super clear so thank you I have a couple
of questions actually
um for the first one in your use case it
seemed that early integration and late
integration performs quite similarly
maybe early integration slightly better
do you think that it is context specific
or which integration method would you
recommend and why
um actually
I will recommend both methods to be
honest because
um like for the early integration here
the advantage would be that all the
modalities will work together to be able
to give the best prediction and for the
late the thing the fact that we have we
perform machine learning methods for
each modality will be specific to one
modality so it can work as well so both
of them work I will recommend both of
them actually because uh they have
advantages so it will just depend on the
data that you will have so I see thanks
and and my second question was um do you
have any idea of intermediate
integration method
so uh we talk about we thought about uh
intermediate integration method
so I didn't do it yet so it's about
kernel uh so uh it will be a mutable
kernel learning when each kernel will
learn a modality I didn't do it yet but
it's an option to try the intermediate
integration method too okay thanks
I am a short question from Isaac great
talk thank you
um have you tried methods like stack
generalization for late integration it's
an old method here I think described in
the 80s where a learner learns from the
predictions basically of the input
learners
like a neural network for example can be
any sort of algorithm because you did
begging if I understood you correctly
yeah
no I didn't try it yet but I would like
to see if there it would be an impact
with that maybe as a comment if you do
early Fusion there is a certain risk in
clinical reality that if you lack data
that the model becomes not applicable
anymore yeah because you need to have
all the data otherwise you cannot apply
thank you for the questions
thank you magette for the talk I have a
very quick question more about Curiosity
on the aggregation methods of the local
explanation could you explain again the
third one I'm not sure I understood
correctly how it works like do you just
average the features over the data set
and then use that as the
I just sent all the uh all the weight
sorry
I sum all the weight of a feature of the
older local explanation for a feature
over like all the incense that we see uh
the the feature oh okay so that's the
the method that will have and even if I
compare with uh here
when I compare with shop we have the
same top features oh okay okay thank you
okay
all right
quick question is when you are comparing
the area and over the perturbation Cube
okay so maybe it should be interesting
for just a feature of analysis because
when you are removing a filter for
example in a base random Forest maybe to
recompute the relevance for the new
model because
uh what could be happening is that the
random Forest just is speaking one the
top video and then when you remove it
there is speaking another
uh filter that is highly correlated with
the one that Jeremy off because random
policy is not very good for fair
attribution social advances and maybe
the the sub if you are using something
like purple version is capturing the top
uh Distributing the relevances across
the correlations more fairly than the
random forest or something in that okay
all right okay
good
hi thank you great presentation uh just
quick question are you also considering
integrating images like mammography at
some point
uh for now I have uh so I have 15 000
patients and I have a mammography for
only 300 of them so
it will be complicated to perform maybe
deep learning with only 300 patients but
we are looking for like Mutual model
data set that have uh images in it and
with more instances but for now we're
just going to stop here with the text
and destructed data because we don't
have enough mammographies
mammography images
thank you
okay thank you a really nice talk you
know I was just a bit um like curious
because you said that you include the
doctor's name or identifier as a feature
and doesn't that introduce some sort of
bias because I mean there is
an inherent bias in how doctors evaluate
and okay it's quite a
yeah yeah that's a good question because
we were thinking about like if uh there
is a doctor that usually take care of
like the most complicated patient it's
normal that we see them here for the
relapse so another another way to deal
with that was instead of taking the
doctor names is to take maybe their
service or or maybe the name of the
specialty of the doctors and replace the
specialty uh into the Corpus and try to
perform that but I have I had this
comment from the clinician
so it's a it's a good question thank you
yes because I mean but even I think with
the specialty if the dog I mean if
someone is specialized in in more
extreme and more difficult cases that
have lower survival rate or whatever
like or yeah recovery then I think the
models will pick up on that or what I
would expect yeah
[Music]
uh so thanks for the talk um just a
quick question following up on the on
the reports uh have you cleaned the text
somehow or just calculated the bigrams
um so how did you process the
um the reports because also thinking
about synonym mapping and so on thanks
so I cleaned that it's hard to clean uh
actually uh reports because we have
different types we have different
doctors and they all have their own
jargon so I did the uh like the basic
pre-processing for text like removing uh
the uh most common words Etc and uh so
the doctor names is inside the the
Corpus so uh I was not aware until the
the results so that's why but I did the
pre-processing like the basic one when
we uh when we deal with uh NLP so
remove all the stop words and all that
stuff
sorry
for for the YouTube I repeated no
synonym mapping was the question
no synonym mapping no
good so then let's thank Maggette again
and move on to the next speaker