Video summary
Pelin Gundogdu presented her work on a neural network project designed to address the "black box" problem inherent in complex machine learning algorithms, particularly within the sensitive healthcare field. While standard deep learning models offer high performance, their lack of interpretability makes it difficult to understand how they arrive at specific conclusions. To solve this, the team integrated biological knowledge directly into the network architecture by incorporating signaling pathways as circuits in the first hidden layer. This design ensures that the model's calculations are grounded in real cellular operations, where each circuit represents a functional unit with receptors or genes that pass signals to subsequent processes, thereby enhancing the explainability of the results without sacrificing predictive power.
The project focuses on three core tasks: cell type prediction, cell type annotation, and interpretation using this integrated biological knowledge. The team utilized two datasets derived from the SCITE paper: a balanced PBMC dataset with seven cell types for prediction analysis, and a melanoma dataset used to test both annotation and interpretation capabilities. Through rigorous experiments involving 100 iterations with certified holdout validation, the network demonstrated high accuracy, precision, recall, and F1 scores across all cell types. The results showed that the model could effectively predict cell types and successfully identify unknown or abnormal samples by clustering them separately, proving that the encoding layer retains sufficient information about cell differentiation even when encountering unseen data types.
Beyond standard prediction metrics, the study emphasized the importance of biological validation through novelty detection and local outlier factor analysis. By calculating similarity scores based on the activation of nodes in the first hidden layer, the researchers could determine which samples were abnormal or belonged to unknown cell types compared to the training set. The findings indicated that while performance was slightly lower for specific cell types like NK cells due to limited sample sizes, the overall accuracy remained competitive with established algorithms like SCITE. Furthermore, the analysis confirmed that the biological circuits actively participated in the decision-making process, as the most active pathways for specific cells, such as B cells, aligned perfectly with known biological functions like proliferation and protein secretion.
The presentation concluded with a discussion on future directions and responses to audience questions regarding model robustness and parameter sensitivity. Gundogdu highlighted ongoing work on cell type trajectory projects using cancer datasets to understand cellular behavior changes during disease progression. When questioned about the network's performance in distinguishing biologically similar cell types, she noted that while the model sometimes struggles with very close clusters, collaboration with biologists to refine definitions could improve results. Additionally, regarding the hard removal of connections not related to specific circuits, she explained that shuffling these connections led to a significant drop in performance, confirming that maintaining biologically meaningful connections is crucial for the network's success, though they did not explore soft regularization approaches as suggested by the audience.
Read the full video transcript
so peline is one of our
esrs at a Spanish node supervised by
joakin and is the final speaker of our
first day of the symposium
that's ready to go yes so we're looking
forward to your talk thank you so much
Carson uh hi everyone
today I'm going to present you
an air Network project we developed and
oh so yeah before I started I'm applying
mention my advisors are Hawking Isabel
and Carlos
and I start I started September 2019.
so as I mentioned earlier before our
project is neural network project and
more space this project are really a
more complex algorithm stand regular
machine learning algorithms and we are
using them and this complexity enables
us to work a really complicated problem
and they produce really well performance
however this complexity also causes an
issue called black box in Black Box
concept and the calculation during
training are well known however the
interpretability of the results is
uncomprehensible
and this lack of explainability
can cause a problem especially if we are
working in healthcare field
and for this reason we are trying to we
are we we are using
we are trying to implement a biological
knowledge into Network more specifically
a signaling Pathways as Hawking
mentioned in his earlier speak
and a signaling pathway is a is a
cellular level operation in an organism
and each each
pathway can either call either creates a
product or an helper signal for the next
process
and we are specifically using circuits
circuits is the smallest element of the
pathway and they have
um they have a factored or receptorgin
information
and we are trying to we are using this
information to improve the interpret
interpretability of the network
the network is uses signal signaling a
single cell earnings sequencing data and
it's uh it's a predict or annotate cell
type information
and to summarize uh the the project I'm
presenting has three main Concepts uh
cell type prediction startup annotation
and interpretation so a startup
prediction is simple supervised approach
it's just predicting cell types cell
type annotation is focusing on the
hidden layer information and the
interpretation part is focusing on this
integrated knowledge which is the
signaling pathways I'm sorry circuits
and our design is fit for nail network
with Dance Connection
and we are using this biological uh
knowledge in the first hidden layer and
we are implementing the implementation
is applied by using
um each node in the first hidden layer
as the circuit which means if the genes
which is the input layer has no
connection with the selected circuits
done we are removing the weights of
course here
and for the data sets
um we are using two data sets for this
project and before giving more details
about data set I would like to mention
one paper which called sipet cybet and
this paper is important uh for us
because both of the data sets are shared
both of the data sets is obtained from
this paper and we are using exactly
exact pre-process step as they did
because this paper also shared some
results according to their algorithm
because this cybit algorithm is one of
the well-known algorithm for cell type
prediction test and they produce really
competitive results
as I mentioned we are using exact same
versions as they they use and the first
data set we called is pbmc it has it is
a balanced data set and has seven
different style types
and the second one second one melanoma
there are two subsets for this data set
there are they called training and
testing uh training a testing has this
exact same data set as training however
there are one additional cell type which
training doesn't don't doesn't have
and that's and we are using pbrmc
dataset for um cell type prediction
analysis and the melanoma data set we
are using both interpretation and setup
annotation tests the task yes
uh the purpose of this analysis is just
to understand how well the how well our
proposed network is because the purpose
is just to get the prediction versus
Grant true and for these purposes
uh we we design an experiment with 100
iteration by using certified holdout
validation and each iteration is uses 70
30 of the data set as a testing data set
and these are the seven cell types which
has the same exact number of sample size
and evolution is um for evaluation we
have five metrics accuracy balanced
accuracy F1 precision and recall scores
in addition to these five metrics we are
also generating a rich confusion Matrix
uh these two figures is the the
evolution Matrix result the top figure
here is the uh belongs to these five
metrics uh actually for balanced and
accuracy balanced are
actually uh yes all of them seems
similar because uh our data set has is
balanced data set however we actually
generate a calculating the the this
paying attention the distribution of
each cell each cell type however we are
seeing a similar results but this is not
an issue because the data set is
balanced and both graphs are taking a
value between 1 and 0 1 is for best
worst for the zero is the worst score
and as we can see here they are quite
close to one and this shows us that our
network is working for Salter prediction
in the bottom figure in addition to
General overall picture we also pay
attention to to Startup details
Precision F1 and recalls course is
calculated for each cell type detail and
they are all close to one however there
are a little bit slight difference for
these three cell types
and so toxic memorizers and regulator
however again like overall scores are
able above than 0.7
so I'm sorry 0.8 so in overall picture
shows that the prediction for each cell
types working well
and this is the confusion average
confusion Matrix and this is a basic run
through versus prediction and the
diagonal shows how well the
predictions are and for these three cell
types uh we are slightly low getting
lower performance however in
biologically
they are quite similar and we as we can
see here each verse prediction is below
is distributed by is distributed by
these three cell types
uh as I mentioned um cyber paper shared
these three results in the bottom these
three algorithms are the most well-known
and most commonly used algorithms in
SATA prediction tests and our results
isn't in the top in the top figure which
is quite similar to cybet and we are
producing really competitive results as
cybed and however in addition to these
three algorithms we are aiming to
interpret our analysis our Network
and for this purpose
um after we saw that our predictions are
really good our assumption was if we are
getting good prediction in output layer
then the encoding player which is the
last hidden layer of the network
should also have the information about
the information about the startup
differentiation
and for this reason we the experiment is
started with in in two steps the first
step is basic visualization just
creating clusters and the second one is
novelty detection
uh which Define each which is we are
trying to find the similarities for
samples and then try to understand which
samples are relatively
um abnormal than the rest of the data
set and interpretation is again looking
the biological layer which is the first
hidden layer in the in our design
and more specifically
and we are looking the activation scores
of each node in the first hidden layer
for each cell type which means basically
we are just getting the most active
notes
for each cell types and according to
this selected cell types selected nodes
we are making the analysis
and for example for B cells the most
active pathway should relate it with
side to side communication prolification
protein expression and secretion and our
results shows that the most active
pathways are related with these
functions cell functions and this output
provides us that during the startup
prediction the biological the integrated
biological layer is taking an entire
account
yes and for next one as I mentioned we
have two steps for this analysis and
both of the analysis is using encodingly
encoding layer yes
and the experiment is designed by using
70 percent of the training set here
and then after model is trained a full
training set and the testing set is used
for this experiment and the as I
mentioned the purpose of this analysis
just to provide a visual proof because
we want to see can encoding encoding
layer can create a clusters for each
cell types one important cell type here
is negative cells as you can see here
this negative cell doesn't exist in
training set and we are trying to while
we are trying to create a separate Cloud
clusters for this cell type while also
getting separate classes for rest of the
cell types
these are the two results as I imagine
70 percent of the training set is using
for the model training and this is the
full sample set for uh for the final
figure
as we can see here it's cell type which
are the non-style type for the uh for
the model can is creating one cluster
for each cell type and for the next one
testing
and as we can see we all we also is we
also can be can see that
um one cluster for each cell type but
also we are seeing that one separate
cluster for unknown setup are unknown
cell type and this shows us uh the
encoding can be used as a is it is a
startup differentiation and for this
purpose
um we are using a similarity score by
from local outlier factor analysis
and um
and the steps are
in this analysis start with calculating
the encoding information
calculated by similarity score by using
encoding information this similar score
is
um shows how for giving sample
uh so this this uh this sequential that
for giving sample uh have a half similar
this sample for the rest of the data set
and it gets a 0 to negative infinite
zero if zero if we are close to zero
zero it means uh this this giving sample
has a relationship or more similar that
or more similar to rest of the sample
set
and the next step is we are by using
these similar score We call we generate
a distribution plot for each cell types
and by using these distributions to
chords we are calculating a threshold
and this threshold will help will is
helping us to decide which samples are
gonna assigned as an unknown and which
samples are going to execute for our
Network
and for this if the calculated similar
to the score for giving sample is above
from this threshold then we are
executing our Network and end of this
end of this network analysis we are
getting a SATA prediction
and if the label if the value similarity
score is below then this threshold then
we are saying that the network didn't
see doesn't see this label before so
that that means like this is unknown
uh this is the final results after
following all steps and as we can as you
can see here
we are getting a high accuracy for
all the cell types and also we are
getting a high accuracy uh for a no
setup identification one issue I would
like to address is the performance of NK
as you know the neural networks work
work better with higher sample set for
this status cell type unfortunately we
have small set a small sample set that's
why the performance is relatively lower
than we compared rest
and this is the comparison between cybet
and our Network and as we can see here
we are getting a higher performance
except NK
and also we are getting a similar
performance for um unknown cell type
annotation
uh the results I showed is still ongoing
one of our ongoing projects and it's
funding by Europe is several European
agency and before uh ending my
presentation I also would like to
imagine our latest project so we are we
are currently working a cell type uh
trajectory project there are several
studies in this field and basically in
this project we are trying to find the
cell type Behavior according to uh yes
for this state for this project we are
using cancer data set because we are
trying to understand uh the
changes in the cell type when it's
called it's became the cancer so the
idea is if we are try if we if we see
this trajectory which is like the pets
we can understand how the cell behavior
for the patient or the treatment all the
or the cancer stage so these are the the
full sample set we are using as you can
see there are multiple
and yes and we are in the initial stage
however we are getting
um the results we get so far is what the
literature says and we are using a
variation of the encoder for this design
and also we are using um signaling
pathway also in this project
thank you so much
[Applause]
foreign
[Music]
now time for questions Lucas
next question thank you very much
billing it was a wonderful talk very
clear and I also love not only the
content but the progress bar at the top
yeah it really helps with anxiety
um yeah I think everyone here is the
same coffee
I'm starting to work a bit with our
single seller and async data and uh yeah
cell type and annotation and prediction
seems to be kind of hard in some
situations I was wondering if you know
how well this algorithms perform in
situations at which these cell types are
not that different for example with
single nuclear rna-seq in which the
different cell types like for example
different types of neurons
[Music]
um
for I mean in biological point of view
I'm not like that expert but I know
there are for example in our result like
confusion Matrix if the cell types are
really close to each other
the network is quite
underperformance so yes for this purpose
I think I mean in my opinion it worked
with some biologists to understand or at
least finding a route for the cell types
might be might be helped for the network
questions
so I have one question you you mentioned
the local outlier Factor yes
um and this includes a number of
parameters are your results sensitive
these parameters or have you explored
this or how do you set them to make yeah
actually this was one of the topics when
we are using this network tool
um how we use the default feature
because our aim is to see can be
separate or candid default I mean if we
are even if we are using simple design
can we find this separation between cell
types by using the simplest model so
just the default parameters in some
software implementation of LOF okay
good further questions really
yeah Giovanni
thank you I have a quick question on the
structure of the first hidden layer
where you mentioned that you keep only
the connections for the genes that are
relevant to the circuits have you
considered doing that instead as a soft
version so instead of literally removing
the the links between the neurons just
adding some regularization term in the
uh in your training procedures so that
the first layer is incentivized to mimic
these connections but also allows for a
little bit of wiggle room to try to find
something extra do you mean what the
regulation do you mean like
um I mean L1 L2 regulations no you can
just add the term sorry to the loss uh
to penalize how far the connection goes
from these hard
um cutoff but then since it's just
determined the loss you could still get
a little bit of other contributions and
try to get a more General result
actually we didn't but we so we make
this analysis which is like we we
Shuffle the connection you know I mean
we assume that I mean we keep the same
number of connection but we shuffle them
and however when we shuffled them the
results we are getting is really
full performance but we didn't
do anything about what you are saying
but when we shuffledam we saw that
um this shuffling is really important
because the discipline it doesn't mean
for the network but keeping the is a
biologically meaningful is providing
performance
thank you
are there further questions
if not then I would like to think
pilling and all the speakers of the
first day of this Symposium thank you
very much
thank you