Video summary
The Concourse Working Group meeting on September 9, 2026, was a brief session primarily focused on the upcoming patch release scheduled for late September. The team discussed how members could submit critical bug fixes or necessary updates by leaving comments in the repository issue tracker, specifically referencing issue number 831. While there were no major agenda items requiring immediate action, the group acknowledged that some requested changes had already been identified but not yet reviewed. This collaborative approach ensures that any urgent improvements can be integrated into the next release cycle efficiently.
A significant portion of the discussion centered on technical challenges encountered during an update to version A3, particularly regarding compatibility issues with new worker versions. The team reported difficulties in smoothly updating their infrastructure, which sometimes required manual workarounds rather than automated processes. Additionally, they identified a recurring race condition involving containerd that manifests during rolling web server restarts. This issue causes containers to be rescheduled unexpectedly, leading to errors where the system attempts to spawn a process that already exists, resulting in lost build outputs and temporary disruptions in production environments.
The developers are currently investigating whether this race condition stems from recent changes in containerd itself or from specific interactions within their own codebase. Suspicions point toward a potential mismatch between how the web server handles database locks and how it communicates with the worker side during restart sequences. While the problem is not constant, it has been observed across multiple environments, prompting the team to attempt reproducing the issue locally to isolate the exact failure point. They plan to open an issue once they can provide concrete steps or log data, inviting other users who might encounter similar problems to contribute their observations to help resolve the mystery.
In conclusion, the meeting ended with a consensus that while the identified race condition is annoying and not ideal for production stability, it is manageable until a root cause is pinpointed. The team agreed to continue monitoring the situation and will report findings as soon as they can reproduce the error consistently. With no other pressing topics on the agenda, the group thanked each other for their attendance and commitment to maintaining the project's health, promising to reconvene at the next meeting with further updates once more information becomes available.
Read the full video transcript
Okay.
Hello.
>> Hi.
>> Hey.
>> Hello. Hello.
>> Sorry, I I'm kind of
mobile
right now. Uh, so I won't be able to
share my screen or anything, but I don't
have too many updates to go over.
No worries.
Do you need us to share anything? I can
share my screen maybe if that's
>> uh I did look at the meeting doc. There
isn't anything on there. Um and I know
Derek's not showing up today either.
Actually, him and I talked last week.
Um, but yeah, there isn't.
Yeah, there isn't much to talk about
besides uh the patch release, which I
think I pinned that issue in the repo,
the 831. Uh, if you guys have anything
that's kind of like critical, like bug
fixes that you think would be nice to
get into the patch release, you can
leave a comment there. I know I think
someone else did. I haven't reviewed
what they requested yet. Um, but yeah,
if there's anything there that you guys
want to get in the patch release that
I'll cut sometime probably uh like last
week or second to last week of
September,
we'll put that in there release.
from our from our side more or less we
were working on updating to A3 but uh we
had some issues because of the new
worker versions and uh it's not going so
smoothly so it would have to be with
some uh shenanigans on our end so or
manually which is not ideal uh because
the workers are required to be with the
new version when web restarts so we're
working around that but Um
uh the other thing uh that we are
currently observing which might be
interesting I don't know if anyone else
reported is there's some kind of a race
condition in containerd
uh sometimes during uh rolling web
restarts we we noticed that uh um
containers get rescheduled uh either on
the same worker or or another one. So I
don't know why another one was selected
but I think once or twice we saw that
and we hit a race condition where the
one of the processes continues working
on the old worker but uh concourse tries
to spawn the process again for some
reason the web loses track of that and
uh we get a container already exists
error from containerd. It's not it's not
all the time and it's pretty easy to
reproduce. I'm currently trying to uh
we're currently trying to reproduce it
in a stable manner maybe with some slips
in the code or whatever to
>> isolate where specifically the issue
occurs from but definitely noticed it on
our production environments and it
causes some somewhat of a problem
sometimes. So uh
>> okay yeah
>> I've been more or less focused on that
after my uh long vacation. So I'm
currently looking into that and if we
have a stable way to reproduce we will
report it and uh but we noticed it on a
couple of environments so it's
definitely there. It's just a matter of
how to accomplish it.
>> Yeah that's that's interesting. Um yeah
I'm wondering what exactly would have
changed like do you think it's something
with containerd itself or is it
something with all of [clears throat]
our code uh related to containerd
>> it's our code or their code
>> I I think I think it might be uh so my
suspicion is since we didn't see it for
the last year or so since we switched
from containerd and we started observing
it quite recently maybe it's either
related to that to some new change in
containerd or how we handle it. But
before I I am sure what exactly it is,
it would be just speculations on our
end. I'm
>> at the moment the the error we're
propagating is coming from containerd
but the error is more or less that such
an uh entry in containerd already exists
and and this specific error happens when
when the same worker is reselected. So
for some reason the worker tries to
spawn the the process again and it's and
it reuses even the UID for for the
container and it says such a container
already exists. So I I I only notice it
when web server restarting. So there's
something in in the way how for example
I I think the first web shuts down.
Maybe just the DB lock was lost and then
web two tries to
>> uh creates the lock. So it says I'm I'm
now the primary and tries to spawn the
process because it doesn't know anything
about it and uh the race condition
occurs. It it's it's still a bit foggy.
I'm looking into it but it's definitely
do the two webs and uh it happens during
one web. So when the webs restart, I
tried with a local concourse and more or
less did some sleeping around before the
container is created in garden. So after
the DB handle is uh to release the DB
handle before the garden container is
created. So the second web picks it up.
But I'm not able to reproduce it at the
moment. So it it's something along those
lines but not not entirely sure what is
it.
>> Yeah. Yeah. There are a lot of parts
there where it could be breaking down.
Yeah, you've got me thinking about it
now now too where it could be breaking
down as well. But okay, cool. If feel
free, even though you don't have like
concrete uh like steps or anything right
now, feel free to maybe just open an
issue about that, Ken. I can maybe like
take at least a look at it. maybe give
some pointers on where to look in the
code as well because I know there's
stuff on the worker side as well that
maybe we're messing up somewhere. But
yeah, it could be just yeah changing
behavior maybe in containerd and it's
not lining up with because I know like
when we start the process on the
container, we always do like a lookup
first to see if there's an existing
process. So yeah, to your point, there
must be some sort of race condition with
either that lookup is failing. It'd be
interesting to see when that happens
too, right? like did that initial lookup
fail? Did we not find the process and
that's why we're spawning a new one and
stuff?
>> So from from my side the one thing I'm
currently focused on is playing around
locally to reproduce it so I can either
dump some logs or whenever it gets
reproduced on production just dumping
the web logs because I I think the web
logs would be key to understand what
exactly happened with that uh
>> container.
uh on the worker side I think everything
is on is is ongoing. The only fact is
that the stream is lost and we stop
seeing the build outputs. But uh under
the hood it keeps going because we see
git uh conflicts. So I assume at some
point the first build reaches the part
where it should get commit.
>> Uh it's just not uh it's just not
visible in the in the UI of concourse
but the worker itself continues working
on that process. So it's something along
those lines. And I I think the the
missing part at the moment for me is
having the web locks uh of that specific
at that specific point in time to maybe
analyze what part of the code
>> uh is uh stale or hanging. But yeah uh
if if I have
at the end of the day I will time box it
to some uh extent and whenever I
whenever I summarize my thoughts I can
open an issue but I'm not sure without
the steps and without anything concrete
I can more or less it would be there's a
race condition sometimes it happens uh
maybe if you have some observations
leave them in the issue so we can try to
resolve it. But yeah, and
>> it's hard to think of uh yeah, maybe if
some other people see the issue, they
can pile up and say, "Oh, we see that uh
occasionally or something." But it more
or less happens during rolling updates,
which I think most people don't do in
the sense we do with Bosch. So that's
might that also might be a way why they
don't see them.
>> Cool.
>> But yeah, I I'll let you know. It's not
uh it's it's not that so it's it's a
pretty uh iffy condition. It's not
happening all the time, but it's
definitely uh not not cool.
>> No, obviously not. All right. Cool. Hope
you're able to find something. All
right. Uh do you guys have anything else
you wanted to bring up?
>> Not on my end.
>> Nope. All right. Cool. Then we will end
it there then. Very short brief meeting.
Thank you guys for showing up as usual.
Appreciate it. Till next time.
>> Talk to you soon. Bye
>> bye.