Beyond the Linear Trap: Scaling CI/CD with State-on-Demand - DevConf.CZ 2026

Most of the problems are not technology-driven, but basically process-driven and people-driven. And my talk is going to not be about technology, but about processes. I noticed that many things were infrastructure-related. So, this is going to be a little bit different here because we are going to talk a lot about testing. And in infrastructure, even test environment is in production environment, whereas from application point of view and from business point of view, before putting anything in production, you need to basically test it. And nowadays, people try to automate everything. So, this is what is CI/CD. It's to automate everything except the last manual step, which should be as early as possible. So, typically today, it could even be your prompt to your AI system is the last manual step, and then everything should be automated. So, this is CI/CD. Automation means and is put in production, it means that customers can use it as soon as possible. Why as soon as possible? Not because of technology and because of pressure, etc. Because of you as developer, you don't want to have anything to remember for tomorrow. So, you finish your work, which is your manual task, and you want [snorts] to put it in production automatically as soon as possible, so that if there are any issues, you have the feedback immediately. Yeah? So, so I guess maybe 1% of enterprises in Europe are have that. Usually, there are always some kind of product managers and testers, etc. involved after building, and on the test environment, they want to verify that things work. And so this this basically the situation, but you know, everybody tries to go to to the CI/CD concept. So, if everything is automated, do we have a problem? You know? That's the question. Because the problem here is this kind of validation before you put it in production. You want to be sure that what you have prepared is going to work, that you are not going to break your product. And this is this is like a risk-based approach. So, what you usually do is you are going to run a lot of tests, automated tests. Uh and those tests will verify the whole system. So, you can imagine in a bank when they do any release to production, which can be on a daily basis, they have to verify that you can create a new customer, that you can you can create an account, that you can order a card, that you can pay with a card, that you can cancel the card, things like that. And all that needs to be verified because otherwise you would never automate it. You would still want some testers that will put their heads on on the table and certify that everything is going to work. The problem is that those processes by design take time. You can't just say, "Okay, we we we will deactivate a card." You first have to have a card. And before having a card, you need an account. And before an account, you need a customer. So, it's by nature linear. It's called business flows. And the problem is that if you want your CI/CD to finish within some short time frame, maybe several times per day, basically you don't have any other option than to parallelize those tests. And that's the main issue of automated testing when you have CI/CD. How to ensure that you can run all tests in parallel while usually only the first test, which is maybe login, can be run as as initial step. So, the idea is that we have everything automated, but today we need to focus on how to automate also the state from from which each test starts. And the idea is what we call it is that you need some kind of test data provisioning. So you you have your test environment, your stage environment or how you how you call it, and you would like to have on-demand test data provisioning. So where where you want to run a test, you just pick a state from which you begin, and then you just validate that some your use use case work. So test data provisioning and state provisioning is almost the same. And that's the purpose of my talk today is to show you how this can be achieved. Um because usually it's it's it's a technical and process challenge. So um Let's go. Uh just for your for your information, I have been focusing on test data management and CICD for really a long time. Uh I'm driving I'm driving a QA community Europe-wide, which is called QA Kitchen. It's It's online. It's a monthly talk. And I like to be challenged. So even today, if you want to challenge me, just come here. And I have uh two gifts. One is uh a coffee a coffee coffee card. The first who challenges me has has a free coffee. And then I have other gifts for everybody. Um back to the end-to-end tests. Uh I like to challenge myself, and usually where I go, people challenge me. I say, "Okay, it's microservice time. We don't need any any more end-to-end tests because you know, it's called contract testing. You have microservices, and each microservice can be tested in isolation. We have advanced concepts like uh contract testing, packed testing, things like that, where you define the API, and then you you have some systems how to create automated mocks. You go you're going to test the client side separately, and you're going to test the server side separately. And that's that's that's something that even the biggest Czech bank implemented and they claim it works pretty well. But, you know, I have been in testing for some time, and I know that even if you have all unit tests pass, all contract tests pass, that's that's almost certain it's not going to work together because the paradox is that the majority of the functionality which can be tested in unit tests does not really match the the minority of functionality that the customers care about. You know, if you want to use your car, you you don't care that you can unlock it, that you can uh have a speed passed in, that you can turn the wheel. What you care about is that you can get to the final destination. So, this is why end-to-end tests are still required because usually it doesn't work, even if you use advanced systems how to split the problem. The challenge number two is uh you know, with Kubernetes and I've I'm I'm sure you have heard a lot of talks about those stuff here on this conference. Kubernetes and all those serverless things allow you to have an a test environment on demand immediately. Yeah, like you can have test environment ready, and then you basically uh run each test on a different environment. Why not? Or you have an environment with data already prepared, and therefore it's not difficult to just have uh have it done for each pull request. The problem is that this works until you're a small startup. Once you get a little bit bigger, you hit the problem that you need all teams in your enterprise to have full automation. And this also means that SAP guys needs full automation, or CRM guys that have some kind of heavy billing system, legacy stuff. And um it starts to be also expensive. Uh so, the the return on experience from the from the practice is that until you're a small startup, you have a client, server, and maybe some small middleware, you can have a femoral test environments, and this is enough for testing your your your delivery, but once you get a little bit bigger, you basically need long-lived environments. So, your stage environment still sits there forever. You need to maintain it, to upgrade it, to clean data, to add data, and things like that. >> [snorts] >> And I have a third challenge, like why why test data on demand maybe is not a good good idea. And this is like legacy approach that everybody has been using, and maybe a lot of people still use, and this is copying data from production. Because if you have production, you have a lot of data there, so why not, you know, copying it on the test environment, maybe run some anonymization, and then you can start testing, right? So, so you can create some scripts, or you can buy a tool. There are plenty of tools out there that can do that. And you you you you you you do this magic that suddenly from production, you have a test database that has anonymized data, and you can run tests, and you can for every test, you can have a different test customer, and things like that. Now, the problem is that uh you need to be able to identify the data. And uh you need the rich data. So, for example, if you are developing very fast, the new feature is not yet on production. So, how do you get data produced by the new feature? It's not yet on production. So, you you get issues like that, but the main issue that I see is again with the modern technology that brought some changes here, and this is with microservices. Because if you have everything in one database, yes, okay, let's create the heavy magic script that will copy the database, anonymize it to test. Now, if you You hundreds of services, some of them you don't even know which database they use. may You don't have access to the database. They are in the cloud, in various clouds, they are on premise, wherever. Well, then you need hundreds of magic scripts that needs to be aligned to do the thing the thing at the same time. And this basically doesn't work. So, we are after the three challenges that I see every day, we are back to the hard facts. We need some test data across the whole landscape, whole microservices that are somehow aligned. We call it referential integrity. Yes. We need [snorts] to stop copying data from production because this is no more feasible with microservice and distributed computing. And we also need this long-lived stage environment because of legacy teams and complex things that are difficult to manage. I can give you an example. I was working for for a Czech Czech company doing uh uh food delivery, Rohlik.cz. They had like ephemeral environments and suddenly they said, "Okay, let's automate our stock. Let's put robots, etc." And they bought they they they they did they did a deal with a with a big company that does these robots and they they came with Windows. Yeah, we are going to to use Windows for that and and we will have a server and suddenly ephemeral environments are gone because you you can't have just like a new warehouse for every team. So, this this is true for in almost everywhere, actually. So, let's go to this on-demand test data. And it's not synthetic because synthetic usually is said to be just, you know, fake in one system only. Here, the concept is that a test, which can be a manual tester or an automated test, wants to start testing and first thing that what it does, it requests some state. Yeah, so you have a kind of request demand system. And then you have a test data service that just basically provision the the the data in your systems. This is let's say the magic and it returns the data set to you. So, the data set can be username and password. But, you know that behind that you have your user created in your user's database, you have your the user has an account in another database, etc. So, this is typically what a request can be like. I want a customer uh that has some kind of subscription, that has access to digital channels, uh AML checks have been done, active card, and some payment history, you know? So, how do you do that in a in an ecosystem which is a little bit more complex? You have microservices everywhere everywhere, you have middleware. So, you can't just insert data in all databases. That's not feasible again because many people see differently, etc. So, the idea is that you know, you do it one by one and you respect the business process. So, that's the idea. You you just follow the business process because this is how thing how things are have be have been made. This is how things work in production. So, if you follow the business process, you have basically a high chance that the data are going to to be created right. So, you you just follow the process, you create the data in each system. So, already now you see that there's kind of orchestration behind. >> [snorts] >> But, the basics are basics, yeah? So, instead of inserting data in the databases, you need to expose APIs because some systems don't don't have databases. Some systems are managed by different teams that don't want you to access the database directly. So, uh so the idea is uh that each system expose APIs. And like some practical advice, if you want to do to go in the direction, usually in a in an enterprise with hundreds of teams, there will be 20 teams that will say, uh yeah, we will not do it. So, then, okay, you can have a workaround. you you take your automated tests which act on those systems, which can be UI automation, things like that, and you can fake the API. You have a You have even some startups that do that, that actually expose UI automation as APIs. So, suddenly you have You have like each system has some regular integration APIs to communicate between themselves. And they have another set of APIs, which is called test data APIs, which you can use to create data in the system. Why does it make sense to have a separate set of APIs? Because sometimes you need to do a little bit more than what you do in production. For example, you want to create data back in time, which is not allowed in production, but in testing you need it. Or you want to create products that no more are active in production. So, therefore you It makes sense to have testability APIs for creating test data. Um then the next idea, once you get that Once you get there, it's still not going to work because it's going to be broken all the time. Cuz on the test environment sometimes systems are broken, that's normal. You have to count with that. So, you need to retry, yeah? So, if you try to provision test data, the system is not available, you retry 5 minutes later. So, okay, you can put a circuit breaker and things like that or some kind of component there. But then there's another idea, and this one is the game changer, and is buffering. Because you can pre-create data ahead of time. You don't have to do it once the test requires it. You can do it the previous day or the previous week. And you only need to store it somewhere in a buffer, and once the test need it, you just deliver them to the the the data that is already ready. And of course, you can have some clean-up and life cycle, etc. So, then the architecture becomes a little bit more complex. You have some kind of gateway or central system or whatever. It It looks like an API gateway, yeah, but it has some more some more features. Where you the requestors request data and they get them pretty fast because maybe those data were created already before. And if the system don't work, there are some retries, etc. So, you decouple the life cycle of systems and of your tests. So, you will tell me, "Okay, so it doesn't it doesn't work because if the systems are broken, basically my test will not pass." Well, you know, for creating this data the data sometimes you need systems that you need only for creating data. For example, in a banking environment, you will have AML systems to check whether you have you are not a murderer or any anybody else. So, those systems are required only to validate that the customer exists and then you don't need those systems for your for your use cases. So, so in many cases, decoupling makes sense. And then once you have that, it comes completely immediately by itself. You can have discovery, you can have a catalog, you can have observability plugged in because you have a central system that enforces some some some kind of some kind of, let's say, homo- homogeneous way how to how to work with the APIs, yeah. And the third idea that comes also there, this is this orchestration. So, each team you can you can you can take each domain is managed by different team having a different manager having a different director, but each team can expose some API, yeah. So, you can this you can somehow enforce. And as told, you can somehow fake it with a test automation. But then you also need somehow to to plug that together. So, this is called orchestration or composition. So, here if you don't if you're not not aware of orchestration pattern, this is something really to to look at. I I will skip that here, but this is a very very powerful pattern that has been there for years, but let's say last 10 years everybody says, "Okay, let's go with even sourcing because even sourcing is decoupling everything. Everything is independent. So it's a perfect. We don't and then suddenly things start to break and there are tools like temporal or persistent workflows that come back in and the orchestration patterns is really something that will that will drive the next years. That's it. This this that's that's my bet. Yeah. So So the orchestration is basically you have a component that just calls other components and is responsible to know in which order and how often on excited. So here you have some kind of orchestration and service ABC and the orchestration knows in which order the the kind of request should be fulfilled. So once you have that you can just play Legos and the initial request we had like a customer with an account with a card with a payment history, you can construct it pretty pretty simply from unit unit API calls like create a customer, create an account, create a card, things like that. Each of those let's say generation APIs can be delivered by different team and you only need this kind of orchestrator somewhere which is of course not given but still manageable. Right. So those are architecture patterns and this is somehow how it can look like. So you have one agile team that cares about systems which is on the top here. And at the same time this team cares about something called generator or a test data API. And the idea is that you can completely plug that together. It can become one system like instead of thinking of it as some external system you can have it part of your system. So the only thing that comes external then is kind of portal or gateway or API gateway that allows you to call all your test APIs and this gateway can play the role of the decoupling, buffering, catalog, discovery, observability, and things like that. So, that's it. That's the this architecture. And the benefits again, uh you can have test data ready in microseconds, whereas before it takes days to weeks to get because I can tell you from practice, if you go to any Czech bank now and you say, "Give me a test customer." You're going to spend days to weeks to get it right because everybody just knows a small bit of it. Then every test scenario can have a new test data set, completely new, starting from a state that that you need. So, you can have a deactivated customer. Let's have it in microsecond. Let's have a a activated and then deactivated and then activated card. Let's have it in 1 microsecond. And you can have any unit any any kind of end-to-end test starting from any state. So, then instead of having 100 minutes, you have 1 minute. And uh the goal was what? It's that the developer can go home and it's done in some production and already customers are using it. So, we know that the probability that it's broken is very very small. And that's the goal of CI/CD, to just be fast and have fast feedback. And the test data were just the last bottleneck, I would say. So, here are your key takeaways. Um this is based on things that we have done in uh several customers because of course I have a company etc., but uh so this is an architecture pattern that I I think is worth doing. You can use your own technology to do that. It's not it's not really um uh there is no lock-in in any kind of o- or closed source stuff. And the ideas are to to have test data APIs, to decouple them and have buffering and retries using some kind of gateway that you probably have to program. Um but it has a lot of benefits like uh some standardization of the test data, etc. And the the core idea is that test data APIs become integral part of the system. So, before the system was made of application code, now we have application code and deployment code and infrastructure code. This everybody knows. We have also test code for test automation. And this is the last bit that was missing, code for creating test data. And this is basically the product for each team. Thank you very much and who wants to challenge me? There's a free coffee here. >> [applause] >> You you Is this a Is this a challenge or >> I don't know if it's a challenge or a question. >> [laughter] >> Uh Yeah. So, you're talking about preparing the data before before you get the data. >> Mhm. >> Or you know what you actually need. You also are talking about the composition of for the user and the account. >> My question is how can you prepare the data before you actually need >> Yes. >> And also in the composition, I can probably imagine that it might work in one domain where you have the customer and the account and one team manages the data and creates some smart stuff that connects together into components. How would you do that, for example, for the payments and the customer data? >> Go through >> Okay. Uh very good questions. I will probably repeat them for for online guys. So, um So, uh one question was how do we know which data to create up front? So, if you have this kind of gateway here, I don't know, maybe I'll put this this this one. You have a test data catalog, yeah? So, basically you are building a some the test data catalog tells you what what test data you are able to create, and each item you can you can think of it as it has a name, like customer, and then it has a set of properties. So, you can model it as a vector of properties. So, of course, you will not do all possibilities up front, yeah? You will try to optimize for some kind of subspace in this vector of properties that is likely to be needed. So, initially you can do it manually, then you can do it by observing what data are going to be are getting used. And third, you can use AI to but not like a big LLMs, you can use like machine learning just to to see what patterns are repeated, and what patterns it makes sense to prepare up front. But, this is like theory, and when we started to do this kind of architecture, we thought of exactly this, and at the end we discovered that this is not the problem. Because you know the tests, like people know the tests that are going to run in the CI/CD. It's not something that is created overnight. It's something that is created over years. So, basically just by talking to people, they will tell you, "Yeah, we we For us it's critical to have a customer with a bank account that starts with this, and card, which is a credit card, yeah." And so, it's rather easy to get over that, to to have the to know let's say the the configuration of your test data you're going going to going to going to need. And second question was how to ensure that you can do to composition across the full organization, yeah? So, and something that was not in my presentation, but is an advanced idea for those that uh understood like what this test data data catalog means. So, here actually it's here it's hidden. This orchestration, which is a small snippet of code. You can imagine that it's just create a customer and add a card, yeah? Two lines. So, instead of calling directly here the guy that creates a customer and adds a card, they can call the test data portal back. And then they have the benefit of the catalog of discoverability, retries, etc. So, you can plug you can you can you can plug the all the features that you have for your requesters here, which are automated tests or people. You can plug them All right. Okay, sorry. I guess this going to Uh yeah. So, so so here, instead of calling directly the directly the the other generator, which is bound to some system, you basically request your let's say subordinate data from the portal. And then the portal can apply all aspects of this solution, which is retries, buffering, and things like that. So, therefore, it's not a big bang implementation. You start by doing a couple of generators like the team for, I don't know, customers will start to create customers. And the next month, you can go to the guys with the the the cards and say, "If you want a customer, just click here, you have it immediately." So, now it would be good that you also expose this kind of API. And suddenly you have another, let's say, item that you add to your catalog. So, you can have have this progressive implementation, and you are not required to do everything. You do only those that where you have requesters that add value to to your stuff. So, so to to be specifically answering your question, it's more a progressive implementation and adoption that that uh that makes this happen. It's not like you you come and you say to tomorrow everybody is going to have data data APIs. >> I understand. Basically, for the first question, you answered this is this test are static >> Yes. Exactly. >> And for the second part, so do you actually have the components draw together create a customer You buffer basically each component >> You you you you can buffer each component on its own. So, if you if you if you show this LEGO composition that was here, you can buffer at any level. You can have pre-buffered customers with many properties. We have And then you can have buffered customers with account because buffering account without a customer doesn't make sense. So, you can buffer customer with account. Again, buffering a card maybe makes sense. You can just assign it to customer, maybe not. But you can buffer customer with account plus card. And that therefore you can buffer at any level. And once you have all those once you have already those levels done and everybody is using that, somebody will come and say, "Now we have a test auto end-to-end test that where we need also payment history, you know." So, they will say, "Okay, we have that already. It's buffered. So, let's just, you know, add payment history API." Which is basically payment API. We just send a payment and you we call it twice and then we have a payment history. And that's it. And then you can buffer it. >> Wouldn't that realization of a of a testing So, if I understand correctly, what was the that create an API fetch data the system of test. >> Okay. >> This introduces a a security risk and if I remove it after the test, I would change the system again. >> Yes, the very very very good question whether doing test data APIs is not a security risk and whether it's not creating side effects to to development. Yes, but it's the best solution. Why? Very easy. It's done by the same people that do the systems. That's the main idea. You don't do it centrally. You do you let it do the the guys that maintain the systems. So, they can of course configure that in production this doesn't exist. It's not deployed. And if they can do it in a way that this will insert data in those databases in a consistent way. So, we don't it's not magical solution where there are some AI will analyze the structure of your database and say, I know how to do it. No, you just let the engineers that created the system also create the data. So, you you consolidate the life cycle and the responsibility for the systems and for the data in in one team. There was >> Yes, so this test data portal would have this one single portal for the API to have a lot of dependencies. So, if you're looking at this it's in a complex data organization where you have multiple teams with different services. Every single team would have to adapt their logic into the test data portal once it's changed. Which is a new problem with the issue with the with this type of complex data setup. >> Yeah, so the question is that we are basically forcing all people to adopt to this concept. My my answer is not not necessarily. It's a progressive adoption and it's built for progressive adoption because I know exactly this problem. Like no big enterprise will adopt something immediately everybody at the same time. It's not it's not feasible. So, here if you go back to to this Lego, you know, maybe uh your your problem is that uh this is done in the CRM system and nobody understands the CRM system. So, once you do at least one piece, which is to create a customer in a CRM system, and you expose it as API consumable in a catalog by other people, there will be already a lot of benefits for all for for some people, not for the whole enterprise. And then, maybe some people will come and say, "Well, we could do the same." And the idea is that uh the pattern is self- extensible. So, the idea is that this a gateway here does discovery, you know? So, discovery is a word, but it's just uh a whole concept behind that. So, they don't have to discuss with anybody. They just create the generator. The generator connects to the portal and tells to the portal what it is able to do. So, if you follow those concepts, you have zero almost zero maintenance centrally. So, that's the idea. Progressive adoption, and you do it only where it makes sense, where there's benefit. Because maybe before you had to create Jira tickets or ServiceNow tickets to request test customers. And suddenly, you don't have any those tickets anymore, because people can do it self-service. So, there's a benefit on both sides. It's not just "Okay, we have all to do those generators, and it's going to be heavy for everybody." No, it's a benefit for the consumers, and it's a benefit for the producers. >> Um maybe to add on to that, when also mention that with all these distributed teams that have to build these generators that are then uh acknowledged, you often have the issue that due to certain priorities, >> Yeah. >> very often, testing is obviously the last priority >> Yeah. >> of the teams, which in this case would lead to uh for example, varying quality of test data produced for different teams, which would also then maybe hurt the the integrity >> Yeah. >> and the trust in the central test platform. Cuz in my imagination right now, we tried doing something similar, I can remember teams saying, "Well, team XY didn't finish their testing yet, so why should we even use it?" >> Yeah, yeah, yeah. Yeah, okay, I have uh so again, for online, I'm just going to repeat it. >> [laughter] >> It's it's uh Let there will be push backs and people having different priorities and no not everybody will develop their generators and their scripts, etc. So, I would say um uh uh uh a last resort as a work around and we have done that as well, actually, is what I said with the test automation. If if you are able to automate the UI, which basically usually you are, then you can automate any business process. And if the guys sitting behind the system, they say we don't have time, you can still take some end-to-end team here and tell them, "Okay, you are going to develop a couple of UI automation scripts." Okay, they will run 5 minutes, but if we do it up front, it's not a problem. Then it's ready in microseconds. And you will create scripts for, I don't know, provisioning a new card. Like, I remember last week we were doing some scripts for to automate uh not provisioning, but manufacturing of of of payment card in SAP, because the SAP team said, "Oh, we we have allowed and we have those kind of stuff. We will not do any automation for you in other languages." So, so we we created a script that just goes in SAP, logs in, and clicks on, "Yeah, okay, manufacture this card." And that's it. And this is like uh the the the the missing piece that allowed uh to have a to have a card done by script, you know. So, you're right. Sometimes it it's it's going to be difficult, but you have work arounds. And W- Why is Why Why is this concept something I think it's scalable and maintainable? Because we just follow the business process. We We And this is This is why I say I I hate technology, because if you follow the business process, it's very high likely that it's going to be right even if you don't have the details, yeah. Like if you go and log as a customer in the self-care and you do something, probably the test data that I created should be consistent across all systems. If not, it's good that it's going to break. It's out of time. Thank you very much and I think you were the first and they were actually all were challenging, so you have a free coffee. >> [applause]