EmberConf 2024 - Perfect CI Pipeline in an Ember Project with Tomek Nieżurawski

Duration: 11:46 1956 transcript words EN summary

Video summary

The video focuses on optimizing Continuous Integration (CI) pipelines for large Ember.js projects to significantly reduce build times. The speaker, Tomek Nieżurawski, begins by highlighting a common issue where standard CI setups can take over an hour to complete due to resource constraints and inefficient test execution. He demonstrates that while modern frameworks offer built-in optimizations, they often fail under the load of large-scale projects unless specific strategies are applied. The core argument presented is that achieving a "perfect" pipeline requires a combination of hardware scaling, intelligent test distribution, and dependency management rather than relying solely on raw computing power. A significant portion of the talk details how to balance the computational load across multiple browsers using tools like Ember Exam. Initially, running tests in parallel on a single machine yields modest improvements, but utilizing multiple containers with an "army of browsers" drastically cuts build times from over 40 minutes to under 10 minutes. The speaker emphasizes that while increasing container size and count solves resource bottlenecks, it is not the only solution; effective load balancing ensures that no single browser becomes a bottleneck while others sit idle. This approach allows teams to save time for other tasks, such as reviewing code or taking breaks, by making the feedback loop much faster. Further optimization strategies involve managing dependencies and caching build artifacts to avoid redundant work. The speaker advocates switching from npm to pnpm, which reduces dependency installation times by roughly 28% by using a more efficient file system structure. Additionally, he explores caching the built test application across containers to prevent rebuilding it in every step, though he notes that excessive abstraction can sometimes introduce overhead. By stripping away unnecessary abstractions and directly copying files where needed, the speaker achieves an impressive 89% reduction in total build time compared to the initial baseline, proving that a mix of tooling choices and architectural tweaks is essential for high-performance CI pipelines. In conclusion, the presentation serves as a practical guide to transforming a sluggish CI process into a highly efficient one without necessarily requiring infinite budget increases. The speaker acknowledges that some problems, like perfectly balancing test execution across files of varying complexity, remain challenging and may require custom tracking solutions beyond standard tools. Ultimately, the talk encourages developers to experiment with different configurations, embrace pnpm for faster installs, and carefully manage caching strategies to ensure their CI pipelines are both fast and cost-effective, turning a potential hour-long wait into a matter of minutes.

Read the full video transcript

thank you folks great to be here there's no technical conference uh without technical problems so we have that checked ER we don't have much time so we are going to focus only on making the pipeline fast there's more about uh pipelines CI pipelines to think about but yeah and I will have to fly through the slides so maybe that's good maybe you see a pattern here uh fast presentation about making things f first of all I'm tomc I'm a principal engineer at Forest I came from Poland here that was a long flight but yeah so if we start a new project uh today we already have some hints how we should organize things we have linting we have testing we have a GitHub preconfigured you can also use Travis I believe and we have linting and testing testing that's that's great start and there are some optimizations inside already I didn't know about these commments and MCI this is like kind of optimized um install for CIS and um yeah not sure if this is performance optimization but you know we have some hints in the uh in this configuration already but let's switch to a big project and this is actually a project of forest and will de optimize things so I did the same on Circle C the same thing and it took 42 minutes to build the whole project you know tested all that stuff uh that that's pretty long I would say the same setup npmc around 2,800 tests to run and uh and I'm already cheating here a bit because I use a bigger container that we usually do so uh for that container that we use large one it doesn't even build so that's I guess that's not great it's faster but it fails so um yeah so we need more RAM we need more CPUs and if we are deploy to that on the pr we build things in the parallel on the main Master we buil you know we require test and linking to to of course to pass before we deploy so this is if you sum up the numbers is one 1 hour and 30 minutes something like that to deploy your stuff if everything goes well and you didn't get any feedback on a PR so yeah that's a really hot hot fix I would say if you are shipping uh but I will what I'm trying to convince you to is that we have great ecosystem for for testing as it's it's great that's it was great to hear from Yuda that about testing in a browser I always thought that we should test in a browser and it's not that common among Frameworks and yeah we have some tools some Ember exam is going to help us if we look at what was happening with our container we can see there was a spike on CPU on RAM probably this is where the large container failed uh but this machine is lazy you can see it doesn't do a lot so with just a simple command with using Ember exam we can split um the load into free browsers the way I imagine it is the number the magic number is n minus one n is CPUs so one Le for the orchestration of um of browsers and three for for these three browsers so this is more effective and look at that we are 40 uh 54% faster just with that with that simple trick you know uh and we didn't we didn't pay a penny for that that that was that was easy and the load is still not optimal if you run the same command uh locally and put dash dash server you will see that the tests are not balanced different browsers have um different amount of tests to run and this is happening mostly because of splitting tests on based on fil files so we can balance the load again Ember exam just a different command is going to balance the load the tests on on a fly in a fly so now it's a little bit weird because it shows that there's one test to run but actually there's one test in the queue let's say one one is to test but we serve them to to different browsers and that again we didn't spend any money and we are 10% faster than the previous step okay that's that's good uh and we now we will spend some money unfortunately we have to but we'll use parallelizations we'll use more containers but we'll save money on the how big the container is so now we are going to use the large one again some code easy and now we have Army of browsers running our code in the browsers you know that that's great that's that's where they should it should happen so six containers three browsers 18 um browsers that run our test and now we are well under 10 minutes which is great right um You can spend the rest of the time on Tik Tok or whatever the the time you just uh saved for the company and we are 62% faster than the previous step right and we really use these containers if we look in inside a lot of spikes to 100% that's great because we pay for that we we use that and money solves a lot of problems right uh so you can say you know just use bigger containers use them more you will have even better results and I agree but that's not the point of this talk is that to use it uh in a good way I would say effectively so here we have 42 browsers to run the tests and we are under 6 minutes 50 seconds faster 12% faster than previously but we kind of pay twice the price so you have to ask your CTO if that's possible um let's say it's not not possible so we spend of course uh more money we don't use all that power let's again save a little bit of money we can look into the testing and we install dependencies with every you know every container uh and we'll introduce we are programmers so we love abstraction H we'll introduce checkout code it's going to install dependencies we'll save that to the cash we can even use smaller container and things are slower but there is potential you know we are reusing some code we love reusing stuff so we are a little bit slower than the previous step but still kind of doing okay right and we are just paying penalty kind of for restoring the cash for saving the cash for having extra step but if installing dependencies is slow can we make it faster and I think we can we just have to switch to pnpm uh and I hope you already did that but uh if not you will see why it you should so in our project this is still everything is on the real project you know uh so I had to run the test million of times uh but just with another simple trick it's not that hard to use pnpm we are 28% faster than the previous uh step and look at the checkout you know 53 seconds against 3 minutes that's great we didn't spend money and not every road is going to give you results and that's okay because uh I thought I will use even newer pnpm I had to upgrade note for that uh there were some optimizations there but it didn't didn't matter after all uh so we are better at building and here when running tests something we don't see is that every container has uh to build the test application and maybe if we're that smart we can learn about you know building um the app in the test environment set it somewhere sa in the cach and then you see that puff this uh we will use that and we are faster again we are under six minutes uh folks H so we are 10% faster than the previous step and of course I'm I prepared like you know the story here so for us at the time uh it wasn't that linear uh for us it g gave us 37% uh bump you know so that was great you should definitely h test uh test that approach but I know I'm flying very fast through these slides so maybe you didn't ask yourself okay you abstracted stuff maybe you maybe if you know every abstractions is costly in a way most of the times that's that's fine you know we don't have to write assembler code H but is abstracting mean means being slower kind of it is I just removed that um that code uh that checkout code uh step and I just copied the stuff there so so I install dependencies again in every step and now yeah it's 16% faster than the previous step so we are 89% faster that at the beginning this is I think quite impressive and there is still room for improvements if we again look at this chart um we can notice that when the tests container are running tests there is some idle time we can see this is not balanced kind of why is this happening didn't we just on the I don't know fifth slide balanced the load we did but on a container level between browsers but not between containers so yeah why is this happening that there is some potential maybe 45 uh seconds so the potential is there but what is happening there is disproportion be uh between test files you can have a test file that is testing your 404 page how many test you will fit there not many right and there can be a test that is you know testing the whole world actually so other folks in the um in the community also had this issue and there was a part from intercom he was he had a presentation uh and he touched on that thing as well in intercom they introduced a rule Max F length Max file length length rule which makes you know guards that your test files will be kind of of the same size so they will be balanced and and you know watch that on on YouTube they had amazing results although I think this is probably not the way to do it like uh we need to start tracking um f executions uh so we we track how fast the tests are in general but we don't know how much we spend in the uh in the files and there's a bunch of things in the play and I really wanted to give you solution but I didn't make it before the conference uh yeah that was that was difficult but I'll work on that and if I have anything to update you on subscribe to Ember Europe um YouTube channel the times won't be good for you to watch live so if I manage to figure out this I will um you know let you know in that uh on that YouTube channel if they accept me and that's it we just improved by 89% that's by the way faster uh it was happening 42 minutes or something like that and the presentation was 16 minutes so you know we made it faster than the first uh first build and we have optimized that thank you very much