Closing the Observability Gap: Building a Kubeflow Plugin for Headlamp - Alok Dangre, NIT Rourkela

Hello everyone. I'm Alok Adhangale. So, I'm here presenting about the KubeFlow plugin, which is made in a headlamp. This is to provide the observability lead lab of for the KubeFlow. For the operators and other. This was the part of Malefix mentorship, by the way. And uh So, the problem is that when we are using KubeFlow, we face some issues and we have to constantly switch between the Kube config other. So, avoiding this, the solution we have got is to make a headlamp plugin. So, you can directly visit and check all the details in very easy user-friendly UI. And uh This This is the complementation, not a duplicate. This This was the project scope. I did not reveal the KubeFlow's authoring workflows. I'm focused on what the Kubernetes API already knows and what operator needs during the diagnosis. And that's what it takes. The plugin grew in a The bug plugins were in a slide like a slide in a side bar. So, I have covered the five components and 16 custom CRDs and 39 detail views and list views out for the whole plugin. So, there is a plugin SDK for headlamp and we directly detect the APIs from your system. When you should You only need to download the headlamp app and then install the plugin. We directly detect all the CRDs present and give you the UI friendly user-friendly UI to navigate you through Kubernetes and Kubeflow APIs to get through all the components and CRDs. So, this sidebar guides you to all the CRDs and the components. A team may run only Kubeflow notebooks and Katib or only Spark. The plugin probes five different API families and parallel exposes the sections that make a sense. So, you can get to the section page and directly navigate it to it. Also, if you doesn't have any some uh CRDs present, so it will directly detect you and let you know about it. So, this is the notebooks overview page presentation. And you can use it for examples like if there is a pending batch, but the pending is not enough. So, why it is pending? So, you will get the uh reasons about it like if the image pulled back is off or crash loopback is off or OOM killed or an unscheduled reason. The overview page helps you and help you to see the CPU aggregates and memory GPU names for distributions and other imaging use. So, you can directly resolve your problem. Here is the detailed documents list of the Kubernetes servers. I mean, the notebook servers present and the notebook detailed page. So, whenever a researcher see that their notebook server never started, use the list to locate the open and locate and open the detailed view page to trace the notebook CR and pods for failing containers. And you can debug it accordingly. So, now here is the resource details of the notebooks. So, if the GPU is the GPU of the notebook is unscheduled, read the condition messages and then compare the resources request selectors and tolerations. The example explanation is that the GPU is contained and this notebook is missing in the required tolerations. You can debug it. So, now this is the profile CRD, which will help you if a teammate cannot see the shared name space. Then start the profile page and verify the owner and contributor list and then move the Kubernetes R back only if the expected contributor is present. So, to debug it if the if by not getting and if you want to edit it. is the uh which have will help you to check notebook if cannot reach the S3, use the default or default view to verify the selector and the secret if it's unmounted volumes. So, there is this issue and admin admin admission configuration problem and application code. So, you can select this and get the uh fix. Now, this is the plug pipeline's component. And this is the KFP CRDs in the component. It has the overview page here to see the pipelines and pipeline version already running number of it. So, you can check what it is. How many of it's running. So, if multiple teams report this failed pipelines, so you can see the control center like this overview page and check whether which one is failing and why investigate the failed workflow. This This is the pipeline version pipelines detail page screenshot and so you can see the annotations and all the details like SDK versions to debug for inspect the pipeline definition parameters and artifacts root file. This separates the definition regression from an infrastructure or credential failure. The This is the pipeline version CRD to get the and this is the diff view which we have made to let you know the differences in the previous versions. So, if the version 8 fails and and version 7 worked, open the comparison and find the change in the missed like a parameter resource limit and connect the change that change to the failed run. This results in a spotted story for the CRD. This is the run CRD details page. So, if the nightly running ex- training running exceeded the limit the SLA, then a start and finish duration check the start and finish duration and status from here. And then move the run CRD to the Kubernetes workload responsible for the delay. They are more sleep details available in this page. The screenshot is like limited here. And this is the recurring CRD Uh the pipeline component. So whenever the schedule is reported did not run, check this enabled state and gone first, then inspect whether the schedule created a run. This tells you whether this failure is schedule or execution. You can actually check all the details and back links for that specific pipeline version which is linked with this recurring CRD. This is the pipeline experiment CRD. It is it This is the detail page of it. This is used You can use it when the team uses similar model and run things. Start from the experiment resource to preserve the intended group name to pick name space and version context before you compare the results. So it eases the task for you. Now we are in the Katib component. This is the dashboard or the control panel for the Katib. Uh it shows all the experiment trials and suggestions running and details about it. The latest running and the health. So if the overnight GPU uses limits expires, use the Katib overview to find active experiments and their trial budgets, then distinguish the health source activity from a controller and worker problem. And this is the Katib experiment CRD. This is the detail page for it. But so whenever all trials completed but accuracy did not improve, inspect this objective metric target parameter ranges and the best trials are narrow or incorrect space is often visible before reading the training log. So, you can debug it much faster. This is the trial CRD details page. This shows one trial reports if one trial reports in a not fine or compare its assignment parameters with the successful trials. Then open the worker order logs. We have all the uh action buttons to see the current logs and the view of the JSON for it. Uh this is the suggestion CRD details page uh showing all necessary things. If the experiment stops creating trials, inspect the suggestion lifecycle and requests count, then for go to the suggestion control for which is the exhausted search from a broken control loop. So, here this is the Spark component uh dashboard page. This list out the uh number of Spark applications and scheduled jobs running. So, if a suggestion uh if a alert suggestion reports execution time, use this control panel to find the affected namespace and application. Then open its topology rather than searching for the label. This is the Spark application CRD. The this is the list view to directly list all the running ETL applications, means use the list to and state duration. Identify the outlier and keep it in space and attempt context as you drill in. This is the details page for this application CRD. And this this is also in that page. So, if the driver is unhealthy, but the job is slow, the topology shows the executor restarts and node placement. Explain how repeated restarts on one pressure node point to an executor resource or schedule drop. Uh this is a scheduled application CRD. And this is the details page of This is We also have a diff view button action button for this. So, if the two details and aggregation jobs overlap, inspect the schedule concurrency policy and generated application history. This exposes a schedule policy issue without treating the child application as unrelated. This is the training component with the number number of training jobs running it and the namespace run times cluster. It also shows the recent training jobs and run time coverages. So, if the CPU queue is growing, use the control panel to count the pending jobs and identify the namespace and run time possible before the investigation investigating schedule capacity. This is the training job CRD list view. If older jobs using a different run time succeeded, then the run time change becomes the leading queue. So, you can directly list here uh according to the condition and go to that specific rather than going to some commands. This is the training jobs uh CRD's detail view page. To see all its details. This uh So, if a full worker job remains in initializing the follow this training job details view page to see the job set and job worker. It's a good pod condition shows the resource request that prevents scheduling. Uh this is the training run time and cluster admin run time details [clears throat] view page. This is for training run time. And uh this is called cluster training run time details page. So, even update upgraded run time breaks all the new training jobs. Compare this run time templates and batch preview. This is uh here we have preview buttons for to see the preview. Highlighted and changed images you can check command environment variables for default resource request. Um So, this is uh so So, the here it is all about like you can always inspect any CRD's by going its overview list and details page. Check the condition and state. What containers running? This viewers are back checks and grants. And you we also have a map integration. So, you can directly check uh the map view for your Kubeflow resources running. So, you can spot the error anytime. Follow the topology and act on context. >> [clears throat] >> So, so you can go and check it out the headlamp plugin. If you have any issues, you can report it there and you can make an issue and uh let me know if anything's. I hope you like this. Uh this all example to guide you so it can help you for the specific uh CRD detail space.