Compute-Scaling Methods at Inference Time - Rutvik Acharya, Nitin Agarwal

Duration: 27:01 5054 transcript words EN summary

Video summary

The presentation focuses on compute-scaling methods designed to enhance the reasoning capabilities of large language models (LLMs) specifically during the inference phase, rather than through extensive retraining. The speaker explains that while standard LLMs can provide direct answers based on historical data, they often lack transparency and reliability for complex queries where users need to understand the underlying thought process. To address this, the talk introduces reasoning models that generate step-by-step explanations, making outputs more trustworthy. However, traditional approaches to achieving better reasoning involve costly supervised fine-tuning or reinforcement learning with large datasets. The core argument of the session is that inference-time scaling offers a viable alternative by allowing any existing model—whether small or specialized—to produce high-quality reasoned answers simply by allocating additional computational time and resources during generation. Several specific techniques are explored to achieve this improvement without modifying the base model weights. Chain-of-thought prompting encourages the model to think step-by-step before answering, which significantly improves accuracy on complex tasks like mathematical problems but increases token usage and latency. To mitigate these costs, methods like chain-of-draft summarize reasoning steps into shorter drafts while maintaining similar performance levels. Other advanced strategies include "step-back leap forward," where the model plans a solution path, executes it in stages, backtracks if errors occur, and iterates until success; and test-time preference optimization, which generates multiple responses, scores them against specific criteria, and uses an optimizer to refine the final answer toward the highest quality output. These methods effectively simulate deeper thinking by leveraging compute at inference time rather than training data. The speaker shares practical experiments conducted with small quantized models on diverse tasks such as natural language-to-SQL generation and counterfactual reasoning scenarios involving physics or economics. The results indicate that while techniques like test-time preference optimization offer the highest accuracy, they come with a significant cost due to high token consumption; for instance, running these optimized methods can incur costs comparable to processing millions of tokens. In contrast, lower-cost alternatives like chain-of-draft and step-back leap forward provide nearly equivalent performance improvements at a fraction of the expense. The findings suggest that organizations should carefully weigh their specific needs: if accuracy is paramount regardless of cost or speed in critical domains, heavier inference-time scaling methods are justified; however, for general use cases where budget and latency matter, lighter techniques offer an efficient balance between enhanced reasoning capabilities and operational constraints. In conclusion, the talk emphasizes a strategic approach to adopting these compute-scaling methods by first determining whether a dedicated reasoning model is necessary or if current models can be augmented through inference-time adjustments. The speaker recommends conducting small-scale experiments with three or four different techniques tailored to specific user question patterns before committing to a production strategy. By understanding the trade-offs between token usage, latency, and accuracy, developers can select the most appropriate method for their chatbots or applications without incurring prohibitive training costs. Ultimately, inference-time scaling democratizes advanced reasoning capabilities, allowing even smaller models to perform complex tasks effectively by simply giving them more time and computational power to think through problems step-by-step.

Read the full video transcript

So we are moving on to next talk uh that is on compute scaling methods at inference trying by Nithan Agraal. Please have a round of applause and welcome Nithan. >> Okay. So uh so today we'll be talking about uh compute scaling methods at inference time. So let me introduce myself first. So I am Nitan. I am currently working as principal data scientist at toast. uh and I have worked with companies like Microsoft, Accenture, KPMG before and also worked on few of the startups uh before uh so I have a a very uh deep interest in geneti after obviously the charge GPD uh 3.5 wave came into us but before that also I was into uh some of the NLP uh models and the use cases that I worked upon and I also have a uh co-author in this particular talk but because of some family reasons uh he couldn't join join us but I'll be covering through the whole of the task with us right so uh yeah so let me uh quickly talk about uh uh why we are talking about this right so what happens is when we have these LLMs and then we ask some of the questions initial stages uh we were getting some answers so let's say if I ask uh where should I go for vacationing maybe it gives some suggestions based on the historical data that it has been trained on But uh what happens is it just give us the direct answer right. So the the reason that we are not able to rely on these kind of outputs there are multiple reasons as a user one is uh uh that uh I feel that from which source it is coming so there's no source around it uh what is the thought behind this answer. So those are the different reasons I was not able to maybe rely on some of the answers especially the complex reasoning kind of answers right. So uh so [clears throat] that's where we are uh thinking and maybe we have these reasoning models coming up these days which are actually answering the complex uh questions right so uh so for the reasoning questions uh just give me a second yeah okay so so let let's talk about like what is reasoning in the language models or the large language or even smaller language models. So, uh let's say if I am asking this question, I earn 60,000 per month. How should I spit my money? Right? So, uh a direct answer would be just say 50% needs, 30% uh wants and 20% savings, right? Uh what what we are expecting to give us more details how it is thinking and provide us the details around the answer. So that's the reason we are looking at the reasoning kind of question where it is actually giving us the steps on how it has calculated the different amounts and then we are getting those amounts. So the the answer that we are getting from the reasoning side is more reliable uh looks more trustworthy and it has more details as well right but obviously there are more caveats to it like it takes more cost more time and everything we'll talk about those things as well. So I think this is the reason that we are talking about this particular topic today. So this is from where it has been generated and during one of our use cases what happened is uh we were trying to uh answer some of the complex questions of the customer through our chatbot we are providing all the data but uh these uh uh what do we say the LLMs let's say ch even 4 3.5 were not able to give the right uh answer in very dep because they're not able to do some iterations they're not able to do the reasoning around it so what we tried Some of the method is we just try to iterate with multiple prompts and say can you deep think about it. Uh one of the simple example is chain of thought. We'll talk about that as well. So those things we tried and we felt okay this is some something that is uh uh kind of giving us more details around the answer and also providing the context how it is thinking to the customer. So they are able to rely on those answers more. Right? So, [clears throat] so let's quickly look at uh how the uh journey of a large language model goes. So, I'm just keeping it very very abstract as of now. I don't want to go into the details of uh transformers and everything. So, so we get the lot of data and then we create a foundation model with uh uh basic training and everything. Then we try to pretend a little bit more on uh like a classifier base or personal assistant little bit more uh towards a particular task and then we also fine-tune on a particular data. So all these things we have done after this what we are stage at right now is we are going towards the specializations right we are thinking about uh rack we are thinking about uh specific topic based chatbots uh even multimodal LLMs assistants agents and all and the interest of our area is the reasoning model right now so when we talk about the reasoning models uh how these reasoning models are coming up right so uh there are few methods uh again [clears throat] this is coming from uh very high level extraction. There could be other ways as well but uh one of the ways I'm going from bottom to top is pure supervised finetuning and distillation. So what we are doing is we are providing a lot of fine-tuning data uh and then it is supervisely train the model on the reasoning kind of a uh prompt right and then we do uh a combination of reinforcement learning and supervised fine-tuning where we are providing the a lot of reasoning data to fine-tune and also we are providing a reinforcement learning based feedback so that it can learn better and answer better right uh another method would be the pure reinforcement learning we are not giving any additional data to it but let the reinforcement learning uh provide the feedback and accordingly learn better right uh so all these methods are basically on the training side of it so we need to train our model so let's say I have a model I wanted to make it more reasonable uh libby answer then we need to use maybe some of these ways or a combination of these methods so what we are looking at what if we don't need to even train the model is there any way that I can use at the inference time right so that's what we are trying to focus on this particular uh talk here right so so the idea in simple terms again the inference time scaling computer time scaling looks a little bit uh technical in terms of terms even for me so what I just when I read about it is basically uh giving it more time to think about it so let's say we are a human right so uh if I just give you one minute to think about a particular answer maybe you just answer in a very short time and maybe don't put a lot of thoughts around it but I if I give you the let's say uh few hours for the same question uh you will be more uh thoughtful about it you will be answering in more detail right so the similar concept will be applied here so we are just giving more time uh and uh resources to uh to our LLM to think about a particular topic so maybe we can go to the the last question that we talked about [snorts] so when I say I earn 60,000 per month how should I spit my money so if I just give you one minute to answer this question maybe you answer me directly and then say okay this is how you should spend right but if I say okay take few hours and then let me know how should I do it maybe you give me these details or maybe tell me also about where to invest and those kind of things as well so this is how we are trying to look at the inference time scaling right so what we have seen is u some of the models use this use this inference time technique as well to get a improvement on the existing models like GPD4 was there and 01 is has used some of the techniques to improve on the previous model to reason better right so we have seen on a different areas uh let's say u ML benchmark on the mathematical data also on the chemistry data all these data you see there would be a delta especially in case of mathematics uh physics and even college mathematics if you see so wherever it is more logical it needs more steps to think about it needs some more detailing around how to think about it that's where these kind of technique would be helpful right uh [clears throat] and if and uh again so there's there's a bit of history to this as well so uh initially we were training all the reasoning models from scratch we are not thinking more about inference time of technique but uh in January uh around this year uh we we had deep seek uh event and that's where uh we all started thinking about uh how we can use some kind of a inference layer or maybe post- training methods to improve on the reasoning uh methodologies. So that's where this all started and all these different uh uh model owners started thinking about these and they started uh bringing out a lot of new techniques as well. So these techniques are coming every month, every week. So we can't uh keep a track of everything but I'm trying to cover some of these today uh in this one right. [clears throat] So uh how this uh inference time scaling helps us. So in a in a very simple terms if I uh talk about u it it lets you use any model and then use your inference time computing or scaling to get the better answer. So you're not dependent on a particular model. So if you're using a particular model and then you train or maybe fine-tune on a reasoning kind of a data what happens is you will have that particular model to be used every time you're using a particular case study or uh a particular u [clears throat] case but when we are uh using the inference uh time scaling methods uh you can use any model in the back end even use a some smaller model as well uh and then uh use these techniques to get the better reasoning answers Okay. Uh and then you don't need to uh uh rely on any kind of a training. You you are saving a lot of cost on uh training uh infrastructures and time and also preparing a lot of data. Right? But uh there are few uh areas where I feel so if you are very specific on a particular domain specific reasoning maybe the training could help out better. uh but that is something that we have not experimented so I can't comment more on that but based on a general understanding if you have a a specific use case uh specific reasoning let's say you are into uh medical industry and then you want to reason on a particular uh topic around it maybe we need to train our model on those kind of topics have a good understanding have a good reasoning how they think about it that would be better but uh in general cases uh we have seen this so one of the use case that I can share is uh when we were generating um uh we were building out a chatbot where we were actually generating the SQL queries based on the user question. So that's where we started using uh some of the iterative methods to generate the query. So for example, I'm asking uh in the prompt itself, I'm saying uh these are the different uh schemas, these are the different uh columns and the tables we have and based on the user query, can you convert this into a SQL query? Uh but again, there could be a lot of u issues with the query that it generates. So within the prompt itself, what we did is we asked it to review the prompt, the query that it is generating and then uh correct itself. So again it was not working all the time but it has improved the accuracy from the past uh direct prompt that we had. So over that what we also did is we also created one more layer of iteration. So it made two calls. So in the first call we were actually generating the SQL query and then let it also assess and then another uh prompt for actually evaluating and giving feedback to the the query it has been generated. Is there any issues? there's any hallucinations and then we are redirecting it to the previous prompt. So it is more like a iterative process. Uh so again it was working out for some cases not working for some cases. So then we need to just iterate on the prompt, it on the experimentation to get it better, right? [snorts] Okay. So let's let's look at some of the methods again. Uh this is not a comprehensive list. Um so uh so there's a article by Sebastian Rashka. uh so there has a lot of papers and articles being mentioned here so please refer to that so while sharing this PPD I'll also share those uh references there but I just talked about few of the techniques that I have explored um on my own so one is a chain of thought it is a very straightforward very commonly used u uh technique to get the better reasoning out of it so what we are doing is we are saying it as a chain of thought so let's talk about this example uh we are saying U [clears throat] we are giving this uh prompt to the uh LLM saying that Roger has five tennis balls. He buys two more cans of tennis balls. Each has um each can has three tennis balls. So how many balls does it have now? So this kind of a complex uh query in terms of that we have created. Uh the answer is 11. Then have more questions around it. Right? So uh the output when we get on the standard way uh could be either wrong or it could be just standard straightforward answers 27 or whatever. Right? In the chain of thought prompting what we are uh putting as few more lines in the prompt we are saying uh whenever you are answering the question think step by step and then answer about this. So even if you add this particular line to it uh it is start creating more steps around it. You can also provide the steps to it like we are doing in the mathematics uh we were doing in our exams right. So whenever we get a question we write all the steps and then we give the answer. You can give those kind of detailing as well. U I give you an example. So whenever uh so some of the customers are asking questions about their past revenues and those kind of things and also trying to look at the relationship between their marketing uh what do we say marketing investment versus the revenue. So on those cases what we did is we asked the customer what kind of answer that they're looking for. Based on that we have created the steps and those steps are being put into the prompt like uh map the mapping of the marketing investment with the uh revenue by the month and those kind of details we put in and then we also provided these are the steps that you need to think about and then after that what is happening the the output of that particular uh uh kind of prompts uh becomes more detailed u becomes more inclined with what user is looking at but more importantly we are letting LLM to think more about it, think step by step to it so that we have uh better uh accuracy in terms of answer right but what it introduces is also if you see the example is not here but if you if you just add more details more steps to the prompt we are increasing the number of tokens that we are sending to LL so uh directly or indirectly we are increasing the cost increasing the latency so those kind of issues are there but we get the better accuracy Yeah. So another technique uh would be around uh chain of draft. So so this is very similar to what we have and the chain of thought but in the chain of thought the problem was uh that we are putting a lot of tokens to it. Uh what this technique does is it creates a draft of uh the the steps and then send it out. Instead of giving each and every detail, it gives us draft of the details. So let's see this example. So as a standard prompt, what we will say is uh Jason had 20 lollipops. He gave Tenny some lollipops. Now Jason has 12 lollipops, right? So how many lollipops did Jason give to Denny? So again it's a mapping question. Some of the numbers basic calculation as a human we are able to do it. But as a LLM it is able to do some of the answering but if you see it's a very direct answer, right? So when we say chain of thought what we are doing is with this particular prompt we are also giving a particular uh if you see this area here I'm not sure we can see the pointer but yeah if you go to line number four something it says let's think through this step by step and then uh it just gave some of the steps that it should follow and then ask the next question maybe but when we get the answer from this particular thing it is most detailed in step by step answer is correct we get more details we are happy about it we get more accuracy but eventually we are sending more tokens to it. So what this chain of draft does is it actually uh uh kind of summarizes that steps and put into a smaller way and uh and then it can also run multiple drafts at a time. So that way we are sending less tokens and when we experimented on this we see the accuracy is very similar to what we have chain of thought right. So again um we are not saying it will be always the case but uh based on the experimentation this is what we have seen u [clears throat] okay so uh yeah so another method that we uh explored is the step uh step back to leap forward so basically what we are doing in this particular method is we have a question so a prompt being asked by the user and then uh LLM creates a uh reasoning path. So it actually creates a plan first. It could be multiple plans or it could be a singular plan to do this. And then based on uh the plan it executes those steps one by one and after let's say n steps in between those u uh total steps it assesses that is the reasoning going fine is it going towards the right answer and then if it is not working out properly uh it does the backtracking where it went wrong and then start reiterating over that. So it it is more like a reiterative process uh happening within itself. So let's take this example where we are targeting a number to be 62 and it could be a combination of 2 39 45 or 10. So out of these numbers we need to actually reach to 62. So there could be different ways that we can do. So all these different uh methods are being created by u as a plan by the LLM and then we are generating different steps and after uh a few steps in between uh start backtracking and checking if the method is going fine is it it going towards the final goal of 62 or not and if not it changes its plan try to go to a different plan and then goes toward that. So those are the uh methods [snorts] and eventually it'll have a plan where we'll reach on the final score. Right. Right. So uh in uh so this is another technique called uh test time preference optimization. Uh in this what we are doing is we are actually um using multiple prompts at a multiple stages to go a little bit more deeper into what is happening with a prompt. Right? So uh let's say we are asking a particular question like what are some of the best universities for uh studying robotics right so what we do is first of all we generate multiple responses for for this particular question right so let's say we make a call to our LLM we let it generate four different responses now for each responses we score it right for again scoring if you see we have a prompt to do it so we are saying uh these are the different variations you need to evaluate based on the uh initial shell prompt and then give the scoring right. So again you can provide the range of the scoring. So all these details you can do but eventually we are scoring all of this and then based on that we are calculating the loss uh and everything is happening basically on the prompt. So it is passing from one prompt to another prompt and another prompt to another prompt and that's how it is kind of chaining the the different outputs. So once we have the the uh scoring done then we pass it to our uh another prompt which kind of creates more like a loss kind of a setup where it says okay uh you are this much closer to your final answer and then uh it goes back to the initial prompt. So we have different stages that we are trying to do right. So first stage is to create multiple responses. Once those are created we evaluate each and every responsive create a score. Once the score is created uh we try to u uh calculate uh some gradient around it. Try to understand how it is uh closer to the uh final answer. Which one is maybe let's say the the lowest uh uh loss one. And then we let optimizer also to see what kind of changes we can do in our answers to get more closer to the the final output. So these are the different things that are happening. So if you see the sample prompts are also there. So all these different details are uh done in a different LLM. So we are making multiple calls here. So we are increasing the tokens, we are increasing the LLM calls. So obviously we are overshooting on a lot of cost. But this actually has proved out to a very accurate uh technique with some of the cases that we have tried. Okay. So uh again there are so many methods uh to this uh we have uh I have tried exploring all these u uh u techniques. So some of the ones that I have colored it out is something that we have tried. So the test time performance operations as we talked about uh if you see the the average tokens it has used is around uh 1 1200 plus. So I've what I have done is I've taken around I'll just explain the experiment first maybe. So what I've done is I've just taken a very simple uh model even quantized model a llama 3.2 to 1 billion model and instruct model. This is not a reasoning model and then I've taken even the quantized model with a 4 bit because we are very uh compute poor uh uh people right now. So so I just use whatever I can do on Google collab and then uh based on this smaller model uh I have created around 50 uh reasoning prompts. So the prompts are like and some examples are like uh uh counterfactual kind of reasoning like if G's uh earth's gravity were certainly reduced to half what are the major effects this would have on the human daily life in the first week. So this kind of questions that I have put in uh again some of the questions I have put in also around NL2SQL where we are giving a simple language to the user uh to the LLM and it should generate the SQL query around it. So similarly different areas of prompts around four five areas of the prompt I have taken and then try to uh score different uh techniques especially these four on those uh prompts and then try to see how uh they are working out and if you see I have also provided uh answer to it. So it becomes our golden data set to uh even evaluate how those techniques are working out and for evaluation I'm using bird score and blue score for now. So based on this small experiment that I have done uh what we have seen is the test time preference optimization technique uh works out pretty good in terms of accuracy it is giving uh almost the highest score across the board. uh but again this is just based on this simple sample. What I would suggest is whenever if you're trying to explore these techniques maybe few or four or whatever numbers you want to try out uh try on your data what you wanted to implement on then then see what kind of scoring it can come out but you can use this kind of a understanding plus experimentation setup to uh perform that experiment right so if you see the tokens count is very high again we are spending uh so based on this average tokens what I've done is I've calculated if we use the same average tokens uh and then how much we'll be spending for 1 million tokens. Uh so that's around $37 we are spending on it for this particular technique. But the accuracy is pretty high. So if you have any critical kind of uh uh cases uh where the accuracy is more important than the the timing and the cost uh please use such techniques. There are one more uh below if you see uh trading compute optimization. So all these different techniques you can explore and then the the ones that we have also explored is chain of draft, step back, leap forward and chain of thought. All these are low cost. Uh chain of thought is still little higher in the cost because we have a longer text that we are writing. But uh if you see they are performing very similar in if you compare the chain of thought and chain of draft, right? So the the bird score is uh very very close. So chain of draft is working out almost equal or better with the prompts that we have tried. uh for these techniques uh and the cost is also low but what I would suggest is try it out in your data and then uh assess it better because this is kind of very tricky. There could be a possibility that the draft that you have created of the steps might not be uh clear enough for the LM and it can hallucinate or maybe give some wrong answers. Right? And even the step back le forward is a very simple technique. it just uh uh takes few uh tokens more than the chain of draft and uh it gives you a very good accuracy as well. So uh [snorts] so so uh overall recommendation what I would say uh before I just uh switch off for today uh is basically whenever you are trying out these kind of technique first of all you understand that uh do you need do you need a reasoning model itself or you can do some kind of a inference time computing like this to get your answers better. Second is assess like if you're creating a chatbot what kind of questions user might ask and what kind of questions might need this kind of a technique because this will increase your cost eventually right and uh third thing is then you can create some kind of a smaller experiment to at least take out three or four techniques and you can also explore more there are so many techniques available u this is what I could explore in some time uh last six eight months right so you can just create a small experiment execute that on your kind of prompts and then see which one is working out better and then create a final trick to it. Right? So, uh this is what I wanted to cover. Thank you so much for listening. Uh so, in case uh you want to connect u can just I'll be available on LinkedIn as well. Thank you.