GenAI+Law Research at MIT: Robert Mahari

hello this is daa Greenwood and I realized as I was editing this video for law. mit.edu that a little preface would be in order to let you know number one fundamentally this is a spotlight on not just the computational law research that's happening now um in the research units that law. mit.edu is part of namely in the media lab human dynamics group and connection science which all of which are somewhat interdisciplinary and um and and cross- departmental themselves but but more specifically in that area of generative AI for law so not the law of generative AI but how it is the generative AI is now being used for Law and for legal processes the other thing is as I'm editing this I'm realizing that um my attempt to uh produce video like this from my local park was only nominally successful and so I apologize in advance for uh some of the audio lag and the um and the poor bandwidth on my side but hopefully that's okay because the main attraction is our star graduate student Robert Mahari so with that said enjoy the show and now a segment that uh you've all been waiting for and that's been long promised a pick a peak rather behind the curtain and the research in into computational law that's happening at MIT and um here in the media lab and the um MIT connection science uh related research group where um where I serve and where law. mit.edu is situated um and I I want to reintroduce you all to my friend and colleague and our star grad student who's um in the midst of a PhD now at MIT and um after a very successful sojourn at Harvard Law School where he picked up a JD um and be before which he was at MIT so I think we we can claim original um that a Providence of Robert Mahari and Academia he he got two degrees one in uh engineering ing chemical engineering at MIT as a youth and uh he's back baby and he's diving with both feet and his whole body into computational law and I couldn't be more delighted than to than to call you a collaborator and so I just wanted to share in in this video uh as I said a bit of a a bit of a look into what is happening in the research in this area that's a little bit different from what you'll be seeing in what what we do in the MIT computational law report and the sort of stuff that we Spotlight in idea flow and in our workshops and so forth this is more what I consider like our day in day out uh at MIT which is primarily a research institution so with that um Robert Mahari thank you so much for taking time out of your incredibly busy days um to to to join again and to and to share with people what it is that you've been working on so uh take it away perfect thank you so much daza uh for the generous introduction as always um and you deserve a lot of credit too uh you are the first person who introduced me to the whole idea of computational law um so uh thanks for that and uh it's been it's been a fun ride so um what I thought I would do today is is just share you know some highlights we'll be moving quite quickly through a couple of research projects just to kind of give people a sense of what the kind of research questions are what the open problems are um and really uh to convey I hope the the kind of breadth that computational law research um offers um and maybe people will be excited and want to um collaborate so um with that uh I will start maybe um by talking a little bit about um legal research um as in the research you do when you try uh to draft a new case and how large language models can um help us with that and so as as a very quick review and reminder we're in a common law system right so uh that is a system that's judg mate that really is uh built on citations to precedent and um it's an exciting opportunity for us to leverage um large language models to uh predict precedent um this incidentally is um one of I think my favorite cases uh from law school that involved um a uh conference that went completely off the rails in a hotel um people were bringing all sorts of animals into the hotel they were firing guns and ultimately a passer by was hit and uh the Minnesota Supreme Court had to figure out whether the hotel was liable and they ended up citing um and quoting a case from New Jersey where something similar had happened and um the conclusion was essentially that well if the hotel was aware of the danger uh that its guests posed then it had a duty to to protect um the innocent passerbys um and so this kind of reasoning right like based on precedent is ubiquities in in common law jurisdictions and um we wanted to see whether we could build a system that retrieves uh past cases um and specifically these kinds of quotations from past cases and then once the retrieval has been done that generates um an argument to be made and so um the task that we're going to focus on is given an legal argument that you'd like to make um can we predict uh passages of relevant precedent um we treat this as a classification problem so essentially we say look um let's just try to predict given a essentially a list of all the possible precedent you could site let's predict um the precedent that's most relevant um to your specific argument um we construct training data by essentially mining published judicial opinions um and finding uh passages of quoted precedent and then looking at the context that surrounds that precedent when it's used and trying to predict the the passage of precedent um given the context that surrounds it um we're able to generate a tremendous amount of training data I'll just pause to underscore like you know legal data is so rich there's quite a lot of it um and so you can create these huge data sets that allow us to do some really interesting things um like uh this uh project on on passage retrieval um anyway so we able to create this big training data set um and then train different models um one kind of more advanced Transformer based model one simpler one um and the the key takeaway is um especially if we look at the kind of more advanc model um the correct passage um from you know thousands and thousands of potential options is among the top 10 or the top 20 um retrieved examples over 90% like 96 99% of the time so really impressive results um large language models appear to be quite good at this um I also flaged like a much simpler um feed feed forward neural network you know this is kind of old school machine learning performs quite well at this task um and then once we have uh the um passages retrieved um which we built this little uh demo to show people how this looks so here we've given an input argument to the Eng and it's retrieving these passages um that you can see here once these passages are retrieved well then we can give them to a um model like chat PT we can essentially cheat uh treat chat GPT like a grammar engine right and we can ask it to um Write a brief based on some uh passages of precent that we provide right that that addresses a lot of the hallucination issues and then it will go ahead and just put it all together and uh Write a brief that um in my opinion is actually quite passive POS is kind of an initial starting point for an attorney um we we've kind of analyzed this quite a lot and it turns out that by providing passages uh initially you you really like address a lot of the hallucination issues but you also get stylistically kind of correct briefs um so something kind of interesting and further to explore um so just kind of concluding this little section here um AI large language models seem to be really good at finding precedent um simple models work quite well and uh we can then combine kind of retrieval tools with large language models um to uh generate some briefs so I'm going to pause briefly here um and uh give Daz an opportunity to to ask some questions oh thank you so much um and you know Auto law is always been one of my favorite projects of yours um and so one question I have for you you you sort of touched on it but could you be a little more explicit about um how the project has itself evolved with the evolution of generative AI so when you started this it was pre chat GPT it was pre GPT 3.5 and some of the capabilities that that now has now we have gp4 um so I recall you're using Bert um initially um to start to identify and classify um you know some of the like the key um kind of Holdings and and parts of the opinions in cases what what's changed in this project with the Advent of this modern generation of um of AI and and and how yeah so um I I think I and we as a community have been quite lucky um to have been around like really at this cusp right where like AI was good enough to do interesting things four or five years ago now it's like really good um and it can do all these new things so um when we first started this it was really focused on retrieving the precedent and the idea was like look let's just focus on doing what a legal research platform does already right like helping you find the precedent you're looking for um we had thought and like considered you know well maybe we could do like some sort of word plugin where you know you're typing along and you press Tab and uh the plugin just suggests what you should site at that um place but what large language models can do is the generation piece right they can generate really high quality text now they they're not necessarily like natively good at uh finding uh the legal precedent but that's the research that we've been doing for years so we are good at that and combining the two um has led to some really exciting things right where we can take all of the infrastructure we've built for uh legal research and then layer on top of that the gener of AI kind of language model um to to produce text um and I think this uh I think we're we're at the start of something it'll be exciting to see how lawyers actually leverage this um but the possibilities I think are significantly expanded now um in good ways and bad that that hopefully y uh people can kind of think about um some of the risks here as well indeed yeah this is foundational um okay so I know we've got a few interesting projects and we're barely scratching the surface of your research lately but let's get to the next one okay um so the next thing I wanted to talk about is um kind of more General um like the opportunities that we have to use uh machine learning large language models to extract data from legal documents and so for for anyone who's like interacted with a legal do in in the workshop you'll know like documents are long uh they are complex you have to go to law school to like really understand what's going on um and so the question is like well can machine learning help us as researchers and as individuals by summarizing by finding questions to specific answers or by pulling out specific information and um the answer is yes uh there there are lots of kind of applications that you can um think of the challenge was before large language models a little bit to your last question daza um what you had to do is you had to kind of collect and label a bunch of information train a model that knew nothing about law um and then th those models would perform well but the precondition was like well you needed to have the training data and that that was expensive to to come by because legal documents are long and complex and you need expertise to understand them now we can just ask right and and machine learning people will call this zero shot learning um you can give a document to a large language model and you can ask it hey can you uh tell me how much the attorney made in this case um and as long as that information is contained in the document you've got a pretty good chance of being able to to get the right answer and then you can do a little bit of kind of prompt engineering that I'm sure daza is going to tell you about um to to get even better results but um this really unlocks a lot so um I'll give you an example of of a research project that was kind of pre pre-chat GPT days where we were interested in um understanding uh the attorney's fees in class action lawsuits and um the cool thing about class actions is in as side is that the final settlement is published and has to be proved by the judge so usually we don't know how much attorneys earned but class actions give us this kind of unique Insight um where there's a Judicial opinion that tells us the problem is that like it's not always super clear um and so you can see from this little extract um like there are all sorts of numbers floating around um it's not entirely clear like what is the correct answer um by the way the correct answer is um 140,000 um which is the last sentence where the judge says I will grant um the petition but there are all these other kind of numbers floating around and there's this load star and like what is that um so what we did um is we just said look you know this is the chat GPT playground um based on the judicial opinion which is long um identify uh the final fee and costs awarded to the attorney um and we can press play and uh lo and behold this will work um and uh the video is a little bit long so I'm going to fast forward um and it tells us dutifully that the final fees uh fee and costs awarded were um 14,622 which is the right number um so that's exciting right and that means that what we can do now is we can use again plugin large language models where we had problems in the past which is like this information extraction step um but then we can go on to do kind of regular um you know regressions or analyses or whatever we want to do um and and this really locks lots of possibilities it comes at a cost like 12 cents per opinion um but usually for for a lot of these applications 12 uh yeah uh this is a trivial amount of money um you know maybe you want to label a hundred or a thousand opinions that's that's doable and if the cost is an issue if it's prohibitive what you can do is you can label a few with chat GPT and then you can train a cheaper model um kind of fine-tune a cheaper model to uh do the labeling for you um and that works uh quite well as well um and then so the takeaway ways are you know we can go from unstructured raw legal data to structured legal data um we can do this kind of quickly and cheaply and now we can do all sorts of interesting U quantitative analysis um you know extracting data different insights uh pulling information from different documents um understanding kind of historical Evolution all sort sort of interesting applications so I hope that this gets people excited about um some of the things you can do and uh I'll hand it back to you daa um thank you so much and this another great example of a project that started with the prior generation of technology and where you really just blew the ceiling off of what was possible um with with the current Generation Um something that I noticed that you men so one thing that I I'll highlight here is I think everybody that knows lot. mit.edu knows we love structured legal data and so I feel like this is just doing such important work to to get these natural language kind of very narrative fuzzy um you know hardto parse um you know legal documents and opinions into something that we can then use as the starting point for proper analytics and can turn that into actionable valuable um knowledge um but there's another aspect of it that you mentioned as well which which I'll just um add as a kind of a segue um you said and now we can use it as training data and I feel like if there's a theme of 2024 it's going to be L's a closer look at the data um underlying these models and and what what having the right kind of data makes possible um and and I think that might be a good segue to your very next project sure um what I'll do is I'll give you a quick um example of how we've used uh this kind of information extraction uh in like a concrete research project um and then I'll move to uh the project that you're hinting at um about uh data provenance so um let's start with um this project project where we were interested in judicial impartiality um and this is you know this like big important pillar of justice that goes back uh to the Book of Leviticus and and the Magna Carta it's hard to study impartiality um for lots of reasons but one of them is kind of data access and so um what we did is we were able to match a couple of databases together um and then uh what we did is we we used the overlap between the structured database and the unstructured database um essentially is training data uh and then we said okay based on this overlap right based on the cases that we're able to annotate using the structure data can we then annotate the rest of the data um and the kinds of things we were after was how was the case decided uh what kind of case was it was it a you know civil rights case a contract's case um and then some other kind of like key uh information about the judge uh that was involved in the case um we were able to create two kind of big data sets one on case data uh did the plaintiff or the defendant lose who was the judge things like that and then on the judges themselves their workload their experience party affiliation um and then we we started being able to do analysis on on like a scale that people haven't been able to do before um about predicting judgments uh using factors that are unrelated to the case details right and uh I won't go into kind of uh the the methodologies too much but the key thing is that this gives us an insight into judicial reasoning and decision-making and impartiality um and uh the the kind of uh bottom line is uh you can see this red line at 50 um% that's uh the accuracy you would expect if you were just guessing if the judge decides for against the plaintiff um and this model trained on the the factors you see underneath um kind of extraneous it appears to to the case uh at hand related of course to Judicial philosophy but not to the specific case does quite well and and for lots of cases we can get you know uh accuracy is approaching like 65 70% um so that's quite exciting um so just kind of an examp example of how you might be able to leverage uh this kind of uh data uh but now let me uh tell you about data provenance which is like this is going to feel a little bit like a pivot um but I hope that uh people will see kind of how this is also a kind of computational law um so let's see um let me close this uh and minimize this and tell you about data provenance so um one kind of version of uh computational is to say can we improve the practice of Law and this is very much like you know in so far as there's like an inter intersection between law and Technology there's kind of the the law of Technology lots of people are interested in regulating Ai and privacy and those are important things then there's the technology of law uh which is kind of like how does technology improve the legal profession and there I would argue fewer people working on that uh and I think there's like a lot of green uh space that people can um can tackle but even in this kind of um law of tech technology area there are some kind of blind spots some like areas especially when you get into like the more technical side where I will say like a computational lawyer can really add a lot of value um and this is I think a good example of this so um you might be aware that um you know generative AI like chat GPT and other models are trained on huge amounts of data not all of that data is uh created equally so you have some data that's kind of scraped from the web you know unstructured large scrapes of the web common craw is a good example there are like scrapes of Wikipedia that's usually what people think of when they talk about training data but there's also a bunch of uh data that was created really kind of in a custom way to train AI models to be good at certain things people will call this fine-tuning data instruction tuning data alignment data um and the recent advances uh in gener of AI maybe not that recent anymore but like the advances of the last year or so in large part uh have been um have been at least catalyzed by um these kind of custommade data sets so we've put together a a team um as part of um this data provenance initiative of machine learning experts and lawyers uh to try to understand and gain insight into where this data has come from how it's being used um and what kind of uh from our perspective kind of most interestingly what kind of legal limitations were placed on this data um this this figure is kind of complicated but the key part is that data sets are are often grouped into collections of data so you have like original sources and then the data will appear on a place like hugging face or papers with code and then someone will take that data and put it into a bigger collection and then ultimately it gets used for for um AI development but there are lots of stages and people kind of get lose track of where their data is going or where the data that they're using to train their model really came from and what we did is we um would find the original licenses from the original sources and then categorize those licenses along a couple of of important Dimensions um one of the ones that I'll talk about is what kind of use the license permits um and so we find that um if we look at the original kind of correct license based on the source and then the license according to the various aggregators where the data set is hosted um there are a lot of errors and you can see kind of in the reddish pinkish triangle these are all the the situations where something is is incorrectly labeled and the First Column was probably the most problematic that's where licenses according to the aggregators say hey commercial of this data set is fine um but actually um the commercial usage is either not permitted um or the you know original Source doesn't mention it at all um and that can pose as you might imagine a real issue in doing this project we also came up with with another kind of interesting finding which is to say well hold on a second if this data was created for the sole purpose of training AI models then the fair use discourse that you'll often hear um applied to training data might not apply in the same way right because the whole kind of idea of fair use is that you have the secondary purpose that is distinct from the primary purpose right I I wrote an article um say to be in a newspaper and now that article is being used to uh train an AI model those two purposes at least appear quite distinct but when you create supervised data well you created that for the purpose of training an AI model um and so fair use might not apply in the same way there's also this question about the market effect right so like if I use your poem to train my AI does that affect the market for the poem of course once I start creating new poems that's a different question but like the moment I've trained an AI it's not like I've created a new poem that competes with your poem um but when I steal or take your your supervised data um in a way that you didn't permit and I use it to train my model well the alternative is I would have paid you for that right like I'm directly kind of competing uh in for for the market um of your data set so this is kind of an interesting analysis we were actually able to um write this up with the help of uh Folks at uh the Buu uh technology law clinic uh as a comment to the US copyright office um so just to highlight kind of completely different but I think still related still kind of in this like Realm of computational law um research um application um of of some of the kind of principles that I hope you know you'll be learning about and thinking about um so heading it back to you daza thank you so much um that is fascinating and incidentally I I was referring to the second to last project when I said there was training data involved um because there was and and you're also correct that the you know the crowning Jewel was the last project you mentioned when it comes to just taking a closer look at this training data uh which is so very essential um let me help you with an an advocate's argument as to why your last project can also I think validly be considered computational law um as opposed to you know like yet another law review article about you know how whatever some legal framework may or may not apply to AI or automation or technology um and that is because um just as you said um being a computational lawyer which is a nice phrase um was some of those skills were needed in order to do this legal analysis the first in the first place you had to find a a way that was um accurate um and and completely captured the relevant legal aspects of these licenses to to represent that information in in in a form that was data that you could then analyze and then show on things like charts and graphs um and you did that um you know you you all your team did the hard work of reading um the licenses and categorizing them correctly and then having the right kind of identifiers and metadata around each one so that you can you could do this analysis and see was there some difference between um you know what the license actually permitted and restricted and how it was being characterized if at all and so that that to me is like that's the hard work of that's the first step of computational law is rending the law in a form that it can be computed um so anyway that's that's my advocacy on on behalf of your project as can also totally be considered computational law although it is you know mostly about the lot of um these data sets so anyway um I think that's incredibly fascinating um and and it makes me want to ask what are you working on now that isn't yet capable of being you know rendered on a slide and and what do you what do you foresee um for the rest of this you know new Dawning year of 20124 and into 2025 like what's on the horizon at The Cutting Edge of MIT research in this area of computational law with a heavy emphasis and thumb on the scale for generative AI that's a good question um there are a couple things so we're you know of course these projects are all like you know it's by definition it's early stage right so there's like next steps for for essentially all of the things I've talked about today um we are for example trying to really build out this retrieval augmented generation platform and really think about how do you design a platform like this maybe even you know what would be exciting would be to go to to like you know prosay litigates and give them a tool like this and see like how does it change how they interact um with the courtroom with kind of legal questions um kind of more of the human computer interaction side of things um another uh you know big kind of research project that we've been grappling with as you know for for quite a while is um we we can kind of start thinking of of laws like a network and we're starting to kind of like think of like well the citation networks and you have all these documents along the way but that begs the question like well you know common law as you know judg made law has evolved in some way can we kind of get a better understanding a better grasp of that Evolution um so I think that's like a key research problem um and and honestly like it's been on the one hand we're overwhelmed because we have all of this new data all these new tools um on the other hand it's like non-trivial to to think of like a system of knowledge and try to say well where do it come from where is it going um who are like maybe the the key people who are changing it or the key events that have changed it like can we find moments in history moments in law that have kind of given rise to new changes all those kinds of questions um seem incredibly Well Suited um to computational law computational legal analysis and then finally there's kind of a broader like what does the practice of law look like like what does the business of law look like and it's early days for that um and we can see lots of folks misusing these tools misunderstanding these tools but I think it's pretty clear that the legal profession will Embrace a lot of generative AI tools a large of large language models and the clients of lawyers which we sometimes forget about but they really matter right like the clients of lawyers will also be embracing those tools and also making the connection hey is my attorney who's billing me like $1,000 an hour or more is is she using these tools because if not then I'd like to know why I'm being build all these hours right so um I think that there's going to be kind of interesting shifts in the legal profession uh that that are worthy of of research in and of themselves right and understanding how you deal with legal risk how you manage Legal Services um understanding the you know the law there's so many questions honestly um kind of an overwhelming number so so um we'll be busy at work and uh yeah if people are excited about this then then I've done my job so hopefully there'll be lots lots more people working on this stuff um so thank you uh this was very fun thank you yeah well I think people I'm excited and I I know other people are too this has probably been one of the most requested segments is like just a research update um so but while I have you before we before we end I've got one sort of like extra question for you and it's partly because you are really still fairly freshly out of law school like what when you graduate a couple years ago or something like that two three so you you probably remember it better than I do um and well the question I would have is what do you foresee not so much in the practice of law which you just started to go over and research of course and industry but what about for law school itself like what what do you think are what would be some of the good directions for law schools to look at as as ways to reckon with to recognize and to start to address and I would say to support and reflect the Advent of generative AI as a part of law practice like what what sorts of activities or courses or skills or or other implications um and let me just start as a starting point with um I'm Acme law school and I've decided to prohibit use of generative AI for like any meaningful aspect of legal education so if you start from that Baseline we what more might be possible with the um liberalization of some of those types of restrictions and what what kind of application of generative AI would be you know kind of beneficial and appropriate or maybe even necessary for competent you know well-educated ready to practice lawyers coming out of law school yeah so I I'm actually probably more receptive to to the argument like hey you know we shouldn't have this these kinds of tools in in law schools because they'll they'll prevent us from learning about the law like I I think that there is a little bit of something to this argument and I don't think that I could have done the research that I've gone on to do without like you know the SLO of like using Lexus and wesla and the other legal research platforms understanding what a key site and what a headnote is and shepher isation like you do have to learn those things the way you have to probably learn long division and other kinds of things even though it's not actually used day-to-day however it also makes sense to me that you would learn a little bit maybe not about the specific tools and vendors but more about like the risks and the limitations and the opportunities like as a lawyer I need to understand um how what kind of precedent I can site right and I need to understand that like some things are good law and some things are bad law well by the same token I need to understand like what are the tools out there um that will retrieve cases what are the tools that will let me you know compile cases into arguments and I really expect that legal practice is going to embrace those um but we need some kind of AI literacy among the lawyers who will use the tools not because that they're they're going to be developing those tools necessarily but because they that kind of literacy is needed to do responsible kind of legal technology usage right to to responsibly use these tools um so that's kind of one side of things so yes it's fine if you need to do manual legal research it also seems highly appropriate for there to be at least one class um that covers some of the kind of more AI literacy topics however the other thing that I think is a real opportunity for law schools with clinics is to say well the purpose of clinics is one kind of practice oriented and two it's to serve clients right and so it seems like there's an interesting opportunity to consider like the existence of like a meta Clinic a clinic that develops and and helps law school students who are interested in developing tools um you know help the other clinics um so like a clinic for the other clinics kind of thing and I think that um especially now that some of these tool tools have become very accessible right you don't need a computer science degree to be able to use chat gbt um so by the same token you don't need a lot of technical knowhow you need kind of design thinking skills but you don't need tech a lot of technical knowhow to build really impactful tools um and uh I think that there's a really cool opportunity for law schools to start kind of not just encouraging their students uh to build these tools to train that muscle if they want to but to do so in a really impactful way um and maybe also to dabble cross over into uh some of the like human computer interaction literature and communities and start kind of exploring like what do these kinds of tools look like um and and how do we design them responsibly um so there's lots of options for law schools uh I think it's an exciting time actually uh to be a law school to be a law school Professor um and uh yeah I'm I'm hopeful I'm you know I remain always The Optimist um that that law schools will find ways to kind of integrate this into the syllabus syllabi um in in productive responsible ways here here um well may it be so and uh you know I I did have a little love Cher motive which is I'm hearing now from more and more of my um friends and colleagues at law schools who are sharing really innovative ways they're they're starting to integrate use of gender of AI into their pedagogy and into their syllabi um and the curriculums and um and you know there's a thousand flowers blooming right now but um one thing thing I can say for sure is I I love what the way that you're incorporating it into your research and um I can't wait to see what you come up with next so thanks very much for taking the time to share um what you've been working on uh Robert and U and you know um don't don't be shy about sharing um the next um kind of flock of projects when they come up and and I'll be sure to to Vector them right into the stream perfect thank you so much for having me take care thanks