Transcript for:
AI and the Practice of Law: from CaseText to CoCounsel, with Pablo Arredondo, VP of CoCounsel

the first flickers of the power of this technology that I saw was suddenly the computer flagging instances and I was like wait how does it know that that means overrule because that is not a normal way to say it and it was because we had moved Beyond literal keywords my co-founder Jake and I were shown a demo of gbt 4 and within 12 hours we had pivoted the entire company around it a lot of times you'll hear people say all that will be automated is the menial work that nobody wants to do anyway right but they can also help for things that are like deeply substantive like really you know Finding inconsistency so you can go do the like famous cross-examination that destroys the witness hello and welcome back to turpentine AI if you're looking for the cognitive Revolution don't worry it's not you it's us turpentine is developing fresh new AI focused shows and this feed is set to become a best of Show featuring highlights from multiple sources meanwhile we've created a new feed dedicated to the cognitive Revolution which you can find And subscribe to at our website cognitiv revolution. just yesterday we posted an interview with Andrew Lee founder and CEO of email client shortwave which has the single best AI email assistant that I've personally used and has become the first email client to effectively replace the Gmail web app as my go-to email experience definitely take a minute to visit cognitive revolution. to subscribe for that and plenty more original content which will only be available on the new feed today exclusively on this feed we're looking at how AI is impacting the practice of law with Pablo arando who helped create and drive the adoption of AI and other Advanced legal search tools as co-founder of case text and today is doing the same with large language models as VP of co-consul at Thompson Reyers which acquired case text last year we begin the conversation with a historical overview of legal research from its pre-digital citation-based Origins to the computerized but still fundamentally keyword driven search era through the last 10 years of case text and the AI powered innovations that allowed attorneys to search no longer just by keyword but by meaning dramatically improving their ability to to locate relevant case law and finally on to the present day in which large language models are beginning to fundamentally change how lawyers perform an Ever growing range of high value tasks including document review and deposition prep contract review and more we talk about Pablo and team's first exposure to gp4 how they immediately pivoted the company to take advantage of this new technology and how they've designed their product and perhaps more importantly their product development and quality assurance processes with reliability in mind we also discuss why co- council's price point which is a couple hundred per attorney per month is not really a problem given the high value use case case that they serve and why Pablo has remained a gp4 maximalist at least as of the time that we recorded a couple weeks ago just before Claud 3 was released Pablo also shares his thoughts on the future of legal billing the potential for AI powered arbitration and the evolving regulatory landscape governing the use of AI in the legal profession while he is as you'd expect from a VP at a major company like Thompson Reuters extremely focused on responsible development and deployment his hope is that large language models can make Legal Services faster less expensive more accessible and higher quality for all as always we appreciate it when you take a moment to share the show with your friends this episode would be an obvious fit for the lawyers in your life and again please do make sure to subscribe to our new feed which you can find at cognitiv revolution. we'll have original content both here and there over the coming weeks so you'll definitely want to stay tuned to both feeds now here's my conversation with Pablo arando co-founder of case text and VP of co- counil at Thompson Reuters Pablo arando co-founder of case text and VP of co-council Welcome to the cognitive Revolution thank you so much for having me I'm excited about this we've got what I think is going to be a really interesting and hopefully you know it'll be a good mix of fast-paced and deep diving uh into all things legal Ai and you know you've had a real front Rose seat in this business over the last 10 years since you founded case text I would love to start if you would give us kind of a backstory on just a brief history of the application of maybe technology more broadly but obviously especially emphasis on AI in the legal profession I realized that you know while I've been into the product and played with it today and we'll get into that in a lot more detail I don't really know that much about how things were before if you know I kind of know the AE Lincoln story of like he had to walk a long way I think to borrow books and return them that's kind of all I know like where did we where are we coming from in legal research and where have you been over the last 10 years with casex then you could take us all the way up to present yeah absolutely so you know last sort of an interesting area because you had some things in law much sooner than you had them in other places so like a citation graph right things citing to other things like that that we now think of was like hyperlinks on the internet not many corporate Min already had that in the same sort of like rigorous systematic way that law had because you were constantly citing to earlier opinions right and citing specific pages on earlier opinions and it was always sort of building on itself and you know very very quickly especially in America which you know had an explosion of lawyers and this kind of of litigation there were quickly more judicial opinions than anyone could read in their lifetime and so you immediately started to have these sort of information retrieval problems and challenges how do you find the right cases and not just finding the right cases how do you know if the case you're looking at is still good law and you know one of the very earliest sort of legal Innovations was a guy named Simon Greenleaf um this was so long ago so he was in the town of gray in the territory of Maine so it wasn't a state yet and it was so long ago that there wasn't any American case law to rely on so we just cited to British case law because you know what we had toci to something and he cited a case that had actually been overruled by subsequent courts essentially the courts had later said that's no longer good law and the judge you know threw him around the courtroom and he was embarrassed and he came I'm taking a little artistic license here but you know he came back he was AG he said you know I never want to feel embarrassed like this again but how do I know what cases have been overruled right like I I can't read every case and know them and so he created this this started to make a list of these cases have been overruled by this case and so this I think in the late 1800s was this sort of early example of a legal tool that then grew and grew and grew and became the resource this sort of meta resource that lawyers could use to try to just better navigate the law and better represent their clients later we're going to see that our first use of large language models at case text was actually because we were putting our shoulder to exactly that challenge that was there in the late 1800s but I think a couple things that in between that I think are worth noting it used to be that K La you know at first judges literally weren't even writing things down it was sort to be like hey I think I remember Fred said something about that you know that that's how common law was handed down then they started writing it down in very hodg Podge ways you know you'd have like crazy things like like an almanac you know and they like the guy doing the almanac would also write you know reports' say like then lord judge Mansfield held that the by the way I think we'll have a great crop of Corin just complete mess until you had this West publishing and this guy Jonathan West who is now Thompson Reuters is sort of the continuation of all of this who first systematized the case law into these reporters and so when you see a lawyer doing his commercial or kind of on TV and he's got those books behind him those are all this this sort of systematically let's take all the case law let's put it into one form into one series of books and start to make it much more navigable but of course this was long before computers and so the question now is how do I find the right case and so you had this taxonomy that was created by humans who went and went methodically and said okay we're going to divide up law into all these different areas and we'll create this sort of flow this taxonomy you know I'm looking for Tor okay somebody's been injured okay now I'm looking for animal attack okay now I'm looking for dog and this was quite useful because it could help you find what you needed but it was also a prison of sorts because however they divided up the law that was what the law was right you had to stick to whatever framework they had and so that was the governing Paradigm for for legal informatics for a while until the I think very late 60s and early 70s when you started to see the digitization of case law and so there you had another company now that is Lexus you know this actually came out of I think a group in Ohio that would started with just like the Department of Agriculture and I don't know I'm blanking on the specifics but essentially sort of small limited project became then suddenly you now had everything digitized and so this was a huge step forward in some ways but also brought in other challenges so now I could navigate the millions of opinions just by searching a keyword right it's whatever keyword I want and then you got into Boolean searching you know patent within sentence of computer and so you could Now search using keywords but as we all know keywords are quite limited they're very literal and you had issues of both Precision which is to say things that came back just happened to have that word but weren't what you cared about and you had more Insidious was issues of recall there were things you you did want to see but you don't see because it happened to use different language and that was really the Paradigm through when I was practicing I was pet lawyer at Kirk Ellis you know that that's what kind of surrounded me during my time practicing law and so okay so now we're kind of starting to get closer into kind of case Tex and these new sort of systems and I'm gonna be nerding out on this because I'm assuming your audience doesn't mind nerding out I hope that's okay yeah no we're here for it people want the Nuggets you know they want to a lot of them are building their own tools obviously in you know many different areas so they want to learn from the depths of your experience in particular all right yeah I mean you know and so so you know the the the early stuff that we were doing with case teex so one of the issues we had was so remember I talked about like knowing does this case over rule this case right that became this very important tool called the citator the famous one was called Shepherds and then that got bought by Lexus conon Reuters has theirs versus keite really essential system but the issue with it with both of them at the time that k6 got it started is that it was only looking at the direct citation path you can only see cases that directly cited to your case to get a sense for that area of Law and how things are being treated and the analogy I make is imagine you went to a video store back some of you have never heard of these but there used to be a time where you'd go in to rent movies and you asked the clerk you know I really am interested in the God I love The Godfather can you recommend any other movies and imagine the clerk said Godfather Part Two Godfather Part three that's the end of the list and you'd say well wait a minute right like that's a pretty impoverished list based on that right what about Good Fellas what about me what about all these other movies and so one of the things early things we did at kch this is last decade was exploit the same patterns that you had been seeing in Spotify and Amazon which are these soft citation relationships right when Spotify recommends a song it's not because that song literally references this other song that you were like it's that people who download this song tend to also download that song right and so that sort of soft citation relationship was this big blind spot and our first commercial product was you could take the brief right the length document that lawyers use when they're trying to persuade the court and we would analyze all the CIT cases you do site and then run a much more robust citation analysis and then we could suggest cases that you had overlooked cases that weren't in the brief but that you should have read these early moments where at the top firms firms that have no you know no money money not an object relative to the stakes of the litigation I mean right they buy all the tools that they thought could help their clients they were saying how we were missing this case how did we miss this case the best attorneys Etc and it was just because the technology they were using had this this blind spot so we started then selling these kind of specific Technologies to the top firms you know the the the very you know the the affluent firms but at the same time we found that there were also a lot of attorneys who couldn't afford the best to you know the the West laws right and they were relying on like Google Scholar they were relying on tools frankly that you know could could have been a lot better and so we then decided we wanted to build out a full-fledged research system that they could use for all of the different things you need to do for research and so we ended up in this sort of bodal place there's this great Jack Daniels commercial where it opens with them like serving Jack Daniels at the Ritz at some wedding and then it shows like a biker bar you know all the bikers and they're also during Jack Daniels and it says like Jack Daniels served in fine establishment and questionable joints since like 1870 or whatever so Kix sort of was this sort thing where we were at like the you know Manhattan skyscraper firms you know where you know represent you know antitrust you know eight figure litigations but also increasingly you could find us in like you know the strip malls of Pasadena or wherever you know solos were hanging their shingle also doing very important work of course right because it's it's also very important that folks who don't have Deep Pockets have representation okay in creating that full-fledged research engine we needed to do what exactly what that lawyer from the 1800 Simon Greenleaf did we had to know does this case overrule this case because then we can warn the attorney when they're reading it and sometimes the court will be very explicit they'll saywe overruled Jenkins great easy to parse but sometimes the court will say we regretfully consign Jenkins to the dust bin of Oblivion because judges can say it however they want and sometimes they like to get poetic and so now we had this really profound information challenge how do you get a computer to detect that kind of overruling that kind of treatment where it's not using the normal words and to be clear this was part of a process that involves humans there was very much a human in the loop but how do you triage how do you identify like humans should take a look at this okay so we trying to do this we're using kind of old techniques and then Rejoice Here Comes Bert Here Comes large language models and our first experience the first flickers of the power of this technology that I saw was suddenly the computer flagging instances and I was like wait how does it know that that means overrule because that is not a normal way to say it and it was because we had moved Beyond literal keywords because this this this language model was starting to actually understand you know you can there's a whole debate about understand versus not and and truthfully when we're talking about things like Bert I think it was more just that you know we had encoded language in such a way that it would draw in these examples and so that was the beginning I think 2018 is Right very very early on of these large language models hey we'll continue our interview in a moment after a word from our sponsors AI might be the most important new computer technology ever it's storming every industry and literally billions of dollars are being invested so buckle up the problem is that AI needs a lot of speed and processing power so how do you compete without cost spiraling out of control it's time to upgrade to the next generation of the cloud Oracle Cloud infrastructure or oci oci is a single platform for your infrastructure database application development and AI needs oci has four to eight times the bandwidth of other clouds offers one consistent price instead of variable Regional pricing and of course nobody does data better than Oracle so now you can train your AI models at twice the speed and less than half the cost of other clouds if you want to do more and spend less like uber 8x8 and databricks Mosaic take a free test drive of oci at oracle.com cognitive that's oracle.com cognitive oracle.com SL cognitive omnik uses generative AI to enable you to launch hundreds of thousands of ad iterations that actually work customized across all platforms with a click of a button I believe in omnik so much that I invested in it and I recommend you use it too use Cog rev to get a 10% discount quick interjection because I I don't know the history on Bert did they open source it for you or was it were you just able to so you were able to get a trained Bert to work from from the beginning yes and we so they this is one of these great moments that really shifted I think Society in some ways and and unfortunately and this is probably there's there's better folks than myself that talk about that sort of aspect of things but less and less I think are people kind of putting out there and open sourcing at least certain aspects of it I know there's a lot of Open Source movement in AI but you know I don't think Google's open sourcing Gemini let's put it that way right um but yes they put out burs as a paper Jacob deblin the code was there the techniques were sort of just it was all just you can go do it and what we did is we actually took that same approach and just trained it on us common law right but basically took directly from Google the techniques the approach and then frankly I think some of the code as well to kind of run it and that was a you know just incredible moment for us and everything we're doing now is really sort of just this Evolution sometimes like punctuated equilibrium right it's not just linear it's like right but very much in sort of the same path that started with bir and so once we saw that it could do this for detecting overruling treatment we said okay well wait a minute now we can do this for search and so we created this engine called parallel search where attorney could just enter a full sentence could take a sentence from your brief and it would find relevant case law even if there was no overlap in the keywords at all right and so like we had a demo sentence we did about Miley Cyrus getting F we used Cyrus as the name it wasn't Miley Cyrus but you know getting somebody being fired for not wearing a mask at work and suddenly it was Finding cases where people weren't printing on safety helmets right like it was Finding analogous case law right things about satorial safety right things like that and and for lawyers who had basically been in what I theatrically call the keyword prison this was as seismic an event as you could imagine at at the time I thought you will never get more seismic than this boy was I wrong and we'll get to 2022 in a second and so we released this technology through the parallel search it you know just immediately was a huge hit courts were using it firms big and small amla adoption amla just means the biggest 200 firms getting lawyers to take time to adopt new technology it's a hard fight right they're they're busy folks they're risk adverse Etc parallel search was this example of one that you just saw kind of spreading and it was absolutely beautiful but it was because it's so often lawyers are trying to find something but they don't know the exact words and so we did it for case law first then we expanded it so that you could upload anything transcripts eisc Discovery right contracts whatever you wanted and apply these these sort of early language models to them gbt 3 happens I think that's 2020 we see it wow that's really neat 10 minutes later though we know we can't use it for lawyers it's just not there right amazing kind of you know kind of sense of like oh wow but not not reliable enough not Nuance enough just not good enough to be something for for attorneys and so we know continued focusing on our search systems and kind of doing that stuff and then one Glorious Day September 16th 2022 my co-founder Jake and I were shown a demo of gbt 4 and within 12 hours we had pivoted the entire company around it it was just so much better than what we had used before or seen before and unlike our bir systems which were really just about search this tool was being used for like you could sum you could create timelines right we had this we had this period where and remember this is before chat gbt came out and chat gbt was based on gbt 3.5 and at some point later Sam said like you know we broke it up because we thought it would be too much to see chat and gbt 4 at the same time well we were the absolute privilege of being the small group that got both barrels so we went you know suddenly we were on a slack Channel where we could ask gb4 to you know create a timeline or change this or I mean and we just the next 72 two hours I I I felt like I was going to burn a hole in my brain from just how many things we tried and how many you know it was just incredible to see what it could do and so we pivoted the entire company around it and we just worked hand inand with open AI you know giving them feedback from lawyers we were like their domain experts for law what we realized though of course early on is you can't use this stuff you can't use gb4 as a chat bot for a laot because it hallucinates because it's not up to I asked it about a case I actually worked on and its answer was so believable that I was second guing myself about a case I worked on and then I kind of I got mad and I was like no you're wrong I worked on this case and I kid you not it said you can sit there and brag about the cases you worked on if you want but I'm right and here's proof and then it Con coed a URL to Nowhere right so you're wet wow this is you know but you can't use it like that it will hallucinate it make things up and so you know from the very earliest days we understood you had to use what now I think is you know very common everyone kind I think you know becoming almost like you know just retrieval augmented generation you have to Anchor the system in a search engine that will retrieve Real Results and then Force gbt for to answer based on what it's seeing in front of it the real case law not freestyling an answer I had a very similar experience you know I was in that same time frame as an early open AI customer you know got a preview and had an uncannily similar experience where I was asking it about chemistry research I had been a chemistry student as an undergrad and asked about the research agenda of the professor that I worked for and asked if she ever had a co-author you know named leens namely you know meaning me and it said yes and what paper and it gave me the paper and I was like wait a second did she put me on another paper that I don't even recall being a part of and then this was kind of the you know the entry point for me to understanding hallucinations not that I you know necessarily thought it was infallible before but it was like whoa you really have to be careful because here I am you know and I think it was the middle of the night is I'm kind of because I was also you know like you just really floored and and just wowed and you know not sleeping a lot for a few days there at least really for a couple months yeah I remember being like this is so believable confused me as to whether or not I was actually involved with this I had to go you know look it up for real and and kind of reground myself on what was I actually a part of you know a few years back so yeah it's very uncanny similarity similar experience I mean we were fortunate that we got to get tricked by it very early on in like the lab you know later you know these Horror Stories of the lawyers who like went into court you know relying on it there were startups that were like putting out marketing with hallucinations in it like they were putting out their screenshots and it's like that's not a real case like so it definitely took a long while to kind of to kind of get there right so so we put this out so then you know and and the truth is like it was so amazing that there was always this voice like is this real can this really be real right like it can't all these things can't really be happening right because it's just it it was frankly so unexpected I think even the really enthusiastic folks weren't anticipating maybe somewhere but I I certainly the folks I mean weren't anticipating the leap that was gbd4 and for me the you know I a co-author of the study where it passed the bar exam right we we got kind of ran that with some colleagues on you from Stanford but to me what really drove home that it was real was when we brought it to legal research and the law librarian community did not rip it to shreds and in fact you know they would say things like this is solid that is the most gushing praise I've ever heard from a law Librarian On Any I mean these they live to tear things like this apart and to see them sort of say wow it actually seems to be able to deal with the Nuance you know of different areas of law and it's understanding these queries that's when I was like okay wow this is actually happening and you know that that um that that really helped so you know early days so first we had six weeks before chat gbt came out so now we're getting to show very limited I mean you know up I understandably there was a very small group that we were allowed to kind of share it with but you know this was a time where you could just demo the poems and people were florid you were showing them everything they' never seen llms in any capacity so you just show them poems and show it translating and things like that even before you got into legal tasks and they were kind of Blown Away they were you know is a cyber dime there was all of that sort of angst you know and it you know one of the unique perspectives that we got because we had this early access was seeing a lot of people's first reaction to just llms as a whole and that mix of excitement and sort of fear right and confusion and I think there's something kind of uniquely human about language so when you see a computer for the first time doing it it's something right even though like obviously it's just guessing the next word but just having it even mimic language so well was unnerving I think to a lot of people and so then we started you know basically we built out these different skills we just took use case after use case after use case packaged them all together as co-counsel was the name of the product you know obviously an omage to to co-pilot and you know that's been basically what has been going on it it's been so different the debut of co-counsel was on Morning Joe on MSNBC the idea of a legal tech product going on national TV for its launch right it would have just been absurd nobody would Morning Joe wasn't calling when we had the citation you know blind spot the just the the the amplitude just how much fervor and excitement and suddenly you know I'm used to giving demos you give it to one law librarian and they go that's kind of neat you know in three weeks I'll let another law librarian and see it right that's your traditional legal Tech here you'd show it and like all the partners would all assemble like in the room like somebody's been embezzling or something you know what I mean it' be like just completely different a sense of it and what we you know really had to focus on is being responsible with it not having it claiming it can do things it can't really putting the guard rails to make sure that it wasn't hallucinating or at least like you know dominous amount of hallucination you know things like policing the results for quote checks like using old school Tech to be like does that quote actually appear in that case and then really creating a system that facilitates oversight because the danger of course his attorneys over relying on this stuff right because it's so seems so good and seems so oh it's done when in fact you know it's not quite a human level I shouldn't say not quite it's it's not near human level ultimately for like really important legal thinking and reasoning I I think it has a ways to go let's get into the product in more depth I mean it's it's really interesting you obviously had very early start and you know I've had a chance to go in and play with the product Hands-On now I'm not a lawyer myself and I also don't have like a lot of legal documents you know lying around so I I couldn't necessarily push it to you know the limits that your actual customers would but definitely a number of things uh jumped out at me one is that you've kind of been building a lot of the same things that the community as a whole has been building but probably in parallel because because being early to it you didn't have the luxury of you know a lang chain or a llama index or whatever you know at the time you're building right so you're identifying these problems kind of in your own you know Lane as the the community more broadly is is also figuring out you know what are the complimentary tools that the language models need I'm interested to get your kind of perspective on a number of different dimensions of that maybe for starters let's just kind of describe what the product experience is I think there's a couple things about it that are notable one is that it is just a lot more structured than your typical chat so it's it's not just there's kind of two tabs is the the main interface there's the chat tab where you're going back and forth and that will feel you know very familiar to anyone who's used chat GPT and then there is the results Tab and this is where you know from the chat tab you essentially can create tasks and you know this is an interactive experience where you know you're having dialogue the system can kind of come back and say okay here's what I what tasks I understand you to be asking me to do and then you can say yep okay go do those tasks and then those tasks actually get run seemly kind of in the background or in parallel and you know and you can elaborate a little bit more on the kinds of tasks and the the volume of tasks that that people are putting through those and then you can come back in a little bit and actually look at the results so that way it is a little bit more of a I I often talk about kind of co-pilot mode being your real time interactive engagement with AI and then delegation mode on the other hand being you know if you're setting up like workflows and you're trying to get to the point where you're not going to check every single output I call that delegation mode and that's like much more prompt engineering much more systems integration etc etc this I think lives in a kind of interesting space in between where you're in that chat real-time interactive mode but you're able to kind of spin off these individual tasks and then they actually live somewhere else that you can come back to and review what are the tasks you know that people are doing and and in what kind of volume you know where are people finding the most value from this product experience today hey we'll continue our interview in a moment after a word from our sponsors the brave search API brings affordable developer access to the brave search index an independent index of the web with over 20 billion web pages so what makes the brave search index stand out one it's entirely independent and built from scratch that means no big Tech biases or extortionate prices two it's built on Real Page visits from actual humans collected anonymously of course which filters out tons of junk data and three the index is refreshed with tens of millions of pages daily so it always has accurate up-to-date information the brav search API can be used to assemble a data set to train your AI models and help with retrieval augmentation at the time of inference all while remaining affordable with developer first pricing integrating the brave search API into your workflow translates to more ethical data sourcing and more human representative data sets try the brave search API for free for up to 2,000 queries per month at brave.com API hey everyone Eric here the founder of turpentine the network that produces the cognitive Revolution this episode is brought to you by odf where top Founders get their start odf has helped over 1,000 companies like tra levels and Finch meet their co-founders and go on to raise over $2 billion apply to the next cohort of odf and go from idea to conviction on what's next startups change the world they can also change your life is it your turn learn more at beond deck.com Revolution when we started at first it was just buttons for each skill there wasn't basically a chat right you had a button called legal research you had a button called review documents and what you know we realize is that like people like the chat flow like the chat is sort of you know just a more intuitive and natural way to kind of do that and so then we we've switched maybe a few months ago to having the first thing you interact with bachat and that Brees challenges right you have to understand the intent you know what are they actually asking you to do right whereas it simpler if you just hit a button and then we sort of realize well wait a minute okay sometimes you might want to go back a little bit to like more structured right so hey I want to review some documents have questions okay then we kind of give you a more like form like ability to put in the questions if you want right so one of our challenges has been how do you strike that balance between the kind of the just wonderful flow of a chat and the intuitiveness of a chat but knowing that behind the scenes we do have these discret skills and these discreet capabilities and how and how you do that is something that I think will frankly be continuing to evolve right I think we're still trying to find exactly the right right balance in terms of the skills you know Loosely you can divide them into you know there's two major flavors of lawyer maybe three if you include criminal folks who do you know crime but for clients there's litigators you know who are you know you Sue this person in court and they're the ones that are constantly searching case law and then there's transactional law which is you know I want to merge companies right and there's a huge amount obviously of law that goes into evaluating those contracts and you know the various due diligence and things like that and so case tags for 10 years had really been focused on litigators because we were legal research and that's although you know it certainly can sometimes impact transactional law really litigators are the ones who are constantly searching case law to find cases to site to the judge but when you know even starting before gbt 4 when we had our sort of our our bbased search suddenly we were being told by these firms hey can our contract you know our transactional guys take a look because they too sometimes need to search for something so our first battery of skills during the beta phase spanned both so we had like you can do legal research where it'll just run rag on legal research we had things like deposition prep so I'm going to go depose so and so and it could suggest a bunch of topics let you alter the topics then suggest a bunch of questions to sort of jump start your ability to to prep for a Depot we had things like timeline so you could upload a big messy Corpus of documents and it would create a chronology which is something that's very labor intensive right to go do as a you know I remember as a first year associate spending a lot of nights just trying to map that stuff out and then on the transactional side we had you know the ability to ask a question of a merger agreement you know like what is what's the this or this term say and then fancier things like one called contract policy compliance so basically let's say you're you know a big Fortune 50 company and you say here's how we do things for our contracts right like we insist that it'd be governed by Delaware law right or we we won't sign unless the IP has this or this you know no what we could do was basically give those policies to gbt 4 and then anytime a draft contract came in it would police the draft not it would sign the relevant Clause say is it kosher does it actually comply with how we do things and if not it would then suggest a redline for how to change the draft to comport with how you do things right so the sort of just it's like you can almost think of it like there was spell checker then came grammar check now thanks to gbt 4 we have like substance check right we have the ability to like check it for like deeply substantive aspects of things to see if it complies and then we you know there's things like what's Market you know you want to see how have other companies handled this or this aspect of a transaction you could we could pull all the the relevant data from the SEC gbd4 would sort of scan it find relevant stuff and then synthesize a report for you right so even though Court transaction law was not really our wheelhouse when you're with the power of gbg4 right it's it's pretty fast that you can start creating pretty valuable skills and then on the litigation side I mentioned a few of them some of the more interesting use cases actually you know during the beta phase we're still we're having yet Productions but one great example we had a a client another for 500 company and you know there's like certain expert Witnesses they make their living just testifying against the company right like the guy that just year in and year out I'm the one that the plain of calls to go say why you know Monsanto do whoever you want you know pick your company and so they said to us hey if we gave you all of this expert Witnesses prior expert reports and prior testimony from like multiple earlier litigations could co-counsel analyze and find contradictions that we could use for cross-examination and to me that was one of the most amazing use cases because that is the kind of thing that lawyers go to law school to do right like a lot of times you'll hear people say all that will be automated is the the you know menial work that nobody wants to do anyway right and look the truth is yes you know llm will help tremendously on that front but they can also help for things that are like deeply substantive like really you know Finding inconsistencies so you can go do the like famous you know cross-examination that destroys the witness right that is like at the heart of what being a lawyer is sometime or certainly being a litigator and so what we found is that you know in just the earliest days the ability to point in at at pedestrian things that are more just tedious than maybe intellectual but also to see how it can help with things that are actually quite intellectual right and quite quite subst so in terms of how that is built you know it's it's crazy that we're you know we're still not even a year into gbd4 being public and there obviously been many versions at this point and you know enhancements in terms of new features such as function calling and also just you know a better understanding of kind of how to do the retrieval augmentation and all that kind of stuff I wonder if you could maybe the best way to ask it would be to give kind of a little bit of a history of where you started and how things have evolved with the product I imagine that and context by the way is another one that obviously is huge right so first version 8,000 token limit you know limited access to the 32 maybe you were using the 32 that obviously could get a little bit expensive so imagine you're like in the early days highly structured and you know big emphasis on managing context perhaps it's still that way and you have like you know a bunch of kind of very discret prompts and it's about kind of chaining them together or perhaps you know because the models are getting a little bit better generally and the contact window is growing you're able to just kind of push more and more onto the models and and rely Less on your own structure yeah so a few things there I mean the context Windows have sort of broken my heart a little bit right because they they turn into a mirage right which is you know they're missing stuff in the middle right they don't they're not as accurate when you fully utilize it so I say it's like saying my boat can seat 100 people but if you put more than 10 people on it will sink it's like well does that vote really SE 100 people in right like so for what we do where we're being right is so important right so far the longer context Windows haven't really yet been there for us now it's getting better I think you know there's there's progress being made and another limitation for us is we do really rigorous testing on a model so open AI will come out with an improvement it's not like we just swap that in and we're not just pointing to whatever their current one is we have to go test the model because it might be better at one thing but worse for what you know the are using it for and so there's always this lag where we know there're these wonderful things yet but we don't yet have them in our system you know there's a process for that I mean I remember one of the great example is you know now there's a Json tacle right just Json we spent like weeks like you know trying everyone begging it please do Jon please do J like just you know that's an example where you know open a like the evolution of just the field I think is sort of what you're pointing to like things are getting better and better and easier and easier for developers and you know believe me well worth it for the Early Access like I'm not complaining by any means but we certainly some of those we were kind of doing the hard way and now there's easier methods but in terms of yeah so I think that's the main thing I think that there's a lag for us because we have to go test and V everything we can't just hot swap it in even though I know there's a lot of great stuff that's coming out right and then there's other aspects like you know law gets really violent there's often violent topics there's often racist topics there's often you know really you know the and so we had to have these filters removed so that tb4 could interact with this stuff in a way that the chatbot you know as an issue of alignment they're basically like steer clear of that stuff so there's a number of ways where like our system which is on our own you know dedicated instances right we have basically our own kind of path that we're doing there's a number of instances where we had to deviate both to kind of protect our use cases and or also just to ensure quality are you fine tuning gp4 that I mean it sounds like there is a slightly different version that you are using as opposed to the Normal public API most of what we've done so far has been relying on just gb4 without any fine-tuning the retrieval engine for some of our rag especially for case law was using a home build system that we trained on common law but very much it was gbt 4 that was doing the heavy lifting now I'm very proud to say that our team as part of our Alpha Testing we got to do some reinforcement learning for the actual gb4 had folks teaching it not to swear and folks teaching it not to you know tell you how to build Molotov cocktails we actually had some of our our very fantastic reference attorneys and you know folks giving an input on you know how best to describe how a document is relying on a case so I just like we got to put one tiny little tile in the alra you know know I mean I just forever like talk about that so you know but but that's the closest you know sort of legal specific stuff now gbd4 was trained on a lot of case law right I think you know it's certainly a lot of this law but we haven't yet done any sort of f you know we don't have a variant of gbd4 that's fine tuned yet certainly as you know like that's something that increasingly opening eyes is looking for Partnerships and and there well could be some things there on the evals it sounds like you have put a lot into that I think that's something that a lot of people are kind of coming around to now even you know in the in my you know the main application that I have built is a video creation app and the core language model task there is video script writing so we don't need to be you know nearly as rigorous in our testing but one thing I have noticed is that the flywheel is getting tighter right it's getting easier and easier to run and we use 3.5 turbo fine tuned it's getting easier and easier to to kind of rerun the fine tuning now you've got a new model we used to not worry too much about the evaluation process because we felt like we were just you know up close and personal with it enough that we kind of would get a feel for how it would go and then we had a couple spot checks that we would run and you know that would be kind of that now with the flywheel getting so much faster it's like gez we do need kind of an automated way to do this not to say that you know I'm always a big believer in not fully automating this stuff you may have a different opinion but I'm always kind of like there's no substitute for for still being at least somewhat Hands-On but I'd love to hear about your system because you know I think a lot of people right now are searching for what is the right balance to strike in valuations how much should they be doing with like modelbased evaluations how much should be objective how much should still be manual what have you learned built are there any good tools that you love I mean everything about ebals I think is of interest you know some of the most important stuff we did as a company the users never see right and that was we built an internal framework so I mentioned we sort of pivoted the entire company within 48 Hours of seeing what gbd4 was capable of part of that was creating these trust teams who's just you wake up and you try to find this thing messing up and by messing up I don't mean crazy hallucination although that too but also you know is it just saying the wrong answer right is it quoting the wrong right just sort of any and all ways that you can find this thing breaking and we just we built a whole framework that had these series of tests that you could run so it like you said it's six mixed you want as much automated as possible but then there is no substitute for also having people using it and in our case you know there's attorneys using it right and so you know you hear very quickly if there's an issue and so you know we aired on the side of of testing right and could we have moved faster probably but we really thought that what it means to be responsible with this is to just go overboard with testing and so when we were first beta when we first had our beta I think we had like 16 some some much larger number we actually launched with a very small subset of that because those were the only skills that we felt and met like the rigorous testing we had done right and there were some that were close and that was painful right like actually our timeline scale was an example you know people were like why' you take that away I love it I love it we said well our testing it's not not there yet right so you know I think for it's use case specific right for law for medicine you know there's certain professions where like just screwing up is got to have a really bad impact and there's other ones where maybe you have a little bit more wiggle room right we like okay right like you know so I think it's use case specific but for law it's just one of these ones where you can't be testing enough frankly right because there's just you know people get hurt or harmed if if you don't so today you use your own framework you have a a suite of automated tools that kind of confirm that you're getting the right still getting the right answer on all the key questions we've merged with Thompson Reuters so now with Thompson Reuters we've like created like the master skills Factory right that both testing prompt you know the whole flow because part of it is like the velocity of creating these functionalities right that's another kind of dimension of competition and well it should be you know how quickly can you go from from a user you know need or right to something that you can put out there and trust and so you know one of the exciting parts of collaboration with Thompson Reuters has been I mean talk about quality control Thompson Reuters is the absolute gold standard and I've been doing it for far longer than there were computers right in terms of really making sure like editorial Excellence you know their site right and no nothing's infallible but Thomson roers part of why Jake and I were and Laura were like you know these are the best Partners we can have is because there's just no better at ensuring that you can trust the output and so that's one area where like our techniques and approaches and our philosophy merges very well with theirs and so I think we're still you know so we've combined it and I think we're going to be amplifying it and magnifying it in terms of all the things that we can do is model powered evaluations part of that framework like we use for example gp4 to assess you know the 3.5 scripts on pretty subjective Dimensions you know and ask it to give us a rating and I'm always like I don't really trust that but I think I at least trust it enough to say if the average rating takes a dive then I should be paying attention we're almost entirely gbt 4 so you're not you know I mean it's not like we have gbt 5 to go police gbt 4 so when you have gbd4 policing gbd4 right it kind of raises right like if right it's possible that that you know it's a different calculus in terms of how useful that could be it may be that we're doing some experiments for that so one of the things we're doing behind the scenes is what happens when you app gbt forward 3 million documents right these like huge corer where you have sh Doc review and so you know you can have it review the documents tell you which ones are relevant and describe why it's relevant and give it a score then what happens is there's a lot of documents that get the highest score and you look at the descriptions and you're like wait a minute some of these like are clearly palpably more important than others so then you say well what happens if gb4 gets involved again and ranks based on the descriptions right and suddenly you have a much more intuitive list so you know using gbt 4 to enhance the output of an earlier gbt 4 is certainly something I think that has a lot of potential and it may be as a complimentary technique using it to police for quality control is important I don't think I see a world where we completely turn things over to gbp4 to do quality control yet although I would be surprised if we're not doing some experiments at least in limited ways that can that can help okay let's do the same thing that we just did for evals on the embeddings and rag side again you've been kind of even 10 years before up you know close to 10 years before this moment working on that now of course there's a rush of you know new embeddings options opening eyes got them other people have got them sounds like you're still using your own core embedding Tech that predates the the gb4 moment what could you kind of tell us about like how do you chunk stuff yeah I have so many questions but tell me everything about rag yeah so so chunking okay that's a great area right because one of the things you want to do is put domain expertise into it right don't chunk where the question from a transcript gets separated from the answer from a transcript right those are better kind of thought of in pairs right so we did have some kind of domain specific chunking that went into it with the embeddings yeah that was one of these rare moments where like our in-house thing you know thing we built we like wow it does seem to be outperforming at least but again this is late 2022 early 2023 which in this field is like the dec it's like a Century right in terms of how things have progressed so but you know there is enough weird Nuance with case law it is a kind of strange fabric for some ways that we did feel like our own home train system was was working better than what we saw we're constantly evaluating that right I mean our our our main thing is just the best user experience so the moment we think that there's embeddings that will create a better experience uh we will and then what's sort of funny with rag I mean I this might be tendentially to what we were talking about but one one thing that's been interesting is you know you have these firms and they're very happy they're very proud of their legacy and they're very proud of what they've collected and built which they shouldn't you know they've been around for 80 years and there was this sense of like well can you create a gbt for with just our data and they're like like I know it's read all of the internet but wait till it gets a hold of our summary judgment motions that'll really kick things the de and you're kind of like no I don't think that's really gonna move the what you want to do is rag you say no no point gbt 4 at your documents that that's how you leverage what you have but it there was a sort of marketing of sort of like will make your own model just for your firm which I think frankly there there was no evidence that that was actually going to have a better outcome but it just fit to sort of the ego of the firm right and their kind of natural desire to want to have a Competitive Edge so that was one thing we sort of encountered early on I think now firms are kind of coming around to let's just use rag right let's just point it at our stuff and how best do we leverage what we have as opposed to I want to go spend a bun of money to create a new model that's everything gbd foresaw plus you know our relatively poultry amount content so what happens when somebody brings their own data to the table like I you know I created my account I've dropped in there it's like okay you can create databases there's six different kinds of databases that you can create I assume that those are like pretty similar core underlying technology but perhaps with some different kind of processing or prompts Nathan because you know listeners your podcast deserve the unver truth I think they might be absolutely identical I think those six options were for education purposes to kind of tell you right like these are things you can do I don't think we're we're switching up the edings maybe for one of them we might have something specializ but but yes to your point working with laot firms security becomes just the number you know the immediate thing that you're always talking about before you know you can be showing somebody the most a time machine and their first question is going to be about their client's privacy right and and you know well it should right this is important stuff so you know we you know our philosophy has just been obviously it does go through gbd4 right so it does leave but then you can have it immediately deleted or if you want to keep it on our system with our embeddings you have that rights but you can have it deleted like every day um you know we went to the sock two compliance you know all of the the various rigorous security stuff you have to go through and of course we don't train any models with your data and there was this little window where that was a differentiator between us and open AI right because it sort of like why don't I just go to open Ai and for this little period it was at least ambiguous whether open AI was training on the data they naturally came to the same place that everyone working with Enterprise comes to which is no we won't unless you know you want us to basically right so I I think that's your point is sort of how do you do with the security and privacy is that the the main yeah that's that's interesting I was actually even more thinking about just kind of the technology like it sounds like you're taking these documents passing them through gp4 to be chunked perhaps there's also like metadata being extracted and then I'm I'm wondering kind of everybody's looking for tips on Vector databases or like a on the future of vector databases so I'd be curious you know which ones you're using if it's a hybrid structure if there's like a graph that's being synthesized all these sorts of you know very practical Lessons Learned I think are great right so I'd say that the we have domain specific chunking men you on our end before so we sort of go there I don't believe that we're we're right now throwing in the latest Vector databases on it so I'm actually not a great person to ask for that kind of stuff because we've just sort of been sitting with our homegrown common law specific one which at least last I checked has been for us more performant than you know better accuracy than than the ones that we've tried you know it's interesting because we were doing this stuff early before gbd4 we kind of set up a bunch of stuff that we're still kind of sticking with right in terms of those embeddings now that could be something where we need to examine that and say no no like just you're doing it early but now it's obsolete but that's just something we're you know testing constantly and so far we haven't seen a real reason to switch we certainly have the case law citation graph going back to like our traditional research tool right that when you're reading a case you have to see who CES to it and that's the whole thing we're not doing anything in the graphs with with gbg4 specifically right we're not leveraging it that way I've always got a handful of different apps that I'm kind of thinking about this with one other one just for context is I'm working as an AI adviser to a company called Athena we're in the executive assistant business and wait a minute I think I use Athena for my executive assistant yeah so this is you know early early days for us as well but of course you know we want to be more efficient we want to bring AI into more things that we're doing one of the experiments that we've been working on is can we create some sort of retrieval augmented chat experience that allows the assistant to get kind of deep context on the client stuff that maybe they've never even talked about before or you know certainly we don't want to be asking the same questions repeatedly if we can avoid that uh this is especially important in the early you know days of relationship and a challenge that we've had in Tred to build something like that is okay you get all this information right and people can upload anything and then we chunk it and you know maybe we could be doing more just to like maybe use gbd4 to do the chunking a little more intelligently than our approach that is straight away could be an improvement but I think a lot too about like if I match on this chunk but where did this chunk come from you know in kind of your naive chunking strategies that stuff A lot of times gets lost and you know also what's the timestamp on that document and you know if you go pure de Vector database like sometimes you do you don't necessarily have all those features you could go to like a postgress you know which certainly is not a native Vector database but is adding and so I'm kind of wrestling with all that sort of stuff like how much pre-processing to do how much structure how much synthetic metadata such a great area so I yeah I'm I'm convinced so right now we're throwing Co counsel at the text and chunking is the basically the only favor we do for it right and the way I think about this is like if you were an assistant at a law firm you just started and they said the night before your first day of work you can go into the office and open all the files what would you do to make your life easier and it's things that you might create a cheat sheet these are the contact information for the attorneys and the current pending legation I'm gonna put that on this I'm gonna create a new document that's just for me to be able to more easily pull that right that's what humans do that's what a roll Index right I think there's a lot there what you where you're looking at which is like can you land gbd4 in a new sort of information environment where at least it knows the general area okay I'm for a lawyer and can it go and like rename files according to like what makes it more efficient like you know look at the first page of each document to then create a new layer that then it can interact with more quickly I think there's a lot there I think there's going to be a lot of like sort of the art of doing this really well and I hearing what you're saying I think you're right completely makes sense for an assistant and and we're doing some experiments with it right where we're you know we're doing things you could like type in the docket which is just the number unique identifier for litigation we pull the docket and now we're starting to parse out a bunch of information even before you run a skill so that when you do run a skill when you say send a letter to IBM's Council it can then go and kind of do a more robust job of that than it would if not it sounds like you are throwing gp4 at kind of every problem as the first approach which is something I've often recommended like don't get too cute too quick see if gp4 can do it you know try to make that work before you really before you try anything else and I think one reason a lot of people don't do that is that they worry that it's going to become too expensive and you know perhaps you know just not viable so I thought actually the co-consul pricing was one of the most interesting aspects that I encountered in the in my exploration of the project it there's a couple different prices you can kind of clarify but basically it comes in at something like 200 bucks a month which is you know obviously 10x your retail chat GPT subscription and you know people may blck at that a little bit but I think that's really smart I've kind of been thinking more generally why don't people just build the very best thing that they can build with the very best models that they can possibly access and charge whatever it takes to make that you know at viable enough that you're not like outright burning money and then you know we've seen like a pretty clear Trend in the models getting cheaper so presumably you know some margin will kind of come back for you I guess you know first of all is that the strategy and you know how has the how have you been thinking about this this question of pricing and and how people responded to it first I mean look we were so spoiled right you get TBT for very early for free without to you know we weren't charged you know and we were just swimming and you just I just love it you just you know it's it's the equivalent of somebody growing up like I guess like a Saudi Prince equ you know I mean like we're just you don't even have a concept of it all you're doing is just the joy of how's a physic mod and you know the the guidance we got from the folks opening eye too is that look first build the best and then optimize right and so that has generally been our approach but then of course the bigger situation is it has to work and it has to work for Law and has to work reliably and a lot of the times when we tried to test other models it failed there and so there you go it's not it's not like it makes it easy in a way right so you know the truth is things are getting things are catching up but for the majority of the last year and Nick tell me if you disagree gbd4 has been like kind of a a big leap ahead of even of what was in second place and tell me three and for us that leap meant the difference between we can put this in lawyer hands and we can't right you know with confidence with reliability now are there subtasks within some of the flows that you could Outsource to you know to delegate to 35 you know and and a lot of our machine learning folks would be like like they're very annoyed with me for how like gbd4 am and they're like prob like you got to start thinking about these other models but for us again it's like it has to work it has to work reliably and so that does often mean you know for us for the majority of tasks gbd4 does it the other ones we try don't end of discussion right again we're we're as it evolves we're constantly looking at this things are scaling up you know so you increasingly you're going to want to look at how you where you can but but yeah basically law itself demands that we always focus on quality and to date by and large that has meant gbt for or or nothing frankly so how have people responded to that pricing I mean I can imagine you know on the one hand people might feel like that's more expensive than other software products I feel confused about maybe the future of legal billing because I'm like if there's one thing that this tool is supposed to do it's supposed to save you time but if you're charging for your time you know how does that work out right like in any other business that I'm in I would say well gez 200 bucks a month if it saves me one hour it allows me to Bill one hour more then great but like wait a second I just sort of those are not like the same thing right in the the legal space traditionally so the the market has responded I say the market for me it's always the profession the legal profession you know again you got to look at like there's a difference between your huge multinational firms right and your you know solo practitioners but the 200 a month pricing generally is kind of what you you know for solos for smaller groups they kind of just want to buy it and I think a lot of it it's like if they think of it in terms of a legal research a legal technology tool they're like wow that is pricey if they start to say to themselves wait I would have to hire a pargal to do that suddenly it becomes quite a bargain right so I think part of why I think the price has been well received is that they're they're getting that these are these are things that aren't it's beyond just having Thompson ruers or Alexis right or like a you know document management tool it's really getting into actually like tasks that you would have to hire a human and pay a lot for we're always trying to make it more affordable and you know I I read that in the early days of electricity only Wall Street and Madison Avenue got Street lamps and to some extent right now there is probably that going on right like especially if you want to do like High throughput stuff these large firms that you know have much deeper pockets and the litigations are such that like you know it makes sense to go spend more money given the risk of losing the case they're right now able to do things with gbt 4 that are probably impractical for your everyday attorney representing you know you or I right right but like you said it's getting better and we continue to try to design prompts in a way that you know lowers the right intensity so that you can you know bring down the costs and things like that but on the whole we were a bit worried too because yeah that is a lot for you know so we seen them Buck at lower numbers when it came to traditional research tools but here on the whole I think they just realize that this is so much more than just once you start thinking of it almost like this weird colleague and not a tool then you know it's been pretty positive so one of my big theories in AI in general is that we'll see a lot of consumer surplus meaning you know in the classic economic definition that you know willingness to pay will greatly exceed the actual price is that basically the trend that you think you are are enabling for your customers clients like are they just getting more for the same bill or they you know their bills are shrinking perhaps because it's just taking less time to do the job right I mean you know it's funny when they with pricing so like I mean Jake Laur and I like we were lawyers by training you know we love legal Tech legal informatics pricing expertise was not necessarily like what we were the best and you know we've joined Thompson Reuters which presents a much more mature sophisticated you know understanding of how you go and test it measure these things if that makes sense so you know I think I think there a couple things first there's going to be competition and well there should be right that this is one of the benefits it's not like only one company has this and therefore can just charge whatever they want right so that I think keeps you kind of honest on it right that said you know you you Tom Reuters you know we trust right you trust us right when your things matter Tom roers if you go with and that that does involve more testing that does involve perhaps more expensive models sometimes and things like that and that will be reflected in in in the price so I think you know first I would defer to like there's probably like literally 50 people at Del are more qualified to talk about how they're going about thinking about the pricing but but generally just that sort of tension between of course there's competition but at the same time like there's a caliber of product that that people expect from us and that people meute from us and then that will always be reflected in the price how about just kind of on how you measure the reliability the obviously a huge you know theme of of your comments has been the critical nature of reliability I think you said you know di Minimus hallucination a little bit earlier do you have kind of metrics you know that you watch or like a a bar you know as long as we're below that hurdle we're good could you be confident enough to say use our product and you know you'll never have one of those embarrassing moments you'll never I mean you can't say that about humans right you can't say that about hiring a human and you'll never have right and I think what's important too is that it's not just these like cartoonish hallucination problems which we avoid not just from the rag architecture but we have the cases beneath with links right and we police it right so you're if you see that leag like that's a real it can't not be a real case right so we are able to sort of give certainty under that stuff but even Beyond this like hallucinate I mean those kind of grotesque hallucinations if you will there's like it's not infallible sometimes it can misread what a judge is writing or misunderstand kind of what the user intent is and so you know that's that's a level of that that's really where we're I think the most Focus right is how can we get better and better there we you know I was always adverse to having the thumbs up thumbs down back when we were just a search engine because I was like we're supposed to be the professionals like imagine if your doctor was like hey Nathan I recommend this prescription hey how' I do what do you think buddy you know it's like you're the pro doing you should right boy did I get over that when it came to Genera you know to llm stuff right so we have like an ability to give real-time feedback and then we just called the followup we have a weekly meeting where we do nothing but talk about any complaints that came up and then we have our sort of systematic quality control constantly you know running tests Etc it's you you're never perfect you know it's going to be something where you're continually striving and striving and striving to get better and better but I you know I'm confident to say that nobody does more than we do to in law to make sure that that it's working for somebody like me so I I have a you know a couple different Ventures that I'm involved with at weark we do not have an in-house Council and we don't really fortunately we don't have to often Avail ourselves of external counsil either I guess I don't even know if you would sell this product to non lawyers it's just straight not available like you have to be licensed to even become a customer so I mean yeah we made the early decision to not go to what called Pro and la which is it's a little bit different than corporations that don't have gc's because it is so it's so misleading because you think look I've got this memo I'm done look it's just really just like a lawyer yeah know a lawyer could look at that and see that there's more Nuance that like there's certain exceptions that the lawyer knows about so we thought would be dangerous now to be clear one of the most probably the most important thing these large language models are going to do big picture is provide services for folks that can't afford lawyers or it provide some form of services some form of help and some I think that then there's a colleague of mine at Stanford Margaret higen who's working in there I'd defer to her I mean go study her work she's fantastic so to be clear don't get me wrong like that is going to be one of the most really important ways that we use llms and I think as a society we should judged by how well do we solve that really critical problem from where we sit at case Tex we did not want to give folks the misleading interpretation like oh I have a lawyer it's called co-counsel and go in there and do it and so we you know and separately from that there's this unauthorized practice of law issue right where it's like are you even allowed to go report to be a lawyer but but frankly even be it wasn't fear of that reprisals from that so much as like a understanding that like we we didn't feel comfortable doing it so like you know look if you're company wanted to use council could you use it are there a lot of skills that are sort of universal that you could like you know you could put in a 100 transcripts from speeches or whatever you cared about yes you can use it but we're certainly not selling it being like you don't need a lawyer because you've got C- counsel you're all set it's just not it's just not there yet I've had a few moments where I've had interesting experiences along those lines where I'm like even and this is even just for personal stuff you know I I get a a contract you know I'm going to do some 1099 work and you know I had one from a big Tech company that has a reputation for having you know a lot of kind of very restrictive Clauses in their contracts or at least that's kind of the sense that I had so I just take them to Claude and to chat GPT and say hey anything jump out of you for the from this contract if neither one Flags anything that seems meaningful to me then I'll usually just kind of say okay that's good enough let's roll with it but I imagine I could do better with a co-counsel seat but I also understand why you would be reluctant to put yourself out there in that way yeah I mean you know it SS like well what jurisdiction are you in right that can impact whether or some Provisions aren't enforcable and some states some right so it's just it gets more Nuance right than just what you're going to get from a chatbot or even frankly from a more sophisticated co-council so what we build is a tool for lawyers it can make your your lawyer bill go down it can make your attorney like you know help your attorney not Overlook something but it's just not a replacement not yet now there are many folks who believe that's the goal goal and that's where we're going to get and you know we we just there's just such debate about how far these llms will go as we scale them up and much much much brighter Minds than mine are debating you know like where does this Plateau so I can't rule out that you know the further generations of this will but but at least for right now we're not there we're not close to there so yeah that's maybe a perfect transition to kind of the future of law section that I thought we might spend a little time on as we get toward the end for starters yeah I sometimes call that the hundred trillion dollar question where will the the models Plateau that's you know inspired by the size of the global economy because it's like could very easily you know do the whole 100 trillion you know with not too many more leaps what are the things that you would say with gp4 are still the weaknesses you mentioned one is like missing things in the middle of context I also meant to ask if you were using function calling at this point or if you're still kind of kind of custom doing your own you know implementation of function calling no I think we're increasingly folding in function calling yeah I think it's like poting some over and then but yeah again all their great stuff we want to Leverage is just we have a slower process because we have to make sure it's not you know screwing anything else up but yes so you know look where it is now it lacks sort of the ability to understand a broader context of what it's working on right which can sometimes you know lead to it that's it can't give you like legal strategic advice right because because it doesn't really understand kind of where these things play in I don't think it can write persuasively enough to really write briefs and but let me car about that I have a bias I people some accuse me of having like a romanticized view of writing but to me writing is thinking like you as you write you think and actually Paul Graham the great YC founder has has written I think omn this of I completely agree with him that like it's very dangerous to let the writing atrophy right we we can't spell anymore because the red squiggly line will bring us home nobody can read a map because why would you when you have GPS but I think writing is qualitatively different like and so wrestling with that blank page I think is something that we actually should jealously guard at least for the types of writing that really involves substance look there are certain types of legal documents nobody car declarations they're supporting documents sure of course but for actual advocacy and the same thing for judges we recently had a a a judge sort of rais the question of like well could this start writing first drafts of opinions I think that's very dangerous I think we lose something about the evolution of law you know like the sort of the way that it it mutates and then changes because of this originality going into it you know other folks would disagree so I think that's going to be one of the most important things in the future of laws like what what happens to that right like do you know are we letting AI kind of just generate a brief and then that just you know essentially with some oversight goes to the lawyer and then the judge is using the two briefs and puts it into AI oh here comes the opinion and there's some oversight that that's one of the profound questions I think you'll start to see more like kind of AI arbitration right but you know the truth is there's a guy in England Sir Richard suskin who was like the sort of OG AI in-law guy like you know going by like 30 years right and he says you know whenever there's a new medicine that comes out the doctors only talk about the patient what what does this mean for patients you don't hear them what does this mean for our billables like what does you know what I mean like and sort of exhorting us and law to have that same you know professionalism because ultimately this is a profession we owe a zealous undivided duty to our client and if that means that we have less money at the end that's actually completely fine relative to our duties right like it's it's not in the and so I think and more serious lawyers among us are going to say like how can we you know better realize the goals that we all set up for ourselves with this stuff and the truth is we're wul behind on this so there's so much room for improvement that I think we'll see AI get a lot of adoption and start to just make process faster less expensive and again if use correctly even increasing the kind of quality of the pursuit of Justice yeah I appreciate that and I I think broadly I really agree with you bringing up the bottom and improving access to expertise for people that don't have it all too often is to me one of the most exciting things about the AI moment and at the same time I think it is at least for now really important that we do not seed control of the frontier and you know where kind of critical aspects of society are going so I would strongly agree you know that like it seems I I guess I was very specifically I was going to ask do you think that there you kind of touched on it a little bit but from the medical profession as you noted I've been very impressed with how positive the reaction to AI has been and how kind of non-defensive it has been from most Corners I would say the same thing broadly true of the legal profession although I don't maybe have as much data on that so I'm curious as to kind of how you see that shaping up like what reaction you do expect from the bar would we get to a moment in the near future where like you'd be guaranteed some right to like legal AI advice as well as you know human advice yeah that's so interesting so maybe I can contrast two essays both written by the same person who the Chief Justice Roberts right the Chief Justice the Supreme Court the end of the year he writes an essay about the topic you know whatever topic he wants and in 2016 he wrote about legal technology and you know I'm paraphrasing a bit I recommend you know go read the real thing as always but you know he sort of almost brags the right word but he sort of says like we're really slow and that's a great thing and he actually points out I don't think most people know that do you know that the tortoise in the hair is actually engraved on the Supreme Court building there's actually a turtle and a rabbit carved into that building right you know they got like Moses up top and then below you've got this Esa table and he sort of points to it and says yep that like you know that's that's the turtle and I always say like there's more options in life than a turtle and a narcoleptic rabbit that falls asleep five minutes before it's supposed to win the wrist like you know like let's so that's 2016 you would you I wouldn't say like anti- anti-tech but very much like talking about how like like we should be slow and nobody you know it's it's almost it's a feature not a bug how much we like 2023 he writes an essay about C of AI and llms and says things you know things that I've been very much screaming from the mountain tops he says the first rule of the federal rules of civil procedure these are the rules that kind of govern the protocols calls for the just speedy and inexpensive resolution of matters like surely AI can help us better realize that and by us he means the courts and you know he says at one point I think he says like legal research will be unimaginable without this now right very kind of pro very much like you know not not Reckless but and you know I think at the end he says judges will still have a job but certainly the kind of exactly as you said like pleasantly surprising in terms of being receptive and sort of like kind of acknowledging the fact that this could bring a lot of value and it's not just something to kind of be scared or contain to me that that that bodess very well and I think generally I've seen it in other parts of the profession I think there is a real willingness now to engage with this stuff and not just from a how do we destroy it or kneecap it but you know where are the ways that we can use it yeah you could almost put that excerpt right on your website I'm not sure how that would uh oh no I went t-shirt website I might you know I might even have the ink you know just tattoo that on no no it's it's it's wonderful to see that um I think Lori this is a time to Rejoice if you're a lawyer this is profoundly useful technology that will help us do our craft and our profession much in a a much better way a much more sane way frankly more enjoyable way I mean just like sort of and so you know I know there are certain billing structures that are built on hey that associate just for 12 hours just did basic Doc review and I was charging 600 an hour and only paying that associate 100 you know I appreciate that those sort of paradigms will probably have to be reworked but I think that that pales in comparison to the transformation that we're going to have with our profession to one that I think frankly is in dire need of that kind of transformation Yeah you mentioned also AI arbitration that's something I've been kind of fascinated with but surprisingly haven't seen I guess maybe not surprisingly because again the timelines are all really short so it's it's going to take time to work this stuff out but I wonder if you could sketch that Vision in a little more detail my conla professor was was a great Larry leig who you know has been done a lot in Cyber Law and things like that you know he was one of the very few people that I showed gbt for to under you know in the earliest days on it and I was expecting you know he's going to calm me down you know tell me to like you know put some breaks on this thing let's really and he immed was like for 35 years I've been thinking about what we could do for like an AI judge like this is I was like Professor lesing that's like way wait oh that's you I guess I shouldn't be shocked that he had even sort of more ambition for it so this is something I think predates large language models this attempt how do we code in Ru making and adjudication right in sort of legal logic and it was all just you know like a lot of these expert systems it was just hopeless like right like laws just so fuzzy and messy Etc but now with the Advent of gb4 and the sort of capability the idea you know both sides consent submit information and what you've done on the background either through Rag and through a lot of trump thing basically encode dispute resolution such that it can spit out an answer and then you know both sides have agreed then I guess that's the answer right if not maybe there's some right to appeal to a human and and things like that again that to me is is is I mean if you take like one very specific type of dispute maybe that has very you know maybe you could kind of get there I I think it's among the more ambitious things that people are going to be are trying to do with gbd4 and if it I'll be honest every time I I think oh gp4 is not gonna do that somehow it surprises me but that's certainly not an area I think you're going to see again like you said it's very early days but folks who have actually been thinking about this a lot even before the llms I think feel like this whole renewed sense of it about what they can do now yeah certainly one of the big lessons of the last few months is that we keep finding new ways to Advance the capability Frontier even just of the the current system so that has been remarkable to continue to watch I guess maybe last question is around the law of AI I wonder if you have any takes on kind of you know should we have standard product liability for AI products how should we think about you know who is responsible when AI contributes to some harm also interested in your thoughts on any if any on things like right to train you know should should all the data that's out there be you know is it fair use in your mind and you know or should there be some sort of profit sharing all right so for fair use what I do is send your listeners to an article co-authored by Mark Lemley who's a professor at Stanford along with I think prely young like with with with actual researchers at Stanford Deep dive on the fair use issue he's infinitely qualified to talk about it than I am you know so and then you know I think the truth is like where I've kind of intersected with stuff like so courts are I've given import to courts and to State bars that are wondering like do we need new rules to govern this right or can we just use our existing rules I'm actually of a mind at least for that aspect of things that the existing rules if properly applied get you 99% of the way there right there's a duty of technological competence that's now in Most states right there's a you know the duty of of cander to the Court the duty you know like not reading the case before you s it to the judge like you didn't need or any of that to tell a lawyer like that's not going to that's not good you know being a proper lawyer so I mean really the the area that I've I've focused has been sort of myopic really just regulation of the bar Etc I look forward to taking a deep breath and diving into these really fantastic Rich issues right antitrust and the you know right to all of these things and certainly Thompson Reuters because we're International right so we just you know launched in the you know Australia Canada you know we're increasingly like facing not just one but multiple different regimes on this so you know what you might want to do is have my colleague Laura on who's our who's our GC at casex she's one of our co-founders and is now I think in global if there something wonderful like that who might dive into the policy stuff a bit more it will keep lawyers busy for a very very long time and actually the the folks at D Piper would be a love firm that is particularly focused on you know making being leaders in this space across a lot of areas they were very with co-counsel they basically helped us co-develop it but at the same time you know then they they finished helping us with the product then they go down the hall and actually represent AI companies that are facing these so let me defer to better folks than myself and I'm happy afterwards offline I can I can introduce you to some folks cool sounds good I'll I'll appreciate any connections and I certainly appreciate some intellectual modesty as as well it's not all about the hot takes I guess I said last question but if I can sneak in one more what's it been like being a acquired by a obviously much bigger much older company you know obviously that's a highly individualized experience but I'm particularly curious about how the acquiring company is thinking about AI like do you have people coming to you from everywhere saying hey you're the AI guy I need your help in my department or are they still kind of waking up to it being a startup guy and never having been you know acquired by a big company I don't have that story to tell myself right yeah well one day you know maybe you will maybe you know maybe I'll just go to the the public markets either way you know Thompson Reuters the reason we were acquired is because they were awake to how important this stuff is and you know although they're a very long and stored company you might think that they would be dragging their feet you know and kind of just relying on their legacy stuff it's been quite the opposite Steve Asar the COO you know just announced look we want to bring this to Bear across all of our verticals in in a very deep way and so and this might be unlike other Acquisitions maybe where you're kind of brought on kind of siloed and your just your thing we're brought on to kind of join forces with what they're already doing which is to just how do we create as much value as possible for our clients and our customers across all of the things that we're doing and so that's been very invigorating right I mean it's it's perhaps unusual and I don't want to tell everyone oh every acquisition will feel like this but in this instance we've been just really encouraged because we're coming we don't feel like oh they don't get it or they're not doing AI enough you know they they call it I think their build by partner approach right like where they're already spending a ton of money doing their own stuff in house they're looking for the right Partnerships and then they're they're making strategic Acquisitions like case text and then on a you know on personal note like you know law there's a lot of content with law and Thompson roer has the most like beautiful content and so I like in it like it's like you've been kind of building with like sticks and mud and then suddenly you have like the Roman Empire with all of its like you know Phoenician dyes and Greek marble and like all these things and so you know a lot of the things that that Jake and I really wanted to build were frankly limited because we didn't have X content and you know and xqu content with tons and roers you really have these just amazing streams of content dockets or law review articles or practical law so that's also just feels great because now you can you know basically this point there's no excuses right whatever we know it's on us because we really now have all the pieces both from the content the support of the larger organization and fantastic colleagues I mean t some really great folks that have been putting shoulder to building to tools for lawyers since you know long before kex before Jake and I were even you know part of the world so so it's been really encouraging so far cool I love it anything else you want to touch on that we haven't got to I'll just if you're out there and you're building llms I consider you an absolute hero of justice and I just hope that you know you appreciate the downstream from your efforts we're putting this to work to really make the world a better place and I just I I applaud you guys I know it it's not easy to do to to push these models forward but just know that you are really truly creating a value at least from My Little Neck of the Woods nothing has been more important and more valuable and we will strive to to really deserve it and to keep you know making the most of it I love it that's a great note to end on Pablo Arondo co-founder of case text now VP of co-council at Thompson Reuters thank you for being part of the cognitive Revolution thank you so much Nathan it is both energizing and enlightening to hear why people listen and learn what they value about the show so please don't hesitate to reach out via email at TCR turpentine doco or you can DM me on the social media platform of your choice omnik uses generative AI to enable you to launch hundreds of thousands of ad iterations that actually work customized across all platforms with a click of a button I believe in omnik so much that I invested in it and I recommend you use it to use Cog rev to get a 10% discount turpentine is a network of podcasts newsletters and more covering Tech business and culture all from the perspective of Industry insiders and experts we're the network behind the show you're listening to right now at turpentine we're building the First Media outlet for tech people by tech people we have a slate of hit shows across a range of topics in industries from AI with cognitive Revolution econ 102 with Noah Smith our other shows drive the conversation in Tech with the most interesting thinkers Founders and investors like Moment of Zen and my show Upstream we're looking for industry-leading hosts and shows along with sponsors if you think that might be you or your company email me at Eric turpentine doco that's e r i k turpentine doco