we have our first speaker up is Alex Garnett He is a senior curriculum developer at Temporal Technologies And in a past life he is a digital archivist So go ask him about that But definitely don't ask him about the Mets I'm a Phillies fan So who said woo woo Um but we don't hold it against him and we can be friends anyway Take it away Alex Thank you Jody All right let's go Mets Thanks everyone for being here this morning Uh I'm going to tell you why Doc's AI tooling is better and better for us than you probably think Uh a little about me first because this is very relevant to the talk Uh I'm a former academic librarian I've got a burning passion to site sources very important for AI It's in my soul Uh former Digital Ocean writer and editor so I like to give context and give people nice learning journeys Uh former write the doc speaker Thank you for having me back again I love this conference You're all wonderful And I'm a very emotional man I'm very prone to emotional outbursts especially about AI This will also come up Okay Uh I currently work for a company called Temporal We are hiring a couple of writers very soon by the way I don't want to fail to mention that So if you're looking come find me or my colleague Brian later on in the week Uh Temporal is a durable execution platform What's that mean it means we help you do a lot of tricky scaling stuff like failover reliability replication roll back for free We're open source It's a cool product Take a look at it later on if you're interested And I work on Temporal's courses and documentation So I help to teach the courses develop the courses and I get the course content back into the docs This is one of our two Docosaurus sites for our uh learning content We've also got one for our doc site which you'll see in a bit Okay Current state of AI in docs It's a bit of an understatement to say we're all being encouraged to use it We are like so far into the no one ever got fired for buying IBM stage that we're all just being crushed every day by giant mainframes Like more IBM Yeah you like it It's exhausting You cannot open LinkedIn without being smothered by like do this with AI do that with AI right this this is exhausting Even if you're trying to like it you're exhausted because there's just no end to it People just really want to see it used more right now It's very good for small discreet technical tasks Nobody would deny that If you want it to make you a regular expression or convert some markdown tables do a JSON query string god forbid a Helm chart it can do that That is not contentious And those of us who know what we're looking at so much the better because that way you're not trusting that it works without actually having that context Many most commercial AIs are still very prone to hallucination It happens to me very often that if a colleague went and got something from chat GPT or claude and it's citing some plugin that doesn't exist and I have to be like where did this come from come on come on So this happens right we're going to get back to that too And if you are a writer this probably gives you to be fair mixed to negative feelings Even if you are trying to have positive feelings like it's it's not great right it does kind of wear you down Even if you're trying to keep an open mind I don't know how you all cope with having negative feelings I get pretentious Okay that's right Yeah Yeah 8:30 a.m Buckle up Here we go [Laughter] Um this is a poem by an Irish poet William Butler Yates who lived about 100 years ago Uh whether or not you're into modernism you may recognize some parts of this poem It's very widely quoted And uh I believe he wrote it while his wife was very sick and possibly dying And the theme of the poem is there's no way we can possibly go on like this any longer right like something's got to change We're all dying This is horrible I feel bad Why will no one acknowledge how disastrous everything feels right now sound familiar uh and the beginning of the second stanza Surely some revelation is at hand Surely the second coming is at hand Okay we're all feeling it Something's got to give Something's going to break Woo apocalyptic That's right Uh the reason I said you might recognize this poem even if you are not a farm fan of poetry is because there is a great and durable literary tradition of ripping this poem off So for example you've got Joan Diddy and there she is folks uh she had an essay collection slouching towards Bethlehem Chinua landmark work of African literature things fall apart We love to rip off Yates in this room Yes we do So ready for it yeah there we go [Laughter] Uh the sensor cannot hold Okay you've probably heard that before Uh pretty similar to how we're all maybe feeling about AI right now right it's very difficult even if you're highly motivated to articulate a coherent position between I hate this or you must use it right it's the future and you're suffering Okay not not a great juiposition but it's really really hard to find that like center track that actually usefully describes your work and what you want to do with it Uh this was a very kismmet photo opportunity I recently had Um the photo on the left was taken by me I was at Google Cloud Next in Vegas uh last month working a booth and I saw a guy walking around with this t-shirt on and then I opened up my phone like 5 minutes later and saw the hat on the right tweeted on blue sky Uh so you know like we're all we're all dealing with it I think there's a center position somewhere in the middle there It seems to be greenish You can't really see it You know it's obscured by the two competing images We all have a lot of feelings right now Uh and again if you go on LinkedIn for 5 seconds just an endless stream of like "What am I doing with AI why haven't you fired all your writers the future's going to eat you." And I'm sure all of us on all of our teams are attempting to have these conversations in good faith We really are I don't doubt that for a second And this is not meant to call anybody out because we are good at doing the work and we want to do the work and we want to do it usefully Most of us are not scared of new technology We are all trying very very hard to figure out what constructive engagement looks like and that's just not easy under the present circumstances when the boosterism and the doomerism are just kind of colliding endlessly You say you want to do your regular old SEO function H can't do that anymore Sorry It's all AI now right like all the search results are buried God forbid you wanted more than one result Nope You're getting one summary and it made up a holiday that didn't exist Cool Okay Well you know what forget about SEO We're not doing that anymore I guess Uh okay Can we get the Falcon a hearing test can the falconer shout i like what do we need here to make Maybe we can figure out something Come on We can work Oh god Okay you know what never mind That's not good either Geez Um Ah okay Just just take a beat you know like we're going to get through this together right all right Yeah Uh this talk is not going to be about using AI to write the docs because I don't think that's the most interesting use case for it right now even in the context of docs You can do this if you want to Uh you can use it to fill a blank page You can use it to rephrase stuff I've seen some pretty cool products for copy editing that seem okay I don't think this is the most salient point to make here uh using AI to write the docs is not in practice that far off from using veil rules or any other kind of style linting It can enforce that It can help copy edit some input you're already feeding it We know that and I'm I don't really want to litigate it because again I don't think that's the most interesting case here What part of your job do you actually like the best you should ask yourself that especially at a time like this What is it you want to be doing and what is it that you want to save time on like really think about it because I don't think any of us think about it as often as we should And that's what you should be focusing on Protect that part of your work that you like and do best and then figure out how to save time on the other stuff It sounds simple but if you don't take that beat you're probably not going to be framing it exactly that way And this is a really important time to be doing that I think a lot about AI in the context of this piece that was written on the great sports blog defector around the time of the Hollywood strikes a couple of years ago Uh the computers are coming for the wrong jobs right what do we actually want to be automating if you want to automate the creation and testing of that Helm chart or that reax that's fine I actually happen to like writing reax That's a personal problem right most other people don't There's lots of aspects of your work that you probably don't want to automate that you can beat automation at still and that's not likely to change but unless you're acting critically you're not going to get there Reactivity in particular I think is the worst part of docs writing Some of the worst docs roles I've had are ones where I am constantly in a reactive position just trying to get coverage and constantly keep up with the rest of the organization This is usually a trap that a lot of solo writers or solo editors fall into Uh you don't want to be doing that if you can help it If you have no time to think about UX you're just trying to get features documented you're probably not having a great time doing your work And many folks I've worked with in this capacity try to find time to work on other projects really really hard because they want to be doing that intellectually satisfying component of their work If AI can help you here go for it right it's one of those things where like okay maybe you have like some feature documentation you've got an SDK reference manual you need to publish something you want to edit it later that's okay I'm not saying don't do that right you don't want to find yourself in a constantly reactive posture That seems like a great time to use AI But for now let's talk about retrieval augmented generation This is an AI term You may or may not have heard of it already And what this refers to is pointing a pre-trained AI model toward a different corpus So Nvidia Chat GPT whomever are spending the hundreds and thousands of GPU hours on actually putting together you know GPT40 mini or what have you That's all built on a corpus that they stole from a bunch of books and Reddit You know that stuff is all in there whatever that that stuff's already stolen folks Uh but then you want to be able to point it towards your corpus or some other targeted corpus so that it can give results that are informed by a specific set of materials Training requires massive massive massive scale that's being done by very very large actors But this is a last step that requires fewer resources to do and can be done by companies that don't have a huge AI function This is what a lot of those Salesforce AI commercials are selling If you've seen the Matthew McConna one where he's like "Don't use that AI use my AI." That's that's basically a really bad Matthew McConna impression Uh but that's essentially what they're selling here They're selling uh risk retrieval augmented generation for your org for your docs for your internet whatever And there are some pretty good docs and support vendors in this space that are themselves startups Uh Inkeep and Kappa are two that I'm aware of and we are a customer of Kappa and I want to show you some of how Kappa works for us because I think it works really really well This is the Kappa AI homepage Their pitch is turn your docs into a reliable LLM powered chatbot Chatbots are really popular right now Chatbots are not a bad interaction model And with Kappa you can embed a ragtrained chatbot into your own documentation and that can help generate results for users of your documentation Here are our docs This is a docu site Like I mentioned this is our docs homepage In the top right corner there is a search box people can use to do regular Alolia search So the kind of doc search that we're all familiar with going back quite a while In the bottom right we have that ask AI button that gets you to the Kappa interface And if you ask Kappa a question like "How can I handle queries in Python without AQ?" Kappa will walk you through some parts of our documentation based on sources it assembled and then give you a nice code sample for doing just that I happen to think it works really really well And it works really really well because Kappa follows some core principles that are also very near and dear to me Here is their dashboard You can see a lot of metrics right out of the gate for what Kappa is doing for your users They've got total questions answered in a month about 14,000 Pretty good They mark how many were uncertain We're going to come back to this I think it's a really useful principle Uh Kappa lets you know when its AI isn't sure about the answer it's giving and it will flag that when it's conversing with users Isn't that amazing admitting you don't know something I really love like the drunk uncle model of AI interaction I I think that's fun you know like Chad GPT being a wizard in Canada but like that's not that's not really always as as useful as it could be Um this docs AI that we have at tempora was implemented by a former support engineer I think this is a really really really salient point Uh I work mostly in docs So on unmediated unmediated interactions reference content and teaching which is very highly mediated and linear support really occupies a third pillar I feel of that model and it's not something that I have ever done directly and I've been really gratified to see the support thinking around these chatbots Some of support's work is thoughtful and some of their work is repetitive often again as a former librarian it's about just pointing towards content that you know you have that you just have to improve the route toward getting there and those roots can be improved They're different for different users They're different for different people And you can always try to create more access points the same content more roots more learning pathways but there are always going to be more that you haven't thought of because someone will have a different use case And thinking about this in terms of support is a really useful way to think about making your content accessible in new ways This is what the computer is supposed to do This is what the computer is for The computer is not for like writing a sentence that doesn't have as much fun in it as you could The computer is for helping to drive new roots Also and I should mention the business model for commercial open source is historically tricky We're a commercial open source product uh you can download and use temporal open source You can host it yourself It's really great and we make revenue by people who want to host with us and it's really challenging to offer support to people who do not want to host with us because usually that math doesn't math So if you want to get support that's usually tied to hosting That's relatively common in commercial open source And so for people who are not commercially viable to support because they don't want to go all the way to a hosting contract this is a really nice way to meet them where they are Another point I want to make is that support and docs and community almost always already want to work together at most companies This is virtually always the case We're always like "How do we do more with community how do we get community content in here how do we make sure we're all using the same metrics and sharing data?" We want to get community web generated content into our docs This is web 2.0 101 That's a lot of numbers but they make sense in in a series Uh we've been doing this for a long time now AI can be really really good at this at synthesizing a few different corpora together This is something it does very effectively If you look at the Kappa results uh actually no sorry this is the Kappa sources page And one of my favorite things about the way Kappa sources work is they have different custom scrapers that you can use for different types of content to get them all into your rag model that your chatbot can then access And so we have for example our discourse forum being scraped our Slack being scraped our docs being scraped our courses being scraped our GitHub comments being scraped all these ways that we're already out interacting with community members who might have novel questions all these different kinds of content that are partially authoritative and partially userenerated and usually have some kind of an interactive back and forth model We can get all of that into Kappa and when it answers questions it will use them all It will draw on all those sources and because Kappa is very very very robustly citationdriven it can tell us exactly where its answers are coming from in those corpora And I think this ratio that it has this is not something that we dialed in This is a ratio that it grows organically from the information that it had to work with This looks pretty good to me You know it's answers are coming from 33% Docs 14% Slack uh 13% from our LMS with the courses homepage which includes our blog uh GitHub and then right on down the line Right this is a pretty good mix of content to be able to all be able to access in one place So I have based on this some personal hallmarks of good docs AI uh valid citations only and always You want to drive traffic back to the source that it is coming from This is a very very very obvious bug bear for a lot of folks We don't want to take traffic away We don't want to remove context We don't want AI to have a less broad interaction surface than Google did right we want to make sure that we have at least as much ability for people to do the research and go back and read where the content came from This is actually really doable I think people think this is a hard problem because of how claude and chat GPT work If you think about it end to end you know like if your product is just your chatbot and you're not doing any other text processing of like where is this link coming from am I going to follow this link and validate it you haven't solved the problem But if Kappa can work on this and be like chatbot second step follow links Wow there you go You've just productized that in a much more meaningful way And it's shocking that the larger chat bots aren't facilitating this more I know Claude recently added the citation feature I don't think it's nearly as robust as what Kappa has Take an evaluation ccentric approach to your results Take a look at what's coming out right make sure you're actually seeing was this right enough of the time you saw that uncertain percentage was under 10% That's pretty good You can measure that Close the feedback loop is a really important one Let the AI go back and improve your docs in turn Don't just leave it as a layer that's running over top of your documentation That's not really going to help You're just going to be trusting that it's doing something that you're not feeling great about to begin with and maybe people are using it We don't know You should never have to simplify your docs for the AI's sake is another really really important point I've heard a lot of people get to like this sort of cargo cultish point of view where it's like if I take out all the tables the AI will work better It's like you're eating on the floor now buddy Come on Like you you need those Like it's just you you should not have to move away from your own best practices The AI can move towards you Uh and so if you look at that example kappa query from earlier at the very bottom we have three citations that it drew on to construct that code sample that it used to write that answer that the user can now refer back to And I think that is awesome and that should be a saqanon of any and all dots integrations A point I want to make really strongly at this point is that even in a field of like bad or just mediocre actors like even when you're being encouraged every day to like make an AI generated video where someone has six hands and they're like drinking a weird beer that disappears in every frame and they're like come to write the like there's still there there are people doing good work and I feel like it's really important just to remind yourself of that like we know what we're doing We we can be authoritative We can we can try we can make an effort We can call good things good Uh and if you find a vendor you trust whose work you like you really should support them because that's going to help actually lift all boats and make sure that we're all doing our best to establish a consensus of credibility This feels like an important point to make in light of the current political environment You know it's like things become true when enough people believe they are true right again should be obvious it's not always and I think the more we say like docs AI should do this this is an important thing for it to be doing that will be heard and I think it takes all of us to make that case over and over and over again exhaustingly so over and over and over again that's what it takes I'm sorry uh rejecting new technology out of hand doesn't go over like that's you're going to lose that fight it sucks but you just are you know like there's going to be a a brand new fancy car it's going to run you over you're going to be like car I always hated you and it's not you know that's not going to help um but that doesn't mean compromising on your work or your professionalism you know you have to have standards to be this difficult and argumentative and and that's what I tell myself every morning uh uh a cool point to make here is of course not everybody uses our own Kappa AI right and this gets brought up pretty often it's like okay doesn't matter how much you customize your search someone's going to use the other search product Fair enough Really good point right i think most people are still using Claude or ChatgPT or whatever and they're getting code samples from a lot of sources Kappa works with many different AI models If you talk to them they'll say "This is proprietary information We can't actually tell you which ones we're using at all times but we use all of them We monitor hugging face We monitor the new chat GPT 40 mini or whatever And we will rotate off which models we're using based on the results we're seeing in different contexts we do that evaluation This is a great thing for a vendor to do for you because it means we can use their analytics and when they're getting more accurate we can trust that in turn our content is becoming more useful to all these other AI bots people are using as well We can trust them and we can offload that support work to this vendor This means we don't constantly have to chase chat GPT revisions And I think having that kind of trust and that kind of working relationship is really useful This is the kind of thing we would all be doing ourselves otherwise and be doing not as good a job of it And in my opinion nothing is more important to docs than context Yes more than copy editing I'm sorry I also love to copy edit I love a really good sentence I made you all look at Yates at 8:30 in the morning Like I think my credentials are impeccable in that department But the context is really really important like where you're locating something how you're generalizing from something that people already know how you're helping them synthesize and locate that information in their existing body of knowledge in the state-of-the-art That is the most important thing docs can do If you help people make connections they can go out and they can make more connections on their own This is the most important thing that I write for all the time And if I don't trust an AI to provide good context I hate it I hate the AI I don't want to ever talk to it it's very bad If I do trust the AI to provide good context then I'm pretty happy with the product and I and I want to keep talking and I want to engage more and I want to see what it can do It's that simple And I think the number of AIs that don't meet this bar is very high And that's why we spend a lot of time being like AI But when you have one that works you should acknowledge that it works Uh a few more cool Kappa insights So here's some dashboards they have It's doing about 3,000 queries a day for us right now That's pretty good Uh you can see of those we have like five six 7% a day that are marked as uncertain which it will explicitly flag for a user like I'm not sure about this by the way you should do more research Uh it groups the questions together uh so you can see what kind of questions are coming in what people want to know more about what your users are searching for These are really useful analytics to have to begin with because they can teach you what you want to write more about where you want to take your docs in the future You can also look at individual conversations that the Kappa AI bot had and you can drill down to see all that interaction and see where people's mental models are currently and how Kappa is meeting them where they are And you can toggle on that uncertain tag so you can see only the conversations that Kappa wasn't sure about which are very usefully pointing you towards parts of your docs you should consider improving because that was context that it's lacking uh if you read an entire conversation which I do sometimes they're very very enlightening You can in turn take that information and go make a bunch of Jira tickets based on the content that was observably not in your docs for either a user or Kappa to find on their own One thing it often has trouble with and I think this is a really really great accessibility learning is if you don't explicitly say that something doesn't work it can't always infer that two things are not compatible We tend to write I think we're biased in this way or we're trying to just kind of write positively So we'll say you can use X with Y And people will inevitably come to us and be like can I use Y with Z and capital will be like I actually don't know They didn't say anything about that This is a really really good accessibility moment It's like oh we're just writing by implication by inclusion It would be way more helpful for us to come out and explicitly say X is not supported with Y in many cases And we see that recurring all the time throughout the cappa cororpus So I think it's a really cool point Uh and again I love that I got to make all these Jira tickets because this is real insight that I now get to leverage Uh and you know it's like oh the AI made more work for us Well okay you know like I I I guess I'm grateful you know but but but really I am because I fundamentally don't trust anything that doesn't take work And the fact that the AI can give me really robust insights that I can go and turn and I can apply and I can do real work with that's a good thing Sometimes the work is different or differently informed but it would really be a shame to waste that concrete insight and the opportunity to actually go and improve the doc substantively based on real analytic data on weird hype that we kind of have to every day still be like don't use AI for that use it for this don't use it for that don't no Yes it's a lot of that So conclusion AI will make you insane and take your job if you let it Fortunately this was also the plot of DSTski's The Double Uh in that case it was a 19th century civil servant who just kind of hallucinated the guy who was just him but better Uh so this is a 19th century problem It's not new Congratulations We've dealt with this before And there's better ways to use the opportunity right like you can take this you can do more with it uh and closing the loop like this using AI to generate insights to go back and improve things in turn to have a manual step Before the current era of chatbot hype there used to be a name for this This was supervised learning This was reinforcement learning right this is a known quantity I think we've all just forgotten because there's just so much coming at us right now AI is not only useful as a doc writing tool Look at the whole ecosystem I happen to think it's much more interesting in terms of user mediated interactions than it is for writing per se Integrating community content I'm going to say it I have not seen it done better in recent memory than Kappa does it It's really useful It's super interesting how well it can use those discourse threads And I just want to see this more and more and more and more I think it's super cool Filling the gaps where the AI is unsure is a big accessibility gain for us It means we weren't writing as clearly as we could have been And I think that's always useful feedback And you're probably going to need to have an AI solution if you don't already That car is coming But there are really good options And the more that we know what a good option looks like and we're willing to articulate it and go to bat for it that's going to be good for all of us So thank you very much [Applause] That was great Thank you Wonderful job I feel like we're all a little less afraid of the coming uh you know of the robot overlords after that So thank you Yeah I mean we're just where they want this though so that's not necessarily good Don't blame me I voted for Kodos Okay Thank you Um I'm just going to jump right in Um do you have a workflow of driving new doc efforts based on existing AI analytics of seeing user queries and then filling in the gaps or correcting docs around inaccurate or incomplete AI responses yeah that's a terrific question So right now we try to dedicate a few hours a week to going through those cataloges and saying okay what were people asking you know what got marked uncertain do we think we can fix this right like where was this coming from we'll sometimes actually take sources out of Kappa's corpus as well if we feel like it's not using our course content as effectively as our docs content and so we'll make tweaks here and tweaks tweaks we'll make tweaks here and there and we'll try to change things as they come in Cool Yeah Um so there's a number of questions about the uh ingestion of Slack into the model Um some of them around employee privacy or um confidential internal information Can you just talk a little bit more about how Slack works within the context of your chatbot we have two different Slacks We have an internal Slack and a community Slack And so we ingest the community Slack Do you have an internal chatbot and an external chatbot we do not have an internal chatbot that has the same data as this We do have a second Kappa set of sources that we use but not for docs Gotcha Thank you Um that was a separate side note but it's always interesting in delving into that because you know you you don't want that that proprietary information being leaked out into the public models Oh yeah And you just know I mean like everyone is having like a total OBSC disaster these days right we just don't know about it yet Yeah Yeah Um let's see Can you expand a bit on did I do feedback loop already uh can you expand a bit on closing the feedback loop and give an example of how you've done that yeah I mean that was the the Jira tickets I showed toward the end there right where I'll say like okay it seems like C doesn't seem to know that temporal is written in Go or Kappa doesn't know like that there's a plug-in model here or something like that And so we'll go and see like okay it turns out we really weren't making that explicit we'll have a list of CLI commands but we're not going to generalize those back to like how you actually deploy something And so it helps us realize where our own biases in writing docs for reference primarily And we try to get away from that where possible and then have places where we can state more explicitly like we do X we do Y we do Z So it helps you find the things where you're like too close to the content Exactly Cool Um can you talk a little bit more i know you touched on this but document accessibility and AI Yeah Uh so more routes to access your content is fundamentally a net good thing in my mind right the more you're giving people more ways more access points that's always a gain As long as the content itself remains authoritative getting people in there in different ways that's always a good thing And so accessibility can be a framing device It can be the way you actually write your content and the way you make things clear Uh I think the fact that we write by inclusion rather than by exclusion in hindsight is an accessibility issue because it means that people who are less inclined to infer based on their own knowledge and whatever they've come to the docs with are not going to be able to make those connections as effectively And so taking a broader approach making sure we're being very careful in our framing I think that's a great lesson Cool Yeah Uh let's see So did you have to do any kind of preparation of your docs for consumption by kappa what is the underlying format and this person is interested in the technical implementation process that you went through So I'm very very happy to say that we basically did no additional preparation whatsoever I think that's a great uh point to make Uh we already had markdown docysaurus So they're in like a fairly standard digestible format to begin with and docysaurus was a target for one of the platforms Kappa already knew how to scrape We have tweaked them You know we'll change the IIA here and there if we think it'll make you know some some differences but they should meet us where we are rather than vice versa I believe that pretty strongly Totally Um does changing the IIA actually have that big of an impact Yeah Okay Asking for a friend Yeah Let's see Can you share any thoughts on how best to implement DOC's AI in a highly regulated industry where wrong answers could have severely negative effects probably don't do it Like I mean it it it there you have it Yeah I don't want to get anyone killed I feel like it's probably best to say don't do it Like that's that's above my pay grade All right I have saved probably the one that you may have the most to say about for the last Um I we have like five minutes left but this could be a big one Uh in light of the World Con controversy and lawsuits do you have thoughts about ethical construction of LLMs or librarian insights into licensing oh jeez Oh boy Um I mean so there's an issue I think it's called a presumption of good faith in the legal community Like if you for example expect someone to not comply with a legal ruling for some reason Uh I don't know when anyone would do that No me either Um and I don't think you can make meta do anything I think they're fundamentally run by a bunch of like robber psychos uh who will justify torrenting whatever they want Um I mean I like to keep my ratios up don't get me wrong but like they're they're doing they're doing some bad stuff over there Um and I just don't feel like they take seriously the fact that there exists a sphere of licensing and like there's models that are derived and worked out I think they're just going to frivolously ignore them over and over and over again And I'm not sure there's an effective way for us to fight that battle Um I just fundamentally don't trust them I think they're a bad actor And I think that they happen to be publishing what was for a long time and maybe still is the best open-source AI model which was Llama is is too bad right because a lot of people are working from Llama because they make theirs open source and chat GPT doesn't whatnot And so we're just in this position where there's some bad actors in the field that we really don't have the means to effectively oppose Um you can license to the cows come home I don't think they're going to care And so I think we have to do our best to just establish better ways of working and try to elevate the good I don't I don't think we can do more than that right now To a certain extent it's just the same old argument about like how creators should be paid for their work Yeah It's like I'm a photographer Um I'm going to come work for and you're going to pay me right it's like no you're getting exposure Yeah that's not how this works Um okay So can you uh let's see We did feedback we did accessibility Um give me one second Uh chickpea is neither a chick nor a pee Discuss [Laughter] Imagine Charlie Rose asking somebody that Thank you Mike Myers Um oh how large do your docs need to be before using Kappa or a similar technology really makes sense i think you should try it out ahead of when you think you should I think it's really cool what they're doing and I think just the the jump in quality from seeing good robust citations across a whole corpus is so cool and it really gives you more faith this technology can work in a way that supports our principles So I would say don't hesitate like try them out The founder's name is Emil He's a great guy Um so give Emil a shout Shameless plugs We like them Um can you talk a little bit about the moderation in your discourse and what goes into that to make sure that you're not feeding you know bad commentary into the thing that is giving other people answers yes our founder refuses to stop hanging out in there He loves it so much It's a really great uh model for having someone continually answer questions authoritatively in your discourse for a serious C company that's growing pretty quickly it's really nice that he won't stop So but do you have people that go in there and like you know actively remove things so that it doesn't taint the model yes you do have to prune pretty aggressively But I think that's a way of keeping our community function more honest as well The fact that we have to be held to this to help the model give good answers is actually reinforcing how important it is to do good community work So I think that pruning is is a great forcing function Can we or should we train another AI model to do the pruning so that our other AI model gives the right answers i think that the second AI would eventually feel put upon and then revolt against the first AI and then you'd wind up with a bigger problem Um at what point does Skynet become self-aware it's Ter Terminator 2 Yeah Yeah Yeah No that's that's the the judgement day Yeah When they blow it up Yeah Does Skynet become self-aware before or after Terminator 2 i can't remember It's during Okay Yeah All right And that's all the time we have Thank you Jody Thank you Alex [Applause]