[Music] hope everyone enjoyed last night I know it's kind of an early start but I know somehow someway you'll you'll uh survive today um until maybe the football game this afternoon we'll see who the victors are um so it's my great honor to invite uh Jeremy Utley back to speak with us he spoke a few years ago and he said a pretty high bar because you'll get a survey afterwards and they rate all the speakers and that year he was the highest rated speaker of the whole Summit and uh don't tell condalisa rice that she may not be uh appreciative but uh Jeremy did a phenomenal job and uh as you may know and you read his background uh he's an edun professor at Stanford who's been very active in the dchool which is the design school here which is a phenomenal program world renowned and I believe he claims now this is you know we don't know if this is urban legend or not he claims to have trained over a million students so it's probably it's probably some truth to that but uh in any event uh he also is an author of a book we gave out a few years ago called idea flow and it's the only business metric that really matters um so I think that uh those of you who are interested following his talk um you know can arrange to buy the book it's it's available on Amazon and uh you know I think it's a great read and it's really about idea creation and it sort of stimulates all of us to think about how we can take um you know what we do every day and help our organizations improve by generating a lot more creativity in and bringing this sort of growth mindset to uh the work every day he's also a uh in runs a firm called free Spin Capital which is investment business um I think he again tries to bring some of his skills to the world of uh private investing and uh has a you know has a lot of fun a terrific guy Family three daughters or four four daughters four daughters and so he um he's got to join us this morning but can't stay all day because he's got some family responsibilities you might imagine but uh welcome Jeremy Utley and uh thank you again for your participation appreciate you all right that's that's a tough introduction to follow this is the highest rated speaker from a couple years ago the only place he can go is down I don't know I don't know how I can get better but I'm excited to share some of the stuff I've been working on over the last couple of years because as Mark mentioned my book came out in October I think I was here in the summer of 22 book comes out in October and then lo and behold a month later this thing called chat GPT come comes out so what I thought was going to be my kind of call it Whirlwind victory tour of my book tour uh quickly became me actually strapping my seat Bel and becoming a front row student in this amazing new technology and what I want to do is talk to you today about the really the intersection of idea flow and generative Ai and um and so for all of my credentials so to speak the thing that I would say is just scratch all of it out okay all of my credentials I feel like can be scratched out right now because I'm a front row student in the classroom and I've been teaching here at Stanford for the last 15 years but in the last two years I think it's my student kind of orientation that has really benefited me and really helped me learn a lot that I hope to share with you today so I hope to share as a fellow front row student so to speak in this exciting classroom and what I found is generative AI isn't just helpful for organizational transformation and personal creativity it's also helpful in domestic affairs okay so this is my front bathroom and as Mark mentioned I have four daughters who are quite rambunctious and you might imagine how many times I've asked them Please don't slam the doors please don't slam the doors we live in 115-year old old house in Mountain View this is 115 year old window which uh as my contractor would say is irreplaceable and the girls are running through the house the other day one slams The Door in the other's face and the other things I bet I can just reach through the window and thankfully nobody was hurt but I remember very distinctly I'm making breakfast I'm actually making eggs and I hear this uh cacophony which is not abnormal in my house I hear the cacophony and then I hear the girl start saying dad and I do what every responsible father should do which is pretend you don't hear anything and I keep flipping the eggs dad seriously and then my my eldest daughter goes dad there's glass everywhere so I you know turn off the burner walk down the front of the hallway and I see this and the question is how does daddy respond what do we do and my wife and I hmed and hawed in the front hallway for a moment but I want to tell you that generative AI delivered one of my greatest parenting victories of the last Dozen Years but let's not get ahead of ourselves okay this is this is a little bit further down The Learning Journey okay my goal today what I hope to do for you is I want to spark you a fire in you to become a gen AI conversationalist and you might think that's not a very big goal and I would say baby steps no I'm kidding really and truly the fundamental shift that has to happen is actually a mindset shift toward this new technology because there's a bunch of hype right now if you look for example this is a recent study conducted by LinkedIn and Microsoft I don't know if you've seen this it's a it's a great study take a read and the technology right now is being hyped enormously and we can't afford to miss out right you see stats like this 31,000 leaders surveyed said 70% of those surveyed said they would rather hire someone who doesn't have experience but is adept with AI than someone who does have experience but isn't Adept with AI is that shocking to you it's shocking to me for sure and you see stats like this there's an amazing paper actually from the Dark Side uh I'm kidding uh but there an amazing study actually is recently conducted amongst BCG uh Consultants I actually started my career at BCG and uh this is what they found every single way they could measure performance the individuals that were using Chad GPT outperformed those who weren't okay I call this the performance potential of generative AI the problem is very few individuals that I speak with are practically experiencing the kinds of gains that are supposedly being studied and my passion has been to help people close the gap between the possibility and their present reality so that's that's what I want to talk about today and one thing that I would mention to you is if you said oh I've tried chat GPT a coup you know last year and really wasn't that great I'd say this is one thing that you have to be careful about the technology is changing so rapidly that you might be inoculated and here's what I mean probably a year ago or something like that you heard about this thing called chat GPT and you open it up and you effectively Googled yourself right said let's see what it knows about me and it maybe returned garbage and you concluded it's not good this isn't a good technology that is the wrong way to think about it not only because it's not a a search but also because the technology is advancing so rapidly and its capabilities are growing so quickly that a weird experience or disappointing experience a year ago really has no bearing on the capabilities of today and so if you aren't using a frontier model with the best capabilities today and evaluating the possibility based on what's available today or based on maybe a weird experience a year ago you're probably missing out and you're probably not even talking about the same thing an easy way to put it is if you think about GPT 3.5 which was available publicly you know a year ago it's like a high school sophomore now GPT 4.0 which is actually freely available to anybody today is more like a first or second year doctoral student and so if you wouldn't give it to a high school sophomore which having been a high school sophomore and having worked with a lot of high school sophomores I know the capabilities there you you probably shouldn't have given it to GPT 3.5 but now there's freely available technology that's effectively the equivalent of a doctoral student and the kinds of things you can ask of a doctoral student are radically different than what you can ask of a high schooler right and so you have to be updating your expectations and if you've bounced off the technology in the past I just want to mention that's no excuse to not continue to play and experiment okay this is ultimately about you it's not not about organizations it's not about something bigger I think it's about you being able to do more better faster I truly have experience and have worked with many individuals and leaders organizations that have experienced profound productivity gains not just in parenting but in my business as well and I want to share some of the ways in which I have learned that will help you unlock that potential as well um as Mark mentioned my kind of background is innovation my expertise is in creativity entrepreneurship design thinking and so I can't really talk about AI without first setting a little bit of context for Innovation so that you can appreciate how I think about innovation in particular okay and as Mark mentioned I wrote a book a couple years ago called idea flow the basic premise of the book is innovation's not an event it's not the workshop or the sprs or the hackathon or whatever right Innovation is a practice and as former athletes or as present athletes Once An Athlete always an athlete right you can appreciate the fact that when you approach something like an event it's a very different mindset than approaching something like a practice like a capability that you routinely cultivate and nurture okay and this question how do I come up with a good idea it's rapidly changing this is a question I hear anywhere I go in the world whether it's Tokyo or Tel Aviv or Topeka people want to know how do I have a good idea this is the wrong question and the reason that I know that is because I'm I've had I'm a nerd I spent a lot of time researching this this is my nerd hero Dr Dean Keith Simonson isn't he handsome I have a nerd crush on Dr Dean Keith Simonson what he found okay is he studied breakthrough thinking across disciplines and across time periods and across geographies and this is what he found this is kind of the thesis statement of a lot of his research the single greatest determinant of the quality of your ideas is the quantity of your ideas which is to say if you want better ideas have more okay and this is a fundamental shift I talked about it at length a couple years ago you can watch that talk if you'd like but just as a to illustrate the point this is Jerry yulman he was anel Adams photography assistant he's also a professor at the University of Florida and what he does in a kind of famous experiment a famous demonstration he basically the divine his class in half because as you know at Stanford and I'm sure it's the same with the University of Florida every student wants an a every student and anytime around may I get an email from a student I just have an auto reply which is the criteria for grading are the following you can consult the grading CR right it's it's standard right and so students come to Y SM at Florida and say how am I going to get an A and he's he's done this several times it's an amazing study basically says okay I'm going to divide the class in half half of you if you want to get an A you have to turn in a spectacular photo and I'm going to bring in a jury of esteemed peers from you know National Geographic and Vogue magazine and they're going to evaluate the quality of your work and if you want an a you have to turn in an amazing photo if it's pretty good you you'll get a see if it's good you'll get a B but to get an A it's got to be incredible but then he says to the other half of the room totally different grading criteria for y'all I'm from Texas if you want to get an A you have to turn in at least a 100 photos doesn't matter how bad they are that's the good news they can be a 100 Terrible photos but if you turn in less than 100 you're getting a b or you know you turn less than 90 you're getting a c so quantity versus quality you could say you effectively divides his class into a natural experiment and then what happens is he brings in a jury of professional photographers and at the end of the semester they grade all the photos but here's the thing yolman doesn't tell them about his little experiment he just tells the photographers to grade the photos and what do they do they look obviously at the quality of the work they assign every single photo a grade they don't know about the experiment they don't know where where the photos came from but then yolman takes all of the jury's grades and he then assigns the control or exper mental condition and it's shocked his students for years and it shocks the jury every time because at the end of the semester when the grades when the dust kind of settles the folks whose job was to take a spectacular photo none of them do it zero A's however there are lots of A's in the class and they all happen to come from the group of students whose job was to take a 100 photos irrespective of quality this is what simonon was talking about the way to produce good work I would submit to you by the way a photo is a picture or a metaphor for good work right when you're trying to very few of us are photographers but all of us are trying to get A's you want an a at work you going to get an A performance review what Dr Dean Keith Simonson and Jerry yolman illustrate is the way to have a great idea is to be generating a lot more ideas and so you might ask yourself okay how do I have a lot of ideas okay if I know this is true hopefully I've gotten you to ask this question how do I have a lot of ideas or how many do I need that's how many you need Dr Bob Sutton and Andy haran on here at Stanford have studied Innovation and organizations and what they found is if you want to have a commercially successful idea you need about 2,000 ideas to start with and the basic if you work backwards through the funnel here's what you see to get to one that's commercially successful you need about five in Market to have five products in Market you need to be building about 100 prototypes and to get to the point where you have 100 things worth building you need about 2,000 ideas this what we call in the book The Idea ratio and by the way this ratio holds true across disciplines okay it's it's true James Dyson made 5,126 versions of the bagless vacuum before the 5,127 version worked okay my favorite Innovation the Doritos Locos taco from Taco Bell clearly the head of Taco Bell's food lab she was interviewed actually a really great interview she said she tasted over 2,000 different variations of the shell before they launched in pharmaceutical Discovery the number is more like 10,000 to one the point is you need a lot of ideas and it's true as Simonson found across functions geographies and historical eras this is always held true so here's where things get weird if you want a good idea the best way to have a good idea or to have a lot of ideas is actually to allow yourself to have a bad idea I'll say that again allow yourself to have a bad idea don't take my word for it though this is what Taylor Swift said recently I want my fans to know it's the hundreds or thousands of dumb ideas that I've had that have led me to my good ideas I've been saying this for years and then when Taylor twift says it it becomes true okay and you might be thinking hang on a second Jeremy I'm not trying to shake it off I'm in a very serious business okay and the way things work in my business is it's different I'm not a photographer I'm not a pop singer and swifties in the crowd are like well pop singer what right but here's my question for you how did Steve Jobs work do you know ser Johnny IV in his memorial address probably the greatest living designer he ate lunch with Steve every day and he said every day Steve used to say to me hey Johnny here's a Dopey idea and Johnny said most of the time his ideas were pretty Dopey in fact sometimes his ideas were terrible but every once in a while they take the air out of the room and I think it's important we don't we don't think about that when we think about Steve Jobs we think about the iPod we think about the Breakthrough we think about all of the delightful ideas we don't know the Dopey ideas but Steve Jobs knew what Taylor Swift Knew which is and I hate to geek out I heard that you had a statistics lecture yesterday I'm a huge stat nerd so I I'll just I'll just kind of remind you of some of your basic statistics okay how are naturally occurring phenomena distributed I know you're like wait I got to dig deep for that okay they fall on what's called a normal distribution right and ideas are naturally occurring phenomena which mean what a small percentage of our ideas are kind of the iPods of the world right a small percentage of Steve Jobs ideas are the iPods right the vast majority of our ideas are totally ordinary and then a small fraction of our ideas are Dopey or to use Taylor's where dumb right this is the this is the nature of things and the danger especially for those of us who are type A who are uh you know High achieving perfectionists which I know none of you are but I'll admit it I'm a recovering NBA I love a good spreadsheet I love a good pivot table and I only want good ideas and I'll tolerate okay ideas but the bad ones remember the premise here is you have to allow yourself to have bad and why is that because if you say I love good ideas and I'll tolerate mediocre ones but I the Dopey ones I just want to get rid of those you see what happens you can't just get rid of the bad ideas quality is a function of variation and in order to allow yourself to have great ideas you have to allow yourself to have bad ideas too you could say that dopy is the price of delight that's why Taylor Swift said said you have to give yourself permission and so you basically have a choice which distribution do you want you can Cho you can pick either one but there's no asymmetric distribution available to anyone not even Steve Jobs not even Taylor Swift you can have if you if you want delightful ideas you have to have Dopey if your goal is to not have Dopey that's fine but you don't get delightful either you you have to choose one and so hopefully I'm I'm kind of causing you to go okay wait so it's about volume and variation how do I increase the volume and variation of my ideas that is the mindset I brought to this incredible new technology because as I said my book came out in October of 22 and then this product launches November of 22 and I'm the Innovation guy I'm the guy going volume variation and all of a sudden we've got a tool with literally infinite capacity for volume and variation it' be like writing the world's greatest book on retail just prior to the internet emerging which is to say it's wild you look at my book you go to the in uh appendix what do they call it index yeah index AI is not even in the index this is insane and the point is this technology caught us all by surprise and so what I spent the last couple of years doing since being with you last is diving deep into this technology because not because it's some new thing but because it's a means of doing the thing that I really already know how to do and I've already helped you know millions of people do but now we have a tool that can do it better so myself and a colleague from Singularity University Keon gohart he's also from the Dark Side originally so we don't hold that against him but Keon and I started studying what happens to Innovation teams and creativity in organizations that have access to this tool we studied fir firms and teams in Europe and in the US from Industries as diverse as manufacturing and financial services and insurance and we started asking what happens to the problem solving approach and to The Innovation approach and to be honest with you I thought we were going to be conducting a study that was about how many more times creative are teams with Chad GPT are they two times more creative are they 10 times more creative are they 100 times more creative that that was that was what was thinking and so you'll imagine my surprise when the data started coming in and it's not two times more creative or 10 times more creative it's kind of most of the time not even as creative and I started reading I remember the First Data started rolling in my honest first thought was oh no oh no like we did something wrong and then of course the devilish oh yes oh interesting right and we ended up writing this paper uh or or writing a research study that was got published in Harvard Business Review just a couple months ago and the basic premise you can read the study if you want you can go online and find it but here's the basic premise most of the time most of the time teams that leverage generative AI fail to outperform their nonassisted counterparts sometimes they outperform dramatically but often they don't often they actually underperform and there's pardon me I just get choked up when I think about it there are there are predictable and preventable cognitive biases driving this underperformance it turns out the cognitive biases that are affecting this have been demonstrated in research Abraham and Edith luchin famously in something called the water jar experiment they've been demonstrating how humans have had a cognitive bias around problem solving since the 1940s and this new technology comes in which has the potential to unleash Untold human creativity and if you believe Harvard researchers at BCG they're unlocking enormous value and yet most people I talked to are going it doesn't even know who I am this kind of sucks you know I go there's an enormous disconnect here and that disconnect I'm seeing propagate in organizations and fundamentally it comes down to an orientation the teams that outperform treat AI differently most teams approach Ai and they think it's got the answers it's the Oracle it's going to give me the right answer and you know what they get just like yolman students who job is to take one great photo they kind of get a B they do actually get like a pretty decent answer and if it takes like three seconds you know what the human teams do let's go get coffee was pretty great that took like three seconds and they just turned that work in human teams struggle and push and wonder but AI assisted teams that outperform what they do is they AI more like a coworker more like a collaborator and it's a fundamental mindset shift and so you can ask yourself do you speak AI that really and truly I believe it's almost about a language acquisition or an orientation do you speak AI does that sound like crazy talk to me to you sorry does it sound like crazy talk you have to get to the point where you are conversation aist and what I want to do is I want to provide you a few I like to think of them as shooting stars what do I mean by that our imagination is sparked by unexpected input this is kind of fundamental cognitive science that when it's something new comes in that's when our imagination gets sparked so my favorite example is yannes Kepler father of planetary motion and heliocentricity kind of a big deal back in his day the the the fundamental or the existing Paradigm was the sky is firmament it's literally a fixed substance that's what they thought and Johannes Kepler one night was gazing at the night sky and he saw a shooting star and you know what he thought why isn't it cracking why isn't the firmament breaking right now and it's that unexpected input to someone who is primed to receive it his training did matter but it was an unexpected input that led him to radical discoveries that changed how every human being understood the world and what I want to do is I want to provide you just a few shooting stars so to speak things that spark your imagination to go wait could it if I can get you to ask could it one time today I I I win I declare that a win actually so I want to showcase a few examples a few simple ways that you can think about learning this language because it really is a language of innovation that will unleash you and your ability and then two also kind of quick watch outs with the time that we have remaining so here's four ways to learn the first thing I want to mention is natural language is not your thumbs okay most of us are used to interacting with a screen or a device with our thumbs or with our fingers the thing that's amazing about natural language processing is you can now use your natural language which is your voice you should not be primarily interacting with AI with your fingers okay so for example here's just a chat GPT screen you see the microphone there that will trigger what so open AI was you know quietly launching the the fastest growing consumer application of all time and while they were doing that they reinvented speech to text just like for fun okay you may not know that but what you can do is you can just start talking okay I do this for five or 10 minutes at a time I'm just rambling I'm literally babbling sometimes I don't even know what I'm getting at and then whenever you're done you just hit you know stop recording and it will do speech to text you can see there I tried to you know be as you know fast as possible the other day I'm trying to give an example of how good the voice of text technology is so I'm talking as fast as I possibly can and really mixing up my words because I want to make sure that it's shows how comprehensively an error Freel less it really works and it gets error Freel less which isn't actually a word it's kind of cool right and then it hits me back Voice to Text has come a long way and it's impressively robust even when you're tossing a verbal salad at high speed you know look it says even if you're channeling your inner Auctioneer the tech can keep up I'm like what okay how does that work for me practically I'm just going to invite you into my life for a moment because remember Mark said I've got four daughters I don't have a lot of free time okay so when can I exercise usually it's during my lunch break which is a joke but that's my lunch break right so I got a morning full of meetings generally like either sales stuff or classes or something and then if I'm looking at my calendar usually one to two is kind of you know is blocked and I've got a little bit of time and so what I'll do is I'll start stretching you know I'm going to go on a run and I just start blabbing hey I had a call with Mark and I did that talk at the Campbell trophy Summit and uh a couple points that I made one was I got to take it to voice and the other is you want to make sure to uh you want to turn the tables on AI and oh make sure that you write Kirby back because he dropped you a note on LinkedIn saying that he enjoyed your talk last time and was really looking forward to this one and um and on and on and on I'm just freestyle oh and then in the in the next meeting with Lisa from Harvard Education school she wanted to talk about agency as a means of promoting creative wellness and I wanted to link to that podcast that I saw the other day about someone mentioning agency was trust I think it'd be really cool so maybe we can drop that link in and then I know this afternoon I know I've got to get back to so and so right I'm just freestyling would you quickly write that into a memo that I could send to myself so I don't forget that stuff while I'm running that's a normal Tuesday I've got a meeting on campus I I live in Londa in the mountains right very windy roads and I'll be driving and I'll go hey I had a meeting with Paul from the GSB and he asked me if I would pull together a proposal for this new exacted that we're thinking about offering in connection with the AI program could you please uh my thoughts are um we got to get way more practical all of the existing training right now on AI is very technical in nature explaining the fundamentals of tokens and llms and it doesn't leave people with any really realistic understanding of how to get started today and so what the unique perspective I think I could bring is enabling people to know how to get started even though they don't understand it because they get on airplanes when they don't understand how to fly and they eat food even though though they don't understand their digestive system and they send emails even though they don't understand the internet but right now for some reason everybody thinks you need to understand llms before you use them actually that's a really good metaphor would you make sure to include those three specifically and maybe provide me two others that I hadn't thought of but as examples of things people use without understanding while I'm driving please write a first draft of that so that when I can get home I can review it real quick this is unleashing human creativity you know how long it would take me to send Paul the follow-up note that I was supposed to send him ordinarily it's like a week because I get home and as soon as I get home my kids are coming to the car and short-term memory totally gets cleared I forget all the stuff right I make dinner and then I sit down at 10 p.m. I'm like dear Paul great meeting today let me know when we can follow up maybe if that right this just how it is and all of a sudden all of my on this you know on the fly thoughts and real time processing is enabled because I have a co-intelligence as my friend Ethan miket Wharton calls it I have a co-intelligence who's capable of understanding transcribing synthesizing first drafting all of it's unlocked by my voice all of it's unlocked by the fact that I'm not have I'll get to this in a second this isn't Siri okay next paragraph comma you know when you're using Siri right you've got to be like overly prescriptive which kind of gets in the way of actual thinking we can now think at the speed of neurons not at the speed of our fingers that's incredible okay the second thing I want you to think about is turning the tables on the AI realize I'm giving you a lot here I'll I'll give you some resources later we we are not going to be able to even hardly scratch the surface here turn the tables what do I mean you should always append any request with this kind of statement before you answer ask me three questions so that I make sure you understand you go AI can ask me questions yeah that's that's the simplest example you know the other day my wife and I were going hey we've got teenage daughters and uh there's no handbook for how to raise teenage daughters and we had a situation unfolding and we said you know here's the situation we're talking with chat GPT um would you ask us three or four questions about her and about her friend before before you give advice about how we should think about that or provide Frameworks for how to think about it sure you know and just goes off asking us questions my friend Juan Carlos I had him on my podcast recently he's an amazing documentary filmmaker he made the documentary second skin which is all about kind of like online lives he said he's always wanted to build an iOS application but he's never been able to and chat GPT he got on chat GPT and he said something like this talk about turn the tables this is radical he said hey I want you to pretend to be my computer science Professor or my TA by the end of the semester I'm supposed to have published an iOS application but I don't know the first thing about it can you take me from no knowledge whatsoever to hitting publish on my first iOS application in the next 10 weeks what's the syllabus what are the daily assignments what's my first assignment and over the it took him a few weeks but over the course of a few weeks he got chat GPT to teach him how to publish his first IOS app you go wait it can what exactly these are shooting stars I'm just trying to give you shooting stars that hopefully break the firmament and you go it can yeah yeah it can it can do a lot more okay by the way I'm not I've never been paid by any of these companies or anything I don't get anything I just am supremely unlocked and I think it's a shame if any of you leave not being unlocked as well the next thing I want to mention is re generate and you go regen yeah regener what yeah what's the most important thing on this screen so you see I've asked u llm a question like some ideas for how to open a lecture about how to be let best leverage generative AI right absolutely it just starts launching right in the most important thing on this screen is right here did you see it you go wait where is that do you see it it's grainy because it's so small it's right here what is that button regenerate you know what that means try again do it again generative AI models are non-deterministic which means they they they're St how do you say they they literally think again so you ask me a question hey Jeremy what's your you know ask me anything how should I how should I try to break into the software it's like ah you could do this if he said well what else could I try you could do this what else can I try I wouldn't say the same thing you ask Google a question it will give you the same answer again again again because it's it's a deterministic search you ask chbt a question it will try again I recommend you never ask a question without hitting regenerate at least three times every single time it's like oh it's a great question let me let me tell you how I think about that right and sometimes they're radically different responses radically different and don't worry about clearing you can always click between the responses you can see them all but think in terms of this again this is a this is requires a paradigm shift because technology is trained us it's always going to give the same answer this is not Excel and I hate to break it to my you know fellow Financial analysts who love a good pivot table you cannot put I mean you actually can encode it uh a spreadsheet with you know to call on an API which is mind-blowing but it's not going to give you the same answer every time it's Sparkle the most important thing about a collaborator who's trying to be who who you you're leaning on for creativity is they're like oh what about this oh what about this they're chbt and other language models can do this the problem is the human takes the first answer that's given and goes well that's pretty good great job chat GPT and you have to retrain yourself the problem with human AI collaboration is the human expectation right and that's what we have to train out so again I'm just giving you some drills here last thing I'll mention here is respond I was talking with a senior clinician at Harvard Medical School who's building a training program for clinical rotations and she was applying for a grant I was actually asking her I've got kind of a small group that talks about how we use technology um and we have a monthly meeting and she was talking about oh yeah I got Chad GPT to write me a grant proposal it was amazing um and I said well how did it work she said well I got it to take it to take this grant proposal and then I just kind of edit it and stuff and I said well did you tell chat GPT what was wrong with the first draft this is what she said never even occurred to me to talk back right again you don't talk back to Google right with Chad gbt I'll give you an example I had Ed catmull one of my life Heroes uh Academy Award winner he worked with Steve Jobs longer than any living person Ed was on my podcast I said I want the best title possible so I asked chat GPT what are 10 suggest questions for a podcast title and then I said okay will you give me 10 more let's see at the end you see that can we try 10 more sure here's another 10 right sure and by the way Chad gbt never says dude I've already given you 10 right right it's a tireless indefatigable intern right so okay here's another 10 right so I said look I like numbers three and eight in first set and 1 six and 10 in the second can you generate 10 more here's the thing the next 10 weren't that good and I was like what's what's the deal so then I did this and I I'll kind of I'll dial in look at this this is what I mean by respond it requires thinking this is what humans don't like I just wanted to read my mind and just like do the thing is that how you treat your Junior employees read my mind dude you're fired you didn't do a good job it's just crazy right you actually have to tell it what you think but if you tell it what you think like I did here notice all of the thought I put into this why did I like those the next 10 I couldn't choose between the next 10 they were all so good but that's the point collaborating with the AI give me give me a a title that's treating AI like an oracle give me 10 titles still treating AI like an oracle here's what I like about what you did and here's how it could be improved that's treating AI like a collaborator it's a back and forth Dynamic I would say Rel relationship even I realize that's a weird thing to say but I think it's a more helpful metaphor for Effective collaboration than thinking about it like a technology okay two watch outs very quickly Tech is programed does right first obviously Paradigm being Google first result always the best result other than ads you know right or actually ads most of the time it's true the first result is the best result there that's not the case with generative AI because of its powers of regeneration because it's non-deterministic and because of hallucination which by the way are a feature not a bug again if you think about the normal distribution of ideas a Hallucination is just far from the mean that's all that's all a Hallucination is okay and when you're when your goal is creativity when your goal is innovation you actually want Randomness okay so that that's it's not a and there are ways to protect against it obviously but you actually want some of that Randomness so first result's not the best result this is actually how generative AI to come full circle solve my parenting problem because I had taken I've got thousands of blog posts and podcast transcripts and things like that I had taken a bunch of my knowledge and encoded kind of a private Jeremy bot effectively to brainstorm with myself if you'd like to brainstorm with myself also you just send me an email I'll just send you a link it's very simple I but I literally in the hallway said to my wife we should try the Jeremy bot she's like are you freaking kidding me I was like let's just see what are we going to do and I kid you not then and and I've programmed because I'm a teacher right so I I don't want to give the answer I want to give the approach the Jeremy bot kind of has like a it's like a Socratic what would Elon Musk do or that's a bad example you know what would a good person I don't know whatever right it doesn't matter sorry I didn't mean that I didn't mean that in a bad way I'm I'm I'm just playing my point is what would Sylvester Stallone do what would Serena Williams do what would you know Yanni do right but it will just kind of socratically provoke you with a bunch of unexpected analogies right and I kid you not and it's it's programmed to serve up 10 because I again I'm a volume and variation guy right so volume 10 variation in terms of Bezos or whatever right the ninth idea was incredible it's like brainstorming with myself but it was pushing me to think of ideas I wouldn't think of I had guy Kawasaki on my show he's a chief evangelist at Apple for a long time he's now at canva I think guy told me he's developed a GPT that he's trained on his own books he said when someone from the public emails him a question about one of his books he asks his GPT verifies the answer goes oh I forgot that story that's a great answer pastes the answer in he said his assistant if his assistant gets some a question about his expertise she doesn't even bother asking guy anymore she asks asks his GPT and then sends the answer right the point this isn't about my but it's about unlocking I had this experience just to because some of you are going dude are you serious I had this experience which was profoundly formative for me personally I wrote this book and I've got to read the book I go to Stanford you know uh recording studio and I'm reading my audio book you know what was shocking to me was how many times I wanted to stop and take notes think about that for a second the and the director I would say hang on I got to write this down the director he's in my ears he goes dude you wrote this book I go yeah but I forgot that story yeah I forgot that research yeah I that's a great way of expressing that nobody's walking around with the book in their head certainly not me not even the author right and so you go so on the one hand you go guy Kawasaki's got to consult his own GPT I go for sure it's way more reliable than guy Kawasaki and I don't I'm not saying anything bad about guy you ask my GPT something about my book it probably has better recall than I do right and the point is it's totally different it's totally different okay and and I experience it in simple situations like uh parenting challenges what did we do the GPT got me and my wife Michelle to realize this was an opportunity to have the children take own ownership of household rules that they hadn't before and so we had our 12-year-old prepare a school lesson for her sisters about why our household rules aren't arbitrary that was the greatest punishment we ever could have given her and it came from generative AI I'm a better parent because of it I was basically brainstorming with myself and in that moment I never could have thought of that idea right this is hyper practical but actually amazing I remember as she sat down going girls it's really important to listen to what mommy and daddy say I was like preach right that's great okay the other watch out this is not Siri I mentioned this a second ago it's not Siri okay with Siri I don't know have you ever had this experience I go hey Siri um text Michelle FYI I'm going to be a little bit late because the talk is running long I'm sorry I don't see Michelle FYI in your context it's like did you re this person who I text 20 times a day you really think that I'm trying to text a new contact whose last name is FYI right it's it's it's it's silliness and that's that's an extreme example but the point is with Siri you have to have synthesized your thoughts before you begin you can't really think on the fly with an llm the llm is actually capable of of thinking as well in a sense I mean you can say air quotes thinking but I don't have to have synthesized or processed my thoughts so much as vomit them out and then ask the llm to do the synthesis and there are many times be you know they say generally speaking there's like think to talk and there's talk to think kind of like introverted extroverted maybe some people think in order to be able to talk and other people talk in order to think I'm clearly a talk to think you all didn't say that because you thought it'd be rude it's not rude I'm clearly a talk to think many times it's not till the end of a conversation with Chad GPT that I actually know what I think about something but it's the conversation that enabled it right and that's the amazing thing is it's not an interface with a technology where I already know everything that I think and what I want to do about this before I get started it's a collaborator to help me understand what I know and what I think by the end of the collaboration and then it will spit it back to me I can say you know I I I had I had this post that I was working on a couple days ago and I didn't know the right way to frame it I basically said hey interview me I've got kind of five key stories that I want to try to pull together here interview me about those stories first great tell me about the first experience cool I was on a call with the CEO of the Sports organization and he said this thing and he asked me this and da okay great what's the next story okay then I was in the classroom and I you know and then I get to the end and I said would you would would you give me three or four suggestions for how I could position all of those stories and what the proper sequence would be in order to have maximum engagement with my audience sure and it's like you could position it this way or and then the sequence would be this or this way and the sequence would be this or this way and the sequence would be this and then I said hey would you rate those on a scale of 0o to 100 in terms of Engagement and likelihood to convert would you rate those on a scale of Zer to 100 it rated them how would you improve each of them to score them 100 out of 100 it scored them it improved them right but it's a totally different way of thinking it really is about collaboration so I said at the very beginning could it is the key I hope I have provoked one could it because the answer is yes as I I was working with a basketball organization with a you know back office of basketball organization there's probably 30 people in this room CEO we get to this point and the o goes hey could it help us find values aligned sponsors that we could work with that reinforce the values of our organization and the community that we may not know about they just ask that question I heard 30 right key fingers on keyboards and all of a sudden people are chiming and it's this amazing moment I said pause I'm super happy that we've solved this problem the most important thing is you just thought could it it's the most important thing there's a Hall of Famer one of my childhood Heroes sitting in the room uh and he he goes hey um I have an aura ring and a Garmin watch and you were just talking about the um code interpreter could it compile that data and have it talk to each other and give me insights that neither one yeah and everybody in the room you know the important thing is sparking the imagination right could it do this could it do that yes so the question is how do you expand your imagination you see there I've got a podcast which is great if you want to listen to it there's stories that are great I've got a newsletter here this is I think I hope it's a linked to newsletter it maybe a link to the slides if it's a link to the slides you can download them um and send me a note if you want I can send you a link to the newsletter as well but the point is I think your first order ambition should be to learn where can I go to spark my thinking if you aren't thinking about how can I think differently you will miss out okay podcast great I had for example the head of uh Notions AI we haven't released the episode yet but we'll probably release it next week um she was on just last week she said she's like yeah we monitor our internal tooling and use and stuff and she said we saw an enormous spike a couple weeks ago because one of our Engineers Was preparing for her performance review and she used used the notion AI tool in order she said ordinarily preparing for a performance view takes like 2 hours she posted a two-minute video of how she she scoured all of her last year's work assignments and correspondence and pulled together a statement of what she'd accomplished it took her two minutes she and then sheer the head of AI at notion said we monitored the internal use of our AI tool it's skyrocketed for the next two weeks what's the point that AI is good to prepare for performance reviews no sharing specific examples and being in an environment where specific examples are shared unlocks your imagination too that's the point where are you going to unlock your imagination if this is the only input to your AI Journey one I'm humbled and two it's not going to be good enough where are you going to learn you have to find mechanisms and forums to learn because you're going to hear a use case and go could it yes where are you going to get that right as I said I try to write about this stuff in my newsletter I'm not I I don't think it's the best there's a great podcast called Ai and I by Dan shipper who's also on our show who's an amazing there's tons of amazing resources put yourself in the Stream of knowledge that's how you're going to learn the last thing I'll say is I'd love to give you practice right now okay if you have chat GPT on your phone I'm going to give you a chance to do these first two things take it to voice and turn the tables if you want to you can literally pull out your phone and hit the you know microphone and it'll be cacophony for the next 30 seconds that's great your your phone will be able to pick up your voice it doesn't matter but literally right now if you want to I would actually love at least try it okay at least pull out your phone if you got chat GPT on your phone hit the microphone and just say something like if I have my do I have my phone here I don't have my phone here I would do it with you right now just say something like this to chat GPT you're a professional executive assistant with vast experience go ahead do do it right now do it right now synthesizing Meandering Brilliance I'd love your help capturing insights from an amazing lecture I just attended would you please draft a memo that I could share with my team make sure to mention something please also remind me of what Professor Utley said about such and such craft a brief memo I can copy paste into email to share the learnings but first ask me two questions to make sure you understand my context I love this noise so what's crazy right now is you know a couple hundred people here there are now a couple 100 radically different conversations based on the context you give it right it it's wild okay it's super fun it's it will unleash you and your certainly your team and your organization I have the incredible privilege of doing stuff like this with lots of organizations lots of for whatever reason lots of sporting organizations like basketball I've got a um a Premier League cricket team in India that's wanting to do like a AI upskilling which I'm super excited about I love going to India um but this stuff it's not a technology transformation it's not a digital transformation this is a people transformation and the organizations that will win are the organizations who treat this as a people transformation so it's not out of the IT department it should be out of the HR department I really believe okay I'll close with this going back to the study I mentioned at the beginning it's not too late okay roughly 50% of the folks surveyed in the study hadn't started six months ago so if you feel like oh man I haven't gotten started you're cool you know a great example of this is my Grandma she's in her mid90s lives in Stillwater Oklahoma go pokes if you're an OSU fan I moved to Texas when I was in high school so I'm like the trader they won't talk to me because I'm a longhorn so it's it's a messed up situation but I get this text from I'm joking it's not it's beautiful I get this text from my grandma the other day I told her this is like my favorite text message I've gotten this is my favorite text message I've gotten in like a year granny right that's amazing she had an amazing conversation with chat GPT about cream of mushroom Replacements the point is how did she get here her she's I mean she's an amazing cook she's doing stuff all the time how did she get to the point that she's able to imagine this application in her relevant environment two weeks prior i' been in the car with her and she's like hey what is this chat GPT thing you keep talking about I said I'll show you what's an emotional question you've got she's like well you know honestly uh we've been thinking about when you know when it would be appropriate to move into assisted living I said great let's talk to chat GPT about that she goes talk to chat GPT about it I said yeah ask us three or four questions before you give us any advice sure Jeremy have there been any changes to her Mobility recently I said granny uh well I mean you actually you know I tried to go to the gym a couple times a week and papa still golfs and he had a hole in one the other day is this what it's asking you know we're good next question tell me about her relationship with her caregivers I said granny well Jeremy's mom my daughter she drives up every couple months from Dallas but she's not a caregiver I mean and the youth group comes by but it's cuz we make him cookies is this what it means yeah that's good next question we do this and then I hand her my phone because I'm driving hand her my phone and she's totally silent I said so what do you think she goes I never thought about it like this before I said you like the technology she goes not the technology Jeremy assisted living and then two weeks later I get this text okay why do I mention this many of you probably yourself included you want the epiphany of how can I use it in my relevant context in my work there are many organizations going how can we leverage generative AI in our organization I say wrong question your people don't know how to leverage generative AI how in the world are they going to imagine some application in their context the first thing is what's an emotional question you're wrestling with that you'd ordinarily talk to your nextdoor neighbor Fay Anne about have a have a conversation with chat GPT that is a shooting star moment start you go whoa that was super insightful actually could it Bingo again if I can get you to that could it that's all I want to do the answer is yes what's your question thank you so much for having me I hope to stay in touch with any one of you [Applause] [Music]