gpt3 came out I uh thought to myself wow this is going to replace Google and any engineer I talk to said you are crazy this is a language model what are you talking about Olga thanks for jumping on a call on uh I don't know how it is for you in um where you're at but here here in North Carolina it's a very cold and dreary day today same same so um in this time between Christmas and New Year's when we're recording this we're gonna just hop in and talk about AI systems thinking whatever kind of whatever comes up so I I generally like to start the conversation just kind of like what is like either so I'll let you pick so one what is a system to you or what are you most interested in lately and I'll let you kind of we'll go from there most interested in AI That's that's kind of what I've been doing living and breeding for the last year yeah so tell me and I guess for for the audience for a little bit of background um Olga and I have been chatting for quite a while I think you originally reached out to me um because of some of my videos on on gpt3 and I think what were you doing originally I think you were doing prompt engineering but you've you've moved on from that since then right yeah yeah so uh I that was my entry into AI was uh doing prompt engineering came across your videos which were very insightful um so thank you for that and uh then I started doing kind of more work in the AI space but on the product side and on the marketing side for AI companies um my background is in marketing and product and from there decided to open up my own company in and Consulting uh companies on how they can build products with AI and having a team that can actually go out and build them so I do have a team of Engineers we build we're building some chat Bots for companies uh we are you know looking into some interesting projects uh that can integrate AI into various kinds of software so that's uh that's been the journey what uh well let me let me kind of take a couple steps back so what about what about this current wave of AI um kind of really pulled you in like what was interesting about prompt engineering or or was it just like what was the signal you were paying attention to or feeling yeah so uh it was the fact that anyone can do it you don't need to have a background in software engineering or computer science degree it was just intuitive in many ways it's seemingly simple definitely far from simple but but on first glance it was very intuitive and I was just blown away by the capability I uh thought to myself wow this is going to replace Google and any engineer I talk to you know gpt3 came out said you are crazy this is a language model what are you talking about uh so it was that it was just almost magical right you you're talking felt like you're talking to a person as opposed to talking to a chatbot and so you know F fast forwarding like so you get into it and you know you discover look look for information you know find find some of my videos and you were doing prompt engineering so what like and and you know you said like it was intuitive and there's a little bit of a low barrier of Entry but then you said it's it's not quite as simple as it seems so what did what what did you find in the early days of prompt engineering kind of moving from from marketing and product to prompt engineering and like what was what was that transition like and what what have you learned what are some of the key key principles or insights for prompt engineering that you've discovered so prompt engineering obviously as you know is very different now than it was then um at that point every punctuation mattered every single you know line break mattered so that was the first thing that I wasn't really expecting until I started actually doing it methodically so a lot of testing uh looking at every possible variation I can think of and trying to figure out well when do I need to iterate on a prompt and when do I need a new approach all together and that was one of the hardest things to figure out uh so what was sorry what was the next part of your question there was a couple things you said there oh it's just you know it started off intuitive and then you find some of the complexity over time and so like what are what are you know like you said iterating on prompts and then um maybe maybe when do you choose an entirely new approach and so that's like that that that's that's kind of like an answer to the question which is um you know what are some of the key principles or insights and like you said sometimes you iterate on the prompt and sometimes you have to choose like a fundamentally different approach right and so but that but then it's changed right because now we're on chat GPT and gp4 because when when you started it was gpt3 which is a different that's so last year right it's yeah exactly Everything Has Changed yep yeah no it's it's all moving so fast and so then you know moving forward I guess there was probably a decision point at some point when you realize like okay prompt engineering and then what was what was the decision to like like start a Consulting agency like what was that process like uh so it was a Natural Evolution kind of as I mentioned I so my my uh contract at that point The Prompt engineering it was short-term contract and expired and I started looking more into the field overall and I decided okay well I want to stay in this field I'm not sure exactly where I'm going to go but I want to stay in this field so started looking at uh marketing specifically product marketing roles and or product management roles within AI companies got a contract there got a couple of contracts in that space as well at the time well it's still kind of the case now I mean we're in the absolute worst like Tech recession since the early 2000s right while full-time work was not so easy to come by obviously especially in Tech but contract were kind of Landing in my lab and I decided okay well now instead of you using kind of just being a contractor I'm going to launch my own company because there are several things that I know I can offer that are different so I'm not just an engineer right I have experience with the you with the technology but I have a marketing background I have a product background so I can help businesses from that business strategy side but I was missing the technical piece which I still needed so that's where I decided okay I need to pair up with technical people and bring them on board into my company having that now full kind of spectrum so we have the technical aspect we have the product aspect and we can even help them go to market aspect then I can actually offer something truly of value that's different that I don't see that many companies out there doing they're usually either just Consulting how do you use AI what do you do with AI or just like an it kind of company Building Products so that's that's the journey that's kind of how we got there that makes sense and you know that I I remember I think it was maybe six months ago or so I was having some I was having like a local meet up with some friends and we realized that there would be such a huge Talent shortage um because as the need for AI skills ramps up so fast that there was probably a huge business opportunity as as you've found to provide that Talent on contract basis because you know I've I've had in the past I've had clients ask me you know how do I find so and so engineer or so and so you know uh you know mid-level management and I'm like well when you consider that Netflix is hiring you know product owners for a million dollars a annual salary you're not going to be able to compete with that and so there's just this huge like this huge systemic shortage like you said the we're in like a kind of once in a generation not since the early days of the internet Tech recession and it's really it's like this weird almost like a paradox right because on the one hand AI is exploding but there aren't that many jobs to be found in some respects but in other cases there's not enough employees to be found so there's this really big like systemic mismatch between skills and jobs and everything else and on top of that everything else is changing so from that like structural perspective from that systemic perspective like what are some of the trends that you see and and how are you like positioning your yourself in your your shop to to either I guess maximize those Trends or or meet those unmet needs I think contract work is just going to continue to accelerate on on the one hand I'm seeing people actually wanting to take a step back as professionals to decide their next move I hear this constantly from my you know my cont tors my Engineers is okay well I'm not sure you know like maybe either they got laid off or it was their decision to move into something else and to leave and they said okay well now I want to take a step back and AI seems interesting but I don't want to make any kind of commitment and I've heard this over and over again and maybe it's this exact kind of Market this exact kind of landscape where you've had hundreds of thousands of people being laid off and now a bunch of them are rethinking their career path they're thinking of starting their own company that's very common you would see you know with massive playoffs is a buding of more you know more startups right so I think uh that's that's going to continue um not sure when it'll level off I think eventually people will probably move back to your standard in-house employment uh I think think AI will definitely be playing a pivotal role in here so there might be some kind of reshuffling of the cards so to speak to figure out what skills companies need internally actually need with AI right like I had a friend who's an engineer he's like my workday is down to like two hours a day and because I have ai do everything for me so he's off working on a bunch of his own AI product you know so I think there's going to be this little bit of a period of uncertainty uh contracts are are going to be there for quite a bit I think yeah no I you know and and that that story of people that are like the new thing is overemployed right people with multiple jobs and they're kind of doing it all remotely and AI is helping do you think just just thinking like okay for employers that are that are that are listening um and other Business Leaders and decision makers I can imagine they're saying like Okay well if people are getting that much more productive maybe we need fewer people so do you think that there's going to be like a like obviously layoffs have already happened so is this part of that Trend do you think like I guess I guess another way of asking that is with the increase of productivity you could either expect people to produce more or have fewer people or a little bit of both um and so that I think that going to be one of the things that has to like the market has to correct itself um before things kind of quote unquote go back to normal but like will there ever be a normal again maybe maybe that's that maybe that's the first question like what is the new normal if anything if anything exactly the short answer is we don't know we don't know what's going to happen in two weeks op AI is dead set on releasing super intelligence right so who knows where when that'll happen probably will we have no idea I I think and also to clarify you know for listening is it like the layoffs haven't happened because of AI so layoffs have been happening for you know a couple years now actually and AI just kind of came on the scene a midst of that uh I think it will depend on the company it'll depend on their Ambitions do they want to grow do they want to make more money do you know what is it they want are they looking to go public maybe there's some own considerations on that side as well uh then you know maybe they want actually more people and they want more people who are using AI because they can be even more productive right maybe you have another company who is like no you know like I kind of want to take a vacation in Bali most of the year so like I'm just going to get rid of half my people so that I can go on vacation it's it's very difficult to say one way or another because companies have such different needs they're different stages of their growth um it depends yeah so so let's that there's two distinct paths from here that that are kind of percolating up so one you know like you said some companies are growing some are changing or pivoting so what can so I've got two questions and let me just lay them both out so we don't forget but one question is what can Engineers do or technical people do to make themselves more attractive because this is a question I don't know how often you get it but I get it all the time what can people do to make sure they you know their career survives their career pivots so if you know anyone who has pivoted who has taken some of that time to step back you know kind of as you did like you you you changed Direction I changed Direction and I became a YouTuber you know because I realized this was just too compelling I needed to talk about it um so like it is it is I guess maybe maybe that's the primary lesson is it is a time to step back reflect and maybe pivot but what are some of the key insights or maybe some of the some of the lessons learned from that pivot and then the then the same question but for companies you know because it sounds like companies are are taking the time to kind of recalibrate or maybe course correct as well so so yeah what what what goes into this period of transition and and change for for both Engineers or individual contributors and companies right so for engineers it's simple learn AI uh start you know learning about large language models start doing projects put them up on GitHub uh participate in hackathons read you know talk to people join Discord groups all of that stuff but obviously create your own projects is going to be the biggest thing so that people can go and see what have you actually done and when we hire Engineers we go and look at what their projects are not just what their resume is so go and create these things um I'm still kind of surprised you know now even talking to Engineers how many of them still have no idea of you know even the fundamental basics of large language models which is mind-blowing but it's true and and it's you know depends on what your background is what kind of engineer you were but that is going to be I think crucial so start now wait start now um that just goes out for companies I think they need to decide what are their goals what are their let's say one month three months six months and one year I don't think anyone can realistically plan Beyond one year in this landscape maybe very very mature Industries can maybe very mature compan companies can but you know because they've survived for centuries you know coola has been around since you know 19th century so I think they'll be fine right but most of us are not the coca-colas of the world so figure out what is your immediate goal your mid goal and then you your like one maybe twoyear goal that's as far as I can see anyone being able to plan in this and then figure out okay well how can we use use AI how can we use our existing resources maybe you don't need to hire new people maybe you can train your employees how to be really great prompt engineers and then they're just that much more productive um I think a lot of people still believe that AI is easy and so we can just get rid of all our marketing people because I can go in and write an ad well you can't I seen the ad I know the ads that Chad GPT produces when you say write an ad they are terrible there it's just so bad right and I've seen the blogs and the articles that it produces if you're you know the person writing them doesn't know prompt engineering doesn't understand or doesn't understand what good content is and I use marketers just as an example because that's my background right um but it's not it's not it's not good quality because you need to know how to use the technology right so maybe instead of saying okay well we're just going to you know hire one person and they're going to do all our marketing for us with the use of AI figure out well like can kind actually be on because from what I've seen no you can't do that um but that could be you know one area is really invest in your people invest in your people using AI how how they can self-develop how they can upscale there's a lot of opportunity for companies to help upskill their employees as you and I both know the simple prompt explain like I'm 10 right yep it it's simple but it works explain like I'm a marketer explain like I'm a lawyer right um productivity can is not necessarily just about how much you can produce but how can people work together how much can they understand each other so there's applications you can use AI for that that could be a really great way for companies to step back and assess what are our actual capabilities how we can use Ai and how do that fit in with our goals so I've heard from from multiple people now that basically like one way of saying that is treat AI like a basic competency right like you said it's surprising ing how many engineers and probably not just Engineers but other stakehold business stakeholders who just simply don't know the basics about language models or other AI capabilities today so like when you say like they need to know the basics like what kinds of Basics because this is this is also a question that I get like do do you mean like building neural networks or just prompt engineering like what's kind of the entry point for for learning the basics I'm I'm imagining prompt engineering is kind of the the starting point promp engineering yeah absolutely and I wouldn't even necessarily say prompt engineering I would say maybe like prompt writing I don't think you need to do like a full-on prompt engineering prompt engineer uh for just an average employee but they should know prompt engineering strategies or techniques so in that sense I think you can say it's prompt engineering but uh they don't they don't wouldn't need no much beyond that so that is crucial right learn what Chain of Thought is learn what you know tree of thought or decision of thought right um meta prompting right getting familiar with all those you know few shot learning and these are things that are simple to learn you just need to learn them and so one thing I picked up on is is as a marketer you've you can spot when marketing copy is read or or written either by by just a language model with no human correction or or by someone who's not necessarily a great prompt engineer or prompt writer yet and so I guess one of the key key lessons there is uh even if it even if it can turn out some content it's not going to fool an expert and so how far below and obviously it's difficult to put a number on this but how far below like expert performance is it is does it look like novice copy or is it just something that's not even useful like what's what's what's kind of the threshold there in terms of quality versus usefulness and and here's here's why I'm asking this is because you know a lot of people obviously a year from now the models might be better and they could be indistinguishable but at least right now you know it sounds like you can't you can't replace experts yet so what's what's the what's the gap between you know typical chat GPT output and expert output at least on some of these some of these benchmarks so I think the term expert needs to be defined right because if you are a marketer and you get paid money to do marketing then you're an expert right that's it's professional so Junior marketers mid-level marketers lazy marketers that is the kind of content that I'm seeing uh to give more concrete examples is these types of posts or ads that are you looking you know to improve your life look no further a um revolutionary technology this kind of trite content that we see everywhere um because it's been that's what it's been trained on right this is the kind of content that's out there that is just bad but it's what it knows it's been trained on that that's why if even gets a whiff of um marketing or social post or blog or anything like that and it's prompt this the kind of stuff it's going to Output you have to get very creative right because there's so much bad marketing out there there's so many bad ads so many bad social posts right this so tough so it is on par with the bad stuff right that's see uh things that are creative things that are actually engaging it's not there yet and the reason it's not there yet is because of the way the prompts I'm seeing are designed so to be even more concrete and a user will say write a prompt you know or or write me a social media post about cats okay and then so writes it writes something about cats instead what is a social media post well it's a piece of advice right right so that's what they should be thinking they should be thinking not like the term post they should be thinking what is this thing that I'm producing fundamentally it's not an engaging piece of content okay it is a thought leadership piece on cats right from the perspective of a veterinarian who sees you know thousands of cats a day maybe not that many that's way too cats sure sure uh so Mass Market bad marketing stuff right it can it can compete with the zero shot prompt without anyone knowing anything about prompt engineering for it to be actually good successful I think we're still aways from that from like the zero shot don't needs to think about any kind of strategy you know something just to me because my my wife and I we're both writers and we notice the same thing which is that the fiction that these models produce looks like the quality of fanfiction it looks like and it's like well what was it trained on what is what is completely free and open source and so you you almost end up with it kind of getting pulled towards an average right like yes it it it it obviously understands marketing copy copy better than someone who doesn't know it at all but it doesn't it's it the the patterns that it follows are kind of just middle of the road you know kind of more amateurish likewise it can write fiction better than someone who's never practiced you know fictional storytelling but for for anyone who's spent any amount of time refining the art they're gonna like this is this is kind of I'm not GNA say it's garbage but it's very it's very then to get higher level quality is better data sets like curating data sets so that absolutely like removing removing the junk from it because then it's like you know maybe as a marketing person like what do you what do you pay most attention to because you don't want to like contaminate your own mind with like bad ideas or bad copy like you say like no you have to be more Discerning about kind of your own human training data that you expose yourself to right uh maybe maybe that maybe not everyone is quite that robotic but it just I don't know got some gears turning in terms of what the machines are exposed to so with with all that in mind if if the language models they have some use some some utility um but it seems like at least right now it's kind of intrinsically limited by the maybe might just by the the distribution of the training data where are you seeing them being most useful in your in your consulting or product development or consult uh um uh like go to market strategies or product development for for your clients obviously I understand you can't share um everything but just yeah what what are you seeing in terms of where people are getting the most like kind of bang for their Buck sure so uh first just to take a step back I think that they are very useful for marketing and for Content Creation in the right hands I think this this is where I'm trying to get to is that a person who either doesn't know the what good looks like or a person who doesn't know prompt engineering won't get the best use of them but in the right hands they are phenomenal like I I do use them for writing all the time I choose it differently so the way that I would approach it is I have my Foundation I I know I know what it is that I want to talk about I know um maybe the the bone for example and I need help like an editor so in that way you can use it and with phenomenal results right um and also no prompt engineering so that gives me the added so in from that standpoint yes you can definitely use it for marketing um for code now I don't write code but from everyone I talk to you know all years I spoken with a lot of them really began to rely on it heavily and have just seen phenomenal results in their productivity um for businesses there's there's a number of areas um research so creating knowledge base right through and we can do something like uh connect your knowledge based VR rag retrieval augmented generation to an llm so you can begin to do research you can begin to find out you know chat with your own documents that is one of the most common ways that I think can be done immediately and it's the most useful right now so having your own information connected to an llm and using it that way um I think education is also a huge one again like we talked about you know explain like right so tailored these kind of tailored individual custom lesson plans um or maybe not even lesson plans but sessions right within llm and your own information to help a person upscale right that is huge right there's Concepts that I can now grasp that I never in a million years would be able to before llms so that is also really really big area um ideation that is really big I I'll constantly use it for ideation and um there's you you can you can definitely get some creative outputs uh you just have to know how to get it to give you those outputs but you can so ideation is a big one um uh summarizing like meeting notes for example right you have so so many meetings that people probably don't need to attend and they can just get a summary and they'll be fine executive summaries are a big one I actually recently even wrote an article I'm happy to share with you on the various ways that they can be applied um yeah I think I think education your own knowledge base and uh synthesizing large amounts of data and research that that uh lines up with there is a there's been a few studies at this point you know they look at productivity of developers and other people um and it can be difficult to measure these things especially because there can be such a subjectively wide range of of quality um and so some some things are not provable but one one study stands out in my mind that it showed that uh that using these these AI tools uh absolutely can make some things go faster particularly like the the the thinking of it like a tool and so you bring subject matter expertise marketing programming fiction writing whatever you bring some kind of external subject matter expertise and you combine that with learning this tool that's how you get the best results I remember I wrote a post about that on LinkedIn a few months ago and someone's like well it's prompt engineering so therefore only Engineers should do it and I'm like yeah but the engineering isn't a lawyer the engineer isn't you know a marketer like it like no you need you need to make these things more accessible to subject matter experts so that's one principle but then it can also raise the floor like you said for self-education you know one example that I give is like IM imagine that you have an engineer who needs to write a pearl script and they've never written Pearl in their life now at least they can fudge through it with the help of chat GPT um or you know plenty of other tools that are rising now um but then you know self-education uh you know there was a video that just came out before we jumped on like you know tech support like if you want to learn how to fix your own computer or whatever there's all kinds of things that it can be that that you can do um with these but sometimes you have to know how to ask the right question and like you said one of the key ones is explain it to me like I'm five or explain it to me like I'm 10 um but but even then learning to communicate with it to express your needs to the machine because sometimes if you tell it right like you just do this for me and it'll it'll give you the answer that you asked for but you didn't ask the right question um but I guess that that also comes down to prompt writing right you know how do you how do you express what you need so anyway it's just kind of working through all that um and I would say also it's not um and a problem that I see a lot of people run into and I think maybe this is also a problem of how we're viewing the technology is over Reliance on the technology so instead of you're the marketer and then like you said you go use this tool to help you you know be better or you're an engineer and you're using this um instead it it's just this constant conversation with the llm to get it to give you like the best output when that's not how you you're going to get the best bank you know the biggest bank for your buck is not by having an endless conversation to get it to give you the perfect output you get it to give you an output and then you iterate on it or you give it some kind of information and then it gives you something and then you iterate on it right like even a year ago if we had a person on staff that was able to cut our productivity or cut our you know the amount of work that we're trying to get done by like 25% 50% 70% right we wouldn't sit there and continue and try to get it to give us you know the perfect output for like 100% you know we we would be thrilled and we would probably maybe even pay millions to have a sidekick like that right and sad what people are doing is just sitting there constantly trying to get it to give you this perfect zero shot you know input output and uh that that is quite a hindrance and I think a misuse a little bit of the technology that makes sense you know it there's there's I don't know if you've heard this joke but it's been around for many years which is um the definition of AGI is whatever computers can't do yet um right exactly I I think that I think that people kind of mentally move the goalposts like oh here's this super useful tool and it's good but I like I get you know I you whoever Royal we get frustrated when it doesn't give us the perfect output but and so then we start wasting time trying to argue with it or trying to coax it to give us but it's like that's just the level that it performs at so you take that you say okay let's do good enough and then move on um so I you know I want to reinforce that idea because both for writing fiction and non-fiction um I use these tools to draft often like one paragraph or one scene at a time because it's like hey here's what I'm trying to say say this for me and then I'll go clean it up and make it make it mine exact exactly for any any profession around writing whether it's fiction non-fiction marketing or whatever like you say you iterate you get some you get you can't edit a blank page is the saying that we have um in in the fiction fiction industry um and likewise this is really good at filling up blank pages so you get it for the first draft knowing that there's going to be five or 10 or 15 drafts before you get a polished product and that's part of the process and you can even use it to get some of the feedback it might not be able to finish the polishing but you can you canuse use it for that so that's a that's a really good like Insight in terms of how to like what to expect of these tools because I I think that some people like you know they'll they'll sit in front of it like oh I asked it one question badly and it didn't give it right haha see it's useless yeah yeah exactly and people think it's it's omnipotent it's not omnipotent right that's the other thing it's not omnicient it's not omnipotent you know maybe omnipresent but you definitely can't expected to um know all of everything that you wanted to know and and just kind of guess your thoughts um there's you know context you need to give it context that's super important and uh and this and this is out also outside of writing right for everything else that we're we're discussing in you know different applications of um llm so in education as well you need to give it the concept you can't just say you know teach me computer you know software right like you need to be more specific you need to explain it and and what do you need and why are you doing this why do you want to learn it right what what are the goals goals are really important um asking when you have you know you're asking questions right you also need to give it some kind of context like why are you asking this right what is the thing that you really want to know unless it's you know maybe just like a simple fact like uh what is the capital of France you don't need to give it context there but for anything more evolved context is is really what you need you know I wonder if there's if the if if a big part of the problem is kind of The Uncanny Valley because you you feel like you're talking to you know a person I remember when when a lot of people first got access to chat GPT there was a lot of like not I don't mean like maybe not conspiracy theory but people are saying like who am I talking to like they were convinced that they were that there was that it was just like a Mechanical Turk kind of thing you know that there was a human that they were actually paying to chat with and they're like how are they doing this and it's like no this is actually AI but it it it's it it falls into that uncanny valley where it's like okay you feel like you're talking to a human-like intelligence but then there are some of those deficits like it it will it can often infer what you need but sometimes it gets it very wrong or it forgets um or doesn't have enough context you know so on and so forth so and and you and I probably take this for granted because we've been using this stuff for a couple years now and so we understand like I tell people like you want to get an intuition for how these things go back to plain vanilla gpt3 where if you do a wrong prompt it'll just start spitting out gibberish like there was one of the very first chat Bots I built um it act I accidentally injected um an error code and so it Swift from conversation to just barfing out error codes you know so it's like you know like go back go back to that if you want to understand like what's going on behind the hood and so you understand there's already a lot of guard rails put in place but of course a lot of people they don't necessarily have time for that or you know but I I think that you know because I have students asking me like should I still learn computer science it's like look I started with Pascal and C++ I don't use them anymore but I understand the the fundamentals I think I think similarly like using using unaligned language models might be the today's equivalent of learning assembly or learning learning ancient you know learning c um but yeah so I guess let's so first just thanks for like all that context as to where we're at today what do you what do you think's going to come in in the next six to 12 months in terms of AI capabilities um company adoption because like it everything's changing at once there is no status quo right there there there's not even a dynamic equilibrium right now everything is changing under us at all times and I think that that's one thing that makes students you know any people anywhere in their career unless they're retired that's I get I get so many comments from people saying like oh I'm just I'm glad that I'm about to retire so I don't have to deal with this but everyone from students through early mid late career like everyone is a little bit you know has a little bit of heartburn about it so what do you with you know being kind of in the thick of it where do you see the the next 6 to 12 months in terms of in terms of AI adoption and learning and capabilities capabilities I'm not going to try to predict no pass a year ago when I was working with you know gpt3 and I would never in the million years imagine that just you know and not even in a year I mean back it was you know just months later right the chat GPT came out after my work with GPT 3 completely night and day right and now we're on to you know multimodal like I never could have imagined that I don't know maybe it can smell things in six months I don't know you never know you never know right like I would I wouldn't have thought that it can see and um so not going to try to predict it uh in terms of adoption it's likely going to be your you know typical curve right um you're going to have your early adopters you're going to have you know your average user you have your lagers um I think probably industries that tend to be slower uh the tend you know that are regulated are more likely to be lagging on how they're going to be implementing it they you know um you for example like even in finance right like you need every single little thing to be approved and reviewed by a lawyer and obviously nothing ever goes anywhere because it's in constant review so those Industries are probably going to be uh lagging I think um small businesses are actually going to be embracing it faster than a lot of the established Enterprises and that's what I'm seeing right now um they have a really hard time making these kind of massive shifts right because they have to retrain thousands of people right they have all of these processes that now need to be completely redefined um tech companies I'm sure going to be adapting it quicker than you know even established tech companies than non Tech uh but yeah I like all the clients that we work with they're small businesses and they are all people that have decided hey I really want to do something with AI and they're in you know wi wide range right um even lawyers you know now talking to someone that is interested in doing something with the ey but it's private practice it's not a you know massive Law Firm that's going to have 20 different you know decision making steps right it's priv practice same thing you know and doctors are possibly going to adapt it quicker um if they are private practices versus uh large Enterprises um unless they can get you know somehow get into medum which is Google released recently so and then that is only you know a handful of organizations that have access to that right now maybe Google who knows maybe Google will go and roll out a metal M for for that's hip a compliant and now every single doctor can go and use it right so they don't need to build their own little systems um it's we we'll see we'll see what's what's going to happen a lot of a lot of things are uncertain I think laws and regulations are going to play a part of it um you know the Biden executive order a couple of months ago and now you know use act on AI are kind of steps in that direction um I hadn't read the actual act from you know the EU act but I read Biden's executive order and most of it is like we have a plan to have a plan in 90 days we will have a plan right I don't know it depends on how fast they get their plan to have a plan maybe um but they are they are putting you know intending to put in a lot of money into small businesses they're intending to put in a lot of money into training people to use Ai and maybe that'll go through very quickly and so then that'll definitely ramp up adoption if we have that Federal funding you know back maybe they maybe they may do it in to industries that have typically been laggard on technology so that that might help kind of also level the playing field a little bit between industries that makes sense so in the so you know sounds it sounds like we're still kind of very in very much in the ramp up phase you would say in 2024 so it's like there's there because there's like the the bleeding edge innovators then there's the early adopters and then early majority so would you say like are we still like is 2024 going to be the the year of early adopters or early majority or is that still kind of too fast sound it sounds like it might be might take a little longer yeah I think it's too fast I think it's easy to get stuck in this like Tech bubble that you and I are in because everyone we talk to knows about Ai and large language models and so we think oh everyone knows about this now when in reality the vast majority of people don't even know the term large language model right um and you they think of AI as I again like some you know omnipotent um kind of omnition deity I I don't know I how to describe it but they don't so I think it's really for all you know the the hype and everything it's still a very very small percent of people that actually are using the technology and the people that we might expect to be using technology are not and the people that we don't expect to use technology they are and they go and they they like go like oh my God I you know want want to be an AI engineer now so um overall I think people know what chpt is many people they know which LGBT is they've probably gone on the platform and tried it you know maybe they used it to write a blog post or something but that's probably the extent of it not much not much more beyond that right uh and the capabilities are just light years ahead of you know what people are currently using you know it it strikes me because because yes we've we've been you know we're we're at like Cutting Edge like very earliest like first million people who knew about this thing or maybe even fewer than that um and so you know by and large and you're right because I had a friend message me just before we jumped on this call um he's like oh yeah I still have friends you know that are and colleagues that don't even know what open AI is don't even know what chat GPT is right so we are like grand scheme of things we are kind of in our own little Echo chamber like like at the tip of the spear and I was I was taking a note as you're talking it seems like there's this full spectrum of people like some of the earliest adopters are the ones that are anxious about it right they don't want to get replaced they don't want to get left behind I've had I've had so many clients over the last year and a half that were you know everything from from lawyers to others that are like I just don't want to get left behind like I see the writing on the wall um and of course like it it may or may not be that that big that bad you know I'm one who predicts that like eventually it's going to just reshape everything and maybe nobody's going to work in five or t years don't know um but then there's people that are excited about it who see nothing but potential so there's like fear and excitement are like some of the reactions but then there's a lot of people that are still in denial or just they don't get it or whatever and they're like H you know it's another Trend it's another fad we still see some people talking like that you know like unironically saying oh this is just another trap you know another fad that's going to fade away or you know it's another Trend and I'm not going to jump on the hype cycle so like what is what like to temper that what is your take like what is hype what is real what is there to be anxious about what is there to be excited about what's what's real what's not from your perspective very good question so I don't think is going to replace everyone in a year five years maybe I have no idea again right like it's it's the exponential evolution of the technology um where we are today I no I don't I don't think it's going to replace everybody uh it's going to be a useful tool and it's going to assist many different professions probably some people that are going to get let go wouldn't wouldn't be surprised if that happens um I think that probably still quite a ways from nobody working so I'm not sure I would go that far um but you know in in reality it can do a lot it can do more than many many people out there I think think people have yet to imagine what it can actually do I think that's what it's actually what I'm seeing one of the greatest limitations is human imagination when it comes to this technology so depends on when when people will kind of figure out what to do with it and yeah I I think um once people start actually using it Mass majority using it maybe that's when there should be more anxiety on Horizon but this current state where very few people are really really using it for work for for you know enhanced productivity um it's it's still kind of going to be hype in terms of like replacement uh but when paired with other Technologies it's uh it's kind of almost like Sky limit a little bit on you know it's what what can you imagine it's not you know not just like going to chat GPT and just talking to a large language model but layering it into different kinds of software and uh that's you know it's one of the things that my company is working on is figuring out how to apply it not just as a straight you know chat bot you're just talking to an llm or just using an llm but using it with other Technologies so that that's actually a perfect segue um because there's the I was just reading this morning open AI paper about like the governance of agentic AI systems and they broke it down into three primary stakeholders so there's the model makers or the model developers like open AI the ones who create the the llamas the gp4s and and so on um but then there's the the systems developers so these are the ones that are integrating it into platforms into Hardware software uh stacks and it sounds like that's kind of where you're at you're you're sitting that between that and the user because you also do the marketing and go to market so let's talk about um let's let's let's talk about that because obviously you know Google's Microsoft's open AIS meta they're all building the models they're not necessarily integrating it so then those of us that are more close to the front end that are in the user space and in the the systems engineering and integration space what do you see happening there because you mentioned like rag retrieval augmented generation earlier we also kind of talked about how making it accessible to subject matter experts is kind of where the kind of The Sweet Spot or one of the sweet spots for productivity because it's not going to replace it an expert or an experienced individual so what on the tech space the implementation and the integration what are you seeing happening right now out there so it's um the people that are coming out of is people who are experts in their domain and they are trying to figure out how to apply this technology and you know what to build so for example emergency preparedness space there's a lot of things that are still kind of being done in you know pen and paper kind of situation right when you can use it's it's partially like automation and partially actually having an llm integrated to help coming up you know with with these different aspect these different ideas so that can be you know one example um or it's a and and this this actually is in the real example there's just something my team and I had been talking about um you know a lawyer who maybe wants to give you know uh content to their clients right like Prep Prep ation notes for example for you know the trial right so there's there's possibilities to build this kind of different software that can help people who are domain experts do either do their job or create a new product offering alog together for their clients that leverages the technology and it doesn't need to be some really like Advanced you know groundbreaking shot in new thing that you've come up with it just needs to be a practical application I think a lot of companies get stuck in this like oh I want to build you know the newest coolest thing right but they don't think about the problem and what problem are they solving and so the the people who are coming out on top using this technology they are solving an existing problem they're not coming out with some you know new revolutionary another large language model model those aren't the people who are profiting you know I think pretty sure all these large language models haven't made a single dollar really so it's uh it's people who can figure out what is the problem that we're solving how can we solve that problem in the simplest possible way but there's still this kind of like added wow factor because a large language model is involved or you can do it faster because large language model is involved so because this this is kind of my mental model and let me know if you agree with this or if you you would change it but it's like you know you're talking about domain experts subject matter experts whether it's a lawyer or a first responder or you know whoever these are these are people who already they know their industry they know their business they know how to create value and so what you're saying is the people who who who really do the best with these Integrations is they they view it as okay how can I use this as a tool to augment what I'm already doing and like you said in some cases they might add a new product or a new service but it's still very much within the within their domain of expertise yeah whether it's like you know hey I'm a I'm a trial lawyer now I can just add a new tool to help clients prepare for trial but they're still fundamentally doing the same thing it's not you know like you said how did you say like you know you're not you're not gonna create some shiny new you know change change the world not everyone's trying to be the next Amazon or the next Tesla or whatever so so there's a lot of value to be found in just uh incremental improvements on existing products and services is that kind of a fair assessment okay yeah exactly it's it's uh customizing it's not just an incremental Improvement but it's it's uh being able to customize that experience for your industry for your customers because people keep s you know saying well how is this different from chat GPT why can't the person just like go on try GPT and like do the same I like well sure like there's that t capability and they can but then there's all of these other aspects that they don't get where you know it's the um the ability to for example reduce hallucination by implementing rag 100% reduction right but you can bring it down things like 97% accuracy with r right so you can connect your own knowledge base to it and that doesn't exist in just going by to ch gbt um and it's uh you know just different product offerings internal or external that people can build fairly quickly um that used to you know could have been millions of dollars to try to implement something like this but is now way way way cheaper than that so it makes it accessible to a lot of different Industries a lot of different professionals um most of our clients are actually solo consultants and they they have their own businesses and this is now like they're beginning to add a different uh product that they can offer to their customers so um yeah you're just exactly exactly you said there are domain experts that are able to spot an opportunity and we help them you know because sometimes I have had someone come to me and say like hey you know I think I want to do something with AI but like I'm not really sure and then you know I would do intake form with them and say okay well the question I ask is what part of your job do you hate takes a really long time and you wish a magic Genie can just disappear and take from you and that is the part that we can probably build right I'm not going to replace your brain but it's a lot of it is in the automation um sphere rather than like human replacement sphere that was actually the those those kind of two points were were one of the primary um set of questions that I would focus on when working with clients was what is the most painful right what is what is the most tedious or painful or draining part of your job and what takes the most time and those are those were two places two where where I would start to Target like let's let's see what what what AI can do to make it a little bit less painful a little bit faster but you mentioned um internal versus external so this is I think this is going to be interesting to a lot of people because you know in in in big Tech you know the shiny sexy thing that makes the news is is usually B to C right the the the the unicorns are often B Toc uh business to Consumer sometimes they're B2B um you know when you get when you get a billion dollar IPO um but really the things that are that are really headline grabbing are B Toc but there's also you can serve internally so you talked about like whether you're serving a customer or maybe the user is your employees and that gets a lot less airtime so what are you seeing on both of those fronts in terms of you know serving external customers versus serving internal customers or employees as as the user what are you seeing in that space so for inter interal um I think EXC me employee onboarding is is a potentially going to be a huge area because you can just ramp up so much faster um upskilling they say your your employees um moving you know just like moving from a junior level person to a mid-level person that can happen a lot quicker because you can have those training materials in place and they can be customized and it can happen in a way that is relatable to a person so when I say education I mean it's like not just like student education you know in our um you know public education system or universities right I'm talking about employee education so employee education I think is uh is a very big um uh team cross team collaboration cross team communication can definitely be improved quite a bit I I built a little like proof of concept early on in my prompt engineering on how you know some of the ways that it can be helped so you can understand a person fun you know basic knowledge basic understanding of your colleagues job and what they are or not capable of doing what is and is not in their sphere in their domain can eliminate hours and hours of meetings so you certainly like reducing the number of meetings that you need to have is a huge huge gain for for companies um you know there's been a number of roles where I had where I you I work with so many different departments 70% of my time was spent on meetings that's insane that's so much money right like right and but there was like no way around it because you you know you had to have all this consensus and people have all these questions and you have to explain it to them and you have to explain it to them in a way that makes sense you know that they can understand and then you have summaries of meetings and then you have to have a meeting about the summary of the meeting and then and there's no summary someone took bad notes right so you can eliminate so much of that so many meetings are just endless chatter and nothing gets done so eliminating that and pivoting your business to more of like a bias for Action mentality as opposed to we need to constantly talk about this mentality can just throw AI into the mix and then we don't need to talk about it anymore so that on the user side um research internal research uh can certainly be um a big one so for example like real estate developers you know they can uh do research much faster and so they can you know like we can go and Implement something for them that will enable them to find like great properties for example and I guess you can say that's like a that can be like a SAS product right that theoretically a company can go and and build and sell right so there's a ton of SAS products they can go and build and sell or companies can just essentially build their own SAS products and use that you know have have that customized which is kind of what we're doing we're building like internal Tas products um I hope I hope that answers your question I hope that gives you some ideas yeah no like that so it seem it seems like AI can help like at every level internally for an organization yeah um like almost creating a new like organizational operating system right like it's it it can it can Grease the wheels of communication make sure that um that unnecessary communication isn't happening because that can be a Time sync or that communication is getting information is getting to who it needs um in a more timely manner but but one thing that you've said a couple times is is is customization and it just occurred to me you're talking about onboarding upskilling like those kinds of things it's not just a matter of having the company data but really like I'm imagining some of that could be much much more easily tailored to a company so that someone goes through a new AI enabled onboarding process or internal upskilling process it's not just an external you know training came and said okay here's a generic set of skills here's actually how we do it and here's how we can make sure that we all do it better or whatever the metrics are um yeah and so however so all that's really fascinating there's two things that I want to drill in on because they I think that they're a little bit more mysterious and they also keep coming up in all the conversations I'm having so one is cross- team Communications so lateral communication and the other is bias for Action so how do you how do you structurally or systematically how do you create the policies the culture the systems the tools for that cross team communication and then also how do you create that bias for Action because those are two really fascinating Concepts to me so I guess a part of it is you know people have to not fear technology in order to be able to use it a lot of people they they really are very resistant um it's a very personal almost philosophical um thing that I'm seeing from people so kind of educating people on what this thing actually is and how it's going to be used and how it's going to be beneficial for them making them feel safe in using the technology and by understanding it at the Fun fundamental level right not like engineering or anything like that um so creating that sense of safety in using AI so many people I think are really like you said they have this anxiety um so so caling that anxiety right um you know beyond that you can build you know like your your own chatot in a way right that that um members can communicate with you know as a way to either prep for a meeting right or if they had some kind of questions they can go and ask the chatbot they don't need to you know poke uh bill from accounting five times a day because they don't understand what this thing is you know right what this number means or um so uh having maybe having a little bit more communication with the AI that can kind of is just like lives in your screen right so it it's just there you know you can just pop it open whenever you want and ask that question right or if there's something you don't understand or something needs clarifying right just have it kind of as a thing that is you know one of the things that gets put on your computer when you join a company right along with Outlook or Gmail or whatever it else they put on your computer so that already is going to just make a as like a standard everyone in the company has their own assistant you know tutor however you want you decide to call it um that I think that will that in and of itself will take it a lot uh you can do lunch and learns um in i' I've done ton of lunch and learns just throughout my career um educating people about marketing because people also have no idea what marketing is and they think it's sales or they you know they have their own ideas of what could be um so lunch and learns are great um and it can be you know fun fun way of uh getting people involved in it um you can maybe have some kind of a slack integration integrating a Chapa into slack I know some uh people are trying to you know work on these different kinds of products people are already doing it so if it's a company that does use slack they can maybe integrate um chaab in slack so those are some some of the ways that I think companies can get started and then they can assess on hey well you know how is how is this going and uh what can we improve you know based on user feedback so really listening to what their employees are telling them as feedback what things they like what things they didn't like and actually listening to that the so the the the the anxiety the fear is kind of one of the first barriers to overcome and then once you overcome that it's just a matter of building familiarity and understanding exactly and and then the third third thing I think is is accessibility just putting it in front of people just just based on ux design or affordances it's in your browser it's in your screen it's in your the the things that you're already using right um and just making it more accessible and adding value there rather than like burning everything down and starting from scratch or or whatever exactly and now um I I can't say who the company was but there's a a company very large um insurance company that had put uh so you know I'm not sure what language model they put I I assume it was probably 3.5 and they may have done some prompt engineering um to limit some of the output you know capabilities or input capabilities but they had created a uh Chad GPT you know customizable Chad GPT for their organization and yet their employees had no idea how to use it they had no idea how to use it they had no idea what it was for they were I actually had someone tell me that they were afraid to use it because they were afraid that they would become addicted to it and it's this you know there's all these um you know fears anxieties and misconceptions and preconceived notions that people have and companies however much they spent on it I have no idea what they spent however much it was was definitely not cheap given their scale and yet they are not actually making any use of it because there's been no um employee training nothing there's no employee education nothing they're like oh yeah I know we have one you know and like oh okay we're you know we're told not to use like chpt we're told to use this and it's it so it's a wa it's a waste it's a it's a waste of time and if anything maybe made things even worse because uh now people think oh it's useless or you know some replace the end of either it's useless or it's gonna replace me right so if anything they made it worse so it's it's not automatic that you expose people and and things get better you can actually have pretty strong adverse experiences yeah Poison the Well and it's just it's mind-blowing to me that there was no employee training whatsoever it was just like hey we have this thing now don't use CH GPT you have your own click this button and that's it and then I would show people you know some some of the employees I showed them they were just just you know friends or colleagues um who worked at some companies that had created these and show them what they can do and and they were just completely blown away like what really like why didn't my you know why did my company tell me this I don't know why they didn't tell you you should ask them I don't know it seems like they should have so you you you really have to be deliberate in in not just making the tools accessible but saying you know step one turn turn the computer on step two like this is a keyboard right so right exactly yeah no that that completely with with what I've talked with other people about which is just treating this like treating Ai and AI tools like a new basic competency like it's just like teaching people Excel for the first time or whatever EXA so as we're as we're kind of winding down I want to ask because you've talked about we've talked about business a lot AI a lot and people a lot so what are your personal Like rules of thumb or best practices for business people and Ai and you in in any particular order H um or Cardinal rules yeah yeah card Cardinal rules learn prompting that's like a non-negotiable you need and and this is part of the offering that we do which is put a part of thing that made us successful is that because we can offer these trainings to people so that is nonnegotiable you have to know how to use it your employees need to know how to use it there's fundamental strategies and then you can dig the deeper in into it so uh every day spend you know a few minutes a day using the technology in some way um Don't Be Afraid right like instead of wondering oh can it go and you know can it do this can it tell me that go and try don't be afraid you're not losing anything like I've talked to people um who were also like really Blown Away by you know some of the things that you can do you know even from like asking advice about something right wouldn't have occurred to them um and they're like well can I do this and then the answer is I know go try right and people are afraid to try so don't be afraid to try it's absolutely important don't uh underestimate but don't overestimate so it's uh very easy to say oh well this should be easy for it to do right so you will you'll see um I'm sure you seen like on LinkedIn it's like LinkedIn and YouTube you know Instagram I've seen like you know the 10 best prompt or you know all all these like you know here's here's what you can do you can give it like you know this information you can like you know give it your diary and it'll output some I don't know Miracle or something like that um so in theory it sounds great that would be cool if I could like you know give it a diary you have to keep a diary to begin with right and then it'll I don't know what it'll output but it'll output like predict my next day or something like that um tell me what's wrong with my life yeah it's not gonna fix wrong with my life right right right it probably I mean it will tell you I don't know if it'll be right or not but it will definely tell you um but don't you know don't think that just because it looks easy or it sounds easy that it is easy and this is um from both just like an and user uh kind of you know interaction with the chat body or interaction with the llm um and companies building the products you know like we've learned as company we've learned quite a bit and there's uh you know there's assumptions that we've made that we thought this is going to be easy and it is not at all like it's like a day worth of work oh no we didn't think about that right jump in the deep end yeah yeah yeah yeah you think because you know you you read stuff and you're like oh like oh this is Theory right so in in theory you know you read papers and you you know see what people are talking about in Discord or in Reddit and um you know AI experts right are saying like oh yeah it's like super simple it's not that simple and uh proof of concept is different from a product so uh that's you know like these like gpts for example they can help you with maybe like a proof of concept get an idea going of um you know how can answer certain you know certain kind of question or certain kind of um domain expertise you wanted to have but it's not a product right so uh distinguishing you know what is a proof concept and what you can kind of do it a very very slow like small small scale and what is an actual product again internal or external um that is going to need to scale so you know that's that that's just like anything with software right you need to factory in scalability um so that is true for llms as for any other kind of thing that you build again internal or external right yeah it's interesting because in some respects there is a lot of novelty but in in other respects it's it's it's just another tool in the toolbox software is not anything new the internet is nothing new data is nothing new it's just another component to to bolt onto that well those are some really great Cardinal rules to follow so thank you for for sharing and um thanks for hanging out with me for I think a little almost an hour and a half now um time time flies so yeah any any closing U questions or thoughts that you'd like to share with the world um Embrace I think that's you know and and and embrace it by with uh childlike Wonder in a way that's great so because um trying you know trying to say oh this is going to like Doom the world and maybe whatever right who knows but like um it's it's exciting there's a lot to learn there's things that are constantly coming out we have no idea where it's going to lead um but for the most part I think it's all going to be positive developments and um yeah embrace it excellent well thank you for a wonderful um very grounded and kind of realistic uh kind of look at everything um very very hopeful and yeah thanks for thanks for stopping in all right thank you this is a lot of fun lot fck [Music] to