Good afternoon, good evening everyone joining. Welcome to another live session at the Reactor. Before we get into the session, please take a moment to read our code of conduct.
We are all here to learn, so please be respectful of other people's views, understanding of differences and be kind and considerate in the way you engage. The chat will be open throughout and we encourage you. to participate, leaving your comments and questions. All right, today we have an introduction to generative AI with Corey.
Hello, Corey. Good afternoon. You're on mute.
Yeah, that's a classic thing. I don't know. So welcome, everyone.
Did we want to throw Mark on as well, or is he going to do a quick intro? Mark is much better. He's going to check that he's not on mute. Okay, great. So we're good.
I'm not on mute. I'm not on mute, and I can do a quick intro. I can certainly do a quick intro. The floor is yours, Mark.
Thank you. Good. Okay.
Welcome, everybody, and welcome to this session. This is London Business Analytics Group. I'm the organizer it's a group that's based in London and I'm sure that many of you to be into our in-person events we're doing a mixture of in-person events and online events we've got some great events coming up in June and July in person so I hope that those of you that can see you in London in June we've got a power bi talk by Prakash Jain and in July we've got an Excel talk by Alan Murray so not yet up on meetup but i'll have those details and we're in the reactor in london but we're starting at the ai series and who better to start off with corey corey is one of the person behind the microsoft uh generative ai for beginners curriculum which is absolutely brilliant so i can't think of anybody better to give us a talk introduce us to that subject so corey without further ado over to you perfect thanks mark uh and i'll I'll apologize for two things. This is a part of my streaming brand now, it seems. One, my video could be really laggy at times or even now.
We are working through that with the platform that we're using for streaming. And the other part of my brand on streaming, especially around this time, is that you'll probably hear children in the background. Those are my kids.
They're very happy kids. No, even if they're screaming going on, it's all out of play. So that's it. I always get some comments in the chat when...
someone says something or hear something and they're like, oh, someone needs to watch those kids or something like that. So that's it. But before we get started, I'd love to see where everyone's joining from.
I see some people in the chat, which I'd like to see. I got Michael from Texas, Ryan from Virginia. So even though this is a London business meetup group, it's great to see a global audience, which is amazing to see because this is a pretty good chat in the market.
We have this private chat, Mark from London. He says, which is great. Hello, Rose. Nice to see you. So yeah, some local Londoners as well.
He's a part of the group. So this is going to be Introduction to Generative AI. This is what Mark has asked me to title this. I imagine everyone has heard or has some experience with generative AI on different levels.
So I want to try to keep this open and universal for everyone in terms of their experience levels. I want to, one, just to see what people's experience levels are so I can know generally what details they have there. But when we talk about generative AI, and I've given a few talks such as this, I think we have like three different reactions, which is the starry-eyed, meaning I'm really excited. You're probably already doing something with generative AI or, you know, you're embracing it. You're using different tools.
Maybe we have this confused face, which is kind of like there's so much out there. don't know where to get started, kind of confused. I came to this workshop, hopefully getting some guidance, or which is the saddest emoji of them all is the sad face, which is basically saying, you know, there's so much going on and I haven't done anything.
I feel kind of behind. So let me know in the chat how you feel. Do you feel, you know, you're working with tools now, and maybe you're building tools with General TVI, your experience level. Maybe you're confused, you don't know where to get started.
Or you're feeling a little bit sad and maybe feeling behind. So right in the chat, so I know kind of what the temperature check is on everyone here. And also someone from Sweden, which I always like to see. Thank you. I can't actually say your name on there.
But all right, this is great. You're actually responding with the emojis, which is even perfect. So Michael has a sad face, which is sad to see as on brand. And Sam, starry-eyed, which is great.
Rose, very enthusiastic, but feel underskilled. So hopefully we fix that today, Rose, a little bit at least. Krunal gave me a little two emojis there. It's the sad face and this like, I don't know what to call that one, Krunal.
Maybe you can tell me. It's like the urr face, the really weird emoji. And I'm feeling, some people are feeling excited. So that's really great to see. Good mix.
Like I said, I'm trying to try to fix the sad faces too. Everyone hopefully at the end of this has starry eyes and something. Maybe that's the end goal. Rowan, sad face and the confused movie.
I love the activity. I'll be reacting to these as I go on. But love it as well. So I think, like I said, I've been at Microsoft enough for about two years.
And I would say, you know, the first six months at Microsoft, we were getting heating up on the world of generative AI. And then things took off. Things took off because of ChatGPT, the OpenAI models. And if I were to say now, in this kind of year since that all happened, is that a lot of people are asking questions because, you know, while things have changed, like people first, it was like very new technology and everyone wanted to just kind of know what's the new thing out there.
Now, I think if we talk about generative AI, there's kind of three distinct areas of why you might want to learn about it, why don't you use it. And the first one is the ubiquity. So it's almost everywhere. And if we specifically talk about Microsoft, we're building these co-pilot technologies. And then almost in all of the products that we have now, where you can basically work with a generative AI model.
and interact with it to generate or ask questions about certain documents, generate new information, process the data, things like that, all within the application. So Ubiquiti, it's almost everywhere now. We're getting everywhere in terms of the type of applications that we're using.
Also, this multi-modality coming up now. So yeah, I think, yeah, I don't know when, I can't remember exactly the day GPT-4 came out. If you know it, write in the chat exactly when, or you can look it up.
when GPT-4 was released. Actually, I should know, have this tattooed to my memory, but I forget. But multimodality, right? So you can take a model like GPT-4 and use it with vision. So you can even ask it to say, you know, what's in this image?
So not only just generating images, but finding information about that, or even voice, text-to-speech, things like that. So even when you go to like a chat GPT, you can give it not only just text, not only asking it stuff, but even data now. And you can get information on that. And then the last one is mainly about autonomy, right?
Not necessarily just specifically about robots, but how do we as people trying to get job done, how do we fit into this story of working with generative AI? How do we use these tools and not lose our autonomy in terms of the processes and things that we build up, you know, we do for our day to day work or daily productivity or even the lifestyles that we build, right? So I think these are kind of the three areas now when we talk about what introduction to Gen and CBI, that's kind of the areas that we're looking at.
And we were saying maybe a year ago, it was more just, what is this? What are large language models? I don't know.
I'm kind of confused. And maybe you feel sort of like that as well. And we're going to cover some of those topics, but I also want to kind of add some freshness to this introduction that we have throughout the time.
Looking like the chat, so some people feeling confused as well. Mark's sad face. So I'm seeing this is not the way, this is not how I planned to start the day with these sad faces. So like I said, hopefully get some starry-eyed converts after this.
So what I think also when we talk about generatory AI, and especially when I talk to people who are looking to ground themselves in this technology to understand what's important, they ask the question, you know what? really can generative AI do for me? And that's, you know, I could probably just throw a whole bunch of, you know, generic use cases at you and you can kind of see, hey, this is what it can do.
And you can't really understand that what it can do for you. And by you, that could be your company, that could be you and your job, could be you and your life. But generally, there's some areas where generative AI performs really well.
And it's in these four key areas. First is this content generation part. I pulled in a few things, some use cases that we've seen from either Microsoft customers or just use cases I've seen out there in terms of working with these models that we're talking about here. And from a company side, we've seen like content.
So whether they're doing like call center analytics, so even automatically generating responses to customer queries, things like customer service, things like that, which were traditionally, you know, we'll say traditionally, but, you know, we've seen this chatbot experience probably for many years now. But we see now with these large language models, they have a bit more conversational tone and also a bit more adaptive into what they're reacting to the. the customer and how they're responding.
So we get a bit more in-depth user experience now in these call center or chatbot type of things when we're using generative AI models. We also see things like generating content within, whether it's a blog post or search queries and taking that data and basically giving that to the generative AI model to generate content that might be geared better to our users or customers or even to ourselves, right? Another big one that I use personally is code generation. Hopefully, let me know in the chat if you use GitHub Copilot.
I hope everyone, even if you consider yourself a developer at once, has used GitHub Copilot or have heard of it or used something similar. There's a few other tools out there as well. But we're seeing, even from the code generation, not just in your IDE, but also being able to open the world up to non-developers in terms of generating code.
So whether it's using natural language. So. So, you know, me just typing out something and I can generate SQL queries. I know this is a data audience, so I'm sure you've done some SQL queries before.
Or even when we're talking about proprietary data models and proprietary data, you can use these generative AI models to generate these queries that you wouldn't necessarily find out if you were to use, you know, a standard tool or, you know, standard documentation that's out there. Another example that we're going to dive deeper into later on is semantic search. But being able to, you know, the world has lots of data out there, some of it unstructured, some of it structured, as we probably have known in the world of data analytics.
And using these models to basically search on that data without necessarily needing to do as much data structuring that we would possibly have. So in the case of here, we have like an insurance company that would use it. be able to get information from this unstructured data to process claims.
So you can understand, especially when we talk about the multimodality of these, let's say I'm writing on a piece of paper or something, some notes, I could give that to a model as a vision model that has vision enabled and being able to understand what that could say on the notes, and then even present that into a structured way for using it within another application. Summarization is another big point, right? In terms of being able to process logs and other data and or reports or analytics and providing key insights rather quickly on that data, whether it's for finance, for personal use, government use, things like that.
These models, generative AI models that we're talking about are, you know, excel quite well with that and also give us an opportunity for as users. of these tools to also participate if we want improvements on the summarization or finding insights rather quickly for us to then look for key areas where we need to take action on. So another big summarization thing that I personally use is whether it's using Teams, for example, we have Copilot integrated in that, and it summarizes all the meetings, and I can also summarize on specific action items that I need to take.
So these are, you know, To answer the questions, and these might be answers that you already have, but this is key areas where we see generative AI being really productive and really effective in terms of applications and even building applications, whether you're a developer or not a developer as well. And what has really changed in terms of the application landscape then? So we've coined this term intelligent apps, which is not necessarily just to say that the other apps aren't intelligent.
but that they have some sort of key difference when we're talking about working with generative AI models. And one of these is, you know, if you look at current applications, they're sort of constrained to the interactions that they already have, whether it's, you know, fitting this text box, write something, where intelligent apps can have this natural language interaction. So you can describe the problem that you're having, the goals that you want to have as a user.
and the model can interpret those things and then bring that about in terms of where that information is going to be. You know, again, we have these hard-coded data sets that are out there in very fixed terms, but then also we can bring about having a better personalization. when we need this sort of improvement over time.
Because getting, we're working with data, data changes over time, and we don't want to have, you know, just a standard way that we're working on and just data there. And then lastly, we can sort of bring in the results rather quickly, whether, you know, we're using the model to interpret new insights, new data. So this is what we call the paradigm shift, which is like, a bit of a buzzworthy type of thing. But I think if we start to look at the applications and use cases that we see, it's quite different to what we've seen before, and even as users or application developers.
Perfect. The test work, London Business Analytics group, I see you in the chat. Now that we looked at the use cases, in terms of actually like, looking at how to get started with this stuff or even build on it. This is kind of a general value versus complexity kind of chart.
I know this is an analytics group, so try to appease the audience here and give you some nice X, Y, X is V, X and Y axis looking at things here. Summarization in QA, it's gonna be highly, it could be highly valuable depending on use case, but complexity is rather easy to implement or use. But as we go up, we have like data-driven decisioning, a little bit harder, right? In terms of analyzing the data.
But again, you can get some valuable insights even on unstructured data. Personalization is another one, right? So how do we tailor these interactions to certain users?
Whether that's recommendations, interactions, and we're going to talk about some of that. And then last biggest one, it's like most value and most complexity is automation, right? So how do we get these models to automate repetitive tasks that we might not necessarily... like to do day to day, but it will make us more productive because it's some test that we need to be completed.
So you can kind of think about if you want to ground yourself and like, okay, how do I get started as a user or even building stuff, right? Maybe summarization and question and answer is where you go. And speaking of question and answer, do feel free to ask anything as we go on in the chat.
I'm watching the chat on the here. So when I'm looking here, I'm looking at you in the chat and I'm looking here, I'm looking at my presentation. So I'll take any questions as we go on, because I know watching these streams can be difficult.
If you have a question, you're like, oh, I'll wait till the end. I can see the chat and I will ping you. I'll answer anything as I go on.
So if you're a developer or you're a technical, you probably know that, like, I showed you this pretty graph and like, look at the use cases. It's so easy, you can just get started. But in reality, we were talking about building these applications, there is quite a lot of tooling involved still in terms of how you want to do this effectively, whether that is using some of the pre-built capabilities that tools have or models have in cases of, let's say, Azure OpenAI, like you're working directly with the OpenAI models.
And then in terms of getting value, we have things like Microsoft Fabric and Power Platform in terms of... being able to deploy and build applications rather quickly. But then even when we go down this kind of stack here, if you will, we have orchestration tools that you'll see, especially if you want to get working as a developer, where we're allowing the model to basically work through certain processes and give us basically visibility on those processes it's doing or interacting with certain types of data. So... For instance, Prompt Flow is an open-source tool developed by Microsoft that allows you to see essentially the flow from both when the user prompts, the user interacts with the model, your application to the response, where does that go in the next in the flow?
Then it's really good to be able to visualize things in terms of that, and as well as evaluate when we talk about that. semantic kernel, LangChain, are also orchestration tools. They also allow for integrations with other tools like, let's say, GitHub or where documents are stored, and allows you to then orchestrate and to get an understanding of what steps needs to be taking in my application next after certain things happen, or certain actions happen.
Then lastly, which we'll talk about even more in depth, is the model. There's a lot of models out there, which I think, is a cause of confusion when we're talking about some of the people who had the sad faces. Maybe you're confused because there's so many models coming out.
And I like to say, especially when we talk about developing applications, whether even though there's a new model that's out there, you don't have to like throw it away like a consumer device or like, oh, you know, I have to use the latest model for everything. There is still a lot of room. So for example, like Llama 3 was recently released, doesn't make it that Llama 2 is like, you know, the old device that you don't need to use anymore. You got to just go to Llama 3. Like Llama 2 is still used, still has some value. So it's one of those things that you have to kind of understand and navigate the world there.
And that's what we're going to cover in today's chat as well. And then if I go to the AI stack as well, like right, there's APIs that you need, you need to deploy them, make sure you have consistent data access. So if we're giving the model access to data, make sure that that's available, but we're gonna go way more into depth, but I just wanna give you like, if I go to this slide and someone says, hey, well, this looks easy.
This is kind of like pulling that under to show you all the parts, but you don't need to know all of these parts to get started and you don't need to know all these parts, especially if you're a user. I see some questions. So Hugo Wing says, I thought Visual Studio Code is all enough, and it certainly is enough to start coding for sure.
But to get all these other things going, you might not do that. And Andy says, where does the model begin? I see that they all interact, but would this start in the top row?
It's a good question, right? So you see here, there has this like, I didn't make this personally, but you see this up and down chart, because what we'll find as well is that, this could use, let's say you started choosing a model. Let's say, oh, I'm going to work with open AI models. What in reality is that you might want to switch or change the models depending on cost or effectiveness.
So you might actually go up and down this stack. So there is no place to start per se. You know, what we see a lot of teams doing if we're talking about development is they will start at the open AI models.
They might use a tool like semantic kernel Langchen and then define a use case. And then they realize maybe they don't need some parts of semantic kernel that they're already using. Maybe they want to limit the cost and use an open source model like Lama 3. But again, it's an up and down kind of thing, right? It's not, you shouldn't think of this linearly for sure. And then Dan has a really great question.
How to choose a model? And that's one of the questions that we're going to answer today as well. So stay tuned, Dan.
So. There's so many things to talk about with generative AI, and I could probably do a four-hour chat on this, but we only have an hour for this conversation. So what I want to do is kind of limit it to the things that I think are parts that when I was starting out here, I got confused on, and I still have questions to this day. I know that this is a very evolving field.
So while I can probably answer some of your questions, you know, there's... these things could change, the answers could change, how we build with these things, these tools can change as well, these tools will change for sure. But I think these are kind of areas. So to Dan's question, right, there's so many different LLMs how to get started. So I'm going to like kind of talk about that.
I'm going to talk about prompt engineering, which is something that even if I'm just a user of these applications is important. I want to talk about responsible generative AI, how to do this responsibly, either building applications, securing generative AI applications. So a lot of times when I speak to technical audiences, they're concerned and rightly so about the security around using these tools.
And then search applications is a big use case. And I think it talks about this audience about data analytics. I think it's a one to really drill down in.
And then lastly, as a kind of a bonus, but kind of a future forward look, but also present day looking at AI agents and what those are and how we can use them. as users or even application developers. Like I said, this is a large field. Mark has already gave me the shout out, but I've already put it in here. But we do have this generative AI for beginners course.
It's a GitHub repo actually, so it's completely free and open source, so you can also contribute to this. But you will see that we have a lot of lessons now and it continues to grow. We have learn lessons, which are lessons that don't require any coding at all, and build lessons, which are mainly geared around building things, building different types of applications. So I do would like to say that this is a good place as a sequel to this workshop to go there and explore more, because I won't cover all of the topics that you see here, and I won't even cover the topics all in too much depth.
This will be kind of a survey of all of the different things I just showed you there. But if you want more information, do check out that repo. But let's start with exploring different LLMs, which is a question Dan asked, and it's the question I ask all the time.
How do I choose a model, and why are there so many models out there? Well, if we want to break down the world into two different worlds, two world model worlds and two different things, I would say there's the proprietary models. And for Microsoft purposes, we primarily talk about the open AI models in this case.
So this is probably familiar faces if you've done any work in this space before, like a GPT-4, GPT-4 Turbo, Whisper. We have the older models, if you will. that open AI model like DaVinci and Babbage. Then we also have these open source models like Lama3. Mistral has a series of models, Command-R, which is a cohort model, and I've grouped up Hugging Face models that aren't actually made by Hugging Face per se, but are models if you've ever done any work with machine learning and AI before you probably are familiar with Hugging Face.
But this is essentially a catalog or a hub for open source models that are out there. So this is kind of a breakdown of those two worlds if we talk about generative AI models, and it continues to grow in terms of open AI releasing new models and open source models being released. Now, how to choose the right model, and I'm glad I actually named this slide the right name. So a lot of times we say, okay, proprietary versus open source.
So do you want to use open AI models or do you want to use open source models? Open proprietary models, by and large, perform better than open source models in certain cases. Not in all cases. And open source models, if you look at certain use cases and certain tasks, perform better than the open AI models in today's world, as I'm saying this. That could change next week.
You never know. A new proprietary model could come out that could wipe up, you know, perform better on different tasks or different metrics. But... That's largely that it's owned by a particular company.
There's a limited license in terms of what you can do with it. Open source models, on the other way, has more free use in terms of what you can actually do with the model, allows for different modifications, especially with fine tuning, which some proprietary models also do, but they're usually slower in terms of releasing that feature. They're also cheaper by comparable by the performance.
So if we take like a cost per... per performance ratio, you will find that especially specific open source models that are out there are cheaper to utilize and especially in applications. And that's largely not because they're worse, right?
Because sometimes you think, oh, that's cheaper, it must be some sort of catch. But because either their architecture is different, again, their hosting structure is different because they're open source models. And open source also has an advantage that they're sort of community-led in terms of innovation. So what you will find, and I'm going to give you an example later on, is that this community-led innovation allows for models that are fine-tuned for a very specific task. So let's say you want to use a model that can do text-to-text, text-to-language, you could find a fine-tuned model that does that really well.
In reality, though, and I think in the past year what I've seen a lot of teams build with is proprietary and open source. So this is actually using multiple models and using them for their best abilities. And also, again, it's just spread out costs in different ways.
So give me an example, like model task. You have a summarization of like you can use one model that performs really well in summarization or the summarizations that you like. So, you know, if I use a GPT-4 and I'm a straw model, like the summarization style is pretty distinctly different in terms of length.
in terms of style delivery and you can probably adjust that in prompt engineering as well but and the whole you know you can kind of look at that um you know even doing text to sql or other languages um is also another thing that models can be very specialized in doing and we'll talk about search and retrieval augmentation augmented generation as well well later on but that's also important and we'll talk about um different things like context with windows so um Use case is going to be really important here in terms of knowing, even talking about latency, so how fast the model responds to queries and things like that could be important, especially if you want lower latency for a certain use case. Also, the expected response type and length. Like I said, some models tend to deliver responses with more length than others.
basically how much data our model can sort of process in terms of the prompt, as well as any data that we want to give it to supplement the prompt and then respond back, which is also going to be kind of important things there. Adam says, can you use generative AI for time series? I'd love to know that use case.
I mean, if you want to generate some synthetic data or you're trying to predict or use time series data. I mean, I think if you go to Hugging Face, I think there are some models that do that really well. But I'd love to know a little bit more about that.
Thanks, Adam, for the question. But just to give you this understanding of like this proprietary and open source. So in the open source model, you can use actually like...
build models sort of like Legos. So let's say I have this, I'm going to use a very local example because I'm based here in Stockholm, but let's say I'm having a Swedish site, a website, and all the content is in Swedish. And I have a model that maybe like chat GPT 3.5 or GPT 3.5 or four really performs well in Swedish. And I'm happy with those results, but the finished content isn't as great.
So I want to make sure that like users can still have the same experience in my website. if they speak Finnish as they would Swedish. So I could actually go in and let's say I take a model, in this case, this Helsinki model that's specialized in translating Swedish to Finnish.
We can translate that and then use just basically a Finnish GPT-3 model that's really also fine-tuned and specialized for that to generate Finnish text, and then use a Whisper large model, which is also a fine-tuned version of Whisper that does speech to text. and I can have basically a customer service bot that uses basically three models in this flow to have a fitness speaker get great customer service that they would if there's a sort of Swedish there. So I think that's one way you can kind of build with multiple models. And I think this kind of architecture is kind of growing in terms of specific use cases when we're talking about using either proprietary and open source or a combination of open source models. Cool, time GPT.
I think that's the one I was thinking about. Thanks, Sam. There's quite a few out there.
So prompt engineering is how to get basically these models perform and complete the tasks that we want to be more effective, I think. And it's available to everyone. So even if you're using, even if you're not technical and you're not a developer, prompt engineering is something anyone can kind of learn. And this is kind of a skill, I think, in terms of interacting with any type of tool that uses generative AI. And there's tons of resources out there in terms of giving you very specific, like, oh, list some details here.
And fine, if you say this and not this, it works really well. I'm not going to give you the whole world of prompt engineering. But I want to say there's basically two areas of focus on making effective prompts. And those two areas are context and constraint. So context is basically this.
You can give the model, whatever tool you're using, a role. You could say you're a helpful assistant. You could say you're a senior business analyst.
You can say you're a developer advocate, whatever you want. That actually gives the context and gives the model a little bit more understanding of what the end's goals might be. So it's a very effective method.
Also being very specific. on what you're trying to do, the content that you're trying to give the model or what it's trying to work with is going to give you better results than if you're just very vague. So I like to say, if you're trying to give this to maybe, let's say, someone that's new to your industry, but has worked in that industry before, you need to give a very specific task to your job, then that's a good method of doing that. And then also very being very open about the end goal.
So what are you trying to achieve? Especially with, let's say, if I'm doing code, like I would like to say when I work with GitHub Copilot, for example, I'm not just saying, you know, generate this text or generate, you know, an array of this, but generate this array because we're going to use this and we're going to loop over this later on or something like that, just to give it more context there. And then supporting content is a very catch-all, but... Giving examples here, whether it's one, two, multi-shot is what they call it in prompt engineering, is going to be super helpful for the model.
So if you have content that is already reaching the goal or something, the style that you like, providing that supporting content is going to be very effective. Then, you know, once you give it all that, also constraints is another really good method of prompt engineering, whether that's formatting. So asking the model to deliver a certain format, whether it's like... Do this in a bulleted list, do this in an unordered list, do this in the table. Formatting is a very great way to make that useful for you.
Also providing the length. I want to summarize this text, but only use one paragraph or something like that. And then prioritization is really important. So even telling the model what to prioritize, or let's say in a certain style. So if you're generating text and you want to prioritize humor, for example, giving that directions and instructions is another great kind of method generally or practice around prompt engineering.
And then this is the last one is actually giving the model an out. So in most interactions that you're going to have with these tools or models and so they're going to try to respond with something. But you can always give in your instructions to the model. So a prompt is basically you when you interact with a tool or model directly, you can basically give the model out saying, you know, if you don't know the answer, don't try.
Don't try. Say you don't know. Or, you know, if this is too much processing or, you know, this this logic doesn't make sense. also point that out.
So always giving the model an out to talk about, to reach your goals. If it can't actually feel a level of confidence in terms of providing an answer is another really great method. Now we're going to talk about responsible AI.
And I tried to put this ahead because I think we have a bad tendency of, oh, I'm going to tell you about all the things you can do with this tools and models. And then, oh yeah, by the way, do this responsibly. So I want to say, like driving a car. Like we don't teach people not to hit people and then, you know, we teach them to drive, right? We want to put that together.
So, making sure you're doing this responsibly is another really great point to focus on and using these tools responsibly even as a user. So, when we're talking about application development, you know, we want to take a human-centric approach to this. It basically summarizes having the user's best interest equals the best results.
Using these tools or even developing these tools, you can really lose the value rather quickly from a user's perspective. Even though we've seen the ability of these tools to create great value, all in either one bad interaction or a bad interaction that people go in and publicize or something like that, or even from a user perspective. So you want to make sure that you take this human-sensitive approach. And impact requires monitoring, right? You need to make sure that you have tools in place to monitor their user interactions, things like that.
And then lastly, intent is not enough. So no one really goes out and like, I don't think at least not many people hopefully go out and say, I want to build like the world's most irresponsible AI application. Like, so even if you say, I'm a good person and I will never make a bad AI application or use it in a bad way, things happen. And intent is not enough.
You need to really design principles and practices around that. And this is all about, you know, potential harms. We know things like hallucinations and fabrications.
So these include things like nonsensical responses or making factual errors, right? Models tend to have these things where they're very confident in their responses. It could be believable, but not be actually factually true.
Contradictions or even presenting irrelevant information. And also, you know, even harmful content, right? You know, having a model produce things like encouraging self-harm, guiding or planning any violent attacks, things like that, or any illegal content. These are also potential harmful content.
And then lastly, some things that maybe other people don't necessarily think about all the time, but lack of fairness. So, you know, making sure that the model output doesn't include any exclusionary worldviews out there for certain groups or bias against certain groups there. This is very much aligned with the responsible AI principles that Microsoft has in place in general for AI. But just to give you more specific genitive AI examples there. And, you know, presenting the harms is one thing, but understanding how to sort of mitigate or measure those.
First, you sort of want to measure those and take an approach that's very akin to software testing. So we have this method called like prompt testing. I might sneeze. Not really good. It's coming.
And we want to make sure that when we do this prompt testing, seeing how the model responds to the prompts that we might want to see or use or our users might want to use, we want to aim for diversity here and not just the perfect scenario. So when we're building applications, we want to make sure that we put in scenarios where things could go wrong or, like I said, the potential harms could come about. And also we want to make sure we start manually here and then scale to some sort of automation of that.
So. really start with some simple prompts, see the responses, and then building it with either tools out there that can automate these sort of prompt testing things. And this is like the layers of mitigation to kind of mitigate against those harms that I explained to you earlier.
And the first one is the model. You make sure you use the right model and the right use case. And the right model could also mean you don't necessarily need to have like the GPT-4, the most powerful model out there, for example, or the most powerful model to get the best results.
Using a less powerful model for that's, like I said, very useful for a specific task. allows you to have a bit more, allows for better results and effective results that are more responsibly aligned to that certain use case. So definitely explore models, especially when you're, you know, after you've kind of iterated or have a proof of concept, because later on, then you could see how other models perform there. And then knowing your parameters of the model and things like that is very important. Having a safety system.
So within Azure AI, for example, we have this content filtering. that's built in. So we're making sure that we're filtering out on certain responses from the model, but also we have things that are responsible AI that has a different scoring and metrics. And like I said, using the model monitoring, things like Prompt Flow allow you to do that more in a scalable fashion than just requesting something from the model and getting something back and seeing how it goes.
We also have this idea of the meta prompt, which is kind of Being able to prompt or design some rules and behaviors around the model and allow it to also ground itself in certain contexts, which is going to be important then to limit the model's behavior and also focus on a key, let's say, data set or use case. And then last is the user experience. So having very transparency to users that they are interacting with an AI model or even constraints on input.
So allowing... even checking what the users are sending in the model is going to be another good way to do some validation on that and limited the amount of harms that can happen or mitigate the harms there. So now that we're talking about responsibility, security is another big one, right? After we've, you know, want to deploy applications or build up with those, how do we make them secure? I'm going to skip this slide, but it just shows you, right, the impact of generative AI in the market is quite high, which makes it clear that there's an opportunity for insecure or attacks against these tools that we use in the models as well.
So OWASP, which is a security, I guess, firm as well as organization that presents a lot of things in terms of cybersecurity and Internet security as well. They've created top 10 security challenges for large language model applications. We're going to look at three more details.
worth a look to look at these all in detail because they have more information on them. But the first one is prompt injection. Basically what this means is using prompts to cause behavior that the application developer or the large language model owner didn't intend to behave on.
So users can come in, craft very prompts and either perform unwanted tests that we don't want in our applications, maybe have better. vulnerabilities in the models themselves or respond again with bias or the misinformation or hateful content there. So be able to mitigate that, as mentioned, input validation.
So even if you're a user, you might experience this where what you type before you even get sent to the model gets validated. Monitoring, right? Logging user inputs, logging responses, even adding watch lists for actors that are known actors that are violating certain applications. is a good way to sort of mitigate those attacks.
Then we have this idea of supply chain vulnerabilities, which isn't necessarily just like supply chain of GPUs or whatever, but actually the tools that we use to build these applications. And like I said, the space is moving so fast that even things like software packages and libraries tend to get updated quite frequently. And they might be updated because security patches or things like that. So making sure that you're using the most secure and not outdated software is going to be very important when you're talking about building applications. You might not get that visibility when you're a user.
But also things like plugins and tooling, right, when we plug these applications into databases or into areas where, you know, there's very confidential information, making sure that the tools and plugins that we use are also secure. Dataset poisoning we won't necessarily talk about, but that's also something around. the data sets that are used to train these models. And also to sort of, you know, kind of mitigate those problems, you know, we want to verify the plugins that we have, or even build their own plugins if you need to. We want to make sure again, that we had used the most secure version of that, or even doing some adversarial testing, like AI rate teaming.
So we're testing applications against adversarial attacks and seeing how they respond. The last one I'd like to talk about is over-reliance. So this is generally overly trusting the output of the models. Cause like I mentioned earlier, right?
These models are prone to hallucination, fabrication and maybe even unsafe answers in certain cases there. So generally how to mitigate these things is, generally just user education, right? We need to make sure that we have a keen eye on where we get responses from an generative AI application or a model.
to have a general level of skepticism or a level of understanding that this model could be prone to hallucinations. Like I said, output verification is going to be a big one, making sure that we're monitoring that when we're building applications, and then doing some testing and evaluation even before we launch this to users to make sure we present very relevant prompts towards the specific use case that we're building so that we know where any of this over-reliance could occur and making sure that we mitigate those harms. before we even get users developing on it or using this for a certain segment of users and then getting the testing results there.
So search applications is a big one that I wanna talk about. I think this is the most, if we wanna say like this is kind of the hello world of building AI applications is this idea of RAG. Let me know in the chat if you've heard about RAG.
I think for me, I feel like everyone knows it but that's just me being working in AI for too long I guess. But basically this idea is we're going to be able to give the model relevant data towards the certain user or the certain question or certain context the user is operating in, rather than just relying on what the model has been trained on. So if you ever ask ChatTPT, let's say a question about, you know, what's the latest tech news that was just released, product release, it's not actually going to know that because the model has been trained on.
a certain amount of data and it has a certain cutoff date. One of the things is, how do we then give that model relevant information, whether it's data that's to the user or basically to information that might be very important for the user to get a relevant response to. That is where we have this rag, retrieval augmented generation, and the R stands for retrieval. we're going to retrieve related data. And this could be, it says here knowledge base, but this literally could be anything.
It could be a certain document. It could be a code repository. It could be an Excel spreadsheet, things like this.
But we can give the model that information and augment the prompt. Like I said, the prompt is how anyone is interacting with the model. This is kind of the question, if you will, or request for data, whatever. And we're going to basically augment it. So we're going to put in that information before it actually gets to the model itself so that you can actually get and say, hey, model, and basically in the prompt, hey, model, this is some relevant information for this information, that data that you could ask or answer the question with.
And then we want to generate a response. So the large language model is good at kind of parsing through this new data that we've given it, looking at the question from the user and saying, hey, great. Here's the response. And we can even tell the model to only limit itself to that data. So again, we're not necessarily pulling out information from the internet or the training model that it's gotten in, training data that's received.
So it really gives a very core personalized experience to the user. And then lastly, we just send the results to the response and we show the answer or the response to the user that they're looking for. To give you a little bit more context of this.
So, you know, we have this user, you know, maybe we have like a search service that's in between there. We have some data sources, like it could be files, like I said, it could be databases. And then we want to make a context aware, a prompt and response. So I gave this example to a telecommunications company. This is why I'm talking about iPhone here.
But you know, let's say I want, I need help with my iPhone. And instead of just saying like, oh, great. I don't know anything about your iPhone.
You know, the agent, in this case, the large language model can come in and say, you know, hey, I'm happy to help them see that your most recent purchase with the iPhone 15 because it's in the database. Customer users, a data product, and then also the product catalog. So they know that, you know, okay, here's their product catalog.
We have all these phones available. And let's say if I were to purchase a new phone, the dynamic data part of this, I can also see what the recent purchase of that user is. So instead of having this static data that the large language model is open to, we can give it more dynamic data and data that could be relevant to the user. So if I put the data into the prompt, does the LLM retain that data?
So not the large language model in itself wouldn't necessarily retain the data. If you were to say like another session or another chat session, the user would just come in and you would need to then provide that data later on. In this case, we're talking about like a cloud environment, right? That data could be housed somewhere else in a database. And then, like I said, you need to use this method, this React format.
again to get the relevant data that the user would have. And you can do that through using like a search service. So if I know iPhone, I can, you know, search all the iPhone catalogs and things like that.
But great, great question there. So lastly, and kind of rounding up here, we only got about eight minutes here. It's this AI agents part.
And there's a fire alarm, a smoke detector going on with my house. No smoke, I hope. So AI agents. I like to show this example.
So this is on the left side is Eliza, which is like probably the first, I think is known as the first kind of like chat interaction, chat bot type experience. And this came out in like the 60s, I believe, if I'm not completely mistaking. This is one that was like this implementation was in 2005. But.
If you look at ChatGPT, it's not too different, right? We're typing in something, we're getting a response, we're getting an answer. And that's kind of how we've worked with interactive as users with AI models in this chat-like feature, chat-like user interface. But if you wanted to give the model a bit more control or a little bit more ability to execute tasks rather than just giving us responses, is this idea of AI agents. And there's kind of three areas that AI agents comes about.
First is like tasks and tools. So we give basically large language models access to these tasks and tools, whether I'm sharing here VS Code, Edge, whatever. But basically, if we look at a large language model and it's interacting with a user, and it says, oh, I would like to schedule a meeting, rather than it just saying, yeah, here's a great meeting agenda, why would it get access to my Teams or another tool, the calendar, and be able to schedule that meeting? And different agent, AI agent frameworks allow either for a user to approve or execute that function, for example, beforehand, or give it sort of autonomy in terms of processing that and showing the results.
That's like kind of one way, right? Is giving it tasking tools. And a lot of these, if I talk about orchestration tools also has AI agents built in.
So semantic kernel on Langchan, if you look at the agents, they basically are giving it tools, whether it's even a search engine to search out in the internet and then using that response from the tools to give back to the large language model to report back to the user essentially. Some of the more future run, I guess more research-based ones, as well as ones are you're going more into production now, allow large language models to do a couple of other things, whether that's reprompting. So basically a large language model gives you a response. And then you say, based on this grade, this response that you've given me and then improve on it. So this is the idea of self-improvement.
There's a lot of research going on now of how effective that is. But I would say that's probably the most in its infancy in terms of making a framework. The other one is I'm seeing more and more in terms of development come into is using the large language model to work with another large language model, whether it's based on a role or purpose. So in the case of we have this framework called Autogen, which was grown out of Microsoft features research, but it basically is large language models, a certain role, or you can give it basically a system message saying like, let's say you're a product manager and the other large language model is a developer.
And then you want to like, okay, create new features on this product. And they can kind of interact as these roles and maybe make a better and effective feature list than if you were to use one large language model and say, act as a developer and then give me your results. And then even on the purpose side, like these large language models, like I said, that were specializing in certain purposes or tasks, you can also give that, have a large language model sort of orchestrate those things.
But AI agents is something that's very interesting to a lot of people. And I think a field that's sort of growing in terms of getting it to be more effective, especially around the different agent frameworks. So I do encourage people to explore that.
And lastly, I'm going to end with this and I'll see if I have any other last questions going on. But like I said, this is a new field and there's a lot of people interested in learning. Like as yourselves who joined this, I'm assuming that's why you came.
So we have this AI community Discord, which you can use the QR code as well as this URL here. As I like to say, this is a community of learners. We do, you can ask any questions. There's no. dumb questions or beginner questions or anything like that, whether it's you trying to build something with generative AI or you working with a certain tool. And then we also have community roundtables on certain topics.
We're going to start also having office hours. So this is a great resource and you're meeting with other learners. So I'd love to all proceed the community activity around people answering questions, asking questions and getting answers from other community members.
So thank you everyone for joining. I would say you could be anywhere on the internet right now, but you spent a whole hour almost with me talking about generative AI. I'll take some questions.
I'll look and see if there's another question here. Sam has asked, what's a good generative AI model for mathematics, finding roots, cubic equations? So I know some of the Mistral models are really good at performing different mathematical applications or operations, but I would say, there's probably a fine tune model and I can't give you the name on top of my head, but if you would go look at hugging face, um, you could probably find a generative AI model that's fine tuned for very specific math operations there.
Um, but yeah, Mistral, I think I've seen some good, uh, results from their evaluations on there as well as, um, giving, even getting a fine tune, maybe a fine tune Mistral model to do so. Um, great. So I think that that wraps up then.
Like I said, thank you everyone for joining. I appreciate the questions. I thought you can see here on my little name thing, Query Space, that's where I'm at on the internet in most places, LinkedIn, Twitter, X, whatever. And I'm also in the Discord.
So feel free to ping me. I'm always open to answer any questions you and chat about Genesee BI. That's my passion and also my job. So I'm very privileged.
Like I said, you can be anywhere. Appreciate your attentive eyes. And yeah, we'll see you in the next one.
If Mark, you want to jump back on, you're free to close out as well. Thanks, Corey. Thanks, Corey.
That was an absolute brilliant, absolutely fascinating talk. Thank you so much for that. I'm being cool in the middle of a smoke alarm. I'm amazed. Yeah, yeah.
No fire. I mean, I would have freaked out by then. I'm glad you're not.
Yeah, absolutely. It was an amazing talk and you know I learned so much. The idea of Llms having a conversation is one that has never occurred to me.
Can I just present something which I'm going to share in this screen with two monitors. I want to share, let's see if I can share my screen. I'm going to share this screen here and just to say you've been looking at the London Business Analytics Group. We have got lots of other good talks coming up. I haven't got it but we've got a series on ai starting in the autumn a lot of that will be online here's a group you probably came for it so you know it anyway um and as well as the group we have a um also a youtube channel hopefully this talk will once it's been recorded will be on there as well as other places so i will what i will do is i'll put the the things in there as well into the um into the chat window but again corey thank you very much uh thank you everybody for joining uh we'll please say that and if i can very quickly just well no i do join us on the on the youtube channel do join us on meetup and what i will do is i will put my those two links and my links as well into the chat window if anybody wants to get into contact with me to maybe offer to give a talk or anything like that or whatever so there's the youtube uh and the meetup in there but that's great except the comments has failed to post my microsoft reactor okay never mind about that again thank you very much corey again thanks again bye everyone thank you and thanks to larissa for keeping us organized and being a great moderator and organizer of this event thank you everyone thank you mark thank you Thank you for everyone for watching our live stream today and we'll see you on our next session here at the reactor.