Laag Overview and Application in Software Testing

I'm going to launch my script and I'm going to pass it the URL as well as the feature file so pretty simple that's what you see on the left and here the feature file gets passed laag immediately starts and we give a simple instruction to our agent we say run this test case and we pass it the scenario you can see here what you see highlighted here is the reasoning behind laag laag uses computer vision to take screenshots of the page and reasons on what it is on the page to generate instructions those instructions are going to get transpiled into selenium code that's going to get executed so as you can see now we're already at the second cycle and it's it has already executed the instruction to click on the cart button right according to the test description now since we want to check that the item is properly added to the cart the next instruction is to click on the cart icon which is it has already done so as you can see our test is successful and since we asked laac to Simply run the test it has and now it knows that it has finished running so it outputs a complete instruction in the world of AI where data flows Daniel and Paul from L AI know they craft Solutions with precision and Care revolutionizing testing everywhere with sharp algorithms and Grand ideas they lead the way a Visionary ban their expertise in testing is second to none with Le AI automation future has begun from codes deep Mysteries insights unfold in their hands technology secrets are told join us today for for a deep dive into ai's wonders and how it thrives I'm Nikolai your guide in this quest with Decades of experience and software I've trained the best 150,000 mines in 190 lands together we explore Tex vast Sands guys welcome to the show super excited to have you all hi Nikolai thanks for having us our pleasure yeah where are you guys joining from we're both in Paris awesome welcome welcome from Paris so we like to start out the show with a coding Challenge and I wanted to see if you guys can use AI to create an automated test from a spec file yeah so we can definitely do that even though laag can do much more than we automation this is a use case that you know we think is a great example of the power of laag especially for front end and visual testing I can show you a a quick pre-recorded demo of us generating a test from a bdd file that's so here we have developed a small proof of concept a small script to see a flag which is more of a general purpose framework for web agents we wanted to see if it can be used to solve problems that are faced by QA engineers and people automating the web so what we've built is a script and everything is available on our open source GitHub this script is going to take your standard BD scenario so you guys are probably familiar with the buin syntax and it's also of course going to take the URL you want to test laag is going to be used and eventually we're going to Output a reusable py test bdd file in this case that you can rerun the great thing about that and you're going to see it in action in a few seconds is that if your interface changes or you want to test it on responsive whatever happens like if you need to regenerate a test you don't have to do it manually you can simply rerun this very simple pipeline Reet a new automated test that you can reuse so our file structure at the start is very simple we have our script and we have our feature file following bdd principles as you can see this is the feature we're going to test it's a simple card feature we're also going to use a demo e-commerce website for Simplicity sake and we're going to generate a test to test your cart and see if you can add an item to cart from a product Details page so I'm going to launch my script and I'm going to pass it the URL as well as the feature file so pretty simple that's what you see on the left and here the feature file gets passed La immediately starts and we give a simple instruction to our agent we say run this test case and we pass it the scenario you can see here what you see highlighted here is the reasoning behind laag laag uses computer vision to take screenshots of the page and reasons on what it sees on the page to generate instructions those instructions are going to get transpiled into selenium code that's going to get executed so as we can see now we're already at the second cycle and it has already executed the instruction to click on the cart buttton right according to the test description now since we want to check that the item is properly added to the card the next instruction is to click on the card icon which it has already done so as you can see our test is successful and since we asked LA to Simply run the test it has and now it knows that it has finished running so it outputs a complete instruction now what the second part does because what you've seen was like laag running a simple objective now what the second part does is we use everything that we've recorded during this execution to recreate the automated test so as you can see we're merging one scenario but you could run it on many scenarios and let's see what it looks like so as you can see it's a pest file we do use the pbd library and it contains your fixtures and your steps and everything you need to do including the assert statement right which is what we need to generate on top of L and the great thing is that our test was directly put into our test directory and as you can see I just ran p test and even though it was very quick my test passed and the the file you can see on the left was executed so again the great thing is that if your e-commerce interface were to change or if you wanted to test it on a different screen ratio or anything you can think about you could just rerun laag to get a new pest file that you can reuse according to your screen dimension or what your product team or design team decided to change on the website so we think it's a first step for you know helping the Q engus job which is Sometimes using recorders and spending a lot of time finding the right cross path and so on that's what we have for now and we think it can already be useful in this state yeah that's awesome so let me just make sure that I fully understand is you basically took a bdd spec file and then laag executed that spec file to figured out the instructions on its own and then based on that it generated test code that can now be ran exactly so maybe one small detail that is quite important is that in a sense laag executes the instuction we gave it and nothing else so here we give laag the instruction to run a test as well as the detailed test case so laag does that and it knows that it has to stop when it's on the last step of the test which was check that the cart contains the product and then we use other llms along with everything that laag has recorded So when laag runs any objective you know it generates selenium code it uses Vision it records the HTML nodes it records a bunch of stuff and we use all of those recordings to build a robust and reusable test file so in a sense laag itself does not generate the test however laag is used to record everything that you need to generate the test he friends we're here to ask for your help this show is free it has no sponsors or Affiliates of any kind the only thing that keeps the show alive is your viewership and subscriptions to the channel Mia and I ask you to like subscribe subscribe and share this content with all of your friends so that we can keep growing and I can continue to create even better and more valuable content thanks so much for your time now back to the show bye that's really awesome we'll get back to exploring this in a little bit I got another rapid fire question for you guys what do you think are the careers that will be negatively impacted by AI it's a long answer but basically anytime you do something you don't necessarily need to like think a lot especially like very mechanical task that will be an impact there this long paper and for instance interesting thing is you can see that every task that intellectual too much can be impacted but not so much very manual task which are less well paid and very high level of thinking which are well paid if you are in that you know in the middle you can be impacted but I don't think the AI we have today is that dangerous in the sense that it doesn't have the same multimodality multi temporality creativity and S that humans have the things we showed can have a lot of edge cases I think it's very useful but I don't think it's going to take your job FR who is a big AI researchers that Google said that yeah today if it the threat your job you didn't do a really good job I guess it's not to be offensive but it's more to say that AI is not that intelligent yet people should not be like you know so threatened Z is however need I guess for humans to really think about okay where is your niche in the sense like ecological niche and as I mentioned I think that the ability to think and solve problems where you need different interactions novelty Cod B change tremendously and so on so I think many professions and I guess Engineers included n not just you know a specific generation even though it's a big part but you know even sof Engineers the good ones I think they spend like 20% coding so rest is more like thinking discussion and so on don't think it's going to be too damaging if you really are in the you know solving complex problems that require a lot of thinking and creativity awesome great to hear and then last rapid fire question for you guys is AI versus traditional testing what are the key differences so the good thing with AI is that it has a lot of semantic knowledge and for instance if we take recorders it's good but it's very brittle you know like change some stuff it doesn't work it doesn't have a good control flow and so on the good thing with AI just like Paul show is that it's much more resilient just like humans are if we move a button we don't break what is this website no we are resident to noise and AI has not the same ability because it can compile on the Fly the instructions and to me it's much stronger because it has a much better understanding and also you don't need to you know do the T yourself or whatever or modify the code itself like the great thing is for instance you can drop all your scenarios at night you go to sleep you wake up in the morning it's all done at least an initial version of it I guess depending on the criticality of your test you might want to double check but it's easy because you have the code is in front of you I think most of the times our job is it's hard to generate it's easier to adjust and I think we can greatly help our draft and then can finish it so it's potentially making the problem of recorded automated tests removing that problem which is as you mentioned is the brutalness it's once you record the chance of it being replicated successfully is pretty low I would say and then of course when the website or the app changes how do you update that script I guess you got to re-record so something like AI can make that task easier yes and um to add to that originally we thought about using laag to direct ly execute the tests because you could almost at this point give a URL to laag and say test my website and it's going to run around it's going to try stuff it's going to click on some buttons we could train it for that however by talking to QA experts and people in the field we discovered that you know right now the cost is an issue the time to execute is an issue you want to be able to execute your test regularly fast some companies and some squads some teams they execute tests on every commit and they have a continuous integration and thousands of tests get executed when you change the codebase so we cannot yet use laak to run those tests because it's too costly and it's too slow however we can use it to generate reusable tests and that's what makes it very strong and also we want to fit in the current test Pipeline and find pinpoints that Q Engineers have for example re-recording every time the website changes we want to solve those problems because we think there's an overlap between the capabilities of our framework and the problems that some QA Engineers are facing but maybe in the future you know you could potentially have a specialized Ai and you can just say audit my website generate improvements for my def team generate IDs for the design and the ux team and just continuously test the websites in so many different ways I don't think we're there yet but definitely laag and its capabilities of navigating web pages is a step in the right direction yeah that definitely makes sense so give me a little bit high level overview of what laag is and how it came into existence and what's its purpose it's pretty simple users provide objectives just like you provide your you know intern or whatever or your colleagues and that is that we have two models so something we called the world model which is a high level AI that takes objectives and screenshots and then know what instruction to do that's what po showed instruction is something low level but still a natural language imagine it's something thing we just mentioned where we use as objective a bdd file and some information about the test to be performed instruction would be for instance Okay click on login then click on car click on Etc and then we take that instruction and the current Dom of the page and we use an AI to generate a code that knows what part of them to interact with to do the instruction this is run in a python interpreter that you know PS a browser and then by exting the action it changes the state of the world because now you know we click on something we're on a new page which means we need to send that new information to the world model and then we loop on it until we are done with the objective so that's more or less how it works but we have a webinar on our website you can find it relatively easily so I've been thinking about that a lot how AI can help solve Legacy processes that we have in the society it goes you know from like filling taxes which I hate to like I don't know booking some friends or find information on outdated documentations where the search bar doesn't work you know and I think humans waste a lot of time on those things well there are so many problems which we be thinking of Education environment ecology you know and so on and we can free that time we don't need AGI or whatever to free intelligent time the llms and so on we have today can do it for us and I think we can unlock tremendous value so that we can refocus that time and energy to other problem that matter or even just spend time with family or whatever and that's why we created laag it started initially as a hackaton in San Francisco know classic startup story we got number one on hacking news like the following week and got we were like okay here seems like it's exciting for people and we also realized that there were not that many Frameworks for web agents there are Frameworks for like retrial M generation here and there but in my opinion there lacks this kind of Apache server of AI web agents is like open source framework that allow developers to reinert the way we enter the web and we find it kind of missing that's why we created this framework and try to you know democratize those agents because I think that's that would be again Cher for our society and and I think one important decision that we made early on as well is that you know there are many companies that are trying to solve the whole web agent and Bridging the Gap between llms just outputting text and actually AI taking action for you so internet is a great example right you do so many things on the internet every day you take action and one decision we made was to make all of that open source we know a lot of people are working on on this problem we want to work on this problem with a community so it's important for us to make all of that open source we have a great active Discord our GitHub is very Lively we make code modification every day our docks are great and that's thanks to the court team but also to all the amazing contributors that have come to help us whether it is giving ideas changing typos in the docks you know there's no small contribution and everything helps and what Daniel has said with his vision I think a large part of it comes to life with the community and as an open source project actually like pretty cool Integrations that we receive for instance play right support started and asked from the community and actually someone implemented for usam and also someone recently helped us have a kind of caching system so you don't make too many course to open because open un expensive so that's very cool too that someone you know provided the caching mechanism to not pay too much to open a and we have others like that someone is for instance looking at how to use more local models so it's both private controllable and cheaper and so it's pretty cool and yeah I really like this aspect of creating something for the developers because I believe it's unlikely that one guy like open Apple do everything because those agents require some specialization either for performance or for control cost or so on and so it makes much more sense that we you know Empower develop ERS to create their own agents so we have you know a very active ecosystem versus just you know being like a tools for Apple to use which is what they propose if you look at Apple intelligence it's just a one way street like developers can provide tools to sir but you cannot control sir it's one way you cannot create this world model your specific vision of an agent with what they provide but what we do people can actually do it so yeah I think it's really important that we have this ability to equip the ecosystem to create alternatives to the big centralized agents so you talked about laag being open source as a developer how would I get started with it and what could I use it for so we have a few examples that you can find on our documentation as we mentioned we have like this keub where you can easily find us we have a quick tool it's really easy you pep inst St and here for instance you can create an agent we have examples about customization and so on and you just do agent. run or even like you can do an agent. demo and it creates like I don't if you know a gradio which is a kind of web app so you can send the agent to create your friends and people can start using it we're going to provide soon a Chrome extension so that developers create their agent their friends install the Chrome extension and then your friends can consume the agents you create it for them for instance we're going to show something where it can get your open a receipts and send an email to whoever you want so that this way you know you don't have to ask your colleague oh yeah send me the reips because you haven't put that in the accounting whatever software which is annoying but as you know a real scenario that's the kind of stuff that we provide and yeah so documentation is rather extensive but maybe not sufficient yet but as Paul mentioned which TR be very approachable react as fast as possible and we have many different use cases right I think the the strength of laak today is it allows you in a very little amount of lines of code to instantiate a web agent that takes an objective which is great and you can think about chaining this with other Frameworks other apps other apis so one example is applying to a job for you what we've done what we wanted to Showcase is that laag is a small part what we did is use laag in conjunction with other tools for example we take a resume so could be a PNG a PDF any format really you give this to an open source Vision model in our case we use edix available on hunging face and this is not laag just yet right the framework is not used we use this open source Vision tool to parse the resume and extract relevant information and then we change that with the laag framework to say to laag hey here's my data that was extracted from my resume go apply to this job and we give the URL and the resume data so the strength of that is that you can use the web agent in conjunction with many other familiar tools and in our case it's a quite a simple example but you can think about scaling it and what if laag could easily parse your resume and apply to jobs for you right that's real added value in terms of the things that are usually annoying to do applying to jobs it's very repetitive it's very annoying so just chain other tools you're familiar with with laag and laag will execute stuff you could also chain tools after laag Execute stuff like we Showcase in the QA automation demo right you use laag to record a bunch of information to navigate the website and then you use a llm local API whatever to generate some automated code so the great thing is that laag is a few lines of code to run an agent and you can think about so many use cases with all of the other tools available today it's a good thing with being open sources you can change it however you want integration is very easy because you know how it works and so and yeah so many ways you can as as agent call let's say Gmail API you can call it yourself if you prefer to do it outside of it you know in your workflow so you can combine those things however you want that support of being open yeah that makes total sense are you guys already seeing customers or companies using laag in different ways to become more efficient shop in VAR parts of the world so we see those kind of examples we also see a lot of people try to use it for personal use cases like get me my receipts try to book some train fan information about that perfume and so on we see AIT bit of everything from like very large overprized use cases to almost b2c like examples and I think it's just a start because laag has been evolving very quickly a month or two months ago we were not there yet and over two months we've accomplished so much so I feel like sometimes people even have trouble following the speed at which we're growing and we're adding so many things and the community is contributing and things are evolving very quickly to his usage and since we're open source we cannot know everything about how people use it because some people can just take our code and do stuff and we will never know that laag was used behind the scene but on our Discord on calls we're making so many people have so many different ideas and sometimes even us we having trouble absorbing all of those use cases and all of those nice idas so right now we're seeing really great traction and and we're really happy about that example when you reach out to us there was no agent thing it was just humans had to type manually click on whatever we did generate the code but there was no higher level AI as I mentioned that can write this instructions but now we have it and soon we want to do something even more meta where there's an AI that provides objectives to other AI so that they understand how the website works in general before asking a very specific thing so it's getting more and more meta I guess you know more higher level it does mean we break some stuff so sorry guys it's not always perfect we are quite optimistic but as you know there are edge cases we don't pretend we going to solve everything we show for instance example where we know it works know it doesn't always work but I think it's doing a very good job now like Paul used it on Amazon so the day it worked directly so yeah it's interesting you know the diversity of the web sometimes things work out of the box and sometimes it doesn't but yeah the goal is really to have this open approach to as many people as possible can tell us what's good and when it's not and we collectively try to solve it that's why also we open again yeah that makes sense you know you guys talking made me realize One manual task that I hate doing is being a developer Advocate I have to submit call for papers all over the world and that's such a manual and annoying process I think I want to use laag for that I assume it's possible from what I saw right if it can fill out a resume form well there's no reason why it can't fill out call for papers for me right for sure and you know we don't pretend that we can handle 100% of modern websites it's still a work in progress so if if you have like complex ey frames and dynamic jQuery elements or calendar Pickers custom youse we're not able to handle everything yet and we're working on it super hard but if you want to fill out a simple form you can definitely try and I'm sure it will go a long way yeah and even like knowing it doesn't work is also super helpful because then we know what to change it's a limitation that you know you need to resolve for example which yeah that's cool the other one that I've actually had I started a project it wasn't AI based it was just code based to do that work and then I ran across some like Google forms types of call for papers and I realized they have a capture on there and I was like crap I don't know how to handle that so you think laag could handle a capture it does actually oh nice it does yeah we're not supporting Z use but it does usually the good thing is that once we have the Chrome extension you just don't need to log automatically because you log with your regular Chrome and then lavac Pilots your browser after you're logged in so you don't need to think about those and one thing we're currently thinking about we have seen a lot of users potentially having issues when using laag on specific websites but one thing we're thinking about is putting mechanisms in place so that when laag is stuck for example on a login page or two Factor authentication cookies captas L understand that it's blocked and asks for user input so you know what if you could launch an agent on a complex task and when it's blocked you receive a WhatsApp notification saying hey what should I do next or what's your login what's your password so you know we're exploring the option of really being kind of like a back and forth between the human and the agent because building an autonomous agent that can handle 100% of the modern internet is really hard and we are a small team so you know maybe it could be back and forth between agent and human and that will solve a lot more use cases going back on captas we've seen laag just taking a screenshot of a capture and reading the letters and inputting them right so it's already pretty impressive although today they are making anti computer vision captures with like pixels to confuse the AIS and stuff so we don't solve 100% but we were already surprised to solve a lot out of the box and just to add on that it's just like what humans do like sometimes you know even your colleagues if you not specific enough they don't know what to do so I think we shouldn't expect AI to solve everything if for instance objective is not well defined so it's not necessar even the agent F sometimes it also forces you to think about how to formulate your problem and something I just want to share because I really like that paper and I think it's good for people to know about it is there's a recent paper by prinston where I looked at the performance of AI on Computing oat and they find that gp4 out of the box is really bad like 8.7% accuracy and anything thing if you provide it more examples and you provide the ability to reflect on its mistakes it gets 20% and the final one is if you put a human in the loop that provides minimal hints so they don't provide the answer it sols 13 out of 15 and those are very hard problem but what does it mean it means that AIS have a lot of capabilities but they need a little bit of human guidance so you know having new interface where you provide just minimal correction to an AI might be much more you know interesting and efficient than thinking it can solve everything that's why I still think we need human is the loop if you want you know very crappy results full automated okay use AI if you want the top performance use AI and put a human to make sure that you're always at that level of quality that's how we think about and that's what people say anyway yeah that's really great Insight thank you for sharing that I think that's important can you give us a concrete example of how we would use an AI with human support in order to get the best results I imagine the future like imagine you're doing yeah web testing you know you have the spec defined or whatever you launch agent and then some you get stuck at some point it will send a WhatsApp and you say okay it shows a screenshot what it thinks you know should be done and you're like okay that's the right test or maybe say oh no actually not go see whatever ticket that you I left somewhere and re run it something like that and I think that can be the future you know I think the chat interface is really interesting and chat really change the game with that and definitely think just like a human would actually I guess sometimes you know you're on a vacation and you know your colleagues are yeah like n say an issue here's what I got and you're okay change that I think that could be the future and as we saw with the paper you don't need a lot of feedback you do need some it's very important to provide feedback you would give to human and then you can see it improve and continue it and then you can store those lessons what it should improve just like a human would there are lot of people about how to manage memory that's just a I learn full time and maybe you know after five trials it's good like 99% of the time that's something to be built you could also think about you know a really General example laag may not be able to do this end to end just yet but what if you ask laag hey book me a flight to Paris tomorrow and it's going to do its thing it's going to load Google flights search for the right destination input the right dates all of that by reasoning and generating selenium code but maybe you'll see that there will be no cheat flights tomorrow so lava could send you a notification saying hey I've stopped I couldn't find exactly what you were looking for but there's a flight maybe the day after that or there's a flight early in the morning what do you want me to do and so this is you know a very concrete example of an AI asking you for input after you've started it which we think is the kind of back and forth you might want to do whether it is on very technical use cases or on very general public use cases we really want the AI to not try to be autonomous too much that you lose performance and quality and accuracy we really want the AI to say hey I have doubts I'm going to ask someone else basically just like as Daniel said you would do in your workplace we're not made to be 100% autonomous it's okay to ask for help and ask for someone else inputs it's also way more efficient sometimes so this is something think we're going to look at for sure yeah I think that makes a lot of sense the example that you gave with booking the flight yeah it made me realize the potential limitations of such a command I mean book me a flight there's so many parameters to take into it right for example for me I prefer to fly out early mornings and so maybe I don't think to say that or maybe it's also important that I don't want to have any stops along the way right when we form these kinds of objectives we don't think about all these parameters that the llm needs to know in order to make its decision so back and forth it almost becomes like a conversation and a personal assistant that you're interacting with in order to achieve the objective and maybe sometimes the questions and the challenges that it has along the way will prompt you to think about things that you didn't even think about before which makes total sense just like a travel agency it may be a bit old school because a lot of people don't even know what a travel agency is you know but you would just call your travel agent and you would go back and forth for 10 minutes and you know even like the most simple example is that let's say lav knew everything about you maybe you just want confirmation to actually buy the flight and not spend like €2,000 Euros because you prompted an AI so maybe you want to put up a threshold like if the action is deemed very important and very impactful always ask for the user permission and that's you know it could be the first step on creating this back and forth a cool I think a cool evolution of that could be if you do keep having these kinds of interactions with it it learns about your preferences so that over time it becomes smarter and more efficient and caters to you exactly that would be amazing because over time you know it knows that you'd like to book early morning flights and you hate overnight flights and you don't want to do stops so yeah it could definitely record stuff about your preferences yeah that's awesome guys where can people learn more about you you all and laag that the kab kab is the starting point the docks are very important we have as we mentioned the Discord where you can find information and there is also like a YouTube channel we have two webinars already about for instance one how the framework works and one about actually how we did it with technical details you can follow us on Twitter and and so on I would say go on the GitHub if you're Dev just to see what we're up building and join the Discord Discord is very active everyone's on there and if you have any question or trouble getting started or any ideas any suggestions just ping us on Discord we we'll answer right away beautiful Daniel Paul thank you guys so much for joining the test automation experience it's been such a pleasure thank you for introducing us to laag I hope that people will use it I'm definitely going to use it I'm actually planning on recording a video around a bunch of the AI tools that I have used for software development and now I'm thinking to add this to one of the tools as well and and explore it in my journey definitely that'll be great and we hope it's going to help you automate stuff and be helpful and if it isn't please tell us what's wrong and we'll fix it sounds good thank you guys so much have a great day thanks bye-bye

I&#39;m going to launch my script and I&#39;m going to pass it the URL as well as the feature file so pretty simple that&#39;s what you see on the left and here the feature file gets passed laag immediately starts and we give a simple instruction to our agent we say run this test case and we pass it the scenario you can see here what you see highlighted here is the reasoning behind laag laag uses computer vision to take screenshots of the page and reasons on what it is on the page to generate instructions those instructions are going to get transpiled into selenium code that&#39;s going to get executed so as you can see now we&#39;re already at the second cycle and it&#39;s it has already executed the instruction to click on the cart button right according to the test description now since we want to check that the item is properly added to the cart the next instruction is to click on the cart icon which is it has already done so as you can see our test is successful and since we asked laac to Simply run the test it has and now it knows that it has finished running so it outputs a complete instruction in the world of AI where data flows Daniel and Paul from L AI know they craft Solutions with precision and Care revolutionizing testing everywhere with sharp algorithms and Grand ideas they lead the way a Visionary ban their expertise in testing is second to none with Le AI automation future has begun from codes deep Mysteries insights unfold in their hands technology secrets are told join us today for for a deep dive into ai&#39;s wonders and how it thrives I&#39;m Nikolai your guide in this quest with Decades of experience and software I&#39;ve trained the best 150,000 mines in 190 lands together we explore Tex vast Sands guys welcome to the show super excited to have you all hi Nikolai thanks for having us our pleasure yeah where are you guys joining from we&#39;re both in Paris awesome welcome welcome from Paris so we like to start out the show with a coding Challenge and I wanted to see if you guys can use AI to create an automated test from a spec file yeah so we can definitely do that even though laag can do much more than we automation this is a use case that you know we think is a great example of the power of laag especially for front end and visual testing I can show you a a quick pre-recorded demo of us generating a test from a bdd file that&#39;s so here we have developed a small proof of concept a small script to see a flag which is more of a general purpose framework for web agents we wanted to see if it can be used to solve problems that are faced by QA engineers and people automating the web so what we&#39;ve built is a script and everything is available on our open source GitHub this script is going to take your standard BD scenario so you guys are probably familiar with the buin syntax and it&#39;s also of course going to take the URL you want to test laag is going to be used and eventually we&#39;re going to Output a reusable py test bdd file in this case that you can rerun the great thing about that and you&#39;re going to see it in action in a few seconds is that if your interface changes or you want to test it on responsive whatever happens like if you need to regenerate a test you don&#39;t have to do it manually you can simply rerun this very simple pipeline Reet a new automated test that you can reuse so our file structure at the start is very simple we have our script and we have our feature file following bdd principles as you can see this is the feature we&#39;re going to test it&#39;s a simple card feature we&#39;re also going to use a demo e-commerce website for Simplicity sake and we&#39;re going to generate a test to test your cart and see if you can add an item to cart from a product Details page so I&#39;m going to launch my script and I&#39;m going to pass it the URL as well as the feature file so pretty simple that&#39;s what you see on the left and here the feature file gets passed La immediately starts and we give a simple instruction to our agent we say run this test case and we pass it the scenario you can see here what you see highlighted here is the reasoning behind laag laag uses computer vision to take screenshots of the page and reasons on what it sees on the page to generate instructions those instructions are going to get transpiled into selenium code that&#39;s going to get executed so as we can see now we&#39;re already at the second cycle and it has already executed the instruction to click on the cart buttton right according to the test description now since we want to check that the item is properly added to the card the next instruction is to click on the card icon which it has already done so as you can see our test is successful and since we asked LA to Simply run the test it has and now it knows that it has finished running so it outputs a complete instruction now what the second part does because what you&#39;ve seen was like laag running a simple objective now what the second part does is we use everything that we&#39;ve recorded during this execution to recreate the automated test so as you can see we&#39;re merging one scenario but you could run it on many scenarios and let&#39;s see what it looks like so as you can see it&#39;s a pest file we do use the pbd library and it contains your fixtures and your steps and everything you need to do including the assert statement right which is what we need to generate on top of L and the great thing is that our test was directly put into our test directory and as you can see I just ran p test and even though it was very quick my test passed and the the file you can see on the left was executed so again the great thing is that if your e-commerce interface were to change or if you wanted to test it on a different screen ratio or anything you can think about you could just rerun laag to get a new pest file that you can reuse according to your screen dimension or what your product team or design team decided to change on the website so we think it&#39;s a first step for you know helping the Q engus job which is Sometimes using recorders and spending a lot of time finding the right cross path and so on that&#39;s what we have for now and we think it can already be useful in this state yeah that&#39;s awesome so let me just make sure that I fully understand is you basically took a bdd spec file and then laag executed that spec file to figured out the instructions on its own and then based on that it generated test code that can now be ran exactly so maybe one small detail that is quite important is that in a sense laag executes the instuction we gave it and nothing else so here we give laag the instruction to run a test as well as the detailed test case so laag does that and it knows that it has to stop when it&#39;s on the last step of the test which was check that the cart contains the product and then we use other llms along with everything that laag has recorded So when laag runs any objective you know it generates selenium code it uses Vision it records the HTML nodes it records a bunch of stuff and we use all of those recordings to build a robust and reusable test file so in a sense laag itself does not generate the test however laag is used to record everything that you need to generate the test he friends we&#39;re here to ask for your help this show is free it has no sponsors or Affiliates of any kind the only thing that keeps the show alive is your viewership and subscriptions to the channel Mia and I ask you to like subscribe subscribe and share this content with all of your friends so that we can keep growing and I can continue to create even better and more valuable content thanks so much for your time now back to the show bye that&#39;s really awesome we&#39;ll get back to exploring this in a little bit I got another rapid fire question for you guys what do you think are the careers that will be negatively impacted by AI it&#39;s a long answer but basically anytime you do something you don&#39;t necessarily need to like think a lot especially like very mechanical task that will be an impact there this long paper and for instance interesting thing is you can see that every task that intellectual too much can be impacted but not so much very manual task which are less well paid and very high level of thinking which are well paid if you are in that you know in the middle you can be impacted but I don&#39;t think the AI we have today is that dangerous in the sense that it doesn&#39;t have the same multimodality multi temporality creativity and S that humans have the things we showed can have a lot of edge cases I think it&#39;s very useful but I don&#39;t think it&#39;s going to take your job FR who is a big AI researchers that Google said that yeah today if it the threat your job you didn&#39;t do a really good job I guess it&#39;s not to be offensive but it&#39;s more to say that AI is not that intelligent yet people should not be like you know so threatened Z is however need I guess for humans to really think about okay where is your niche in the sense like ecological niche and as I mentioned I think that the ability to think and solve problems where you need different interactions novelty Cod B change tremendously and so on so I think many professions and I guess Engineers included n not just you know a specific generation even though it&#39;s a big part but you know even sof Engineers the good ones I think they spend like 20% coding so rest is more like thinking discussion and so on don&#39;t think it&#39;s going to be too damaging if you really are in the you know solving complex problems that require a lot of thinking and creativity awesome great to hear and then last rapid fire question for you guys is AI versus traditional testing what are the key differences so the good thing with AI is that it has a lot of semantic knowledge and for instance if we take recorders it&#39;s good but it&#39;s very brittle you know like change some stuff it doesn&#39;t work it doesn&#39;t have a good control flow and so on the good thing with AI just like Paul show is that it&#39;s much more resilient just like humans are if we move a button we don&#39;t break what is this website no we are resident to noise and AI has not the same ability because it can compile on the Fly the instructions and to me it&#39;s much stronger because it has a much better understanding and also you don&#39;t need to you know do the T yourself or whatever or modify the code itself like the great thing is for instance you can drop all your scenarios at night you go to sleep you wake up in the morning it&#39;s all done at least an initial version of it I guess depending on the criticality of your test you might want to double check but it&#39;s easy because you have the code is in front of you I think most of the times our job is it&#39;s hard to generate it&#39;s easier to adjust and I think we can greatly help our draft and then can finish it so it&#39;s potentially making the problem of recorded automated tests removing that problem which is as you mentioned is the brutalness it&#39;s once you record the chance of it being replicated successfully is pretty low I would say and then of course when the website or the app changes how do you update that script I guess you got to re-record so something like AI can make that task easier yes and um to add to that originally we thought about using laag to direct ly execute the tests because you could almost at this point give a URL to laag and say test my website and it&#39;s going to run around it&#39;s going to try stuff it&#39;s going to click on some buttons we could train it for that however by talking to QA experts and people in the field we discovered that you know right now the cost is an issue the time to execute is an issue you want to be able to execute your test regularly fast some companies and some squads some teams they execute tests on every commit and they have a continuous integration and thousands of tests get executed when you change the codebase so we cannot yet use laak to run those tests because it&#39;s too costly and it&#39;s too slow however we can use it to generate reusable tests and that&#39;s what makes it very strong and also we want to fit in the current test Pipeline and find pinpoints that Q Engineers have for example re-recording every time the website changes we want to solve those problems because we think there&#39;s an overlap between the capabilities of our framework and the problems that some QA Engineers are facing but maybe in the future you know you could potentially have a specialized Ai and you can just say audit my website generate improvements for my def team generate IDs for the design and the ux team and just continuously test the websites in so many different ways I don&#39;t think we&#39;re there yet but definitely laag and its capabilities of navigating web pages is a step in the right direction yeah that definitely makes sense so give me a little bit high level overview of what laag is and how it came into existence and what&#39;s its purpose it&#39;s pretty simple users provide objectives just like you provide your you know intern or whatever or your colleagues and that is that we have two models so something we called the world model which is a high level AI that takes objectives and screenshots and then know what instruction to do that&#39;s what po showed instruction is something low level but still a natural language imagine it&#39;s something thing we just mentioned where we use as objective a bdd file and some information about the test to be performed instruction would be for instance Okay click on login then click on car click on Etc and then we take that instruction and the current Dom of the page and we use an AI to generate a code that knows what part of them to interact with to do the instruction this is run in a python interpreter that you know PS a browser and then by exting the action it changes the state of the world because now you know we click on something we&#39;re on a new page which means we need to send that new information to the world model and then we loop on it until we are done with the objective so that&#39;s more or less how it works but we have a webinar on our website you can find it relatively easily so I&#39;ve been thinking about that a lot how AI can help solve Legacy processes that we have in the society it goes you know from like filling taxes which I hate to like I don&#39;t know booking some friends or find information on outdated documentations where the search bar doesn&#39;t work you know and I think humans waste a lot of time on those things well there are so many problems which we be thinking of Education environment ecology you know and so on and we can free that time we don&#39;t need AGI or whatever to free intelligent time the llms and so on we have today can do it for us and I think we can unlock tremendous value so that we can refocus that time and energy to other problem that matter or even just spend time with family or whatever and that&#39;s why we created laag it started initially as a hackaton in San Francisco know classic startup story we got number one on hacking news like the following week and got we were like okay here seems like it&#39;s exciting for people and we also realized that there were not that many Frameworks for web agents there are Frameworks for like retrial M generation here and there but in my opinion there lacks this kind of Apache server of AI web agents is like open source framework that allow developers to reinert the way we enter the web and we find it kind of missing that&#39;s why we created this framework and try to you know democratize those agents because I think that&#39;s that would be again Cher for our society and and I think one important decision that we made early on as well is that you know there are many companies that are trying to solve the whole web agent and Bridging the Gap between llms just outputting text and actually AI taking action for you so internet is a great example right you do so many things on the internet every day you take action and one decision we made was to make all of that open source we know a lot of people are working on on this problem we want to work on this problem with a community so it&#39;s important for us to make all of that open source we have a great active Discord our GitHub is very Lively we make code modification every day our docks are great and that&#39;s thanks to the court team but also to all the amazing contributors that have come to help us whether it is giving ideas changing typos in the docks you know there&#39;s no small contribution and everything helps and what Daniel has said with his vision I think a large part of it comes to life with the community and as an open source project actually like pretty cool Integrations that we receive for instance play right support started and asked from the community and actually someone implemented for usam and also someone recently helped us have a kind of caching system so you don&#39;t make too many course to open because open un expensive so that&#39;s very cool too that someone you know provided the caching mechanism to not pay too much to open a and we have others like that someone is for instance looking at how to use more local models so it&#39;s both private controllable and cheaper and so it&#39;s pretty cool and yeah I really like this aspect of creating something for the developers because I believe it&#39;s unlikely that one guy like open Apple do everything because those agents require some specialization either for performance or for control cost or so on and so it makes much more sense that we you know Empower develop ERS to create their own agents so we have you know a very active ecosystem versus just you know being like a tools for Apple to use which is what they propose if you look at Apple intelligence it&#39;s just a one way street like developers can provide tools to sir but you cannot control sir it&#39;s one way you cannot create this world model your specific vision of an agent with what they provide but what we do people can actually do it so yeah I think it&#39;s really important that we have this ability to equip the ecosystem to create alternatives to the big centralized agents so you talked about laag being open source as a developer how would I get started with it and what could I use it for so we have a few examples that you can find on our documentation as we mentioned we have like this keub where you can easily find us we have a quick tool it&#39;s really easy you pep inst St and here for instance you can create an agent we have examples about customization and so on and you just do agent. run or even like you can do an agent. demo and it creates like I don&#39;t if you know a gradio which is a kind of web app so you can send the agent to create your friends and people can start using it we&#39;re going to provide soon a Chrome extension so that developers create their agent their friends install the Chrome extension and then your friends can consume the agents you create it for them for instance we&#39;re going to show something where it can get your open a receipts and send an email to whoever you want so that this way you know you don&#39;t have to ask your colleague oh yeah send me the reips because you haven&#39;t put that in the accounting whatever software which is annoying but as you know a real scenario that&#39;s the kind of stuff that we provide and yeah so documentation is rather extensive but maybe not sufficient yet but as Paul mentioned which TR be very approachable react as fast as possible and we have many different use cases right I think the the strength of laak today is it allows you in a very little amount of lines of code to instantiate a web agent that takes an objective which is great and you can think about chaining this with other Frameworks other apps other apis so one example is applying to a job for you what we&#39;ve done what we wanted to Showcase is that laag is a small part what we did is use laag in conjunction with other tools for example we take a resume so could be a PNG a PDF any format really you give this to an open source Vision model in our case we use edix available on hunging face and this is not laag just yet right the framework is not used we use this open source Vision tool to parse the resume and extract relevant information and then we change that with the laag framework to say to laag hey here&#39;s my data that was extracted from my resume go apply to this job and we give the URL and the resume data so the strength of that is that you can use the web agent in conjunction with many other familiar tools and in our case it&#39;s a quite a simple example but you can think about scaling it and what if laag could easily parse your resume and apply to jobs for you right that&#39;s real added value in terms of the things that are usually annoying to do applying to jobs it&#39;s very repetitive it&#39;s very annoying so just chain other tools you&#39;re familiar with with laag and laag will execute stuff you could also chain tools after laag Execute stuff like we Showcase in the QA automation demo right you use laag to record a bunch of information to navigate the website and then you use a llm local API whatever to generate some automated code so the great thing is that laag is a few lines of code to run an agent and you can think about so many use cases with all of the other tools available today it&#39;s a good thing with being open sources you can change it however you want integration is very easy because you know how it works and so and yeah so many ways you can as as agent call let&#39;s say Gmail API you can call it yourself if you prefer to do it outside of it you know in your workflow so you can combine those things however you want that support of being open yeah that makes total sense are you guys already seeing customers or companies using laag in different ways to become more efficient shop in VAR parts of the world so we see those kind of examples we also see a lot of people try to use it for personal use cases like get me my receipts try to book some train fan information about that perfume and so on we see AIT bit of everything from like very large overprized use cases to almost b2c like examples and I think it&#39;s just a start because laag has been evolving very quickly a month or two months ago we were not there yet and over two months we&#39;ve accomplished so much so I feel like sometimes people even have trouble following the speed at which we&#39;re growing and we&#39;re adding so many things and the community is contributing and things are evolving very quickly to his usage and since we&#39;re open source we cannot know everything about how people use it because some people can just take our code and do stuff and we will never know that laag was used behind the scene but on our Discord on calls we&#39;re making so many people have so many different ideas and sometimes even us we having trouble absorbing all of those use cases and all of those nice idas so right now we&#39;re seeing really great traction and and we&#39;re really happy about that example when you reach out to us there was no agent thing it was just humans had to type manually click on whatever we did generate the code but there was no higher level AI as I mentioned that can write this instructions but now we have it and soon we want to do something even more meta where there&#39;s an AI that provides objectives to other AI so that they understand how the website works in general before asking a very specific thing so it&#39;s getting more and more meta I guess you know more higher level it does mean we break some stuff so sorry guys it&#39;s not always perfect we are quite optimistic but as you know there are edge cases we don&#39;t pretend we going to solve everything we show for instance example where we know it works know it doesn&#39;t always work but I think it&#39;s doing a very good job now like Paul used it on Amazon so the day it worked directly so yeah it&#39;s interesting you know the diversity of the web sometimes things work out of the box and sometimes it doesn&#39;t but yeah the goal is really to have this open approach to as many people as possible can tell us what&#39;s good and when it&#39;s not and we collectively try to solve it that&#39;s why also we open again yeah that makes sense you know you guys talking made me realize One manual task that I hate doing is being a developer Advocate I have to submit call for papers all over the world and that&#39;s such a manual and annoying process I think I want to use laag for that I assume it&#39;s possible from what I saw right if it can fill out a resume form well there&#39;s no reason why it can&#39;t fill out call for papers for me right for sure and you know we don&#39;t pretend that we can handle 100% of modern websites it&#39;s still a work in progress so if if you have like complex ey frames and dynamic jQuery elements or calendar Pickers custom youse we&#39;re not able to handle everything yet and we&#39;re working on it super hard but if you want to fill out a simple form you can definitely try and I&#39;m sure it will go a long way yeah and even like knowing it doesn&#39;t work is also super helpful because then we know what to change it&#39;s a limitation that you know you need to resolve for example which yeah that&#39;s cool the other one that I&#39;ve actually had I started a project it wasn&#39;t AI based it was just code based to do that work and then I ran across some like Google forms types of call for papers and I realized they have a capture on there and I was like crap I don&#39;t know how to handle that so you think laag could handle a capture it does actually oh nice it does yeah we&#39;re not supporting Z use but it does usually the good thing is that once we have the Chrome extension you just don&#39;t need to log automatically because you log with your regular Chrome and then lavac Pilots your browser after you&#39;re logged in so you don&#39;t need to think about those and one thing we&#39;re currently thinking about we have seen a lot of users potentially having issues when using laag on specific websites but one thing we&#39;re thinking about is putting mechanisms in place so that when laag is stuck for example on a login page or two Factor authentication cookies captas L understand that it&#39;s blocked and asks for user input so you know what if you could launch an agent on a complex task and when it&#39;s blocked you receive a WhatsApp notification saying hey what should I do next or what&#39;s your login what&#39;s your password so you know we&#39;re exploring the option of really being kind of like a back and forth between the human and the agent because building an autonomous agent that can handle 100% of the modern internet is really hard and we are a small team so you know maybe it could be back and forth between agent and human and that will solve a lot more use cases going back on captas we&#39;ve seen laag just taking a screenshot of a capture and reading the letters and inputting them right so it&#39;s already pretty impressive although today they are making anti computer vision captures with like pixels to confuse the AIS and stuff so we don&#39;t solve 100% but we were already surprised to solve a lot out of the box and just to add on that it&#39;s just like what humans do like sometimes you know even your colleagues if you not specific enough they don&#39;t know what to do so I think we shouldn&#39;t expect AI to solve everything if for instance objective is not well defined so it&#39;s not necessar even the agent F sometimes it also forces you to think about how to formulate your problem and something I just want to share because I really like that paper and I think it&#39;s good for people to know about it is there&#39;s a recent paper by prinston where I looked at the performance of AI on Computing oat and they find that gp4 out of the box is really bad like 8.7% accuracy and anything thing if you provide it more examples and you provide the ability to reflect on its mistakes it gets 20% and the final one is if you put a human in the loop that provides minimal hints so they don&#39;t provide the answer it sols 13 out of 15 and those are very hard problem but what does it mean it means that AIS have a lot of capabilities but they need a little bit of human guidance so you know having new interface where you provide just minimal correction to an AI might be much more you know interesting and efficient than thinking it can solve everything that&#39;s why I still think we need human is the loop if you want you know very crappy results full automated okay use AI if you want the top performance use AI and put a human to make sure that you&#39;re always at that level of quality that&#39;s how we think about and that&#39;s what people say anyway yeah that&#39;s really great Insight thank you for sharing that I think that&#39;s important can you give us a concrete example of how we would use an AI with human support in order to get the best results I imagine the future like imagine you&#39;re doing yeah web testing you know you have the spec defined or whatever you launch agent and then some you get stuck at some point it will send a WhatsApp and you say okay it shows a screenshot what it thinks you know should be done and you&#39;re like okay that&#39;s the right test or maybe say oh no actually not go see whatever ticket that you I left somewhere and re run it something like that and I think that can be the future you know I think the chat interface is really interesting and chat really change the game with that and definitely think just like a human would actually I guess sometimes you know you&#39;re on a vacation and you know your colleagues are yeah like n say an issue here&#39;s what I got and you&#39;re okay change that I think that could be the future and as we saw with the paper you don&#39;t need a lot of feedback you do need some it&#39;s very important to provide feedback you would give to human and then you can see it improve and continue it and then you can store those lessons what it should improve just like a human would there are lot of people about how to manage memory that&#39;s just a I learn full time and maybe you know after five trials it&#39;s good like 99% of the time that&#39;s something to be built you could also think about you know a really General example laag may not be able to do this end to end just yet but what if you ask laag hey book me a flight to Paris tomorrow and it&#39;s going to do its thing it&#39;s going to load Google flights search for the right destination input the right dates all of that by reasoning and generating selenium code but maybe you&#39;ll see that there will be no cheat flights tomorrow so lava could send you a notification saying hey I&#39;ve stopped I couldn&#39;t find exactly what you were looking for but there&#39;s a flight maybe the day after that or there&#39;s a flight early in the morning what do you want me to do and so this is you know a very concrete example of an AI asking you for input after you&#39;ve started it which we think is the kind of back and forth you might want to do whether it is on very technical use cases or on very general public use cases we really want the AI to not try to be autonomous too much that you lose performance and quality and accuracy we really want the AI to say hey I have doubts I&#39;m going to ask someone else basically just like as Daniel said you would do in your workplace we&#39;re not made to be 100% autonomous it&#39;s okay to ask for help and ask for someone else inputs it&#39;s also way more efficient sometimes so this is something think we&#39;re going to look at for sure yeah I think that makes a lot of sense the example that you gave with booking the flight yeah it made me realize the potential limitations of such a command I mean book me a flight there&#39;s so many parameters to take into it right for example for me I prefer to fly out early mornings and so maybe I don&#39;t think to say that or maybe it&#39;s also important that I don&#39;t want to have any stops along the way right when we form these kinds of objectives we don&#39;t think about all these parameters that the llm needs to know in order to make its decision so back and forth it almost becomes like a conversation and a personal assistant that you&#39;re interacting with in order to achieve the objective and maybe sometimes the questions and the challenges that it has along the way will prompt you to think about things that you didn&#39;t even think about before which makes total sense just like a travel agency it may be a bit old school because a lot of people don&#39;t even know what a travel agency is you know but you would just call your travel agent and you would go back and forth for 10 minutes and you know even like the most simple example is that let&#39;s say lav knew everything about you maybe you just want confirmation to actually buy the flight and not spend like €2,000 Euros because you prompted an AI so maybe you want to put up a threshold like if the action is deemed very important and very impactful always ask for the user permission and that&#39;s you know it could be the first step on creating this back and forth a cool I think a cool evolution of that could be if you do keep having these kinds of interactions with it it learns about your preferences so that over time it becomes smarter and more efficient and caters to you exactly that would be amazing because over time you know it knows that you&#39;d like to book early morning flights and you hate overnight flights and you don&#39;t want to do stops so yeah it could definitely record stuff about your preferences yeah that&#39;s awesome guys where can people learn more about you you all and laag that the kab kab is the starting point the docks are very important we have as we mentioned the Discord where you can find information and there is also like a YouTube channel we have two webinars already about for instance one how the framework works and one about actually how we did it with technical details you can follow us on Twitter and and so on I would say go on the GitHub if you&#39;re Dev just to see what we&#39;re up building and join the Discord Discord is very active everyone&#39;s on there and if you have any question or trouble getting started or any ideas any suggestions just ping us on Discord we we&#39;ll answer right away beautiful Daniel Paul thank you guys so much for joining the test automation experience it&#39;s been such a pleasure thank you for introducing us to laag I hope that people will use it I&#39;m definitely going to use it I&#39;m actually planning on recording a video around a bunch of the AI tools that I have used for software development and now I&#39;m thinking to add this to one of the tools as well and and explore it in my journey definitely that&#39;ll be great and we hope it&#39;s going to help you automate stuff and be helpful and if it isn&#39;t please tell us what&#39;s wrong and we&#39;ll fix it sounds good thank you guys so much have a great day thanks bye-bye

Transcript for:Laag Overview and Application in Software Testing

Transcript for:
Laag Overview and Application in Software Testing