if you watched my videos I'm sure you've noticed me shout out this goddess string Library it's very useful and brings in the expected behavior of strings into HK stuff like methods to Strings which is something that exists in other languages but doesn't ahk well I like that Library so much that I actually kidnapped this colada today and I haven't with me the scalada introduce yourself hi guys I'm descalada and I'm here today to talk about the UIA library for uh ahk B2 so what is UIA it's UI automation which is a framework or Library developed by Microsoft for blind people and today we are the blind people yeah so for um yeah so for blind people uh because uh they they need to have screen readers uh which read the text of screen uh so the screen reader needs to see what is going on on the screen and then Microsoft decided that hey let's add user interface testing capabilities also to that framework and then we got UI automation and we're gonna take a look at it today yep so eoya is a library naturally so you can include it but if you run it as a script you get this nice UIA viewer which is actually all we need to get the information to make something work so uh uh first we'll start with an example to press this exact button it's very very simple you start capturing move your cursor onto the elements that you want to catch give it time to construct the tree and okay it can start at the tree you will actually see a tree of first of all the window and all the elements that go into it and the element that you caught will be highlighted in Gray at least for me it's highlighted in Gray but even easier is the fact that you can just uh choose the action that you want to make with that element then click add elements and here you go you have the code uh naturality already copy it and made it so it first highlights and then clicks coming back to the Explorer let's try this out it highlights and then clicks that Watson which is wow beautiful foreign so shall we go over what exactly happened here yeah so please open your vs code window with the code and we can see that the first line is UIA dot element from handbook so the UIA is the class name for for UIA which is used to access all the starting methods so you need to start from somewhere right in this case we are start starting from a window element if we wanted to we could also get an element from point or get focused element which would what is element from points now on the screen element from point gets either the element under the cursor or it it gets from uh it gets it from a specified point so you can put any like 100 and 200 for X and Y coordinates and then it gets a element from that point but yeah in this case we are using element from handle uh and it's very simple it just takes a win title just like many things in HK do so if you use the HK for some time you know how to get a win title so that's pretty easy and we end up exactly within UIA element of the entire window like in Wind exist yes uh yeah but then we have some other generative code which you will naturally will not be able to write yourself so uh let's show off how to uh write your own code rather than just take Auto generated code good idea let's take a look at your Spotify example that's perfect so I had one issue with Spotify or rather something that was not perfect um I had a hotkey that opens the context menu once eventually Spotify loads perfect So currently say I'm listening to a track and it's a banger so I want to right click add to playlist to best a playlist that contains all my bangers naturally uh so doing it without you it was not really viable but with yoy I've been able to make it so let's get to My Method get context menu first of all uh what are the actions that I want to do uh essentially right click here so we get that menu then we click add to playlist then we click best so what's happening here once again we're using element from handle to get the yoye object instantiated so we can use it later then we use find element but how do we get the information to actually get it let's go back to Spotify and you are a viewer star capturing in this case I want this or rather let's actually catch add to playlist immediately it constructs the tree okay we got it so let's scroll and find the actual thing that I'm looking for as you can see Spotify is a huge ass window so it can take some time to load it all and eventually we will find what we're looking for hmm so essentially the Crux of the issue is that you are starting from the window element and then you see that we have some child elements underneath the window and then the children also have children of their own and resulting tree is very big so if you want to move inside the tree to get an element of interest in this case your song song name element or the menu element then we we want to limit the search as much as possible yes and in your example we could see that you you use both the element type it was first type document and you had specified scoop children but we'll get to that uh the reason why we need to narrow down our search so much is because if we try to specifically search for best you can see this huge S3 it would have to go through all of those to eventually find best which would take a lot of time and the script would be slow but if first we find the document meaning the window essentially since Spotify along with most apps that you use day by day are browsers essentially so document then this menu then add to playlist and then best rather than trying to find best immediately so that's exactly what we're doing in code first we need to find the element of document um and having a look at the tree indeed that is the first thing that we come across so can you talk about scope and why it's important to keep it in mind so if our element if our element is very deep inside the tree or very at the end of the tree then the search might take very long because by default find element goes through all the children and the children of the children so in this case this tree is very big and the search would take very long so we want to limit the search as much as possible in this case we are specific by I think the scope hold on it's hard to hear you your sound is choppy can you talk again sorry can you hear me now yep perfect can you hear me hello okay um so uh so in this case we are using scope children which means that find element is only looking at the direct Children of the element we are starting from and to show that off in this case to show that off in a more visual sense so we start at pain essentially we start at the highest element and now we need to find documents because the menu thing this thing was in document as most things are uh so we need to find the document and now we're limiting the search up to only the direct children right now it only needs to search through these five things rather than all of that entirety so by putting the scope of children or two they're the same uh we limit the Search massively improving speed well in this case document is the first ever thing so it wouldn't matter but it's a good thing to keep in mind when you're trying to look for Speed then weight element s wait till I met this uh is exactly what it sounds like it waits for the element to appear uh and the element has to match the criteria you have given so in this case you are waiting for a menu type element and the scope of the search is again children so only it's only waiting for an element inside the direct children so that menu should be somewhere here uh basically um now that we found the elements of the document and the reason why you use find element instead of weight element is if the window exists then document is definitely there but this context menu might not be there so we need to wait for it to access it and that's where weight element is very useful this is because UIA cannot see elements that don't yet exist on the screen it can only interact with stuff that is visible on the screen and so wait element might be used a lot with UIA yes so we specified that menu type but how do we even get it so let's search for add to playlist first um is constructing the tree which once again is pretty huge and here it is add to playlist We want to find this menu first to limit down our search and here we have properties which is the important thing uh we specified type in our code that's a cute rectangle so we specified the type but what if that's not enough in that case you can also specify the other values that you have for it so for example localized type I guess right or name or name yes in this case there is no name so we can use localized type to specify menu as well uh remember the commas naturally great it disappeared um but that element of the menu could also have a name in which case you do name and then well the name and sometimes there won't be enough to just have one thing specified so you might need to specify multiple just like in win titles by the way you specify like the actual win title then HK exe maybe HK group HK class what who even uses that but maybe you do uh same idea in finding and waiting for elements and once we found what we need to uh we can do an action so it could be highlight which is that uh box that's jumping around when you're trying to find the element or the most useful one is obviously click two click on that um okay but if you want you can test out some of the methods inside the UIA viewer as well so once you have selected the element of interest in the UI A3 when you click on it and it highlights blue uh then on the left side there are the patterns so in UIA an element is described by two things first it's the properties so for this we have the name type location Etc and then there are the patterns which describe some specific ways of interacting with the element so in this for example we have uh I think we have the selection item or select somewhere if you scroll down we have scroll item text child uh uh Legacy I accessible and invoke yeah so we can use for example if you double click the in a book with the parenthesis then it will try to invoke it that means click it in this case I think it's already clicked so it doesn't do anything but yeah but that information is a bit uh more complex it's just needed to be explained so you know uh where to go off uh from so uh here we found this whole menu and now we get an object of this menu that we can use for anything else so maybe I create uh more functions that maybe add to Q like remove from this playlist so on so we have the object of this entire menu uh great now let's go to add to playlist the actual meat and potatoes of this whole thing we regain the song menu by using the previous method that we just looked at uh and continue finding and waiting for elements now we need to find elements of add to playlist let's boot up our uoa viewer and start capturing but now for add to playlist so essentially you capture things one after another to get the needed information and here you have name add to playlist um matter of fact you can right click on it to copy that value to then paste it into your script so here I specified the name to playlist great we found that um elements of add to playlist and once we do we have this menu where we find or rather we wait elements because it still takes time for that window to appear of the name of the playlist that I specified in this case best so uh let me remove the highlights and run this and then you'll see how fast this thing actually is okay I reloaded the script let's start playing the track because of how my pot key works that doesn't matter and let's add it to best that was almost immediate and as you can see it just appeared there but that was a bit too fast so let's bring back the highlights um reload the script so you can see what's actually happening uh let's do it again hotkey finds the menu finds add to playlist clicks on it finds best and clicks on best so that is the idea that you can use to do pretty much anything this idea spreads to other things you could do with yoya as well oh Excel why didn't you use regular ah key to do this why did you have to use UIA because I can't use regular HK for this I there's no such Behavior if maybe you read the docs and they lie to you about HK underscore class being useful you know that pain of no this isn't really a thing that you can do but with this Library you have all the power to do pretty much anything that you want so is there anything else that we need to um explain in our beginning tutorial I think that's about covers it for now there's a there's much more to learn yes and uh if you if you want to read more information about it then on the on the main GitHub page for this Library there is also the Wiki page which explains how most of the methods shown here work yep in this video we just wanted to have some beginning because that's already very much enough to do amazing things and then you can discover more and more stuff uh via the amazing documentation I'm sure that there's a lot of roads and naturally you can find the link to that GitHub repo in the description so if you enjoyed this video press a like type some comments if you have a question or suggestion definitely subscribe so you don't miss my content but most importantly stay fresh cheese bags now I'll see you next video bye bye