[Music] welcome to slp nerdcast i'm kate and i'm amy and we appreciate you tuning in in our podcast we review and provide commentary on resources literature we discuss issues related to the field of speech language pathology and you laugh a lot can use this podcast for asha professional development for more information about us and ceus go to our website www.slpnerdcast.com slp nerdcast is brought to you by listeners like you even small contributions keep our ceu prices low and our program ad free you can contribute on our website by clicking support our work you can also go to our website to find permanent products notes and other handouts some items are free others are not but everything is affordable visit our website to submit a call for papers to come on the show and present with us contact us anytime on facebook instagram or at info slpnercast.com we love hearing from our listeners and we can't wait to learn what you have to teach us just a quick disclaimer the contents of this episode are not meant to replace clinical advice slp nerdcast its hosts and guests do not represent or endorse specific products or procedures mentioned during our episodes unless otherwise stated we are not phds but we do research our material we do our best to provide a thorough review and a fair representation of each topic that we tackle that being said it's always likely that there is an article that we've missed or another perspective that we haven't shared if you have something to add to the conversation please email us we'd love to hear from you we are obviously excited for today because we couldn't get through the intro without laughing so we are very very excited to welcome back dr trina spencer and dr doug peterson welcome trina and thank you it's exciting to be here again exciting i'll be back trina and doug you guys are here to discuss dynamic assessment related to language and literacy and we were lucky enough to have you guys join us once before and listeners if you haven't checked out our previous episode on dynamic assessment please do doug and trina for our listeners who haven't listened to our previous episode can you tell us a little bit about yourselves uh yeah sure so i'm a professor at brigham young university and uh let's see what else do you want to know about that i i work in the department of communication disorders i'm a speech language pathologist i worked in the schools for several years before i went and got my phd and it was real world issues that drove me to pursue a career in research and teaching and yeah do you want more amy is that enough i mean i think it it helps it helps set the stage you're you're a guy who's actually worked in real life you've worked in schools um in terms of your research what are the areas that you have been the most interested in focused on yeah so i am pre so it trina and i both have very um similar interests but i have definitely gone down this path of assessment lately i mean i've always been interested in assessment and she is too not to not to say that trina isn't into assessment as much as i am but i'm just saying like it is a major focus of my life right now it's okay intervention is way more fun doug i do love doing intervention research too it's true i i think here's what it is when i was working in washington someone was presenting about a test and they were talking about measuring comprehension which is language right language comprehension and they were talking about doing it in a way that in my estimation was totally not valid and it drove me crazy and it made me feel like we have got to do a better job and that so like i think that thread of assessment has been woven throughout my whole career as an slp as a researcher it's been festering and that's why i'm super excited about talking about assessment today too because i'm going to explode about it i just got things i need to say well we're so excited to give you a platform to say them this is going to be great some of our best motivations for learning come from a place of like an angry place i feel like yes when you have a bee in your bonnet you need to get it out is how i feel about it trina how about you tell us about yourself all right well right now i'm an associate professor at university of south florida i am part of a research center called right path research and innovation center and our mission is to reduce disparities among vulnerable populations and you know that kind of means do whatever it takes to to get kids performing well in school and socially i'm also affiliate faculty in communication sciences at our university so i do a little bit of like uh work with slps educators you know psychologists whatever and my clinical background is also pretty diverse i was a school psychologist in new york city and i also am a board-certified behavior analyst and i've done my fair share of being a preschool teacher special ed teacher all of those kinds of things and i also had a similar like impetus to get to a phd program because i was in the schools as a school psychologist and you know they do a lot of testing but none of it really made sense to me i was like i am so tired of these iq tests i'm so tired of these achievement tests because they don't actually give us what we really want and need and i was kind of frustrated by that and it was interesting because at one point i realized that we needed to know how children learn not what they had already learned and that key right there was like well i've got to do something about this because there's thousands and thousands of teachers in school psychologists in the schools and now slps of course who need better tools to get the job done right and so you know i did my phd at the same place as doug i know you can feel bad for me later but what came out of it was a pretty good complimentary partnership like we knew we both knew something about assessment and we both know something about intervention and these things are you know inexplicably linked we they need to be linked and so doing them together in our kind of collaboration works out really well for us and hopefully for the people who use our tools a pretty good collaboration that's how you're going to characterize our relationship yeah it's just so right i love it i mean that's really what we all hope for yeah it's just pretty good that's pretty good you amy you're a pretty good co-host thanks buddy rip i got you you're pretty good solid medium okay well before we get to entirely sidetracked with laughter which is a risk let's just be honest it's a risk i am going to go through our learning objectives and disclosures quick quick quick so that we can get into the good stuff learning objective number one explain how dynamic assessment differentiates between language disability and language difference learning objective number two describe different applications of dynamic assessment that identify a disorder learning objective number three explain how dynamic assessment can inform intervention financial and non-financial disclosures dr peterson and dr spencer's financial relationships they are co-authors of the story champs curriculum and pearl dynamic assessment they receive royalties from the sale of those items dr spencer and dr peterson have no financial relationships to disclose no non-financial relationships to disclose uh kate grambois that's me i am the owner and founder of grand blog therapy and consulting and co-founder of slp nerdcast my non-financial disclosures i'm a member of asha sig12 and i serve on the aac advisory group for massachusetts advocates for children i'm also a member of the berkshire association for behavior analysis and therapy mass eba the association for behavior analysis international and the corresponding speech pathology and applied behavior analysis special interest group uh amy that's me i'm an employee of a public school system and co-founder of slp nerdcast and as far as my non-financial disclosures i am also a member of ashes sig12 and i also serve on the aac advisory group for massachusetts advocates for children all right trina doug take it away tell us all the things so we're going to talk about validity yes you need to talk about validity first can i talk about star trek just a little bit star trek for 30 seconds go yes okay 30 seconds just just bear with me i i know this is weird right and i'm okay with that as long as you are but so um i show my students the opening scene from star trek to the wrath of khan did we talk about this last time we did not but no no but i'm very curious i really think that we should this is maybe the best intro we've ever had because i have no idea where it's going oh well it's going somewhere so trina wouldn't let me talk about it last time that's right i was constrained so um this is an epic movie i don't get any money for saying this okay but look this is star trek to the wrath of khan 1982 and it opens up with the scene where kirstie alley is lieutenant savage she's in the captain's seat okay and they have to go into the neutral zone to rescue a ship that's lost life support and there's like 360 people on this ship she knows she has to go somewhere illegal to go rescue the ship um and so she does it as this as this new captain goes in with the enterprise to rescue the ship they get into the neutral zone immediately the ship disappears and three klingon warships these are the bad guys they show up and they just start attacking the enterprise things are blowing up all over the bridge spock is dying in the corner like everybody is it's it's legit right here i am i'm like 10 years old i'm in the movie theater i'm freaking out like everybody in this movie is dying in the opening scene and then all of a sudden you hear somebody over the loudspeaker captain kirk of course he says lights and all the lights come on and everybody comes back to life and um you realize it was just a test the whole thing was just a test it was a simulation and i love to highlight the expense and the effort that these people went through to make this test for lieutenant savic right because what they wanted to do was they wanted to test her reaction in a no-win scenario like what is she really going to do in a situation where there's nothing you can do but essentially face death and um i mean it they they have explosions going off in the on the bridge in this simulation it's as real as can get and and i think okay why did they do that why not just do a paper pencil you know multiple choice true false oh lieutenant savic in this particular scenario what would you do right that has this whole scenario though has everything to do with validity right what they wanted to do was measure her response to that sort of situation and the best way to do it was to put her in that situation as much as they could so they could get a valid interpretation of her response and and so and that's what validity really is validity is truly measuring what you want to measure and the people who created this test called the kobayashi meru test they thought we can't really measure lieutenant savick's ability to respond to a known scenario based on her responses to a piece of paper and a pencil so um so i kind of like to highlight that like this is what validity is you're really legitimately measuring what you want to measure and your interpretation of those results are interpretable or there's something that makes sense to you and you know what to do with those results okay so that's that's a really weird way to talk about what validity is i think right but if you you can immediately talk about and connect that to language right like if you are interested in measuring how someone communicates using language in a functional way in the real world how weird would it be for you to sit across a desk from them and ask them to just point to pictures right like we're talking about a potential mismatch here right and what you want to measure and the way you're doing it so i'm not saying that every test has to be totally ingrained in the real world but i'm just saying what you're interested in measuring needs to align with the test and okay i want to add right is this a good time doug yeah please do okay so i want to add when we're talking about maybe like educational assessments or speech language pathology assessments oftentimes what we're really after the true construct is how someone learns language right but we're trying to measure that construct by actually measuring how much language do they know and what we've got is a construct mismatch like doug said right so we and these are different learning language and knowing language are different constructs and that's why we need a more valid approach to measuring the construct we're actually interested in yeah yeah yeah right isn't it isn't it true that a language disorder is a learning disability right like we just don't characterize it that way but that's what it is it's a difficulty learning language and and so then when you go to say i want to measure if someone has a language disorder what you're really saying is i want to measure if someone has a language learning disability and so then you don't measure that just by seeing what language they currently know that's not measuring that construct of learning you want to actually get into the learning process the irony of this is oftentimes we refer to like norm reference standardized tests as the gold standards unfortunately they are not the gold standards they are simply proxies of the actual construct we're interested in which is learning language and we have to make assumptions about a child's history before they take that test and with the assumption is that they had really good um you know verbal community and they had role models or verbal models right and so the end that everybody who takes that test has the same history and exposure and experience to a verbal community but like that's a crazy assumption we know that's not true but that's that type of assumption is embedded in the use of norm reference standardized tests that was said so succinctly and i fully support your star trek example just for the record i think that painted a great picture and set the stage contained you should see it with james horner's soundtrack and all the pickles oh it's so much better than my little story but yeah you can youtube it i'm sure i can find that scene if you want okay well so doug when would an slp want to use a dynamic assessment and for what purposes would they use it okay okay yeah so i i think we're going to talk about two major purposes right one is for diagnosis to identify disorder or no disorder and then the second one is for um informing instruction and instructional planning uh now i tend to focus on the first one i think that's probably because of my slp background right we seem to be hyper focused on this diagnostic process when we think about assessment that tends to be where our minds go um and i apologize for that but like that's what wait just a second doug it might also be because those are the kind of assessment tools that slps have been given yeah right it's not just that's where you're going that's where you've been trained to go because that's what's available and that was one of my issues with school psychology is like that's all i got is a bunch of norm reference standardized tests i want to do a different type of assessment that tells me what this kid needs to be able to learn language or literacy or whatever not just you know a norm reference that's all i've got you know right right you're like the first school psychologist i ever met who wasn't just obsessed with assessment it's like i thought that's all they did was just go and you know diagnose i i think maybe i should stop talking i'm gonna get in trouble i had a lot of friends who were school psychologists but i mean that's like and i remember when rti came out side note i remember when rti came out i remember thinking huh i wonder what my school psychology friends think about this is this going to work them out of the job like you're not identifying disorder anymore with some kind of iq discrepancy anyways so the purpose is yeah of dynamic assessment can be more than that right but i think those are the two that we primarily want to focus on is diagnosis and informing instruction so i have a question before you move past that doug um we'll talk about validity is is a test valid or is it something else when we're talking about validity yeah that's a real leading question i like it yeah it's because you didn't you didn't say this and i wanted you to i kind of did it's it is the interpretation of the results that have to do with validity so it's not really the test itself but it's what you do with that test and the purposes for which that test is administered those are the things that have validity you want to tighten that up trainer do you feel like that was pretty good mm-hmm i do want to tighten it up wait like our vernacular our vernacular we're just so used to saying things like is that test valid right well what we should be saying is is this decision valid using that test right so it's more about decision validity and that's why we need to understand that there are different purposes of assessment in general because it's those purposes that tell us what kind of decision needs to be made so for example uh i would not use a norm reference standardized test that was designed to i identify normal and not normal to be able to inform instruction or intervention because that's not the tool's purpose right so it we would say that that decision would be invalid using a norm reference standardized test and you know on the flip side i wouldn't be using a classroom you know spelling test to identify a spelling disorder right that decision would be invalid using that particular assessment tool so it's really about purposes decisions and making sure that the tool is intended to do that decision and then like i like to think about it as decision validity not test validity you know one of the examples i like to use is the peabody picture vocabulary test which has been a very popular test in the speech-language pathology world for a long time and for certain purposes it can be very valid say you wanted to get an inventory of a student's receptive vocabulary of ma whatever we want to call this mainstream american english or something you want to get you know an inventory of their ability to understand a particular set of vocabulary words that are in the ppvt then that test could have the results of that test could be interpreted with validity right that oh okay this is what the child can understand but if you want to use it for another purpose like to identify disorder notice how it's driven by the purpose if you want to if you want to use that test to identify disorder then the validity evidence is against you for doing that and um and i i mean we don't have time to go into all the psychometrics of the ppvt and why that might be um and we don't have time to do you know salmon tests or things like that that i like to do but it's just like if the extent to which you've been exposed to vocabulary has so much to do with your inventory and not necessarily whether you have a disorder or not in language but okay i don't want to i don't want to take us down too many side paths i kind of want to do that today i can tell it's because you're having so much fun or like a medium amount of fun pretty good you don't have to answer that as you were so when we think about norm reference tests and you guys are saying they're not sufficient on their own for identifying disorders can you talk to us about that like like why can't i just go in and do this one test do the do the peabody and be like yup here's a vocabulary problem right here i see it what am i missing by doing that or am i is that no i don't we we know the answer or we have an answer it's just i'm trying to let doug talk well i i've been talking too much just i want you to say something okay fine i'll i'll say one thing but then doug i'm going to hand it over to you to talk about our culturally and linguistically diverse kiddos okay so the first answer the first layer of the answer is that again norm reference tests are not actually measuring whether or not someone has a difficult time learning something right so it's still a construct mismatch um and the second layer of the answer is that those types of tests normal standardized tests are actually actually have some challenges for many kids in our schools or in our in our country yeah i think that um we should be careful not to like make some kind of blanket statement like norm reference test can't identify disorder because that's not true at all right and i think that if you had a super homogeneous population and you knew their background and their history and you went and administered a norm reference test to students or individuals from that population you very well could identify a disorder right easily and and um you could interpret that with validity but but the united states in particular uh is not a homogeneous population of individuals in any way right we have so much diversity in this country uh man i mean i don't want to get in trouble here but even the way that we norm our norm reference test the normative samples don't make a lot of sense to me we'll we'll say i so what what well what test makers will do is they will go out and administer their tests to a lot of people hopefully a representative sample of individuals across the country and um and they'll try to get a representation of the diversity of the country so they'll try to get like 16 percent hispanic and 12 african-american and 5 asian etc in their normative sample and um and then what they do is they give you the mean performance accumulated mean right like the mean of all of those people and then and you get standard deviations around it from there you derive your standard scores and your percentile ranks and all of that right but who is that mean representing right like who is who is sixteen percent hispanic and twelve percent african american and five i'm sure that we have some people who have that kind of makeup right but i mean it's really odd if you really wanted to do a norm reference test in my estimation accurately you would try to find micro populations get norms for those populations so that they can more accurately reflect the diversity that we have in the country um so i i i can't believe we haven't even talked about um elena plant's study that she did with um spaulding and fernella in 2006. but i think we should talk about it i kind of like worship her and she doesn't even really know me this is amazing but i talk i probably say her name every day it's crazy but um i love dr plant's work and she did this study with colleagues uh where they analyzed 40 some odd published norm reference language tests and and they're i love the introduction of this paper because it's just so obvious they say um if you have a language disorder you should score low on a language test right this is sort of like axiomatic they're like language disorder your performance should be low on a language test and then they went to see if that actually was true out of these 43 norm reference tests that are published for language how many children with languages or what percentage of children with language disorder actually do perform poorly on those tests and they found that there was tons of overlap the children with language disorder were scoring in the same range as children without language disorder and vice versa and so what that means is of course this test these tests are not very good at actually doing the one purpose that they have which is to identify disorder like that is why you norm a test that's crazy that makes no sense yeah that is dumb that is just that makes no sense it's a tragedy it's a tragedy and and so um and a lot of that has to do with just the way the test is constructed or the diversity of individuals that we have um and uh yeah so it is it's unfortunate and so they looked at two things that are particularly important to analyze when you're determining the validity of the interpretation of test and those are sensitivity and specificity and sensitivity is how well the test accurately identifies the individual with the disorder and specificity is how well the test accurately identify as individuals who don't have the disorder you have to have both you have to have adequate sensitivity and specificity they kind of play against each other and um so they they only found nine now this this study was done in 2006 so it was a while ago right but they only found nine out of those 43 tests that even provided evidence of the one thing they're mostly supposed to do which is show that they have adequate sensitivity and specificity and then out of those nine tests only five of them had 80 sensitivity and specificity or higher um so and that's not even looking at individuals who are culturally linguistically diverse and just just to make sure i'm interpreting this the right way if if a test doesn't have both of those things and it doesn't have highly accurate sensitivity and specificity we're either going to be over diagnosing using air quotes but like over diagnosing people with a language disorder that they don't have or we're going to be missing identification of people who do have a language disorder because it's not picking them up exactly okay and to say something else back to you to sort of recap all of this as my my small very general population brain is sort of absorbing all of this norm referenced assessments are not doing the job they're supposed to do number one they're the bias in the way that they're norm referenced is a significant problem in in identifying individuals who have a language disorder and yet they are what we are taught to use and what as a as a as a culture within our profession what we rely on to identify language disorders so i have to assume that dynamic assessment has a place here and i feel like it's coming so so i mean how does dyna i feel like it's like the magical unicorn that's gonna like ride in and solve all the problems notice i said unicorn and not knight in shining armor so how can how tell us about the role of dynamic assessment here unless you have other knowledge and science that you want to drop on us about norm referenced assessments and all of its baggage so i i we can make that transition and i think it's a good time to make the transition and i i also with doug i was like i don't we don't want to say that norm reference tests are not useful right it's just those are the tools that have the psychometric um qualities that are preferred right so they have good reliability they have good you know evidences of validity you know to say to say something you know they do have their their uses um but especially when we have such a diverse school population or children child population one of the challenges that we have is that children who don't speak english as a first language or even children who are economically disadvantaged they tend to have low skills when they enter school right and b if they don't speak english as a first language everyone's really hesitant to identify them as a with a disability partially because the tools that they have available to them are not well normed with that population right with a subset of that population and another is that they can't actually tell the difference between a disorder or just a difference right and so like just because a child doesn't speak english as their first language does not mean they have a language disorder and so we've got to be really careful about that and in fact they are and children who are uh language minority students or you know don't speak english as a as a first language they tend to be under identified for language disorders or even reading disorders in um you know in kindergarten and first grade however that means that their difficulties are going unidentified for so long that by the time they're in third grade kids who probably really don't have a frank disorder are now so far behind that they qualify for special education which is then what we get is over representation of language minority students in special education and in fact we see that huge increases because we haven't taken care of their problems so there's a real need to identify these kids earlier right and this this idea of like under representation and over representation it in the sequence or in the across the grades it's really a general education problem not just a special edge so it doesn't all fall on you know slps and school psychs and whatnot but it's a it's a problem nationwide it's one of the reasons why response to intervention or multi-tiered frameworks have have a really important place is that they are able to kind of catch those kids be and give them an option for intervention before identification of a disability but dynamic assessment actually does the same kind of thing that response intervention does it gives them um we we kind of test out how they learn so that we can uh improve or hasten their identification oh this is the this is the cartoon okay so we recently had um we recently interviewed uh chelsea privet chelsea privette who's a doctoral student um and she was talking about this heart cartoon she came out to talk to us about language ideology um and linguistic diversity and i had never seen this cartoon before i'm hoping that maybe we can describe this cartoon for our listeners and then maybe put it up on our website okay yeah so what i'm showing you here is this it's a very famous cartoon unfortunately i don't see this well it was famous enough for me to not know it yeah so if you don't know this cartoon it's okay yeah so there's you know a psychologist or some you know nice man it looks like uh and there are animals all lined up you know we've got birds monkeys penguins elephant a fish a sea lion it looks like a dog and the um the man is saying uh for a fair selection everybody has to take the same exam please climb that tree well clearly some of those animals cannot climb a tree but the test is the same for everyone which is what kind of what a normal standardized test does it gives the same test in the same way to everyone right so in our last episode with you guys you described what a dynamic assessment is and some of the different components of dynamic assessment and i am making i'm jumping from stone to stone here in that then the the concept or construct of dynamic assessment is to bridge this gap so that everyone doesn't have the same test so the in other words using this analogy with this picture that poor fish doesn't have to climb that tree because the tree is the standardized or norm reference assessment right like that's that's the tree and that's cool if you fit in the tree box but not if you're the fish yeah it just doesn't account for uh differences in experience in particular uh so i i like to use an example of alaska salmon i give my students a salmon test all the time because i want them to try to identify alaska's salmon and they can't do it and and they can't identify the salmon not because they have some kind of disorder it's just because they haven't been exposed to it it's sort of like unfair for me though to say okay everybody we're all going to take the same test you're all going to identify these salmon and my friends in alaska are like oh yeah sweet easy i got this right that's not a problem for that but for other individuals it's totally biased against them it's unfair and a lot of times our norm reference standardized assessments have this bias where we're asking children to do things that they can't do not because they have a disorder but because they've never been exposed to it never seen it before or or whatever it might be right some other external factor that's not within them yeah so dynamic assessment is obviously the thing that both doug and i have landed on um and there's good reason for it like dynamic assessment has actually been studied for a long time a long long time there are tons of publications about it especially in the psychology journals um but in terms of speech language pathology there's a number of really great studies that use a dynamic assessment approach to identify a language disability and a dynamic assessment large you know let me just give you an overarching definition of what i'm talking about it's really a learning assessment right it's not a performance assessment it's a learning assessment like how we need to assess how well this child learns blank whatever that could be so there are lots of forms of dynamic assessment could do things like a test teach test in a small amount of time which is very similar to a response to intervention framework where they get screening in the fall they get instruction in throughout that semester and then screening again in um winter and then the same for the spring semester right so there's a test we teach in between and we post-test but in a dynamic assessment that would be collapsed into a much smaller time period now some of the studies that are out there for dynamic assessments actually take a month to do so like i think some of pena's earlier work was you know a test teach test model that spanned maybe four weeks i believe and others are like an hour long or sometimes they're like three days whatever it is but doug and i have really worked hard to try to get that down to a very short period of time so that we do a test teach test in a really really small but important set of skills so that we can quickly more quickly and more efficiently and accurately i predict which kids are going to have troubles learning and mostly because the kids don't have time like every minute they are not learning in their classrooms or not getting the right amount of instruction is detrimental to them in their futures not just the kids that don't have time it's the clinicians also right i think that conceptually dynamic assessment makes a lot of sense to to all of us but clinically you know like the feasible implementation of it has been somewhat problematic um and and so yeah like trina said we've tried very hard to make these things efficient while simultaneously not losing you know sensitivity and specificity or or validity can you tell us a little bit more about just how so we as as somebody who's been doing a lot of standardized norm reference tests since i since i got out of grad school i i kind of know the process although i'm learning the process is a flawed process but i know the process for identifying a language disability in that paradigm right like your x standard deviation's below the mean this is your percentile rank i've identified you in these specific ways what is how how do we figure that out through the lens of dynamic assessment like how are we making those determinations those like assessment determinations about language disability right so the interpretation of the dynamic assessment can be either through criterion referencing or norm referencing i don't know anyone that's done a norm reference dynamic assessment we're working on it but traditionally it's through a criterion reference lens that's what you do with the data you take them and you reference them to a criteria and um so the the way it works in the real world is you administer a dynamic assessment i just got an email yesterday from a colleague who administered a dynamic assessment and she said um this student got a post-test score of 11 and their modifiability score which is the score you get from the dynamic assessment that modifiability score was a 17 and their overall ability to learn that score was a three out of a scale of zero to four and so she said how do i interpret those results right and i didn't come back to her with this child has a standard score of 70 and they're out below the first percentile or at the second percentile right i said um for for good sensitivity and specificity students who have a language disorder their uh post test score is usually a nine or lower and their modifiability score is an 11 or lower i'm just kind of making that up but notice how i'm just giving her a cut i'm giving her a line to draw and i'm saying if the child's any child who scores below that then those are the kids that we identified in our studies as having a disorder and we're like 90 accurate or so in doing that so that's the very same information you put in your report and you can say i administered a norm reference test this child scored with a standard score of 80 etc etc and you can also say i administered a dynamic assessment and based on the results of that assessment they um they were identified as having the language disorder or classified as having a link disorder based on the results of this dynamic assessment they didn't make the meet the cut point i would also say that like in the speech language literature the um the assessment tools or the instruments that are being used for dynamic assessment do not necessarily meet the criteria to be like broadly um disseminated okay so they're just trying testing it out to see if it works to predict which kids are um have a language impairment and it it depends on whether they can create an instrument that is standardized right because we still need to have standardization now keep in mind standardization means the administration and scoring procedures are standardized it doesn't mean that there's a norm group right so we we do have to have that and there needs to be sensitivity and specificity data right if you're going to use a dynamic assessment for identifying a disorder those are critical right and so there aren't very many of those available that's why slps are probably not using dynamic assessment to identify disorder because there are too few of them available now the work that doug and i have been doing is to create such tools that will be will be like have enough psychometric qualities to be able to be used and that are efficient feasible to use in schools or wherever and that has the great sensitivity and specificity data and like i'm sure we can talk about some of those because some of those are available and some of those are we're still working on but they are definitely in the literature right so sometimes research we can publish the studies before the actual instruments get available broadly so i'm looking at our learning objectives and thinking about all of this information about dynamic assessment and you know the pitfalls and and drawbacks of norm reference assessment and i'm wondering if you can tell us a little bit about our second learning objective in terms of how dynamic assessment can be like what kinds of applications of dynamic assessment will specifically identify a disorder sure doug i think you should answer that at least try it first yeah [Laughter] so so there are a lot of really amazing researchers that have done work in dynamic assessment and they haven't only focused on narrative language as the medium whereas the topic of that dynamic assessment capansculo restrepo thompson and several others have focused on word learning of our or vocabulary um like liz pena litz those researchers and they found that they can identify language disorder and separate language difference from language disorder with accuracy by using word learning measuring someone's ability to learn new words and they find that children with language disorder have a harder time learning new vocabulary so that's one way the dynamic assessment has been applied narrative has been focused on maybe the most out of all the dynamic assessment of language research and um there have been many many studies on dynamic assessment of narratives there's there's there's a new meta-analysis out by orianna at all i think it's too bad i think it's 2020. um they actually do a meta-analysis of dynamic assessments of language and and the results are pretty clear that they're accurate um tracy ukrainians and her colleagues did a study on categorization again a vocabulary task so um so this so you can see how dynamic assessment can be applied in many different ways but ultimately you're still i think this hasn't been confirmed but i just i'm so suspicious that you're really in the end measuring essentially the same thing you're measuring the ability to learn language and it seems like whether it's vocabulary narrative it's still tapping somehow that same construct of of difficulty learning within at least the the domain broadly of language so like that the answer to that question is the literature's talked about word learning narrative and categorization um but there's also different types of um you know dynamic assessment formats really so you could do the test teach test you could do graduated prompting where you increase or withdraw your prompts within the test while you're teaching or you could test the limits right but one of the things that's key is that we have some sort of measure about modifiability or responsiveness to the uh training or instruction or intervention that happens during the dynamic assessment so there's always that's a key feature of dynamic assessment that there is some instruction going on and oftentimes the examiner or the interventionist in the dynamic assessment um rates the child on measures related to modifiability like how many prompts did they need how quickly did they learn this you know what was their frustration level things like that that kind of give you a clue about their learning right it's more about process and not as much about product and um consistently in the literature when the process is measured like through something like the modifiability scales or something that is more accurate than the product like pre to post test gains even right but we also know that if we combine them sometimes we get superior classification so if it's a test teach test kind of format and we put the child's gains with their modifiability we can be even more accurate in the classification right and for our listeners trina this is you talked about this quite a bit in our in our first podcast with you guys right this was the cake and the sandwich and i didn't remember it just because it's food but maybe a little bit because um so if anybody is listening and you want to hear kind of trying to give a um a more detailed explanation kind of of what that might look like listen to the first one and listen for cakes and in my mind it's grilled cheese but it's probably just the same if it makes you feel any better i remembered the cake one but not the sandwich one what does that tell you okay so and again oh sorry doug go ahead oh no i was just going to add that you can also measure reading and decoding i just can't wait to talk about that that's all i wanted to say and i just waiting for permission to be able to talk about it right so we we should be careful to say that dynamic assessment is not just for language right there are other researchers and other content that can um is amenable to dynamic assessment actually pretty much anything is but there are researchers researchers that do math they do um phonological awareness even inference making and decoding of course so but kate i interrupted you and i'm sorry no no it's fine i'm just you know i'm thinking about you know how something that we said earlier in the episode about how slps may not be familiar with dynamic assessment i know i definitely wasn't before i met youtube glorious folk um and how you know in listening to this dynamic assessment clearly deserves a place at the table we should be using them and educating ourselves about them um and i want to talk about our last learning objective also in terms of how dynamic assessment can inform intervention but i also want to ask for our listeners who might not you know are listening to this and thinking okay i need to incorporate dynamic assessment more in my regular practice but they don't know exactly what to do or exactly how to modify those norm referenced assessments to use them in a more dynamic way or you know what do you do what what should people phys what can they take away from this episode to actually do yeah can i talk can i talk about decoding just real fast so um i'm i'm perusing through facebook and somebody posts how do i identify dyslexia at the beginning of kindergarten and all of these people post these things like you should do dibbles you should do aims with you should do acadiance you should measure phonemic awareness you should measure letter names letter sounds and i'm like what are you people talking about how are you going to identify dyslexia at the beginning of kindergarten by measuring someone's phonemic awareness or their letter names or their letter sounds you that is not going to identify dyslexia what that's going to do is over identify a million kids who don't for whatever reason know their letter sounds or their letter names or don't have adequate phonemic awareness it it's oh i just don't instead of measuring what they currently know which is confounded by so many potential factors why don't you instead measure their ability to learn to decode and that's one of the things that we've done so trini and i have done this study we followed these these 600 kindergarten children from kindergarten all the way to the end of fifth grade and we predicted at the beginning of kindergarten with a three-minute dynamic assessment which just simply asks children to read nonsense words at pre-tests which they could not do and then we taught them very quickly how to read those nonsense words and then we tested them again and we measured their modifiability how hard was it for them to learn to read those words and we've got like 80 accuracy in predicting future difficulty with decoding with a three minute test minutes six years later right so so we're gonna have this reference for years we will put this reference in our show notes for everyone who's listening whose brain just exploded like a teeny bit because that that's like a monumental thing in my limited estimation and i also want to say that we compared because these kids also got dibbles you know i love dibbles people but it is not accurate at identifying at the beginning of kindergarten they know that right it does not accurately identify them the dibble's accuracy was less than 50 why aren't we all doing this why aren't we why aren't we all doing this from the beginning this is nuts it's just crazy right so so the the implications are profound right you can iden you can take a kindergartner at the beginning of the school year and predict which ones will have difficulty learning to continue to code and then you can go in and do early intervention before the manifestation of a disorder ever emerges and totally prevent it from ever happening that child will never ever know in their life that reading was going to be hard for them so that's one of the ways dynamic assessment can inform intervention if you do it early enough it can tell you hey which kids need it and they get it immediately now our dynamic assessment of decoding is available through language dynamics group and it's paired with our dynamic assessment of narratives right so then at the beginning of kindergarten you can do the three minute decoding one and the seven minute language one and be able to sort kids into the right intervention boxes so you're going to have maybe 75 percent of the kids 60 percent of 60 to 75 of your kids at the beginning of kindergarten are not going to need intervention or they're not at risk of having a disability you're going to have another 20 that have decoding only problems right those are likely going to develop dyslexia then you're going to have about 10 to 15 percent of kids who have difficulties in language only so those are the kids who are going to have comprehension difficulties and or language disorders and then another 10 percent that are likely have difficulties in both the coding and language comprehension so you can sort those kids at the beginning of kindergarten and start intervention immediately in september okay so i see the power of dynamic assessment in terms of you know informing intervention but holy cow that is nuts am i the only one that's having a reaction but i no well i think it's because we've heard it before people go no that's too good to be true and we just kind of go all right if you say so but where's the paper on it here's the data data data data you can't argue with a number yeah that is amazing and i'm sorry in addition to that all of our participants in these studies you know they are bilingual um you know low low-income children native americans african-americans you know we're talking about a population that is very difficult to identify using traditional tests so we're intentionally choosing participants that might be hard right and we report the data that way so that you can see it's just as reliable and it's just as valid when we're predicting children who are normally um not very well placed or identified properly i'm just so glad that this is being recorded so everyone can hear it this is like just so important and i feel like i mean i was educated you know a long time ago and i am assuming and hoping that education and graduate school about assessment is different now but this is a this is huge this is really really important and and tell me more okay tell us more about how dynamic assessment informs intervention and then before we do wrap up i want to hear the answer to my question about like what clinicians who are listening to this now who want to incorporate dynamic assessment into their practice what can they actually go do you can choose which one to answer first so so we have mostly been talking about dynamic assessments that have decision validity for classification okay and and you have to have you have to probably purchase something that somebody a test developer has taken care of right they've taken care of the reliability and validity studies so doug and i have dynamic assessments that are commercialized that have been through those psychometrics so been through the psychometric studies that are designed for improving classification we talk about our kindergarten screener it's called the pearl now that is one way to sort kids at the beginning of kindergarten and get them started intervention now there are other mechanisms or other types of dynamic assessment that are not necessarily standardized and they're more like clinical in nature right so for example you can use a dynamic assessment in an aac evaluation where you're really looking at what grid size is appropriate for this particular child and so you may go you know present one and see how the child performs present a different one see how the child performs and so on and then in that type of a dynamic assessment you're looking for where is this child zone of proximal development with aac use and by the way i'm actually talking about a recent article by oh let me get her name right uh gawarder and colleagues from new mexico and there are other aac dynamic assessment studies that they're not their goal is not to identify disorder but to be able to uh identify the child zone of proximal development for instruction right so there's a lot of ways to use dynamic assessment for that way and if that's the case then they could do something as simple as using one of our narrative language measures stories right doing one of an assessment doing an nlm retell and then trying to teach them and then do another nlm retail right well and trina if i could just interrupt for a moment we do provide very detailed procedures in the research on how we've done this dynamic assessment so someone could pick up our research on identifying decoding problems or identifying language disorder using narratives you could pick up the research and you could replicate what's in there and you don't have to buy anything you can follow step by step um how to do it and so i realize a lot of people don't want that right but like that's like monday morning you want to do dynamic assessment download one of those articles from from the asha webpage just keep in mind though if you use something like that it has to be a supplement to other things right or you're using it to inform maybe goals right you could identify some really good iep goals using that method right because that whatever that slp does if it's not the exact scripted procedures from the instrument that was developed with the classification accuracy and reliability and validity evidence then you know you're kind of using it for clinical purposes not for diagnostic purposes i wonder though if that's a good place to start because i feel like as someone who's worked in schools and worked in outpatient settings you know there is this expectation kind of by the funding agency or by the employer that assessments look a certain way and i i wonder if you guys were able to talk a little bit just about how incorporating dynamic assessments is not just going to be something that gives us better information it's also moving toward a more universal design and access for everybody right so it's you know in your study where you're looking at people who are dual language learners if we just do dynamic assessment for everybody it's more accessible for everybody similarly you know if you're an aac user and maybe you have a complex body and you haven't had these same life experiences dynamic assessment is going to be more universally accessible i don't know if you guys had thoughts about kind of as us clinicians kind of boots on the ground in the school like how we can also shift not just the quality of the information that we're getting but sort of shift the machine a bit yeah i really think that it's important to emphasize the autonomy that speech language pathologists have you are the professional in your system and you understand language and articulation technology etc more so than anyone else and you are a licensed professional and there is no law anywhere that indicates that you have to qualify someone for services based on a norm reference standardized test and so it's time for you to just step up and say i'll use a norm reference test that's fine but it will only be part of what i do i'm going to use evidence-based practice and here's the research it's unambiguous now it is clear that dynamic assessment should be used in addition to other assessment approaches to help identify our children who need help absolutely and i think now that most schools are converting to like an rti or mtss framework they're going to understand the need and that construct that you're really after how well they learn and that that information from a dynamic assessment is going to give them so much more than disorder not disorder it's going to give them information about what the child can do in class or what kind of supports they're going to need like is this kid gonna need a one-on-one assistant or can he be fine in the mainstream classroom no i think that that i think that saying it makes sense is redundant it's powerful is what it is and i am really hoping that you know everybody who has listened to this takes these clinically actionable steps and puts them into practice um to improve our way of evaluating and our way of looking at these um you know in improving the culture of our assessment skills as a whole i think that's that's critically important um we i know we mentioned a thousand resources and references all the literature that we mentioned will be listed in the show notes so anyone listening can go dig themselves into a little nerd hole and do some additional reading trina and doug is there anything else that you want to leave our listeners any more gems of wisdom that you want to leave our listeners with before we before we say goodbye gems of wisdom only that i really do encourage you to try to adopt some aspect of dynamic assessment because like amy mentioned there's a universal design feature there right where it is far more inclusive and far more ethical and less biased than a lot of the assessment approaches approaches that that many of us are using right now and i think my little gem is going to be like don't limit yourself to just the language construct either right it's anything and everything that you might need to teach an individual so i once used dynamic assessment to identify which vocational jobs would be appropriate for high school life skills students right it's the same idea i need to put them in and try to teach them this task and see how they do did they like it did they like throw the broom or did they hit me with it right that kind of information is very useful so open your mind to all the possibilities and all the things that you might be able to teach in the context of a dynamic assessment you guys are awesome thanks for coming by this was really great um anybody who has additional questions you can reach out to us anytime uh you cannot you can use this episode for point one asha ceu you if you're interested in doing that you can take the quiz through our website www.slpnercast.com as i mentioned all of the resources and references that we listed throughout the episode will be in our show notes so that you can read on if you feel inspired um again trina and doug thank you so much for joining us and hope everybody enjoyed learning a little bit today thanks for joining us [Music]