thank you very much uh my name is David men I teach a class called cs50 at Harvard which is our introductory course in computer science for majors and non- Majors uh the slides that I'm about to share with you live at this URL cs50. ai4 and if you've missed that at this point just turn to the person next to you uh but what I thought I'd share with you is a bit about how we uh within cs50 at Harvard have been really steering into AI particularly as recently as the past few months most recently beta testing some of the ideas and some of the experiments I'm about to share with you with our most recent summer school students which literally wrapped this past Friday so all of this is hot off the press so this is cs50 at Harvard um in healthy times at least um when we have about 700 students on campus undergraduates and graduate students taking the class uh it's a fairly traditionally structured class in that we have lectures with me once a week in which we introduce the coures concepts and then we have sections or recitations with the coures teach teaching fellows or TFS who leave review and Q&A and dive into the material more deeply um all of the course though is freely available and has been for the past 16 years as open courseware so to speak whereby all of the lectures the videos the audio files the PDFs the assessments the software literally everything technologically and curricularly has been freely available to anyone around the world at that URL youtube.com/ cs50 and also via platforms like edex if familiar which are platforms for massive open on online courses like at edx.org cs50 which is to say that we have quite a few students both on campus and off in fact in terms of this coures demographics um there are indeed about 700 students at Harvard College being a residential uh University we uh are fortunate to have quite a bit of support structure in the way of humans so some 40 teaching fellows who lead those sections grade work answer questions hold office hours and also another 40 CA as we call them course assistants who only lead office hours working with students oneon-one we also have 200 students through Harvard's extension school which is our continuing education program both in Cambridge Massachusetts and online and also curiously about 300 students at Yale University where the course has been offered in collaboration with our colleagues there as well uh there we have about the same ratio too but on YouTube nowadays there's some 1.5 million subscribers folks who are engaging in some sense with the class even if they're not necessarily submitting all of the work and through edx uh roughly 5 million registrants over the past few years which is to say at any given time we might have hundreds of students literally on campus especially nearing a deadline asking questions trying to write code solve bugs um and online in a given uh period it might be 40,000 active students at any one time so support structure has been at Forefront of mine for us for quite some time and being computer scientists we've tried to address the need for a support structure through software so for years we've given students now a commandline tool familiar we can type textual commands and get automatic responses from the computer called check 50 which is a tool that all students can use on campus and off to check the correctness of their code automatically before they actually submit it before the deadline we give them a tool called style 50 which allows them to automatically assess the quality of formatting the style the Aesthetics of their code so if you're not a computer person per se it's the analog of writing like an English essay and capitalizing words properly indenting your paragraphs and so forth sort of nicely formatting things is style in the world of code so both of these tools students can use to sort of teach themselves and get iterative feedback on the work that they're doing beh before a deadline but we also more technically have given them a tool called debug 50 which actually just opens up automatically an industry standard debugger special software that just helps programmers find mistakes in code and this just makes it a little easier for them to pull up that user interface to dive into troubleshooting some problems that they have but I dare say it hasn't been enough and so over the years too we've embraced something that programmers call rubber duck debugging uh so the idea is you take literally a rubber duck or really any inanimate object you keep it on your desk next to your laptop or desktop and when you have some logical problem with your code the problem that you're trying to solve we encourage students literally to talk to the rubber duck and the idea is and there's a whole Wikipedia article on this the idea is that in talking through the process verbally in the absence of a ta a teacher a friend a family member you eventually odds are hear some illogic in your state in your thought process and ah the proverbial light bulb goes off and you of appreciate where it is you've gone wrong so for past couple of years we've had a virtual version of this this is a screenshot of what's called an extension or a plugin and an industry standard tool called vs code from Microsoft which lots of pro s included for the class and if students were up until 2022 to ask this duck for help akin to the physical rubber duck all they would get back is three two or one random numbers of quacks so not exactly AI um but over the past few months literally really a few weeks into 2023 have we really been focusing on trying to bring this duck to life so to speak so that not only do they have this physical Incarnation at least on campus but also a duck that talks back to them in English and to some extent in other human languages as well so this was in large part motivated for us not only by a desire to provide students ultimately as you'll see with all the more of a support structure but also to try to reframe The Narrative that at least we have seen in higher education in K12 over the past few months which has been very negative a lot of fears understandably around plagiarism and using chat GPT and the like to just do your homework for you I mean we within cs50 and dare say introductory computer science classes for years for decades have been sort of wrestling with academic dishonesty plagiarism copying someone else's code in person or online for years and so dare say this isn't really a new problem and I do think that tools like chat GPT are poised to exacerbate the problem because you can do it a little more easily eily a little more anonymously without actually making a friend or a classmate complicit but it's not really a new problem and I did really think the opportunity is to focus on the more positive opportunities here which would be not only the what the tools can do but also weaving as in our case all the more of a conversation around ethics and academic honesty into the process and academic Integrity in so far as these tools aren't going anywhere and we're already pretty close to the point where you have this subject matter expert at least in some domains next to you as you're writing your code as you're writing your essays or solving some problems so I'd like to propose that there's some really exciting opportunities ahead that even we have gotten a taste of I think in our own microcosm of cs50 itself so what have we done on that front and what are the tools we've been working on so this is the language we had in our summer school syllabus over the past few weeks whereby we deem it by policy disallowed to use chat GPT the new version of Bing GitHub co- pilot and similar tools on the premise that they're just too helpful right now at least out of the box they're not really configurable they pretty much try to not only just finish your thought for you but hand you your second and your third thought as well trying to do the work for you which is wonderful for productivity certainly in an industry context and frankly probably in higher level classes where you just want to solve problems more quickly after you've developed some muscle memory and some fundamentals yourself so this is language we now have in our syllabus but we supplement it with this language as well that we consider it now reasonable for our students at least to use our own AI based tools built on top of platforms like open Ai and Azure in bing and the like in the coming months so what does this mean in real terms for us so these are some of our works in progress and I thought I'd give you a tour via screenshots and URLs of what it is we've been thinking about our use cases absolutely skew a bit more toward programming and computer science But ultimately I'd like to make the claim that I think a lot of what we've been thinking about a lot of what we've been experimenting with is certainly generalizable not just to other stem Fields but even non- stem in Humanities arts and Beyond so explain High highed lines of codee this is the first problem we bid off a few months ago uh explain highlighted lines of code is a little Works a little something like this this is a screenshot a vs code the programming environment free and open source that our students use and lots of people online here's some code written in an older language called C and suppose that students have downloaded this or we've given it to them and they don't understand one or more lines of those code they can highlight those lines they can rightclick thanks to one of cs50's own extensions or plugins select explain highlighted code roughly there in the middle and voila they get a chat GPT like explanation of what that code is even if we humans have never seen precisely that code before and if no human is even awake at that hour to answer that question so it's AK kin if you will to just copying and pasting code like that into chap GPT but it keeps students in situ in our own environment and it also as you'll see tries to explain it in introductory terms not necessarily as an industry professional might like it explained but a student who's been programming for one or just two weeks advising students on how to improve their code style the formatting or the Aesthetics thereof so here's another darker screenshot here where this code let me just stipulate especially for the non-programmers this is kind of messy it's all flushed to the left it's not nicely indented so if a student were to now click a button at top right that says style 50 the moniker we use for improving code style you'll see rather dimly on this screen that there's some green text that's been uh some green shadowing that's been added to say add White space here hit tab hit the space bar hit the enter key or the like but moreover at top right is there a button for explain changes where they can similarly get a Chachi like explanation of what they should change if not why as well so that they're not just doing a but clicking a button in Auto formatting as they will eventually in higher level classes in the real world but again we're sort of introducing them to the muscle memory that we want them to develop this one I dare say is the one that's perhaps most generalizable to Fields well beyond CS so trying to answer most of students questions in the context of not a physical classroom but a virtual one even for on-campus students that have some kind of online Q&A discussion form or whatever software you're using we use something called Ed another one popular is Piaza Blackboard and similar all have their own equivalents of online Q&A platforms and um here's a little something like so we rolled out in integration into this thirdparty Q&A tool this summer whereby as soon as a student asks question and hits enter their uh platform sends the question to our server where we've written some of our own code on top of open ai's own platform we talk to open AIS apis application programming interfaces hooks that you can use to leverage their services and their model so to speak and then the student within a few seconds gets back an automated response and that response to a simple definitional question like this is fairly straightforward this is kind of a softball question and even if unfamiliar let me just stipulate that flask is in indeed a it is indeed a micro framework dot dot dot but you would expect being in Google a year ago not to mention 10 years ago to be able to answer definitional questions like this so more interesting is a question like this from an actual student though we've anonymized them as John Harvard as well let me just stipulate for the non-programmers this is a pretty sophisticated question the student even included some code or an error message that's fairly Arcane in there and the last question they ask is is there a more efficient way to write this code which itself is a pretty broad question but I dare say that the tools response in this case built on top of these apis is pretty darn nuanced and in fact this is just one of the questions that we humans endorsed at top right so this same tool independent of us lets a human say yes I like this answer and so we hijack that mechanism to say yes we like this AI based answer to signal to students that they can at any hour trust this answer even if no human has actually chimed in with English but we've at least clicked the button to Signal as much that this is in fact a good answer um this was one of my favorite responses from a human student in response to the robot or the duck in this case which again we virtualized via this platform um so another mechanism helping students find bugs or mistakes in their code via ta like rhetorical questions here too the goal as a any good teacher would ideally do is not just answer all of students questions and tell them what to do or what to fix but to sort of guide them to that kind of solution and so we have this larger interface similar to going to chat GPT Itself by which students can ask questions like this in the middle my code is not working as expected any ideas that's about as specific as many students tend to get even with us humans but even these tools do a pretty darn good job at analyzing the code inferring what's actually gone wrong especially in intro classes where these are common mistakes and frequently asked questions and for our purposes today let me stipulate that this is a good and correct answer that said the first thing students always see from ddb the duck debugger at top is just a disc claimer that I do think we'll eventually get rid of but that we encourage students nonetheless to always think critically and especially if they're asking questions that see whose answers seem a little suspect they might indeed be inaccurate but that's indeed been a corner case and not a common case for us so if unfamiliar with how writing software that uses uh Azure Services open Ai and other platforms you typically right now in 2023 feed these uh models a prompt of some sort where English which where you give the the AI personality if you will and we have literally given it a personality of a rubber duck but a friendly and supportive teaching assistant for cs-50 that answer students questions about cs50 in the field of computer science do not un uh answer questions about unrelated topics do not provide full answers to problem sets as this would violate academic honesty so this is a bit of a hack the the way the world is using prompt Engineering in this way to sort of just tell the software to do something and then hope that it does but it does as of now work pretty darn well and I think this kind of tech technique is only going to get more robust over time explaining now Arcane error messages in the world of programming um it turns out that when you run commands you very often get cryptic response that's no big deal for a seasoned industry professional but for green students very much so and so what we're about to release for students this fall is another tool that when just something goes wrong they don't even necessarily have to ask the AI about it we can sort of detect that something went wrong in the command line returning a non-zero exit status for those un for those familiar and invite the duck to sort of help them along the way here too trying to chip away at the volume of frequently asked questions and now is where in our final three bullets things get a little more aspirational for the coming months and if not the coming years assessing the design of programs and providing qualitative feedback we've indeed automated correctness with check 50 we've automated Style with style 50 and we've always dreamed frankly for years of having a design 50 tool that provides students with more subjective qualitative assessment of their code like yes this works and yes it looks pretty but you can do better sort of like in an English essay making a stronger argument even if grammatically everything looks Rock Solid so we're close to this now because we also have years of data uh assignments that have been submitted by students that have already been assessed by humans data that we can now use to train these kinds of models and our hope frankly is to next chip away at the most time intensive use of our ta's time on campus which is grading and providing long form comments which might take three three to six hours of their human time per week when we know empirically from instrumenting tools students spend 0 to 14 seconds consuming and reading that same feedback and it's always been out of skew but it's been a missed opportunity until I dare say now we hope to generalize a lot of what we're doing not just to other CS courses but to other non CS non- stem courses and faculty as well by enabling them to sort of fune so to speak our own tools on top of azure and on top of open AI so that it's a little more turn key for people who are not computer scientists and a little less technical who just want to embrace AI in some positive way and then ultimately two I think this is the thing we're really thinking about and wondering about is what this means for assessment because we've seen empirically over the past 10 years that our students independent of higher education grade inflation our grades have been going up and up and up because of these tools like check 50 and style 50 logically because they're getting iterative feedback throughout the week before they submit and we conjecture that as soon as we provide students with a design 50 like tool that provides them with even more subjective feedback along the way it stands to reason that by the time they submit at midnight on a Sunday evening their code really should be in really good shape after all what was the point of all of these tools otherwise and so in my mind that calls into question well what is it that we are then assessing do we bring back sort of yester Year's traditional exams timed exams which two we've moved away from and sort of the arbitrary um sort of constraints that those provide on students and so we're thinking quite a bit about about what this means for grades what it is we want to assess and for this I don't have an answer just yet but it's the kind of question that this is now inviting not necessarily A worrisome one um if at the end of the day these tools are impa enabling learning all the more effectively but it's certainly a side effect of what we're seeing already so if you'd like to play with if you're more of a computer uh scientist type programmer person you're welcome to use any of these tools cs50. deev is our entry point to that tool I call VSS code that has our extensions baked in automatically cs50. a is the full browser version the conversational version of the same tool that you can ask questions about at the moment just cs50 but eventually we do thank other courses as well and I thought I'd end with just a few quotes from students that they blessed by sharing anonymously from our most recent summer school offering here's one from a student the duck was great and I wish other classes had such a tool this one was even more nuanced felt like having a personal tutor I love how AI Bots will answer questions without ego and without judgment generally entertaining even the stupidest of questions without treating them like their stupid it has an and one as one could expect an inhuman level of patience and then lastly love love love duck we are friends now so for all that and more for these slides in particular feel free to scan that URL and I think we have a couple of minutes remaining for any questions but thank you so much for coming