DAVID MALAN, All right. Well, hello, world. My name is David Malan. I teach computer science at Harvard, and in particular, a class called CS50, which is Computer Science 50, which is Harvard's introduction to computer science for students who both want to major in and focus on computer science in college, but also who want to just learn a little something about college, about CS50 in college.
We, CS50's team, have very much steered into artificial intelligence over the past six months or so, thanks to some of the team members that are here and back and in front. And I thought I'd share with everyone a bit about how we are using AI, not only for CS50 at Harvard, but also at Yale University, also among our CS50x students, our CS50x teachers, and really students around the world of all ages. Very much applicable then to secondary school and K-12, so to speak.
Before we begin, just wanted to highlight some of CS50's team members here. Nima here is one of our CS50x teachers who attended a workshop at GIS a year ago. So happy that she's back with us. So happy that Margaret Tanzosh from New York City is with us here as well. Andrew with the camera over on the side, if you don't mind smiling when you see him coming.
Miss Bernie over at left, who used to be a guidance counselor here at GIS, you might remember. We have Yulia here in back, who just co-led a computer science class. Carter in way back.
Some of you might have seen some videos of his that are starting to pop up online. Rong Xin, who helped write much of the AI you're about to see today. and our friend Ramon, too, with the camera as well. So maybe if we could, just a round of applause for CS50's team for coming this whole way and joining.
Please interrupt at any point if you have questions, whether you're a teacher or a student. The goal is really to share, for better or for worse, what our experience has been with AI, answer any questions you might have, and give you a sense of some of the good and some of the bad. So CS50 is very much a programming class.
And it's a course in which we use languages called C, which is an older, more traditional language. We use a bit of Python nowadays, JavaScript, but we even use Scratch and SQL for databases. So it really is a broad focused class for which students have lots of questions.
And it's a thing in programming, if you've not heard this expression before, called rubber duck debugging. The idea being that if you don't have a teacher or friend who knows more than you about some subject, if you don't have a parent or even necessarily something like ChatGPT to ask questions of nowadays, you can literally are encouraged to have a inanimate rubber duck. on your desk, or really any object that's not actually going to talk back to you. And the idea is that you actually do talk to the rubber duck when you have confusion.
And you explain to the duck what problem you're having. And the presumption is that just by verbalizing the confusion you're having, eventually that light bulb goes off. And you're like, oh, I'm an idiot. Like, now I understand what's wrong, even though the duck has not actually said anything back to you.
And so for CS50, for a couple of years, we actually have had, within the programming environment, a textual version of this where you can ask questions of the virtual duck. And for many years, a couple of years, it just really quacked back at students, either one, two, or three quacks, and that was the extent of the feedback. But in the context of ChatGPT, which has kind of been taking over the world and AI more generally in the past few months, we've started to rethink just what this duck and what AI can do for us more generally.
However, as some of you might have discovered, at least with an educational context, ChatGPT, in our mind at least, is just too helpful. Like, it's all too willing to answer most any questions you might have, even if it's a homework question and even if it's really something you should be giving more thought about rather than just asking someone else for the answer outright. It's like a friend who's being a little too generous just handing you the answers to some problems. And so with this in mind, what we essentially set out to do was disallow use of ChatGPT at Harvard within CS50, this class I teach, just through policy, not necessarily technology, but through messaging to students that this is not really in its current form ideal for education.
So we have this language in CS50's policies or syllabus. Using AI-based software such as ChatGPT, GitHub Copilot, which is fancier and code-specific, Bing Chat, and others that suggests or completes answers to questions or lines of code is just not reasonable. Like you should not, you may not use it. And using it, by contrast, would be considered academic dishonesty within the class. But we do consider it reasonable for students to use CS50's own AI-based software.
And this is where, especially if you're programmers or aspiring programmers yourself, it's really exciting nowadays that not only do tools like ChatGPT exist, but APIs, application programming interfaces, exist whereby you can sign up with Google, with OpenAI, and others, and write your own software that uses OpenAI or Google or some other underlying AI technology. So you can focus on the application. and less on how the actual AI itself works, which means within hours, days, weeks, you can build really interesting projects, as Rongxin and our colleagues back home have as well.
We do at the end of this class allow students... Encourage them to use ChatGPT and similar tools only once they've gotten to their final projects. Because the reality is by the end of the class, once you've understood some of the basics and fundamentals of programming, the hope is that you will learn to and get accustomed to using tools like ChatGPT that honestly I think are going to be with us in some form really forever here on out.
And so it's certainly a tool that we don't want to completely take away from students, but our goal as teachers really back home has been to try to find that balance where even though it'd be nice if the AI would just do the homework for you, there's got to be some sweet spot there, where you're actually intellectually getting something out of taking the class or doing the homework without necessarily spending minutes, hours, trying to solve a problem for which you really need help. And there's no teacher in that moment available. So we've been trying really to strike that same balance.
So up until about six months ago, the duck that CS50 had was literally physically inanimate. And even the virtual version looked a little something like this, like I described earlier. If you were to say something to this virtual duck, I'm hoping you can help me solve a problem, it might respond with literally one, two, or three quacks.
And we know anecdotally that this alone was a sufficient implementation of a virtual duck for that light bulb to go off for some students. Because if you've ever done this before, like just typing out an email maybe, or posting on a message board if you've done that, oftentimes, at least for me, once I finished organizing my thoughts and writing things out carefully, I'm like, oh, I am an idiot. The whole answer was there in front of me the whole time, and I just needed to think it through a little more carefully. But we can, dare say, do better than this. And so thanks to Rongxin and others, now the duck has sort of been brought to life, such that pretty much overnight this summer, the duck stopped replying with quacks and started replying in English and now other languages as the underlying API support.
So our goals within CS50 have been to do this, to provide students with virtual office hours 24-7. So at the college level, office hours, and some high schools do this too, it's just opportunities to knock on your teacher or your professor's door, sit down and ask them questions about the homework or the material if you're having trouble with it. And maybe it's just you and them, maybe it's a few students doing the same. So virtual office hours would just be an opportunity to do that. But literally all the time, all day long, even when you're home from school, even when it's a holiday, even when it's a weekend or night, and your teacher, of course, is not necessarily online or available.
And for us, and this I think is kind of the holy grail, so to speak, like the thing to really aspire for educationally with AI, is to approximate a one-to-one teacher-student ratio, which even in the strongest, the best of schools, is just not really commonplace. Even in your smallest classes, you might still have eight classmates, maybe 16 or 20. I mean, those are considered, at least in the US, pretty small classes. But imagine if it was just you and the teacher, and that's it. I mean, sort of the apprenticeship model of yesteryear, where you have one master, one student, sort of learning from teaching each other.
That's sort of a possibility now with AI at our disposal. So we built this, Rongxin, and some of our undergraduates this summer built this, CS50.AI, which those of you who are students, if you already have GitHub, Anyone in the world can use this for free and play with it at CS50.ai. And it gives students a user interface quite like this, whereby we first disclaim to students that the tool you're about to use is still very much experimental in beta, so to speak.
But we encourage students, therefore, to reach out if it ever seems to be hallucinating, so to speak, sort of giving them advice that maybe is not necessarily accurate. But here, for those of you who might know a little bit of Python code, is kind of an interesting technical question. It's asking in line one for user input from the keyboard, in line two for the same, storing the strings that the user types in both x and y, which are called variables. Down here, you're just doing some boring math like x plus y, storing it in another variable called sum.
And then here, presumably, we're trying to print that x plus y is that sum. But just a little brain teaser, for those of you who do know some programming, particularly Python, if I were to type in 1 for x and 2 for y, What will this specific python code print if you know? Yeah. Ideally it would print that but there is a bug in this code.
And so this is why the human has asked the A. I. Yeah, it would appear to say that X plus Y is 12 or 12. Do you see why now? I think you have the right instinct here.
Yeah? Say again? Yeah, so there's no int function in python.
Everything you type at the keyboard comes back as what's called a string, which is just like text in english or any other language. So if you want to treat the humans input as numbers, you have to convert it to what's called an integer. So an actual integer using a function in python called int.
So long story short, this is buggy code and that's why the human has written. My code is not working as expected any ideas, which is kind of as specific as like many students yourself. Included probably tend to get like you don't necessarily tell the teacher like what's going wrong or what the symptom is But there too is where software can sort of help by knowing that wait a minute Lots of humans have asked this kind of question in the past I bet I can figure out what you mean by not working quote-unquote and indeed the solution here as Generated by this virtual duck debugger otherwise known as DDB is this so the no human wrote this This was dynamically generated by something similar to chat GPT Quote unquote, it looks like you're trying to add two integers, but the input function in python returns a string. So when you try to add X and Y, you're actually concatenating two strings as you inferred earlier, not adding two integers to fix this. You can use the int function and so it gives you some sample code with which to fix this.
So in this particular case it was a narrowly scoped problem. It wasn't just saying asking how do I add two integers in Python in handing the code, the student had gone to the lengths of writing four lines of code, which is not that much, but is essentially most of the solution to this problem. And the AI tried to get them to the rest of the way. So this is the interface that students now see within CS50. But it's indeed very similar in spirit to chat GPT without necessarily trying to do the whole homework for them.
DAVID MALANI-Now, those of you who are gamers might notice a sort of system up here of hearts. Like, Zelda has this, and lots of games have HP in some form. And this exists within this interface for a couple of reasons.
One, we've noticed that some students are way too excited about AI now being here. And we've seen students ask as many as 120 questions in a day. That's not a bad thing, per se, but it feels like a lot, right?
And can you imagine sitting down with a teacher in class or during office hours and asking them 120 questions? I mean, feels like that's probably excessive, if only because you should probably be taking some time back at your desk or at home thinking about the problem that you're asking so many questions about. So we, through software, have tried to simulate this by having the hearts sort of fade away with each question the students ask, but they regenerate every few minutes or hours or so. So eventually, we're trying to find the sweet spot of sort of asking you politely via the duck, maybe you should go back to your seat and think about this for a little bit, as opposed to asking the software too much. And frankly, underneath the hood, this actually just helps with the engineering of it all.
Like it saves costs, because the more questions that get asked, like the more money OpenAI or Google or Microsoft charge us or anyone using this technology. So it too is sort of a way of using software to balance a couple of goals, pedagogical and technical. For the curious, because CS50 is freely available to teachers and students around the world, that software you just saw has been used by about 90,000 users so far. We are seeing about 20,000 prompts or questions being asked per day. and about 3.4 million questions so far, which from a computing perspective is kind of fun and exciting because it's some really juicy data and an opportunity to figure out what's working well and not at scale.
So I thought we'd give you some technical insights here and give you a bit more vocabulary in the AI space as it relates to this particular duck. And let's see, Andrew, what time do we have to be mindful of? The end time here? Yeah, I think we're good.
1225, perfect. So system prompt is sort of an emerging term of art, like something that technical people know in the world and toss around in the context of AI. And what does that mean? Well, it turns out if you want to write your own software that uses AI in some form, you don't have to write that much code. And in fact, has anyone signed up for not just chat GPT, but to use chat GPT to create your own GPTs?
You can make your own chat bots, essentially. So what that invites you to do is not even write code, but write English or some other human language that just really tells the AI how you want it to behave. And in our case, we want the AI to behave like a rubber duck, if you will. But you could have it behave as an expert in science of some sort, or in the humanities, or the arts.
You can sort of tailor it to only ask questions about a certain field by just telling it as much. So for instance, here is a representative string of English text that we feed to OpenAI. to get our duck to behave the way it does, focused on CS50, the class that we teach. Quote, unquote, you are a friendly and supportive teaching assistant for CS50. You are also a rubber duck.
Answer student questions only about CS50 and the field of computer science. Do not answer questions about unrelated topics. Do not provide full answers to problem sets, as this would violate academic honesty.
Answer this question, colon. So in essence, any time a student asks questions of our duck at CS50.ai, and we ask them to answer them, we will provide them with a free answer. instead of just sending it to ChatGPT and allowing any and all answers to come back, we sort of coach the AI to behave like this and to, frankly, make its answers not necessarily as helpful as ChatGPT out of the box, so to speak, would, because we don't want it spitting out like a 20-line program and saying, here's the answer to your homework problem that's due tomorrow.
We don't necessarily want the AI to just spoil answers to intellectually interesting problems outright. We want to kind of like guide students towards solutions and try to figure out who's been wasting minutes, hours, trying to solve some bug, at which point it's probably best educationally just to help them get over that hurdle, but without just handing them what the answer is. And so we then copy-paste effectively what the student's question is here so that the AI knows ideally how to behave in the context of this specific class.
For those of you who do write code or want to learn code, we use a very popular product called Visual Studio Code. It's free. It's called open source, which means anyone can download it, even modify it if they so choose. It's made by Microsoft, and we host our own version of it at this URL.
CS50.dev, which if you're curious, you can use your own GitHub account and sign in later today for free and play around with it. But it's just a cloud-based version of VS Code. So students don't have to install anything on their Macs or PCs to use this tool either.
And I thought I'd give you here a tour now of some of the ways we're using AI in this particular software. It is admittedly very computer science and programming centric, but halfway through the list, I think you'll see, even if you teach another field, that a lot of these features are generalizable to other fields in the sciences, humanities. and beyond as well.
So this was the very first thing we started playing with last summer. Can we get the AI, the duck, to explain highlighted lines of code? Much like a student might ask a teacher, like, what does this do? Or how does this work? So here is a screenshot of that same tool called VS Code.
Here at top left is just some very simple program written in a language called C. It's only supposed to prompt the human for their name and then print out literally, hello world. But there's a lot of stuff going on.
And if you've never programmed before, all of these parentheses, curly braces, and syntax is confusing. It's not obvious at first glance what it does. So what students can now do in CS50 is highlight one or all of the lines of code, right click on it, and choose Explain Highlighted Code, which doesn't come with VS Code. We added this menu feature by writing code ourselves.
And then voila, at the right hand side of the screen, you'll get a chat GPT-like explanation in English that no teacher, no human wrote in advance, because we didn't necessarily know what code the human was going to type in. But it does its best now to explain very much like a teacher would, step by step. what this code is actually doing.
So it turns out that was actually relatively easy to do. And so the next thing we tried to implement was this. How can we get the AI to advise students on how to improve their code style? Like the formatting of it, the aesthetics, the stuff that doesn't fundamentally matter, because the computer doesn't care how pretty your code is, but trying to guide students to writing more readable code for themselves, any colleagues in the future, their teachers for now. So here's another screenshot of some code, also written in that language called C.
And I'll just stipulate for today, it's messy. Like, it should not be all aligned on the left. Like, this is just sloppy.
The human should have been hitting tab or the space bar a bunch of times. This would be like in an English essay, like not using any indentation or proper spacing. Like, it reads like English, but it just looks like a mess, which that code does. So what students can now do is click a button. And on the right-hand side, they'll first see what the code should look like.
according to some rules that we've configured. The green suggests that you should be hitting the space bar or the tab key there. If there's any red, it means to delete it.
But if students don't quite recognize what is really different, what matters, they can click that button at the top called Explain Changes. And this, too, will pop open a window now, a lot chat GPT, and explain to students how the code looks on the left but should look on the right, much like a teacher might sit down with you side by side and sort of walk you through your code and explain how it could or could not work. or should be better.
So this now is where we start to veer into territory that isn't necessarily CS or programming specific. It just happens to be for us. Can we get the duck to answer most of the questions asked online by students, be it our students on campus at Harvard and Yale or anywhere around the world online? So here is a tool that's called Ed.
It's getting more popular in the United States among universities for Q&A, questions and answers, because it just lets you ask questions all throughout the day. And then the staff, when they're awake and online, they can reply with answers like on reddit or stack exchange or any tool like that but classroom centric. So here's a question that an imaginary student called John Harvard might ask via this tool.
What is Flask exactly? Now, a human for years I've been teaching this class would have had to be up or wake up, answer this question by just typing out a few sentences of explanation. This is kind of an easy question.
It's very definitional. Like most of us could just Google this to get the answer too. But we have integrated the duck into this third party software, so to speak, and have the duck now within three seconds to answer.
really type out this answer for the students. Still guided by that system prompt, so it's trying to help the student understand the topic, but it's kind of keeping it bite-sized, so it's not like a massive list of search results that the student will have to flip through to figure out what it is Flask is, but it tries to really hit the nail on the head. So underneath the hood is the CS50 duck, but this is just some third-party software. Here's another question that's a little more sophisticated.
John Harvard again asks this question, which isn't so important for today's purposes, but this is a juicier question. Question and it ends with a more qualitative question Is there a more efficient way to write this code and students are increasingly asking this like they're gotten they're getting pretty good at writing Code, but they want to know like can I improve it? You know am I going to get an A on this or is this like B quality code and trying to take it further? To the finish line, so this is not the sort of thing that you can just google really unless someone asked exactly that question about exactly that code, but through AI you can sort of infer what the code is trying to do.
And here's a screenshot of the duck's reply to that question generated within three seconds. And let me just stipulate, it's a pretty good nuanced answer. And even though it shows students how to begin to answer the problem here, this is just a comment, so to speak, in programming, which means, like, to do this next.
It doesn't provide students more than a few lines of code. And so that too has become part of our system prompt. The Dux should rarely, if ever, give you 10 lines, 20 lines, 50 lines of code. But it should frequently give you three or five or so, something that a teacher might, to give you a hint to help unstick you. Here too, there's another feature that we've used in the short term that will eventually phase out, but this tool called Ed.
lets humans endorse posts to say, like, this is a good answer. And a few years ago, this was just to endorse, like, another human's answer as being good, especially if the answer came from a student or maybe just another member of the teaching staff. We have co-opted this over the past year so that even though the duck responds to students within seconds in this tool...
If they want to wait and be sure that the duck isn't lying to them or misleading in some way, we also have our human staff, myself included, click endorse to signal like, yes, this is a good AI answer. And this helps mitigate what some of you might have heard about, read about, that like sometimes the AIs these days just make stuff up and we're trying to, you know, have some filter against that. But that's a problem that I do think will go away over time. So what has some of the feedback been?
Among our own students at Harvard this past summer, one of my favorite quotes was this from this student. It felt like having a personal tutor. I love how AI bots will answer without ego and without judgment, generally entertaining even the stupidest of questions without treating them like they are stupid.
It has, and as one could expect, an inhuman level of patience. So this is interesting, because even the best of teachers occasionally might have that bad day, or maybe you are kind of overstaying your welcome in office hours, asking a few too many questions. And even I can distinctly remember not even in college, but even graduate school, where I certainly had my own insecurities about not understanding certain topics.
And I would go to a professor's office hours, one on one. It was just me and him. And the guy was so smart in my mind that I didn't want to look like an idiot that even if I still had questions, I'd be like, yep, I get it. Bye.
And then walk out, which is just stupid educationally, right? Like socially, there were these pressures on me, or I felt that I should just be turning around and leaving. but that's got to happen to other people as well.
And how counterproductive is that if your teacher is there, like, ready and willing, but you feel like they're judging you, even if they wouldn't ever even think that? There's something about, then, using software to just take that out of the picture altogether and just give us all the comfort of asking questions that we might be embarrassed to ask other people, even though we really do want to. understand the question. So there's something I think healthy about that too. Another student wrote, the AI tools gave me enough hints to try on my own and also helped me decipher errors and possible errors I might encounter.
And then a third student wrote, I also appreciated that CS50 implemented its own version of AI because I think just directly using something like ChatGPT would have definitely detracted from learning. So we being a computer science class are a bit unique in that we have the technical skills to go build these tools ourselves. But I do think, and you're starting to see this in the real world already in the coming months and years, there's going to be lots of companies, big and small, that are going to create tools like this that will just exist within your LMS learning management systems or freely or for subscription purposes online.
So I think what you're seeing here is a taste of something computer science specific, but that's very soon going to broaden to other fields as well, whether it comes from people like us or others. This is more anecdotal than anything, but some of the data we started looking at in response to students feedback was well How frequently did you use the duck within CS50 and you'll see that frequently was the number one answer with about half of students using it frequently 30% were using it constantly and then there was a smaller tail over here In terms of how helpful students found, this too was encouraging because this was version one of the duck in CS50. Frequently helpful was over 50% of students, 40% always helpful, and then these are sort of been the focus of our work.
If students are finding it infrequently, if they're ever finding it unhelpful, we keep track of all the questions asked and the answers provided to see how we can improve over time and the team has been tinkering with the answers over time. This gets a little more quantitative, but we rolled this out to about 70 students last summer, so just nine or so months ago, and of the questions that we had, we They asked these were the results for the duck so 88% of curricular questions were answered in version one correctly by the duck Slightly fewer 77% administrative answers were answered correctly the latter was not that interesting or worrisome to us This was more a function of what students would ask like when is the homework due and? Because the AI had been trained on the previous version of the class like dates were wrong and certain policies had changed So the administrative stuff wasn't that interesting But the fact that it was getting 88% of the curricular questions, the content questions correct out of the box, was pretty encouraging and inspiring for us.
Interestingly, when we looked at the data with about 1,000 students this fall at Harvard and Yale, it looked like the ducks somehow got worse because only 39% of the answers were deemed by us humans to be correct. So we dug in a bit more to this data, and what we found was this. Whereas in fall 2022, pre-COVID, pre-ChatGPT, pre-Duck in CS50, so a year plus ago, students were asking via our Q&A tool online, Ed, 0.89 questions per student, which meant on average every student was asking roughly one question online during the term.
They could still ask QMNTFs. We were not measuring in-person interactions, just online. In summer 2023, apologies for the typo there, students were asking roughly the same, so 1.1 question online via students.
But as soon as we rolled out the duck this fall en masse to many more students and had been fine tuning it, that dropped significantly to about 25% of what it was. So this would mean out of every four students, only one of them was asking a question online of the duck. So what we think is happening here is that students were actually in a positive way gravitating now toward having conversations interactively with the duck, which we don't monitor for correctness, which itself is a trade-off. because you get an immediate answer, but you don't have a human waiting to click on the Endorsed button.
But what I think we're finding is that we just have less juicy data here, and students who were escalating their questions to us humans were perhaps asking the harder questions, the more policy-oriented questions via the same tool as opposed to synchronously via the doc. So it doesn't necessarily seem a bad thing. If anything, it indicates a behavioral change that happened even faster than we might have thought.
And just to provoke any questions or conversation here, I thought I'd just give you some projection as to what we think is going to happen further as side effects, good and bad, of this kind of technology, Rduck in particular. So the impact on grades. So I actually think within CS50, grades are just going to go up and up and up. And we have data already to suggest this. Within CS50, which is again very much a programming class, we've traditionally graded students on three different bases.
Like one, correctness of their code. Does it work the way we ask them to make it work? Two, style. Like how pretty is it?
How well written is it formatted? Which is fairly black and white too. Does it look good or not based on certain rules? Three is design, which is how well written is it logically?
And is this the best you can do, actually? And that's more subjective. That's where a good teacher would have you not only get the code to work, but really get it to work really well, such that it's impressive to other programmers, it's maintainable, it's scalable, and the like. So for many years, CS50 has graded correctness automatically and style automatically, with tools called Check50 and Style50, for the familiar.
And we've seen quantitatively students'grades for the past 10 years have been going up and up and up. But this is kind of logical, because if we're giving them immediate feedback before the homework is due, well, of course, by the time they submit it, it should be better and better and better, because they're getting this iterative feedback. Not feedback that a teacher with a large class could necessarily provide to all students, but software can certainly do this. We're now reaching the point where within a programming class, we're probably pretty close to having a design 50 tool, so to speak, or just a duck. that can answer questions like, how can I make this better?
And if you do have a smart AI giving you more and more good advice before you even submit, I mean, honestly, like the rational conclusion here is like all of your homework will be 100% because you've had opportunities to refine it and improve it over the course of the week. I don't think that's necessarily a bad thing. I dare say it might be an inevitable thing, at least for certain fields where you do have these opportunities for feedback.
What does this mean then for assessment? Because if everyone's getting 100%, which as an aside, at least within the U.S., a lot of students are getting close to 100% because of grade inflation, so to speak. So there's other problems in this context already.
But how do we actually distinguish whether a student is good, great, fair, poor? I mean, some way of evaluating students so that, one, they have good feedback as to where they can improve. And two, if they are applying for jobs or pursuing other courses in a field, are they prepared for it? Are they ready for it? So we don't yet know exactly what our own answer is going to be, but I do suspect, if I had to guess, that we will probably move away from sort of traditional exams, paper, or online that are asking open-ended questions, because honestly, even AI can start answering questions.
open-ended questions that teachers have never asked in a test before because it's brand new and made up I think what we'll probably get is to something more conversational whereas CS50 students at least in the future will probably have like an hour-long conversation with a chat bot initially by the keyboard eventually probably verbally and we'll use voice recognition too and have a conversation because I dare say most any of your teachers too and sitting down with you one-on-one seeing lots of your work over time having a conversation asking you to explain something back to them You get a sense of like who is the A student, who is the B student, even though that's not necessarily as rigorous a process as something more procedural like a formal assessment. But I do think we'll be able to leverage the AI to give a sense of how conversant a student is literally in the material and therefore where the opportunities are for improvement. So, so much is changing and I think there's even more to come.
I do think that education, at least in the U.S., has at least... nine months ago, really focused on the negatives, with so many teachers and even schools saying, like, no ChatGPT, like, simply can't use it. And I don't think that's unreasonable right out of the gate, because all of these industry tools are, again, just all too eager to be as helpful as they can be. And that's great for someone in the real world who's just trying to be more productive.
It's not necessarily great in the educational world, where you actually want to learn the material, not just get the answer per se. So at this point, I'm happy to refer you to our whole experience, which some of the team, Carter and Rongshin, will present at an education conference in the US in a couple of weeks'time. We'll share these slides out with anyone who would like afterward, but that same paper is at this URL, especially for teachers who might want to dig a bit more into what it is we did well and not well, but other than that, I'm happy to entertain and have any discussions that folks might like. But thank you again for having us all here this week.
Question? No, it's okay. Any questions at all, students or teachers alike?
Yeah? What are my thoughts on AI replacing programmers? I think this will happen to some extent, but particularly for work that humans didn't really want to do or enjoy doing themselves. And I say that for a couple of reasons. So back in my day, when I was learning how to do websites in a language called HTML, and CSS, back in my day, we wrote it all from scratch.
We'd open up a simple text editing program and literally write out all of the tags and attributes and do it by hand. Pretty quickly, the world realized this isn't the best use of human time. And so a very popular product called Dreamweaver came out for Macs and PCs, where now you can click and drag and sort of create the website, and it really just generates the code for you underneath the hood, but it was never exactly what you wanted, so you still had to know some HTML and CSS, and you could tinker with it and make customizations and refinements. Nowadays, though, if some of you have websites personally or professionally of your own, like Squarespace and Wix in the US are very popular websites, GitHub Pages and the like, it's pretty easy to pay someone else to generate the website for you through software, so you don't even have to go through the process of creating a website. even have to install anything.
You just use the browser and click, click, click. It actually helps you get work done. So what I think you'll see in the near term with AI, I think AI is just going to make us much more productive. And it's going to allow people like us, maybe not necessarily in the educational context where you're trying to learn and master fundamentals, but as soon as you start working on your own personal projects or professional projects, it's going to make it a lot easier and faster and thus less expensive to build more features into your software, fix more bugs more quickly.
Because you kind of have like a smart artificial intelligence sort of next to you virtually of whom you can ask questions, and software like GitHub Copilot is built into the tools and will even point out when you're doing things poorly or can make things even better. Long term, maybe there will be a need for fewer humans doing any of that, like piloting the system. But honestly, the world is only getting more and more technological.
There's so many companies out there that haven't been optimized with software and hardware. That I don't see that happening anytime soon and I think this will, in the whole, really improve our productivity and frankly if it can enable humans around the world to work less, you know fewer hours, fewer days per week, that's probably a good thing up to some balance so I don't think it's overall a bad thing. I imagine that CS50 has a semi-automated system for looking for similarities between students'code and sort of suspicious submissions that might not have been fully generated by the student. And you mentioned that students often... authenticate to the rubber duck.
So I further imagine that probably the duck has resulted in some submissions being flagged, and that you're then able to follow back into the logs to see what's a student ask the duck. Is that an accurate assumption? And if so, have you seen any kind of interesting anecdotal things as part of those cheating flags? DAVID MALAN, Yeah, really good question. This in CS50 and a lot of CS courses, certainly at the university level, has always been a problem.
It's been a problem whereby students are copying code from each other sometimes, or Googling, or from Stack Overflow, or the like. With the duck, we have absolutely come across instances where students have crossed the line policy-wise, where they've just asked the duck to answer a question for them, and we didn't necessarily have enough guardrails in place for the duck to recognize, oh, wait a minute, that is a homework problem. I shouldn't even be answering this in the first place.
So we can do better at that. That said, it has made it harder. I think to detect some instances of academic dishonesty, not so much because of the duck, but because of ChatGPT and these other tools. Because at least for the processes that Harvard and we CS50 have in place, it has been, I think, in terms of fairness and due process, a good thing that we can point to an online URL where that exact same code is and say, with high probability, yes, the student copied and pasted this resource.
When it's being generated dynamically on the fly by AI and therefore might differ from some of the online sources. There's no smoking gun necessarily. And so we've started to rely all the more on some of our instincts to at least, even if in the cases where we're not quite sure, we might with somewhat higher probability soon have conversations with students as opposed to more accusatory statements because the data is just not as clean.
It's never been a fun problem though. I don't think this is creating any new problems, but it might very well exacerbate them, certainly in some circles. Question here?
Oh, it's a good question. Short answer, no. I mean, certainly not anytime soon. There's just too much to educate humans on. I do think there, too, with the right tools and with enough time, I think teachers like me, like us, will just be able to reach more students.
or focus our most precious resource, human time, on the kids that really need it the most. Because there's certainly a categorization among students that you could probably come up with over time, whereby some students are actually going to do quite fine, even if they don't have as much of that interpersonal connections. They're fine. talking to the duck or some AI in some sense.
But there are students who really just kind of need to want to have that more human connection. And I think what AI is going to allow a lot of teachers like us to do initially is sort of prioritize our precious resource of time on the kids who do need or benefit from it the most while still helping to uplift and even move further along those students who are actually okay relying a bit more on software. I mean, honestly, this has been a challenge even in the most luxurious of classrooms where you only have 10 students or 20. Like, that's still a lot.
Like, there's only so many minutes in the classroom hour that you can spend with each and every student at the table. And so my gosh, if you could actually be more of a facilitator, bouncing around, knowing that students have a virtual teacher working with them intimately, but you can swoop in too and really help them if they need that more interpersonal thing. I think that's a net positive. The world has too few teachers as it is.
I don't think we're putting any of them out of business any time. Yeah. It's a good question. So the examples I showed were relatively few in their lines of code.
How is AI handling much larger projects? Better and better. We at CS50, most of our programs are fairly short in nature.
And that's where, truthfully, we have focused. However, colleagues of mine in the real world that are using commercial products like github copilot and others report that they're getting very very good because they not only look at the current file or program that you're trying to write they look at all of the other files on your hard drive they look at the API documentation that companies provide to help you actually use their interfaces and such it's getting very good and perhaps that's where the biggest returns are is here's a code base I'm brand new to the code I have no idea what's going on tell me what's going on And my god, if you can have a piece of software do that instead of imposing on another human whose time is very precious, very expensive, that too I think is going to be a productivity booster. And I want to be mindful of our time.
Should we officially wrap here? But happy to stick around up front here with any questions. But thanks so much for having us again.