Hey everyone! If you're interviewing for a product manager role soon, you're probably going to get a trade-off question that asks you to consider trade-offs between multiple options and figure out how you would decide what option to end up launching with. A couple example questions would be...
How would you decide between showing ads in the beginning of the video versus the middle of the video when trying to monetize a video platform like YouTube? Or for Instagram stories, how would you decide between showing the story for 24 hours or 48 hours? Trade-off questions tend to be the hardest questions within a product manager interview.
And that's because there's so many variables at play. Keep watching and I'm going to share with you a framework that I use to tackle these type of questions and walk you through an example. Hey everyone! I'm Diana and I'm a senior product manager at a big tech company in Silicon Valley, California and I bring you the best tips and tricks to get into product management and teach you how to succeed once you've made it. So today we're talking about trade-off questions.
These questions come up quite often because product managers actually have to deal with these considerations on a day-to-day basis because when you launch a product oftentimes there can be positive effects but also negative effects. So oftentimes you're having to weigh the multitude of effects to try to launch with an optimal solution that's going to hurt your product and company the least while trying to maximize the value. And sometimes you're going to have to make really hard calls.
The best product managers put their hypotheses to the test by using A-B testing instead of assuming that they can predict human behavior, which we all know Humans say one thing and they do another. That's just a joke, but it's true. So let's start with a simple framework.
There's five steps here. The first one being understanding the product and the value. Basically, what is this product we're talking about to get alignment with the interviewer that you guys are talking about the same thing?
And what value does it bring to users? Because once you understand that, are you able to tackle the second part of this framework, which is Key metrics. What are the North Star metrics that are important to this product and to the company? And what might be other supporting metrics that you'll want to analyze in your A-B test?
The third is coming up with your actual hypotheses of the positive and negative effects that will come from the options that you're considering. The fourth is setting up the A-B test to test the hypothesis. For five is figuring out how you come up with a data-driven decision to land on an experience to launch with.
Now let's put the framework into action and use a real-life question. So say you're a product manager at YouTube and we're launching ads for the first time. How would you decide between showing ads in the beginning of the video versus in the middle of the video?
Okay, so we said the first step. was aligning on the product and the value. YouTube is a platform that allows creators to share video content.
With over 2 billion people on the platform, we're able to discover a diversity of videos where they can learn or be entertained. Identify both sides of this ecosystem, whereas most people would have spoken about the people that are watching videos only. And I'm going to call them watchers going forward.
And what value... are both of these sides getting? So the creators are getting a way to share their artistic creations with a wide audience and they get to monetize their content if they have at least a thousand subscribers, which is our goal here too on the channel. Watchers get a diversity of content that's easy to search and personalized to you after YouTube's algorithm learns what you like. So again, this alignment with the interviewer is so important.
One, to help you understand the value that this product is creating, to then help you more easily think about the key metric that this product is trying to optimize for. So the second step, let's talk about the key metrics that matter for YouTube and for this ads feature we're considering. So first I'm going to focus on the North Star metric. And usually I like to think of a good North Star metric as one that intersects the value of the multiple sides of the ecosystem. So for creators, they get value.
When people watch their videos and not just watch 10 seconds of it, but watch it end to end or as much of the video as possible. On the users, we can tell that they're finding the video valuable the more time they're spending watching it. And for YouTube, YouTube earns more money when people spend more time on the platform. So from looking for one good North Star metric that represents the value for all three sides, It would be something like total watch time per day in hours.
Some other key metrics would be the number of videos watched per day, defining watched as in more than 10 seconds, let's say. Another metric would be the average percentage of video watched. And when we're talking about ads, usually the measurement is impressions, which means when an ad video is seen, and ad clicks, which means When a user actually clicks on the ad to go to the destination that the advertiser wants you to go to.
Take a second to like this video and subscribe to the channel so that you don't miss out the next video. Alright, the third part is talking about our hypotheses. So what positive and negative effects do we think will happen between showing the ad in the beginning of the video versus in the middle? So let's start with the beginning of the video. A couple hypotheses I have showing ads at the beginning is we'll get more ad impressions if we show it at the beginning because more people are clicking on videos to watch versus people who are watching it through to the middle.
So we'd likely see a bump in ad impressions. However, I can see people getting annoyed by the ads and end up, let's say, closing the browser because they don't want to watch the ad, which means there's a possibility that they end up not even watching the content that they clicked on, which would decrease the total watch time. Or let's say people do engage with the ad and click on it, and that takes them to another destination, which means they forget. Or they don't even get to watch the video that they had clicked on in the beginning, which means the content creator loses out, thus again decreasing total watch time. Another hypothesis for putting a video at the start is because by that point the user has not even gotten the value from the video that they wanted to watch.
So there could be a chance that you see less completion rate or less watch rate per ad. Another hypothesis is we might see people seeing the ad. But not necessarily engaging with the ad because it's happening in the beginning and people get annoyed and they close the ad immediately. And there, if the video is not watched, then YouTube isn't actually able to charge the advertiser. So here you might see impressions, but you might not get the necessary watch rate that will allow YouTube to grow their revenues from it.
Some possible downstream effects. We've mentioned. This could cause the creators content not to be viewed because watchers get distracted by the ad or they close YouTube entirely because they get annoyed by the ad.
And if content creators don't get views on their video, that might discourage them from creating further videos. And if they get discouraged, that means less videos. If there are less videos...
I'm sorry. That means less content on the platform that people can discover, which ultimately reduces total watch time on the platform. Most people will just talk about the direct effects, but what I just shared was what we call downstream effects of what could happen in the ecosystem.
Now let's evaluate what would happen if the ad was in the middle of the video. So a couple of hypotheses. You probably have a significant drop off.
of people who click on a video versus actually watch it to the middle. So hence, we would hypothesize that putting it in the middle would decrease the number of ad impressions compared to putting it at the start. But we might hypothesize that you actually get people watching the whole ad through if it's in the middle because by that time, the user, if they've watched it to the middle, they're engaged enough in the original content that they might find it valuable to actually wait after the ad ends to resume the content that they were so engrossed in, which would translate into higher ad engagement defined by clicks and watch.
rate. And because you're putting the ad in the middle, you're not obstructing them up front from watching the content. And people get to watch the video at least till the middle, which ultimately increases total watch time than putting it in the start of the video.
Another hypothesis is waiting to show video in the middle allows people to get the value that they came for when they clicked on the video before they got interrupted. which could increase the total watch time. I want to call out these effects really are dependent on several other factors, such as a person's sensitivity to ads, whether they just opened YouTube or not, or watched another video without an ad before getting this ad.
It could be affected by how long the video is. It could be affected by where I actually placed the video. And here's another opportunity that you can be a standout candidate by calling out these other alternatives to consider. Because oftentimes you're not just weighing A versus B, but there's all these other different variations to create a product that could be better than A or B. Step number four is where you show that you know how to A-B test.
It's hard to really know whether the hypotheses we just talked about are actually going to hold. And that's why it's so important to A-B test so we can validate or invalidate the hypotheses. And more importantly, measure the effects to know the weight of the effects.
For example. Did it cause a 5% decline or 25%? Each will be treated differently if it was a 5% drop versus a 25% drop and thus each would affect your decision on whether you launched product or not. So how would I set up an A-B test?
In this case I would set up the control group or the first group of people to see the video ad at the start of the video and the second group would be people who see the ad in the middle of the video and I would even show a third group of people You'll see the ad at the cut-off right before where the majority of people drop off when watching videos. So that might be 30% in instead of 50% in. And I would ask our data scientists to do some analysis to find out what that drop-off rate is. Because that could help us capture some of the losses in ad impressions waiting for the middle.
I would make sure to hold everything else constant. For example, I might test this. on a few set of videos and compare the results across the same set of videos I'd make sure there's enough power in the test which means that it has enough people in each of the test groups so that I feel 95% confident that the effect didn't happen at a random chance but happened because of a causal I would run it for a few weeks to a month or whenever the numbers stabilize we really want to avoid novelty effects which is when a behavior is different from when you first see something shiny and new versus after some time when you get used to that feature being there.
And then evaluating the metrics so I'm able to compare apples to apples. And some of the metrics I would look at to compare the effects to each other is one, ad impressions. Two, ad clicks.
Three, percentage of ad watched. Four, total watch time. And number five, the total number of videos that are watched. Now the fifth step is how would we come up with a decision that's data-driven?
So here you want to consider some scenarios that could happen after you run the test. So for example, what could happen comparing adding the ad at the start of the video versus in the middle or some other time before the middle is we could see ad impressions go up, but total clicks go down. But as long as the total volume of ad clicks or engagement is going up, that's good.
as long as it doesn't change total watch time. But if total watch time goes down significantly, then we might want to reconsider shipping this because total watch time has a huge effect on engagement later down the line, as we talked about. And if people are not staying on the YouTube platform, there's no way we can show them ads. In this case, it will really depend on how much the drop in total watch time is.
If it's something like 1%, that's going to be very different from something like 25% because a 1% drop might be acceptable for a, let's say, 5% to 10% increase in ad impressions or ad clicks. So it really depends on the percentages of the drop or the increase in these cases. So that's why it's very hard in this question to actually get to an answer. And that's not the goal of the question to provide an answer.
It's a show the interview how you're thinking through it and considering all the different cases. All right, so back to evaluating the scenarios. There's also a scenario where we see putting the video ad in the middle or before the middle is leading to a ton of extra watch time, but also a ton of total ad impressions because of what our hypothesis was that people are willing to watch through to the video. And that although impressions... are lesser than if they were at the start, the engagement on these ads are much higher.
So in this case, this might seem like a more obvious one that you could launch. This is hard. Okay, not gonna lie, that was pretty complex.
And there are parts of it where you might have not totally gotten it and it's all right. If you didn't get it the first time, don't worry, you can go back to watch that section and ask your questions in the comments so I can clarify anything. Again, this is one of the hardest product manager interview questions.
So don't expect to get it the first time. And I've been training my clients over months to try to tackle this question. So again, to summarize the framework to help you tackle this question is first, start with aligning the interviewer on the product and the value that the product provides. Second, identify key metrics, including the North Star and other metrics that you would want to measure in your A-B test. Third, you want to come up with their hypotheses of what you think could happen between the different options.
Fourth, you then want to set your A-B test to help you test your hypotheses. And fifth, you want to come up with different scenarios on what could happen to show that you can make a data-driven decision. So if this video was helpful or if it was confusing, doesn't matter. Take a look at these other videos to help you in the product manager interview process.
And make sure to support the channel by liking this video and subscribing to the channel. See you guys next time.