[Music] [Music] I said just quickly in case people don't know what deliverer is we are an on-demand food delivery company we set at the center of a freeway marketplace between the riders the restaurants and the customers and so we partner with around 10,000 or so tens of thousands of restaurants quite a lot more than 10,000 across 13 markets and we work with a fleet of tens of thousands of riders who form our delivery network to deliver these orders right so here we have the rough flow of our customer they come to our site so first who are hungry customer you want some food that comes to us a or app and order our food and this is when my team the logistics algorithms team starts to get involved and so from there we have machine learning algorithm that predicts how long the food's going to take to prepare we also have algorithms to predict how long riders gonna take to travel so we can assign riders to get to the restaurant exactly when the food should be running we don't have algorithms to like at home they're gonna wait and how long they're going to take to deliver the food and so Frank an order assignment algorithm it's the collection of all the orders that we've received and all of ours and finds the optimal assignment between these such that all of the customers get a Worman they're happy like they're lovely happy come over here and to give you a bit of an idea of the scale of this problem this is a map from central London Astoria all of the delivery logos on here is match the app writers who were online some weekend other and I think in the summer some time and all of the orange knives and forks are the restaurants that we have on the platform this gives you a bit of an idea of the scale of the problem we're dealing with and we're trying to choose which writer should deliver which order that's that's logistics what about load management what am I talking about there and so load roughly speaking is how we tell and how many orders we have competitor riders how our capacity to deliver orders that is much right of time we have compares to the number of hungry customers we have you want some food and if management only really comes into effect when we have a lot more orders than we have roughly speaking our predictive algorithms that predicts the order volume that we expect day-to-day hour-to-hour and we have our rhythms that work out how likely it is that riders are to attend as well as system for riders to request when they want to work all of this means that we typically have the right number of riders on the road day I'm to deliver the orders that we receive there sometimes things happen that means the the right number of riders aren't there and so load management really only kicks in when something's gone wrong and one of the typical situations that we like to talk about when this happens is sort of bad weather and bad weather we we do things that trying extra riders in the road the at the end of the day if it's snowing outside riders don't really fancy going out and getting wet and cold and so there's less riders they go out well so customers don't really want to go out to get food and so we get more orders and this sort of double whammy means that we no longer have enough supply to deliver to other people who would like food another example other than the weather would be something like events like if we have predicted an events going to happen and everyone's watched the same football game and they may order at the same time and suddenly would just get this wave of orders which we can't handle at that time and so we have these algorithms in place to manage this and our goal is to make sure that we don't have customers saw their home waiting for their food hungry not being able to get their food so to give an example of what this sort of looks like and what we do about this here I have a graph at the start of the day where on the y-axis there we have the delivery network load and so when it's low we're nice and quiet we've got a good number of riders to deliver the orders and then it was a customer you're going to see a large number of restaurants yeah nine dispar visualization and as we start to get busy we start to not able to deliver the orders as quickly as we otherwise would and so we start to reduce the number of restaurants you see there's a busier and busier we reduce it further and further eventually I'm showing well together this is so the every customer who orders from us has a good experience and if the algorithms work correctly then it should come back down again and you'll be able to see all of your restaurants as you did at the beginning let's on the graph I just have this delivery network light and what do I mean by that how can you measure that if we're gonna have an algorithm that works we need to quantify that somehow and so when we first started working back to this graph of ordered vs. writers we took the ratio of the number of orders we had to the number of writers we had and use that as sort of an a way to inform some of our human operators who would manually stop people from being able to order and this worked in the early days for as we scaled up this doesn't really scale with the company and just looking at the number of orders relative to the number of writers you start to run into problems that not every order is the same and so you can kind of see in this part map here we have a writer a a restauranteur B and two customers at C and D now they cost to us in terms of the amount of time it's going to take a runner to deliver this order for C is going to be less than it is for D just goo they've got to travel further and further more in this example this is where the writers actually delivering both these orders so we offer them the chance to deliver both of C and D from B ever seen from at the same time and this will reduce the per order cost to us in terms of writer time and just that ratio of orders to writers doesn't take into account any of this information it doesn't capture that and so it becomes a kind of a bad proxy for how busy we are so instead if we said that they take different amounts of time and that causes us to be different amounts of busy maybe we should look instead the amount of time it's going to take and before I said we had this algorithm Frank that decides which ordered writers we offer which orders Frank actually looks right into the future for all the orders we have and all the writers we have and decides roughly which orders we're likely to assign to or offer to expire in the future and as you can see here with three example writers they all have two orders but they're going to take quite a different amount of time between them to deliver these two orders that ratio is not the right measure here in fact maybe just time third until the writer is free there could be a better measure so if these were all in a similar area we could say that well the next time we're going to have a writer free it's going to be in roughly 25 minutes and this could be a better measure of how busy when we had this restaurant list the customers had a large selection of restaurants they could choose from maybe they're really fancy one specific restaurant and maybe that restaurant is here and the final customer on the previous chart could be where these three writers are now and here we see that second writer the motorbike is actually much further away from this restaurant the customer will still order from and so instead we can add that travel time until whenever and let's get the time that the rider would arrive at the restaurant if they finished what we already had planned for them this is a much more realistic view of how busy we are in terms of a specific restaurant and we do this we fare every restaurant that is open we look at which how long it's going to take us to get right us there after they've completed their orders and use this metric in order to determine how busy we are and so here we will reduce thirty five minutes and just let show you that this sort of metric makes sense I have some data from a few weeks ago so on the x-axis here we have that metric I was just describing yes amount of time it would take to get a rider to a restaurant I don't know you have a couple of measures for how busy we are or how degraded our services so on the Left we have delivery network delay this is the amount of time or how delayed the writers going to be arriving at the restaurant and we can see that's really nice sort of linear correlation linear between this metric I defined and how late they get to the restaurant I've just read up concerned the delay gets higher there but we do also protect this delay and we tell restaurants how late the riders expected to be to make sure that the food is prepared at the right time and customers still get hot food it doesn't just sit there for this time and on the right hand side we have quite a similar metric in some ways the order duration so the amount of time it takes from a customer placing an order until they receive it and again we have this nice sort of linear correlation and so this is our nice grounding of a metric we can use to try and decide how many restaurants we show and all the ones going to tell you about how we decide which restaurants to show hello good evening so if I were to summarize Penn has so far explain to us what note actually means and we can see from this these charts that if float is high then delay would also increase and it also takes longer for the order to be delivered and what I would like to cover now is how we actually manage load okay so far what is load how to measure a lot and now the second part of this talk is about how we want to manage load so what we see here is a map ok colors notice I expected but so imagine we have a customer you see in the center and this customer wants to order food and the normal circumstance is we would show basically all the restaurants there where we can deliver food from within a reasonable amount of time and that is true as long as we have the right amount of riders okay so if we have enough riders we can actually deliver from wherever the customer wishes from but Spano I said but there are sometimes unexpected events such as rain and it rains and not all riders will turn up and what we have is then a situation where load increases so we get busy because a lower number of riders have turn up and presumably the Imola have to deliver the same number of orders if everyone gets hungry and and this of course cause our causes a problem to the network so how can we solve this so if we only look at the restaurants ok where we think we can deliver food from within the reasonable amount of time given now the shortage in supply then we would tend to pick the restaurants okay sorry the the darker one brogre symbols that are closer to the customer because therefore if we choose only to show those restaurants to the customer then we can see that the travel time or the time that it takes from the restaurant to travel from the restaurant to the customer is shorter therefore we would use less of the resources we have the fleet so we can manage load by shrinking reducing the delivery area so whereas before if we had a sufficient number of riders we weren't allowed to travel longer distances allowing the customer to order from restaurants which are further away now we are saying if we do not have enough riders if the load in the network is too high we would then say ok and show smaller number of restaurants and preferably those who that are closer to the customer such that travel distances are shorter making sure that the network during high load is more efficient because if we already have a lower number of riders we don't want to additionally ask them to travel for longer distances the question then arises the so how should we then set this optimal radius okay so how can we possibly do that there are two extremes okay so one would be to say well just show there a customer one restaurant which is the closest ok this would surely make sure that the network would recover faster because the writer has to travel very very short distances of course that's not optimal for us because if I were to show every customer just one restaurant the chances that he will he or she will order from this one restaurant is not very high so that in that case we may have recovered the network but at the cost of actually not getting any orders and that's actually of course not ideal so if we want to have more orders what should we do well we could possibly order actually offer everything ok even distances that we further have not considered that would surely increase conversion because we show there were customer much more restaurants and which would also ding the chances that the customer order order from but of course that goes against our original goal as to making the zone more efficient as to managing the load in the network efficiently so these are the two extremes okay either making very very short distances showing only the closest restaurant or allowing the customer to order from almost everywhere showing very very long travel distances and somewhere in the middle should be a nice niche area where we are aiming for so we have an algorithm that's trying to figure this out for us and in this particular case we use a regular eyes logistic regression what we can see here is that on the x-axis we have the livery Network load the feature that been explained to us and this is one of the input features and what we want to do with is we want to predict as to how high the load will be in X minutes okay so I phrase it that short has to what is the probability that we will be busy in X minutes okay so the higher the load is the higher the probability is that we will be busy in next minute and the lower the load is the lower is the probability that we will be busy and that red vertical line is the maximum or network load that we are able or they were happy to endure to serve customers under because we know that if we deliver order up to that Network load the service levels are still within agreement but if we go any beyond then you can see from one of Ben's previous slide that if we if never was just slightly been higher all of a sudden we shouldn't show any restaurants and customer can't order from anything so that is then a scenario which of course we want to avoid so we have I recognized rajasic regression and network load is just one of the features that we put into this model now of course also other features such as the time that it takes for the writer to travel from the restaurant to the customer the actual delivery area at the customer and the right iron time of day okay because there are peak times such as lunch and dinner where we experience different levels of busyness and a of week okay so weekend's are not DIF different to weekdays and a variety of combination and interactions effects of these and other features so again well pointer is not working so there are two extremes that we want to avoid so we do not wish to be in an area where we really have no very very low load such that just offering one restaurant and the other extreme would be that we do not want to go to the other extent that we actually offer everything sorry so that we end up with a very high load State so how are we going to do this if we have such a model how are we going to use this to manage load so as we do with any other train model we first train the model and then use it to predict so I in this graph I just show one of the features which is the restaurant - customer time so having trained this model we would be able to predict if we just feed in the features as to say okay if the feature is this how likely is it that we will end up in a busy state once you have used this model to predict it we can also use the same train model and invert it such as to say okay now if that's the case if I am likely to end up in a busy state with a probability of X what could I possibly do to end to end up in a non busy state okay so if which means it here I wish to leave a busy state I wish to enter a non busy state what is the corresponding feature that I should actually aiming for such that I can leave the current busy state and then a non busy state so we have trained a model inverted and from this inverted model we can then read out the feature that would allow us to get the desired result result and then use this feature to set the maximum travel time we are looking for so this can work like this this is a just an illustration okay so what do we see here is the number of riders on two consecutive Saturday afternoons the blue one is the number of riders on a rainy day and orange or yellowish one is the number of risers on a sunny day and we can see here on rainy days let's write a show up what happens to the load well following the discussion we have from before when it's sunny then network load is lower than when it's rainy because if we have less riders on rainy days we are getting more busy if we are more busy the network load will increase okay so far so good now let's focus only on the rainy days because those are the days where load management or load management are going will become most active again what you see here the red dotted line was that line that we are able to happy to surf orders if the load in the network gets any exceed this particular threshold we would then decide not to show any more Kuster on to the customer such that the the delivery network can recover so how does it work now so what let's combine this what with what we have discussed before okay so our goal was to figure out how many restaurants or actually which restaurants we want to show to the customer okay so the customer the middle those are the burger restaurants in the area that the customer could have ordered from and how using the model a lot gistic model inverting it will allow us to derive a particular number of the restaurant to customer and it works as follows so when we have a high number of riders then we have of course a low load because we are not so busy if we are not so busy then we said before we would be very happy to show a selection of restaurants where the rider could travel very far away from the customer because we have the work feed we this we have the capacity to do so okay ordering food allowing the customer to order food from for further a restaurant does not cause any harm to the network if we are have the right number of riders in the network however if we enter state where the number of riders drops and as a consequence of this the network load increases then we have said and we need to shrink the zone such that the customer can no longer order from restaurants or further away and that's how the algorithm works so it it's used as a trained model and in every cycle it keeps continuously predicting in real-time as to how should I set the optimal radius if you're so ish of the delivery area for such that the customer can always see the optimal number of restaurants such that the ideal load doesn't go any beyond the threshold we should be clear that it's not our goal to kill for a killed load okay so there needs to be some load so we are not aiming for a scenario where we have zero orders okay because that's where load is lowest but that's not what we're aiming for so in a way you can see this as I opt Amis a ssin algorithm where we wish to increase or maximize the number of orders during high load and the last chart is where you see in the blue solid line the average number of restaurants that customer can see in that particular delivery area and the green dashed line is the maximum number of restaurants that are in that particular delivery area so meaning when load becomes high we have to manage it such that we can't no we can no longer show everyone all possible restaurants we have to show them less restaurants ideally those which are close at the customer in order to make the network more efficient of course this is just a temporary measure and once load recovers then of course we are very happy to show again all available restaurants to the customer so this is just one illustration at one example from one Saturday afternoon and then you might want to ask it does it always work so all algorithms that we design and test on delivery always have to go through a experimentation framework and this algorithm was just tested month ago with the following results so this is the new algorithm where we exactly do this dynamic load management where we always dynamically calculate the optimal or restaurant to customer travel time to make sure load is optimized and what we can see here is the new algorithm manages to avoid accepting orders when the delivery network is already suffering from extreme high load okay meaning if we are already busy we shouldn't additionally accept more orders and but it is not that the new algorithm is always closing the door and pushing customers away the algorithm is also intelligent enough to say when we are not busy however we should show more restaurants to the customer and we can see in this light green area that we do get more orders when we are not busy ok and as a result of this this has shown the following improvements in the network such as when the network is extremely under pressure and we have Network load the new algorithm manages to decrease the delivery time by shrinking the zone and at the same time we can also shorten the delay that customers are experiencing while on the other hand side although we do accept more orders when we're not busy we are not increasing those measured those measures so this is one example of the algorithms that we do here at the live room other colleagues and other teams working on other data science projects such as recommender systems and personalization so when you go to a web pages you are always greeted with a very very delicious selection of food but depending on your profile you want the list and the order of restaurants that you see will differ we're also working on projects on forecasting and scheduling optimization meaning here the task is to forecast what the order volume in future will be such that we provide the right amount of riders and schedule the supply in advance ok showing allowing riders to book particular slots in particular areas such that we can measure the expected order volume with supply we also work on dynamic pricing meaning we want to always calculate the optimal price we can show to riders such that they can that they are most likely to accept the order so not every order is equal ok sometimes writer prefer orders that are have a short distance or sometimes it is the restaurant that they wish or do not wish to go to or the area time of the idea week so there are variety of features that go into different algorithms and models in this area but this is just one of the projects in the primary pricing that my colleagues are working on just to also show you a small selection of the other design projects that we are working on so I hope you enjoy at the top [Music] [Music] [Music] you