hello everyone in this video we are doing a overview of VPC endpoints we're going to talk about what they are why they are useful from a networking perspective and then at the end I'll show you a demonstration of creating a VPC endpoint and interacting with it from a Lambda function that's located within a private subnet so that's the agenda for this video Let's jump into it and the way I'd like to explain the utility of VPC endpoints is by first talking about the problem that VPC endpoints attempt to solve and so let's assume for a moment that we have a VPC and we have two subnets we have a public subnet and a private subnet and then within our private subnet we have an ec2 instance and our goal is to make it so that our ec2 instance has the ability to speak to Amazon S3 or any other AWS service that exists in the AWS Cloud so by default it is not going to have the ability to do that because it's in a private subnet and you're not allowed outbound internet traffic if you're in a private subnet so what folks would typically do in order to make this work using kind of a naive approach without VPC endpoints is the first thing you do is you'd create something called a Nat Gateway and that stands for Network address translation and then from there we'd have our internet gateway which is just a default component that comes with our VPC and once we have these three components set up we would wire them all together so our ec2 instance whenever it tries to make calls to the internet that traffic would flow through the NAC Gateway and onto the internet gateway and then from there the traffic will be forwarded out to the public internet and here's the key the traffic would exit the AWS Cloud so that traffic that is destined for S3 would exit the AWS Cloud into the public internet before finally being able to interact with the Amazon S3 service and then once it does all that information is going to be returned following the same path that we just took back to the ec2 instance so this is how you would typically solve this problem from a networking perspective if you have a private subnet and you're trying to talk to to a service like Amazon S3 and so what are the issues with this approach well there's two important ones and maybe a third if you like to nickel and dime but the first one is in terms of cost not gateways do cost a pretty penny to set up and maintain you're charged by the hour as well so these can eat into your expenditure and they're kind of not really a component you want to have to have unless you really need it the second one is in terms of security so in this case the traffic is leaving your VPC it's also leaving the AWS cloud and then it's coming all the way back in so not ideal from a security perspective there could be some vulnerabilities or loopholes that hackers can attempt to exploit and the third arguably maybe this one is a concern to you maybe it's not is that a not Gateway is just an additional point of failure although this is a managed piece of infrastructure from AWS and the likelihood of it going down are slim to none can't be ignored that if a Nat Gateway breaks or goes down or there's some kind of Hardware failure then you're going to have some intermittent outage in your connection so this is is the problem that VPC endpoints attempt to solve let's take a look now and see how we would do this using VPC endpoints and how much simpler it is okay so same diagram we just made some extra real estate to demonstrate some Concepts here so the first thing that you'll note is that we do not need to speak to the public internet and so the first thing we would need to do setting this up is that we would need to create a VPC endpoint and then VPC endpoints are kind of mapped to a particular AWS service so if you want to talk to different AWS Services then you need to create separate VPC endpoints so in our case we would wire this up to S3 in this example since that's the service we want to speak to and then from there it's not completely over yet there's one small little thing that we need to do we need to take care of what are called security groups when you create a VPC endpoint it has a security group that you need to associate with it there's also a security group that's associated with your ec2 instance and you can think of these as rules for your pieces of infrastructure that help govern the networking traffic on those particular instances of infrastructure so we have two security groups in this case one for our endpoint and one for our Lambda function so to set this up correctly we kind of need to set up some mutual rules between these security groups so we would need to say from the VPC endpoint perspective allow inbound traffic from the ec2 security group and then from the ec2 security group perspective allow outbound traffic to the VPC endpoint Security Group so only after you do that will you be able to establish network connectivity from your ec2 instance to your S3 service and that's all you really need to do to set up connectivity to a supported AWS service and so as you can tell this solution is a lot simpler a lot more secure as well your traffic is staying within the AWS Cloud no longer leaving the public internet via your internet gateway you also do not need a Nat Gateway in this instance in since it serves you no good and you're not relying on it to call out to the public internet so overall a whole bunch of goodness by using VPC endpoints one small thing that you should know about as well is that with VPC endpoints you do need to enable DNS host resolution that's a setting on your VPC if you go to the console just go to editvpc there's two options in terms of DNS resolution that you need to allow or else this solution will not work all right now that we understand how VPC endpoints work let's head over into the AWS console and I'll show you how to set one up using step functions as our destination service that we want to talk to and then instead of using ec2 we're going to use a Lambda function so I'll see you over there in a moment alright folks so here we are in the AWS console now just a couple prerequisites for this video in order to follow along you need a VPC that has a private subnet I'm going to be using the default VPC in this video but I do have another video where I show you how to create a VPC with a public and private subnet so if you need to set that up go ahead and watch that before you watch this section things will make a lot more sense okay so let me just show you what we have here really quick so first of all we have a VPC here if we go into your vpcs so this is my default AWS VPC and if you go into the resource map here and you go to subnets you can see that I have a private subnet here and this private subnet is configured to have a custom route table called private demo Dash RT you can see that it does not have internet connection to this internet gateway basically everything stops at this route table and the rules in this route table are pretty straightforward I'm just going to open this really quick it's just for local traffic so everything within my local network will be allowed using this default route table all right so what we need to take care of first is we need to create those security groups the ones that are going to be attached to our VPC endpoint and the one that is going to be attached to our Lambda function and the reason we want to do that first is because if we try to go and create our VPC endpoint first it's going to ask us for a security group and we don't have that yet and it's going to cause us some headaches if you try to come back to this later and edit it and the same thing for our Lambda function when you create a Lambda function within a private Subnet in a VPC it's going to ask you for your security group right out of the box so if you don't have that you're kind of out of luck so let's go into the security section over here and then we're going to click on security groups and so we're going to create two security groups so let's go ahead and click on create Security Group and we're going to call this security group name so vpcnpoint Dash demo and then description we can just copy paste this if we want and then for inbound rules so this is the inbound rules for our VPC endpoint now the interesting thing is that we don't have the security group as of yet for our Lambda function so this isn't going to work get right we haven't created the security group for our Lambda function quite yet and that's going to come later so instead what I just want to do is create these two security groups as placeholders then once we create all the infrastructure We'll add all the rules and I want to show you how this works when we don't have the security group set up correctly as well you'll see that we get a timeout exception when we're working with our Lambda so that's what we're going to do here so we're going to actually delete this rule for outbound rules just get rid of everything so we have a clean slate gonna go to create Security Group okay that got created let's create another one going back to security groups and create Security Group let's call this Lambda Dash VPC endpoint demo and copy this as well and similarly let's get rid of everything here so these are completely blank security groups now okay so we have those two things set up that's perfect now what we can do is we can go ahead well you can do two things you can go create your Lambda function first or you can create your endpoint first whichever you prefer I'm going to create the endpoint first and then create the Lambda function and then we're going to mess with the security groups to make it all work all right so let's go to endpoints endpoints or endpoint Services um yeah it's just endpoints actually and then we're going to go to create endpoint in the top right here so we're going to click on that we're going to call this the step functions VPC Dash endpoints and like you can see here like select the service category we're going to use AWS Services there's a whole bunch of AWS Services here there's over 23 pages each have about like 10 or so so like 230 or so service right here 221 services that are supported the thing that is a little bit confusing is that the names aren't always that clear so like we're searching for step functions and there's no step functions here and this kind of took a little while for me to figure out but what you need to search for in the step functions case is like the the name is is kind of weird it's this so you type in States and it's you know com.amazon aw us dot usc21 dot States and if you want to call synchronous step functions it's a different endpoint if you're using Express workflows and so that requires setting up a different VPC endpoint so this is the one that we're going to need here this top one so just for normal state so we're going to click on that make sure you click on the little radio box here to specify it then it's asking us which VPC do you want this to be applicable for this is our VPC our default one you can mess with the additional settings if you want we're going to leave this as default on for DNS name and then you need to select the subnet that you want to place it in or the availability zone of that subnet and so we're going to select Us East 1A for availability Zone and select our private subnet here and so IP address type you need to select one here it currently supports V4 so we'll select that and then for security groups this is where you need to put in the VPC endpoint security group that we just created so that's not going to work what did I call it VPC Dash and I I hope um oh boy which one was it oh it was this one VPC endpoint demo you can see I was doing some experimentation earlier oh this is the one that we created for the Lambda so we'll come back to this later anyway select the security group that we just created and then we are going to go down and click on create endpoint so what that basically did is that it created a VPC endpoint and then assigned a security group to it so if you select this now and you scroll to the right you should see security groups okay they don't have it here they should have it down low yeah um right here under security groups wow the UI is terrible here yeah so here's the security group that we just created and we assigned it to this VPC endpoint and as you can see it is currently in pending modes we're just going to wait a moment or so for this to complete and then we're going to move on to creating the Lambda function in that private subnet okay after a couple of minutes of diligent refreshing we can see now that is currently available so we can move on so let's go to our other tab here and we are going to create our Lambda function in our private subnet so we're in the Lambda section of the console all right so we're going to go ahead and click on create function in the top right here we're going to give this function a name let's call this step function VPC endpoint demo and we're going to be using python 3.10 here architecture doesn't matter permissions don't matter and you don't even need to set up the IEM rule to have step functions permissions in this specific case since it's not going to be required to illustrate our network connectivity example although you can set that up if you really want to but we don't need to in this case make sure you go to advanced settings and you enable VPC here and now we need to select our VPC and then we need to select the subnet which is the private subnet which is this one here which I labeled private subnet that's asking us to choose two we're going to ignore that although in production I probably would and then for security groups we are going to filter down to our VPC endpoint Security Group yeah so this was the one we created for it so so Lambda VPC endpoint demo so hopefully this illustrates what I mean if we didn't create these security groups in advance we wouldn't be able to select anything here and it would kind of be a bit of a problem so that's all we need to do and you can actually just see we have no rules for this Security Group which is perfect it's desirable so let's go ahead now and click on create function now because we are creating a function in a private VPC this does take a little bit longer than normal to create your Lambda function so I will fast forward this when it's done and we will move on to the next step alright so after a while we finally see we successfully created this function so let's go and test out right now let's try to call the step function service and see what happens and should be no surprise here so I'm just going to paste in some code if we just take a look at this really quick we're kind of making a client here for the step functions um photo 3 client and then we're calling the start execution API you can put in your arm here and your your name and your input if you want but we don't need that to demonstrate this so let's just deploy this really quick and then try to create a test event and test this out gonna click on Save and we're going to click on test now and you'll see one seconds two seconds three seconds and error message tasks timed out after three seconds you can change the timeout to whatever number you want but this will never work because we're not allowing the correct connectivity based on the security groups of our VPC endpoint and of our Lambda function so that's what we need to correct in order to make this work so let's go and do that now in order to do that we're going to just open up a separate tab here and then we're going to go back into security groups and we're going to modify those security groups that we just created so I think it was these two so VPC endpoint demo or Lambda and then VPC this UI man I swear okay so here are the two so this is the one for the endpoint so let's go ahead and modify this one one first so what we need to do for this endpoint is that we need to allow inbound rules for all traffic from our Lambda Security Group so let's go and change that so we're going to go to inbound rules and then we're going to add inbound rules and we're going to add a rule and we're going to say all traffic all traffic and for the The Source you're going to put custom here and then just click on here and then we're going to find the name of our Lambda VPC Security Group which is it currently attached to it so it's this one here Lambda VPC endpoint SG so we're going to click on this go ahead and click on Save rules and so now it has the correct permissions let's do the opposite now for our Lambda Security Group so this is the one right here I'm going to click on this and this one needs outbound access to the VPC endpoint Security Group so we're going to go to outbound rules we are going to edit outbound rules and similar story we're going to do all traffic all traffic and then we are going to select the VPC endpoints which is this one this is the security group for the VPC endpoint that we just created in the earlier sections of the video okay so that's all we need to do now let's go ahead and click on Save rules and that's good so that setup is done let's go back to our step function go back to our code I just want to make a small change here and redeploy because sometimes if you don't do that right away it doesn't pick up the new changes so we're just going to do that and then now if we test this out remember last time we got a timeout after three seconds we're probably going to get an error here because the Arn doesn't exist and there you go so an error occurred in valid Arn when calling the start execution operation that aired out but at least the network connectivity is working so that you know we got everything right so if you enjoyed this video please check out the other ones on my channel and please don't forget to like And subscribe I'll see you in the next one