hello everyone Google Cloud offers a variety of services for you to transfer files from on premise to gcp or from different Cloud vendor to gcp it could be visa versa from gcp to on premise and different Cloud vendors such such services are storage transfer services Cloud SDK transfer appliances you probably are familiar with GSU till out of all these Services my personal favorite is storage transfer service which is really robust and it can solve most of the file transfer uh use cases so in this uh video I'm going to show you how storage transfer services work and how you can set it up with a very simple example it can be applied at a very production level solution as well so without any further Ado let's jump right into it so for the sake of this discussion I am going to uh uh assume that I have a virtual machine on premise for the demonstration purposes I'm going to spin up a virtual machine with within this gcp project where this VM does not uh is not aware of the this storage transfer service and we will make sure that this transfer agent which is which is the main uh component of the storage transfer service can read files read pickup files from this virtual machine and drop into G GCS bucket so I'm going to show you the VM that I have in this case this is the virtual machine it's up and running on the left side of the screen I have the virtual machine and I'm going to go into the storage transfer service and create uh agent pool so before I create an agent pool let me show you what exactly the storage transfer Service uh console uh console would look like so there are two components here the agent pools and transfer jobs uh let's go into agent pools so agent pools are basically lightweight agents which sit on your compute instance it could be a VM on your on premise it could be a VM in a different Cloud vendor and that age agent is tagged along with a pub sub service in gcp where it constantly looks for changes in that specific folder location and if if it detects any change in that folder location be it a new file get loaded into that folder location or a change in that change in that specific folder it picks it up and drops into uh it picks it up and sends a message to storage transfer service where storage transfer service picks it up and drops into the GCS bucket it could be GCS bucket or it could be another posix file system so let's go ahead and create one agent pool so I've already created a couple of them but but let me show you what it takes to create an agent pool so I'm going to create uh select on agent pool it will ask you to give the pool ID and it will ask you to set the limit I'm going to select no limit in your case if you have a limit bandwidth limit go ahead and use it once you create that agent pool you will see the agent gets created like this where the connection parameter shows no connection we will I will show you how you can how how we can basically connect this to the virtual machine and a location where file needs to be picked up so let's select the one that we just created and select on install agent in here it basically asks you to provide specific directory or a root of a file system where files needs to be picked up so I'm going to select posix file system and give pick the specific directory where I where I want the files need to be picked as you can see there are a couple of folders that I've given already so feel free to name whatever you want but the important thing here is to pick up this g-cloud command so this is exactly what you need to install the agent in the virtual machine where files need to be picked up so I'm going to select this and go back to my VM and run this command so just for the demonstration purposes I've already installed an agent prior to this demo so I'm going to show you the history of the commands that I ran so this is exactly the agent installation command but if You observe here in the install agents uh guidelines it will tell you that that you need to install gcloud CLI and Docker into that specific VM to have this agent to be installed so since this is a VM within gcp my g-cloud uh CLI is already installed but I need to install uh Docker so the these are the instructions you will get prompted when you run this command so don't worry about having uh knowledge about these things so I first went ahead and uh ran this command which is when it asked me to authenticate here this is pretty well-known command where you run this it will create a very huge link to authenticate once you select that link it will generate a code copy that code and paste it into the into into the command and then it gets authenticated automatically and once that's done try running the command again which will prompt you saying hey you do not have Docker installed so let's just go ahead and install Docker so this the this is the command it will prompt you to do this is basically two commands one to install the docker command two in two the second one is to install the agit so I'm I went ahead and ran this Docker inst uh Docker enable command and once this is done the docker should be enabled and this is the final g-cloud install agent command which is exactly what we've copied from here and once this is done the agent is successfully installed into your virtual machine once this is done you uh the way how you can check if this agent is successfully install is to go back your go back to your agent pool section and check if the connection says connected or not so this is my agent that I successfully connected to this virtual machine so we are good with with our agent so let's just move move ahead with transfer job so in this case I am going to create a new transfer job as you can see I've done I've I have tested a couple of transfer jobs before but why don't we just go ahead and create one so I'm going to select create transfer job and select Source type in this case my source type is posix file system and my destination type is GCS bucket I want to pick it up from my uh VM and drop it into the GCS storage bucket so let's just select next uh step and then select the agent pool that we just created which is uh this and then the source uh directory path in our case the source directory path is where we I need the agent to pick up the file from so give the source directory path here and select the next step so I'm going to just give home here but in your case give the source directory path and then next and then give the destination location in this case I'm going to select my GCS bucket one of the GCS buckets that I have and move forward with the next step this is where you can configure this uh transfer job to run once in a day once in a week or you know if you have a custom frequency if you want to run it ad hoc and if you if you can also set it up based on you know time uh you know you have several different options where you can run this so let's move uh let's pick something and move forward so I'll show you what are other options you need to pick up so I think I'll just pick up 26 and select give a name for your job and there are a bunch of parameters you can select when to overwrite when to delete based on your logic based on your requirement select the option that you want and once this is done as you can see I've scheduled a job before this this this demo and this is successfully uh executed so this is how you get a prompt when a job is successfully executed within the cont console so once this is done just go ahead and create and your transfer job within the transfer transfer job section you will have another one listed here which says hey your job is scheduled okay so this is this is what you need to do from the storage transfer service from the VM perspective I scheduled previously I scheduled that I I need my file to be picked up let me just clear this and I'll show you what exactly I uh dropped into this folder location so I I dropped a file called first file and then I dropped a file called second file and based on my schedule the agent pool will look for this folder and see if there is any changes in it and picks up that file and sends uh a message to storage transfer so behind the scenes agent pools are tagged along with the pub sub service where it looks for a change and notifies the transfer service to to to to see the changes and if there are any changes pick it up and drop it into the GCS bucket or wherever your destination location is so let's just see yeah I'm going to show you yeah that's pretty much what you need to do from the VM perspective and from the storage transfer perspective so I'm going to walk you through an example of what a successful transfer job would look like you have as you can see there are three jobs here and I'll show you what a successful job would look like this is a successful transfer job where it will show you what exactly my scheduling mode is in this case it's a batch but not recurring and if my Pro progress is 100% or not and the size of the file and date time Etc so I'm going to select on this I think this is pretty much what it says let's check into my GCS storage bucket and see if this file gets loaded uh I'll show you what exactly the storage transfer service creates when you set set this up so I'm going to go back into the cloud storage and select where is it this is the file that I created this is the bucket uh folder in the bucket I just I created for this transfer job and this is the file that got picked up from this VM and loaded into here in addition to that it creates some internal files some um some files which I haven't explored much for you to explore you know B I think it's about the logs as such but yeah you can you can feel free to explore and see what exactly this is but yeah at a very high level this is how you can use storage transfer service to move files from on premise to gcp and it could be from a different wherever you have a compute engine wherever you have a file system that needs to be mve into GCS bucket or vice versa storage transfer service is very ideal and that is something that I would highly recommend for you to use for file transfer this can be used for your data migration strategy as well so yeah that's all for this video if you found this video helpful and informative please consider subscribing my channel I'll try to make more and more informative videos like this until then keep learning