💾

Google Cloud Storage Transfer Service

Jun 28, 2024

Lecture on Google Cloud Storage Transfer Service

Overview of File Transfer Services in GCP

  • Various services for file transfer:
    • Storage Transfer Services
    • Cloud SDK
    • Transfer Appliances
    • GSUtil
  • Favorite: Storage Transfer Service (Robust and versatile)

Demonstration of Storage Transfer Service

Assumptions

  • Using a virtual machine (VM) on-premise
  • VM within a GCP project
  • VM unaware of Storage Transfer Service

Main Components of Storage Transfer Service

  • Transfer Agent:
    • Key component
    • Reads and picks up files from VM
    • Drops files into GCS bucket
  • Agent Pool:
    • Lightweight agents on compute instance (VM on-premise or other Cloud vendors)
    • Tagged with GCP Pub/Sub service
    • Detects changes in specific folder location, picks up new/changed files, and notifies Storage Transfer Service

Steps to Set Up Storage Transfer Service

Creating an Agent Pool

  1. Go to Storage Transfer Service Console
  2. Look into Agent Pools and Transfer Jobs
  3. Create Agent Pool:
    • Provide pool ID
    • Set bandwidth limit
    • Connection parameter initially shows "no connection"
  4. Install the agent on the VM:
    • Select directory/root of file system on VM
    • Run provided g-cloud command on VM to install agent
    • Ensure VM has gCloud CLI and Docker installed
    • Authenticate and install Docker if prompted
    • Check agent connection status in Agent Pool section (should show as "connected")

Creating a Transfer Job

  1. Go to Transfer Jobs in Storage Transfer Service Console
  2. Create Transfer Job:
    • Select Source type (e.g., Posix file system)
    • Select Destination type (e.g., GCS bucket)
    • Configure frequency (daily, weekly, custom, etc.)
    • Set transfer parameters (e.g., when to overwrite/delete)
  3. Monitor transfer job for successful execution
    • Scheduled jobs will appear in the Transfer Job section
    • View job details for progress, file size, date, etc.

Example

  • Files dropped into specified source directory on VM
  • Agent detects change, notifies transfer service, and files transferred to GCS bucket

Verification

  • Check GCS bucket for transferred files
  • Internal/log files may also be created

Use Cases for Storage Transfer Service

  • Ideal for file transfer tasks
  • Useful for data migration strategies

Conclusion

  • Highly recommended for transferring files from on-premise to GCP or between different Cloud vendors
  • Offers a robust and reliable method for file transfer

Subscribe for more informative videos!

Full transcript