Transcript for:
Real-time Sports Data Processing with AWS

imagine watching your favorite sports life while watching this you get realtime informations like players movement heat maps and even AI predictions today in this video we are going to talk about an architecture through which we are going to create a platform using that we'll be able to analyze and get all these information on Real Time basis we are going to use AWS event driven architecture through which processing realtime data will be much more easier if you someone who is interested or who is looking forward to learn how you can process real time data this is the perfect video for you let's Deep dive into it let's start with understanding the problem which we are trying to solve when you watching a Live Match there is massive amount of data which has been generated consider player who is having a suit fit with all iot sensors that generate in data fan interaction and even life Updates this amount of data needs to be processed on Real Time teams rely on this data to make predictions to do further changes so on and so forth this makes this data very much important and very casual to handle it in this we are going to create a platform through which we'll be able to manage this massive amount of data on Real Time basis now that we have understood the problem that is having this huge amount of data and that's need to be processed on real time so for this we are going to leverage AWS event driven architecture through which we'll be able to analyze and get this data on real time without losing the accuracy speed and scalability of the application we are going to use a lot of different AWS Services which includes AWS Lambda functions Amazon Genesis ands we are going to talk about each of them step by step but before going to that let's understand what is an event driven architecture so when we talk about event driven architecture we are creating a system which is based on an event now when I say event event is nothing but an action or a trigger so if there is any action which is getting performed based on that particular action I'm going to process my information this helps us to make our application much more scalable because we are not relying on predefined infrastructure we are defining our infrastructure based on the requirement which we have let's just Deep dive into how we can implement this entire architecture and understand how this platform will work the first step of our architecture is event inje when I say event inje this is nothing but we have to ingest the data that is we need to capture all the data which is coming in now in a sports event there can be multiple data sources it can can be from the iot device which are connected it can be from internet which is giving by fan interaction or live stats which are all these data needs to be streamed in such a way that there is no datab bo using Amazon Genesis we'll be able to stream all the data and then process it nearly real time I'll be able to get this maximum amount of data and then I can store it somewhere so that I can do further operations as well can I helps us getting data in real time as well as we'll be able to process it on the flow kesis handles the flow of event with no delay when there are crucial moments like a penalty shootout and extra time every second counts and Genesis handles this data very smoothly once skesis has captured all the data now the next step is to process this information and to do so we are going to leverage Amazon's Lambda function Lambda is a service which is used as a serverless when I say serverless it's nothing but having a enhanced computation power wherein we don't have to manage any servers Lambda helps us to manage serverless the compute power which we are using imagine a situation where in your sports s there is an extra time going on and at the very last minute there are lots and lots of users which are coming in and watching your videos for that we need an extra power after a while there's a half time for that the audience will be very less Lambda will automatically scale down and it will be able to process that information within that given time frame so the scalability is been managed very powerfully when we have AWS Lambda now that we have defined our processing power so Lambda we need to Define what will be the trigger for myda function consider each event such as pausing of the game any half time any score are an event this will trigger my L function but to manage this trigger we will be needing an architecture which will decouple my process and to do that we are going to use a service called as sqs that is simple queuing service this will help us to create a queue through which data can be processed on each and every step imagine you have each step after each step is been performed your Cube passes to the next and then process that same information similarly once the process has been complete we are going to use another service called as SNS simple notification service this will help us to get notification about the insights which we have generated so now our architecture will be where we are getting an Insight from our lbda the trigger will be created through sqs that is let's say an information about half time which will hit a Lambda function once it hits it will move ahead to an SNS notification the data is been processed and then I'll be able to get that information through a notification that information has been processed now that we have processed of information and also got notification of the same we need to store this information now storing this huge amount of data can be a tedious task and we need a resource which is scalable enough to manage this huge amount of data and for that we have a service called as Amazon Diamond D it's a no SQL based database which will help us to store all this information on the real time so imagine you have information of your player let's say we can manage and store information like how much he ran how much calories he burned so on and so forth that can be very easily managed and stored in no SQL database that is Dynam TBL now there is another type of information that we need to store that is historical information consider you have stored you have an existing information that you need to revise or take analysis at a later point of time for that we are going to store this in Amazon S3 buckets so whichever let's say there is an match this has happened 2 days back and you want to get that information we are going to store it our S3 bucket so first we'll store it primarily in D cables later on moving it to S3 bucket so that we can use it for further analysis when I have to analyze this data from S3 bucket we can leverage another AWS service called as AA which will help us to quer this information and get the exact information what we want speaking of analytics we can analyze using a that is the data which we have stored previously but we also need to process or visualize the data which is real time for that we are going to use Amazon Qui this helps us to create realtime dashboards which will be used by maybe coaches or the sports enthusiasts who can keep track of players quick site can be connected with my databases and then from that data I can create dashboards or charts which will help me to analyze information we have created the back end of the application but we need a front door through through which users can go ahead and access this information for users to process or to get this information we are going to use a service called as API Gateway this will become front for a mobile application a website so on and so forth through which user can come in and get the information this application is going to work for millions of users they are going to get information on real time and information when they actually need it finally let's talk talk about scalability of the application when we have scalability first we need to monitor how my application is performing for that we need to add a service that is Amazon Plum watch through which I'll be able to keep an eye on my resources if there are any problems or any alerts it will notify me to handle this notifications or alerts we can add an autoscaling group through which I'll be able to scale my application on Real Time basis so let's say if there is any World Cup going on and my number of users increasing autoscaling group will be managing all these number of users now that we have created a fundamental application we also need to manage the security aspect of this here we can use AWS Cognito to create a user pool that is user authentication system through which user can come in and authenticate themselves and then use the application we can also add a WS B that is firewall which will act as a firewall between our application to protect from cyber threats this architecture is a game changer when we are talking about event driven architecture if you have to create an analytical dashboard in which you want to implement decoupling and create everything based on event you can refer this architecture based on this you can create many other platforms and leverage AWS power so that you can make your application much more scalable and process the data on Real Time basis if you excited about learning AWS architecture or any architecture about realtime application hit that subscribe button happy learning