how do you write data to Kafka using python it's one of those things it's easy when you know how so I'm going to show you how step by step we'll do it completely from scratch we'll go through every line of code until we've got some kind of interesting working application that writes to Kafka so to fulfill that promise let's create a brand new directory with nothing in it and prove that we're doing everything from scratch if you already know how to set up a python environment you might want to skip ahead a minute or two in the chapter Market but assuming you're still with me I'm going to say Python 3 use Virtual M to create me an environment to develop in isolated from everything else Source M bin activate and now I can do pip install a couple of libraries the first one I'm going to use is the standard requests library because I'm going to get some interesting data from the web for this and the second one is I want to talk to Kafka using python so I'm going to install quick streams and with that I can actually start writing my main script so that was Step Zero step one is to get some interesting data worth writing into cafka so let me import requests library and say requests.get some URL with some parameters and that will get me a response which I will print and then we'll know we're happy is what data am I going to get I don't go around memorizing Json API details for fun I promise you I promise you I'm not like that but there is one I know which is useful to keep around which is api. open.com V1 forecast is a really simple open weather forecast API useful for things like this thank you open Meteo for existing if I give it a couple of parameters uh the latitude that I'm interested in the guess what longitude I'm interested in um and then I'll be able to say I would like something like the current temperature at an altitude of 2 m above sea level if you're an American viewer you'll be wondering what 2 m is it's roughly the temp temperature that your Smartwatch is going to record when you hail a taxi for a person of average height standing up that's about 2 m give or take latitude and longitude again I promise you I'm not the kind of person that memorizes these things but I happen to know that 51.5 and minus 0.11 is London is enough for me it's walking distance to a part of London I really like um I could get that more accurate but then I'd have to be the kind of person that memorizes latitude longitude to six decimal places and it's not that bad yet okay with that why don't we run that and see what we've got and see if we're in a good place um python clear the screen Python 3 main I get 200 okay so um response. Json yes okay that's some data I can stick into Kafka we move on to part two and if you've just come back from the chapter markers we have some Json data that we'd like to write into CFA now we're all set up to go so let me edit that file once more and there are three steps to getting this into CFA I need to create an application object which I will import uh from Quick streams I will import the application object and then with that app I'm going to say app.get producer which will give me a producer which is CF cap Palance for a righton connection I'm going to take that producer and I'm going to say producer. produce something I'm going to give it the data at that point and then there's one kind of gotcha with connecting to CFA you really want to make sure that when you're finished with the connection you close it cleanly so I could at this point call producer. flush and that would close it cleanly before the program exits but I'm not going to instead I'm going to make sure that python calls it for me using with if I use the with syntax that will guarantee that any cleanup that needs to happen happens after this block and I think that's a much nicer way to write it because you can't risk forgetting if you do it that way okay so it's really just those lines provided I fill in these holes correctly so let's do that and application for for development time at least it only needs one argument which is where do I find cfer on the network what's my broker address and I'm running CFA in development on Local Host 1992 so I could do that there's one other thing I'm going to set which is going to make my life a bit easier I'm going to set the log level to debug for this so I'll get some more interesting output and you'll see what it's doing a bit better but that's enough for the application I can move down to ucing to writing to Kafka and for that I need three things I need to specify which Kafka topic I'm going to write to I think I'll say weather data demo for this I need to give it a kit well I don't actually need to give it a key keys are optional in Kafka it's a really good idea to supply one something that identify as this data uniquely a unique ID I'm going to say London I could give it something more fiddly and precise but I think London will do for this demo and then I can actually write in my value what shall I have for my value I think what I'm going to do is this I'm going to say um whether is that whether is the Json response and my value is take the Json library and dump as a string that weather and if I Jason up here now you could argue that's a bit strange I've passed the Json here just to re-encode it as a string here I actually like that because it's doing two things for me it's checking that what I got back was valid Jason before we try and write it and it's giving me access to the object because I'm going to want to extend this a little bit down the line into something more fancy but we'll leave that for now just to say do a round trip make sure everything's okay for now with that unless I miss my guess I think this is going to work let's find out so let me rerun that and see what we get yeah that was fairly fast so if you have a look here the app is starting it's starting um it is getting a delivery receipt for sending that data to the weather data demo topic a partition has been assigned which I take as proof that it's actually been written and there's the key in value we specified and as I hoped CFA producer has been correctly flushed before the script exits so with that I think we can call that a working how do you write to CFA with python app and we could stop there and if you're stopping there and you've got all the information you want please click like on your way out I'm glad it was helpful but give me a couple more minutes cuz I think we can do something slightly more interesting with this and make it look like a real application so I'm going to tidy this up a bit bit of refactoring I think this is worth making into its own function so let's do that and um that and then down here we will say weather is a call to get weather that's a bit neater this I feel should be a main function so let's do that and we'll do the standard python Preamble here if Dore name isore main then call Main why don't we add in a little bit of extra logging import loging and say loging basic config uh level is debug as well and we'll say uh logging do debug got weather like that and what's going to make this feel more application is um let's run this in a loop so if I say while true do that produce it and they then say logging doino produced sleeping and then time do sleep for 30 seconds let's say 10 seconds but if you're running this please be nice to open Meto and dial that down to something sensible and that should be something that continually pulls weather data and wres into CFA good let's run that and verify I didn't import time let's fix that import time seems to be doing something got weather produced sleeping okay let's go over to one more Tab and just verify this independently here's a little debugging tip for cfar applications Kat if you don't know about Kat it's really useful it's a command line app to just inspect topics and I'm going to tell it that my broker is on Local Host 1992 I can tell it to act like a consumer consuming from the topic weather data demo and start watching from the end of the topic and we'll just watch that for a couple of minutes to watch the data stream in except I have to watch it you can skip forward in time by me speeding up the video like this and we are done look there's some weather data coming through I think we can call that a proper working python cfra application you can easily change that to fetch whatever data you're interested in I suppose the next step is to read it back out using python or do some processing on it using python but we're going to save that for another video so if you've enjoyed this if it's been useful click like subscribe for those forthcoming videos and I'll see you again then