you might think big companies would use languages like rust and go to scale and sure plenty do but there's a secret weapon that a lot of these companies are using to let them scale much bigger and much faster with teams that are much smaller and more focused that language has been around for a lot longer than you might think too it's a language I love a lot well not it but the things that you can build on top of it that language is earling from the 80s it was built for phone companies but since then it's been used for a hell of a lot more including my favorite language Elixir yes not typescript El I love it so much the things you can do with that are incredible and we're going to read an awesome article all about how these companies are using this stuff so let's get to it how WhatsApp Discord and other companies all have managed to scale to hundreds of millions of concurrent users taking advantage of earling in the technologies that it is enabling huge shout out to H we for writing this I'm very excited for this article as a solo full stack Dev one of the most popular projects our clients ask us to build is a real-time app oh you know the ones that use every day things like WhatsApp Discord slack twitch chat Etc when building real-time apps you need to consider for various factors like scalability fault tolerance responsiveness and distribution and that's a tough job especially if you're a small team or even a solo team one of my favorite fun facts is that when WhatsApp was acquired they had I think six Engineers total maybe a bit more but they had less than six working on the infrastructure and man the WhatsApp team killed it and they were able to scale to unbelievable numbers of users with such a small team because earling made it easy what if I told you that you could build a real-time app that can easily scale to more than a million users with just a few deaths and not only that but you could also deploy them with almost zero downtime at extremely low costs the secret weapon I'm talking about here is the erling virtual machine or beam the Bogden or Bjorn earling abstract machine remember earling is ancient earling comes from like the ogc days so they didn't have a real term of virtual machine yet like that concept was still new largely being invented by erlang so the term was abstract machine because their goal was to make a that they could write code for and then they could just Port that layer over kind of similar to things like jvm from java This was meant to be an alternative to how C worked where the C language had to be ported to every different architecture so that you would write code and it would compile to the correct assembly beam was what were focused on being a slightly higher up level so you don't have to compile different code or compile your code for different platforms you compile the runtime and you can stuff the code in again similar to Java but at the time that was revolutionary and also very slow cuz computers sucked which meant that earling did not get adopted that much but since then the power of having that abstraction in the networking benefits and all the crazy stuff that got built into erlang have proven to be very very useful at the time more for the telephone companies but since for the internet we now know and love because believe it or not erling was invented before the internet so yeah this will be a fun one in this article I'll explain how erlang in its modern counterpart Elixir have become the secret weapons behind real-time apps like WhatsApp and Discord I'm especially excited to share this because I've been using Elixir for more than 3 years now and I can say it's one of the best decisions I've ever made in my career yep there are very very few people who can say they regret learning Elixir I'm not going to promise you a bunch of jobs and opportunities as a result but it will make you it will make you appreciate functional programming and scaling and networking and all these things in a different light and it was certainly a very powerful thing for me to learn when I learned it so let's dive in the origins of earling and elixir that's such a good meme that's so true I would I was about to give the history but I'm going to let the author do it and just add my details before we go into the history of Elixir cuz we're going to have a lot of fun there I have to plug Elixir conf it's happening August 27th the 30th in Orlando so if that time's already passed I'm sorry but if you want to come to a cool place to talk about Elixir and hear Keynotes from Jose valim Chris McCord the guys who made Elixir and Phoenix respectively as well as this guy who I don't know how he snuck in but he did come check it out I'll even leave a discount code in the description so use that if you want to get some money off anyways earling was developed in the late 1980s by Ericson a Swedish telecommunications company the goal was to create a language that could handle massive concurrency and high availability requirements of telecommunication systems again they wanted to make this work for phone companies the internet was not even a thing they just wanted phone companies and their stuff to work properly earling design focuses on lightweight processes message passing and fault tolerance making it ideal for Building Systems that require continuous operation as well as high reliability Earl's lightweight process allows for thousands or even millions of concurrent activities within the same system without significant performance segregation this is kind of the magic of llang the idea of like what we now know in JavaScript with the event Loop where you can have something going on and process other stuff at the same time you can have a a function that is running that isn't reserving a meaningful amount of CPU or Ram at any given time it's just IO that's being waited on the idea of having more processes running than you have processors and threads in your processor that concept was unbelievably powerful and still is to this day especially when you consider out the time there was no such thing as multi-threaded processors and multiple cores like your distribution method was multiple computers and the only thing that knew how to even start doing that was earling but it could also spin up multiple processes in quotes on the same machine and just wait for the first one to finish before doing the next one all of these Concepts were largely invented by earling because this idea of a process not being the thing directly tied to the machine itself but being an abstract thing that you could run pause resume cue deal with later deal with sooner all of these ideas came from earling which is so crazy it's a great quote here we'll get to in a second it message passing model enables these processes to communicate efficiently without sharing memory which reduces complexity and increases fault tolerance the let it crash philosophy which is this is like core to earling let it crash it's fine because you can recover from the crash don't go out of your way to make sure edge cases are handled in every single detail because you can just catch at higher level and try again the let it crash philosophy encourages devs to write systems that can recover gracefully from failures contributing to the robustness of the earling based applications Elixir created by Jose valim in 2011 Builds on the solid foundation of the earling VM but it introduced some more modern and developer friendly syntax so I'll dive in a little more on here check out my interview with Jose if you want to see a bunch more cuz it was awesome I love him so much he was one of the most relevant devs in the Ruby world he built the ooth implementation for Ruby which was a huge deal he realized that Ruby's scaleability was like absolute garbage it's as single threed as a thing can be so Ruby on Rails sucked to scale we invented all sorts of new technologies to abstract on top of rails like outside of the server to tell you which server to go to because it was the only way to load balance hell the concept of load balancing as we know it today was largely invented to deal with Ruby's as such Jose was annoyed and wanted to find something better he Soul searched he looked at every language and he fell in love with what erlang could do but then he had to read erlang code and I should probably have started with this but it's important to understand how cursed earling syntax is this doesn't look too too bad but like accessing sub values is largely done with like the number sign there's a lot of code that has a semicolon at the start instead of the end this is very modern earling okay a good bit of this is relatively modern but the dash for modules also the concept of modules being something they had was insane this looks not as bad as it usually does when I look at old orline code maybe my brain's being rotted by FP it used to be weird okay now we're talking with the brackets the underscore ATS like what the is this now we're talking yeah it's one of the earliest languages with pattern matching I it might have invented pattern matching what language was first to have pattern matching ml in the 70s had it damn did not know that good to know yeah llang was one of the first languages with it not the first but one of the first anyways the the point I was trying to make is that Jose saw erlang realized it was miserable to write and read and wanted syntax didn't suck and let's be real Ruby syntax is beautiful it's like if python didn't hate you the promise that python made of readable code like English was actually delivered on by Ruby which is funny because the guy who wrote Ruby is not a native English speaker the point being Ruby had great syntax Jose missed it when he started playing with and decided he would build a language on top of llang so he could add the beautiful syntax of Ruby while also having the performance benefits in ecosystem of llang and he pulled it off with Elixir for sure especially once he started adding all the fun functional programming things on top it was great together earling and elixir provide a powerful stack for building scalable reliable and maintainable apps their features make them a perfect choice for companies like WhatsApp and Discord which require systems capable of handling a large number of concurrent users with minimal downtime we need to actually talk about those features though cuz right now I'm just glazing the language so let's dive in a bit the five core features that make beam a secret weapon for real time apps there's a reason these companies are using it excited to dive into it first big feature is Fault tolerance see this we got a supervisor keeping track of the workers if a worker dies supervisor sees it and they spin up a new one love it also love the escal draw reliability is crucial for applications like WhatsApp and Discord which it must operate continuously without significant downtime Earl Lang's let it crash philosophy ensures that individual process failures do not bring down the entire system instead processes are isolated and supervisors can restart a failed process automatically this design minimizes the impact of crashes and contributes to the overall stability of the application the the fact that everything in llang is its own like isolated worker always is magic it's so powerful you can do crazy like hot swap the code out live like you can just change what code is running because again Earl's like a virtual machine in layer so you can just throw different L code in and it will interpolate it and use it for whatever you want to that's nuts it's so cool it's like one of those things that you don't realize cuz it it could work in Java but nobody uses Java that way I don't even know if it can actually do that the thought of things like hot swapping code restarting individual processes in workers without hurting the system crashing without the system going down oh all magic and hugely useful at scale especially if you're moving fast scalability scalability not only needs the ability to expand physical servers horizontally but also methods to communicate between those servers this is one of those things that llang just does and once you see how llang does it you get mad that everything else sucks so hard we'll see what the article has to say first but Communication in distributed systems always brings several issues from race conditions to node discoveries orchestration failures Etc there are lots of reasons why these issues are brought up but most of them come from the ill-managed complexity Beam on the other hand achieves scalability without any fancy yet dangerous tools out there it handles internode communication like a charm let's see if simple example of how to communicate between two nodes in Elixir so you have to spin up two nodes so here's two different nodes spinning up here this one is named node one at here is just where it's located has a cookie and we spit up another one here named node two so all I have to do to connect them it's inside one of them say connect to the other one node. connect node two with the IP and here is where gets actually magical do you know how you run different code on that node from this node call node. spawn and just pass it a function it will just run over there what the why does no other language have this work this easily these aren't necessarily different processes on the same machine these could be different machines entirely and you can just spawn a process on a different machine from the one you're currently on it's insane it's so powerful they only you get into like gen server show me an Elixir gen server example it distributes the compute to process an array across multip multiple nodes gen server is nuts it's a generic server that makes it easy to distribute cool showing all that cool as you see here looks a lot like Ruby sad that there's no Elixir syntax highlighting on chat GPT but I get it and all you have to do to distribute these we have our nodes which we just get from calling node. list we now know how much how many nodes there are by calling length of nodes we break up the array we then chunk every for this amount so there's now chunks of however many like like let's say there's five nodes we now have chunks of five so for each Chunk we then spin or spin off an async process for each of those it's that simple it's like real easy like hilariously so and we just have the Gen server that creates the link and actually spins up all of these processes and we can just Chunk on an array which by the way the ability to Chunk on something like this is magical to just split up your data in groups of five with a helper like that oh oh so good for those wondering how does this work it assumes the same source code is available and tagged properly on all nodes which it pretty much always is and it's updating itself as far as I understand to make sure all the nodes are on the most upto-date Source it's not actually sending the source code across the wire it's sending a tag to what source code to run across the wire from my understanding I could be wrong but it's it's these are the things you don't think about when you use Elixir because they're all just kind of solved for you you have the node one node two here example node. connect to node two we can distribute a compute. start link now we have the two nodes result distributed compute. process array the array can you give an example without a module where you do it all as one big function with pipes the pipe operator is one of the coolest things in Elixir by the way Yep this is how I would have written this now we're talking so we have the node list and we can cat self to it so now we have all the nodes including this one chunk size again we're dividing the length of the array by length of nodes create the chunks .zip nodes map so the zip is attaching the node in between each element so now we know if we grab two it's the chunk then the node task. async we pass this thing through the RPC call the distribute example process chunk chunk end and that's it and if you want to see how the chunk processed that's this code here the module def process chunk chunk does whatever you do here and all you have to do to call that somewhere else because again we're calling this function on a node we don't know the node is this node so we just call that by passing this node and then calling colon process chunk and also distributed pipe we need to tell it which module CU we're in this distributed pipe example module and we're telling it in this node call this module specifically this function and pass it this data it's actually really nice like magically so it's oh you know I I will just sit here glaze Elixir all day if I keep looking at code Snippets so back here as I said this is really powerful and scalable and you can avoid using complex distributed systems like Pik Kafka rabbit mq and things like that the amount of time I spent avoiding these things because Elixir and earling just did them for me probably hurt my career as much as it helped getting some good questions from chat already like does this relate to the wild fly does yes this is how fly Works fly functions entirely because they're building everything on top of Elixir people saying that my hair is convincing to check out Elixir oh boy I'll take it the hair seems less sincere the hair's convinced you I'll take it anyways distribution this is where it gets even more fun cuz again you can run this on any node and it's very trival to spin up realtime Global applications like what's in Discord require distributed architecture to hindle users from different regions this needs a system that can distribute workloads across multiple nodes and ensure seamless communication between them beam Schuler can not only distribute processes across multiple cores on a single machine but also across multiple machines in a network so cool and also it treats every process the same whether it's on the same machine or a different one this is kind of the magic you can wait for a result of a process and call it as though it ran on the same machine even if it ran on the other side of the world it doesn't care and obviously this makes it very easy to build distributed systems without worrying about the underlying infro computer one computer 2 computer 3 all getting things passed to them from theer it doesn't matter this ease of distribution also connects to the scalability feature when you need to scale your app you can easily add more nose to the cluster and distribute the workload across them no need to worry about complex load balancers or sharding strats beam takes care of it all for you yep you could hire a team of devops to do all these things for you or you could just use a good language it's also worth noting I'm going to drop a spicy take when we're here performance means a lot of different things I find a lot of companies getting excited about languages like rust because they want to squeeze out all the performance they can but man performance is not that simple performance means that your app runs well or that the thing you're building runs as expected it doesn't mean that the amount of time it takes to do a specific math function is as short as possible it means the whole system runs well so as much as rust might make a given function compute faster than other languages certainly faster than Elixir cuz Elixir is not a compiled language it won't benefit once you want to go past what your current processor does one of the most important things to know about performance is that performance is often a function of scale how many processors and threads do you have how well are they being utilized I've seen a lot of code in go and in Rust where the go code is faster not because it can compute a function or get data faster because it's easier to parallelize the things you're doing in go then it is in a language like rust I know I've gotten flamed for the rust suck at acing takes but I don't care I'm right rust suck async it doesn't do concurrency well for it certainly doesn't do parallelism well for you can brute force it in but good luck have fun earling just does it it's just part of the language and that's the magic is while yes if we're comparing how much compute can get done on One Core Elixir loses it solves a much more complex performance problem that matters a lot more to businesses that are actually making money which is when we hit the cap of how much performance we can get out of one box how easily can we have have a second box and sure if you can if you could theoretically only ever have one server rust is a great language for that but as soon as you might need a second server at some point Elixir is the best thing you can be using and if you want to be able to spin up multiple nodes and distribute your processes so that you can get responses to users faster without micro optimizing every single line of code in your app that's the performance win that you get from elixir in earling they make it easier to scale up to address Performance needs rather than rust which makes it easier to write individual lines of code and functions that resolve way faster as great as that is and it is awesome it is not going to solve the problem as indefinitely is something like elixir and lling does because these mean whenever you need more performance just spin up another server no other language has this this good period unless it's also built on top of beam So speaking of performance and how quick things handle stuff let's talk about responsiveness real-time communication platforms must deliver messages and updates to users quickly and reliably I mean don't tell that to slack or GitHub anyways Earline takes the execution of multiple processes into its own hands by employing dedicated schedulers that interchangeably execute many earling processes the Schuler is preemptive it gives a small execution window to each process and then pauses it and runs another process this is so magical you have no idea so here's an example theer allocates an execution window to process one which allows Max 500 function calls per process so if this needs a th000 function calls but it can only take up to to 500 at once then after it's hit that 500 it pauses and goes to the next process since process one needed a th000 it's paused it then tries process two which only needs 100 so it finishes that and then it can go back and finish that this sound familiar if you're deep enough in JS you're familiar with the event Loop this is like a Brute Force scalable multi- Noe version of the same thing it's so cool just got the question how does gleam compare it's the same because both llang or both Elixir and gleam are built on top of earling and beam the same way I'll say gleam whole like compiles to JS thing won't get any of these benefits but if you're using it on the server side with earling it's the same thing it all compiles the same thing the same way that like cotlin isn't faster than Java all the languages built on top of java are the same because the execution window is small a single long running process can't block the rest of the system furthermore iio operations are internally delegated to separate threads or a kernel pole service on the underlying OS is used if available that means that any process that waits for an IO operation to finish won't block the ex ution of other processes again it's doing a lot of the same things that we love JavaScript for and the things that actually make JavaScript surprisingly fast but it's doing it across machines that's nuts that's so cool speaking of cool Live code updates nothing else has this maintaining real-time applications requires the ability to deploy updates without disrupting the user experience usually in a containerized cloud environment you would use techniques like blue green deployments Canary deployments Etc to achieve this but with beam you can update code on a running system without restarting the application do you understand how magical this is Imagine something like a websocket server that you're connected to to get new chat messages now imagine you're the dev of that server you need to push an update to change how the processes are being shared or what database they're connecting to or something like that normally you'd have to take down the server and put up a new server and then hope the user automatically reconnects and write a ton of client side code to make sure they can reconnect to the right thing now you can just put up the new code you're done it's nuts this feature is known as hot code swapping and it's crucial for real-time applications where continuous operation is essential it allows for devs to fix bugs add features and optimize performance without interrupting the service oh boy we're getting into case studies now this will be very fun to see how these things are actually being used I love the the ABC here of context on what this is examples of why it's powerful now examples of how it's actually been used how WhatsApp leveraged earling to scale real-time messaging with more than a billion daily active users that's so hard to comprehend in 2009 WhatsApp was founded by Brian Acton and John com who wanted to create a messaging app that was crossplatform fast reliable and secure they chose earling as the foundation for their real-time messaging platform because of its fault tolerance scalability and distribution capabilities the basic architecture and codebase for whatsapps messaging systems were inspired by an open source project named ejab it was an xmpp server written using earling Whatsapp growth was exponential and by 2018 it had over a billion daily active users thanks to the efficiency of earling and their careful refinement of both ejab they managed to handle millions of messages per day with minimal downtime while keeping the team lean and the hardware and Engineering costs low yeah I I want to emphasize like how many Engineers worked at whatsapp when acquired they had 32 Engineers if I recall six of them or so were on infra and that was for a$1 199 billion acquisition and that was sixish people covering 450 million active users that's so hard to fathom that's so crazy that ratio is unbelievable it's possibly the craziest of like number of devs to number of active users connected unbelievable according to Anton lavick a WhatsApp server engineer one of Ling's best attributes is the concurrency he says that other languages or platforms can try to achieve the same level of concurrency but they often require a lot of effort and resources since earling was built to solve the problem of concurrency from the beginning it's much easier to achieve the same level of concurrency with Earline yep in case two is Discord which uh is a relatively well- covered story even on the Elixir blog back from 2020 Jose wrote an article about why Discord was on Elixir so that's a really cool thing if you want to check that out great reference let's see what the author of this has to say Discord founded in 2015 by Jason and Stan is a realtime chat app designed originally for gamers also devs by the way t3g Discord if you want to hang out with us back then they chose two main languages to build their infra Python and elixir this is particularly funny cuz since then they've done go and rust and a bunch of other the python codebase in Elixir Still Remains for powering API Rel Rel ated stuff most of the core features including realtime chat are still powered by Elixir so the python I don't think the python codebase is being used for anything but the realtime chat absolutely is elixir powered discord's popular servers such as those dedicated to popular games like fortnite and Minecraft can handle hundreds of thousands of users chatting simultaneously hence it should be very good at broadcasting messages and user status updates in real time without any delay alixir concurrency model in Fault tolerance features made it an ideal choice for building discord's realtime chat system the ability to handle massive conc and distribute workloads across multiple nodes allowed Discord to scale its chat platform while maintaining high performance as well as reliability and then in 2019 Discord reached a new Milestone they had to handle way more concurrent users than ever before this time they had to use not only the powers of beam but also the powers from other more so-called performant languages they use rust to build their new member list that can rapidly update the status of 11 million plus concurrent users and the cool thing is that they adopted pre-compiled Russ modules into the Elixir code base using a feature called NF which is native implemented functions really cool that for the things that again Elixir doesn't do fast because it's not a fast language it's a scalable language you can still dive into something like rust to do certain things fast and then send out the response across the crazy node management stuff that you can do in earling this way they could leverage the performance of rust while also using the productivity and scalability concurrency and all of that of alexir super cool case three this is the case that the author worked on how I buil built a realtime networking app with elixir in Phoenix last year I had a chance to build a fun project called food Focus which broadcasts a live video of a user cooking a dish and other users can join the live video and chat with the host not only that they can also connect their smartwatch to the application the application will show the synchronized recipe instructions as a timer on their watch by the time I started building the project I had to choose the most cost effective Tech stack to build it most important traits of the tech were the following need to be easy to build real-time features it should provide ready to use tools so they don't have to bring in a lot of third part stuff should be fun to work with nice DX oh we were doing so good so good threw it all away at the end this hurts me after some research I decided to use elixir in Phoenix for the back end and make a mistake on the front end I I'm sorry I I Love You author you've done a great job with this the fact that you were willing to try out new things and find elixir in Phoenix is also why you were willing to try flutter but I hope you take the time to reflect on why flutter sucks especially if you need a mobile like a watch app flutter is going to only get in your way there I I don't want to ran to about that cuz I love this article so though it was my first time building a production ready app with Elixir I was amazed by how easy it was to build real-time features with Phoenix these are some of my favorite features and tools that I found very useful while developing the project with Elixir Phoenix channels which are a great way to build real-time features they're built on top of websockets and they provide a simple and easy to use API for building real-time apps ETS which is the earling term storage it's a built-in key value store in earling that's very fast and efficient no need to bring something like grus for caching or storing temporary data also post back tools like Ecto for the database operations Oban for background jobs electric SQL for running local first queries with better ux good stuff thanks to these features I was able to deliver the project on time and within budget if I had chosen a different stack I might have spent more time configuring and integrating thirdparty libraries and have less time to focus on building the actual features yeah conclusion WhatsApp and Discord are exemplary cases for how the earling VM in Elixir can be leveraged to build scalable real-time apps however they are not the only companies harnessing the power of these Technologies other prominent companies like Spotify Pinterest PepsiCo Financial Times Heroku they miss fly iio but they're a great example too these companies have all adopted earling based languages to achieve High concurrency fault tolerance and scalability in their respective domains also twitch twitch isn't using it anymore but we use it a bunch for the marathons and it was so good for that it was so good in conclusion whether you are building a new messaging platform a real-time gaming service or any other hyperconnected app I strongly encourage you to consider the earling VM and Elixir is your secret weapons for succcess their unique features and capabilities can help you establish a strong foundation for your app and scale it to new heights without compromising on performance or reliability thank you Hoon for writing awesome article I loved every word of this other than the word flutter I am so thankful you wrote this and that you gave me the opportunity to once again rant all about how powerful earling and this whole ecosystem is let me know what you guys think in the comments I know erlang and elixir aren't for everybody but I think it might be for so let me know until next time peace NS