many of the ideas that you actually were in place in Spotify are also in place in that nexus framework which we're not going to talk about too much about today the real story here's the real short summary yes whatever you have heard or said in videos blog posts etc it is true and it's also not true most companies they have one operating model something that unifies everything that pretty much everybody try to apply to it's not like that 100% right what I learned in Spotify is that it's like yeah if 20% are doing the same thing that's kind of the situation they're in they're changing all the time they were experimenting with everything with their product obviously they have so many AP tests that if we pull out the phones right now we open Spotify I would guess we have at least 20 different versions in this room right now they have an insane amount of data collection from doing a/b testing comparing version a in comparing version B which one do we like the most so a what the experiments so much that means that there is no one standard there is no one way that Spotify does everything so know whatever you hear is also false it's false somewhere somewhere in perhaps in many places in Spotify they do something really great though regardless of this high speed of change and experiments and crazy stuff happening they have really satisfied entries it says 95 percent of the people here are super happy about working at Spotify would you recommend it to your friend so they do something great right so my idea was to give away the key takeaways in the McGuiness then we can talk about it so this is from as you walk him sundean joachim is one of the agile coaches has been the longest with Spotify he has quite a lot of material out there so if you want to go with one person Google Joe walk him because he knows the stuff he really knows the history he's been there he's been fighting in the pits so say so this is from one of his presentations one identify the challenges to solve it great now we can go on now seriously this is from the claws that he has a two-day claws with Jimmy John Lee and another coach on how to scale with this Potiphar in influenced by Spotify so in the two-day claws this is what they end with and start with pretty much yes there is something called the Spotify model primarily other people call it the Spotify model not Spotify themselves but there are some keys there are some principles there are some values there are some really hard truth they optimize for you can do that too if you want speed for instance because they want speed they're fighting Google Amazon Apple big players Spotify was a small company how are we gonna be able to fight globally on the music market well they say there's only one way we can win and that is by having the best speed by learning the fastest so whatever we do and everything we do it is about speed quickly learning quick feedback local decision-making speeds up and feedback so this is what I thought and this is what I prepared if we don't we might have to change it and that's fine but I want to see show of hands how many of you have read the PDF written by under SH and Henry Keaney by the Spotify model PDF okay good and how many of you have seen the to Spotify culture engineering videos about the same so I think those two videos pretty much sums it up in a good way so I actually think that we should watch those two it's not very long and afterwards I will complement it and add a bit more information to it but those kind of summarizes the gist of what they strive for so instead of me doing a crappy presentation here is a well-prepared presentation that someone's spent weeks on doing and then I'm going to show my examples and some ideas Jason ship is another coach in New York we're going to look at some of the challenges that he has also presented to do everybody we can talk about and then we'll take a break and then we'll do some Q&A one of the big success factors here at Spotify is our agile engineering culture culture tends to be invisible we don't notice it because it's there all the time kind of like the air we breathe but if everyone understands the culture we're more likely to be able to keep it and even strengthen it as we grow so that's the purpose of this video when our first music player was launched in 2008 we were pretty much a scrum company scrum is a well-established agile development approach and it gave us a nice team-based culture however a few years later we had grown into a bunch of teams and found that some of the standard scrum practices were actually getting in the way so we decided to make all this optional rules are good start but then break them when needed we decided that agile matters more than scrum and agile principles matter more than any specific practices so we renamed the scrum master role to agile coach because we wanted servant leaders more than process masters we also started using the term squad instead of scrum team and our key driving force became autonomy so what is an autonomous squad a squad is a small cross-functional self-organizing team usually less than eight people they sit together and they have end-to-end responsibility for the stuff they build design commit deploy maintenance operations the whole thing each squad has a long-term mission such as make Spotify the best place to discover music or internal stuff like infrastructure for a be testing autonomy basically means that the squad decides what to build how to build it and how to work together while doing it there are of course some boundaries to this such as the squad mission the overall product strategy for whatever area they are working on and short term goals that are renegotiated every quarter our office is optimized for collaboration here's a typical squad area the squad members work closely together here with adjustable desks and easy access to each other screens they gather over here in the lounge for things like planning sessions and retrospectives and back there is a huddle room for smaller meetings or just to get some quiet time almost all walls are whiteboards so why is autonomy so important well because it's motivating and motivated people build better stuff also autonomy makes us fast by letting decisions happen locally in the squad instead of via a bunch of man and committees and stuff it helps us minimize handoffs in waiting so we can scale without getting bogged down with dependencies and coordination although each squad has its own mission they need to be aligned with product strategy company priorities and other squads basically be a good citizen in the Spotify ecosystem Spotify is overall mission is more important than any individual squad so the key principle is really be autonomous but don't sub optimize it's kind of like a jazz band although each musician is autonomous and plays its own instrument they listen to each other and focus on the whole song together that's how great music is created so our goal is loosely coupled but tightly aligned squads we're not all there yet but we experiment a lot with different ways of getting closer in fact that applies to most things in this video this culture description is really a mix of what we are today and what we are trying to become in the future alignment and autonomy may seem like different ends of a scale as in more autonomy equals less alignment however we think of it more like two different dimensions down here is low alignment and low autonomy a micro management culture no high level purpose just shut up and follow orders up here is high alignment but still low autonomy so leaders are good at communicating what problem needs to be solved but they are also telling people how to solve it high alignment and high autonomy means leaders focus on what problem to solve but let the team's figure out how to solve it what about down here then low alignment and high autonomy means teams do whatever they want and basically all run in different directions leaders are helpless and our product becomes a Frankenstein we're trying hard to be up here aligned autonomy and we keep experimenting with different ways of doing that so alignment enables autonomy the stronger alignment we have the more autonomy we can afford to grant that means the leaders job is to communicate what problem needs to be solved and why and the squads collaborate with each other to find the best solution one consequence of autonomy is that we have very little standardization when people ask things like which code editor do you use or how do you plan the answer is mostly depends on which squad some do scrum sprints others do Kanban some estimate stories and measure velocity others don't it's really up to each squad instead of formal standards we have a strong culture of cross-pollination when enough squads use a specific practice or tool such as get that becomes the path of least resistance and other squads tend to pick the same tool squads start supporting that tool and helping each other and it becomes like a de facto standard this informal approach gives us a healthy balance between consistency and flexibility our architecture is based on over a hundred separate systems coded and deployed independently there's plenty of interaction but each system focuses on one specific need such as playlist management search or monitoring we try to keep them small and decoupled with clear interfaces and protocols technically each system is owned by one squad in fact most quasi on several but we have an internal open source model and our culture is more about sharing than owning suppose squad one here needs something done in system B and squad two knows that code best they'll typically ask squad two to do it however if squad two doesn't have time or they have other priorities then squad one doesn't necessarily need to wait we hate waiting instead they're welcome to go ahead and edit the code themselves and then ask squad two to review the changes so anyone can edit any code but we have a culture of peer code review this improves quality and more importantly spreads knowledge over time we've evolved design guidelines code standards and other things to reduce engineering friction but only when badly needed so on a scale from authoritative to liberal we're definitely more on the liberal side now none of this would work if it wasn't for the people we have a really strong culture of mutual respect I keep hearing comments like my colleagues are awesome people often give credit to each other for great work and seldom take credit for themselves considering how much talent we have here there is surprisingly little Eagle one big aha for new hires is that autonomy is kind of scary at first you and your squad mates are expected to find your own solution no one will tell you what to do but it turns out if you ask for help you get lots of it and fast there's genuine respect for the fact that we're all in this boat together and need to help each other succeed we focus a lot on motivation here's an example an actual email for the head of people operations hi everyone our employee satisfaction survey says 91% enjoy working here and 4% don't now that may seem like a pretty high satisfaction rate especially considering our growth pane from 2006 to 2013 we've doubled every year and now have over 1200 people but then he continues this is of course not satisfactory and we want to fix it if you're one of those unhappy 4% please contact us we're here for your sake and nothing else so good enough isn't good enough half a year later things had improved and satisfaction rate was up to 94 percent with this strong focus on motivation it's no coincidence that we have such an awesome reputation as workplace nevertheless we do have plenty of problems to deal with so we need to keep improving okay so we have over 50 squads spread across four cities some kind of structure is needed currently squads are grouped into tribes a tribe is a lightweight matrix each person is a member of a squad as well as a chapter the squad is the primary dimension focusing on product delivery and quality while the chapter is a competency area such as quality assistance agile coaching or web development as squad member my chapter lead is my formal line manager a servant leader focusing on coaching and mentoring me as engineer so I can switch squads without getting a new manager it's a pretty picture huh except that it's not really true in reality the lines aren't nice and straight and things keep changing here's a real-life example from one moment in time for one tribe and of course it's all different by now and that's okay the most valuable communication happens in informal and unpredictable ways to support this we also have guilds a guild is a lightweight community of interest where people across the whole company gather and share knowledge within a specific area for example leadership web development or continuous delivery anyone can join or leave a guild at any time guilds typically have a mailing list biannual on conferences and other informal communication methods most organizational charts are an illusion so our main focuses community rather than hierarchical structures we've found that a strong enough community can get away with an informal volatile structure if you always need to know exactly who is making decisions you're in the wrong place one thing that matters a lot for autonomy is how easily can we get our stuff into production if releasing is hard we'll be tempted to release seldom to avoid the pain that means each release is bigger and therefore even harder it's a vicious cycle but if releasing is easy we can release often that means each release is smaller and therefore easier to stay in this loop and avoid that one we encourage small frequent releases and invest heavily in test automation and continuous delivery infrastructure release should be routine not drama sometimes we make big investments to make releasing easier for example the original Spotify desktop client was a single monolithic application in the early days with just a handful of developers that was fine but as we grew this became a huge problem dozens of squads had to synchronize with each other for each release and it could take months to get a stable version instead of creating lots of process and rules and stuff to manage this we changed the architecture to enable decoupled releases using chromium embedded framework the client is now basically a web browser in disguise each section is like a frame on a website and squads can release their own stuff directly as part of this architectural change we started seeing each client platform as a client app and evolved three different flavors of squads client app squads feature squads and infrastructure squads a feature squad focuses on one feature area such as search this squad will bill ship and maintain search related features on all platforms a client app squad focuses on making release easy on one specific client platform such as desktop iOS or Android infrastructure squads focus on making other squads more effective they provide tools and routines for things like continuous delivery a be testing monitoring and operations regardless of the current structure we always strive for a self-service model kind of like a buffet the restaurant staff don't serve you directly they enable you to serve yourself so we avoid handoffs like the plague for example an operation squad or client app Squad does not put code into production for people instead their job is to make it easy for feature squads to put their own code into production despite the self-service model we sometimes need a bit of sync between squads were doing releases we manage this using release trains and feature toggles each client app has a release train that departs on a regular schedule typically every week or every three weeks depending on which client just like in the physical world if trains depart frequently and reliably you don't need much upfront planning just show up and take the next train suppose these three squads are building stuff and when the next release train arrives features a B and C are dumb while D is still in progress the release train will include all four features but the unfinished one is hidden using a feature toggle it may sound weird to release unfinished features and hide them but it's nice because it exposes integration problems early and minimizes the need for code branches unmerged code hides problems and is a form of technical debt feature toggles let us dynamically show and hide stuff in tests as well as production in addition to hiding unfinished work we use this to a/b tests and gradually roll out finished features all in all our release process is better than it used to be but we still see plenty of improvement areas so we'll keep experimenting this may seem like a scary model letting each squad put their own stuff into production without any form of centralized control and we do screw up sometimes but we've learned that Trust is more important than control why would we hire someone who we don't trust agile at scale requires trust at scale and that means no politics it also means no fear fear doesn't just kill trust it kills innovation because if failure gets punished people won't dare try new things so let's talk about failure actually no let's take a break get on your feet get some coffee let this stuff sink in for a bit and then come back when you're ready for part two [Music] so get on your feet talk to your neighbor for two minutes is this your culture that he's describing is something different two minutes let's look at the second part is 13 minutes and then I will add some of my own experience hey you're back great now you've probably forgotten all about part one so let's do a quick recap our culture is based on agile principles all engineering happens in squads and we try to keep them loosely coupled and tightly aligned we like cross-pollination and have an internal open source model for code squads do small and frequent releases which is enabled by decoupling our self-service model minimizes the need for handoffs and we use release trains and feature toggles to get stuff into production early and often and since culture is all about the people we focus on motivation community and trust rather than structure and control that was part one and now I'd like to talk about failure our founder Daniel put it nicely we aim to make mistakes faster than anyone else yeah I know it sounds a bit crazy but here's the idea to build something really cool we will inevitably make mistakes along the way right but each failure is also a learning so when we do fail we want it to happen fast so we can learn fast and therefore improve fast it's a strategy for long-term success it's like with kids you can keep a toddler in the crib and she'll be safe but she won't learn much and won't be very happy if you instead let her run around and explore the world she'll fail and fall sometimes but she'll be happier and develop faster and the wounds well they usually heal so Spotify is a fail friendly environment we're more interested in fast failure recovery than failure avoidance our internal blog has articles like celebrate failure and stories like how we shot ourselves in the foot some squads even have a fail wall where people show off their latest failures and learnings failing without learning is well just failing so when something goes wrong we usually follow up with a post-mortem this is never about whose fault was it it's about what happened what did we learn what will we change post-mortems are actually part of our incident management workflow so an incident ticket isn't closed when the problem is solved it's closed when we've captured the learnings to avoid the same problem in the future fix the process not just the product in addition all squads do retrospectives every few weeks to talk about what's working well and what to improve next all-in-all Spotify has a strong culture of continuous improvement driven from below and supported from above however failure must be non-lethal or we don't live to fail again so we promote the concept of limited blast radius the architecture is quite decoupled so if a squad makes a mistake it will usually only impact a small part of the system and not bring everything down and since the squad has end-to-end responsibility for their stuff without handoffs they can usually fix the problem fast also most new features are rolled out gradually starting with just a tiny percent of all users and closely monitored once the feature proves to be stable we gradually roll it out to the rest of the world so if something goes wrong it normally only effects a small part of the system for a small number of users for a short period of time this limited blast radius gives squads courage to do lots of small experiments and learn really fast instead of wasting time trying to predict and control all risk in advanced mario andretti puts it nicely if everything is under control you're going to slow alright let's talk about product development our product development approach is based on Lean Startup principles and is summarized by the mantra think it build it ship it tweak it the biggest risk is always building the wrong thing so before deciding to build a new product or major feature we try to inform ourselves with research do people actually want this does it solve a real problem for them then we define a narrative kind of like a press release or our elevator pitch showing off the benefits for example radio you can save or follow your favorite artists we also define hypotheses how will this feature impact user behavior and our core metrics will they share more music will they log in more often and we bill various prototypes and have people try them out to get a sense of what the feature might feel like and how people react once we feel confident this thing is worth building we go ahead and build an MVP Minimum Viable Product just enough to fulfill the narrative but far from feature complete you might call it the minimum lovable product the next stage of learning happens once we put something into production so we want to get there as quickly as possible we release the MVP to just a few percent of all users and use techniques like a be testing to measure the impact and test their hypotheses the squad monitors the data and continues tweaking and redeploying until they see the desired impact then they gradually roll out to the rest of the world while taking the time needed to sort out practical stuff like operational issues and scaling by the time the product or feature is fully rolled out we already know it's a success because if it isn't we don't roll it out impact is always more important than velocity so a feature isn't really considered done until it has achieved the desired impact note that like most things in this video this is how we try to work but our actual track record of course varies now with all this experimentation going on how do we actually plan how do we know what's going to be released by which date well the short answer is we mostly don't we care more about innovation than predictability and 100% predictability means 0% innovation on a scale we'd probably be somewhere around here of course sometimes we do need to make delivery commitments like for partner integrations or marketing events and that sometimes involves standard agile planning techniques like velocity and burn up charts but if we have to promise a date we generally defer that commitment until the feature is already proven in close to ready by minimizing the need for predictability squads can focus on delivering value instead of being a slave to someone's arbitrary plan one product owner said I think of my squad as a group of volunteers that are here to work on something they are super passionate about so where do ideas come from an amazing new product always starts with a person and a spark of inspiration but it will only become real if people are allowed to play around and try things out so we encourage everyone to spend about 10 percent of their time doing hack days HAC weeks that's when people get to experiment and build whatever they want like this dial a song product just pick it up and dial the number of the song you want to listen to is it useful does it matter the point is if we try enough ideas we're bound to strike gold from time to time and quite often the knowledge gained is worth more than the actual hack itself plus it's fun as part of this we do a spotify wide hack week twice per year hundreds of people hacking away for a whole week the mantra is make cool things real build whatever you want with whoever you want in whatever way you want and then we have a big demo and party on Friday we're always surprised by how much cool stuff can be built in just a week with this kind of creative freedom whether it's a helicopter made of lollipop sticks or a whole new way of discovering music turns out that innovation isn't really that hard people are natural innovators so just get out of their way and let them try things out in general our culture is very experiment friendly for example should we use tool a or tool B don't know let's try both and compare or do we really need sprint planning meetings don't know let's skip a few and see if we missed them or should we show 5 or 10 top songs on the artist page don't know let's test both and measure the impact even the Spotify wide hack week started as an experiment and now it's part of their culture so instead of arguing an issue to death we try to talk about things like what sigh pollicis and what did we learn and what will we try next this gives us more data-driven decisions and less opinion driven ego driven or authority driven decisions although we are happy to experiment and try different ways of doing things our culture is very waste repellent people will quickly stop doing anything that doesn't add value if it works keep it otherwise dump it for example some things that work for us so far are retrospectives daily stand-ups Google Docs get and guild on conferences and some things that don't work for us our time reports handoffs separate test teams or test phases and task estimates we mostly just don't do these things we're also strongly allergic to useless meetings and anything remotely near corporate bias one common source of waste is what we call big projects basically anything that requires a lot of squads to work tightly coordinated for many months big project means big risk so we are organized to minimize the need and instead try hard to break projects into a series of smaller efforts sometimes however there is a good reason to do a big project and the potential benefits outweigh the risks and in those cases we've found some practices to be essential visualize progress using various combinations of physical and electronic boards do a daily sync meeting where all squads involved meet up to resolve dependencies do a demo every week or two where all the pieces come together so we can evaluate the integrated product together with stakeholders these practices reduce risk and wastes because of the improved collaboration a short feedback loop we've also found that a project needs a small tight leadership group to keep an eye on the big picture typically we have a tech Li and a product lead and sometimes a design lead working tightly together on the whole we're still experimenting a lot with how to do big projects and we're not so good at it yet one thing we keep wrestling with is growth pane as we grow we risk falling into chaos but if we overcompensate and add too much structure and process we risk getting stuck in bureaucracy instead and that's even worse so the key question is really what is the minimum viable bureaucracy the least amount of structure and process we can get away with to avoid total chaos both sides cause waste but in different ways so the waste repellent culture and agile mindset helps us stay balanced the key thing about reducing waste is to visualize it and talk about it often so in addition to retrospectives and post-mortems many squads and tribes have big visible improvement boards that show things like what's blocking us and what are we doing about it we also like to talk about definition of awesome for example awesome for this squad means things like really finishing stuff easily ramping up new team members and no recurring tasks or bugs and our definition of awesome architecture includes I can build test and ship my feature within a week I use data to learn from it and my improved version is live in week 2 keep in mind though awesome is a direction not a place so it doesn't even have to be real stick but if we can agree on what awesome would look like it helps focus our improvement efforts and track progress here's an example of an improvement tracking board inspired by a lien technique called Toyota kata top left shows what is the current situation in this case the squad was having quality problems bottom left shows definition of Awesome in a perfect world we'd have no quality problems at all top right is a realistic target condition if we were one step closer to awesome what would that look like and finally the bottom right shows the next three concrete actions that will move us towards the target condition as these get done the squad fills it up with new actions boards like this live on the wall in the squad room and are typically followed up at the next retrospective all right I realize that maybe this video makes it seem like everything at Spotify is just great well truth is we have plenty of problems to deal with and I could give you a long list of pain points but I won't because it would go out of date quickly we grow fast and change fast and quite often a seemingly brilliant solution today will cause a nasty new problem tomorrow just because we've grown and everything is different however most problems are short-lived because people actually do something about it this company is pretty good at changing the architecture process organization or whatever is needed to solve a problem and that's really the key point healthy culture heals broken process since culture is so important we put a lot of effort into strengthening it this video is just one small example no one actually owns culture but we do have quite a lot of people focusing on it groups such as people operations and about 30 or so agile coaches spread across all squads and we do boot camps where new hires form a temporary squad they get to solve a real problem while also learning about our tech stack and processes and learning to work together as a team all in one week it's an intense but fun way to really get the culture they often manage to put code into production in that time which is impressive but again failing is perfectly okay as long as they learn mainly though culture spreads through storytelling whether it happens on the blog at a post mortem a demo or at lunch as long as we keep sharing our successes and failures and learnings we each other I think the culture will stay healthy at the end of the day culture in any organization is really just the sum of everyone's attitudes and actions you are the culture so model the behavior you want to see that's it we're done I hope you enjoyed this story thanks for listening okay so let's do the same thing to not fall asleep this late in the evening stand up talk about something that's different three minutes so and I'm gonna add some information to this and obviously everything that he talks about I have seen every piece of it in real life this actually exists everything they do everything he talks about someone is doing it somewhere inside Spotify I have never seen everything everywhere not one single squad is doing everything that he talks about so there's exceptions to exactly everything as well and perhaps the only exception which there is no exception for so the one thing that is true is this thing about experimentation everybody has their in the DNA in their culture the boot week the boot camp when you're thrown into you're gonna do something and you're gonna ship it at the end of the week everybody in the company is gonna see what you built in this boot camp because when you do stuff and when you produce work on the real product it's going live as well so start getting used to it you need to produce stuff quickly so experimentation is is really key part of the culture it happens everywhere even inside finance and those areas very interesting so quick something I'd like to focus in emphasizing is not already clear he said many times this is part of what we are today this is part of what we want to become they want to see much of this everywhere but it's not really what happens not every squad works the same ok so what is a squad did everyone get this or any question about what the squad is or how they work what it looks like in real life they work together it's a cross-functional team they do have a mission every squad does have a very clear mission it's clearly stated and typically hangs on a whiteboard or someone has made a really cool poster about it or a fun poster or a boring poster because they think is boring but important so wherever you go you can quickly say yes this is what they're working on when to be worth more time when to decide how to build a new squad who decides so I have a slight later on that says agility what what agile so agility to me it talks about three things I'm gonna draw and planets but so one thing they really so agility for me it's about technical agility being able to quickly ship with high quality or good enough quality they it doesn't break we learn quickly if we gathered metrics another phrase for this is technical excellence are we good at doing user studies are we good at programming are we good at you know all of those things that you do as a team to build a product so technical agility is one area and I will get to your answer and to your question another area is organizational agility how quickly can we align when we decide that hey let's give up this idea of building this product it doesn't work one example is running with Spotify they had this app part of the app was optimized for running for when people go out jogging so it could attack how often you put your feet down and they would find a music that will follow your pace and stuff like that it worked great product runners loved it but not enough people so decided to kill it because essentially it was just removing focus from the organization on what they wanted to accomplish so they they closed that down and they decided to go for a new bed a new initiative and I've never seen an organization so quickly reorganize to get people to understand what do we do when why we're doing it to find the new missions for every squad so whose job is this Oh someone doesn't and it's about this one here which is market or product or business agility whatever you want to call it this is top CEO pretty much Daniel with a couple of people and doing product management whatever you want to call it they have company bits there's information about this if you google as well and it's visible to everybody everybody understands what we're doing and how long if we're going to continue to invest in it for another perhaps quarter or six months or something like that so people see when actually the bets are starting to go down and they realized something new is going to happen and here are a couple of things the tribes take on these bets they break parts of it down or they take the entire bet and they make it theirs and then they break that down into what can we do with squats do we need to reorganize do we create new sports so it I would say it happened inside those tribes but again Spotify doesn't work in one way so you might discover that one one squad decides actually we don't need to exist anymore so we're going to close it off we're going to go find something else there's something here that alone an idea so there's been a try but then the tribe decides how I squats that are needed there was a long question so Lina congratulations so basically there is an idea yeah and then there's a decision made there is needed to try so the tribe decides how many squads I needed to solve to create the idea to make it live something like that I'm serious I cannot give you a perfect answer because it changes I was there for some time and during that time I see I saw different ways of these decisions being made but typically Daniel it has a strong vision for what we're going to do as a company and this trickles down into the organization in terms of goals and bets and we bring data to those bets to see if it's actually true etc but the tribe has a tribe leader that person makes sure that the environment works the tribe has chapters and in those chapters you have chapter leaders there kind of like a lion manager they help make create meaningful ways for people to work essentially find where do you want to work can I support you in moving to another squad can I help you in some way personal professional development but again the this role called chapter leader that's the one that has it's so different bit depending on who did the job some work very classical you can find the exact same line manager role in Ericsson so that I saw that some chapter leads were programmers that spent 90% of the time programming in 10% doing line management which essentially was Hey are you feeling okay good thank you well that was a very short version but perhaps they were spent like so one day per every sprint and then they had one too once with everybody in their chapter and that was what they did and they didn't have to do anything else what exactly such person I don't know more like management or is it more like I don't know yes somebody who manages squats yes no primarily no they don't define know the tribe leader doesn't define the squads doesn't define like I said what to do they do this would help they are primarily a servant leader meaning they likely have some time perhaps budget that they can invest in what other people say they should do so the tribe leaders try to not have ideas they really are the servant leader they go and say hey what do you need how can I help you yeah and that becomes their their backlog on what to do but it is a very a lot of this is self-organizing people figure this out and it's a lot about the communities and when you have so many communities that people travel to and they I mean I'm sure you heard that anyone is what if I can travel wherever they want to when they need to and when they think they need to you don't need the proof of this like yes why are you asking me if you can go you know better I had that started to change during the time I was there because they were introduced to the what you call it to the market IPO initial offering etc so by then they started to look at what are some structures we perhaps need to change to be more predictable and on economic terms so I think that started to change the culture bit and I was not with them when they did the final stuffs in final stages so but that started to change but before that for so many years if you needed to go and be somewhere you went just get the ticket here's the receipt and then you go no questions asked a lot of trust and to be able to have that sorry and a lot of spare cash yeah yes I mean spotty I'm sure you've seen that let Spotify has not made money from the first ten twelve years or something like that and that is true I mean they were negatively the red numbers but the owners were pretty much the in music industry right so the music industry they invested in Spotify and what they got in return was yeah people use their services their music instead of Napster or some other pirate and peer-to-peer solution so the music industry was yeah well yeah we're losing some money but the people use our services and they like our music and we can learn about our uses so and they thought that eventually it will start leveling out I guess why examples of tribes I haven't prepared this the one I was working closely with was the core experience if you google this you can find it from mg Joaquim Cindy and he has public information on this so one tribe I was working what was core experience which was part of an alliance which was much bigger I don't remember the exact name but like the offering to two customers so in core experience we had stuff like clay back making sure that music actually streams how can we optimize this I mean if I hit play on Spotify you will have music within a number of milliseconds ideally hundred 100 300 milliseconds is okay different for different markets etc so there was that was one tribe focusing on stuff like that another one was more on user experience so whenever you like of say this is a favorite it gets saved somewhere that information is available to other squads so they can find out some some things like your daily mix and all of those things so again how do you organize into tribes who decides Water Tribe is going to take care you start by looking at what do we need to accomplish the work that we're going to do what does that look like and this can be tuned and turned in different directions and after that you start looking at okay what about what are the teams or tribes that we need that we think is necessary for this this is a coordination of the people that are more closely to the to the problem people who decide other people from this organizational part who are they we go into the tribe or for the next six months we go into the tribe organize into the tribe for someone decides okay let's create another tribe for this you typically ask again this is very different from what most people are organized they're used to how it was so based on our experience how it works we have this picture I don't have it prepared but there's a slide on Google Docs with these are the company beds it's only ten here which we're focusing around right now here are the next time we might do and here is an unfiltered list of ideas for the future so you can see exactly what we're doing right now and what tribe or alliance is focusing on what and people who see what's coming next and so this information is publicly available and whenever you see hey this is something we want to start up and you it bet initiative we want to focus on essentially the question goes out is there anyone here that is a that thing they can start take this on because that means they wouldn't stop doing something right even though they're expanding super super super fast we still typically need to stop doing something so this is just an open question which is there a tribe here that could take on this bet is that one tribe here which is better position to take down this bet can you do something what do you need to stop to doing can you someone else take that I can't talk about all the experiments I seen but but we did close down a couple of things in order to get on a new a new initiative and you bet and after that we thought about do we have the right organization to do this do you have the right sports can we change this can we keep the team but we get the new mission so that typically happened both see our CEO or some group of people organizations that someone selected joins the conversation as I captured from this two short movies yep there's one big role mentor in Spotify organization and it was said that everybody can be a mentor to other person as I [Music] understood this metal is a key role in this age right organization am i right I imagine that to most people here this sounds like a very very strange thing like a fluffy pink this thing in the air which doesn't exist but it does and yes it is very different I have a quote here from the HR manager Catalina body she says this let's be clear our way of working is unique but it is definitely not perfect we make mistakes and we realize that we don't have the answers to all the questions as we're growing so quickly new challenges arise every day and the biggest challenge are to attract the right people and to harness our innovation agility and unique culture while welcoming 100 new employees to the team every month so whatever I observe during my time by at that place it is different now not everything but a lot of things are different because they move so quickly and they need to move so quickly in order to ramp up to fight Google etc so - I guess the trend when I was there was that they were moving towards more classical decision-making and structures but I don't think they moved all the way I hope they did not and yes a lot of decision making happening and on the floor I'd like to go through some of the things I have here and then I'll take more questions there okay yeah I think it makes more sense so chapter strides against we can talk about that later if you want to but it's just go out and see this is pretty looks almost like traditional matrix right with line management etc and it's for many parts it's not that different this idea though I wanted to focus on this at some point in time this was true is what it says in the PDF the thing that documents how they work it's not exactly so back in 2012 that's seven years ago that's what it might have looked like back then but then they've had seven years to try new things out so whatever we are reading and we're doing it's likely or watching it slightly different now how fast are they growing we talked about this just previously this is information back in 2012 this is when this paper was published 24 million users six million users about 300 developers this is the most recent information I've seen that is public two years old 17 million subscribers so from 2012 you see that's a huge growth it's like exponential not real exponential but it's really fast and the same thing goes for employees and this was up to 2012 and I can assure you they continued when I was there son yeah so how can you make how can you make this work if you have a classical structure with well it you don't you need to flip the pyramid and really have servant leadership and make as many decisions as possible locally otherwise you cannot scale doo doo doo doo clues so how do you get people to align if everyone is making their own decisions how do we know we're not breaking it forever anyone else so I'd like to do a quick check here and you need to help me out with this one - two - two - Chris Anya and I'm going to need your help so if you all think about a really great project that you have seen been part of perhaps hopefully a really great project what do you think of just shout things out now a good problem to be solved great people sorry what about budget unlimited budget okay great meaningful goals excellent by the validated learning happy customers happy customers alright new tech whose satisfaction our and everyone's satisfaction including have employees right and did you do two to everyone's satisfaction all right perhaps room for two more heaven points what is heaven points okay so this so this actually and when I go when I come to heaven I get some good score okay yeah oh my god yeah okay this is good enough can I ask you to cut so thank you all for contributing to this list can you hang it on this one behind you now forget everything about what you just thought of and think about something perhaps more personal think of a product that you like a product that you use perhaps every day something you would really be sad if you lost you couldn't use it anymore it could be a product or a service so what do you think about now what comes to mind when you think about this great product design is one thing I heard what about the design simple intuitive someone said lovable great user experience some content always available fast unbreakable sounds like a movie furious awesome useful some what's that right which is something we are a touch attached to I'm gonna summarize it with that yeah it improves if it doesn't improve you or does it the product improve improve living it makes my life better in some way yeah improve or easier etcetera yeah and someone said something about free I'm gonna say yeah worth it right it's worth it product understands me thank you I'm gonna say that's enough understands me so we have two lists now and this is related to Spotify someone tell me what the difference is a way a goal okay nothing else you say in great projects deliver great products mm-hmm are you sure it might help right if people are engaged happy have fun times and good goals we could get a great product yeah now we're getting there hey just said this is what we do this is what we want to get and I would say if you get this this will work out how do you align 2,000 people making daily decisions well one thing I learned is that whenever you make a decision in your squad you send an email you send an email to decisions at Spotify dot-com or something like that everybody shares their decisions because there are so many decisions made and you're not gonna you're not supposed to wait for a decision to be made you don't wait to synchronize with people unless you absolutely have to make the decision and send it out to everybody and then everybody can see what decisions haven't been made and how to adjust if needed or get some feedback I'm simplifying a bit but be honest this email list was super interesting to watch and it goes out to everybody who reads those who reach those if there's so many decision everybody a day I say somewhere between 5 and 50 I'm just guessing now but yes lots of decisions were made and yes this was one way to start sharing those decisions what did we learn here's decision we made if you really want to read more talk to this person or watch this slide or this video or whatever how many years ago about two years ago so what is different here I'm just gonna emphasize again this is about internal stuff right this is how we think about projects we focus on validated learning having meaningful gold budget is our concern great people if you think about technology and happy customers that's in there as well is this what people buy is this what people buy when they buy it Spotify no this is what they buy right so this is the thing we need to align on so Spotify made this very clear these are two of the most important metrics inside Spotify how many people are actively using our product on a daily basis on a monthly basis whatever change you do to your product to our product these two are not supposed to go down so whenever they do changes and do a/b testing etc you will see something that looks like this this is for the a group we can see that when we introduce functionality to this was used much more and this was reduced this came up this went down and the other ones here are not statistically significant so they are not data driven but they are data informed heavily data informed this might be an okay thing if this is a very big new thing that we believe in right but at least we get this data and we want to be statistically verifiable so to speak so here's another thing that they use quite heavily they use the dibs we want to have some data insights beliefs and bets and this is a very old example they saw that actually people looking listening to music not looking at music but listening to music on the desktop it's decreasing and mobile usage is increasing but our staff looks like this 95 percent work on desktop stuff and they have desktop skills perhaps that's a bad match and I think this was at least 10 years ago so the insight is as mobile and mobile is overtaking desktop perhaps we should change we're optimizing for the wrong thing so let's start to create a better around this higher mobile devs retrain people build infrastructure to iterate fast on mobile tu-tu-tu-tu-tu-tu so that's a couple of things that I really learned here invest super heavily in architecture that supports teams to do the valuable work so that was one example in the previous slide how can we iterate quickly on the mobile phone when we have to go through Google and Apple every time we make a release Google and Apple kind of controls our release pace and how often we can deliver and we can't have every Squad make a new release of the app because then we'll have new release like ten releases every day no know who is saying the Spotify user is going to be happy about having updates every minute every hour right so how can we iterate faster without having people say hey you have a new version you have a new version you have a new version you build architecture and platforms and tools that allows you to deploy in the server side and the client just picks instructions and renders it kind of like all the web technology but it really works and this allows them to iterate much much much much faster they don't have to wait for a synchronized slow update and this was not for sheep not for free but it was totally worth it it gave them a lot of speed so investing in this architecture to allow you to your teams to really operate and do great work that is something I've never seen another organization do this as heavily I talked about decisions being shared coaching is natural everywhere you go is you have coaching some of you talked about mentor over here before so inside Spotify this is people learn this I don't know if they learned it through boot camp etc but of course we need coaches of course we have mentors and these are the classical four stances of a scrum master for instance coach that's me up here I'm a coach I have no idea what is important to the person I'm coaching or to the tribe lead that I'm coaching I don't know better than the tribe leader or the chapter lead or this QA person but they have some ideas so we look at their agenda and their questions and they have the answers I as a coach I just help them get the answers out of them make it visible transparent to them so this happen everywhere as I Spotify you have you one-to-one sessions every week with your chapter leader or a coach they help you figure out how you are going to take your next steps mentor yeah there was absolutely mentorships things happening as well I would say it was more informal in terms of you show up in this guild because you're interested and you learn and people are happy to share people are super happy to share it's just part of their culture we're very open with what they do their blog is vibrant it actually has useful meaningful stuff happen being shared all the time mmm-hmm okay so what are some other things I learned I'll talk about the top things Google tools that it works I've never seen a company where you can go in and you say I'm gonna have a meeting in ten minutes and everybody the technology works everybody's on the same page everybody looks for a great video it just works never at technology issues just a side note but it really works and amazingly it will post mortems hand up if you have been in a post-mortem so this is really part of their culture as well he talked about it in the video when something goes wrong do a post-mortem if you don't know what that is google it and there's going to be examples from Google engineering etcetera at the Google engineering book so they do it really well action points gets implemented things happen it's not just a place to go to wine and then nothing happens people do this they change some of the things I was not so happy about which is also known now they started introduced a technical owner next to a product owner very interesting concept so instead of the product owner having to optimize you know maximize value and optimize total cost of ownership that full life cycle perspective they kind of divided it now so there's a technical owner thinking about how to do things technically well and the product owner thinking about market business and user acceptance etc they are they work in Paris yes and no and this was happening about when I stuff stops yes I'm sorry to give you the sno answer but that really what it is there was a lot of people that said yes I take the technical owner role and I'm not gonna do anything because I think it's stupid that happened right so that person didn't do a lot and not every team is the same what I can say is that when you're operating on global scale which they certainly are with over a hundred million active users technologies matters and to expect a product owner with no technical background to be able to make sound technical decisions you likely need a partner you know another thing I learned it when you do a be testing properly and you don't want your monthly active users to go down that means at least from one month you need to collect data right to see to see a change or not that meant whatever you did is gonna last for at least four weeks before you build something one day and then wait four weeks to see the change they don't do this for everything of course that will slow everything down immensely but they do it quite often so they had something called a road road manager is not road map sorry a road manager or something like that a person that stays with this thing make sure that we collect the data and checks in every week to make sure that there is nothing strange with the data coming in etc and then at the end of it presents it back to anyone who cares about it I do a presentation make sure it's valid and if they think it's yes we had good results then we decide are we gonna roll this out or not and then they start rolling out gradually beyond the a/b testing and that often meant rewriting parts to allow global scale because you don't need to write for global scale right away before you know it's a good idea right so I learned that this means a team will revisit an item for a very long time and sometimes that can be demoralizing everybody understood the value of it ozone they accepted it hack week Wow I've heard about it and I read about it I didn't see the results but when I saw this energy of two hundred I'm I don't know how many thousand people that actually got to spend the week doing exactly what they wanted to drive their own ideas as fast as far as possible I was overwhelmed just trying to go through all of those experiments at the that Friday the party Friday Wow some really crazy and super doable stuff so I say here at least 50 of those that I saw were ready to go to market nothing holding them back they had this architecture set up to very quickly go out with new ideas so many people they had already done it started collecting a/b tests it's a time you're in just one week the real pain here was to figure out which one of these are we gonna support there's so much creativity how do we make sure that well we can sustain it in the videos has no external testing that started to happen while I was there know what does a tech lead do etc they're always changing sorry yeah can you elaborate a bit more on the technical owner issue what's your take on the on the on the situation we hear when we have two owners and each of the each of the owners can have their own agenda yeah or I'm here control so the problem is this I have a technical product technical owner that's you and I have a product owner that's you so obviously you like black shirts right and you don't like black shirt you like these striped shirts so I am a developer and I need to know what I'm gonna focus my two days left in this printim am I gonna do technical stuff removes technical depth and improve quality in scaling etc or am I going to do a new feature development tell me so sorry so which one do I listen to it depends if one of the ominous is your boss so they're in the same team the same squad right who is the boss the technical owner or the product owner I don't know to be honest I mean that'sthat's conflict that arises when you have two people that are supposed to own something so that was the beef and the issue I saw I hope they sold it and someone could play and I don't think that had one solution again for the playback squad one of them based in Gothenburg I'm sure that since it's so technical how to deliver music really fast that person already pretty much was the same technical owner so I know that he assumed both the product on a roll and the technical on the wrong problem solved for them right one more thing one example how things don't quite work at Spotify so this is Jason ship he presented this back in 2016 yeah so Jason ship is a consult he used to be a consultant with thought works a really great guy bright and here are some of the things that he saw I'm gonna go to slide 36 so pretty much everything of what I've been telling you is already out there so this is quite a lot of text to read but essentially it scans back saying if I walk into a room within fifteen seconds I want to know what they're doing and how it's going most squads don't have that that's what Jason saw some squads especially new ones they're not even familiar with agile basics to do and not everybody understands autonomy versus you can do whatever you feel like so here's another issue that he saw in the New York office I did not see that in the Gothenburg office I did not see it much of this in the Stockholm office so but some of this is absolutely I saw on the pains as well so that was actually all I wanted to show that they have growth pains they have paints like everybody else the grass is it greener on Spotify of course it is is it greener somewhere else yes of course it is I think the grass is always green it's just different green and different stickiness and different long and here is more Brown and you know we have different brown spots of non green grass so eat but they do have some magic juice because they have a lot of people being really happy about what they do so suggesting that we take break because you've been listening for a long time