Security and Compliance at Datadog

[Music] welcome everybody again thank you so much for joining us today uh we're going to be speaking a bit about security and compliance at datadog i think one of the really interesting things i've seen spangler and his team do is use an engineering first approach to security and compliance in a way that i really appreciate having been on the other side um so with that welcome to another episode of datadog on uh this is a series about engineering at datadog uh we go through real world examples of the challenges faced by engineers at datadog and again we we have a url there where you can go and sign up to be notified when more of these are coming up so introductions first of course i'm kirk kaiser i'm a technical evangelist at datadog and i lead the evangelism team at daylor andrew you want to introduce yourself yeah sure my name is andrew spengler i support the governance risk and compliance team at datadog and we sit within the information security and compliance organization cool so backing up a bit a little bit of context about datadog and what we are where we're coming from so datadog is a monitoring and analytics platform that helps companies improve the observability of the infrastructure and applications what does that mean we help people build software more effectively and we have a bit of scale to contend with we have over 12 000 customers millions of hosts and trillions of data points per day and internally we run mostly on kubernetes um the the data point about kubernetes is significant in that it gives you a bit of the operational overhead that we're dealing with and a bit of the complexity and in addition to kubernetes and uh kind of the scale of numbers we're also multi-cloud and we have over 400 different integrations that are used to ship data to datadog um so you kind of get an idea for the space of datadog in general having to deal with complexity and then especially for andrew's case trying to reason about risk in a place where there's so many dimensions for variables and to have to consider and to throw another wrench in uh andrew's job one of the the key components of how datadog works is that there's this agent binary installed on every single one of our customers production systems i say every single one that is a bit of an exaggeration but every single one you need to collect metric data from you're going to be installing an agent binary on um adding another one just for andrew is the fact that our customers send us some of their most critical software data right like when they need to know when things are down with systems break um they need to have the context and the right data presented for them to be able to reason about what is happening within their software system and so our customers have an extremely high level of trust for us and so andrew certainly has his work cut out for him and one of the core tenets of running a business like this of of having this level of trust of having our binaries installed everywhere it is earning and keeping the trust uh with with our customers and i think earning and keeping trust in my mind is one of the things that andrew has balanced really well um so with that um andrew i'll let you chat a bit about what compliance means at data. thanks kirk so i thought to kind of set the stage we might uh first give a little bit of background on how we've structured the information security and compliance team at datadog where we sit and kind of give a sense to build on what you've talked about give a sense of the scale over the last uh several years so i i was able to join datadog right near the end of 2017 in november at this point in time uh datadog was about 500 people and and the i.t uh security and compliance teams uh was you know between 10 15. um and so over the last uh three and a half years we've seen the the total head count of the company go from you know right there around 500 to over 2 000 we've also seen a dramatic expansion in our technology footprints so uh three and a half years ago datadog was was solely hosted in aws and this as kirk mentioned uh we're now multi-cloud spread across uh you know all the major cloud service providers um which has added uh some interesting nuance and additional complexity to uh the the challenges that we face in protecting customer data and building and earning that trust and in you know building strong security programs as we've as we've scaled across multiple data centers the service that we offer has also continued to grow and evolve um back in uh 2017 uh we kind of had this this model of the three pillars of observability with infrastructure metrics apm and logs management and while those are still very core tenants at datadog we've really evolved into a platform with with multiple products multiple types of data multiple ways to enrich and present that data back to our customers and so again just kind of getting a sense of the scale of growth in both the infrastructure that supports the service as well as the service itself as those things have grown as the companies continue to scale so has our general function in security we now have uh just under 100 folks across our corporate i.t teams our security engineering team and then our infrastructure information security and compliance group in our compliance group specifically that i work in um we're north of 20 and looking to grow so if you're if you're here and interested check out our careers page right now we're structured in kind of two primary pillars so we've got the security engineering function this contains uh teams that focus on things like offensive security our active defense team we have a security development group that includes uh folks who manage our security platform and a growing number of services that uh support both the infrastructure and the application teams on our side of the house information security and compliance we've got a number of teams we have a customer trust team we'll talk a little bit about them here in a moment privacy function that continues to expand and evolve our our team grc are responsible for the governance risk and compliance functions of datadog and then as we've moved into some highly regulated markets recognizing the need for dedicated expertise we've built up a federal program's team to support datadog's engagement with with the government and then a compliance operations team whose mission and mandate is to automate our compliance processes to implement new technical capabilities and then build tools that that support our goals as a company we also work really closely with our corporate i.t and our i.t security teams one of the benefits of i.t and security engineering and infosecond compliance all rolling up to our cso is get the opportunity to work kind of cross-functionally within those teams and try to all drive towards a common goal um so some of the guiding principles i wanted to talk about um with our uh with our scale and as we continue to build out and think about how to evolve with datadog um there's been maybe three things that have informed how we put these things into practice so first we were always conscious and mindful of of risk and our goal of minimizing risk understanding uh the context of that risk and then being able to reduce and manage uh residual risk across the company our position in engineering so we report up through the cto star our position in the engineering function of datadog has has really helped to facilitate um partnerships with our engineering teams which has been really key to the success of our work and and i'd like to think as well key to the success of datadog in a way so minimizing risk preserving velocity data dog is a fast-moving company we do lots of things very quickly and that's that's been an important value at datadog from an engineering perspective this kind of brings an interesting wrinkle as a security and compliance person because uh we're typically and and maybe not unfairly often seen as a drag on velocity or as something that you know will limit the ability to continue to move fast so one of our values has been we want to be able to preserve that velocity maybe functioning a little bit like like airplane flaps helping to generate lift at the right time and then also helping to generate maybe a little bit of caution or slow down where that's appropriate and important and then the last guiding principle so minimizing risk preserving uh the velocity of our engineering teams and as this of the business as a whole and then practicing empathy um practicing empathy for our colleagues recognizing their workload and their uh their own goals uh as teams understanding what the limitations are of the technology and of the resources that we have available i'm trying to really take a look at uh what's our current state what's our aspirational state and then how do we come together to set mutual goals to achieve those those aspirations we also look at practicing empathy for our customers recognizing they're also thinking about security that they're subject to their own compliance and regulatory requirements and then you know in that lens thinking about how we can establish trust and continue to build on the trust foundation and confidence in data dog as a partner so kirk had the slides up uh three three kind of ways that i wanted to talk about how we've how we've implemented those guiding principles and the first tier is thinking about security by design so this is a definitely a common theme across at least our little chunk of the industry you know there's uh there's momentum for security to shift left in the the build process and the development process in business process um and so as we think about security by design at datadog uh what does that really look like and what does that mean i think um for us it continues to evolve and it's something that needs to be dynamic but it really looks like as we as we think about our infrastructure as we think about our processes as our products and different business capabilities that we have that that those teams and and those capabilities incorporate security and compliance as a fundamental component of the design implementation and operation of both our technology and our and our business processes there's really been a natural expansion of that mandate as we've walked through the process of becoming a public company and then entered into some of these more highly regulated markets i mentioned earlier with federal government thinking about fedramp and then expanding our customer base and demonstrating uh that security by design through things like iso 27001 and our software report customer trust uh so we talked again a little bit about the importance of earning and then maintaining the trust of our customers as we as we continue to expand the footprint of datadog as we look into these different regulated markets we really have seen the need to evolve this into a dedicated team so we've got an excellent team in the infosecond compliance group that is is wholly focused on on that customer trust aspect they're really the public face of data dog security organization they're involved in a number of things everything from inbound customer audit requests to tight engagement with our legal and product teams during the due diligence phases of a contract negotiation um they'll also be responsible or they also are responsible for uh the education and engagement of our customers so datadog uses a lot of of uh modern or cutting edge novel technologies to take that risk-based approach to how we how we secure our customers data how we secure data dogs data and so being present to walk customers through walk our colleagues through the different approaches that we've selected to to controlling that security is a primary function for them another function of our team uh is is really enabling the business we talked about um our kind of broad engagement across engineering we also have uh you know many touch points in in the g a side of the house um we contribute i think directly towards the revenue efficiency of datadog how quickly can we turn those deals around how can we anticipate what customers want to know about the security programs at datadogs so that they can they can build that trust um unlocking new markets i've talked a couple times about about fedramp and opening up the government market we also offer a number of products that are hipaa compliant for the healthcare market and then as we continue to look out across the uh the different customer types um the data dog wants to pursue uh you know many of them have associated regulatory requirements or just security requirements in general that [Music] directly with with our groups to to make sure that we can meet those needs i think again focusing back towards towards the engineering side how can we enable the business how can we enable the engineering function within the business to continue to move forward and and to accomplish the product goals that they have the the reliability and resilience goals that we have some of the things that we've taken into consideration here and are actively evolving is this concept of self-service compliance or self-service uh deployment guidance for new products that are rolling out new data centers new infrastructure this has been i think one of the key points that we've identified as critical to our ability to continue to scale across multiple clouds um moving into new countries uh we're as we mentioned earlier a growing team and really relatively a large team for for a security compliance group but even with that right we can't uh we can't keep up with everything and so how do we make sure that our colleagues have the resources they need so that we're not a single point of failure or a bottleneck in that deployment process another thing that we've uh that we focused on is being able to plug into some of our existing engineering workflows um i talked a little bit earlier about the the security engineering function and the partnership that we have with them uh this has also been one of the uh one of the impetus for our compliance operations team so recognizing as data dog scales as we add new customers new data centers we need more robots to do the work there's plenty of compliance that i think traditionally has been done manually we're certainly no strangers to spreadsheets but we also want to take advantage of the modern tooling and the automation capabilities uh that that are present and then where it's where it's appropriate and possible to to build some new tools to operate at datadog scale yes 100 i think one of my own personal experiences with change management has been the idea of somebody who has you know no experience no uh domain specific knowledge for for what i'm shipping that doesn't know what i'm shipping but is in the way of me deploying my software and it's a checkbox and they're not going to look at anything and so i think one of the things that's really exciting about the engineering based approach to compliance for me is that all of this stuff happens and it happens in a way that's more effective than kind of the manual looking and signing off and so i think this is one of those cases where the automation is giving us a better product with a better outcome so all around yes 100 on board for this so let's talk a bit about the experience with corporate laptops i think for me personally having been a a startup employee and having been a employee at a medical device company i i've experienced both sides of the corporate laptop so um from from prior experience you know when you're at a startup uh provisioning and getting a new laptop tends to be maybe you're using your own laptop maybe you're not even giving a company specific laptop and whatever laptop you're given you tend to have admin rights and there is kind of that hybrid approach of well a company's given me a laptop i'm going to play my games on it and i'm going to set it up as if it was a personal laptop i i'm curious kind of your perspective on when it matters for a company when does it begin mattering uh how you view your computational inventory uh from your perspective that's a great question i think there are probably a couple different inflection points on on when you start to see the cultural mentality shift with things like uh with things like laptops one is you know datadog's very responsive to customers and as we've engaged with you know an increasing number of enterprise customers their expectations are that our security process and our footprint looks similar to theirs they have put things in place to protect data that they're responsible for and the expectation is that data dogs sitting downstream to your point earlier processing some of their more sensitive data that we're taking similar precautions so i think i think as a company begins to really engage with uh with enterprise com customers that that's definitely a piece i think you know as we set our own internal security objectives um we we have different benchmarks that we want to hit while at the same time um supporting people in using the tools and the software and the capabilities that they're used to that they're comfortable with familiar with that helps them be most efficient um and so i think you have to you have to identify um when that makes sense for you i think there's also that external driver of customer expectations and then um you know again since we're talking about compliance i'll continue to come back to that i think as we've um made these commitments as a business to engage and serve these highly regulated markets there are then um you know in some cases laws that require you to to put some of these restrictions in place and so really we viewed this at least in part as an opportunity for us to help guide and shape organizational change management at datadog and make that happen in a way that preserves the cultural values that makes datadog i think a really great place for engineers but also then meet those expectations and kind of balance that line between internal risk and customer expectations gotcha so i think there's an interesting thing that has occurred here right uh as we've grown we've also needed to target multiple operating systems can you can you chat a bit about growing beyond the macbook for everybody to to something else yeah so um like a lot of uh i think startups these days datadogs started out with uh everybody gets a macbook and that uh you know simplifies to an extent uh simplifies the uh the management process um for uh you know fleet management um as we've grown um there have been you know folks who come in and say hey i'm really good and and most familiar with with windows and i really would would most benefit from a windows laptop we now have uh and i think for a while i've had folks who have said hey i'm really good with and my job would be most efficient with access to a linux workstation and so those each bring some nuance and complexity and add some overhead um certainly mostly for our for our it team but also for our compliance function as we think about providing guidance on how do we establish common security baselines across these these sort of disparate operating systems and then how do we find that tension between uh you know things like everybody gets local admin to hey now we're going to need to rethink or or revisit what makes the most sense and so being able to put in place sort of logical restrictions based on on on security concerns again both for our customers and for our regulatory requirements yeah and i think it's a good spot to pause you there in in talking about what level of restrictions you put on a laptop because i very distinctly remember the next product we'll talk about rolling out which i might as well introduce it um actually let's let's skip ahead jamf i distinctly remember our cto pushing back on the some of the data points that we were originally going to collect and i think that this was unique from two perspective from from two dimensions from my perspective so the first is that this is happening in public having worked at a medical device company this was very much not a piece of software that there would be a public discussion about it is something that would be hidden away from you that you wouldn't want to be probing for um and then the other thing was that we were talking about collecting less data and i think like with with a tool like this and you know the the general established perspective you're going to say collect more we'll figure out what we don't need later and we'll be more covered in general um so maybe we can chat a bit about what what we're referring to here and how this was rolled out yeah absolutely so i think to your point kirk one of the one of the great things about the way that we're able to practice security and really engineering across the board of data dog is is that it is transparent that we do a lot of these things in public that we really value conversation and feedback and again back to that idea of organizational change management where if we're going to make these changes and you know retain people and continue to preserve velocity and practice empathy that we need buy-in we need people to to agree with the things that we're putting into place and how we want to move the company forward together and so with things like things like jamf we've also implemented cis benchmarks across our workstation fleet these are baseline standards security configuration standards very common in the industry they're you know open source so as we uh as our corporate i.t team begins to make some of these changes and how we uh look at and manage the workstation fleet those conversations happen you know again like you said publicly and github rfcs or you know really big slack channels uh and uh you know there's an opportunity for a push and pull there and lots lots of different considerations right we've got a highly distributed workforce again we have those customer expectations we have our own internal security and management objectives as we try to think about how to go from 500 to 2000 plus to the future okay so that's a good spot sorry to interrupt you to go back to the other side that i skipped over and talk a bit about the encore to just interject there yeah yeah so beyond corp again is is another another strategy um for managing a highly distributed and sort of a modern ite environment um we've uh and in particular our excellent i.t team has has really put a lot of work into implementing some of these strategies uh beyond corp thinking about honest security um from an endpoint managing perspective um and so these are things like post monitoring and configuration management transparency through that rfc process and brown bags and then as you know maybe deviations from our target baseline whether that's software or some configuration or maybe just a version behind as as those things are picked up by the the fleet management system uh that there's a kind of a an unobtrusive but still um persistent uh notification so uh something like uh slack uh notification for a software update or you know almost a push notification in your in your workstation another interesting example i think in this kind of same thread is browser are extensions worst i mean the best yes both at the same time um really interesting and kind of uh unique attack surface for a company i think there's some some excellent tools um and uh you know also an interesting problem for it and and security and compliance teams to try to figure out how and support folks using those tools that they want but then also recognizing you know those those kind of trust dynamics that we talked about earlier so actually just this last uh this last quarter our i.t security team has pushed out some new tooling that will give them the ability to enumerate chrome extensions we run some some tests and some scans on those chrome extensions to get an understanding of the the risk baseline of the security um dynamics of those uh of those browser extensions and then uh send people a slack message if they're using something that's risky and and then eventually um you know having that ability to maybe remove or prohibit that if it presents a risk to the business wow that sounds really cool um i i can think of a few products namely ones that um you know help you write better that that are key loggers ostensibly um and so i can imagine there are uh products which seem to be great that you know from a regulatory standpoint for an enterprise laptop uh are terrifying um and and i'm really now curious about how this extension how this works checking the browser extensions but we will save that for for afterwards all right there's a there's an rfc up in github oh perfect so i can go redo it all right so um let's talk about automated enforcement and i think this this to me is one of the more magical products um from a compliance perspective that i've interacted with or experienced um so why don't we we talk a bit about what this is and what this means yeah so back to um you know back to maybe one of our earlier points about the value of the the partnership with our engineering teams in particular our security engineering teams who write and maintain a lot of tooling for visibility and observability uh across our our production environments our our various pre-prod environments um one of the uh one of the tools that they've put together uh is is this concept of automated enforcement so maybe a quick example of one of the the ways that we've used that as a compliance team is with the integrations into let's say aws being able to go out and look at security group configuration and you know we want to have a sort of an explicit allowed default deny policy on security groups we want to have visibility into what ports are open which resources are are being allowed to communicate across the network and so uh the tooling that they've developed um will not only alert if a a port or a given service is allowed through that's not sort of explicitly allowed if there's a configuration change that's made it also has the ability to revert that change notify the engineer hey you made a change that's kind of against our our standard patterns um if you if you need this change here's who to reach out to um and here's how to kind of walk through the process of of making that change and we use a lot of a lot of tools like this where um you know at least historically in in my sys admin days um you know you'd output a long list of firewall rules and configurations and you kind of have to go through and manually review these these cumbersome documents and so being able to shift that responsibility at least mostly to automation has been a huge benefit i think for certainly the security function but also for us over here in compliance yeah i think on the flip side of that as a developer shipping things and um you know not being aware of the things where that we're allowed to do and not allowed to do or that are okay from a security perspective and definitely not okay from a security perspective i think having that automation in place to tell you hey did you mean to do this because you just did this um one of the things that we have now is you know their script for automatic system management things like terraform and potentially you're applying a script to a piece of infrastructure and you're not as a developer potentially aware of all of the pieces of infrastructure being created um so to me that that is like another layer of security as a developer for helping you ship things uh confidently and knowing okay i'm not going to be doing something that is terrible because i will immediately get notified and it will get shut down right away um so let's talk about one of these tools that you mentioned a bit early here um one of the things that we said at the beginning is datadog runs on kubernetes uh kubernetes means this thing called containers can you tell me a bit about uh how you verify containers or what how do you even reason about containers in security space yeah so i think this is another another great example of how datadog is a company and how the compliance function has really evolved and continues to evolve um we have uh work that was done uh last year that will i think continue into the future with in particular the the compliance operations team one of our one of our senior folks who's actually now the team lead for compliance ops had the opportunity to embed with our build engineering team for a number of months last year build engineering does a ton of really great stuff one of the really interesting things that they've been working on is modernizing and tailoring the deployment process for infrastructure and and code datadock and so you know back to that idea of how do we shift security and compliance left what better place uh to start and and to work with than the folks who are helping us build and deploy software and so uh we had uh we had bill in bed with the build team um last year and uh it was a it was an opportunity for us to not only build relationships and and cultivate a kind of two-way trust uh with with a really core team at datadog but also then uh build in some capabilities and define a roadmap for what the future of compliance evidence and security controls could look like at datadog so you've got uh represented here uh trivi has a sort of just-in-time uh vulnerability scan for for deployments that are moving through our software delivery machine this also gives us the ability to kind of tap into the the main line of code and changes and compliance requirements as they flow through flow through this machine and so thinking about how we can define policy as code leveraging things like the open policy agents and some of the capabilities with uh containers and and microservices and being able to uh standardize um around our security controls in that multi-cloud environment for sure i think it has a lot of parallels to the the infrastructure being automated to containers and upstream dependencies um automatically being flagged as potentially problematic um another thing that that we talked about before is the idea of uh github code owners can you chat a bit about what this means uh in general yeah so this was this was a an interesting uh lesson that we learned um going through the uh the ipo process um so one of the uh one of the things that you pick up as a public company is requirements to adhere to to a law called sarbanes-oxley and without getting into too much kind of wonk detail about socks there's a number of requirements that public companies have to have around their i.t systems that guarantee or that validate your security around financial systems so how do you meter customer data or customer usage how do you turn those uh the data into dollars basically and so one of the one of the particularly i think challenging controls is for specific changes to these regulated systems you have to implement some additional change management control so datadog's code workflow generally encourages or requires two people to approve a change before it is pushed into our production environment socks adds another kind of nuance on top of that where the person uh pushing the change to a financial system that's in scope for these socks requirements has to be qualified to approve that change it needs to be an owner of that system it needs to be someone with you know kind of the authority and accountability um to validate that changes being made to these systems is are approved and expected and meet all these criteria and so again going back to the data dog scale and the emphasis on velocity we landed on building out some functionality around code owners so being able to take that code owners file in github and define some workflows around identifying who are the authorized persons who can approve these changes to our revenue systems and then how can we generate compliance artifacts in a programmatic way have that audit trail have you know any uh any deviations reported allow us to go and do some remedial work if necessary and again this was uh really the result of a close partnership with our engineering teams with our revenue engineering team and some really great work by uh by the folks in grc yep great all right now now for the the hot topic for sure uh supply chain security uh i will turn it over to you and you can talk a bit about um one of the in-house built systems at datadog maybe explain the space a bit at a high level yeah so you you mentioned um at the at the beginning of our of our talk here kirk that uh we're we have over 400 different integrations that folks are installing the data dog binary the open source agents on their systems and so as we think about how to [Music] build a compromise resilient system the one of the one of the really interesting pieces of work that's been done by a number of folks and internally led by one of our staff engineers has been the implementation of tough the update framework and in total which has has allowed us to gain a high level of confidence about the the authenticity and the integrity of agent integrations from the moment that our developers commit source code to the point that our end users install them as packages and um without going into too much detail i'd definitely encourage folks to to check out this blog post by trishank a really interesting read he's he's also got a couple of uh i think video talks that dig a little bit deeper but really neat capability that allows us as compliance folks to represent um the the security capabilities that are built in uh natively to our development process all right so so kind of flipping it around right we introduced kind of a tool that we use to to verify that the software we built is the the software that ends up on people's machines um let's talk a bit about how datadog evaluates vendors and downstream vendor risk what does that even look like how do you evaluate risk in general from from a external vendor standpoint i think that's uh that's the question of the hour especially after uh you know the last uh the last several months i think have have shown a new spotlight on the importance of supply chain risk and supply chain management and how we think about vendors and sending data uh downstream to to other folks and so uh this this process continues to to evolve that data dog it's really grown organically you know every talked a little bit about how over the last three and a half years we've added you know north of 1500 people each of them again has their own tools their own vendors their own sort of preferred way of doing things that ultimately help make datadog a better place for everybody but we recognize the responsibility of one protecting datadog's corporate data information that we kind of generate and maintain internally but then also more and more uh things like uh privacy data and personal information and and our customers data that's uh that's used um uh for for enrichment or uh as as a sub processor so we try to think about this in a couple different ways one we want to be proactive and risk based we want to understand the context of uh what specific data is being sent to a vendor what does that sort of data flow look like and how can we be predictive about some of the potential risk with with these vendors we also want to be responsive so things like solar winds breach like shell shock and heartbleed being able to quickly respond and validate the security capabilities of our vendors moving more and more towards towards automation and trying to remove the manual work there and and then sort of as a part of normal operations walking through a regular at least annual assessment of these vendors so building relationships with with some of our key vendors understanding what their infrastructure and kind of security space looks like um and then uh building uh this assessment process into uh into our own workflows both from an engineering and a business side so uh integration with procurement and our our legal colleagues focused on that security of of data flowing downstream yeah i think one of the most challenging uh parts of your job as an outsider is trying to figure out all the things you haven't thought about yet and trying to think about all of the mental gaps that you have i think that is probably one of the most challenging things to do in general um so this this was great thank you so much for sharing this andrew um for everybody on the call thank you so much we're going to turn it over to questions now um so there there's already a few questions and um again if you you want to kind of join this challenge of reasoning about all the things you're not thinking about and building out these automated systems to both enhance developer impact internally and externally datadog is hiring let's start with one of the questions i think we got a couple really really good ones um first of all andrew what kind of changes did you have to do to work with the united states government where to begin um so so datadog uh last year received a an ato and authorization to operate um from fedramp um and so fedramp uh if you're if you're not familiar is both an organization and a compliance framework uh published by by the u.s government um specifically for uh cloud companies and so um there's three different uh again i'm speaking simple simply here but there are three different tiers to fedramp low moderate and high and so as you walk through those those processes or the gap analysis of where are we at now where do we need to be in order to do these do this business with the government um there's there's lots of small changes i think one of the one of the benefits for for david dog is the the security programs that we've built are not based around check box compliance so we've we've tried to take a look across the industry from the experience that the folks in our department have had and set a high bar for uh for our security programs that then allow us to kind of map some of these compliance frameworks back to uh the things that we already do and then certainly you know closing gaps where they exist so some changes necessary i think specifically in terms of you know access management and how we monitor and log access to certain systems um doing that in a way that uh kind of lines up with the federal expectations uh has been an interesting shift yeah i think as an outsider one of the things that i've noticed is that the the access to information about the status of the systems is more limited than the greater systems in general so so it tends to be that the people working on the system know about the system and inhabit the system and it's it's a little bit less clear from people outside of working in that environment what's going on um okay so we got two more pretty great questions i'm gonna start with the one that came in first so does datadog have international employees if so how does this impact the availability uh this employee has to systems and data and what tools do you use to facilitate these controls that's a that's an excellent question um so to answer the first part of that question absolutely datadog has many international employees we we have a an engineering hub in paris we've got another office in dublin we've got growing presence in apac so truly a truly a global company with a global workforce we use a variety of tools i think one of the one of the maybe more interesting tools for for this conversation is is a tool called appgate appgate is a software-defined perimeter kind of a it's a little bit of a disservice but think of it as maybe a modern-day vpn that allows you to granularly granularly provision access for folks um to just the resources they need so instead of um you know like a vpn kind of dumping you off into um you know a subnet where you have access to any system sitting in that network appgate gives you the ability to say hey andrew can interact with one or two of these systems in this network and that's it and yes the the tool the name of the tool is applicate yep appgate it's super cool access to what you need and can't even see what you don't so the ultimate again going back to enhancing developer productivity the person who is responsible for shipping things if i don't need to see it great don't don't make me look at it and see what it is and try to remember what it is and try to remember that i shouldn't be trying to touch that thing so again a more proactive uh background tasks that that enhances productivity for developers okay so another question it's important to hire security engineers to scale security capabilities to build tools reproduce reported vulnerabilities and fix them rather than relying on engineers who are focused on building features and fixing bugs but how do you ensure security doesn't get siloed within the security team in general man what a great question i think this is um this is a challenge for many organizations and not a stranger at datadog either i think maybe two things that i'd uh call out as beneficial that we've seen some some return from in in avoiding silos and breaking down silos uh from a security perspective um the first is as kirk mentioned earlier that that concept of transparency and having these conversations out in the open this gives i think people who are not focused on security for their full-time job the opportunity to get some insight into what the security and compliance teams are focused on what tools are being built what new um processes are being designed and developed and so i think that that idea of transparency and trying to do things out in the open configuration to an extent right there's some things that that by necessity need to need to remain uh least privileged but in general um i think having those those conversations out in the open has been really beneficial um i think another thing that has been really valuable both on the security engineering side as well as infosection compliance is this concept of partnership and embedding so i talked about how one of our folks had spent some time uh truly embedded with our build engineering team attending the stand-ups uh going in their sock channels he had work signed on their jira board he was truly you know integrated into that into that group and uh the the kind of visibility of both teams work i think was elevated through that process and so um avoiding these silos is a i think uh there's really no end state it's something that you have to consciously um think about and uh you know fight against um as a security team um and and i think those are two ways that we found some success doing that i i can add a another anecdote um having not been on the security team uh there's a couple things that i think the security team has done well we first of all we have an informal space to talk about security in an informal slack channel to talk about security so we can talk about what's going on in the industry i think regardless of what your role is security matters to you right for every engineer you have to care about security um it just it's it's part of being online in general right it's thinking through the implications what what could go wrong what am i not thinking about um the other thing that was really a different mindset for me coming from a medical device company background is when i joined and i joined as a remote employee one of the things a leader at the company told me was to separate my work networking from my personal networking and to keep a completely separate network for my work laptop versus my personal in-home network and so i think the idea of having transparency again to reinforce andrew's point of look we're not here to gather data to have a file should we need to take some action and genuinely saying look we don't want to have this data anywhere near us keep it on your own separate network and you're good um and so i think that that approach in general of having an informal space to discuss where you don't have to be an expert and you can come and ask questions about security events that happen externally um and also being proactive and being a genuine ally outside of the specific security framework um okay so it looks like one more question came through um i like the idea of making everyone responsible for his or her own security compliance with slack alerts etc how do you know where to draw the line between employee lead compiling compliance and force compliance i.e push software and force updates so i i have personally seen force updates so i will let you answer that it is a thing um i think maybe the simplest answer to that is where there are regulatory requirements um that that becomes sort of the minimum baseline or the minimum bar for um hey these things have to be done um we uh to kirk's point this idea of democratizing and sort of distributing the security awareness and security responsibility to our teams has been uh really a great way to facilitate a security conscious culture but at the same time there are still some things that that have to be black and white right we have to have these phone scans done we have to have these change management requirements we have to have uh you know these types of access control things in place and so where where that line is drawn kind of has to be a bright line at meeting those compliance requirements i think one of the benefits or one of the values that our team brings is being able to interpret those in the data dog context and so again back uh to that bright line there's also some ability to use technology to help clarify what that looks like at a company like datadog agreed and i have never felt that it was too heavy-handed um as a personal anecdote in general all right so i don't see oh wait there's one more question let's see this will be the last one potentially and again i think somebody asked earlier if this is recorded yes it's recorded um it will be available on the datadog on youtube uh channel all right while using terraform what is a good datadog tool to monitor my infrastructure i assume the infrastructure tab within datadog would put me on the right path um so yeah i can probably take this one um so you using terraform um there is datadog has a very robust from my perspective integration with aws and so there are multiple ways for you to ingest data from aws number one is what we talked about before kind of having that that agent installed on each one of your your system so potentially each one of your ec2 instances etc um or you can have a container within your um with your within your kubernetes cluster it's scheduled as a daemon set which means it runs the container once per node and sends up all that information but the other thing you can do and that is super powerful is there is an aws integration specifically so rather than installing a binary we get metrics out of aws and so as soon as you start creating anything yes it will start populating within the infrastructure tab within datadog and so there is a bit of latency it's a couple minutes hypothetically of you you enable the integration and then we start adjusting data um and so yes the infrastructure tab will start you down okay but we got one last one we'll see if we can screw it in all right but but yes you'll be able to see it in the infrastructure tab using the aws integration again it will show you on so hopefully that answered last question is there a quick and easy way for datadog to log and alert potential os top 10 related requests to your application any any response there andrew uh so i would say on this one um last year datadog announced a new security monitoring product and i know that there are uh configurations that you could put into place for for things like the owasp top ten um this would be a great question uh to to reach out to our support team and get some insight into the best way to to configure that but if it's not something that can be done now i would keep an eye on the roadmap that's a that's a great idea yep cool thank you so much everybody for attending thank you for sticking through with us um yes thank you glad to have you here glad to have spent the time together and again thank you to andrew for sharing all of this great insight and this will be available on youtube any more questions reach out like andrew said to support and they can route it for you bye everyone have a great one thank you thanks kirk cheers you

[Music] welcome everybody again thank you so much for joining us today uh we&#39;re going to be speaking a bit about security and compliance at datadog i think one of the really interesting things i&#39;ve seen spangler and his team do is use an engineering first approach to security and compliance in a way that i really appreciate having been on the other side um so with that welcome to another episode of datadog on uh this is a series about engineering at datadog uh we go through real world examples of the challenges faced by engineers at datadog and again we we have a url there where you can go and sign up to be notified when more of these are coming up so introductions first of course i&#39;m kirk kaiser i&#39;m a technical evangelist at datadog and i lead the evangelism team at daylor andrew you want to introduce yourself yeah sure my name is andrew spengler i support the governance risk and compliance team at datadog and we sit within the information security and compliance organization cool so backing up a bit a little bit of context about datadog and what we are where we&#39;re coming from so datadog is a monitoring and analytics platform that helps companies improve the observability of the infrastructure and applications what does that mean we help people build software more effectively and we have a bit of scale to contend with we have over 12 000 customers millions of hosts and trillions of data points per day and internally we run mostly on kubernetes um the the data point about kubernetes is significant in that it gives you a bit of the operational overhead that we&#39;re dealing with and a bit of the complexity and in addition to kubernetes and uh kind of the scale of numbers we&#39;re also multi-cloud and we have over 400 different integrations that are used to ship data to datadog um so you kind of get an idea for the space of datadog in general having to deal with complexity and then especially for andrew&#39;s case trying to reason about risk in a place where there&#39;s so many dimensions for variables and to have to consider and to throw another wrench in uh andrew&#39;s job one of the the key components of how datadog works is that there&#39;s this agent binary installed on every single one of our customers production systems i say every single one that is a bit of an exaggeration but every single one you need to collect metric data from you&#39;re going to be installing an agent binary on um adding another one just for andrew is the fact that our customers send us some of their most critical software data right like when they need to know when things are down with systems break um they need to have the context and the right data presented for them to be able to reason about what is happening within their software system and so our customers have an extremely high level of trust for us and so andrew certainly has his work cut out for him and one of the core tenets of running a business like this of of having this level of trust of having our binaries installed everywhere it is earning and keeping the trust uh with with our customers and i think earning and keeping trust in my mind is one of the things that andrew has balanced really well um so with that um andrew i&#39;ll let you chat a bit about what compliance means at data. thanks kirk so i thought to kind of set the stage we might uh first give a little bit of background on how we&#39;ve structured the information security and compliance team at datadog where we sit and kind of give a sense to build on what you&#39;ve talked about give a sense of the scale over the last uh several years so i i was able to join datadog right near the end of 2017 in november at this point in time uh datadog was about 500 people and and the i.t uh security and compliance teams uh was you know between 10 15. um and so over the last uh three and a half years we&#39;ve seen the the total head count of the company go from you know right there around 500 to over 2 000 we&#39;ve also seen a dramatic expansion in our technology footprints so uh three and a half years ago datadog was was solely hosted in aws and this as kirk mentioned uh we&#39;re now multi-cloud spread across uh you know all the major cloud service providers um which has added uh some interesting nuance and additional complexity to uh the the challenges that we face in protecting customer data and building and earning that trust and in you know building strong security programs as we&#39;ve as we&#39;ve scaled across multiple data centers the service that we offer has also continued to grow and evolve um back in uh 2017 uh we kind of had this this model of the three pillars of observability with infrastructure metrics apm and logs management and while those are still very core tenants at datadog we&#39;ve really evolved into a platform with with multiple products multiple types of data multiple ways to enrich and present that data back to our customers and so again just kind of getting a sense of the scale of growth in both the infrastructure that supports the service as well as the service itself as those things have grown as the companies continue to scale so has our general function in security we now have uh just under 100 folks across our corporate i.t teams our security engineering team and then our infrastructure information security and compliance group in our compliance group specifically that i work in um we&#39;re north of 20 and looking to grow so if you&#39;re if you&#39;re here and interested check out our careers page right now we&#39;re structured in kind of two primary pillars so we&#39;ve got the security engineering function this contains uh teams that focus on things like offensive security our active defense team we have a security development group that includes uh folks who manage our security platform and a growing number of services that uh support both the infrastructure and the application teams on our side of the house information security and compliance we&#39;ve got a number of teams we have a customer trust team we&#39;ll talk a little bit about them here in a moment privacy function that continues to expand and evolve our our team grc are responsible for the governance risk and compliance functions of datadog and then as we&#39;ve moved into some highly regulated markets recognizing the need for dedicated expertise we&#39;ve built up a federal program&#39;s team to support datadog&#39;s engagement with with the government and then a compliance operations team whose mission and mandate is to automate our compliance processes to implement new technical capabilities and then build tools that that support our goals as a company we also work really closely with our corporate i.t and our i.t security teams one of the benefits of i.t and security engineering and infosecond compliance all rolling up to our cso is get the opportunity to work kind of cross-functionally within those teams and try to all drive towards a common goal um so some of the guiding principles i wanted to talk about um with our uh with our scale and as we continue to build out and think about how to evolve with datadog um there&#39;s been maybe three things that have informed how we put these things into practice so first we were always conscious and mindful of of risk and our goal of minimizing risk understanding uh the context of that risk and then being able to reduce and manage uh residual risk across the company our position in engineering so we report up through the cto star our position in the engineering function of datadog has has really helped to facilitate um partnerships with our engineering teams which has been really key to the success of our work and and i&#39;d like to think as well key to the success of datadog in a way so minimizing risk preserving velocity data dog is a fast-moving company we do lots of things very quickly and that&#39;s that&#39;s been an important value at datadog from an engineering perspective this kind of brings an interesting wrinkle as a security and compliance person because uh we&#39;re typically and and maybe not unfairly often seen as a drag on velocity or as something that you know will limit the ability to continue to move fast so one of our values has been we want to be able to preserve that velocity maybe functioning a little bit like like airplane flaps helping to generate lift at the right time and then also helping to generate maybe a little bit of caution or slow down where that&#39;s appropriate and important and then the last guiding principle so minimizing risk preserving uh the velocity of our engineering teams and as this of the business as a whole and then practicing empathy um practicing empathy for our colleagues recognizing their workload and their uh their own goals uh as teams understanding what the limitations are of the technology and of the resources that we have available i&#39;m trying to really take a look at uh what&#39;s our current state what&#39;s our aspirational state and then how do we come together to set mutual goals to achieve those those aspirations we also look at practicing empathy for our customers recognizing they&#39;re also thinking about security that they&#39;re subject to their own compliance and regulatory requirements and then you know in that lens thinking about how we can establish trust and continue to build on the trust foundation and confidence in data dog as a partner so kirk had the slides up uh three three kind of ways that i wanted to talk about how we&#39;ve how we&#39;ve implemented those guiding principles and the first tier is thinking about security by design so this is a definitely a common theme across at least our little chunk of the industry you know there&#39;s uh there&#39;s momentum for security to shift left in the the build process and the development process in business process um and so as we think about security by design at datadog uh what does that really look like and what does that mean i think um for us it continues to evolve and it&#39;s something that needs to be dynamic but it really looks like as we as we think about our infrastructure as we think about our processes as our products and different business capabilities that we have that that those teams and and those capabilities incorporate security and compliance as a fundamental component of the design implementation and operation of both our technology and our and our business processes there&#39;s really been a natural expansion of that mandate as we&#39;ve walked through the process of becoming a public company and then entered into some of these more highly regulated markets i mentioned earlier with federal government thinking about fedramp and then expanding our customer base and demonstrating uh that security by design through things like iso 27001 and our software report customer trust uh so we talked again a little bit about the importance of earning and then maintaining the trust of our customers as we as we continue to expand the footprint of datadog as we look into these different regulated markets we really have seen the need to evolve this into a dedicated team so we&#39;ve got an excellent team in the infosecond compliance group that is is wholly focused on on that customer trust aspect they&#39;re really the public face of data dog security organization they&#39;re involved in a number of things everything from inbound customer audit requests to tight engagement with our legal and product teams during the due diligence phases of a contract negotiation um they&#39;ll also be responsible or they also are responsible for uh the education and engagement of our customers so datadog uses a lot of of uh modern or cutting edge novel technologies to take that risk-based approach to how we how we secure our customers data how we secure data dogs data and so being present to walk customers through walk our colleagues through the different approaches that we&#39;ve selected to to controlling that security is a primary function for them another function of our team uh is is really enabling the business we talked about um our kind of broad engagement across engineering we also have uh you know many touch points in in the g a side of the house um we contribute i think directly towards the revenue efficiency of datadog how quickly can we turn those deals around how can we anticipate what customers want to know about the security programs at datadogs so that they can they can build that trust um unlocking new markets i&#39;ve talked a couple times about about fedramp and opening up the government market we also offer a number of products that are hipaa compliant for the healthcare market and then as we continue to look out across the uh the different customer types um the data dog wants to pursue uh you know many of them have associated regulatory requirements or just security requirements in general that [Music] directly with with our groups to to make sure that we can meet those needs i think again focusing back towards towards the engineering side how can we enable the business how can we enable the engineering function within the business to continue to move forward and and to accomplish the product goals that they have the the reliability and resilience goals that we have some of the things that we&#39;ve taken into consideration here and are actively evolving is this concept of self-service compliance or self-service uh deployment guidance for new products that are rolling out new data centers new infrastructure this has been i think one of the key points that we&#39;ve identified as critical to our ability to continue to scale across multiple clouds um moving into new countries uh we&#39;re as we mentioned earlier a growing team and really relatively a large team for for a security compliance group but even with that right we can&#39;t uh we can&#39;t keep up with everything and so how do we make sure that our colleagues have the resources they need so that we&#39;re not a single point of failure or a bottleneck in that deployment process another thing that we&#39;ve uh that we focused on is being able to plug into some of our existing engineering workflows um i talked a little bit earlier about the the security engineering function and the partnership that we have with them uh this has also been one of the uh one of the impetus for our compliance operations team so recognizing as data dog scales as we add new customers new data centers we need more robots to do the work there&#39;s plenty of compliance that i think traditionally has been done manually we&#39;re certainly no strangers to spreadsheets but we also want to take advantage of the modern tooling and the automation capabilities uh that that are present and then where it&#39;s where it&#39;s appropriate and possible to to build some new tools to operate at datadog scale yes 100 i think one of my own personal experiences with change management has been the idea of somebody who has you know no experience no uh domain specific knowledge for for what i&#39;m shipping that doesn&#39;t know what i&#39;m shipping but is in the way of me deploying my software and it&#39;s a checkbox and they&#39;re not going to look at anything and so i think one of the things that&#39;s really exciting about the engineering based approach to compliance for me is that all of this stuff happens and it happens in a way that&#39;s more effective than kind of the manual looking and signing off and so i think this is one of those cases where the automation is giving us a better product with a better outcome so all around yes 100 on board for this so let&#39;s talk a bit about the experience with corporate laptops i think for me personally having been a a startup employee and having been a employee at a medical device company i i&#39;ve experienced both sides of the corporate laptop so um from from prior experience you know when you&#39;re at a startup uh provisioning and getting a new laptop tends to be maybe you&#39;re using your own laptop maybe you&#39;re not even giving a company specific laptop and whatever laptop you&#39;re given you tend to have admin rights and there is kind of that hybrid approach of well a company&#39;s given me a laptop i&#39;m going to play my games on it and i&#39;m going to set it up as if it was a personal laptop i i&#39;m curious kind of your perspective on when it matters for a company when does it begin mattering uh how you view your computational inventory uh from your perspective that&#39;s a great question i think there are probably a couple different inflection points on on when you start to see the cultural mentality shift with things like uh with things like laptops one is you know datadog&#39;s very responsive to customers and as we&#39;ve engaged with you know an increasing number of enterprise customers their expectations are that our security process and our footprint looks similar to theirs they have put things in place to protect data that they&#39;re responsible for and the expectation is that data dogs sitting downstream to your point earlier processing some of their more sensitive data that we&#39;re taking similar precautions so i think i think as a company begins to really engage with uh with enterprise com customers that that&#39;s definitely a piece i think you know as we set our own internal security objectives um we we have different benchmarks that we want to hit while at the same time um supporting people in using the tools and the software and the capabilities that they&#39;re used to that they&#39;re comfortable with familiar with that helps them be most efficient um and so i think you have to you have to identify um when that makes sense for you i think there&#39;s also that external driver of customer expectations and then um you know again since we&#39;re talking about compliance i&#39;ll continue to come back to that i think as we&#39;ve um made these commitments as a business to engage and serve these highly regulated markets there are then um you know in some cases laws that require you to to put some of these restrictions in place and so really we viewed this at least in part as an opportunity for us to help guide and shape organizational change management at datadog and make that happen in a way that preserves the cultural values that makes datadog i think a really great place for engineers but also then meet those expectations and kind of balance that line between internal risk and customer expectations gotcha so i think there&#39;s an interesting thing that has occurred here right uh as we&#39;ve grown we&#39;ve also needed to target multiple operating systems can you can you chat a bit about growing beyond the macbook for everybody to to something else yeah so um like a lot of uh i think startups these days datadogs started out with uh everybody gets a macbook and that uh you know simplifies to an extent uh simplifies the uh the management process um for uh you know fleet management um as we&#39;ve grown um there have been you know folks who come in and say hey i&#39;m really good and and most familiar with with windows and i really would would most benefit from a windows laptop we now have uh and i think for a while i&#39;ve had folks who have said hey i&#39;m really good with and my job would be most efficient with access to a linux workstation and so those each bring some nuance and complexity and add some overhead um certainly mostly for our for our it team but also for our compliance function as we think about providing guidance on how do we establish common security baselines across these these sort of disparate operating systems and then how do we find that tension between uh you know things like everybody gets local admin to hey now we&#39;re going to need to rethink or or revisit what makes the most sense and so being able to put in place sort of logical restrictions based on on on security concerns again both for our customers and for our regulatory requirements yeah and i think it&#39;s a good spot to pause you there in in talking about what level of restrictions you put on a laptop because i very distinctly remember the next product we&#39;ll talk about rolling out which i might as well introduce it um actually let&#39;s let&#39;s skip ahead jamf i distinctly remember our cto pushing back on the some of the data points that we were originally going to collect and i think that this was unique from two perspective from from two dimensions from my perspective so the first is that this is happening in public having worked at a medical device company this was very much not a piece of software that there would be a public discussion about it is something that would be hidden away from you that you wouldn&#39;t want to be probing for um and then the other thing was that we were talking about collecting less data and i think like with with a tool like this and you know the the general established perspective you&#39;re going to say collect more we&#39;ll figure out what we don&#39;t need later and we&#39;ll be more covered in general um so maybe we can chat a bit about what what we&#39;re referring to here and how this was rolled out yeah absolutely so i think to your point kirk one of the one of the great things about the way that we&#39;re able to practice security and really engineering across the board of data dog is is that it is transparent that we do a lot of these things in public that we really value conversation and feedback and again back to that idea of organizational change management where if we&#39;re going to make these changes and you know retain people and continue to preserve velocity and practice empathy that we need buy-in we need people to to agree with the things that we&#39;re putting into place and how we want to move the company forward together and so with things like things like jamf we&#39;ve also implemented cis benchmarks across our workstation fleet these are baseline standards security configuration standards very common in the industry they&#39;re you know open source so as we uh as our corporate i.t team begins to make some of these changes and how we uh look at and manage the workstation fleet those conversations happen you know again like you said publicly and github rfcs or you know really big slack channels uh and uh you know there&#39;s an opportunity for a push and pull there and lots lots of different considerations right we&#39;ve got a highly distributed workforce again we have those customer expectations we have our own internal security and management objectives as we try to think about how to go from 500 to 2000 plus to the future okay so that&#39;s a good spot sorry to interrupt you to go back to the other side that i skipped over and talk a bit about the encore to just interject there yeah yeah so beyond corp again is is another another strategy um for managing a highly distributed and sort of a modern ite environment um we&#39;ve uh and in particular our excellent i.t team has has really put a lot of work into implementing some of these strategies uh beyond corp thinking about honest security um from an endpoint managing perspective um and so these are things like post monitoring and configuration management transparency through that rfc process and brown bags and then as you know maybe deviations from our target baseline whether that&#39;s software or some configuration or maybe just a version behind as as those things are picked up by the the fleet management system uh that there&#39;s a kind of a an unobtrusive but still um persistent uh notification so uh something like uh slack uh notification for a software update or you know almost a push notification in your in your workstation another interesting example i think in this kind of same thread is browser are extensions worst i mean the best yes both at the same time um really interesting and kind of uh unique attack surface for a company i think there&#39;s some some excellent tools um and uh you know also an interesting problem for it and and security and compliance teams to try to figure out how and support folks using those tools that they want but then also recognizing you know those those kind of trust dynamics that we talked about earlier so actually just this last uh this last quarter our i.t security team has pushed out some new tooling that will give them the ability to enumerate chrome extensions we run some some tests and some scans on those chrome extensions to get an understanding of the the risk baseline of the security um dynamics of those uh of those browser extensions and then uh send people a slack message if they&#39;re using something that&#39;s risky and and then eventually um you know having that ability to maybe remove or prohibit that if it presents a risk to the business wow that sounds really cool um i i can think of a few products namely ones that um you know help you write better that that are key loggers ostensibly um and so i can imagine there are uh products which seem to be great that you know from a regulatory standpoint for an enterprise laptop uh are terrifying um and and i&#39;m really now curious about how this extension how this works checking the browser extensions but we will save that for for afterwards all right there&#39;s a there&#39;s an rfc up in github oh perfect so i can go redo it all right so um let&#39;s talk about automated enforcement and i think this this to me is one of the more magical products um from a compliance perspective that i&#39;ve interacted with or experienced um so why don&#39;t we we talk a bit about what this is and what this means yeah so back to um you know back to maybe one of our earlier points about the value of the the partnership with our engineering teams in particular our security engineering teams who write and maintain a lot of tooling for visibility and observability uh across our our production environments our our various pre-prod environments um one of the uh one of the tools that they&#39;ve put together uh is is this concept of automated enforcement so maybe a quick example of one of the the ways that we&#39;ve used that as a compliance team is with the integrations into let&#39;s say aws being able to go out and look at security group configuration and you know we want to have a sort of an explicit allowed default deny policy on security groups we want to have visibility into what ports are open which resources are are being allowed to communicate across the network and so uh the tooling that they&#39;ve developed um will not only alert if a a port or a given service is allowed through that&#39;s not sort of explicitly allowed if there&#39;s a configuration change that&#39;s made it also has the ability to revert that change notify the engineer hey you made a change that&#39;s kind of against our our standard patterns um if you if you need this change here&#39;s who to reach out to um and here&#39;s how to kind of walk through the process of of making that change and we use a lot of a lot of tools like this where um you know at least historically in in my sys admin days um you know you&#39;d output a long list of firewall rules and configurations and you kind of have to go through and manually review these these cumbersome documents and so being able to shift that responsibility at least mostly to automation has been a huge benefit i think for certainly the security function but also for us over here in compliance yeah i think on the flip side of that as a developer shipping things and um you know not being aware of the things where that we&#39;re allowed to do and not allowed to do or that are okay from a security perspective and definitely not okay from a security perspective i think having that automation in place to tell you hey did you mean to do this because you just did this um one of the things that we have now is you know their script for automatic system management things like terraform and potentially you&#39;re applying a script to a piece of infrastructure and you&#39;re not as a developer potentially aware of all of the pieces of infrastructure being created um so to me that that is like another layer of security as a developer for helping you ship things uh confidently and knowing okay i&#39;m not going to be doing something that is terrible because i will immediately get notified and it will get shut down right away um so let&#39;s talk about one of these tools that you mentioned a bit early here um one of the things that we said at the beginning is datadog runs on kubernetes uh kubernetes means this thing called containers can you tell me a bit about uh how you verify containers or what how do you even reason about containers in security space yeah so i think this is another another great example of how datadog is a company and how the compliance function has really evolved and continues to evolve um we have uh work that was done uh last year that will i think continue into the future with in particular the the compliance operations team one of our one of our senior folks who&#39;s actually now the team lead for compliance ops had the opportunity to embed with our build engineering team for a number of months last year build engineering does a ton of really great stuff one of the really interesting things that they&#39;ve been working on is modernizing and tailoring the deployment process for infrastructure and and code datadock and so you know back to that idea of how do we shift security and compliance left what better place uh to start and and to work with than the folks who are helping us build and deploy software and so uh we had uh we had bill in bed with the build team um last year and uh it was a it was an opportunity for us to not only build relationships and and cultivate a kind of two-way trust uh with with a really core team at datadog but also then uh build in some capabilities and define a roadmap for what the future of compliance evidence and security controls could look like at datadog so you&#39;ve got uh represented here uh trivi has a sort of just-in-time uh vulnerability scan for for deployments that are moving through our software delivery machine this also gives us the ability to kind of tap into the the main line of code and changes and compliance requirements as they flow through flow through this machine and so thinking about how we can define policy as code leveraging things like the open policy agents and some of the capabilities with uh containers and and microservices and being able to uh standardize um around our security controls in that multi-cloud environment for sure i think it has a lot of parallels to the the infrastructure being automated to containers and upstream dependencies um automatically being flagged as potentially problematic um another thing that that we talked about before is the idea of uh github code owners can you chat a bit about what this means uh in general yeah so this was this was a an interesting uh lesson that we learned um going through the uh the ipo process um so one of the uh one of the things that you pick up as a public company is requirements to adhere to to a law called sarbanes-oxley and without getting into too much kind of wonk detail about socks there&#39;s a number of requirements that public companies have to have around their i.t systems that guarantee or that validate your security around financial systems so how do you meter customer data or customer usage how do you turn those uh the data into dollars basically and so one of the one of the particularly i think challenging controls is for specific changes to these regulated systems you have to implement some additional change management control so datadog&#39;s code workflow generally encourages or requires two people to approve a change before it is pushed into our production environment socks adds another kind of nuance on top of that where the person uh pushing the change to a financial system that&#39;s in scope for these socks requirements has to be qualified to approve that change it needs to be an owner of that system it needs to be someone with you know kind of the authority and accountability um to validate that changes being made to these systems is are approved and expected and meet all these criteria and so again going back to the data dog scale and the emphasis on velocity we landed on building out some functionality around code owners so being able to take that code owners file in github and define some workflows around identifying who are the authorized persons who can approve these changes to our revenue systems and then how can we generate compliance artifacts in a programmatic way have that audit trail have you know any uh any deviations reported allow us to go and do some remedial work if necessary and again this was uh really the result of a close partnership with our engineering teams with our revenue engineering team and some really great work by uh by the folks in grc yep great all right now now for the the hot topic for sure uh supply chain security uh i will turn it over to you and you can talk a bit about um one of the in-house built systems at datadog maybe explain the space a bit at a high level yeah so you you mentioned um at the at the beginning of our of our talk here kirk that uh we&#39;re we have over 400 different integrations that folks are installing the data dog binary the open source agents on their systems and so as we think about how to [Music] build a compromise resilient system the one of the one of the really interesting pieces of work that&#39;s been done by a number of folks and internally led by one of our staff engineers has been the implementation of tough the update framework and in total which has has allowed us to gain a high level of confidence about the the authenticity and the integrity of agent integrations from the moment that our developers commit source code to the point that our end users install them as packages and um without going into too much detail i&#39;d definitely encourage folks to to check out this blog post by trishank a really interesting read he&#39;s he&#39;s also got a couple of uh i think video talks that dig a little bit deeper but really neat capability that allows us as compliance folks to represent um the the security capabilities that are built in uh natively to our development process all right so so kind of flipping it around right we introduced kind of a tool that we use to to verify that the software we built is the the software that ends up on people&#39;s machines um let&#39;s talk a bit about how datadog evaluates vendors and downstream vendor risk what does that even look like how do you evaluate risk in general from from a external vendor standpoint i think that&#39;s uh that&#39;s the question of the hour especially after uh you know the last uh the last several months i think have have shown a new spotlight on the importance of supply chain risk and supply chain management and how we think about vendors and sending data uh downstream to to other folks and so uh this this process continues to to evolve that data dog it&#39;s really grown organically you know every talked a little bit about how over the last three and a half years we&#39;ve added you know north of 1500 people each of them again has their own tools their own vendors their own sort of preferred way of doing things that ultimately help make datadog a better place for everybody but we recognize the responsibility of one protecting datadog&#39;s corporate data information that we kind of generate and maintain internally but then also more and more uh things like uh privacy data and personal information and and our customers data that&#39;s uh that&#39;s used um uh for for enrichment or uh as as a sub processor so we try to think about this in a couple different ways one we want to be proactive and risk based we want to understand the context of uh what specific data is being sent to a vendor what does that sort of data flow look like and how can we be predictive about some of the potential risk with with these vendors we also want to be responsive so things like solar winds breach like shell shock and heartbleed being able to quickly respond and validate the security capabilities of our vendors moving more and more towards towards automation and trying to remove the manual work there and and then sort of as a part of normal operations walking through a regular at least annual assessment of these vendors so building relationships with with some of our key vendors understanding what their infrastructure and kind of security space looks like um and then uh building uh this assessment process into uh into our own workflows both from an engineering and a business side so uh integration with procurement and our our legal colleagues focused on that security of of data flowing downstream yeah i think one of the most challenging uh parts of your job as an outsider is trying to figure out all the things you haven&#39;t thought about yet and trying to think about all of the mental gaps that you have i think that is probably one of the most challenging things to do in general um so this this was great thank you so much for sharing this andrew um for everybody on the call thank you so much we&#39;re going to turn it over to questions now um so there there&#39;s already a few questions and um again if you you want to kind of join this challenge of reasoning about all the things you&#39;re not thinking about and building out these automated systems to both enhance developer impact internally and externally datadog is hiring let&#39;s start with one of the questions i think we got a couple really really good ones um first of all andrew what kind of changes did you have to do to work with the united states government where to begin um so so datadog uh last year received a an ato and authorization to operate um from fedramp um and so fedramp uh if you&#39;re if you&#39;re not familiar is both an organization and a compliance framework uh published by by the u.s government um specifically for uh cloud companies and so um there&#39;s three different uh again i&#39;m speaking simple simply here but there are three different tiers to fedramp low moderate and high and so as you walk through those those processes or the gap analysis of where are we at now where do we need to be in order to do these do this business with the government um there&#39;s there&#39;s lots of small changes i think one of the one of the benefits for for david dog is the the security programs that we&#39;ve built are not based around check box compliance so we&#39;ve we&#39;ve tried to take a look across the industry from the experience that the folks in our department have had and set a high bar for uh for our security programs that then allow us to kind of map some of these compliance frameworks back to uh the things that we already do and then certainly you know closing gaps where they exist so some changes necessary i think specifically in terms of you know access management and how we monitor and log access to certain systems um doing that in a way that uh kind of lines up with the federal expectations uh has been an interesting shift yeah i think as an outsider one of the things that i&#39;ve noticed is that the the access to information about the status of the systems is more limited than the greater systems in general so so it tends to be that the people working on the system know about the system and inhabit the system and it&#39;s it&#39;s a little bit less clear from people outside of working in that environment what&#39;s going on um okay so we got two more pretty great questions i&#39;m gonna start with the one that came in first so does datadog have international employees if so how does this impact the availability uh this employee has to systems and data and what tools do you use to facilitate these controls that&#39;s a that&#39;s an excellent question um so to answer the first part of that question absolutely datadog has many international employees we we have a an engineering hub in paris we&#39;ve got another office in dublin we&#39;ve got growing presence in apac so truly a truly a global company with a global workforce we use a variety of tools i think one of the one of the maybe more interesting tools for for this conversation is is a tool called appgate appgate is a software-defined perimeter kind of a it&#39;s a little bit of a disservice but think of it as maybe a modern-day vpn that allows you to granularly granularly provision access for folks um to just the resources they need so instead of um you know like a vpn kind of dumping you off into um you know a subnet where you have access to any system sitting in that network appgate gives you the ability to say hey andrew can interact with one or two of these systems in this network and that&#39;s it and yes the the tool the name of the tool is applicate yep appgate it&#39;s super cool access to what you need and can&#39;t even see what you don&#39;t so the ultimate again going back to enhancing developer productivity the person who is responsible for shipping things if i don&#39;t need to see it great don&#39;t don&#39;t make me look at it and see what it is and try to remember what it is and try to remember that i shouldn&#39;t be trying to touch that thing so again a more proactive uh background tasks that that enhances productivity for developers okay so another question it&#39;s important to hire security engineers to scale security capabilities to build tools reproduce reported vulnerabilities and fix them rather than relying on engineers who are focused on building features and fixing bugs but how do you ensure security doesn&#39;t get siloed within the security team in general man what a great question i think this is um this is a challenge for many organizations and not a stranger at datadog either i think maybe two things that i&#39;d uh call out as beneficial that we&#39;ve seen some some return from in in avoiding silos and breaking down silos uh from a security perspective um the first is as kirk mentioned earlier that that concept of transparency and having these conversations out in the open this gives i think people who are not focused on security for their full-time job the opportunity to get some insight into what the security and compliance teams are focused on what tools are being built what new um processes are being designed and developed and so i think that that idea of transparency and trying to do things out in the open configuration to an extent right there&#39;s some things that that by necessity need to need to remain uh least privileged but in general um i think having those those conversations out in the open has been really beneficial um i think another thing that has been really valuable both on the security engineering side as well as infosection compliance is this concept of partnership and embedding so i talked about how one of our folks had spent some time uh truly embedded with our build engineering team attending the stand-ups uh going in their sock channels he had work signed on their jira board he was truly you know integrated into that into that group and uh the the kind of visibility of both teams work i think was elevated through that process and so um avoiding these silos is a i think uh there&#39;s really no end state it&#39;s something that you have to consciously um think about and uh you know fight against um as a security team um and and i think those are two ways that we found some success doing that i i can add a another anecdote um having not been on the security team uh there&#39;s a couple things that i think the security team has done well we first of all we have an informal space to talk about security in an informal slack channel to talk about security so we can talk about what&#39;s going on in the industry i think regardless of what your role is security matters to you right for every engineer you have to care about security um it just it&#39;s it&#39;s part of being online in general right it&#39;s thinking through the implications what what could go wrong what am i not thinking about um the other thing that was really a different mindset for me coming from a medical device company background is when i joined and i joined as a remote employee one of the things a leader at the company told me was to separate my work networking from my personal networking and to keep a completely separate network for my work laptop versus my personal in-home network and so i think the idea of having transparency again to reinforce andrew&#39;s point of look we&#39;re not here to gather data to have a file should we need to take some action and genuinely saying look we don&#39;t want to have this data anywhere near us keep it on your own separate network and you&#39;re good um and so i think that that approach in general of having an informal space to discuss where you don&#39;t have to be an expert and you can come and ask questions about security events that happen externally um and also being proactive and being a genuine ally outside of the specific security framework um okay so it looks like one more question came through um i like the idea of making everyone responsible for his or her own security compliance with slack alerts etc how do you know where to draw the line between employee lead compiling compliance and force compliance i.e push software and force updates so i i have personally seen force updates so i will let you answer that it is a thing um i think maybe the simplest answer to that is where there are regulatory requirements um that that becomes sort of the minimum baseline or the minimum bar for um hey these things have to be done um we uh to kirk&#39;s point this idea of democratizing and sort of distributing the security awareness and security responsibility to our teams has been uh really a great way to facilitate a security conscious culture but at the same time there are still some things that that have to be black and white right we have to have these phone scans done we have to have these change management requirements we have to have uh you know these types of access control things in place and so where where that line is drawn kind of has to be a bright line at meeting those compliance requirements i think one of the benefits or one of the values that our team brings is being able to interpret those in the data dog context and so again back uh to that bright line there&#39;s also some ability to use technology to help clarify what that looks like at a company like datadog agreed and i have never felt that it was too heavy-handed um as a personal anecdote in general all right so i don&#39;t see oh wait there&#39;s one more question let&#39;s see this will be the last one potentially and again i think somebody asked earlier if this is recorded yes it&#39;s recorded um it will be available on the datadog on youtube uh channel all right while using terraform what is a good datadog tool to monitor my infrastructure i assume the infrastructure tab within datadog would put me on the right path um so yeah i can probably take this one um so you using terraform um there is datadog has a very robust from my perspective integration with aws and so there are multiple ways for you to ingest data from aws number one is what we talked about before kind of having that that agent installed on each one of your your system so potentially each one of your ec2 instances etc um or you can have a container within your um with your within your kubernetes cluster it&#39;s scheduled as a daemon set which means it runs the container once per node and sends up all that information but the other thing you can do and that is super powerful is there is an aws integration specifically so rather than installing a binary we get metrics out of aws and so as soon as you start creating anything yes it will start populating within the infrastructure tab within datadog and so there is a bit of latency it&#39;s a couple minutes hypothetically of you you enable the integration and then we start adjusting data um and so yes the infrastructure tab will start you down okay but we got one last one we&#39;ll see if we can screw it in all right but but yes you&#39;ll be able to see it in the infrastructure tab using the aws integration again it will show you on so hopefully that answered last question is there a quick and easy way for datadog to log and alert potential os top 10 related requests to your application any any response there andrew uh so i would say on this one um last year datadog announced a new security monitoring product and i know that there are uh configurations that you could put into place for for things like the owasp top ten um this would be a great question uh to to reach out to our support team and get some insight into the best way to to configure that but if it&#39;s not something that can be done now i would keep an eye on the roadmap that&#39;s a that&#39;s a great idea yep cool thank you so much everybody for attending thank you for sticking through with us um yes thank you glad to have you here glad to have spent the time together and again thank you to andrew for sharing all of this great insight and this will be available on youtube any more questions reach out like andrew said to support and they can route it for you bye everyone have a great one thank you thanks kirk cheers you

Transcript for:Security and Compliance at Datadog

Transcript for:
Security and Compliance at Datadog