[Music] [Music] so I don't think Jeff Bezos needs much of an introduction you perhaps heard of this small company he runs out of some city in the west coast where there's another company the I took a poll of the faculty recently and and it's at a faculty lunch and we were talking about what people did online and I sort of asked people questions and eventually I said well how many of you have bought something at amazon.com and it was actually a hundred percent and I think raised their hands I guarantee it was not one of those any rate Jeff is a graduate of Princeton and I'm pleased to say he was telling me upstairs he graduated from Princeton when it was still an East and electrical engineering and computer science department before the department bifurcated and so he does have an electrical engineering and computer science degree and I think he's he tells me he's glad he has one of those and has he should be after leaving Princeton he went to to Bankers Trust worked there in the development of computer systems so presumably if you need help with code he can help you with some of your programming and stuff of that nature after that he went to de Shaw well known for their high tech approach to finance and in fact one of two financial services companies three I guess that belongs to our industrial program in the department and then he left de Shaw to start amazon.com and I'm sure he can tell you more about that than I can Jeff okay thank you you guys it's great to be here and so I start out telling you a little bit about the founding of amazon.com that I'm gonna do my favorite thing which is to give a live website demo this is the house where Amazon was founded on the left-hand side here is this little enclosed garage and I was living in New York City working for Deshawn Coe and came across the fact that web usage was growing at 2,300 percent a year this was one of those weird things where you could measure the rate of growth without actually knowing the baseline usage and people were doing that because they could put sniffers at various points at various nodes and they could sort of see how fast the traffic levels were growing past those nodes and they could get a statistical sample of those but they actually had no idea what baseline usage was but you could look around the web already in the spring of 1994 and see that it was big enough that if it was growing at 23% 2,300 percent a year pretty soon was gonna be huge and so I you know kind of if we packed up my wife and I and we flew to Fort Worth Texas where my dad gave us a car 1988 Chevy Blazer which by the way Consumer Reports recommends not to buy used under any circumstances for any price and we drove that car from Texas to Seattle Seattle was chosen as the the best place to start amazon.com because it was a large pool of technical talent and also nearby what at the time was the largest book warehouse in the world in a town called Roseburg Oregon and so we got there my the first order of business was to recruit a VP of engineering and I talked to several people and I found the guy I really liked his name was shale he was with the company for six years and was a he of you know huge instrumental force in the company and in the early days you know not single of course but in large part led all the software engineering that that was amazon.com and he had nothing to do with this you know the look of this website though I'd like to get on that this is what this is what our website looked like when we launched in 1995 and I wrote all of that HTML myself please hold your applause and but but shall part he wrote all of the all the code that actually made it work and but before hiring shel we're making I I was making our deaths out of doors and four-by-fours and it took me three months to convince shel to come join this company because it was such a you know it's really just a one-person company it was me and it was a piece of paper and a name and the name wasn't even Amazon at that time it was Cadabra Cadabra got changed because whenever I would say to anybody over the phone they would think I said cadaver and I realized that wasn't going to work very well Amazon was incorporated by the way by Ike I didn't know I only knew two people in Seattle so we were driving out there I wanted to be moving fast well in the company be incorporated by the time we arrived so I called one of two friends I had in Seattle and said could you recommend a lawyer who could help me you know get a company incorporated and open bank accounts for the company and do some stuff so I can hit the ground running my friend said sure but so amazon.com was incorporated by my friends divorce lawyer not like some big high-powered attorney and but anyway I finally talked to shell into you know coming and joining this intrepid adventurer and but before he had moved from the Bay Area up to Seattle he was you know getting ready to do that move I called him and I said so how tall do you want your desk to be and because I'm making him out of doors I bought at Home Depot and four by fours and he was like 10 seconds of silence on the line and I was thinking to myself oh great now he realizes what a two-bit operation this really is and he's gonna change his mind he eventually answered the question and about a year later I said shell you're important I asked you I'll tell you on your death to be a year ago and you were quiet for like 10 seconds were you like reconsidering your decision he said no I was just considering how tall I wanted my desk to be and so we got the software put together took about a year to get the software put together and to write this astonishingly beautiful HTML and it's really ugly isn't it it is and if you'll see I think the next slide coming up know that well this is our next row this is what I don't have a what our website looks like today but you guys know what our website looks like today you will note that has minor improvements like a search box on the whole on the on the on the main page here you had to actually click through to search you had to click on that link that says 1 million titles so things have gotten better by the way since people don't know this but our number one the number one thing we've invested in amazon.com has been technology and in fact since 1997 we have in the last five years we've spent 800 million dollars on technology so when you look at what what it is today I look back at this that's that's what eight hundred million dollars will buy and we've only spent three hundred million dollars on five million square feet of fulfillment centers for example and only six hundred million dollars on marketing so technology has been our single biggest investment and the company basically runs on computer science and it's either that or it's just my bias since that's my background but at any rate that's what we do is we really focus on trying to differentiate our experience with the with the engineering and the algorithms that we put into it so anyway we we got this company just about ready to launch and we're looking at it when looking at this little tiny fulfillment center we were here by then in this building we had a basement fulfillment center to call it a fulfillment center what's very grand there's a lot of puffery it was four hundred square feet which is about the size of a one car garage and we launched this business and we're looking at it and we didn't if anybody would order from us we really didn't and in fact one of the software engineers looked at this little space and he said I can't figure out whether this is incredibly optimistic or hopelessly pathetic and indeed we didn't know we had no idea whether anybody would want to buy things in this way the business plan called for generating sales very slowly as customers changed their attitudes the original business plan which I thought was very optimistic at the time called for amazon.com to generate 70 million dollars in sales in the year 2001 we actually generated an excess of 3 billion dollars in sales in the year 2001 and we knew by the way that we were really onto something in those first 30 days in the first 30 days we got orders from all 50 states in 45 different countries with not a dollar of advertising just all word-of-mouth the first year we didn't spend anything on advertising all of it was word-of-mouth and that really forged the company as a that we were gonna focus on customer experience cuz we saw the power of word-of-mouth so so very very clearly in those early days there's something great about word of mouth online by the way and feedback from customers which is that email turns off the politeness gene in the human being it's wonderful so people actually tell you what they really think usually in all caps and so we get lots of info and feedback from our customers and we try to try to use it well well in that first few months we got one order it was literally in the first six months again where exactly was an order from Bulgaria and I didn't even know they had internet access in 1995 in Bulgaria but they did and this person did not pay with a credit card they paid with cash yeah which is a method of payment we discouraged and they had taken to crisp $100 bills and folding them up in a tiny little package and they took those $100 bills and put them inside a floppy disk they opened the little metal door and slipped them in the floppy disk and then they mailed us the floppy disk and on the note on the floppy disk was a little note and it said the money is inside the floppy disk and then it went on to say the customs inspectors steal the money but they don't read English and so we opened up the little door sure enough there was $200 in there they had written their order number on it so we could associate it with and they shipped him their books and always well our vision starting about it has changed a little bit but it hasn't changed in about four years about four years ago we came up with the notion that we wanted to be a place build a place where people can come to find and discover anything that they might want to buy on a line literally Universal selection anything with the capital a we had already been on the path of trying to be what we call Earth's most customer centric company and we mean three things you have a very precise definition for what customer centricity is at amazon.com and it means listen invent and personalise you and the basic idea is you can't really run a business of any kind I think in most people's opinion if you're not going to listen to your customers that's sort of fundamental but invention is equally important an invention that goes far beyond listening to customers because oftentimes customers don't really know what it is that they want it's your job to invent on their behalf and we've done a tremendous amount of invention and innovation of different kinds in the company I'll show you a little bit more about that when I give you the live live demo and then finally personalization the notion of customer centricity is to build a place where each individual customer has his or her own website and to do that in large part not by explicit information that the customer provides but through implicit observation of their behavior on the website so that we personalize that experience based on the actions that they take and we've put a lot of effort on into that we've been working on that 404 all but for the first year of the company's operations so for I guess about six years now it's where and a lot of some of our smartest people have worked on that they're a bunch of areas where we're we're good algorithms and good heuristics and and so on turn out to be very important for driving our customer experience and that kind of discovery accelerating discovery for customers which is what personalization is largely all about is critical when you use this when you decide to be selection intensive so we have over twenty mean it's not 28 million or something like that items in our catalog we have in across all categories and and it's not when you have so many items you have to work hard to build tools that help customers find products but you also have to work hard and something which is a little less intuitive and actually a little technically even more challenging which is to help products find customers so we reverse the sense of that we've put a lot of energy into helping products find customers and it's paid off this is the basic thing that we work on every day so what we want to do is have the world's best customer experience and there are a few things that feed into that massive selection which requires great discovery tools and low prices those are the two biggest things low prices and both of those things the heart of both those things is technology it's very interesting because I often get asked and it get made it's just my bias because you know I'm an I'm an X gig but how our business differs from traditional retail and the answer to that question I happen to know the guy who does real estate for Starbucks and believe me this is this is like the guy who has the metaphorical corner office I mean that business is all about real estate this is why the old retail saw the three most important things in retail are location location location and in our business it's technology technology technology and one of the great things about technology as opposed to real estate is that real estate on average economists will tell you we'll get more expensive at the rate of inflation so real estate gets more expensive every year and that's the core ingredient of physical physical retailing whereas our core ingredients get to follow Moore's law so you know diskspace getting twice as cheap every 12 months bandwidth getting twice as cheap about at that rate CPU every 18 months this is a tremendous advantage and as those things get cheaper and cheaper and cheaper in and of themselves they don't offer any real benefit to customers but what we get to do is take all that compute power and layer innovation on top of it to really change the customer experience game and that's a lot of fun we have invested enough in our software that we've become a more insert in a big part of what we've done have become a more classic e-commerce come more classic technology company - or we actually now service make available as a as an application service provider using an ASP model our e-commerce technology to others and we're also I've done and I think Rob is going to give you guys a quick demo of our Web Services stuff in a minute - we still doing that competition oh good this is what instead of our four hundred square foot fulfillment center we now have five million square feet of full of fulfillment space it turns out that operations research and all the logistic stuff we do here is astonishingly similar when I started learning about these techniques I was like wait a second this is computer science they just have different words for everything and we do one of the things that's very interesting is in some of these fulfillment centers we have two million different items in the fulfillment center and you can order any two of those and somehow we have to marry those two items together and get them into a single box and ship them to you that sortation is very challenging and there are many pieces of it that are challenging to do it in an efficient way all the most obvious things you would try first turn out they they work but they don't work efficiently the cost would be too high and what you end up doing is you know is using very sophisticated algorithms to calculate an optimal path through the fulfillment center and you pick a bunch of customer orders at once and just you and and and a decentralized swarm of people actually pick all these things and then they have to get married in sort of a two-stage sort later but the the picking algorithms are like a dynamic Traveling Salesman problem where the cities occasionally move and disappear it's it's like unbelievably it's it's it's it's actually very fun and I didn't know when I started the business I I thought the front end of the business would be very fun and intellectually stimulating and had envisioned a lot of the kind of personalization features that we could build online but one of the things that I had not known and not envisioned is how intellectually stimulating inventory optimization and pick path optimization and these kinds of things would be - this is you know kind of like airline routing and so on it's it's a very interesting problem we wrote our own software for that so it's a great question why don't what in fact we we bought off the shelf inventory optimization soft there is no good picking software for the two million items so that we built from the very beginning the we did buy inventory optimization software and it didn't work very well for such a large skew base one of the things that we found is that our operations are significantly different from most companies for a variety of reasons your typical mail-order company most of these things are made for like you know warehouse management system and made for mail order companies and you can buy those things off the shelf but they typically have a different number of SKUs measured in the tens of thousands and a lot of things change when you scale that up to millions of SKUs instead of tens of thousands of SKUs and so we had to do a lot of ourself we bought we early on we we bought some personalization software that also didn't work and one of the things that's interesting about personalization and that's where we've put a huge amount of energy and attention is that well first of all has to work at scale with millions of customers and in and you know on peak days we will or you know take orders for and ship two million items and and and so it's very very sensitive to doing that kind of do and we do for unrecognized customers we do real-time personalization based on their clickstream and so there really is nobody doing that the other thing is that we can we can do active experiments this is like a really big point that most people miss but it's one thing to take a huge data set and sort of do backward looking data mining on it and you can actually find some interesting things that way but what's but what you can't get from that is anything that has a feedback loop where if you change something changing that thing is it's gonna change the customer behavior and so one of the most fascinating kind of tools we have at our disposal is the ability to do active experiments it's you know it's kind of this huge laboratory with literally you know 30 million visitors a month coming through and every time we make a little change on the website we roll that change out to a randomly selected group of people and hold constant the rest so we make you know we call it a B testing we may do half the people C version a and half C version B they're seeing it simultaneously so we're holding all the factors constant and we can actually tell so if we develop a new personalization algorithm that you know some smart person inside the company comes up with and they're really jazz and they think it's better we never have to argue about that we said well let's try it and you know a few hours later we will know whether it's better and that is that is huge because this kinds of arguments are so you know useless of energy-draining come on up I'm gonna introduce this this is this is Robert Rock you're an alum and Roberts been with us for quite a while how have you been three and a half years we've been to Japan together we've been through several trials together all men all good things you've worked on our wireless business and now you're doing a lot with web services and let me turn it over to you so again my name is Robert Frederick I manage Amazon comms Web Services program how many of you of heard of web services or our doing things with web services okay great Amazon Web Services is something that we launched this year in July we found that by providing an API a solid SDK or an API to our broad website developers as well as our associates that we would be able to get more and more users to interact with our features all over the Internet rather than just that Amazon's website so what we were thinking of doing is we had we had three goals we wanted to define a product and that product is the ability to create your own store on your own website to expose Amazon services and features on I believe it at this point in time is nine hundred thousand websites that interact with us on a day-by-day basis so what we were doing is we were defining a capability to interact get search results interact with our similarities engine to find products and to build those storefronts on external websites all over the internet in various countries all through exposing a solid API and SDK so again we had a market of 900,000 websites we have developers who wanted to build soap packages scripts Perl PHP you name it dotnet applications and we wanted to provide those applications to each one of these website owners so that they can use our tools and interact with us via soap or via XML over HTTP in a very easy and viable solution so the economic model for this as well or why are we doing this why would Amazon will ever want to expose the services that we have on our site to all of these other web sites on on the Internet and the reason is very simple the more people that are interacting with our the more items that will potentially sell and then we also have the capability to provide these products on their website in a way that you know doesn't have to make them recreate the wheel it's easy to implement literally like you know anywhere between five minutes to you know a couple of hours and people are able to have a very robust storefront a very robust way of interacting with our products in our selection so what we have what we ended up doing was we came up with a soap interface in an XML over HTTP interface and I just wanted to go through a couple of examples of some of the things that people have built before turning this over to Jeff again so so here is a very simple script it's called Amazon Lite simple search what we did here was we provided a the capability for this one developer to put this little search box over here on his site and if I do a search for dogs sorry so as you'll notice over here on the right and this is kind of cool over here on the right when you type in dogs you will be able to interact with us in a very like quick manner send a request over to us via XML over HTTP or via soap it all depends on the backend system that request is then parsed by our back-end services where it retrieves information from our back-end systems our databases in our search engine and then sends back an XML file to this particular websites script they then use that information and embed it into that existing site the very cool thing down here is it says QuickStart and this is where people will be able to configure their website offering down here by answering a couple of these search questions or these questions right here and then they'll be able to copy the code and just embed it into their site and automagically it now allows them to do search on their own site with having without having to actually build the whole search engine the whole technology you click on that link the link up here for these top search results and you're brought over to an Amazon site or you're brought over to a Amazon Lite site that is powered by that third-party let me go ahead and click on one of the links so this will bring you over to Amazon now the reason or why in the world would anyone want to embed this on their website well we've tied the web services offering to our Associates Program which would allow the developer or the website owner to earn a percentage for every item sold on their site so every time they're bringing a customer over to Amazon they actually are given the opportunity to earn a referral fee anywhere between 5% and 15% of the actual price of the item so there's like a built in model a built in reason why people would want to act want to use this I know I only have a little bit of time so I'm going to go really fast so there's also a very interesting implementation called book search or book watch as soon as it opens up and what this what this site will do when it actually changes is it's going to make a call to Google's web services offering and Amazon's web services offering as well as this other web service offering called book watch plus and it'll go or onfocus sorry and it'll go through and find various topics all on the internet via various people's web logs then it'll call Google and then determine like what the subjects are for these people or the this particular topic and then it'll call Amazon and find products related to those books that people are talking about in their web logs so what you have here is an opportunity to see the item that people are talking about in the web logs to see the news items that are all over the Internet concerning that particular item and then to see Amazon's similarity engine being used to provide products that people are also buying on our website so this is an amalgamation or a combination of three different web services displaying a page so that a customer would have a different user interface that would allow them to find out more information or to have a different customer experience a better customer experience for the things that they're interested in this is the surprised us type of thing this is what we were very interested in seeing people coming up with new ways of consuming our web services offerings to do things that we had not even thought of ourselves and then the last one is my favorite and as Jeff said before well actually this isn't last I'm gonna do one more but as Jeff said before we we worked we we did it something very interesting called an AI mode launched in Japan which was for our device access group that's also a group that I managed and here in the US that we found a company who knows what's playing on any radio station all across the u.s. now they have a web service and but they don't have products and they don't have a business model for selling those products so what did they do they actually came up with another build a link or build a script type of interface where individuals are able to enter a city of frequency choose a design this is actually what would appear on any web site enter in an associate ID that is that idea I was telling you about before that would allow us to track a purchase made and then give credit to that particular associate now the very interesting thing about this is if I click on if I change this to a different station didn't someone give me a Boston station just off the top of their head any chance any Boston station MBR that's a tough one can you nine what was it ninety-two five things I needed a frequency MBR was eighty eight point five or anyone yeah that's right it's been a while this one's taking a little bit of time ah wouldn't you know when you're doing a demo but one of the things that I was trying to say here is that this particular implementation is very unique this is yet again another business model that we had not thought of this one provider this yes net whenever it does come up here this flow web service is not our website exactly this you've got some standards this particular web service offerings and I don't know why it's not coming up right now they came up with a model where they give away their script for free and one out of five links on the page will be they will replace that person's associated ID with their own associate ID that way they're able to just earn that revenue so they get 20% of the exact revenue share and it shared all across the internet where people who have websites are able to embed this on to their web on to their own websites and not even have to worry about paying for it because it's already built into the system sorry here oh it was okay and this is their little detail page where you're able to see songs with clips people who listen to this song also listen to these songs it's been cool I really like that one and then here are if you here are some links over to our website so if you want to see the last few songs that were played you can see the song name and then buy that album from amazon.com very last and I promise it'll be the last is a whole store a whole store different from Amazon built using our web services offering buy it at a 21 year old German student who just wanted to play with it and see what he can build and he was able to build this simple camera shop in a matter of I believe three hours that's what he said and he he did something over here where you're able [Laughter] yeah well anyway what I was gonna show was are you sure that I have a connection here but what he what he has at this point is the ability to compare products this is what this line over here is and if you would press this Go button and then the next page would allow you to see a comparison across all the different products with a check mark and I just wanted to show that hopefully we have we still have a connection so let me bring it back over to Amazon so one of the really cool things about doing web services for amazon.com okay this is looking better is that we have a competition going on right now for developers students people all across the country to be involved in where we're giving away $5,000 to the best storefront builder developed by a developer or student or whatever the case may be I have some more information about that so if anyone afterwards would be interested in learning more about the competition about Amazon Web Services about device access or just anything at all just come and see me thank you thank you all very nice there's one more there's one more thing that I wanted to do and I wanted to recognize some of the people who we have given offers to we are still hiring of course one of the let me just I don't know if everybody everyone on the list is here today but these are some of your classmates who have job opportunities with us and hopefully they'll join Allen McConnell is he here buddy - okay Alex Dinu Joe Hastings Andrew Sutherland Jonathan Bruns Minh and Nicholas Hansen's so I don't know if you know these people but we're very excited about them having the opportunity to come join our company thank you and if you if you win the web developers contest then you can have a Segway it's $5,000 it's perfect all right I'm just going to show you a few things I I want to first I'll show you I've been talking a lot about personalization I can you know it's really some of the most impressive things that our architects and engineers have done actually doesn't demo well because a lot of it has to do with scalability and what happens if a particular service goes down you know how does it feel gracefully and some of these things are very complicated to do but not that interesting to demo but if I come in let me look at my recommendations for a second and I'll tell you a story the very first time now it wasn't the first time but one of the first times that I read that I demoed recommendations live with a live web site I was being too a group of Wall Street analysts and and my number one recommendation number one personal recommendation was the DVD called slave girls beyond infinity which was this is much less embarrassing and you never know what you're gonna get I never know exactly what I'm going to get because it is live and it does change you know from minute to minute but I'm glad to see that today I've gotten you know Apollo and then I can see of course why was I recommended this because I bought cosmos and angle of attack and another book called Apollo so but slavegirls beyond infinity I'll just show you was was a little embarrassing what happened to that laser pointer thing oh here it is do not look directly into me okay and there it is slave girls from beyond infinity just go over here this is pretty interesting it's so you can buy this DVD with assault of the killer bimbos by together it's only 1996 all of that product clustering is done automatically of course it the you come down and you'll notice that you know customers who bought slave girls also bought sorority babes in the slimeball bowl-o-rama and my personal favorite cannibal women in the avocado jungle of death and there's this interesting thing here we we even know when things are part of a series so we say and also they bought DVDs from the barbarian queen series so we can quick through a quick click through and look at the barbarian queen and there's a customer review here it says for Lena Clark's and fans only this is barbarian queen - this is not even barbarian queen it's an incredibly bad film whose only redeeming feature is laying at Clarkson in and out of some skimpy outfits she's completely gorgeous but even she can't save a disaster of a film then here comes the great recommendation by Deathstalker instead in fact were you trying to be funny no I didn't think so no it's not it's very interesting we use a technique called random stow so all the items in the warehouse are stowed randomly there is no rhyme or reason as you yeah they're hashed so as you walk through they all have locations they all have bins so you know where everything is but only the computer knows where everything is as you walk through it'll be you know slave girls from beyond infinity right next to you know dr. Seuss you know they'll be side by side and so it does make walking through does it does is like an incredibly eclectic experience when you go walk through the fulfillment center but these similarities are incredibly hard to compute you know and there's a lot of stuff you have to look at how much information theory there is in each I mean how much information there is an each purchase because the you know you don't want you know if you do the simple straightforward thing then every book just recommends oh and customers who bought this also bought Harry Potter and that's not very interesting so you have to take that into account you have to do a lot of things and we have to compute millions and millions of these we have to do it very rapidly and we like to do it on small machines so we figure out how to do that and only two gigabytes of RAM so there's a lot of there's a lot of cool stuff along that there was a great similarity I remember one time I was looking at this book and it was a book on Zen and I looked down at the list of similarities customers who bought this book on Zen also bought and there were five or six similarities as there usually are and five of the six were other books about Zen and then the sixth one was how to have a clutter free desk and that's the kind of connection that only a computer can make I mean no human editor it's going to say they would always pick six other books on Zen but you know people who like Zen also buy books on how to have a clutter free desk and that comes out let's see and show you one of my favorite features I can show it's just about any product I bought so many things on the site I'm going to show you the Aqsa salad spinner in our kitchen store here is the Aqsa Good Grips salad spinner now this is interesting for a couple of reasons but one is that the customer reviews on this product read like a sexual experience I mean people love the Aqsa sound like people who don't eat salad love the Aqsa salad spinner that's how how potent of a product this is and we do a couple of things here that are interesting one is you can see here it says these friends have purchased these items these are mine these are my of mine and so you know as long as we're here and as long as I've already identified my friends we might as well whenever I stumble across something I get to see which of my friends have bought this thing by the way doing that in real time isn't that easy remember this is a big decentralized system with thousands of web servers and you don't want to go back we have what happens is we as a pub sub technology and when you first come to our website the instant you're recognized we sent we publish a message to all of the different services that subscribe the ones that subscribe to that message recognize it and they start immediately the second you hit our gateway they start filling a high-speed cache of everything that might be interesting that you might want to see on this visit to the store so that's one of the ways that we're able to do you know millions and millions of visits provide this level of personalization and customization and have it not be really really slow the other thing that was very personalized on this page and this is I told you earlier that we very carefully tasked all of our features and you know and when we do this testing we have a very simple objective function it's it's a little more complicated than this but basically our objective function is you know we prefer the thing we're if we're testing feature a against feature B we like the one that generates more sales you know we're pretty simple-minded in that regard and but this what I'm about to show you is one of the few features we've ever left on the website even though with statistical significance it slightly reduces sales and it's called the instant order update so you can see up there at the very top it says Jeff Bezos you purchased this item on May 16th mm so you no more than two years ago I bought the oXXO salad spinner now with salad spinners it doesn't happen very much but with music CDs and a whole bunch of other products people forget they bought them and they buy them a second time so they're like you know I don't remember if they buy that Madonna CD and then they're like they get it home or maybe they never notice it but they buy it accidentally a second time and all of us do this I mean unless you're like incredibly organized or something it's like hard to avoid this if you do so you know you're busy but people buy books a second time and so on so on and so anyway we thought about this and we thought well what we can do unlike any other store anywhere in the world is every time you come to a page if you've already bought it we'll just tell you then he won't buy it a second time and we even though this slightly reduces sales we get so much positive feedback from this we get so many email messages to customer service where people said I was just about to buy that Enya CD a second time and I didn't thank you it's like they're thanking us because they didn't buy something what's that well you know the amazing thing is most people don't return things very much even you know what they will do because it's a bit of a hassle to return something if they got a second copy of that Enya CD typically they would give it to a friend or something they don't really want it but it's not for an expensive item yes people would return it but people also tend to keep track of their you know they're not going to say oh I already bought a digital camera Oh silly me that's right I have a Segway shoot but for you know a twelve dollar CD people don't put the same amount of attention on it so anyway I just show you this is just amazing to me every time I look at this there are new reviews so this one was only written on November 9th and it's from this guy named Warren holls holls them this is one heck of a spinner I've darn near got enough to centrifuge speeds with just a few easy pumps great for drying off all kinds of fruits and vegetable and it just goes on and on literally there are no let's see it should say down here I don't know how many customer reviews there are but there are hundreds you know they well here the list of 74 so but don't know why they're only 74 I thought there were more but anyway there are 74 customer reviews here look this one is labeled sand from Georgina this is a very good spinner at a good price most of the previous reviews are dead on except they neglect to mention that this spinner does the great job of removing sand from red lettuce and other exotic greens including dandelion radish not to mention spinach I bought it for that reason alone it's some people are it's just fanatical salad spinners look at this it's all five-star reviews oh here's somebody good but questionable durability this is you know there's always one in the crowd here and you're good and we can sort these reviews we use it's not a totally straightforward algorithm but we can basically we're trying to put the reviews if you come back to the main page some of the top reviews are from people who've gotten lots of helpful votes so you can always review vote was this review helpful to you yes or now one of the nice things about voting on amazon.com is we have customer identity because you know everybody who's using our system has a credit card number and so we can kind of keep track of we don't let like somebody stack the deck and vote 100 I mean I guess they could if they have a hundred different credit card numbers and a hundred different accounts but you know it's not really worth stacking the deck for the salad spinner to get so many credit cards well I guess that's true if you make it then why not if you're make it and you're evil we did have we did have it was very funny we've we have we do remove them and I often think we should at least build a graveyard for them somewhere but we do have people who who put up spoof customer reviews and they're hilarious we've had God review the Bible we had we had remember we had a camera which Bronte sister it is the one who really hated Jane Austen it was either Emily or Charlotte does anybody know is it Emily and so we had Emily where there were a few years ago there were like there were like two miniseries and a full-length motion picture based on different Jane Austen novels it was just a few years ago and somebody started put somebody started posting customer reviews pretending to be Emily Bronte and she was just like I hate Jane Austen to miniseries in a full-length motion picture in one year yeah they were it was quite amusing but we take those down and we also had one guy who said his review was incredibly honest it wasn't a spoof but it was also kind of like damning with faint praise his review started out this is the best book my brother ever wrote like golly tough family there unlike all my other brothers books this one might be worth reading so let's see I don't know I might just switch over to Q&A you know once you go in I can show you the page you made this is the real time clickstream stuff it's more useful for customers who we don't recognize but it's you know trying to make sense of these bizarre pages that we've looked at the dog listener slave girls and the oxxo salads we're not giving it much to work with here but but it's recommending the microplane grater zester which by the way I have that product that's a great product and it's recommending assault of the killer bimbos The Dog Whisperer it's also recommending the mini salad and herb spinner so if you need a herb spinner and addition to your salad spinner but I think I will end the demo there and let me see if I can give back I don't really even need this the customer recommendations are based on what we're trying to do is get the probability what we want is to increase the probability that somebody will add an item to their order even if it's an inexpensive item so it actually turns out to work in our interest so what we're trying to do is we look at for personalization we basically calculate you know for every customer a probability that they will what's the most likely product that you will buy and then we try to take out the most obvious you know sort of low information content products like Harry Potter or some because you probably you've already run across that a hundred times in other ways in the supermarket and everywhere else so putting that in front of your something gonna say Oh Harry Potter four is out you know you'd have to be living under a rock so so that is but we try to get the things that are a little bit more obscure products and then and then within that we optimize those algorithms and we're certainly doing it out of our own self-interest I mean we're not you know with there's no question we are in this business to make money but the way that we do it is by trying to help customers make purchase decisions and if you can do that and in small ways as well as I showed you at the instant order of day we've been willing to accept some features that decrease sales but that's because we're long-term oriented we believe that if we give up those small amounts of sales today the people who are reminded not to buy the Enya CD a second time we'll think that we're so great that they'll come back and buy more in the future so it's just you know it's a very simple model which is if you if you if you treat customers right try to do the right thing and actually provide real value great prices we've been very focused on cost structure making the prices so that we can afford to lower prices so we've got you know free shipping on orders over $25 30% off books over $15 great product prices on electronics and DVD players and digital cameras and so on and and we're the only place you can buy the Segway human transporter now I realize this is a little out of reach of your average undergrads budget but nevertheless it's a cool cool product and then you know I just like to offer up my email address I won't answer your email if you email me but I will forward it I do read my emails and if anybody is interested we are always looking for you know the best of the best in terms of engineering talent computer science talent so if you have any interest at all you know you can email me and I'll I'll ship it off right away to the right people anyway with that I will end and I'll open it to questions and however much time we have I'll leave that up to you and and who's got a question yes please something about different customers receiving different prices on the same problem yeah we did a and a B test with best-selling DVDs or what we did was we took what we wanted to determine was the shape of the of price elasticity in best-selling DVDs so what we did was we randomly took we actually subdivided customers into three groups just based on like the last digit of their customer ID so was completely random and we showed some people 30% off some people 35% off and some people 40% off but what happened is the press got this all confused and they believed that there was some devious algorithm where we were trying to charge like people in rich zip codes more like you know trying to which we weren't trying to do we were actually just trying to determine the shape elasticity curve but it was so confusing for people that we decide not to run that kind of experiment so we still do all the a/b tests but we do them with features we don't do them with prices yes things that you know are very simple basics like treat your customers well didn't you develop all these great innovations and I'm wondering is that all internal and customer base or do you benchmark against traditional companies and how do you keep this sort of spirit of innovation alive well that is a truly great question the first of all the way to get a lot of innovation in a company in my opinion is to try and is to work very very hard to reduce the cost of doing experiments because the problem is if experiments are expensive then very few people are going to get to do very few experiments that's just the way it works so if you have a big group of very creative people which we do we have a tremendous number of smart people who love to innovate on behalf of customers and vente news things you've got to set up this kind of like what I was sorry about the a/b testing framework something like that so that if somebody comes up with a new personalization algorithm I think will outperform the current one you don't have to you know move heaven and earth to get it tried so it's important to reduce the cost of experimentation and then to have some objective standards about what's better than what if you can do that then you can do a lot of innovation now some of the things that we experiment with are really hard to do and the cost of experimentation is high so for example we don't really and and with consumer behavior you know it is based with anything totally new it is really very difficult anywhere between very difficult and impossible to guess how consumers are gonna behave how a mass audience is going to do anything in the future is very hard to determine the easiest way to determine it is just to do it and see what happens and I think a lot of companies get that wrong they put too much energy into like you know arguing about how consumers are gonna behave and by the time they're done arguing they could have just done it and see what see what happened and so sometimes like we but we spend a huge amount of effort building customer self-service tools so the customers could cancel their own orders and and we did some research up front in the kind of classic way to try and determine whether customers would want to do this and we got some encouraging results but focus groups and things like that are fraught with peril they're the people people don't know really typically what what they will do if you ask them what you think they'll do they'll try to tell you the truth but they don't really know and so we decided to just take the gamble there was no way to build this in a simple way and sort of do a/b tests that we could think of so we just went ahead and built you know some heavy-duty infrastructure to allow people to cancel their own orders by the way internally we argued about should we let people cancel their own earners you know these are the kinds of things you get you get cold feet about you're like maybe they will cancel their orders and and maybe we'll have no sales you know but in fact of course if you give people more control you know our basic theory was if you give people more control over their environment if they know they can cancel their own orders then maybe the order more and in fact you know it that seems to have been correct that by giving people these these self-service tools we're empowering them and they seem to like that but that was an example where he couldn't really do a low-cost experiment kind of had to build the whole thing in order to test it yes companies were able to look to see from their organization what people were buying and I'm wondering with all of the personalization issues do privacy issues like that I'm not sure what's your I think you're referring to purchase circles which actually has been quite successful so I'm not sure before purchase circles this is a very long time ago there was a news story Xerox look to see what but its employees of are buying somewhat yeah but that's still up there you can do that no no no no Xerox can't look to see what their employees are buying and I and I don't I don't doubt Slashdot reported that with what you're thinking of I know exactly what you're thinking of is that we aggregate the data and so companies can opt out if they want but like I think like four companies have opted out you can go online I could show you and you can see like what are the best selling books from people who you know at mit.edu you can go online and see what are the best selling books at you know from amazon.com employers or from Microsoft employees and so on and so on but we have rules about aggregation and we did from the very beginning so unless the group is at least 250 customers we don't aggregate that we don't put the data out there I think it's 250 it's some large number so the people can't back into who bought what so and it turns out to be really cool data to see you know what these companies are buying what the best seller list is and we do it geographically so you can go in and look at you know see what's the best selling book in Matt and what are the best link books in Massachusetts and we do not only best-selling but we do a special algorithm we call uniquely best-selling to sort of scrape away all the Harry Potter's and so on so you can see what's uniquely best-selling in Massachusetts what's uniquely best-selling in Boston what's uniquely best-selling in Cambridge what's uniquely best-selling at Harvard what's uniquely best-selling at MIT and he can compare those and it's actually kind of fun and then we put the best Arliss wheat then we tie that back to the detail pages so if you're stumbling around you looks find a detail page you may see something like this is the number seven bestseller at MIT edu and it's anyway that I think that's what you're thinking open at that time we didn't make money off books either the question was why why did we expand into things like electronics tools in kitchen since those businesses are not today profitable and the answer is that well that it increases our addressable market size so we can ultimately build a much larger company by entering into those new product categories and and second while it's true that those new categories which are only three years old and by the way electronics tools in kitchen is now a six hundred million dollar business at amazon.com which makes it one of the largest on its on a standalone basis it's probably in the top four or five ecommerce companies anywhere in the world so that business is growing and growing very rapidly and it has it one of the big advantages it has it has a huge audience of early adopters which is why companies like Segway choose us as the exclusive launch platform for products like that because really early adopters are the kind of people who buy stuff like this so it's but that business ultimately will be electronics in particular in my opinion will be our largest most profitable business one day when you start a new business there are certain fixed costs that come along with starting that business and then the business has to reach enough scale so that the profit from the business the variable profit can cover those fixed costs there's no sidestepping that and it's just a simple investment issue the company now in the LAT in a trailing 12 month basis generated 120 million dollars in free cash flow after interest expense so it's you know and and really were just at break-even the company is doing very well financially yeah in the back how do you deal with the new technologies going out there and how do you compete against that despite the fact that you guys already invest a lot internally well really that is them I mean the good news from a technology investment point of view is that compared to most of our competitors well actually really I guess compared to all of our competitors we're spending so much more you probably have to add up the next ten eat Ahlers together to get anywhere near our technology spending budget so at 200 million dollars a year you know most of our competitors don't have 200 million dollars in sales so they can't spend 200 you know we have 4 billion dollars in sales so 200 million dollars a year is just 5% of sales you know at Barnes and noble.com 200 million dollars would be half of sales so that's a a significant advantage for us as we continue to build out we want the gap between our customer experience and those of our competitors actually widened you know with with these kinds of personalization features and and there's lots of stuff that I can't talk about on the drawing board that we think is gonna make the experience you know much much better in the future well we the question was of that 200 million how much of it is R&D I never know exactly how to answer that question I mean oh as opposed to disk drives almost all of it is I mean there's very little the good news is you know we're a total Linux platform and you know so it's very cheap it's all commodity hardware we have a few big database servers but you know this stuff has gotten so cheap I don't know exactly what the fraction is but a lot of a big fraction of that expense is as people expense it's you know smart people yeah well there are there are two groups of competitors there that our big competition is in the physical world so if you in there and we online is still so small it's let you know retail sales worldwide or five trillion dollar market and so an online is you know less than a percent of that it's very very tiny today so you know you don't want to take sales away from online competitors just because there isn't that much sales to take away so you look at physical all the sales are in the physical world so that's where the dollars are that's where you need to focus your primary competitive energy the the you look at online companies not because you want to get their sales but because sometimes you might be able to learn from them I mean they're watching customers to hopefully and and and so you need to see what they're doing how are they innovating what are they thinking of what kind of lessons can we learn from that incorporated into doing things the way we do things rarely by the way have I seen it successful online to do direct copies of things usually you get some inspiration from something but then you have to kind of make it your own and and and in you make it be part of doing things the way we do things for our customers so in in and then once you're in the physical world the big competitors there's a different one for every category so we would look at you know Best Buy in electronics as our top competitor Best Buy in the physical world Barnes and Noble in books and you can go right down the line in every category and you can it's usually the kind of the the selection intensive you know superstore type competitor defined by category information that example I know many product designers you know for baby toys you know go to Amazon they look up their customers you figure out how people feel about it and I'm wondering so now I were to start a web use the web services from Amazon for trying to resell it repackage it you know sort of how did you guys what's that interplay the answer is right now you have to try and use the web services as part of our Associates Program so the there's a contract you sign when you do when you sign up to get a developer token there's some there's some rules and one of the things you cannot do is download the data and repackage it for the sort of purposes that you're talking about and you can't cache the data there's there's several things that that you know because the data is valuable we don't want you to just to write a program that like sucks all the data out and then you know then it's your data that can't be but you do but if you're you know playing by the rules of being an associate you can invent a lot of cool stuff you know you own that stuff you read the contract you know you own your application and the things you build on top of it we just reserve the right to you know kind of take the data back yeah do some more down here in front desk top desktop yes what's what's uniquely interesting about that world that makes so the question is is the mobile wireless world interesting presentation logic how to write the I think the answer is that that world is not very interesting today with the exception of Japan where for whatever reason it's worked a lot better and people are willing to you know the real problem is the devices so the devices have very small input device a versatile output device and as we get to see better web pads that you know like this is a great device the BlackBerry for doing email is this is not my most value but I give him my cell phone before I would give up my blackberry and that that you will see devices with this sort of form factor with decent input devices that have you know decent color screens with enough resolution so you can actually look at enough data to actually do something on the web right now you know if you're using your cell phone to do things on the web you know you're like you know hitting the three key three times to get a particular I mean it's just it's just bad it's a bad customer experience it's the best you can do perhaps with that physical device but that stuff will get better and that's a pretty big deal the thing that's a bigger deal in the shorter term is more people getting instant on at home it's not even the extra bandwidth that you get at home it's the fact that you can leave your computer on I think the thing that will happen next is that people will have multiple computers in their home so I already see that starting to happen we know one of our best selling electronics products for years has been wireless routers a lot of people installing 802 11 networks in their home and it's and it completely changes your behavior when you get you know two or three or four computers in your house instead of sort of one in the office or one in the bedroom or whatever it is so when you look at your average household you know you're gonna start to see that the number of computers just like with phones there was a time when people said you know why would anybody ever have two phones in their house that's sort of ridiculous absurd luxury and now people put a phone in every bedroom they put them in their bathrooms they put them in their kitchens they you know they put them everywhere in the kitchen is actually the key room so I put a computer in my kitchen and it doubled my amazon.com purchases I I strongly recommend you need an instant-on computer in your kitchen that will be a big deal not serve behavior when you're in the kitchen different enough that into consideration and making even company by making what recommendation recommendations I hadn't thought about that recommendations by room right we need to be able to impute your room though otherwise we'd have to ask people what room are you in or have you like register your computer this is my kitchen computer will make the Ox of salads been a recommendation in there yes what I think is my position on this is very clear which is I think patents in general are really very good and you know if you look at the kind of the history of the Patent Office and of innovation in this country it's hard to unlink the two and now the one of the things that I think is if you look back historically the rate of change was slower so if you look at the lifetime of a patent it's actually increased a little bit because of some international treaties and it's basically gone from 17 years to 20 years the and for some industries 20 years is a very appropriate lifetime so for example if you are you know manufacturing a new pharmaceutical you probably need you may need 20 years you know you've got to go through ten years of FDA testing and you know there's all this stuff going on you may need a long lifetime to recoup your R&D costs and you need you know you need some exclusive period where all that R&D benefits only you and but when you look at software patents it seems to me that 20 years maybe too long and so that's the point I made and yeah this it's really the same thing but but it's it's I would say that there is no chance of that changing and the reason is when you go look at this and I I did I actually went to Congress and I made my case in front of a few people that this seemed like too long a period and the the problem is this thing is now so well-established not just in the US but via international treaty you know literally hundreds of Nations that have all entered into mutually binding patent treaties and it's very hard to get that whole web of stuff change so that's why it's it's people tend to leave it alone yeah so getting stuff to me that it seems like you know your business will scale your computers will scale but the you know one roadblock between me buying stuff on Amazon all the time she's getting that stuff from Washington to to my home and you know I applaud you all for making that $25 free shipping stuff I think it's amazing I don't know how you do it but where do you think that's gonna go like how do you think those cost scales because well it's a great question the question basically is is there some way to you know dramatically change the delivery part of the business model so that you are and the answer is at certain scale levels maybe so already it's changed a lot because we first started we shipped only out of Seattle you know now we have you know five but you're not gonna have one in every city well why not did you get the hospital well it's all question of scale the question was you're not gonna have one in every city and you probably won't have one in every city but you very well might have one and I mean you need a lot of scale at four billion dollars in scale we've got the right number of fulfillment centers right now but if you take that from 44 billion to 40 billion things would start to change significantly you'd probably put it in New Hampshire [Laughter] well the well that's why you should shop at Amazon you're looking particularly handsome today it's the the it is no the $25 Super Saver Shipping is incredibly expensive so it has to be it's the most expensive price reduction we've ever made and we're still hopeful by the way that we will leave it in place permanently we're gonna make a decision at the end of the year about whether to make it permanent we were at $49 if we can't make $25 permanent we'll go back to 49 but I'm very hopeful we can make the $25 level permanent and the key there is being is be is being very efficient so it the economics can be made to work it's just a question of does it generate enough incremental lift in volume to justify the huge price reduction that it effectively is yes if you were a 2002 college graduate you know with your newly minted engineering degree what are some of the fields you'd be most interested in working in newly minted 2002 college student with an engineering degree what are some of the fields that I would be most interested in working in well you know I think first of all what I would say is you you the best advice and gift somebody is do something that you think is interesting and let the waves catch you you know something that I see people do which I don't think is is a good idea is they try to chase the current thing I saw this so much in 1997 in 1998 1999 with respect to the internet there were you know in 94 and 95 96 97 the mostly the people who were you know the kind of the earlier you get the more true this is but you know the people who were sort of looking at this space and starting companies were genuinely interested in it they thought it was really cool by the time you got to 1999 well doctors were stopping you know screw that doctor thing I'm gonna start a dot-com company you know and so that really made no sense whatsoever you know people were sort of abandoning the things they were genuinely interested in trying to catch a wave and whenever you try to catch a wave if you're almost always too late so it's it's better to just I mean if you're paddling after it you basically have to wait in place and let it come to you and so just to just sprang something that you're passionate about I would definitely on the career advice side the most important thing about your first job out of school is to pick a place where you think you're gonna you're learning per unit time is gonna be very high you can optimize for a bunch of variables and it can get very confusing you know where it's location you know what's the salary going to be and all those things are worth considering they're all important but make sure you pull back after you've done your analysis and say am I also picking the place that I believe my learning per unit time is gonna be the fastest and you learn much more from a best practices company than you do you learn no matter where you go so if you go work for somebody you're like god these people are all crazy this is insane you'll learn a lot of negative lessons that you can use later and you can not do those things but that process of elimination method takes a long time so it's better to try and find a place has really high hiring standards some of the basics especially you know a place that's trying to recruit only the you know the you know is hire A's and B's higher seas and C's higher D's so you've got to hire only A's and you need to pick a company that's doing those kinds of fundamental practices well because then you'll learn so much it'll set you up to do other things so I would say that's probably the best career advice again time for one more question okay yeah the sights are similar yeah our international sights are similar it's a very good question and I'm very pleased to note that you have found the sites to be similar because we we insist all of our software development is done in Seattle we don't allow any software development to be done we have we do have big physical operations and great marketing teams and so on outside of the US but the software is so core and so expensive to do well that you want to have exactly one set of source code that gets used all over the world globally and many companies have you know made the mistake of allowing their you know different sets of source code in different countries to go out of sync and they end up with two completely different systems now they're getting no fixed cost leverage across those two systems they've got two different teams developing two different systems and then they decide one day this is terrible and stupid how do we let this happen and then they want to recombine those systems and the systems by that point have diverged so much the recombining them is very difficult so it's it's very important to have one sort of source code around the world and and and and that's thank you for noticing that we do that we did but the software that runs the the fulfillment centers is also the same all over the world so the key is you know what happens to software software is fascinating in a business because not only do you you know you do all the things that you're trying to do but some stuff what happens is business knowledge gets overtime embedded in the software so little bits and pieces the software ends up knowing this is my opinion again I admit to being sort of computer centric but in my opinion the at the end of the day the software knows more about your business than you do and so it's really important that you only have one such system because it's going to be complicated and it's going to be very interesting and it's gonna have lots of knobs and it's gonna and the reason it has all the snobs because it knows a lot about your business thank you you guys [Applause]