We’ve cut down the second week of lectures to be even shorter and combined them into one podcast.
00:00 – Intro
00:43 – Michael Seibel – How to Plan an MVP
1:25 – The goal of a pre-launch startup
3:10 – Iterating vs pivoting
3:59 – Lean MVP
5:06 – Heavy MVP
6:19 – Launching
7:35 – Learning is easier with an MVP
8:07 – How to build an MVP quickly
10:17 – Adora Cheung – How to Set KPIs and Goals
11:51 – What are the right KPIs to set?
15:50 – Revenue or active users
16:28 – Why choose active users?
18:55 – Biotech or hardtech KPIs
20:25 – Secondary metric
21:25 – What if you haven’t launched?
22:20 – Setting goals
23:47 – How fast should you grow?
25:13 – Defining your own goals
28:40 – Tracking progress
30:27 – Ilya Volodarsky – Analytics for Startups
30:46 – Why analytics?
31:36 – Funnel
32:51 – Collecting data, analytics, and data flow
34:16 – Metric – Signups per week
35:28 – Metric – Retention cohorts
36:30 – Which metric to pick?
36:57 – Have I reached product market fit?
37:27 – Metric – Revenue
37:51 – Dashboards
38:35 – Advisor updates
39:01 – The startup stack
42:13 – Recommendations for the MVP process
Craig Cannon [00:00] – Hey, how’s it going? This is Craig Cannon, and you’re listening to Y Combinator’s podcast. Today’s episode is a recap of the second week of startup school. I’ve cut down the second week of lectures to be even shorter, and combined them into one podcast. First, we’ll have the lecture from Michael Seibel. Michael is CEO, and a partner at YC. His lecture is about how to plan an MVP. Then we’ll hear a lecture from Adora Cheung. Adora is a partner at Y and a co-founder of Home Joy. Her lecture is about how to set KPIs and goals. Last, we’ll have a lecture from Ilya Volodarsky. Ilya is a co-founder of Segment. His lecture is about analytics for startups. All right, here we go.
Michae Seibel [00:43] – My name’s Michael. I work here at Y Combinator. I help run the accelerator. Before that, I did two YC startups, one in 2007, and one in 2012, and today I’m going to talk to you about minimum viable product, so MVP. We always yell at founders to not use jargon yet we have this whole set of stupid startup jargon, and MVP is one of them. When you think about MVP, you should think about something ridiculously simple, this is the first thing you can give to the very first set of users you want to target in order to see if you can deliver any value at all to them. That’s all it is. It’s extremely simple. Okay, so the goal of a pre launch startup is extremely simple. Step one, launch quickly. This is something that’s been part of the YC ethos from the very beginning, and it’s been great advice for 10 years, and it continues to be great advice. If you can walk away from one thing from this presentation, it’s launch something bad quickly. That’s it. Literally the rest of what I’m going to say is basically going to be re summarize versions of that same thing. The second thing that an early stage startup needs to do is get some initial customers, get anyone using your product. You don’t have to have a vision of how you get everyone using it, but just anyone interacting, and seeing if they can get value out of the product. You’d be surprised at how many founder’s journeys, end before a single user has actually interacted with a product they’ve created. It’s very, very common. Please get past this step. It’s extremely important. The next one is talk to your users,
Michae Seibel [02:27] – any of them after you’ve launched this MVP, and get feedback. This is one that’s also extremely common mistake, because most founders in their heads have a idea of what they want to build, and so they kind of have this weird feeling that if I haven’t built the full thing yet, getting feedback on the shitty initial thing is kind of useless. Of course, it’s not going to work. It’s not the full thing. The full thing’s going to take three years, $10 million, a whole team, so feedback on the little thing is useless. The reality is that in some ways the full thing is this really awesome idea in your head that you should keep in your head, but it should be very, very flexible, because it might turn out the full thing that you want to build isn’t what your customers want at all, and last is most important, iterate, and I like to kind of distinguish between iterating, and pivoting. A lot of founders, once they figured out how to build something, fall in love with it, and so if it doesn’t work for a certain set of users, they start thinking, well, I wonder what other problems this thing can solve. Well, the screwdriver is not actually good at screwing in anything, but I wonder what other problems that could solve. And they’re like, “Oh, maybe you can use it to cook. Maybe you can use it to clean,” and it’s like, “No.” The problem was I need to screw something in. The user was like a mechanic, and if your screwdriver doesn’t help the mechanic solve the problem, keep the mechanic, keep the problem. I need to screw something in, fix the fucking screwdriver.
Michae Seibel [03:52] – That’s the thing that’s broken, right? The program thing is not the mechanic, and it’s not the fact that they need to screw something in. In most cases, most people should be building a very lean MVP. So by that we mean you should be able to build it fast in weeks, not months. This can either involve software, or honestly we see startups just start with a landing page, and a spreadsheet, but most startups can start very, very fast. The second, extremely limited functionality. You need to condense down what your user needs, what your initial user needs to a very simple set of things. A lot of times founders want to address all of their users’ problems, and all of their potential users, when in reality they should just focus on a small set of initial users, and their highest order problems, and then ignore the rest until later. You should have a vision of everyone. You should have an MVP, very small. All this is, is a base to iterate from. That’s it. It’s just a starting point. It doesn’t, it’s not special in any way. You just have to start, and so please make sure you don’t feel like your MVP is too special. In very few cases you have to build a heavy MVP. I just invented that term heavy MVP when I made this presentation two days ago. So you know, maybe it becomes a thing. If you’re in an industry with significant regulation, like insurance, or banking. Sometimes drones, although sometimes not. It’s hard to launch. It’s harder to launch. You have to pass through a bunch of regulatory bodies first. If you’re doing hard tech, if you are building rockets,
Michae Seibel [05:36] – it is hard to build a rocket in a couple of weeks. Biotech, it is hard to invent a cancer drug in a couple of weeks. Moonshots, well fill in all the other blanks. It’s hard to bore tunnels in the earth, and have extremely fast vehicles that replace cars in a couple of weeks. So if you’re in that situation please remember that your MVP can start with a simple, simple website that explains what you do. It’s helpful when you talk to people, and interact with people that they can refer back to something. That can be your start, and you can build that simple website in days, not weeks. Maybe your heavy MVPs are faster than your lean MVPs in some weird, strange way. Now I want to talk about launching for a second, because a lot of founders have this misconception about launching. They see big companies launch stuff, and they assume that’s what startups do. In fact, they see companies, they kind of think about like startups, and you know, Facebook’s not really a startup anymore, but they see them getting a lot of press, and getting a lot of buzz, and yada, yada, yada, and they have in their head that that’s what a successful company looks like when they launch. Well, let me ask you this question. How many here, remember the day that Google launched? How about Facebook? Okay, how about Twitter? No, great. It turns out that launches aren’t that special at all, okay? So if you have this magical idea of your magical launch you want to do, throw it away, it’s not that special. The number one thing that’s really important is to get some customers.
Michae Seibel [07:12] – To make people feel better let’s use different terms. How about launch is when you get any customers, and how about like press launch, or press launch, really impressive, is when like people write about things, and it’s all exciting, and you get all this buzz. Let’s push the press launch off, and let’s push the, get any customers launch really, really soon. That’s our goal here. It’s a lot harder to learn from your customers when they don’t have a product they can play with. You know, you can talk to your customer all day, but you have no idea whether the thing you want to build can solve their problem. If you put the thing in front of them, and it doesn’t solve their problem, you know right away, and so all the research in the world is good, but until you can put something in front of people, you have no friggin’ idea whether it’s going to work. Spending all that time on a pitch deck is not as valuable as spending your time building anything that you can give to a customer. Finally, some hacks for building an MVP extremely quickly. First time box your spec. Your spec is the list of stuff you need to build before you launch. Timebox it. Say okay, what happens if I want to launch in three weeks? Okay, well the only things that can be on my specifically are things I can build in three weeks. That makes your life a lot simpler. It allows you to remove all the features you can’t build in three weeks. Second, write your spec. This seems really straight forward, but most people fuck this one up. It’s really easy to change
Michae Seibel [08:41] – what you’re working on before you ever launch it, because you never write it down. You start working on something. You talk to a user, they say, “Oh, I would never use that,” or God forbid you talk to an investor, and they say, “Oh, that could never be a company,” because investors know everything, and so you decided to change what you’re working on, and because you never wrote it down, you don’t even really realize you’re changing it, and so your three week plan turns into a three month plan. If you write shit down, at least you can be honest with yourself that you’re changing your spec all the time. The next one is cut your spec. A week into your kind of three weeks sprint, you probably realized that you added too many things to your spec, and you’re not going to make your deadline. That’s okay. Just cut the stuff that clearly isn’t important, and if there’s no non-important things, start cutting important things. Most of the goal here is just to get anything out in the world. Once you get anything out in the world, momentum to keep anything going is extremely strong. If you don’t have anything out in the world, it’s very easy to just delay, delay, delay, delay, and then last, don’t fall in love with your MVP. So many people fall in love with the vision in their head, and none of the products I showed you before was the initial vision, what it ended up being. So please don’t fall in love with your MVP. It’s just step one in a journey. You wouldn’t fall in love with a paper you wrote in the first grade,
Michae Seibel [10:07] – and like that’s like the level of impact often your MVP has.
Craig Cannon [10:12] – All right, now for Adora’s lecture on how to set KPIs, and goals.
Adora Cheung [10:16] – I am going to be talking about setting your KPIs and goals for early stage startups. I’m going to be pretty pedantic in this lecture, and the reason why is doing this correctly is a necessary condition for starting as successful, or building a successful startup. The acronym KPI stands for Key Performance Indicator. If you Google around for it, there are actually many definitions of what this actually means, but for the purpose of today, for this context, I’m going to define it as a set of quantitative metrics that indicate how healthy your business is doing. This is important, because obviously you should know what state your business is in at all times. Setting the right KPIs, and goals will objectively tell you if you’re doing well, just okay, or bad. Nothing keeps you more grounded, humbled, and realistic about where you are than a bunch of numbers, because if you interpret those numbers correctly, they don’t lie. It’ll also act as a feedback mechanism for whether your current strategy like user acquisition, building new features, launching new features, and so on, and so forth are actually working. If you do something, and things go up, that’s probably good. If you do some things, some things go down, that’s probably bad, and will not only help you prioritize your time, but also course correct. It follows if you do this incorrectly, you’re setting your KPIs and goals incorrectly. You can direct your startup into a bunch of circles, or if you do it for too long, or onto the wrong path, it’ll lead to it’s unnecessary demise.
Adora Cheung [11:51] – What are the right KPIs to set? I’m going to break this down into two pieces. Primary metric, and secondary metrics, and most of today is going to be focused on the primary metric. Every week in startup school, we’ve asked you in a software to fill out to define your primary metric, and then update its current value. By definition, you can only pick one, one prior metric, and it’s the metric if you had to, you’d be willing to bet the whole company on. Why just one metric? It’s a way to focus, and keep things very simple. If there’s a way to get 90% of the job done with just one variable, that’s better than having a bunch of variables that gets, let’s say 91% of the job done. In this case, the job to get done is quickly determining how well your startup is doing. What are the characteristics of a good primary metric? There are four of them. One so your primary metrics should quantify how much value you’re delivering to your customer. That is, you obviously wanting to build something that people want. Now how much do they actually want it? And users often indicate the value through either trading you through money, or time. Revenue is always the best metric. I pay you $100 to use your product, your software, I must at least value that $100. Active users using the product once a week, or a once a day, we call that weekly active user, or daily active user is a weaker, but in other a good decent indication of whether you’re delivering value, or not. The second one here is it your primary metric must capture whether your product has recurring,
Adora Cheung [13:32] – or enduring value to your user, or it should anyway. So for example, on a SAS tool, most SaaS tools use MRR, monthly recurring revenue as their primary metric, I commit to forking over $100 bucks a month, continuously every month, because your product demonstrates to me every month that it has value to me. Another example is if you’re building an online digital daily newspaper, then obviously DAU, daily active user is a good one, because I expect you, I expect to be delivering content to you that is valuable to you every single day. Hopefully he’ll come back every day. The third one here is your primary metric should be a lagging indicator for success. A common trap that founders do to trick themselves is by picking a primary metric. Let’s say something like email signups. Because one, it’s easy to move, but while it may eventually influence revenue, or actual usage, it actually doesn’t represent real value the best. The best indication is when the value has already been delivered, it’s already occurred. When someone has already forked over their time, or money then to use it, then that’s a definition of what a lagging indicator is. If revenue increases it’s because more customers have already paid for the product’s value. Versus a potential customer who came to your site, gave you an email, and maybe they’ll sign up one day, or maybe they’ll use your product one day to buy something, and lastly your primary metric should be usable as a feedback mechanism. That is, it helps you prioritize strategies, and make decisions quickly.
Adora Cheung [15:08] – In a startup, one of the key things to being successful, and getting past product-market fit stage is to iterate very fast, right? While you want it to be a lagging indicator, you also don’t want it to lag too much. For example, a lot of people pick MAU, monthly active user, but this is usually not a great metric, because it takes time to understand the impact of movement. Especially in a startup this early as in your startup, and so many things can happen within a month, and also another reason why I don’t like MAU generally is, because if your user only comes back once a month, they only value something that you’re building once a month, I really questioned actually if you’re solving a real problem. All right, so you may have guessed from me talking about these four characteristics of a primary metric that there are really two primary metrics to pick from. One is either revenue, or active users. Ideally you’re picking revenue, because nothing tells you more about delivering real value than people forking over handing over real hard earned dollars to you, and even better is picking revenue that people keep giving you over, and over over again, like monthly recurring revenue, MRR. It’s the best test for whether people really want what you’re making. All right, so what are reasons why? Kevin, in an earlier lecture said 99% of you should actually use revenue as your primary metric. What are reasons why you should consider active users? So one main one is, because you, because building a large audience
Adora Cheung [16:42] – is actually a prerequisites to monetization. An example of this is if your business model’s advertising based, like a Facebook, or Google, then yeah, you need millions, and millions of users coming back to your site every day before you can actually get brands, and people to buy ads. And so in this case, active users is actually a reasonable proxy for revenue, because eventually when your start up starts making money, it’s usually just revenue’s just a multiple of your active users, and another reason is also, but much, much more, much more rare, is if you have very strong network effects. That is if your like a marketplace that requires tons of users, did you get to get the flywheel going, and grow, then maybe that’s a reason for you to focus on active users today versus revenue, and then just do revenue later down the road. Now that being said, if you’re using active users as a metric, it’s important that you define user appropriately. I hear often I ask, “Okay, how many users do you, or what’s your primary metric?” Active users, how many users do that? I have 100 users. What is users mean in situation? Sometimes to people it means 100 users that just signed up, and gave you an email. Sometimes it means 100 users that signed up, and started using a product, and come back every day for about 10 minutes a day. Which is by far much better than just people just like little dabbling on your site, right? You really need to get that definition correctly, and don’t trick yourself by just saying users, and get having a really easy definition of users.
Adora Cheung [18:14] – Another another example of users where it’s not exactly users, is if you’re in a marketplace, and there are two types of customers. There are two types of users. So a good example is Airbnb. Who are your two users? You have not just the guests, but you also have the hosts. What do you do? How do you pick just one? Well, you pick a value that actually represents them both getting value. In Airbnb’s case it would be nights booked, right? And another example is Uber. Who are your two users there? You have riders and you have drivers, and so an example of a primary metric you could pick there is weekly trips, okay? All right, so there are always exceptions to the rules, and there is one exception in which your primary metric is neither revenue, or active users, and that is if you run a biotech, or a hard tech business, and you’re still trying to figure out whether the science, or tech is actually going to work. Can you actually build a product? And another definition of this is for biotech businesses, it often takes a lot of time, and money to get your first product to market. What’s a founder to do, especially when you have little little funding? So there’s two answers to this. One is if there are no regulatory issues to doing sales pre product, you should actually do the same as everyone else. It should be most likely revenue. Your prior metric should be revenue in the form of paid contracts. LOIs, POCs, Proof of Contracts, proof that if you build it, they will actually come. Now, if you are in a space with regulatory issues, meaning you can’t sell it at all
Adora Cheung [19:56] – without having to go through like FDA, or some kind of body like that, then your product primary metric is actually less quantitative per se, and more of a binary thing. It’s about figuring out the technical milestones that you need to demonstrate to mitigate the risk of whether the drug, or tech is working. If you had to think about experience to prove this out you can ask a question like what are about minimal things I need to do to truly answer the question of whether this works, or not. So one suggestion I have is to select a set of three to five other metrics, secondary metrics to pair it with your primary metric. This gives you a good 360 degree overview of the health of your company. So there are a ton to choose from. So many choose from. What you choose is actually very dependent on your business. Next week we’re going to have two lectures on these sorts of metrics for consumer, I’ll be giving one on consumer startups, and another YC partner will be giving one on B2B companies, and so we’ll deep dive into these metrics next week. The key here though is just picking a few, right? At most, five, three to five. Closer, probably the three. You don’t want to boil the ocean, and pick everything. It’s totally fine to track all this kind of stuff. But it’s really not a good idea to optimize too many at once. Still really just suffer from analysis paralysis. All right, so a common question I have when I say you should, what is your primary metric, or you should have set one is, well, what if I haven’t launched yet? Well obviously, metrics don’t matter if you don’t know
Adora Cheung [21:34] – what the problem you’re solving is, you don’t even know who your customer is yet. You should really just focus on that first. You’d be really putting the cart before the horse by worrying too much about this kind of stuff. That said, once you get to the point where you’re building the product, it’s really good idea to get this down, even if you haven’t launched it. By at least defining your primary metric, you’ll be able to think about who your user really is. You get everyone the same page on who you’re targeting, and even you can hypothesize the metrics, and goals to hypothesize on how you might get your first few users, and trust me, nothing is more motivating than staring down the barrel of zero users, and $0 of revenue for weeks on end. You’re going to get very antsy about launching very quickly, and that’s actually the effect that you want. All right, so I’m going to go into how do you set goals for your primary metric for your KPIs? Paul Graham actually wrote a great essay a few years ago called “Startup Equals Growth” and explains why startups should focus on growth, and I really urge you to go read it. And this section of this lecture draws a lot of insights from it. The goal of your startup is to grow your primary metric. By doing this, it does two things. It proves that you’re making something lots of people want, and second it proves you’re making something that has a possibility of reaching, and serving all those people. Each week, your goal should actually be to set a weekly growth rate. Now we use weekly increments,
Adora Cheung [23:02] – because startups early on need frequent feedback from their users to tweak what they’re doing, but also we use weekly growth rate, because it helps do divide up the progress you need into doable chunks. Say your goal in a couple of months is to get 10,000 daily active users, which requires growing new users, let’s say 10% week over week. To grow 10% this week may amount to actually just getting 100 new users, which is a different problem to solve than trying to get 10,000 new users, right? You should be focusing on what’s directly ahead of you in that week. Do things that don’t scale today, if that’s actually the best way to get those hundred users, and don’t worry about the eventual goal of 10,000 too soon. Naturally, the next question is how fast should I grow? What should this rate actually be? Well, there’s no good formula. There’s no right formula for this, but one angle that we could tackle it from is looking at good startups, and seeing how fast they were growing in the beginning stages of their life. I actually went back, and I looked at the good startups who pitched in recent YC Demo Days. If you think about these startups, they were three months prior, they were all in the phase that you are probably in today, and it turns out the growth rates range anywhere from 20% to 200% month over month, but clustered more closely to 20% to 50% month over month. Which if you back up, back it out, it amounts to about 5% to 10% week over week. Growth is a little hard to grok, but if you look at this chart,
Adora Cheung [24:36] – you’ll see that how small variations in weekly growth rates can make a huge difference on the monthly, and yearly time horizon. You also get the sense that to get big fast, it actually seems doable if you have something people want. On the flip side, if you only manage 1% weekly growth, it’s a sign you haven’t figured out things yet. It doesn’t mean that you have a horrible business. You can run a great small profitable business growing 1% week over week, but it’s not a good sign that you have a startup with a billion dollar potential. You should think about that trade off there, and what you really want out of your business if you’re growing at that rate. That said, the main thing in terms of setting your goals is, is to think for yourself, is to define your own goal based on not what others are doing, but what you think is ambitious, and achievable based on the product you’re building. So you know your users, and business better than everyone else. What does success look like for you, and what does being on track look like to you? Here are some general guidelines when defining a goal. All right, first if you’re solving a real problem in a large market, then that means there’s a ton of latent demand out there. People would use just about anything to use your product, even if it’s half broken, half-baked, or just solves a bit of their problem, which means that startups usually have fast initial growth. That said where you are today matters. So if you have a ton of users, and a ton of revenue, you will probably know that at that volume, as a volume increases
Adora Cheung [26:08] – what you need every week to grow, gets harder over time. So again, most startups, they grew very quickly, and then over some time, the growth rate kind of slows down a little bit. The second one is time to sale. So when you try to set your goal, you need to consider how long it takes to acquire user, and make a sale. So for a consumer startup, generally you have an app, or a website. I show up to it, I look at it, I see if I want it, and then if I do, bam, I buy it, or I sign up for it, and so it’s instantaneous. For an enterprise startup where you’re actually probably going through some red tape, you a bunch of stakeholders that you have to deal with, and like you, you can show up to the company, and they’re not going to be even buy it right away, because you’re maybe not even talking to the right person. So it might take some months to actually get your first sale. So you will have to take that into account. Over time, this time to sale should actually decrease over time. Good enterprise startups, that time to sale goes from months to hopefully days, if not hours, and so it shouldn’t impact your growth rate in the future, but in the near term it actually might. Third is you really want to focus on organic versus paid users, or paid growth in the beginning. Organic means they discover it through word of mouth. Basically you’re not paying for the user. They kind of just maybe you searching for it, and using it themselves. I think in the early days, using paid users is actually cheating growth,
Adora Cheung [27:32] – and you should avoid it as much as possible, and finally, because your startup startups equals growth, you should focus on exponential goals, and not linear goals. All right, so in terms of picking the goals, I think there’s two ways to do it. One you can just pick a growth rate, and then pick a growth rate that you think you can hit, and if you hit it, great, you probably shouldn’t change it. If you’re hitting it consistently to something higher. If you don’t, if you’re not hitting it, then you should be a little bit alarmed, and you should figure out why, and other way to do it as timebox an absolute goal. What I mean by for that is say for the purpose of startup school, at the end of startup school, how many active users, or how much revenue do you want to have? what would it look like? What was something meaningful look like at the end of 10 weeks? Then go back out your weekly growth rate, and then go week to week. Figure out the obstacles, and how you should hit that, hit that weekly goal. In the beginning, if you’re somewhere close to zero users today, often you’ll get something higher. If you do this method, then 5% to 7% week over week. Tracking progress. Metrics and goals obviously don’t mean anything if you don’t leverage them. Use these as a motivational tool. So one way to do this is get a piece of paper, draw a forward looking graph of what the growth you want to hit in the next 10 weeks printed out, and put it everywhere. Put on top of your desk, on the bathroom mirror, put on the fridge, and update once a week. This is in fact what the Airbnb founders
Adora Cheung [29:06] – did in the beginning, and if they hit the numbers, great. If they did not, and that’s all they would talk about, and so I would follow something too like this. Now you want to leverage your prior metric, and goal to help you prioritize your time week over week. Week to week you should be stack ranking all the ideas you have of how to grow it, and make a good guess on what’s going to have the biggest impact for the next week to meeting your goal, and then choose accordingly. Occasionally you won’t hit your goal for the week. We can dream that our growth will be flawless, and look like this, but in reality, in the beginning, it always looks something like this. It’s okay if you don’t hit your goal one, or even two weeks in a row, as long as you understand why. You should be always asking yourself, what is the biggest obstacle in my way of hitting my weekly target? How do I overcome this, and be obsessive of of this? If you don’t know the answer, then the answer is go talk to more users, and don’t spin in circles trying to figure it out yourself. A good startup idea will keep growing at some point, so not hitting your weekly targets week on end will maybe just help inform you you’re not working on the right thing, or even the right idea.
Craig Cannon [30:23] – All right now for Ilya’s lecture on analytics for startups.
Ilya Volodarsky [30:27] – Hi everyone. My name is Ilya. I’m one of the co-founders at Segment, and I’m here to talk to you about how to set up analytics, and the analytics foundation to build your MVP, and to measure these primary, and secondary metrics. This is going to be a little bit more of a tactical guide around what tools are there in the analytics space, in the marketing space. Which one should we actually be using? How do I set them up? So why even focus on the analytics? Obviously primary, and secondary metrics drive the MVP, and product-market fit process, and you’re using that to actually test product-market fit. You’re also using it once you get out of product-market fit strict search to actually focus the team. Maybe there’s going to be an acquisition issue in the company that’s preventing your growth, or maybe the users that you’re getting aren’t as engaged, or maybe you’re having some monetization issues. The funnel actually is a forcing function to understand your business, and where founders should be actually spending their time, and then finally all the way from, you know, two, or three person team to a Google with 1 million employees, you’re actually using metrics to operate, and drive teams. So eventually you have an engineering team, you have a marketing team, and so what goal do you send in front of the marketing team? Use analytics for that. Okay, so to start, always start at the funnel. So we’ll make an example funnel for Netflix, which is a company in the world super familiar with. Any B2B product, or B2C product actually has
Ilya Volodarsky [31:48] – this type of funnel where you acquire a user, you engage a user over a period of time. That loop is called retention, and then finally you monetize the user. And then metrics, both primary, and secondary are performance indicators on top of each stage in the funnel. On top of acquisition you can ask yourself, how many net new users did I get this week versus last week? And what’s my growth rate there? For engagement, you take a cohort of users. So from you know, Sunday to the following Monday you, you have, you know, 16 people sign up, and then you can track that cohort of users week over week, and see what percentage of them are still using the product four weeks later, which is a good example of how to track retention, and then we talk about monetization, which is how much net new revenue did I make this week versus last week? Okay, and then you apply your own custom business funnel to this. If you’re Netflix, we’re all familiar that users sign up for Netflix, then they play videos in a loop. Netflix is obviously a very sticky, watching it a lot, and then finally when the trial runs out, you do subscription, upgrade, and you get access to more content. Okay, so how do you collect data once you have this funnel? So there’s analytics APIs out there. I’m using Segment as an example. You basically want to say a user, user one, two, three in this case, has done user sign up event, and they happen to be an organic user, which means they’re not invited by someone else. Then if you’re Netflix, you might say the user’s video played,
Ilya Volodarsky [33:13] – and then eventually subscription upgraded, and so this is how you instrument your tracking in your mobile app, or your web app. Then you think about event properties. So imagine you’re Netflix, and you’re holding one of these video played events in your hands, and you’re wondering questions about it. So you know, what video is the user actually playing? How long is the video? How far did the person get inside of the video, right? Equivalent, if you’re holding a subscription upgrade event, you’re going to want to derive monetization as a north star metric. So if you’re a subscription business, you want to send your monthly recurring revenue. If you’re a transactional business like ecommerce, or retail, you want to send the actual value of the transaction. Okay, so then you push this out into your web app, your mobile app. Then you start seeing the data come in. You look at the bugger, you see, okay, user signup is here, everything looks good. You add your first analytics tool, we’ll use Amplitude as a good example here, Amplitude and Mixpanel are pretty awesome analytics tools out in the market right now, and then you start seeing data flow inside of one of these analytics tools. This is Amplitude. You can start seeing user signups growing as soon as you launched the real like a mobile, or web app. Okay, so now that you have analytics set up, it’s time to focus on three different metrics. The first one is acquisition metric signups per week. It’s really nice if you’re a B2B business to cut this by the invite type. So you have organic users, which you’re just signing up
Ilya Volodarsky [34:31] – from coming to your website, direct sign up, and then some users are inviting other users. So those are invite type, right? When you’re thinking about growth, it’s really important to think about the organic user in that case, right? Another example why event properties are important. The way you create this is you go to Amplitude, you say event segmentation report, user sign up, next, here’s your graph, right? So it’s as easy as that. Just website, send data, gets to Amplitude, and then you can see the amount of organic users every week, and then if you’re working on the acquisition step as a secondary metric, you can basically say today, 218 users organically signed up in the last week, but by the end of the month, we want that to be at 300, and we’re going to execute projects A, B, and C this month, and they were going to watch this graph every single day on a TV dashboard in our office, or our apartment, wherever we work, and then we’re going to see if our efforts are actually driving this, right? That’s an idea of data-driven operation of a team. You set a metric, and a goal, and then you drive towards that every day. Okay, the second one is retention cohort. So someone recently asked about retention, we’ll talk about that right now. So with retention you want to think cohorts of users. You want to say Monday to Sunday. Let’s say December 10th through December 17th, 16 users signed up, and then you look at those 16 users as they use your product on week zero, which was their sign up week,
Ilya Volodarsky [35:52] – week one, week two, week three, and week four, and the general idea here is like you can convince your mom, or your grandma to use your product once, but even your mom, or your grandma won’t continue to come back, and use your product over time every single week, right? And so if you see users that are addicted that are coming back week over week, that’s a really good sign of product-market fit. This business can see that that December 10th cohort, only 6.25% of those users are still around on week four, and that’s a pretty low amount, right? You’d probably want that to be somewhere between 20, or 30 at least. And so you can set a goal saying, I’m going to talk to a bunch of these target users, and try to figure out you know, why they’re not getting value out of the product, and then make some changes as well. Okay, so what metric do you actually pick? This is taken from one of Gustaf’s slides. Pretty awesome. You think about what value your company is giving to your users. Airbnb gives you value by letting you stay at different rental properties around the world, right? And they want you to do that at least one time a year. Otherwise they consider you a churned user. Equivalently, Facebook gives you value by letting you look at the newsfeed, and you know, connect with your friends, and they want you to do that at least daily, or monthly once, and so when you think about product-market fit, you basically have these two different curves that happen. We have that cohort of 16 users that signed up in one a week, and we track them over time,
Ilya Volodarsky [37:07] – and so what ends up happening is for products that don’t have product-market fit, they end up tending to go to zero, because people just don’t care about the product, right? And that’s definition of product-market fit. For those tools, and the products that do have product-market fit, you’ll see some kind of natural plateau. Don’t mind this axis. It should be somewhere, you know, between 20% and 30% at least. Okay, finally, revenue. This is the primary metric that they want to be thinking about. You’ll use for a subscription business, the subscription upgraded event, you’ll do a property sum over new plan, monthly recurring revenue. You press next, and outcomes your weekly that your revenue graph, and then you could set monthly goals on top of this to make sure you’re growing at the rates that you want to be. Okay, finally if you have a founding team, it’s really good to basically put all this stuff on a dashboard, and then put this dashboard on a TV in your office. It’s incredibly, incredibly important. Basically, this is kind of the difference between being a data-driven team, and not a data-driven team. A lot of founders actually set up their analytics, but then don’t look at them ever again, because it can be painful, right? While a data-driven team will put it on a TV, and talk about projects, talk about what this project is actually changing the metrics that they’re trying to drive, and then just completely understand the business every single day, right? And this kind of company, and this kind of founded will actually scale
Ilya Volodarsky [38:25] – to build better high performance companies, because the next team of employees they hire, will also be looking at that same TV dashboard, and be driven off those same metrics. Really important, get a TV. Next, what you want to do is have some kind of social accountability around your metrics. If you have your friends, your parents, your advisors, your investors, package up, how your businesses doing into an email, this helps you synthesize what is actually happening, and then send it out to those advisors, and tell them where the business is struggling, and what your plan is to fix it. This allows the advisors to quickly understand the business, and then respond back with a much more appropriate advice. Cool, and now we’ll go into the startup stack. So these are tools that we recommend that help this kind of tactical process of setting up these metrics. I’m going to talk a little bit about that MVP business workflow that that Michael talked about earlier. Initially you’re building an MVP. Segment built about seven different MVPs before we actually found Segment, and all of those failed, and eventually we found one that worked, and the process of actually building the MVP’s incredibly important. Once you have that little experiment built, you want to enter private Beta, which basically just means getting like 10, 20, 30 customers to actually try this product, and then having very direct lines of communication open with them. What Segment does nowadays, every new product we ship, we opened Slack channels with each one of our customers,
Ilya Volodarsky [39:49] – and we have the product managers sit in those Slack channels, and talk with the customers. For the products that don’t get product-market fit, the customers just stop responding, and we’re asking, asking, asking, they’re not responding, right? And for the products that do have product-market fit, the customers are immediately being like, “Oh, why don’t you have this feature? This is broken. I tried inviting my team, this is not working.” Instead of you kind of pulling at the customer, the customer starts pulling at you. That’s a good feeling of product-market fit. Okay, so at some point the private Beta is going well. You feel like people really care about this, you understand your target customer, then you want to get a larger market segment to use it. That’s the launch that we talked about earlier. Try to get there as quickly as possible, and then a launch is just more users that you get to test product-market fit on, and so if you feel product-market fit there, then you can start scaling the company, right? And you hire salespeople, and you start doing paid marketing, and things like that. Different tools will guide you throughout this process. As you’re building an MVP, and you’re about to give it to the first group of customers install Google Analytics, install Amplitude, Google analytics will tell you who’s coming from the internet to your website, and then Amplitude will tell you which features are they using, how engaged are they with that feature set? Unless you’re able to stand over the shoulders of all of your users 100% of the time,
Ilya Volodarsky [41:04] – analytics is the next best alternative for that. We also install live chat on the page. So either Slack with your customers, or if you can’t do that, then maybe have a live chat available. In the beginning of Segment, customers would paying us, day, and night, and that’s where we got the most valuable feedback from them. So just as many open channels of communication as possible. Next Data Warehouse, this is something that we recommend used to be expensive. It’s no longer expensive today. Basically if you have a non technical co-founder on your team, but want to ask questions around the data, and they’ll always ask the technical co-founder who will have to provide the answers. So data warehouse kind of democratizes the data, not only for the non technical co-founders, but for everyone else in the company that you hire after. Company Dashboards, obviously we should probably move that to the left, email, and a push tools. So as soon as users sign up, you want to send them an email, I’ll talk about that in a second, and then a help desk. At some point you’ll have so much support tickets. If you start feeling product-market fit, and if they’re all going to your Gmail, one founder will just get overwhelmed, and not be able to answer them. You want to have a shared inbox where multiple founders can respond. Okay, this is my recommendation of what tools, are the best for the MVP process. I’ll walk you through them right now. Google Analytics is for understanding what users are coming to your website. Amplitude is for feature analytics.
Ilya Volodarsky [42:28] – Google BigQuery is to democratized data access with a Data Warehouse, which is just a database of data. Mode is the the tool you use on top of that to ask questions on top of BigQuery. Intercom is like a list of all of your customers. They’re really good CRM for a early stage folks. FullStory is for improving product usability, and Customer.io‘s for emailing your customers. Once you get big enough, you can start using Google Ads, Facebook Ads to actually do the paid acquisition.
Craig Cannon [42:56] – All right, thanks for listening. As always, you can find the transcript, and the video at blog.ycombinator.com. If you have a second, it would be awesome to give us a rating, and review wherever you find your podcasts. See you next time.