October 11, 2017

Lean Startup says "test everything". How do you balance developer work with proper testing?


Hey all,

I have bootstrapped my own company, and so far things are moving along. However I struggle with something that I am not entirely sure how to solve.


The book "Lean Startup" recommends testing every new feature, and the feature's impact on cohorts. For example:

  1. I want to implement social login on my website

  2. My hypothesis is that social login will increase user sign-ups

  3. I can validate the hypothesis over a specified time period (say, two weeks)

  4. If sign-ups as a percent of total visits increases, then the experiment is true

The Problem:

I have one developer who finishes features that I lay out for her. Ideally, I would be getting feedback from users who are providing me with features that I should be building. I want my developer to have consistent work, so I come up with features that I think may be useful, but have no backing behind customer input.

How do I create a system where I can give my developer user-driven features, or do I just have to rely on gut feeling for features to be tested later?

  1. 4

    Keep in mind you'll need a lot of traffic to get a statistically significant result of a test. Without that, it's mostly gut feeling, expert advice, customer interviews, etc.

    1. 1

      This is a great point, chroma. Thank you.

  2. 2

    If you already have users or a mailing list you could set up a "feature voting" tool or use a public trello board to upvote features: https://blog.trello.com/3-ways-to-manage-feature-requests

    With a quick search for "feature voting" i did not find a free/useful tool, but I am sure there are some. Similar to StackOverflow or this one here on github: https://feathub.com/SickRage/SickRage

    If you find a useful tool / plugin, let me know!

    1. 1

      I'm using UserReport in a project I'm working on, just needs a bit of JS and pops up a really nice feedback forum.

      1. 1

        Thanks for the hint ttarik!

    2. 1

      These are great, actionable ways to increase the actual information from customers. I will test out some of the recommendations.

  3. 2

    I've taken a couple of approaches on my side projects. Depending how strongly I feel about a new feature, I won't test it's impact on my KPIs. Instead I'll make sure to set up a custom event in my analytics platform to see if people use it. If adoption is limited, I may kill it in the future, or at least not maintain it.

    For things based around signups, monetization and social sharing, I prefer to test using a proper a/b testing tool, like Optimizely. I think doing pre/post analysis doesn't give you a proper read if usage of your product is spikey. That being said, a/b testing a lot of variations can take some time to get to significance...

    Hope that answered your question.

  4. 2

    As a one man band, I test everything. In the market where I want to launch, my product is - as afr as I know - unique. And people know each other and talk.

    For me, then, there is no option to launch with any untested features .. so it takes me longer than the "I coded this over the weekend" fighter pilots of the Indie Hacker world.

    The balance is : how much time do I have before the financial pain of not launching really, really hurts?

    1. 1

      Hey Thomas, how's it going? What's your product, market, and customer pain point?

      1. 2

        Hi CodeforCash,

        I'd prefer to keep my product to myself for the moment as it is a somewhat complex one and still some months away from market.

        Ny pain point is not so much a customer one as a financial one. Basically, some years ago, I built a website for free for a friend of a friend who had the bailiffs knocking on the door. As a result , I turned his existing business around and, by way of a "thank you" he has agreed to keep me financially afloat for a specific period of time. He can afford to do that from the earnings of his bricks and mortar business (installing intruder alarms in one of the UK's most crime-ridden cities) as a result of the traffic which comes to his website.

        Once I start earning, he will get a cut for life for taking the chance on me so he is not being entirely altruistic. So my particular pain point is to deliver and start earning before that financial lifeline comes to its contracted end.

        1. 1

          OK, pardon me for asking the obvious, but why haven't you used your skills to expand to other crime-ridden territories?

          Anyway, if you're dead set on avoiding doubling down on something that already works, (I was for many years – ashamed to be a software guy, "just a coder")–

          I've been reading this book recently called Nail It and Scale It, which is an entrepreneur's (with the help of an academic's) guide to customer development. Their recommendation is Keep Things Simple.

          In fact there's another book about getting people to remember your ideas that offers the formula SUCCES or Simple Unexpected Concrete Credible Emotional Story.

          The story about the friend and the alarms is such a simple, unexpected, concrete, credible, emotional story...

          1. 1

            Because I built the original website to get the bloke out of a financial mess, nothing more. I was working out my own ideas and he was a useful test bed to learn how to program.

            I can tell you that I look on that programming with horror but it is still good enough for him to run his front and back-end business processes and he is funding me now for the simple reason that I dug him out of a hole by bringing business to him.

            For myself, I want to run a business that I care about and where I am my own master. It has taken me longer but I am, I hope, close to financial and geographical independence as well as personal satisfaction.

            As for keeping things simple, it is a good maxim. What I have found, however, is that often a lot of complex thinking and experimentation has to go into boiling concepts, processes and code into their simplest, purest forms!

            1. 1

              Top 10 London crime hotspots (burglary)

              1. Redbridge IG4

              2. Whetstone, Totteridge, Oakleigh Park N20

              3. Chadwell Heath, Marks Gate, Little Heath, Goodmayes (north), Hainault (South) RM6

              4. Hawksworth, Kirkstall LS5

              5. Clayhall IG5

              6. Mortlake, East Sheen SW14

              7. Dulwich, Dulwich Village, West Dulwich, Tulse Hill (part) SE21

              8. Winchmore Hill, Bush Hill, Grange Park N21

              9. Herne Hill, Tulse Hill (part), Dulwich (part) SE24

              10. West Norwood, Gipsy Hill (part) SE27

              He could use geotargeting and automation for Twitter and Instagram in those areas.

              Cold email business owners in that area.

              Institute a referral program for existing customers.

              What if you valued his business using comparable valuations, and then struck a deal that said if you generate at least $that amount in sales, you will become equity partners?

              OK, I get it – you're mates, bonded for life, but as a rugged and brilliant individual, independence is a beautiful idea.

              I totally agree about the experiments. Just remember it took 10,000 tries to get the lightbulb right. How quickly can you run through experiments? Can you publish the results?

              1. 1

                He could, but he won;t because he lives nowhere near London and his business is very much limited to the areas he can physically service. In his case the internet supports his business but it is not actually his business.

                Having said that, in three years he has gone from Court appearances and having the bailiffs knocking on his door to employing three service engineers and a full time assistant so he's heading the right way!

                By the way, cold emailing is illegal in the UK. Firms can be and are fined very heavily for it.

                My friend is an old-fashioned type so is not really amenable to the sort of ideas you have proposed. Besides, it is his business and livelihood, not mine. I have no passion or real interest in the security alarm business but, in doing him a good turn at a point in his life when he needed it most, he has given me the time and freedom to pursue my own dream.

                Once I have launched and been going for a few months (planned for March / April next year), I 'll write an article on Indie Hackers about how I got there.

                The "experiments" now are more along the lines of "how the hell do I make the software do that rather than A/B testing with feature sets. I have a very clear idea of what I want to produce; the real challenge is to make it as lightweight on client and server ends as I can so that it will scale. Also, as I mentioned before, it is of necessity quite complex so it simply does take time to code and test.

                1. 1

                  Ok, sounds like you know what's up. btw, the legal website I read said that unsolicited b2b email solicitation is legal in UK whereas unsolicited b2c email solicitation is not.

                  1. 1

                    True but you can still fall into traps if, for example, the business is a sole trader or partnership.

                    See https://www.lawdonut.co.uk/business/marketing-and-selling/marketing-and-advertising/your-email-marketing-and-anti-spam-law for a better explanation.

                    In my mate's case, he is B2C. He sells, installs and maintains domestic intruder alarm systems so he would not be able to cold call in the way you suggest. In extremis, theCourts can actually fine you up to £5,000 per unsolicited email! In practice, of course, they would only do that in the case of persistent and deliberate offenders. A recent case saw a company fined £400,000 for doing just that.

                    1. 1

                      I would do the instagram automation anyway (following people in the high crime geo's). SocialGrow is a good tool for that.

                      Anyway, I'm not really buying your 'complexity' as an excuse for launch. Just FYI. And the reason I'm telling you my honest opinion (for whatever that's worth) is just to reciprocate, because you provided me very in depth feedback when I launched my book here nearly 10 months ago (it's now #10 in Kindle Store > Kindle eBooks > Computers & Technology > Programming > Software Design, Testing & Engineering > Software Development)

  5. 1

    I think if you have your developer trying to build it before you've actually ran any tests you're not thinking lean enough.

    I would probably just put the social buttons on your signup page and see how many people engage with them. Have it redirect to a static page that says that the feature is coming soon and please create an account with email and password, log the results and you have validated your hypothesis.

  6. 1

    For each feature that you decide to build, you should have some sort of hypothesis about what kind of impact it's going to have on your business, and where.

    For example, if you have the idea to implement social login, and you think it's going to increase signups by 10%, then you can use your business's LTV economics to understand what the value (Net Present Value) of this investment could be.

    For any other feature idea, whether user driven or gut intuition, you can make an estimate according to this and prioritize the features with highest ROI.

    Of course, in a startup environment you don't have any knowledge of actual ROI, so you have to rely on testing hypotheses and investing based on data

    1. 1

      I guess the main point I wanted to get at is if it's ok for most of the features in the beginning to be off of "gut-feeling" as user feedback has not accumulated just yet?

      1. 1

        Take your gut feelings and build wireframes. Take said wireframes to users and get their feedback. Ask, if they could spend $100 to fund development of features, where would they invest the money? And prioritize those things in the actual product.

      2. 1

        Search Amazon for the book Nail It and Scale It. I think it will be a game changer!

  7. 1

    It always an balance between effort and impact. Adding social login to a website is not a big effort with potentially high impact. (higher visits/sign ups rate).

    You should ask yourself those questions:

    1. What is the effort to implement social login,

    2. What is the minimum I can do to validate the need of social login? What button my users may most need?

    B2B - Google, LinkedIn, Github. B2C - Facebook, Pinterest.

    Pick one, that seems you most suitable. implement it and start get signals. Good luck!