the software development pendulum

TLDR;

You can take data driven decision making too far. The same way you can take intuition based decisions too far. A balance of the two is needed to make a product or tool feel like a human has solved a problem instead of a machine.

Intro

My entire career I’ve considered myself a developer instead of an engineer or a programmer. Software Engineer always sounded too official, too grown up. Holding on to that title of developer is more important now than ever. A developer is part engineer, part artist. Someone who has decided to try and master a craft of balancing what people want and innovation, with making money either directly or indirectly from the products they create.

Data

Let’s start with data driven decisions. This is something I looked for relentlessly at the beginning of my career because all the cool kids were doing it. Facebook, Twitter, Amazon to name a few were all developing hypothesis based on data they’d collected and reacting quickly with lean features based on that data to prove or disprove those assumptions. They managed to cut out a lot of overhead when developing new features because of this and to be completely honest this sounded fucking awesome. I mean who didn’t want develop something, release it and start measuring its impact in the same day. It sounded like a developers dream.

After finding it however I was disappointed to find that there are some serious downsides to following nothing but data. Someone once said to me “The road to hell is paved with good intentions” and this could not be more true here. Following nothing but data quickly leads to an empty feeling as though all decisions are being made for me. There’s a lack of focus that is put on how the product feels to use instead focusing on how much money it might make.

Intuition

The other direction, intuition based decision making, is equally dangerous when taken too far and is far more common. This is also something I’ve experienced and is used often when there’s either a lack of data available to help inform a decision or the company already has an incredibly large market share. The biggest problem with this method of development is the large amount of time that is wasted seeing features through that haven’t been proven because it’s considered a good idea by “the business” or a competitor is doing something similar.

The Balance

Striking a balance between these two development methods leads to a much more fulfilling job day to day and is equally rewarding knowing that you’re innovating by taking small leaps of faith and then validating those leaps with faith with data. I always refer back to the iPhone whenever we talk about the dangers of leaps of faith, this product was complete overkill in the market but it completely changed the way we interact with mobile phones and the internet. This leap frog product has since been iterated on, and data has been used to help make it better. The interplay of these two concepts is a hard balance to find and will often swing a little in one direction or the other. Keeping both in check will undoubtably result in a much more fulfilling, innovative product.