Manage complexity and find flow

Starting an engineering project from scratch can be challenging. There are so many components that still have to be implemented that it can be difficult to start. Writers refer to this phenomenon as writer's block, but engineers are not immune to it either.?

This week, I encountered such an issue while experimenting with data flywheels. Constructing a successful data flywheel is a complex task. A data flywheel enables you to build a competitive advantage by solving the problems your application is weak at. To achieve that, you need many pieces. However, as the number of pieces grows, keeping them all in memory becomes more complex (one of the pieces you need is hard-negative mining which relates to evaluation; and the reason I wrote about evaluation in my last two posts). It becomes increasingly difficult to retain all the information in your working memory. This overflow results in a situation where the cognitive load exceeds your brain’s capacity.

Often, when I start a new project I get too bogged down in the details. I think of the perfect solution, which, let's face it, is an elusive concept in a constantly evolving landscape.

This morning, I spent all my time thinking about the components needed for building a data flywheel. My thoughts went from frameworks to databases and from evaluation criteria to infrastructure deployment. The list was extensive. Unfortunately, I spend some time on each of these aspects, without making significant progress. The only outcome I had was that I knew what to write about…

To find flow (a state in which you are highly productive and time passes by without you noticing), we need to understand cognitive load a bit more. If you have to keep too many concepts in mind, your brain overflows, you get stressed, can’t focus and lose your productivity.

To overcome this, you have two options:?

1. Reduce the complexity of the concepts?

2. increase your skill

My morning was scattered across all building blocks, where I should have focused on a single component.?

I know there are more effective ways to build, which I normally tend to follow. Here is an approach that works for me (if I apply it).

One principle: done is better than perfect

One process:

1. Isolate the component you want to work on. Don’t think too much about the other pieces just yet. (reduce cognitive load)
2. Timeboxing limits the time you are allowed to spend on a component. If you’re still stuck after that time. (increase skill)
3. Ask help from a peer. (increase skill)
4. Repeat

If applied correctly, you gradually increase your capacity, remove blockers, improve the thing you are working on and grow as a person. Ooh, and if you’re lucky, you might come close to perfect