Being adults

True "adulting" happens when you realise after a lengthy period of "getting into your hobby", you stop and start the next difficult cycle of life called "maintaining". Of course, when I first started the plant fiasco as a way to regulate my stress, I wanted to buy and propagate every damn plant I have. I'm no content creator, but I just wanted to understand how plant propagation works. Then I slowly got into succulents after killing many, and terrariums, also after failing multiple times (I got shrooms one attempt!).

Until one day, my friend scolded me, "Your house is in such a mess now with so many plants everywhere! You have to stop, Alex!" That hit me. As I slowed my impulse buying, I start to understand my plants more, and that helps with my care regime. I also can now appreciate how slow/fast certain plants grow their new leaves, like this philodendron MacDowell that I got during a plant fair, from 2 leaves now 6 in less than half a year! I now learn to just SIT AND WATCH, and sort other things out during my downtime (like learning Python while doing my contract job right now).

Why I am sharing this here? This is not Facebook!

Well, I don't have Facebook, and most importantly, this is how I realise I am approaching problems at work. When I started learning Power BI, I just wanted to hoard ALL the damn projects and automate EVERY. SINGLE. PROCESS... Because it's fun to see how fast things get. Yeah, that gives me the bragging rights too, by saying "What took me 4 hours every day is now done in 10 minutes," but I realise this isn't going anywhere in my career. I'll just forever be "the good report builder" and I cannot grow, both technically and in my leadership experience.

So I started diving DEEEEEP into the Vertipaq engine the past 3-4 years, while still researching for better alternatives in writing DAX functions. Here's also where I met really cool people like Brian Julius, Imke Feldmann , Melissa de Korte , Rick de Groot who write passionately about Power Query. Of course, I must not forget Owen Price who writes extensively on Python too. There are just so much that I am learning just by following them here. Sure, you can write 10 steps in Power Query and it would achieve what you need as compared to nesting them. The same holds for Excel, that you can create as many helper functions as "needed" to get to your final output.

BUT IS IT EFFECTIVE & EFFICIENT?

That's the question I always ask myself. Let's take an example from something relatable and simple: reading a CSV file from a folder. Spoiler alert: The function does not promote header in the argument, unlike Excel.Workbook() that allows you to set 'True' in the second argument to promote headers before it's being read in the terminal.

// I prefer this approach over the click and load, because I faced partial load issues in the past

let
// get data from folder before proceed to execute 'content'
content = Csv.Document([Content],[Delimiter=",", Encoding=1252,   QuoteStyle=QuoteStyle.None]),
promoteHeaders = Table.PromoteHeaders(content)
in promoteHeaders        

I used this religiously when I started learning Power Query until:

1. I need to compile A LOT of csv files in the folder. You cannot create 1000 scripts as this is burdening the Vertipaq engine by reading the same folder 1000 times and you eventually give up because your 16GB RAM cannot handle the big process.
2. Not all fields in all files are mandatory. Some would be skipped and missed. This becomes an issue much later when you're transforming data types.

This dilemma can be solved quickly by creating a function that takes the file_name as a parameter, and you need to make your custom function as lean as possible as it would be invoked and applied through every single file.

// Nesting Table.PromoteHeaders() would do the trick here, and you can apply similar approaches of nesting further down your script

let
// get data from folder before proceed to execute 'content'
content = Table.PromoteHeaders(Csv.Document([Content],[Delimiter=",", Encoding=1252,   QuoteStyle=QuoteStyle.None]))
in content        

During my handover in Heineken, my Data Engineer asked why does it matter that you chain (Python's equivalent to nesting in Excel/DAX/Power Query) or not? It only makes the code easier to read, and not really drastically improve the performance. I told him, "That's because I am bored with nothing else to play around with codes. So, I would make my code shorter whenever I get the chance to. There's only so much more you can do with new shiny stuff."

I hope this weekend read helps you to reconsider some parts of your work that you do too. For me, it always goes back to the question, "Does my effort commensurate the impact I am delivering? Can my solution scale while maintaining the same effort put in?" If the question to the first is no (i.e.: I am putting in a lot of effort, but it only impacts a team of 5), and the second is yes (i.e.: the same effort is required to scale this automation to impact an entire department, say Supply Chain, for example), I would seek opportunities to go "out of my role".

So, does your effort commensurate the impact you're delivering? Can it scale while maintaining the same effort you put in?

Recommended reads:

I highly recommend you to read Reza Rad's blog here because it helped me to understand how Web API works, and how parameters work together to form a function in Power Query.

Marco Russo explained in depth on Vertipaq engine here that I had to rewatch multiple times just to get a high level view of what it actually is. I am no expert, but I understand this better now than yesterday. It is a very long YouTube video.