Data & Philosophy

At first glance, "data" and "philosophy" might seem like an odd pair. They aren't opposites, nor do they appear to have much in common. But they do share a crucial pursuit: the search for truth!

Let me explain.

ISE233-Operational Data Analysis was an elective I chose last semester while pursuing my MS in Engineering Management at San Jose State University. I was eager to learn how data is collected, stored, analyzed, and used to gain insights that could solve business problems. I imagined diving into data analysis algorithms, writing Python code, and finishing the semester with a solid project. But I discovered there was more to it than just technical skills.

"How do we ensure that our model doesn't overfit?" my professor asked one day, prompting us to think about ways to improve our models' performance.

"Maybe try increasing our sample size?" a classmate quickly suggested. Indeed, that’s one way to reduce the probability of our model memorizing the training dataset, thereby skewing the results on the test dataset. As the discussion continued, my thoughts drifted in a different direction.

Isn't this how life works as well? The training data—our life experiences—teaches us lessons and gets us accustomed to certain patterns, which we often cling to, calling it our comfort zone. When faced with a new challenge—the test data—we tend to rely on our comfort zone, even when the situation demands that we change and adapt. So, how do we break this pattern? How do we avoid overfitting in life? By exposing ourselves to new experiences, expanding our knowledge, adding more data points, gaining fresh perspectives, and using these insights to train our minds to handle challenges differently. I made a note in my book: more data -> more knowledge -> better decisions.

As the weeks went by, our professor guided us through Exploratory Data Analysis (EDA), teaching us to interpret what our data was trying to tell us. What questions should we ask? What insights can we draw? They encouraged us to iterate, learn from our mistakes, and quickly identify and correct errors. The key lesson: no matter what model or parameters we choose, the only way to know what works is to try.

"Truth emerges more readily from error than from confusion!" - Francis Bacon

We often find ourselves suffering from decision paralysis. To do or not to do, do this or do that, what is the best option, what is the right decision?! It doesn't matter! Most decisions in life aren't do or die, we choose one, try our best, learn and repeat no matter what the outcome is.

In another class, we discussed classification algorithms and how they measure different outcomes: True Positive (TP), False Positive (FP), True Negative (TN), and False Negative (FN). Both FP and FN are generally considered undesirable, but I started wondering: which is worse? Is it better to be optimistic and risk a False Positive (assuming something is present when it isn’t) or to be pessimistic and risk a False Negative (assuming something is absent when it is present)?

In most cases, living with regret is more emotionally taxing than dealing with failure. Being pessimistic and assuming the worst can lead to missed opportunities, whether it’s a model failing to diagnose a disease or your mind convincing you that you lack the skills for a task. The opportunity cost is high in both scenarios—a lesson I learned through data analysis.

I was particularly intrigued by the Neural Network algorithm, which is inspired by the human brain. Its structure and function mimic how we receive, process, and act on data. While working on my project, I tested various algorithms on their accuracy and adaptability to new data. Random Forest, Gradient Boosting, Logistic Regression, and SVM performed well, but they weren’t as reliable when new data was added to the training set. The Neural Network model showed more promise, adapting more effectively.

"It is not the strongest of the species that survive, nor the most intelligent, but the one most responsive to change!" - Charles Darwin

In a world where everything changes at lightning speed, we need more adaptive models—both in our systems and in our minds. I made a quick note of this as well.

As the semester came to an end, I was surprised at how much I had learned in just a few months. Yes, data analysis was fascinating, but the parallels I drew with life were even more valuable. Learning doesn’t stop in the classroom, and questions aren’t answered solely in textbooks. As I approached my project deadline, this was my takeaway: whether we’re dealing with data or life, we’re all trying to uncover the hidden truth behind the information, make sense of things, sift through the noise, and find answers—existential or analytical, it’s all the same!