Data Scientists and ML Engineers Are Luxury Employees

Machine Learning sits at the top of the software?chain

There are many versions of the hierarchical structure within the tech field, but a key takeaway for me is the positioning of AI/Machine Learning at its pinnacle. Achieving success in machine learning requires a sturdy base: comprehensive data pipelines must be built for data collection, transformation, and loading. The data must be meticulously cleaned and stored in orderly, well-documented repositories. Efficient tools are necessary for querying this large volume of data. Additionally, a strong infrastructure is essential to support the computational demands of running algorithms. Frequent, repetitive training and inference cycles necessitate robust pipelines. Orchestrators are crucial for managing these processes, and it’s important to maintain records of past experiments and the data utilized in those instances. While this list is not exhaustive, it highlights the extensive technical foundations required for effective machine learning.

Correspondingly, Google’s research on the hidden technical debt in machine learning systems is particularly enlightening. It reveals that ML models constitute only a small fraction of the complete framework, emphasizing the need for greater focus and development on the supporting non-ML components.

There it is?—?the models, though just a minor component of the overall system, capture significant attention. But what exactly draws data scientists and ML engineers to this field? What do ML practitioners eagerly seek in their professional roles? The answer lies in that small, critical segment of the system. It’s this core element that we, as professionals, are most passionate about and value the most in our careers.

We all want to practice machine?learning

Machine learning and data science are uniquely situated at the intersection of computer science, mathematics, and business acumen. This positioning allows for immense personal and professional growth. Transitioning from software engineering to machine learning, for example, significantly expands one’s scope of work and learning opportunities. It transforms the job into an intellectually stimulating experience where one can create sophisticated systems and fulfill their curiosity by leveraging vast amounts of knowledge and standing on the shoulders of giants in the field.

There’s a practical aspect to our engagement with machine learning. After investing considerable time and energy in mastering complex subjects like statistics, calculus, and various branches of computing and machine learning, it becomes crucial to apply this knowledge regularly. If these skills are not used, they begin to fade. This is a common frustration when tasks assigned at work do not align with the skills we wish to maintain and enhance.

The pace of innovation in machine learning is unparalleled. Each day brings news of breakthroughs, cutting-edge techniques, and essential new resources for professionals. This dynamic environment is exciting but also demanding. If one’s job does not involve working with machine learning, staying current requires dedicating personal time to study and experimentation, which is often unsustainable in the long run. We all need downtime to recharge and connect with our loved ones.

Beyond intellectual satisfaction, machine learning professionals often enjoy lucrative salaries and the potential to significantly impact their organizations. The strategic importance of AI and its projected future growth add to the allure of building a career in this field.

Together, these factors create a strong pull for professionals to immerse themselves in machine learning, despite the challenges that come with the territory.

Getting the?Job

Congratulations, you’ve landed the job and are now officially a data scientist. You’re handed a project with a clear business objective. What’s the next step? You find a source, perhaps a website, to extract the necessary data. You swiftly write a Python script to scrape this data and organize it into a dataframe. Next, you open a Jupyter notebook to begin exploring and visualizing the data to understand its nuances and patterns. Through this process, you run various experiments, tweaking and testing different approaches until you develop a robust model using your preferred techniques and tools. Once satisfied, you serialize the model into a pickle file and deploy it for production use. This quick journey from data extraction to deployment is a classic example of bootstrapping your way into the practical side of machine learning.

Companies claiming to use AI often overshadows the reality of actual AI application

Initially, my data science journey began with an internship at a consulting firm, where my assignments included basic data cleaning and creating simple visualizations using R. It soon became apparent that the more sophisticated techniques I’d studied weren’t going to be applied. After expressing my concerns, I was tasked with modeling the price elasticity of demand, which while interesting, primarily required basic statistical methods, not the advanced machine learning skills I was eager to use. This experience made me question whether such roles truly required a data scientist or just a capable analyst.

My subsequent role at a startup as a data scientist and ML engineer was more aligned with my aspirations. The startup’s innovative culture and resource constraints allowed me to experiment extensively and tackle genuine machine learning challenges. However, in my enthusiasm to apply machine learning, I sometimes overlooked the foundational aspects, which later led to issues with the robustness of my solutions.

Another stint at a different startup further highlighted the mismatch between company claims and reality. Tasked with overhauling a poorly maintained data pipeline critical to the company’s AI claims, I found the work unstimulating and largely focused on data engineering rather than data science. This role clarified how companies often embellish their AI capabilities to enhance their marketability and attract funding, despite the operational reality.

Reflecting on these experiences, I appreciate how they’ve sharpened my ability to assess a company’s true engagement with machine learning versus its public portrayal. This insight is invaluable, not just for my career development but also in advising peers about the realistic expectations they should have when entering the field of data science and AI.

In conclusion, while it’s true that life, much like the tech landscape, is rarely black or white but rather shades of gray, it’s important to proceed with caution. Employing a data scientist can indeed offer versatility and enhance a startup’s appeal to investors, thanks to their broad skill set that intersects various disciplines. However, it’s crucial to recognize the potential downsides. Balancing the immediate benefits of hiring a data scientist against the long-term demands of their skills and the robustness of your technical infrastructure is essential for sustainable growth and innovation.