Exploring the Evolution of the IT Landscape: Changes and Constants

I decided to revisit the IT landscape since my initial article, delving into my textbooks from the Data Warehouse Institute of Technology to gauge their relevance. Over the years, startups have driven innovation amid a pedestal of growth and low interest rates. Looking ahead, I anticipate a shift with big tech taking the lead. Here are a few themes outlining the changes and constants in the IT landscape.

Continuing Strong with a Hint of Change:

Data Warehousing Resilience: The once hotly debated claim that "The Data Warehouse is Dead" has subsided after five years, affirming the resilience of Data Warehousing. Contrary to predictions, the vitality of Data Warehousing remains intact, with platforms like Netsuite's NSAW and Snowflake experiencing robust growth. In the past, the primary focus was on offloading data to enhance application performance. However, the industry now places a stronger emphasis on organizing data centrally, reflecting a growing need for a unified version of the truth. Also, the explosion of vendors directly selling into business units has exacerbated challenges like security, data silos, and integration issues, necessitating a resurgence of proper data governance. Looking forward, there is an anticipation that companies will adopt a collaborative approach involving business users from various departments and IT to safeguard valuable data. Vendors will need to engage IT again to effectively sell into the enterprise.

Demand for Visualization: The Total Addressable Market (TAM) for visualization tools continues to expand, despite the saturation of hundreds of tools in the market. While more vendors are entering the Business Intelligence (BI) space, the challenge persists in achieving meaningful visualization without proper data organization, affecting the quality of results. As the market matures, companies are shifting their focus from tool adoption to prioritizing practical business uses for BI tools. Starting small projects with tangible Return on Investment (ROI) has become the new norm.

I’ve also seen growth in specialized solutions tailored to specific industries. Mid-sized market vendors are increasingly verticalizing their offerings, developing playbooks and tailored solutions to meet the specific requirements of a target industry. David O. Sacks exemplified this trend by releasing DataGrid, BI tool offering specialized formulas and integrations specific to SaaS. I believe this shift towards tools that address industry-specific needs will continue to grow.

Quality Imperative: The mantra of "garbage in, garbage out" resonates even more profoundly with the advent of Machine Learning (ML) and Artificial Intelligence (AI). While the recognition of data's value is not new, the scale of AI/ML applications has heightened the consequences of poor data quality. The industry witnessed a surge in projects initiated without consideration of the order of operations, driven by the pervasive notion that "data is the new oil." Large enterprises, equipped with integration specialists, navigated these challenges more effectively than mid-market counterparts. Executives, in a knee-jerk reaction, greenlit projects without fully contemplating the second-order effects, leading to the use of spreadsheets for accuracy checks, even when BI tools were in place. As a consequence, companies are now prioritizing proper processes, recognizing the pivotal role data quality plays in daily operations.

The Real Shifts:

Compute vs Storage: Isolating storage from compute has emerged as a norm in the Data Warehousing industry, improving operational efficiency and delivering substantial cost savings. This shift has allowed companies to collect more data without the need for implementing a data lake. Most data warehouse companies charge via usage-based billing on the compute side, replacing fixed-rate models of on-premise solutions. This provides flexibility by tying costs to specific usage and ensure vendors are providing consistent value.

Cloud-based data warehouses, exemplified by platforms like Amazon Redshift and Google BigQuery, underscore this trend, enabling organizations to scale storage and processing capabilities independently, optimizing resource utilization and costs based on specific needs. With escalating competition, it's likely that vendors will slash prices for Data Warehousing consumption usage, aiming to encourage companies to retain the majority of their data within their systems.

Data Scientists: The role of data scientists or statisticians with Ph.D.s has evolved significantly, marked by skyrocketing salaries. Initially accessible only to large companies, newer technologies have democratized access, making data scientists or data citizens as a service more readily available for mid-sized businesses. In the near future, individuals capable of translating complex mathematical concepts into practical business use cases for smaller projects with shorter Return on Investment (ROI) periods will be the most valuable.

ETL in the AI/ML Era: AI/ML applications have become mainstream, thanks to improved underlying technologies. While Extract, Transform, Load (ETL) remains relevant, cloud-based tools have considerably enhanced the user-friendliness of data integration. Noteworthy acquisitions, like Salesforce's purchase of Mulesoft, haven't brought about significant changes in the market, as many expected, in my opinion. Looking ahead, AI is expected to materially change integration vendors, making the transformation of data more straightforward. Natural Language Processing (NLP) is envisioned to play a crucial role in cleaning and translating data, with AI identifying mislabels at both the application and database layers. The parallels drawn with technologies like the Oracle Autonomous Database suggest a future where integrations become more seamless and intuitive.

Core Analytics: The core questions of analytics—Descriptive/Diagnostic (What happened and why), Predictive (What could happen), and Prescriptive (What should we do)—remain unchanged. However, there's a notable increase in the emphasis on Prescriptive Analytics, driven by the integration of AI. Take, for instance, a retail scenario: analyzing past sales data (diagnostic analytics) to predict future trends like increased demand during holidays (predictive analytics), and recommending proactive measures such as stocking up on popular items before peak seasons (prescriptive analytics). Stocking up of those items is becoming increasingly done in real time and with much more precision as retailers capture and analyze data such as social, past buying trends, etc.

For another example, let's examine healthcare, recognizing that they might be trailing in adoption due to HIPAA compliance, and are unlikely to implement such practices in the near future. Clinics can analyze past data to understand health issues (diagnostic analytics), predict future trends based on historical data (predictive analytics), and suggest proactive measures like increasing medication stock (prescriptive analytics). The accuracy of medication stock is likely going to improve as AI enables healthcare providers to have algorithms that learn as they capture and analyze external data about the patient.

?

Real-Time Analytics: Back when I worked at Solver, I wondered why companies insisted on having everything in real time, especially for financial data. However, I did recognize the importance of real-time updates for day-to-day operations, such as swiftly analyzing customer feedback data that companies use to make timely improvements to their products and services. That said, it wasn't easy back then because of technology limitations.

Over time, the technological impediments have been overcome, rendering real-time analytics feasible. This means organizations can quickly and accurately make decisions. Thanks to tech like in-memory databases, in-memory processing, parallel processing, and columnar storage, looking up information and analyzing data has become much faster and more efficient.

Ease of Use/Friendly UI: User interfaces, even in technical products like data warehouses, have evolved to become more user-friendly. This democratization of data access allows business users and analysts to access and analyze data without extensive technical expertise. Success stories, such as Airtable, underscore the impact of beautiful, easy-to-use products. What we've been discussing for a while is now becoming a reality with AI – developers are starting to automate themselves out as tools become more user-friendly.

?

One Vendor Approach: The concept of having "one vendor" for support, management, and more has proven effective, as demonstrated in Oracle's sales pitch. The advantage of a single point of contact, or "one throat to choke" in case of issues, has proven advantageous. In 2007, when the initial analysis was conducted, Snowflake and Tableau were considered complementary Data Warehouse and BI tools. However, the landscape has evolved, with vendors now bundling both solutions and emphasizing the ease of dealing with a single vendor, alleviating concerns about integrations.

Summary:

Looking ahead, the dynamic IT landscape shows subtle shifts alongside consistent themes. Centralizing data, enhancing collaboration, and prioritizing efficiency emerge as key trends. Visualization tools evolve toward practical business uses, tailored to specific industries. Quality data governance gains importance and democratization of data science and AI-driven analytics reshape roles. Real-time capabilities and user-friendly interfaces become standard, fueled by AI with a unified, one-vendor approach simplifies operations.

As developers automate, adaptability to change becomes crucial. What's truly valuable now is a holistic grasp of both IT and business, enabling creative adaptation to the ever-evolving landscape.