What is Data Management doing in the kitchen?

After work, you arrive home and your significant other asks about your day. It’s an exciting time in the fast-paced world of digital technology and you are happy to share the details. But instead, you hear: A lake? A Lakehouse? Are we going on vacation? Wait, what? No! But how do you explain a data lake, a data Lakehouse, and a data warehouse to someone who doesn’t have a clue what you are talking about? The following might help.

As it's dinner time, you’re hungry and you want to make a meal. You begin by thinking about what food you have in your pantry. These items come in a multitude of different shapes, sizes, and colors. They are stored in packets, wrappers, boxes, cans, cartons, bottles, jars, or unpackaged and stored in baskets. They are pickled, dried, perishable, salted, canned, or preserved. Even though these items have similar attributes and are stored with their kind, each item is uniquely different.

The food items in your pantry make up your data lake, a mixture of unstructured, semi-structured, and structured data. Some data lakes are open and allow you to share your food, others are not.

To prepare your meal, you select the ingredients you need from the pantry and place them on your countertop. This is your workspace for preparing your food, your data warehouse. This is the place where your ingredients are stacked, organized and ready to be used—they are structured.

You are the data Lakehouse. You have the ability to take items from both your pantry or your countertop and transform them into something that can be used and consumed.

You may use a recipe, a paring knife, scales, mixing bowls, and a chopping board. Depending on the recipe, your prepared ingredients may need to be grilled, fried, or boiled. You may need to use a frying pan or several pots and pans.

The recipes, utensils, pots, pans, chopping boards, and your knowledge of how to cook are the metadata, data catalogs, tools, and applications that you need in your data Lakehouse to help you reach your end product—a fully cooked meal.

The appliances in your kitchen may be different to your neighbor’s kitchen. You may have a gas stove, an electric induction cooktop, or a wood burning stove. Each piece of cookware that you use on these appliances has a specific purpose. For example, the pots and pans used on a gas stove cannot be used on an electric induction cooktop. Despite these differences, they all perform the same function—they help you cook your food.

The different appliances are your cloud providers. Azure may be your electric induction cooktop, AWS your gas stove, and Google your wood burning stove. Each cloud provider has their own fully fitted kitchen and they would prefer you to use their whole kitchen to store, prepare and cook your ingredients to make a meal. They would, however, rather you bring your own utensils and use their cooking device than cook elsewhere.

Databricks has a huge pantry with a vast array of food, your data. The data is stored in an open file format making the pantry open to all just as long as you have access. As a Databricks chef, you can select the ingredients from your pantry and prepare your meal on your virtual countertop. Magically, you can return these ingredients to the pantry after your meal has been prepared and cooked. You can also make your meal on any one of the cooktops in the cloud using the same Databricks pots, pans, and cooking methods, no need to buy new ones or retrain yourself if you switch cooktops.