What do you do if your promotion hinges on being a problem solver in Database Development?

The first step is to get some context. People tend to talk about problems, especially the ones they need solved. Since we're talking about promotions here, it's important to focus on the value for the people who can give you the promotion. If you focus just on the technical problems, it might not be the one your boss wants you to solve first. Let's say your boss complains that the data is not up-to-date when he needs it. In addition, he wants to change some existing business rules and add some new features to the database.

The problems your boss needs solved: - Outdated data - Changing business rules - Adding new features to the database Before you promise anything, you need to understand the data infrastructure architecture: - The data update issue may not be a database problem itself, but could be related to the scheduled refresh, the API, or the system from which you receive the data. - Changing the data in the database might not be the best way to provide your boss with the necessary insights or reports. Consider keeping the raw data untouched and creating views or utilizing BI tools to serve the data. - For quality and maintainability, you might want to keep the raw data and the transformations separate when adding new features.

Recently, I had to solve a database problem involving scalability. My initial approach was to generate Parquet files on the data lake and then use a traditional cloud database with hourly billing. The idea was simple: run the pipeline, consolidate the data, generate the Parquet files, and then push them to the database. This approach can work well, but as data volume increases, partitioning the Parquet files becomes necessary for smooth performance. Additionally, depending on the frequency of user queries, hourly billing for the traditional database might be inefficient. Fortunately, there are serverless options available. These options scale automatically and charge based on the amount of data processed, staying online only on demand.

Before implementing i strongly advise you to read the documentation, search online proofs of concept and try the solutions yourself. Because here things can get tricky. A simple example: The SQL flavor and architecture of the system you chose to implement may create a problem depending on your user and use cases. If your users rely on functions unavailable in your chosen environment, all their queries may need to be refactored. Crazy, huh? They are going to lose hours of work, dashboards are going to break, and reports will stop work... f you don't consider that. So before implementing any solution, it's wise to test and think.