Use SQLs: Harness the Database Processing Power

Off late, the disturbing trend is to treat a database as a place where you can just dump data. Nowadays, developers somewhat do not feel the need of learning SQL and data modelling skills.

Evolution of internet scale applications led to an era of NoSQL databases where the requirement was to store the data as key/value, time-series or document.

Internet scale applications were designed using microservices for which the data stores were mostly non-relational.

There was an era when talks started that object store databases will lead the world and death of relational databases and SQL is certain.

However, soon the industry realized that data hasn't value when it is accessed in silos. To derive value from data, relationship between the data and entities is utmost important. RDBMS kept reinventing itself to host multiple data types and now multi-model databases have emerged. SQL hasn't lost it's relevance and is the most efficient way of deriving value from data. SQLs are not going anywhere soon.

New age databases now have embraced SQL capabilities on JSON and new ways of scaling SQL has taken hold again.

However, still many developers do not have required SQL skillset to harness the power of database processing and many want to treat the database as a key-value store only.

Database servers are generally high end servers and you must harness their processing power to use like high performance computing servers.

When you are moving code outside of database, you are actually putting more load on the database. Application code causes row-by-row and single table processing which results in highly chatty applications and that leads to huge load on the database.

Imagine making millions of lookup calls to be made to the database and the to and fro movement of data, when a simple SQL at DB can do that operation very optimally.

By moving the data processing at database layer you:

·       Get benefit of using SQLs to reduce complexities in application code

·       Database optimizations and query optimizations by database engine

·       Utilize the parallel processing at the databae server, while most of the application processes are single threaded

·       Utilize the database scalability features

·       SQL is fast and powerful in data aggregation/manipulation

·       Utilize pre-computation data aggregations and manipulations if they are queried frequently by users

·       Save the cost while saving the compute by moving the processing at the database layer