DATA OR TEXT VECTORIZATION

Data Vectorization is something very common now a days specially since the inception of Big Data or Hadoop. It’s not like, it was not in-use in past, but I would say it was not famous.

All of us must have heard about MPP (Massive Parallel Processing) right? But do we know how it work at back end? Data Vectorization is about enabling parallel processing to fetch data.

To understand MPP, let’s refer to four types of instructions to pull data in computer architecture i.e., SISD, SIMD, MISD and MIMD (all explained in separate topic). In this topic, we will be referring to SISD and SIMD only.

Traditional, a computer or machine or a server works in SISD mode i.e., Single Instruction Stream and Single Data Stream means each instruction fetch required data one by one.

In Data Vectorization, we change our approach to MPP mode i.e., computer or machine or server starts working in SIMD mode i.e., Single Instructure and Multiple Data, means if one query is executed and data is residing in multiple data nodes, data from all nodes will be pulled in parallel making computation must faster as compared to SISD.

Data Vectorization has become key component of any data solution specially since Hadoop, No SQL databases and Cloud has surfaced. Now most of the databases are Data Vectorization or MPP enabled.

Question is, why Data Vectorization is so important? Please note, in Big Data era since social media, CCTV, Audio etc., kinds of datasets are also able to produce valuable insights, organizations have started to store and utilize those. But to utilize one must process those. To process such kind of huge datasets, non-data vectorization was not a suitable technical as processing Gigabyte and Terabytes of data in sequential mode would take days to process so SIMD or MPP or Data Vectorization has been a chosen technique which could process data in massively parallel mode making computation 100s of times faster as compared to SISD or non-data vectorization.

Cheers.