BIG QUERY- Part 1
SHOAIB SHAIK
?? Published author, AI and Deep learning; fascinated with technology with a deep passion to use technology to make the world a better place. Time, space, Quantum computing are traits I am happy to be involved.
Simple Answer to understand Part 1
what is bigquery
big query is a fully manged service that helps the user and eng to mange and analyze data with the prover of the SQL it include feature of data sharing machien learning geo analysis and BI
Concurrent answer
it help user to ingest store and do a bi to unlock insights across the peta byte of data
How the data is stored is in big query
the data is stored in the columer format that is optimed for analytics on the structure data and semi structure data.
What is structure and semi structure data
the data the are in the format of row or column where it can defined as a schema oriented area. with a proper defination.
semi structure data
With some processes, you can store them in the relation database (it could be very hard for some kind of semi-structured data), but Semi-structured exist to ease space. Example: XML data.?
data is there now what.
Now, we need to process how data needs to be loaded in the BQ
if you want to load a data in the BQ you can use the like Flat file or batch load or a by a streem load.
Now shoaib what is batch or streem. flat files we know it is a CSV file right.
Yes, lets dive what is Batch and streem load.
in simpler term let say you have a multiple files and you want to have BI for all of them so you load all the data in the BQ and process them at ones and will use it.
Streem Load.
That data that is live generate and pushed as the data is generated into the system like 10 sec or 1 sec depends we can also call it a batch it self as the data that are loaded as a block of or chunk of data. Clear ?
领英推荐
Yes…
so shoaib what language you use to get the insight from the BQ
Cool we use Plain SQL to View the data.
So shoaib i understand that the data and language we use, But how will you access it, COOL good question.
We use GCP console
so i know GC what is GCP it is nothing but the Google cloud platform.
so is there any other way we can access?
yes we can use the BQ commad line..notebooks and client libraries as well as BigQuery’s REST API and Remote Procedure Call (RPC) API to transform and manage data. .
We can also use the ODBC connection to connect with any third party app..
cool what do you think shoaib will this be a scalable and cost effective solution
yes man, if you ask me can scale indiviually with data and compute power as well as this two are indepent in the GCP.
ALLL this is ok what are the Component does the BIQ query have.
Good question Lets dive, what component we have.
BQ has DREMEL, COLOSSUS, BORG and JUPITER