Building Your First Database - Part 2

Okay, so as we talked before in the first part , we are going to build a very very simple version of CrankDB - the Key Value based database I had created. Today, we will be going to talk about the storage layer of our database.

The (not so) Complex Data Layer

How do you think we build a key value store? Where do we keep the data stored? How did you do an O(1) lookup when solving your CP questions? Maps...!!! Obviously..!

{
    "key1": "value1",
    "key2": "value2"
    ...
}        

As simple as that, I started CrankDB with this same approach..! Checking the source code of CrankDB (it's in Golang), it's as simple as -

type Database struct {
		store map[string]*dbObject
}        

I just defined a Database structure having a variable store which is nothing but a Map we talked about...! The keys are string based and the value of each key is a dbObject type. Let's see what is this dbObject.

Almost everyone might have seen Redis..! It does support a lot of types of data? Well, similarly over here, I tried to push CrankDB as a Key Value based "Document Store", whose ultimate aim was to build a mix of Redis/MongoDB on single source of truth. (probably I'll finish that goal somewhere in future haha)

So if you see, the dbObject is nothing but another structure I have defined -

type dbObject struct {
		key     string
		valType cql.DataType
		value   interface{}
}        

Now if you look at this closely, the value of each key in our Database is kind of like a JSON / dictionary, having multiple things (metadata) attached to it -

• key -- It's a duplicate of the key, probably should remove this.
• value -- This is the actual value of type interface{}. Not this means that in Golang, every other datatype can be represented in interface{}, very similar to Java saying that each Class is a child of Object class.
• valType -- The type of value this key in the database is storing. If you check the cql package, I have defined this as -

const (
		DataType_BOOL   DataType = 0
		DataType_BYTES  DataType = 1
		DataType_INT    DataType = 2
		DataType_LONG   DataType = 3
		DataType_FLOAT  DataType = 4
		DataType_DOUBLE DataType = 5
		DataType_STRING DataType = 6
		DataType_JSON   DataType = 7
	(        

If you see, CrankDB supports all these types of values, where DataType_JSON is the enum and correspondingly DataType (integer value) is 7. This tells me which type of data is stored in each key.

Where is datatype useful?

Datatypes are very useful when you need to support multiple operations in the future such as Redis has increments, MongoDB has groupBy, aggregations, etc. Now you'd be thinking, why do we have a Range query / groupBy query in a Key Value store? This is because that is what I was trying to build -- A Document database with power of Key Value store.

Let's ignore my ambition and focus on the simple Key Value store right now haha.

So, now we know how we are going to store the data in our system. Be a Python Dictionary, or a Golang Map or a Java HashMap, maps are the base of any Key Value store..!

In the next blog, we will see how I started using basic TCP connections (first version) and overtime I switched to gRPC and what why and my decision making to chose the same.!

If you like this blog, Subscribe ? to my newsletter, Like ?? and Share ?? it with your network ??