Lessons learnt using Golang and MongoDB

This past quarter I have been experimenting with Golang and this past week in particular, my downtime from work, I have been re-working on my pet project. The pet project is a bot developed through node.js, web sockets and Mongodb. I wanted to replace the database access layer with Golang to leverage the goroutines and some of its other key features. The database access layer earlier developed in node.js had too many nested callback functions, some of which were 4 levels deep. I must say it has been a fun experience and learning unlimited. It was an exercise of migrating 16 node.js functions, approximately 800 lines of code. It took me almost a week to complete the same. Accessing mongodb through node.js is almost a cake walk, as there is abundant documentation and if you know what needs to be done. The only thing that becomes a pain point with node.js is nested callbacks but I must say this can be handled differently as well but the nested callbacks seemed like a natural flow. I have a minimal experience in Golang and have developed modules for data monitoring (referencing the link here https://www.dhirubhai.net/pulse/golang-quirkiness-vishwanathan-raman/ in case it interests you). However developing a rest API engine accessing Mongodb using "quirky" Golang was a tough journey as the documentation around it is very limited but one can find a few relevant articles here and there.

These articles are good to read before you begin your journey into Golang and Mongodb

• https://www.mongodb.com/blog/post/mongodb-go-driver-tutorial-part-1-connecting-using-bson-and-crud-operations
• https://www.mongodb.com/blog/post/quick-start-golang--mongodb--data-aggregation-pipeline

Here is list of my learnings

• Let me begin with the end first as this is critical. Once I was done with all development, I found something weird in the entries of Mongodb, the field names of the document were all lowercase and hence a few of my functions were failing as my original design had camel case notation. This was a huge shock as my struct definition had the right field names. By this time, I had already encountered a lot of quirkiness of Golang so I knew there had to be something of a fix. The following article came to my rescue https://jira.mongodb.org/browse/GODRIVER-607. The fix was to use "struct tags" adjacent to each attribute in struct, something like this `bson:"userName"`. I had to revise my code to include this across the entire codebase.
• Structs play an important role in reading data from Mongodb. If you have a simple or complex document I would suggest one to use an abstract definition to load data before using the actual Struct definition. This makes the job of querying a cake walk. Here the dataLoaded is the abstract definition of data. Once you are able to see the data in dataLoaded you can start defining the Struct. The next point is also critical in defining the right Struct

var dataLoaded []bson.M

curr.All(ctx, &dataLoaded)

• This is something very unique to Golang. The Struct definition for loading data from Mongodb has to follow this outline else you will never realise what's going wrong. I can't place why is it the way it is, yet this is a mandate. If you look carefully I have capitalised the first letter of every attribute of the Struct though the actual field name has the first letter small case. This was the topic in my earlier blog though on a different context. In this particular context it might sound confusing as the field name has a lowercase first letter.

type userlogmodel struct {

  UserName       string   `bson:"userName"`

  ChatGroup      string   `bson:"chatGroup"`

  CurrentLearningStep int64   `bson:"currentLearningStep"`

RecordedOn interface{} `bson:"recordedOn"`

}

• As shared earlier the query construction in Golang is a little different. node.js follows a different approach hence a 1 to 1 conversion is not possible. The CountDocuments is shown as below, here its a typical function call where parameters are the filters and options. In node.js the flow of query construction is more natural where the filter precedes the function. If you are doing some aggregation that can become a little tricky if you have many fields to aggregate on.

     countOptions := options.Count()

    curatedcontentsCollection := client.Database("botI").Collection("curatedcontents")

    filterStage := bson.D{{"contentSubjectArea", tokenStringData.ChatGroup}}

    documentCount, err := curatedcontentsCollection.CountDocuments(context.TODO(), filterStage, countOptions)

• Here is a sample of a not so simple Aggregate, as you can see bson.D follows everywhere we are defining a new operator or having more than one field.

contentqueriesCollection := client.Database("botI").Collection("queries")

filterStage := bson.D{{"$match", bson.D{{"learningContentId", currentLearningStep}, {"contentSubjectAreaSubArea", tokenStringData.ChatGroup}}}}

aggregateStageforContentQueries := bson.D{{"$group", bson.D{{"_id", bson.D{{"chatGroup", "$contentSubjectAreaSubArea"}, {"learningContentId", "$learningContentId"}}}, {"totalQueries", bson.D{{"$sum", 1}}}, {"maxQueryNumber", bson.D{{"$max", "$querynumber"}}}, {"minQueryNumber", bson.D{{"$min", "$querynumber"}}}, {"maxQueryGroupNumber", bson.D{{"$max", "$queryGroup"}}}, {"minQueryGroupNumber", bson.D{{"$min", "$queryGroup"}}}}}}

contentqueriesCurr, err := contentqueriesCollection.Aggregate(context.TODO(), mongo.Pipeline{filterStage, aggregateStageforContentQueries})

If you are venturing into Golang and Mongodb, my best wishes on your adventure. I love the channels and goroutines in Golang, through which I can get the activities done concurrently but it needs to be handled carefully. I have not measured the timing of the performance but Golang is definitely light weight, fun and quirky to work with.