Data Engineer internship at The Commons XR

The Commons XR represents an advanced behavioral analytics tool, seamlessly blending AI and XR to produce real-time insights for educational and therapeutic sessions. It delves into the assessment of individual behaviors within group activities, all within the confines of a secure and impartial 3D environment.

Check them out at : www.thecommonsxr.com

As someone who wanted to contribute to the education domain and learn about the intricacies involved in the work, this was a fitting place to be.

During my tenure as a Data Engineer Intern at TCXR, I was immersed in a dynamic environment, dealing with high-velocity, high-veracity, and high-volume data sourced from a Unity AR system.

My primary focus was on conducting Extract, Transform, Load (ETL) processes, data profiling, and validation to identify anomalies and discrepancies within the incoming data streams. The first intent was to familiarize myself with the databases and its contents and how they are shared within the ecosystem. After getting a good understanding of the data, I could figure out the areas where we were losing information and also the areas to focus for building our behaviour analytics models. I worked with a highly effiecient data team that collectively worked towards transforming the legacy data to establish an architecture that would be best suitable for deriving relevant features for the machine learning models.

Next, to manage and optimize this diverse and complex data, I implemented custom SQL and NoSQL queries, tailoring them to the unique characteristics of the Unity AR data. This not only involved data manipulation and management but also included a keen focus on optimizing database performance to ensure seamless operations.

Microsoft Azure played a pivotal role in our data ecosystem, and I utilized Azure Data Factory and Azure Logic Apps to construct robust data flow pipelines. These pipelines were instrumental in handling data migration and merging tasks, contributing to the overall efficiency of the data engineering processes.

Another key aspect of my role was addressing challenges related to data synchronization and security. By executing virtualization and VPN gateways, I was able to design secure communication channels between different subsystems. This not only enhanced the overall security posture but also facilitated seamless data exchange between various components of the system. Additionally, I was also involved in developing logging and monitoring workflows to facilitate downtimes and detecting arising issues in real time. To accomodate the heartbeat and exception alerts for all subsystems, I proposed comprehensive strategies that could be administered by the concerned authorities.

Beyond traditional data engineering tasks, I actively contributed to the development of administrative React Web Apps and automated Python workflows. I was responsible for leveraging Azure DevOps tools and building a resource management system to help with the HR operations. I led the development right from database design to structuring backend services with Azure Management API and Azure Functions. This interdisciplinary approach allowed for a more cohesive and integrated system, where data engineering seamlessly intersected with application development.

Lastly, recognizing the importance of data analysis and decision-making support, I implemented interactive visualization dashboards using Python and PowerBI. These dashboards provided stakeholders with intuitive tools for exploring and interpreting data trends, thereby enhancing the organization's data-driven decision-making capabilities.

In the context of the internship, where time constraints are prevalent, I gained insights into the pivotal role of task prioritization. Recognizing the significance of focusing on essential tasks, I honed my ability to deliver key objectives that wielded the most significant impact. Furthermore, I realized that success in data engineering extends beyond technical proficiency. Skills geared towards navigating ambiguity in requirements and overcoming data challenges are equally indispensable.

In summary, my experience as a Data Engineer Intern was marked by successfully navigating the complexities of Unity AR data, implementing innovative solutions, and actively contributing to the development of a robust and integrated data ecosystem.

Collaboration and adeptness in data discovery emerged as critical elements for success. This underscored the importance of teamwork and adaptability in accomplishing goals within the realm of data engineering. I learnt a lot from the leads Johannan Hjersman , Cody Chan and Sai Ruthvik Thandayam (TSR) and their mentorship and support was of utmost value. I would be forever grateful to Ray Freiwirth for giving me this opportunity and believing in me! I'm happy that I grew multifaceted skills and would continue to leverage this diverse expertise to excel in my future endeavours.