From Upskilling Challenge to Vegas Adventure: A Data Team’s Journey

Hackathons are often the breeding ground for innovation, collaboration and unexpected victories. However, what starts as a simple desire to test skills (and perhaps earn a few bragging rights) can transform into something much greater.??

Such was the case for a team of three data engineers from Skopje, Elena Donev, Slave Tasev and Katerina Stankovska. Here, they share their journey from late-night coding to the bustling streets of Las Vegas.?

The quest begins?

It all began one February evening when we decided to participate in the global online AI hackathon, organised by Microsoft. The hackathon was designed to explore the possibilities of the new Microsoft Fabric analytics platform for developing AI applications called?PIInovators. This cloud-native data solution is developed in Microsoft Fabric, integrating Open AI for document analysis, particularly for detecting Personal Identification Information (PII) in files and images, categorising the data and masking (blurring) the PII data in the images.?

Our motivation for taking part? Initially, it was the allure of badges and the chance to showcase our skills. Little did we know, we were about to embark on an adventure that would lead us to the grand prize.??

Navigating the hackathon??

Over the course of three intense weeks, fuelled by coffee and determination, we crafted a solution from scratch. From brainstorming sessions to coding marathons, we left no stone unturned.??

The first week began as you might expect – we explored various directions and brainstormed potential solutions to real-life challenges. The following week was dedicated to familiarising ourselves with the platform's capabilities, tying in our existing knowledge from the Azure portal services.??

In the third week, it was time to put it all together. We finalised the architecture, developed transformational notebooks and seamlessly integrated them into a lakehouse. We did this following the medallion architecture, preparing the data for analytical use in reports generated by Microsoft’s Power BI data visualiser.??

The final weekend we dedicated to recording a compelling demo that not only showcased the full capabilities of our solution, but also captivated and appealed to our audience.??

We sent off our hackathon entry with hopes high, and we waited.??

From coding to conferences – a prize on the other side of the world??

Only a few days later, we received an email. Disbelief washed over us as we opened it to read that our solution had been selected for the grand prize.??

The good news spread quickly, and we were astounded by the flood of praise and support from our Endava colleagues in the following days, not only from Skopje but from various locations around the world. Excitement quickly turned to logistical challenge, as we realised our prize was tickets to the first ever Microsoft Fabric Community Conference in Las Vegas.??

With no time to delay, we set about obtaining visas, book accommodations, and secure plane tickets. Thanks to the support of our Local Head of Data, Aleksandra Sretenova Matev, and Group Head of Data, Adriana Calomfirescu, within two weeks we found ourselves boarding a flight bound for the entertainment capital of the world.?

Viva Las Vegas – insights and celebrations?

Multiple layovers and long flights later, we arrived, weary but exhilarated. The conference began with engaging keynotes from Microsoft technology leaders before we met Paul De Carlo and Jasmine Greenaway, both hackathon judges, along with other hackathon winners.?

We discussed our experience with the platform, delved into the hackathon process, and shared insights on our project's workflow.

Having the opportunity to meet the PIInnovators team in-person, in Las Vegas, was a surreal experience. From our perspective,?just months before, we had literally invited the entire world to compete in a technical challenge and this was the best of the best out of over 100 teams that submitted projects to the Hackathon judges. Now here they are, standing with us,?sharing their project with Arun Ulag (Head of Azure Data, Microsoft) and discussing plans to productionise the solution in the Azure Marketplace.? When you realise the impact that these challenges have in driving innovation in addition to creating impactful opportunities for participants, it is truly inspiring for all involved.??

Paul DeCarlo , Principal Cloud Advocate, 微软

The PIInnovators team set a high bar! It was an eye-opening opportunity to learn the limitless capabilities of Microsoft Fabric and Microsoft technologies. It was a wonderful experience to share space with others who also share the passion to innovate, no matter where we are in the world. I'm looking?forward to seeing their solution in the marketplace and learning about the impact they are sure to make.

Jasmine Greenaway , Senior Cloud Advocate, 微软 ?

Eager to glean as much as we could from the experience, we split up to attend conference sessions. Some of our key insights included:?

• Transformation of the Microsoft Fabric’s CI/CD experience by supporting data pipelines and data warehouses in Fabric Git integration and deployment pipelines?
• Spark job definition and Spark environment becoming available in Git integration?
• Providing support for both Fabric Git integration APIs as well as Fabric deployment pipelines APIs, enabling to integrate Fabric into these familiar CI/CD tools.?
• Improvements on the Power BI reports and their integration into Fabric’s notebooks. Data modelling with Copilot and enhancing security with Purview.??
• Table clones in synapse data warehouse within Microsoft Fabric now offering automatic transfer over the row-level security (RLS) and dynamic data masking (DDM) from the source to the cloned table almost near-instantaneously.??
• Autotune Query Tuning feature for Apache Spark which leverages historical data from the Spark SQL queries to automatically fine-tune the configurations with the usage of the newest Machine Learning algorithms, ensuring faster execution times and enhanced efficiency.?

The insights gathered by the team during this journey will be shared with data and AI communities through ‘pass-it-on' sessions, facilitating the development of enhanced solutions to better serve our customers. This demonstrates the value we bring to our clients.

Adriana Calomfirescu, EMBA , Group Head of Data Delivery, Endava ?

After the conference agenda, the nights were reserved for exploring the wonders of Las Vegas. The second day we attended a conference party in the company of ‘Elvis Presley’ and ‘Bruno Mars’ and enjoyed their fabulous performances. We spent the rest of our free time in exploring the city’s iconic landmarks, including replicas of the Eiffel Tower, the Colosseum and the great sphinx of Giza. ?

Reflection and renewed motivation??

From hackathon victory to Vegas adventure, we returned home with a renewed sense of motivation. Armed with newfound knowledge and experiences, we were eager to implement our learnings into our own projects.??

Witnessing the PIInovators journey, unfolding from a mere seed of an idea, nurtured true usage of newly available technology and engineering determination, which led to winning the first prize, was priceless confirmation of the engineering practices our teams are deploying in clients’ engagements and communities.

Aleksandra Sretenova , Head of Data, Analysis and PDM, Endava

This journey served as a testament to the power of collaboration, determination and seizing opportunities. From humble beginnings to grand achievements, we’ve been truly inspired by the endless possibilities for those willing to embark on the journey of innovation.?