Digest for September 2022
For a number of years, we provided a monthly digest of all our blog posts, talks and news, to customers and to subscribers of our mailing list. Something happened in 2020/21 (??) that disrupted that routine, but it's time to get back into the habit, especially as we have something to celebrate... ??
News
...Two "TENdjineers"
This month we reach a significant milestone at endjin - two of our employees have just celebrated their 10th anniversary. Congratulations to Mike Larah and Paul Waller! It's amazing how quickly a decade has flown past, and how much fun it's been. Paul joined us just as he became a father for the first time, so every work anniversary also marks a birthday; time flies and children grow up fast! Mike joined as our first apprentice straight out of university and celebrated his Civil Partnership this year.
Big Data London
Ed Freeman, Barry Smart and Eli Gascon attended the Big Data London conference. Expect blog posts about the talks they attended soon.
Programming C# 10 Book, by Ian Griffiths, published by O'Reilly Media, is now available to buy.
Over a year of work has gone into the latest edition of this book, covering all the new language features in C# 9.0 & 10. Fellow endjineer Howard van Rooijen acted as a technical reviewer.
Ian Griffiths presenting at Visual Studio Live! 2022 San Diego. Ian is presenting 3 talks about cold starts in .NET and Azure Functions, high performance serialization with C# 10 and high-volume event processing with Reaqtor.
Endjineers at Visual Studio Live 2022 Orlando - workshops & talks. Ed Freeman and Ian Griffiths are running a Modern Data DevOps in the Azure Cloud workshop,
Ian Griffiths is recognised as a MVP for the 15th time. For the 15th time over 19 years, Ian has been awarded Microsoft MVP status.
Howard van Rooijen's 7th renewal as an MVP for Azure and Developer Technologies.
Talks
This month we have two new talks, from sessions at Data Scotland 2022 and the Virtual Data Platform Summit 2022.
Fake it 'til you make it - generating production quality test data at scale
Many organisations provide digital products or services that need to handle personally identifiable information. The challenge is providing product and engineering teams with a sufficient volume of realistic looking synthetic data to enable them to design, develop and test their solutions.
Barry presents open-source tools and open data sources that can be used to tackle this challenge, and then demos this in action to generate thousands of synthetic customers.
He describes how this approach can be used to build better products, to test products using production quality data at production scale and embed data quality and best practice information security practices in your engineering processes.
This talk was originally delivered in Glasgow at Data Scotland 2022.
Do those numbers look right?
How to ensure quality and avoid inaccuracies in your data insights.
The need to validate business rules and security boundaries within a data solution is important, as well as the need for ensuring that quality doesn't regress over time as it evolves. Data insights are useless or even dangerous if they can't be trusted, and in this session, James will explain how and why they should treated just like any other software project with respect to testing - by building automated quality gates into the end-to-end development process.
During the session, he'll walkthrough some practical examples and proven techniques around testing data solutions - including Power BI reports, Azure Synapse Pipelines and interactive Spark notebooks.
This talk was originally given at the Virtual Data Platform Summit 2022.
Blog Posts
We've had a busy month, publishing at least one new post per day.
Azure Synapse Analytics, Power BI & Power Platform
Sharing access to synchronized Shared Metadata Model objects in Azure Synapse Analytics
The "Shared Metadata Model" is a powerful feature within Azure Synapse Analytics that synchronizes Spark database objects with SQL Serverless. This article, by Ed Freeman describes how to give non-admin users access to these synchronized objects in a least-privileged manner.
Performance Optimisation Tools for Power BI
Performance tuning is an important aspect of Power BI report development. This blog post, by Jessica Hill, highlights useful analyzer tools you can utilise to optimise the performance of your Power BI reports.
How to enable data teams with the design assets required for impactful data storytelling in Power BI
Paul Waller demonstrates how to expand on a data team's creative skillset, without access to specialist photo editing software such as Photoshop or Illustrator.
5 tips to pass the PL-300 exam: Microsoft Power BI Data Analyst
Recently Elisenda Gascon studied for and passed the PL-300 - Power BI Data Analyst exam. Here are top 5 tips if you intend to take the exam.
Automating Excel in the Cloud with Office Scripts and Power Automate
The combination of Office Scripts, Power Automate and Excel Online enables the possibility of automating many manual, Excel-based business processes. In the blog post Liam Mooney deep dives into the topic with a fully fleshed out demo automating a retailer's sales data report.
How to apply behaviour driven development to data and analytics projects
Barry Smart demonstrates how the Gherkin specification can be adapted to enable Behaviour Driven Development (BDD) to be applied to data engineering use cases.
Insight Discovery (part 1) – why do data projects often fail?
Many data projects fail to deliver the impact they should for a simple reason – they focus on the data. This series of posts, by James Broome explains a different way of thinking that will set up your data & analytics projects for success.
Insight Discovery (part 2) – successful data projects start by forgetting about the data
In part 2 of his series about our tried & tested Insight Discovery process, James Broome says that if you want your data projects to be successful, you need to forget about the data. Instead, start by understanding the business objectives, and work backwards from there to discover what insights are needed, and what data is
Insight Discovery (part 3) – Defining Actionable Insights
In order to make insights actionable, we need to identify the questions that we want to answer and the evidence that we need to support the decision making. We also need to consider the notification mechanisms and feedback loops to make sure that we're moving in the right direction.
Insight Discovery (part 4) – Data projects should have a backlog
Insight delivery should be focussed on business needs, not data availability. Insights should be prioritised by business impact and complexity to deliver.
DevOps, Cloud Native & Azure
Continuous Integration with GitHub Actions
Liam Mooney provides an overview of Continuous Integrations and shows how you can implement it with GitHub Actions, with an accompanying example Python project.
Publishing Scripts to the PowerShell Gallery
Publishing individual scripts to the PowerShell Gallery can be an effective way to share pervasive functionality without the overhead of maintaining a full-blown PowerShell module. In this post James Dawson looks at how we can take a function that is already published as part of a module and also make it available as a standalone script, without having to maintain two versions of the code.
Using Azure CLI Authentication within Local Containers
Have the recent changes to Azure CLI on Windows broken your dev inner loop when working with containerised applications that need access to Azure resources? If so, this post provides a workaround that will unblock you without having to be stuck on an old version of the Azure CLI.
Bye bye Azure Functions, Hello Azure Container Apps: Introduction
The first in a series of posts, by Jon George talking about how and why we migrated an application from Azure Functions to Azure Container Apps including cost modelling.
Bye bye Azure Functions, Hello Azure Container Apps: Migrating from Azure Functions to ASP.NET Core
In the 2nd part in the series on migrating an application from Azure Functions to Azure Container Apps, Jon George shows how to rehost a Function inside ASP.NET Core / Kestrel, in preparation to be containerised.
.NET
C# has gradually been adding support for discards. This article, by Ian Griffiths, explores how this evolution has led to some surprises.
Service Lifetimes in ASP.NET Core
There are three lifetimes available with the Microsoft Dependency Injection container: transient, singleton, and scoped. In this post, by Elisenda Gascon Arnal, she explains their different behaviours and the importance of being aware of the lifetimes of the dependencies in our applications.
IT Strategy
A simple toolkit for IT budgeting and planning
This article describes a free, easy-to-use approach that will enable you to optimise your IT budget. In the current economic climate, technology budgets are likely to come under increasing pressure. This toolkit can be used to optimise your budget by identifying opportunities to both root out waste and direct investment to where it is needed most. It achieves this by taking an evidence-driven approach to engage senior decision makers, reach consensus and make fundamental decisions about future investments in technology.
What is the total cost of ownership (TCO) and why is it important?
Understanding the total cost of ownership is key to making informed decisions about technology investments. Read the first in a 3-part series by Barry Smart
Newsletters
We publish two free weekly email newsletters: Azure Weekly on a Sunday, and Power BI Weekly on a Tuesday. Here's your chance to catch up with all the updates and announcements!
Azure Weekly
Issue 385: 4th September 2022 - Highlights:
- Azure Virtual Machines with Ampere Altra Arm - based processors goes GA
- Azure Cosmos DB integrated cache goes GA
- Azure Managed Grafana goes GA
- App Configuration references for App Service and Azure Functions Public Preview
- Introduction to Azure Automation
- Building Microservice Applications with Azure Container Apps and Dapr - Parts 1-5
Issue 386: 11th September 2022 - Highlights:
- Generally availability of the Kusto Emulator
- Reverse engineering the Azure Application Insights Transactions URL
- Walkthrough of Durable Entities
- An overview of Azure Chaos Studio
- Azure Container Apps with Dapr
Issue 387: 18th September 2022 - Highlights:
- Bicep is in Visual Studio
- API Management DevOps Resource Toolkit v1.0.0
- Azure Code Signing, democratizing trust for developers and consumers
- A guide to programmatically generate Cloud Resource Names
Issue 388: 25th September 2022 - Highlights:
- Improved Append Capability on Immutable Storage for Blob Storage
- Immutable storage for Azure Data Lake Storage
- Azure Blob Storage and Life Cycle Management
- Azure Payment HSM achieves PCI PIN certification
- Defining Actionable Insights with Azure Synapse
- Observability with Azure Container Apps
- Azure Container Apps Networking: A condensed view of concepts
Power BI Weekly
Issue 173: 6th September 2022 - Highlights:
- What's New in Datamarts – August 2022
- A deep dive into Power BI incremental refresh feature
- How to Set up Power BI Gateways on Azure Virtual Machines
- Changing a Report Tooltip Visuals Axis Field with a Slicer
Issue 174: 13th September 2022 - Highlights:
- AAD-based filtration for Power BI Datamarts
- Power BI component for Vue.js applications is now available
- Power Automate + Power BI Enhanced Refresh API Part 5: Getting A Dataset’s Refresh History
Issue 175: 20th September 2022 - Highlights:
- Power BI September 2022 Feature Summary,
- Hierarchical style axis
- Automatic display names for summarized fields have been adapted
- Consume datasets you have access to in external tenants
- "Horizontal Fusion" a query performance optimization in Power BI and Analysis Services
- USERCULTURE DAX function now supported in Power BI Premium
- Power BI report and dataset management in Power Apps Solutions
- On-premises data gateway August 2022 update is now available
- Power BI data types in relationships – Does it matter?
- Calling the Power BI Enhanced Refresh API From Power Automate, Part 6: Cancelling Dataset Refreshes
Issue 176: 27th September 2022 - Highlights:
- What's New in the September Release of Power BI Report Server
- On-premises data gateway September release
- Power BI Metrics (previously referred to as Goals) survey
- How OFFSET in DAX will make your life easier
- Defining Actionable Insights
- How to enable data teams with the design assets required for impactful data storytelling in Power BI