How to deal with sensitive data in .NET

In the realm of .NET software development, the mishandling of sensitive data is a pervasive issue that compromises security and privacy. This concern spans various aspects of the development process, from initial coding to deployment and testing. Bad practices often emerge not from malicious intent but from a lack of awareness or understanding of proper security measures.

1. Bad practices

1.1 Hardcoding sensitive dataOne of the most glaring mistakes is hardcoding sensitive information, such as passwords, API keys, and connection strings, directly into the source code. For instance, consider a sample .NET console application using Entity Framework to read data from an SQL Server database:

using System;
using System.Linq;
using Microsoft.EntityFrameworkCore;

namespace SensitiveDataExample
{
    public class Program
    {
        public static void Main(string[] args)
        {
            using (var context = new UserDbContext())
            {
                var users = context.Users.ToList();
                foreach (var user in users)
                {
                    Console.WriteLine($"User: {user.Username}, Email: {user.Email}");
                }
            }
        }
    }

    public class UserDbContext : DbContext
    {
        protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        {
            // Hardcoded sensitive data - bad practice example
            optionsBuilder.UseSqlServer(@"Server=myServerAddress;Database=myDataBase;User Id=myUsername;Password=myPassword;");
        }

        public DbSet<User> Users { get; set; }
    }

    public class User
    {
        public int Id { get; set; }
        public string Username { get; set; }
        public string Email { get; set; }
    }
}

        

This approach not only makes this information easily accessible to anyone who can view the code but also complicates the process of updating or rotating secrets, leading to potential security vulnerabilities.

1.2 Improper use of configuration files

Moving towards improving security practices in handling sensitive data, we can refactor the example above to externalize the sensitive information into an appsettings.json file. This approach avoids hardcoding sensitive details directly in source code. However, remember that storing sensitive data in appsettings.json without proper encryption or access controls still poses security risks, especially if the file is included in source control. So this technique is especially risky in open-source projects or when repositories are not properly secured.

First, create or modify your appsettings.json file to include the database connection string:

{
  "ConnectionStrings": {
    "DefaultConnection": "Server=myServerAddress;Database=myDataBase;User Id=myUsername;Password=myPassword;"
  }
}
        

Next, adjust your .NET console application to read the connection string from appsettings.json:

using System;
using System.Linq;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Configuration;
using System.IO;

namespace SensitiveDataExample
{
    public class Program
    {
        public static void Main(string[] args)
        {
            var configuration = new ConfigurationBuilder()
                .SetBasePath(Directory.GetCurrentDirectory())
                .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
                .Build();

            var connectionString = configuration.GetConnectionString("DefaultConnection");

            using (var context = new UserDbContext(connectionString))
            {
                var users = context.Users.ToList();
                foreach (var user in users)
                {
                    Console.WriteLine($"User: {user.Username}, Email: {user.Email}");
                }
            }
        }
    }

    public class UserDbContext : DbContext
    {
        private readonly string _connectionString;

        public UserDbContext(string connectionString)
        {
            _connectionString = connectionString;
        }

        protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        {
            optionsBuilder.UseSqlServer(_connectionString);
        }

        public DbSet<User> Users { get; set; }
    }

    public class User
    {
        public int Id { get; set; }
        public string Username { get; set; }
        public string Email { get; set; }
    }
}
        

This version of the application leverages the Microsoft.Extensions.Configuration package to read from appsettings.json, avoiding hardcoded sensitive details in the source code. The ConfigurationBuilder loads the configuration file, and the connection string is fetched using the GetConnectionString method, which is then passed to the DbContext.

While this method is a step forward in securing sensitive data by externalizing it from the codebase, it's crucial to manage the appsettings.json file carefully, especially regarding source control and deployment processes. Further security measures, such as encrypting the connection string or using more secure storage mechanisms like Azure Key Vault, are recommended to enhance data protection.

1.3 Other common mistakes

Developers might also overlook the need for secure communication channels, transmitting sensitive data over unencrypted connections. This leaves the information vulnerable to interception by malicious actors.

Neglecting environment-specific configurations is another pitfall. Using the same credentials or keys across different environments (development, testing, and production) increases the risk of exposure and misuse. It not only poses a security threat but also hinders the identification of issues specific to particular environments.

Furthermore, the failure to implement proper access controls and audit trails for sensitive data access can obscure the detection of unauthorized access or data breaches, complicating security monitoring and response efforts.

These practices not only endanger the security and integrity of the software but also violate compliance standards, potentially leading to legal and financial repercussions. Addressing these issues is not merely about adopting new tools but also about cultivating a security-aware culture among developers and integrating best practices throughout the software development lifecycle.

2. The ultimate solution -- User Secrets

Introducing User Secrets is a significant step towards enhancing security in .NET applications, especially in the development phase. User Secrets provide a mechanism for storing sensitive data outside of the project's source code, preventing the data from being accidentally shared or checked into source control. This technique is particularly useful during development, as it allows developers to keep sensitive information such as API keys, connection strings, and other secrets out of their codebase and configuration files like appsettings.json.

The User Secrets feature is part of the .NET Core framework and is accessible via the .NET Core CLI or Visual Studio. It stores secrets in a separate location on the developer's machine, which by default is a JSON file in a system-protected user profile folder.

Let's refactor the previous example to utilize User Secrets:

2.1. Enable User Secrets in the Project

To start using User Secrets, you need to enable them for your project. This is done by adding a UserSecretsId to your project file (*.csproj). Right-click on your project in Visual Studio > Properties > Right-click on the project file in Solution Explorer > Edit Project File, and then add the following inside the <PropertyGroup>:

<PropertyGroup>
  <!-- Other properties -->
  <UserSecretsId>YourProjectName-3f5b48b9-9f2f-4098-846b-5465195390c1</UserSecretsId>
</PropertyGroup>
        

The UserSecretsId value is typically a unique GUID. You can generate one using online tools or Visual Studio.

2.2. Store secrets using the .NET Core CLI

With the project configured, you can now add your secrets. Open a terminal or command prompt, navigate to your project directory, and run the following command to store the connection string as a secret:

dotnet user-secrets set "ConnectionStrings:DefaultConnection" "Server=myServerAddress;Database=myDataBase;User Id=myUsername;Password=myPassword;"
        

2.3. Modify the application to use User Secrets

The great thing about User Secrets is that you don’t need to change your code to access them if you’ve already refactored your application to use configuration providers. The ConfigurationBuilder in .NET Core automatically includes User Secrets when the application is running in the Development environment. Here's a reminder of how your Program.cs file looks:

var configuration = new ConfigurationBuilder()
    .SetBasePath(Directory.GetCurrentDirectory())
    .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
    .AddUserSecrets<Program>(optional: true)
    .AddEnvironmentVariables()
    .Build();

var connectionString = configuration.GetConnectionString("DefaultConnection");
        

In the code above, the .AddUserSecrets<Program>(optional: true) line tells .NET to load configuration values from the User Secrets store when the application is running in the Development environment. Note that you should ensure the application environment is set to Development for User Secrets to be included.

2.4. Accessing the secret in your application

Since the configuration setup hasn’t fundamentally changed, accessing the secret remains the same as in the previous example, demonstrating the simplicity and effectiveness of the User Secrets mechanism:

using (var context = new UserDbContext(connectionString))
{
    // Your database operations here
}
        

2.5. Manipulating your secret.json file directly

Managing User Secrets efficiently is crucial for .NET developers, especially when it comes to editing them. There are several ways to manage these secrets, including using Notepad for direct edits or leveraging Visual Studio's integrated support.

The secrets.json file is where User Secrets are stored locally on your machine. This file is associated with your project via the UserSecretsId specified in your project file (.csproj). Here's how you can manually edit this file:

2.5.1 Locating thenbsp;secrets.jsonnbsp;file

1. Find the User Secrets ID: Open your project file (.csproj) and look for the <UserSecretsId> element. This ID is used to name the secrets.json file and its directory path. For example:
2. Navigate to the Secrets Directory: User Secrets are stored in a system-protected directory on your development machine. The path typically follows this pattern:

2.5.2 Editing thenbsp;secrets.jsonnbsp;file with Notepad

To edit the secrets.json file using Notepad or any other text editor:

1. Open File Explorer (Windows) or Finder (macOS/Linux) and navigate to the path where your secrets.json file is located, based on your operating system and the User Secrets ID.
2. Open the secrets.json File: Once you've located the file, right-click on it and choose Open with > Notepad(or your preferred text editor).
3. Edit Your Secrets: In Notepad, you'll see your secrets stored in JSON format. You can add, modify, or delete secrets as needed. For example:
4. Close Notepad: After saving your changes, you can close Notepad. The changes to your secrets will be automatically picked up by your application the next time it runs.

2.5.3 Editing User Secrets in Visual Studio

Visual Studio provides a more integrated and user-friendly approach to manage User Secrets, but it requires additional setup and might have compatibility issues with older or unsupported .NET Core versions.

1. Ensure Compatibility:
2. Add Required NuGet Package:
3. Edit User Secrets in Visual Studio:

2.5.4 Important Considerations

• Security: While accessing and editing the secrets.json file directly can be convenient, it's essential to handle the file carefully to avoid exposing sensitive information.
• Environment Specificity: Remember, User Secrets are primarily intended for development environments. For production, consider using more secure mechanisms like environment variables, Azure Key Vault, or other secure storage options.

Editing secrets.json directly offers flexibility in managing your application secrets, but always prioritize security best practices, especially when dealing with sensitive information.

3. Providing secret data in the CI/CD pipelines

Integrating User Secrets into the build process, particularly for Continuous Integration/Continuous Deployment (CI/CD) pipelines in Azure DevOps, requires a strategy that securely passes these secrets during build and test executions. Since User Secrets are primarily a development-time feature and not meant for production or CI environments, you need to use Azure DevOps features to handle secrets safely during these stages. Here's how to manage secret values during the build process, focusing on Azure DevOps pipelines:

3.1 Using pipeline variables

1. Define Secure Pipeline Variables:
2. Accessing Secure Variables in Your Pipeline Tasks:

3.2 Integrating Azure Key Vault

Integrating Azure KeyVault with Azure DevOps pipelines is a powerful method for securely managing and accessing sensitive values such as secrets, passwords, API keys, and other encrypted data during CI/CD processes. Here's how to integrate Azure KeyVault with Azure DevOps using Variable Groups, allowing you to securely manage secrets outside your pipeline code.

1. Set Up Azure KeyVault
2. Configure Access
3. Create an Azure DevOps Variable Group
4. Connect to Azure KeyVault
5. Use the Variable Group in Your Pipeline
6. Secure Access

4. Providing secret data in a production environment

Let's suppose that we are deploying a REST API as an Azure Web Appication, and securely manage the ConnectionStrings:DefaultConnection for production environment. Leveraging Azure Key Vault is the best practice for this. Azure Key Vault provides a secure way to store and access secrets like connection strings. Your application can access these secrets at runtime, which enhances security by avoiding hard-coded credentials or storing them in configuration files.

4.1 Integrating Azure Key Vault Without Code Changes

If your application already uses the Microsoft.Extensions.Configuration infrastructure (which is default for ASP.NET Core projects), integrating Azure Key Vault can be seamless with minimal to no code changes for fetching configuration values like connection strings. The Azure App Service can be configured to load secrets from Azure Key Vault as application settings, making them available to your application, just like environment variables or values stored in appsettings.json.

Steps to Integrate Azure Key Vault with Azure Web App:

1. Set Up Azure Key Vault:
2. Assign Managed Identity to Azure Web App:
3. Grant Access to the Key Vault:
4. Configure Azure Web App to Use Key Vault:

5. Conclusion

Managing sensitive data, particularly secrets, is a crucial aspect of modern software development that requires attention to security and best practices. We explored common pitfalls such as hardcoding secrets or improperly managing configurations, and highlighted the importance of utilizing User Secrets for local development to mitigate these issues. By leveraging Azure DevOps pipelines and Azure Key Vault, we demonstrated secure methods to provide secrets in both CI/CD processes and production environments without necessitating changes to the original codebase. This approach not only enhances security by centralizing and securing access to sensitive information but also maintains consistency across development, testing, and production environments. Adhering to these practices ensures that applications are developed with a strong foundation of security, making them robust against potential vulnerabilities and breaches. By integrating these techniques, developers can focus on building features and functionality, confident in the knowledge that their application's sensitive data is well-protected.

Article by Gergely Tóth .