Navigating Common Pitfalls and Future Trends in Azure Pipelines

In this post, we will explore the best practices to provide insights on optimizing your workflow for maximum efficiency and reliability. This guidance will help you to understand how your pipeline operates at its full potential. We will also explore future trends in Azure Pipelines by the end of this post. Lastly, we will know how to avoid common pitfalls, optimize your workflow with best practices, and adapt to the evolving landscape of this essential tool.

This post will cover the following topics:

- Common pitfalls

- Best practices

- Future trends

Common Pitfalls

Understanding and proactively addressing common mistakes and issues in Azure Pipelines is essential. When we learn how to recognize and resolve these issues promptly, we can save valuable time and resources.

In real-world projects, timely resolution is crucial to prevent delays and costly setbacks. This allows you to maintain project momentum and focus on other critical aspects.

Let’s discuss some of these common errors:

- Pipeline Queued: Sometimes, you may start a pipeline run and see it is queued. This could be due to a lack of available agents on Microsoft-hosted or self-hosted agents, insufficient concurrency, or unmet demand for agents. You can identify the error immediately after checking the status of the job in your pipeline, as shown in the following screenshot:

-This might indicate that you need to purchase additional parallel jobs to prevent queuing and ensure that your pipelines run as quickly as possible.

- Incorrect Branch Filtering: A pipeline may not run because of incorrect branch filtering in YAML files or disabled triggers. Review the values under the branch section in your YAML files to ensure they are correct.

- Access Permissions: Pipelines may fail to run due to access permissions for variable groups or secret files. To resolve this, navigate to Library > Secrets > Variable Group > Pipeline Permissions.

Then, you can do either of the following:

• Open all access for all pipelines. This option will allow all pipelines access to secrets in the

variable group. If all pipelines need to use the same secrets, then you should select this option:

This option will limit access to only the required pipelines that can access secrets. Otherwise, anyone could access your pipeline and use your secrets.

Best Practices

Following best practices becomes crucial when we encounter issues during a pipeline’s lifecycle. These practices will enable your team to maintain pipelines with ease and swiftly identify and address any problems that arise.

Here are some practices to help your team find and resolve issues efficiently:

- Use YAML Syntax: Opt for YAML syntax over the classic version. YAML can be version-controlled in repositories such as Azure Repos, GitHub, and Bitbucket, allowing you to track changes and introduce updates gradually with a branching strategy. To enforce this, you can disable the creation of classic build pipelines at the organization or project settings level.

- Modularize Common Tasks: Separate common tasks into individual YAML files that can be referenced from your main template. This approach simplifies maintenance and changes in large multi-stage configurations.

- Create More Stages for Long-Running Jobs: If you have long-running jobs, consider creating more stages to improve manageability and performance.

- Keep Pipelines Small and Focused: Use templates, stages, jobs, and steps to keep your pipelines small and focused. This enhances clarity and maintainability.

- Optimize Build Stages for Large Applications: Large and complex applications may require more resources to compile and package faster. While Microsoft-hosted agents run on general-purpose virtual machines with limited resources, you might need self-hosted agents on more powerful virtual machines to improve build performance.

- Use Incremental Compilation: Consider using programming languages that support incremental compilation. Projects benefiting from partial incremental builds should be hosted on self-hosted agents to ensure filesystem resources are retained between builds.

- Utilize Variable Groups: Reduce duplication by using variable groups when multiple pipelines share the same values. This makes your pipelines more maintainable.

- Secure Private Values: For private values or keys, use secret values. Navigate to Library > Secrets > Variable Group and click on the lock icon to limit access to these variables.

If you have variables from Azure Key Vault (KV), Azure’s Key Management Service (KMS) will encrypt and securely store the secret keys. This ensures that all variables are protected, and no one can access your secret keys without permission. It is recommended to link all variables from KV to the variable group, as shown here:

Consider applying the principle of least privilege, a security concept that dictates users should be granted the minimum access levels needed to perform their job functions. This restricts access rights to only what is necessary, minimizing the risk of malicious activity or accidental damage and reducing the chance of incorrect pipeline usage.

To investigate the problem of failed pipelines, you can download the log directly from the pipelines. This can be done by following these steps:

I. Select the pipeline for which you need to download the log.

II. Click on the three dots icon and then Download logs:

If you use PublishBuildArtifacts@1, you can download artifacts directly from a pipeline if you need to inspect them, by following these steps:

I. Select a pipeline, and click on the value in the Related column, # published:

II. Click on the file to download it:

For cost optimization, you can adjust the retention policy of Artifacts to reduce the storage size:

I. Go to Project Settings | Settings.

II. Adjust the parameters of the retention policy, depending on your business objectives. For example, if your business plan requires you to retain Azure pipeline results for only five days due to the need for cost optimization in storage, then it’s essential to adhere to this timeframe. Keeping data for longer periods would result in additional storage costs:

? Consider validating the number of parallel jobs you have. It will help you to understand better how many processes you can run on the pipelines in parallel mode:

I. Navigate to Project Settings | Parallel jobs.

II. You can see an overview of all the agents you have:

Future Trends

Azure DevOps is continuously evolving, with new features and capabilities introduced regularly, driven by the Microsoft product team's efforts to improve the platform and user requests.

Here are some of the latest features related to Azure Pipelines:

Enhanced Authoring Experience: Improvements have been made to the authoring experience for Azure Pipelines tasks to support newer versions of Node.js. This facilitates the transition to newer versions and allows you and other community contributors to create your extensions.

GitHub Advanced Security for Azure DevOps: This new security suite aims to enhance DevSecOps practices. It includes tools to scan your code and prevent common vulnerabilities, such as cross-site scripting, SQL or XML injection attacks, and other common issues developers may overlook. This feature supports many programming languages and is continually evolving to offer broader support.

? Reduce the gaps in features between YAML pipelines and classic release management deployment pipelines for CD scenarios. Several capabilities to improve the checks will be released in the coming year.

Looking into the future, other features will also become important as you expand your usage of Azure Pipelines:

? In-product recommendations to highlight the best practices when configuring your pipelines around security

? Deprecation and removal of unsupported versions of Node.js for the agent software included with both the Microsoft-hosted and self-hosted versions The Azure DevOps features roadmap is available at: -

Where you can see all the product capabilities that are in progress or planned to be released by Microsoft in the near and long term.

Summary

This post will help guide you in your journey, empowering you to use Azure Pipelines with confidence and expertise. You will be better equipped to recognize and address common pitfalls and challenges, saving valuable resources and maintaining a seamless workflow.

By following best practices, you can optimize the functionality of your pipelines and enhance team collaboration, ensuring efficiency and reliability in your pipeline operations. We see a glimpse into the future of Azure Pipelines, enabling you to adapt and stay ahead in this dynamic field.

Good luck on your journey!

Microsoft Learn Microsoft Azure Microsoft Azure DevOps