In this post, I want to share my experience on how to ensure Secure DevOps practices that enhance software quality in DevOps environments.
What is DevOps
I have seen many variations of the term DevOps. Some organizations restrict teams from accessing the production environment. In these cases, the work is divided among different teams, which may even belong to separate organizations. The development team creates a new release candidate, while the operations team handles the deployment in production. This separation can be necessary in sensitive domains. However, it also has disadvantages: troubleshooting requires multiple teams to communicate and collaborate. The development team understands their software, while the operations team is experienced with the runtime environment. Finding problems and bugs necessitates collaborative interaction between these teams.
Typically, it’s more efficient when one cross-functional team takes responsibility for the entire product lifecycle, embracing the philosophy of “you build it, you run it.” This approach doesn’t mean that every team member must possess skills in all domains; rather, the team as a whole should encompass all necessary skills. This model offers several advantages:
- The team can learn from bugs, problems, and incidents, leading to better software design.
- The team can resolve all issues related to the product and its runtime environment without needing to collaborate with other teams.
- The team can deliver independently, without external dependencies.
- The team will be motivated to design a maintainable and stable product because they must maintain it themselves.
The question is: How can a team take full responsibility for the software and its entire runtime environment without introducing risks or security vulnerabilities? Software infrastructure and libraries are increasingly targeted by attacks. Fortunately, many tools and methods can help ensure a secure software development process, often referred to as Secure DevOps, which reduces the risk of potential security vulnerabilities.
In this post, I want to share my experience on how to ensure Secure DevOps and maintain high software quality.
Secure Devops & Software Quality with Static Code Analysis
There are many reasons for static code analysis. Some of them are:
- The code should be formatted according to team alignment.
- Static code analysis can identify certain bugs.
- It helps pinpoint complex and redundant hotspots.
- It uncovers poorly readable code and “don’ts.”
- It identifies potential security vulnerabilities, such as hard-coded secrets.
- It can reveal architecture violations.
A commonly used tool for static code analysis is SonarQube. It supports many programming languages and rulesets developed by a large community.
Measurement of the Test Coverage for a good Software Quality
It’s clear that a good software product should include automated tests to ensure the software continues to function after changes and refactorings. Test coverage can serve as a useful indicator of the amount of testing performed, but teams should use this metric pragmatically. There are various types of coverage, such as path coverage, branch coverage, and condition coverage, to name a few. In my opinion, aiming for 100% coverage of the source code is a waste of time. The product team should discuss which elements of the product require high test coverage. Some indicators to consider include:
- Complex hotspots
- Important use cases
- Cost of potential damage
- Frequency of changes
- Security-sensitive parts, such as authentication and authorization
SonarQube helps measure test coverage. By excluding certain files, teams can disregard code that should not be considered, such as accessor methods without logic or generated code, which can waste valuable testing time.
Quality Gates with DevOps
It’s possible to define quality gates that contain rules specifying when a gate is passed or not. For example, critical bugs identified by static code analysis, poor security ratings, or low test coverage can cause the quality gate to fail. If the deployment pipeline is configured with a build-breaker, the pipeline will fail if the quality gate is not passed.
Dependency Check as part of Secure DevOps
Another significant aspect of Secure DevOps is verifying external libraries. More and more libraries contain security vulnerabilities that attackers can exploit. One helpful tool is Nexus IQ. This tool integrates into build pipelines, and during the build process, the Nexus IQ Auditor checks all libraries. The tool generates reports for each build and highlights license issues and security vulnerabilities. It also recommends newer or older versions of a vulnerable library where the issue does not exist. I recommend configuring the pipeline to break in the case of critical errors.
Secure the Runtime Infrastructure as part of Secure DevOps
IIt’s crucial to secure the runtime infrastructure, especially in public cloud environments. As humans, we can make mistakes, but fortunately, many ways exist to significantly reduce the risk of potential security vulnerabilities.
There are two main strategies to achieve this:
Policies: Another approach involves utilizing the runtime infrastructure’s capabilities. For instance, Azure provides the ability to define policies, with hundreds available for configuration. One example is a policy that ensures every database is protected by a firewall. If a user forgets to configure a firewall – whether via CLI, API, or UI – the provisioning of the corresponding resource will fail. In addition to existing policies, you can also create custom policies tailored to your specific needs.
Static Code Analysis: Many infrastructure tools, such as Terraform and Pulumi, allow you to test infrastructure code. This testing helps ensure a high level of security. For example, you can verify that databases have firewalls properly configured, that anonymous access is disabled, and that secure transport protocols are in use. Many potential security risks can be addressed through these tests.
Summary
There are many ways to ensure secure DevOps and maintain good software quality. You can analyze the code for potential bugs, maintenance issues, and security vulnerabilities. Relevant modules should be covered by automated tests, and quality gates ensure that the required test coverage and software quality standards are met. It’s also possible to identify vulnerable libraries.
Instead of splitting development and operations into two teams, it’s often more effective to assign full responsibility to a cross-functional product team. Security and infrastructure specialists can utilize tools like Terraform or Azure Policies to ensure that the runtime infrastructure maintains a strong security posture. This approach significantly reduces the risk of human errors.
All these actions should be integrated into the pipeline. If problems occur, the pipeline should break to prevent the deployment of flawed software.
While it’s impossible to eliminate the risk of security issues entirely, these actions can help to significantly reduce that risk. For me, this represents the current baseline for a secure DevOps environment, but many more tools and methods are available in the market. This field is rapidly evolving in response to potential threats and the tools designed to prevent them.