At Hdiv we talk to lots of developers and infosec professionals. We have more than a few funny stories about the friendly rivalry -and sometimes not so friendly- between these two teams. Some people even call them tribes!
Like Switzerland, we like to take a neutral position, and we strive for the common goal of building secure applications.
That’s why we are so excited to help teams adopt DevSecOps methodologies. We believe that DevSecOps is a great way of creating applications that are built with security in mind since its inception: we call this approach ‘security by design’.
DevSecOps is such an important concept for us that we want to give it plenty of attention. So we are going to break our message across a few individual stories: In this first post, we will introduce the overall vision and our recommendations. In subsequent posts, we will expand each of our individual recommendations in a dedicated post.
Table of contents
The DevOps Methodology
What Is DevSecOps (aka SecDevOps)?
DevOps vs DevSecOps
DevSecOps Business Benefits
DevOps Security Pain Points
From DevOps to SecDevOps: The 7 Key Steps
Make DevSecOps a reality today: How Hdiv can help you
The DevOps Methodology
What makes DevOps different? The first keyword is speed: DevOps philosophy takes the principles behind agile methodologies to a new level of speed by feverishly increasing the rate at which code is pushed to the repos, and subsequently released into production. The adoption of continuous integration tools, such as Jenkins, enables ever faster releases. The application release cycles are moving from “a few releases each year” to “weekly releases” –or even more frequently.
The second keyword is scale: DevOps supports systems with hundreds (or thousands) of applications built by hundreds of developers, that run on dozens of servers that can increase from 10 to 100 in seconds.
When speed and scale become such an important factor, dev teams naturally respond by introducing as many optimizations as possible to eliminate bottlenecks in the SDLC. Very often, the security activities in the software life cycle are deprioritized or overlooked. After all, we can’t blame for “giving up” after seeing a list of hundreds false positives that sometimes code review tools produce.
BONUS: Free White Paper
Accelerate your agile AppSec practice. The 7 key factors to successful DevSecOps.
Get Your Whitepaper
What is DevSecOps (aka SecDevOps)?
DevSecOps, also known as SecDevOps, is a software development philosophy that promotes the adoption of security through the entire software development lifecycle. DevSecOps goes beyond a single tool or specific practice, but generally speaking, it favors security automation, communication, and scalability. This development philosophy scope includes all teams involved in the SDLC, such as development, operations, and security.
In the past, the assessment and validation of the security of the applications happened only at the very end of the project. The release cycles were longer, such as months and years, so the postponement of security was not ideal, but it was manageable to some extent. However, the acceleration of the software release cycles has made this legacy development approach impractical and even inapplicable. Nowadays, it is not uncommon to release new code to production on a weekly and daily basis. This means that security automation and scalability are required, and are no longer optional.
DevSecOps favors the inclusion of application security activities from the very beginning of the development stages. Automation is therefore key to allow this agile methodology. Awareness and security culture should be the first priority, but certain AppSec tools will make this security automation easier.
DevOps vs DevSecOps
DevOps is a software engineering methodology that unifies the development and operations activities, with the goal of enabling agile continuous delivery. Generally speaking, teams that adopt DevOps practices experience productivity and performance metrics improvements and release better quality software. Other benefits are:
- Time-to-market reductions
- Application stability increases
- Increased agility to respond to competitive shifts
On top of these DevOps benefits, DevSecOps introduces security automation as well as new security processes wrapped around the DevOps flow. For instance, the most basic implementations start by adding DevSecOps elements to the CI-CD pipeline.
DevSecOps does not just add security elements, but when done properly, it makes security an integral part of the entire process, from the beginning to the end.
As a consequence, the security team becomes much more engaged with the rest of the teams involved in the SDLC, including Development and Operations. This removes friction as the natural tension between speed and security is shared by all.
DevSecOps Business Benefits
Adopting DevSecOps often delivers immediate business benefits. The first, is a general improvement in the security position of the applications. As DevSecOps favors early identification and mitigation of security risks, this also means faster time-to-market and as a result, reduction of development costs.
In order to have an smooth adoption of security automation culture, we recommend a few essential DevSecOps best practices to get started:
- Short and frequent development cycles
- Incorporate security from the very beginning
- Leverage technologies that foster agility, such as containers and microservices
- Make an organizational priority to instill a culture of proactive collaboration between the teams involved in the SDLC
- Automate security as much as possible to facilitate agile development
DevOps security pain points
Based on our experience, we see the following security pain points in DevOps environments:
1. Inefficient SAST tuning & false positives
Static AST tools (SAST) are the most widely used solution in terms of detection of security issues during the development phase. However, SAST tools usually require tuning processes to integrate with each different development architecture, and they present a high rate of false positives. Many developers find the long and static reports produced by SAST tools frustrating and ignore them.
In consequence, SAST tools consume tuning time for the tools and review time to clean false positives from real vulnerabilities. Dynamic AST tools (DAST) are also extremely popular but they lack source code visibility and in turn they present a very low detection rate of existing vulnerabilities (18%). Also, the detected vulnerabilities are not matched to the location in the code, and many of them remain undetected.
2. Lack of coordination between infosec team and developers
Security issues are treated separately from other software quality issues. QA and penetration testers typically produce long static reports that contain the security issues. As a result, developers must check different systems to review traditional functionality bugs vs. security issues. Developers naturally tend to resolve functionality bugs first, and only after, they focus on security issues. This means that many security issues are not fixed during development. Once a critical issue sneaks into production environments, it takes an average of 129 days to get fixed [1].
3. Continuous deployment chaos
Teams do not have an efficient way of evaluating the security level of their applications, and as a result, security issues sneak into production. Source code compilation issues stop builds and continuous integration pipelines, but there is no automatic way for security bugs to block the deployment to production. This means that teams must manually ensure the compliance of security, which introduces delays and/or the deployment of insecure applications.
4. Manual pen-testing becomes a bottleneck
First of all, it is important to note that detection solutions (AST vendors) only detect one kind of vulnerabilities: essentially, code syntax issues, also known as security bugs. Examples of security bugs are SQL injection, XSS, etc. and they follow specific patterns that are exactly the same across all applications.
Unfortunately, there is another important group of security vulnerability: business logic flaws or design flaws. They can not be detected by tools. We are talking about security vulnerabilities such as access control, binding, abuse, etc.
In view of this situation and because automatic detection tools only detect security bugs (50% of the total problem), companies and organizations perform manual reviews to detect business logic flaws. Current agile development methodologies that generate new releases almost every day do not adapt well to this manual review. Even when review budgets are not an issue, the review impacts severely on the time to market of your applications.
5. Poor visibility of security track record
Teams lack access to smart tools to monitor the evolution of security vulnerabilities from build to build. Most teams are not aware of the number, severity, and age of the vulnerabilities.
Fixing all security issues is next to impossible, but teams lack a clear way to rank the existing vulnerabilities by their severity. As a result, teams waste time fixing issues that are not exploitable while leaving serious issues unresolved.
6. Lack of cloud support
Cloud environments do not have a perimeter that can be defended with traditional tools such as WAFs. Cloud providers and/or deployment conditions can change dramatically over time. Most security tools do not adapt to these changes.
Frequently, security defense is trusted to an edge boundary that has no context and has no visibility of the app. The data flow is only inspected at one point, which limits the security analysis accuracy and introduces false positives.
7. Systems that are not scalable
External AST review strategies such as testing-as-a-service introduce lags in the deployment phase, ranging between 4 and 24 hours. This lag reduces scalability and increases time-to-market. Once in production, it is common to abruptly scale from 10 servers, to 100 servers within seconds. DevOps teams complain that security tooling is a bottleneck and lacks flexibility to scale dynamically as the application load grows.
From DevOps to SecDevOps: The 7 Key Steps
The Application Security community has reacted to the challenges and pain points described above by wrapping the DevOps philosophy with a security blanket: DevSecOps.
DevSecOps is all about increasing speed and productivity by introducing automatization, monitoring, and tight integration of security. DevSecOps leverages Secure SDLC tooling and at the same time, it ensures that the cadence of releases is not hampered.
In subsequent posts we are going to cover in detail each of the of the following 7 best practices but here is the summarized version of our recipe for DevSecOps success:
| DEVOPS PAIN POINT | DEVSECOPS SOLUTION |
| Inefficient SAST tuning & false positives. | Step1: Use an IAST Tool. |
| Lack of coordination between infosec team and developers. | Step 2: Integrate security issues within your general bug tracker. |
| Continuous deployment chaos. | Step 3: Define metrics and thresholds to assure quality. |
| Manual pen-testing becomes a bottleneck. | Step 4: Automate the protection of business logic flaws. |
| Poor visibility of security track record. | Step 5: Better reporting. |
| Lack of cloud support. | Step 6: Embedded security. |
| Systems that are not scalable. | Step 7: Linear scalability and affordable cost. |
Pain Point #1: Inefficient SAST tuning & false positives
Solution: Use an IAST tool
Most software development teams do not have dedicated security experts; as a result, security protection will be implemented by non-specialists. Including AST tools in your toolchain empowers developers to create secure code.
To avoid the lack of accuracy and the labor-intensive reviews of SAST reports, use a more direct detection tool such as an Interactive-AST (IAST.) IAST tools do not require tuning or manual reviewing of false positives since they do not generate them. Use the OWASP Benchmark coverage indicator to rank vendors.
With the help of IAST tools, you remove slow code scanning activities and instead receive real time information on your security issues as you code. IAST tools help to push security testing left.
Pain Point #2: Lack of coordination between infosec team and developers
Solution: Integrate security issues within your general bug tracker
Communicate the bug tracking tools that your team is using, such as Jira, with your security tools so that developers can see security bugs as regular tasks. Automatically create new bugs and tasks to represent security vulnerabilities as they are found during the reviews and audits of the application.
The goal behind this recommendation is that developers never have to leave their continuous integration and/or continuous deployment toolchain environment [2]. As a result, more security issues are resolved during the development phase, which is much better than resolving them later in the process.
Pain Point #3. Continuous deployment chaos
Solution: Define metrics and thresholds to assure quality
The same way code compilation errors stop your deployment, security issues should block the deployment as well. Known as “security controls” these automated checkpoints ensure that the code pushed downstream the CI/CD stream respects security standards.
Create automatic security checkpoints to enforce quality goals, and stop the pipeline if the number of security bugs is higher than a goal. Seek solutions that integrate with your deployment server (such as Jenkins).
It is important to adapt the thresholds to the team’s maturity regarding security, starting with a more flexible threshold and using very strict thresholds at the end.
Pain Point #4: Manual pen-testing becomes a bottleneck
Solution: Automate protection of business logic flaws
Business logic flaws (also known as design flaws) are an important group of security issues that detection tools can not identify. To mitigate the bottleneck of manually reviewing business logic flaws, automate the input validation using solutions and architectures that are secure from the start. Create feedback loops to help pentesters focus their energies on the right parts of the application. When the pentesting team has a clear image of the attack surface, they are more productive.
Then, integrate the output of the solutions with the audit tools. The combination of automating the protection and connecting the protection with audit tools allows automation of most manual pen-testing activities.
Pain Point #5: Poor visibility of security track record
Solution: Better reporting
Versioning your releases should not be an afterthought. Implement metrics and reports that track the evolution, number, and severity of bugs for each release. Enhance solutions such as Jenkins Reports & Web Reports to include security issues.
Identify metrics that increase the productivity of developers and remove critical issues first. Instill a culture that incentivizes progressive reduction of the number of security issues, and celebrates positive improvement trends towards perfection. Infrastructure artifacts such as deployment descriptors and build scripts should be subject to strong version control. Postmortem analysis should include reports with security issues as well.
Pain Point #6: Lack of cloud support
Solution: Embedded security
Build applications with agile security built-in. When the deployment conditions and cloud providers change all the time, applications that have security embedded are able to adapt to the changes.
We believe in security as code, and vulnerabilities as defects. By incorporating security early in the development process, your applications are born secure, and remain secure wherever they go.
Pain Point #7: Systems that are not scalable
Solution: Linear scalability and affordable cost
Ensure that the security infrastructure in your application is not a performance bottleneck. Seek security solutions that scale in constant or linear time. Monitor the evolution of the added latency of the security solution and choose those that perform better.
Make DevSecOps a reality today: How Hdiv can help you
The foundation of our recommendations and best practices described above is the idea of empowering developers to build secure code. This means helping your developers with the right toolchain, that provides helpful input (low false positives), it is integrated in their work pipeline (security issues as defects in the bug tracker), and creates a culture where writing secure code is recognized as a goal.
From the very beginning, Hdiv was created by and for developers. The practical recipe above, and more broadly our company DNA, supported the DevSecOps philosophy even before the term was coined. We are proud of this heritage.
Thanks for reading!
[1] WhiteHat Security Inc., “2017 Application Security Statistics Report”
[2] Gartner Inc., “10 Things to Get Right for Successful DevSecOps”
