At Hdiv we talk to lots of developers and infosec professionals. We have more than a few funny stories about the friendly rivalry -and sometimes not so friendly- between these two teams. Some people even call them tribes!
Like Switzerland, we like to take a neutral position, and we strive for the common goal of building secure applications.
That’s why we are so excited to help teams adopt DevSecOps methodologies. We believe that DevSecOps is a great way of creating applications that are built with security in mind since its inception: we call this approach ‘security by design’.
DevSecOps is such an important concept for us that we want to give it plenty of attention. So we are going to break our message across a few individual stories: In this first post, we will introduce the overall vision and our recommendations. In subsequent posts, we will expand each of our individual recommendations in a dedicated post.
The DevOps methodology
What makes DevOps different? The first keyword is speed: DevOps philosophy takes the principles behind agile methodologies to a new level of speed by feverishly increasing the rate at which code is pushed to the repos, and subsequently released into production. The adoption of continuous integration tools, such as Jenkins, enables ever faster releases. The application release cycles are moving from “a few releases each year” to “weekly releases” –or even more frequently.
The second keyword is scale: DevOps supports systems with hundreds (or thousands) of applications built by hundreds of developers, that run on dozens of servers that can increase from 10 to 100 in seconds.
When speed and scale become such an important factor, dev teams naturally respond by introducing as many optimizations as possible to eliminate bottlenecks in the SDLC. Very often, the security activities in the software life cycle are deprioritized or overlooked. After all, we can’t blame for “giving up” after seeing a list of hundreds false positives that sometimes code review tools produce.
DevOps security pain points
Based on our experience, we see the following security pain points in DevOps environments:
1. Inefficient SAST tuning & false positives
Static AST tools (SAST) are the most widely used solution in terms of detection of security issues during the development phase. However, SAST tools usually require tuning processes to integrate with each development architecture, and they present a high rate of false positives. In consequence, SAST tools consume tuning time, and review time to separate false positives from real vulnerabilities.
2. Lack of coordination between infosec team and developers
Security issues are treated separately from other software quality issues. Developers must check different systems to review traditional functionality bugs vs. security issues. Developers naturally tend to solve functionality bugs first, and only after, they focus on security issues.
3. Continuous deployment chaos
Teams do not have an efficient way to evaluate the level of security of their applications, and as a result, security issues sneak into production. Code compilation issues stop builds and continuous integration pipelines, but there is no automatic way for security bugs to block the deployment to production.
4. Manual pen-testing becomes a bottleneck
First of all it is important to note that detection solutions (AST vendors) only detect one kind of vulnerabilities: essentially, code syntax issues, also known as security bugs. Examples of security bugs are SQL injection, XSS, etc. and they follow specific patterns that are exactly the same across all applications.
Unfortunately, there is another important group of security vulnerabilities: business logic flaws or design flaws. They can not be detected by tools. We are talking about security vulnerabilities such as access control, binding, abuse, etc.
Due to this reality, and taking into account that automatic detection tools only detect security bugs (50% of the total problem), companies and organizations perform manual reviews to detect business logic flaws. Current agile development methodologies that generate new releases almost every day do not adapt well to this manual review. Even when review budgets are not an issue, the review impacts severely the time to market of your applications.
5. Poor visibility of security track record
Teams lack access to smart tools to monitor the evolution of security vulnerabilities from build to build. Most teams are not aware of the number, severity, and age of the vulnerabilities.
6. Lack of cloud support
Cloud environments do not have a perimeter that can be defended with traditional tools such as WAFs. Cloud providers and/or deployment conditions can change dramatically over time. Most security tools do not adapt to these changes.
7. Systems that are not scalable
External AST reviews strategies such as testing-as-a-service introduce lags in the deployment phase, ranging between 4 and 24 hours. This lag reduces scalability and increases time-to-market. Once in production, it is common to abruptly scale from 10 servers, to 100 servers within seconds. DevOps teams complain that security tooling is a bottleneck and lacks flexibility to scale dynamically as the application load grows.
From DevOps to DevSecOps: a quick Hdiv recipe
The Application Security community has reacted to the challenges and pain points described above by wrapping the DevOps philosophy with a security blanket: DevSecOps.
DevSecOps is all about increasing speed and productivity by introducing automatization, monitoring, and tight integration of security. DevSecOps leverages Secure SDLC tooling and at the same time, it ensures that the cadence of releases is not hampered.
In subsequent posts we are going to cover in detail each of the of the following 7 best practices but here is the summarized version of our recipe for DevSecOps success:
Pain Point #1: Inefficient SAST tuning & false positives
Solution: Use an IAST tool
To avoid the lack of accuracy and the labor-intensive reviews of SAST reports, use a more direct detection tool such as an Interactive-AST (IAST.) IAST tools do not require tuning or manual reviewing of false positives, since they do no generate them. Use the OWASP Benchmark coverage indicator to rank vendors.
Pain Point #2: Lack of coordination between infosec team and developers
Solution: Integrate security issues within your general bug tracker
Communicate the bug tracking tools that your team is using, such as JIRA, with your security tools so that developers can see security bugs as regular tasks. Automatically create new bugs and tasks to represent security vulnerabilities as they are found during the reviews and audits of the application.
Pain Point #3. Continuous deployment chaos
Solution: Define metrics and thresholds to assure quality
The same way code compilation errors stop your deployment, security issues should block the deployment as well. Create automatic security checkpoints to enforce quality goals, and stop the pipeline if the number of security bugs is higher than a goal. Seek solutions that integrate with your deployment server (such as Jenkins).
Pain Point #4: Manual pen-testing becomes a bottleneck
Solution: Automate protection of business logic flaws
Business logic flaws (also known as design flaws) are an important group of security issues that detection tools can not identify. To mitigate the bottleneck of manually reviewing business logic flaws, automate the input validation using solutions and architectures that are secure from the start. Then, integrate the output of the solutions with the audit tools. The combination of automating the protection and connecting the protection with audit tools allows to automate a large part of the manual pen-testing activities.
Pain Point #5: Poor visibility of security track record
Solution: Better reporting
Versioning your releases should not be afterthought. Implement metrics and reports that track the evolution, number, and severity of bugs for each release. Enhance solutions such as Jenkins Reports & Web Reports to include security issues.
Pain Point #6: Lack of cloud support
Solution: Embedded security
Build applications with agile security built-in. When the deployment conditions and cloud providers change all the time, applications that have security embedded are able to adapt the changes.
Pain Point #7: Systems that are not scalable
Solution: Linear scalability and affordance cost
Ensure that the security infrastructure in your application is not a performance bottleneck. Seek security solutions that scale in constant or linear time. Monitor the evolution of the added latency of the security solution and choose those that perform better.
Security is everyone’s responsibility
The foundation of our recommendations and best practices described above is the idea of empowering developers to build secure code. This means helping your developers with the right toolchain, that provides helpful input (low false positives), it is integrated in their work pipeline (security issues as defects in the bug tracker), and creates a culture where writing secure code is recognized as a goal.
From the very beginning, Hdiv was created by and for developers.The practical recipe above, and more broadly our company DNA, supported the DevSecOps philosophy even before the term was coined. We are proud of this heritage.
Thanks for reading! To learn more about Hdiv and get started with an online demo, visit our website at https://hdivsecurity.com/demo