Published on
This post was written by Chris Romeo during his tenure at Security Journey.
This article was originally appeared on at TechBeacon.com on January 6, 2021. You can access it here.
Will DevOps and DevSecOps still be relevant in 50 years? Today's DevOps technology will be long gone, but some cultural pieces may still be around. My best guess on the part of DevOps that will still exist: DevOps security culture.
What is DevOps security culture?
- Knowledge, because there are things that you must learn and understand to build secure software fast.
- Experience, because you improve the process and tools the more times that you fail.
- Art and creativity, because the innovation of process and tools requires exploring new security ideas, not just relying on the status quo.
- Science, because what matters gets measured, and DevOps security improves by measuring input and output and making adjustments.
DevOps security culture is defined by what happens with security when people are left to make their own decisions. We all do the right thing for security when a security team member is standing over our shoulder. But your true security culture is revealed on Friday at 4:30 p.m. when a feature promised to a customer must either get pushed to production or shelved until next week.
If there is a security issue, and the customer requirement trumps the potential platform weakness, you have a weak security culture. If a developer can say, "Nope, this has a security issue, and we are unable to push to production," you have a strong security culture.
You want your DevOps security culture to continue to grow because it is crucial to your success in building secure software fast. Your DevOps security culture is a living, breathing thing. Security culture can stagnate, but it can also continue to grow if you can absorb others' learnings and failures.
Here are 12 DevOps security culture fails—and how to change.
1. Name and brand
The industry created a monster when it agreed to accept the term DevSecOps. The debate over the name—DevSecOps, SecDevOps, SecOps with dev, dev with a side of sec—has been a distraction, time misspent worrying about what amounts to a marketing term.
Let's just call it DevOps and teach everyone that there is no DevOps without security integrated.
How to change culture
- Embrace and believe that security is not an afterthought with DevOps. Stop using any term other than DevOps.
- Teach all product and application-adjacent people this new definition. Your teams look to you for guidance. Standardize, and reap the reward of less confusion.
- Create a "DevSecOps" swear jar, and make people add money to it every time they use the term.
2. The infinity graph
The infinity graph is the never-ending visualization of DevOps that shows you'll never complete anything. That is perhaps an exaggeration, but it's not a good representation of building modern software in a development organization.
To make matters worse, folks will wrap a rectangle around the infinity graph and write "security" on it to represent how they integrate security into their software.
There is much more to integrating security than a rectangle.
How to change culture
- Ban the infinity graph from your world. Choose an image or illustration that matches up to what you do with DevOps.
- Embrace pipeline visualization. DevOps with security is always a pipeline because a pipeline is a factual representation of how we build software with security integrated.
3. Security as a special team and lack of collaboration
"Special team" is sometimes used to describe the security team. This is especially true in older companies transitioning from other software development methodologies toward DevOps, where a security team was a separate entity, but it can also occur in startups if they're not careful.
Security as a particular team is a failure because when security is "special" and separate, it spends its time governing the DevOps process and the build pipelines instead of collaborating with the developers and operations folks to improve flow and security in tandem.
Another failure with security teams is having team members who do not know how to code. Successful application security people can read code in an object-oriented language. To gain this level of competence, app sec people must learn at least one language. Once you discover one, you can apply that knowledge to other languages and become a trusted adviser to development and operations.
Collaboration among development, operations, and security is challenging when the security team thinks it's unique.
Siloed organizations and lack of communication result in more bugs and slower building software cycles.
How to change culture
- Security isn't special; lean into the fact that developers want to be responsible for security.
- Responsibility requires knowledge and experience.
- Teach everyone the foundational lessons of security.
- Teach security folk how to code. They must speak the language of the developer.
- To foster collaboration, institute communication paths among specialties and share both the good and the bad; no hiding anything.
- Build DevOps strike teams—dev, sec, and ops together.
- Foster a mutual partnership aimed at serving each other.
4. Vendor-defined DevOps
Who defines the DevOps standard? Where is the governing body that determines what DevOps is? Here's the catch: There is no DevOps standard. Vendors define what DevOps means in the context of their product offerings. The things that vendors want to sell represent the category.
There must be a vendor-independent architecture for what DevOps means to customers.
Build the best possible solutions from your architecture, not from a vendor's playbook.
How to change culture
- Realize that there is no DevOps standard.
- Define your DevOps, and do not allow vendors to define DevOps for you within the context of whatever solution they provide.
- Study and teach the bigger, contextual reality of a vendor-independent DevOps throughout your organization.
5. Marketing term infatuation
Marketing terms are a necessary evil, and they're not something that can be wished away, no matter how hard you try. "Shift left" and "shift right" are the two marketing terms that are driving the DevOps space.
Shift left means you start at the beginning of your process and consider security at each step of the way. Shift right means you adapt security tools and technologies in your production environment to bolster your capabilities. Neither is wrong, but better advice is to start left and start right. The catch here is that when you start left and start right, you're back to what's been called the "secure development lifecycle" for the past 20 years.
When organizational strategy is driven according to marketing terms, you set yourself up for failure.
Follow the best practices, but leave the marketing terms to the marketeers.
How to change culture
- "Start left" by embedding security into every section of your DevOps pipeline.
- "Start right" by integrating runtime application self-protection (RASP) into production.
- Meet in the middle and call what you do secure development lifecycle.
6. Big-company envy
Everyone wants to emulate the best. The acronym FAANG (Facebook, Amazon, Apple, Netflix, and Google) exists because small and large teams look to these giants as setters of the standard.
The failure here is trying to base your DevOps security program on what the FAANGs are doing. The catch is that you're not them. You're a different organization with a different culture.
Leave big-company envy for other people.
How to change culture
- Avoid distracting yourself by fixating on what those at the top of the industry are doing with security.
- Make incremental progress: a tool here, a tool there.
- Cast a vision for where your DevOps with security needs to go.
7. Overcomplicated pipelines and doing everything now
DevOps and pipelines are complicated beasts. If you're not careful, you can create pipelines that are so complicated, nobody really knows how your software gets built. A lack of understanding is a tough thing to admit, and the other danger is that if you are teetering on this edge of the cliff, you're not going to be willing to accept it.
The other challenge is doing everything now. As you build and mature your pipelines, the temptation exists to deploy all the tools and all the tech immediately.
Immediacy sets you up for failure because adding all these moving pieces and building complexity means it's even more challenging to understand the big picture.
How to change culture
- Realize that there are dozens of categories of security tools.
- Start with a small subset of security tools and grow the pipeline over time. Software composition analysis (SCA) and static application security testing (SAST) are the best places to start.
- Keep it simple; the pipeline must be simple enough for all product and application-adjacent people to visualize and explain it.
- Avoid the pressure to solve all problems immediately. Take a phased approach.
8. Security as gatekeeper
The process of gatekeeping is adding manual checkpoints to an automated process, and it's a dangerous temptation for old-school security people. Security people that have been doing security for more than a decade are used to a gatekeeping approach. When things moved slowly, it was easy to make the security checkpoints exist at the main gates in a process.
DevOps goes fast and deploys often. If you're gatekeeping, there is no way things are going fast.
How to change culture
- Practice empathy. Walk a mile in each other's shoes to truly understand the challenges of each role.
- Move security toward development, development toward operations. and operations toward security.
- Work together to find solutions, not roadblocks.
- Provide coaches from each discipline. Coaching is a teaching method, and collaboration occurs when all the roles learn from one another.
9. Noisy security tools
Noisy security tools hamper the productivity of developers. A loud security tool runs in the build pipeline and then sends a 10,000-line report or generates 1,000 tickets requiring resolution.
The over-generation of data results in developers hating the pipeline security tools or, even worse, ignoring the output of the tools wholeheartedly.
How to change culture
- Tune the existing security tools that exist in the pipeline before adding any new tools.
- Never waste anyone's time with security findings that do not matter. Your team members measure the value of every tool you employ, and if you send them garbage results, they conclude that the tool is garbage.
- When adding new tools, start with a minimal policy; this is one where you tune the tool to look for one specific type of error. With a new tool, most people fall into the trap of turning on a maximum policy or everything that the tool supports, and they get 10,000 results when they run the tool, a quantity that no developer could ever take action on. By setting the policy at a minimal level, the developer learns to trust the tool, and over time you can increase the number of things the tool is checking for.
10. Lack of threat modeling
Threat modeling is analyzing representations of a system to highlight concerns about security and privacy characteristics. Threat modeling teaches everyone in your DevOps process how to think through and mitigate every new feature's security challenges.
The failure here is not tapping into the cultural eye-opening and knowledge building that security threat modeling unlocks. When all team members grasp threat modeling concepts, they start to see potential security issues and can mitigate them and make them go away early in the process.
Threat modeling makes for more secure features, applications, and products.
How to change culture
- Perform threat modeling outside the scope of the pipeline. Threat modeling is not a pipeline activity, because it's more process than it is an automated tool. Address it correctly to maximize success.
- Connect threat modeling to your approach for doling out new feature assignments. Make threat modeling inherent in your process, and threat model all things.
- Teach everyone how to threat model and include the Threat Modeling Manifesto as a reference.
11. Vulnerable code that's in the wild
Open-source and third-party software is both great and terrible, at the same time. It's great because it provides functionality and saves us from having to create the same feature functionality over and over again. It's terrible because of vulnerabilities in the software supply chain.
Vulnerable code could be a library or package with a known flaw, or a package with a backdoor.
The failure from a DevOps perspective is allowing vulnerable code to slip into the wild.
How to change culture
- Embed SCA into all your pipelines. In addition to commercial packages, open-source options are available, including bundle-audit for Ruby, Npm audit for Node.js, and the OWASP Dependency Check, with numerous plugins.
- Set your SCA policy to fail and break the build when a vulnerability is detected.
- Ensure you have a time-boxed filtering policy. Sometimes there is no fix for a known problem yet, but you have to move forward. Instead of having a policy that allows filtering out vulnerabilities forever, time-box it so that after seven, 14, or 30 days, the issue starts to break the build again, causing developers to check if the fix is now available.
12. Lack of security retrospective
The final failure is forgetting to step back and examine how well security has worked out for your team. Security retrospective is about having a defined time to collect data about what went well and poorly, in a safe and blame-free place.
Failing to perform retrospective creates a culture where learning from your failures is impossible because you never stop long enough to determine how you're doing.
How to change culture
- Always practice security retrospective and consider security failures as you would any other failure.
- Leave the blame at the door; growth comes from the ability to discuss failures freely.
- Measure output from retrospectives and manage process and pipeline change.
Go all in on resilient code
Some of the above failures discussed are outside of your control, but you can change how you present and use them for your teams. Naming, infinity graphs, and marketing terms are things you can fix to create an image for DevOps-plus-security that will stick with everyone.
Remember that security and coding are for everyone. You must define your DevOps, not the DevOps that your vendors want you to live. Keep up the collaboration, keep it simple, lower the noise, add the best practices, threat model, add software composition analysis, and do blameless retrospectives.
You must define your DevOps, not the DevOps that your vendors want you to live.
Embrace and laugh at your failures. Life's too short to let your failures define you. Failure determines how you get to success.
The only impact DevOps could have 50 years from now is on security culture. DevOps security culture is a work in process, so continue to create it. Continue to strive to meld together dev, sec, and ops in a shared mission to secure everything you build. It's a worthy cause and a mission with multiple levels of rewards.
Keep learning
- Get up to speed fast with TechBeacon's Guide to App Sec Testing and Gartner's 2020 Magic Quadrant for AST.
- Dive in app sec testing, with this free report, The Forrester Wave: Static Application Security Testing for Q1 2021. Plus: Learn how a SAST-DAST combo can boost your security in this Webinar.
- Build your app sec foundation with TechBeacon's Guide.
- Take a deep-dive into the tools landscape with our Application Security Trends and Tools Guide.
