Transcript
Episode Six: AppSec Tech Below the Deck
Mallory cracked her fingers in anticipation of the night’s work to come. Like a panther lurking in dark canopies of ones and zeros, she had stalked her quarry, MegaHelix Corporation, for days, and was now ready to pounce. She had preyed upon MegaHelix in the past, finding old APIs left abandoned on the network edge that had crumbled swiftly before her bag of tricks. But tonight there were bigger stakes.
The company had just launched a brand new piece of software, an upgrade complete with expensive marketing launch and a bright new UI festooned everywhere with diamond-dazzled AI-assisted iconography. With a few hours on Hell-a-Fam, she had managed to trade for the credentials for a Not-For-Resale account left over from the site’s beta period. The new product was taking off. It featured ways for customers to bring their own AI API keys “to leverage AI-powered insights and unlock synergies”—but to her, it meant a treasure trove of keys in one place, ripe for a well-executed heist.
Her goal was simple: get remote code execution, and exfil hundreds of AI API Keys for resale.
She began the dance.
Hours had passed. Mallory had tried everything to no avail, but at last had found something interesting. The system had a dusty corner that allowed a user to import and export a customization template. Mallory practically purred as she read the first four bytes of the file: AC ED 00 05. These magic numbers might not mean much to most people, but to her, it smelled of opportunity. Like the predator stalking the herd, she had found the vulnerable, weak member of the pack. She laughed aloud for a moment, only to get so serious and down to business. A few keystrokes later in a popular tool, and she had an Apache Click gadget chain ready to exploit unsafe reflection in TemplatesImpl.getOutputProperties().
A setback: there was no shell on the instance. Must be a hardened container image. Everything that would have made her life easier had been vacuumed out of the container’s file system, leaving it cold and sterile. But then she got lucky. Mallory found a discarded tool left on the otherwise spotless virtual workspace: an HTTP testing tool that was able to send a single lonely UDP packet spiraling out to port 53 on a DNS server under her control. It wasn’t much, but it confirmed the vulnerability. If this was what she had to work with, so be it. It was going to be hard. Locks could be picked with a swiss army knife. Difficult, but doable. And what a tale to tell. She decided to celebrate first with some pizza to be paid with one of her crypto coins that hadn’t yet been rug pulled.
Hawaiian or chicken barbeque ranch though? Decisions, decisions…
—
Meanwhile, at MegaHelix HQ, it was Friday evening, but Alice had some news for her CISO that just could not wait.
Alice: Hey Bob, Alice here! Sorry to call you at this hour, but I have some really good news! The new release is out the door! We ended up finding something pretty nasty using that DAST tool. There was a ton of debate on the dev team. You should have seen faces in the virtual meeting when we tried out the proof of concept it showed us ourselves. The Dev team was shocked that a serialized save file could execute code!
Bob: Yeah, yeah, um… I’m glad it got fixed.
Alice: Right? There was a merge request opened in like twenty minutes. One of the juniors even got inspired and rigged up some CI/CD to scan all of the APIs for the new project before we scan everything deployed with the UI in stage.
Bob: Oh? That’s… great. Actually… that really would be great! Maybe we can merge that back to the infosec scripts…
Alice: Definitely! I just thought you’d like to know. Have a good weekend, Bob!
Bob: You too! Thanks so much, I’ll rest much easier knowing we’re secure.
—
In her secret lair, Mallory cursed and sighed in exasperation. Only twenty minutes ago, the gadget in her attack payload had been working perfectly… But now? In the time taken to write the custom payload designed to lift all the API keys, someone had fixed the vulnerability. Her victory was snatched away at the last moment! MegaHelix must have patched it! She was so close… But no. This time, she had lost.
And worse, the pizza ended up being terrible.
Frank Catucci: Hello, welcome to another episode of AppSec Serialized. My name is Frank Catucci, CTO here at Invicti Security. For my special guest today, I’d like to introduce Paul Good. Paul, can you introduce yourself and maybe tell us a little bit about yourself here at Invicti Security?
Paul Good: Yeah. Hi, Frank. Hi, Dan. Thanks for having me. So I’m Paul Good. I’m an AppSec engineer here at Invicti. I’m originally from Ireland, as you might know from the accent, and I’m living in Malta working with Invicti. Yeah, so just a typical day in the life of an AppSec engineer. I start off like any engineer, a big jug of coffee, get me going in the morning. Yeah, that’s always, always the first thing. Emergency coffee.
And yeah, so I kind of start out reviewing, obviously. Any requests that come in, Slack, e-mail, Jira, that kind of stuff. If there’s any fires to put out, check our kind of metrics from our AST tooling. Again, we might kind of engage in some kind of initiative that’s upcoming, get the ball rolling and start some scripting or helping out our devs.
Dan Murphy: Cool. So Paul, I know that I sense in you you have this kindred spirit. I’ve always been a builder and I noticed that in you—you’re a builder at heart as well. And one of the cool things that I’ve noticed is that you have built various things to help you do your job better here at Invicti, to be able to kind of shift security a little bit more left. What are some of the things that you have built, whether they are wrappers on top of commercial tools or integrations into CI/CD pipelines? How have you made AppSec here at Invicti part of our continuous integration strategy?
Paul: Yeah, so I think scripting is really important. It allows you to kind of use the tools that you have and get the most value from them. So the value is in data and knowing what you have, as well as kind of building that seamless integration into pipelines. So in AppSec, CI/CD is your bread and butter. You want to get in there, you want to shift left closer to devs.
One of the things that we’ve done is build a uniform AppSec CI/CD. This is kind of an easy-to-implement GitLab pipeline that has all of the logic hidden. So it kind of sits in the background. You have all your installation and deployable Docker containers. You have all of the commands to run your various AST tooling, all within this. So the devs don’t really see that, they just integrate it. It’s going to run all your tools. It’s uniform, they inherit it, they can configure. So it kind of makes it quick to add. It doesn’t add a headache to those devs. And it works fast. You can keep iterating on that, make sure it’s fast at the right stage for devs. So if it’s SAST, you might not rely on any jobs—just kick it off straight away during the pipeline. Devs don’t have to wait, it’s going to run with their pipeline rather than against it.
And then with that, we’ve kind of built up to breaking builds. Obviously, you want to break builds, but you don’t want to slow down developers and annoy them as well. We’re kind of breaking builds, reducing that friction, so it’s only on critical projects and for higher severity findings. And then you can work the other ones into your normal vulnerability management procedure. So it allows us to understand and work with devs rather than working against them.
Dan: Right. And you’re also meeting devs on their native tools. So as a developer, if I’m going to git push some code, I watch that pipeline start to cook away. And the failure of a security tool is no different than the failure of a unit test. Both of those things are just going to stop the pipeline and I get to fix it. And I get to fix it without having to wait for a meeting be scheduled or having someone nag me or having you tell my boss or my boss to come down. It just gets done very, very quickly, which is really, really nice.
Paul: Exactly.
Frank: One of the key things that you mentioned was friction, right? So if we’re talking about friction with developers, if we’re talking about the ability to do things like breaking builds and maintaining that level of communication, a key part of that, Paul, has got to be visibility, right? So one of the toughest challenges, I think, for anybody implementing any type of AppSec program historically has been that visibility—specifically, visibility with identifying where to draw the perimeter, visibility as far as your engineers, visibility for your developers, and visibility for the security team.
If we’re going to focus on visibility specifically, do you have any tips or things that have worked for you and worked for Invicti or other areas that you might say is really a great place to start tackling that problem or a strategy for tackling the visibility problem as a whole?
Paul: Yeah, so the visibility problem is the first thing you’ll start working on. You need to know what you’re protecting and where it is. Initially, it’s about building up that wide-range visibility—knowing what platforms and code repos you have and what source code managers you’re using. First, it’s about cataloging the high-level places where your assets are. From there, you can build into that. All the big source code managers have APIs, so you can build scripts to pull this information. We’ve built around that, starting with visibility and then categorizing risk.
Different products your company ships or different areas of the product might be riskier or more business-critical, so knowing them is important. That gives you a high-level idea of what’s critical. Getting that information is really about talking to teams. Devs know their work like the back of their hands, so they can tell you. Then, you can work that into a standardized approach. We’ve looked at tagging and logical grouping of assets, using tools and services at our disposal to get the most value.
It starts there—building different logical groupings—but it never stops. Visibility is all about keeping visibility. so you need to iterate, you need to automate. One missed project could be the weak point that leads to a disaster. You have to have scheduled reports, automated tagging, and build it into the culture so that as new products or projects are added, they’re tagged and come straight into visibility.
Dan: Speaking of visibility, one of the cool things you built a while ago was providing visibility into SBOM—automating the process of identifying the components that make up the models we build and ship to customers. Can you talk about that SBOM tool and how it works? Do developers have to log into some vendor tool, or how did you build it?
Paul: Yeah, we’ve based our SBOMs around our SCA tooling. We’ve done that logical grouping, knowing which projects relate to which products we ship. We’ve built out different organizations or logical groupings within the SCA tool we use. From there, we know which packages relate to which product, and we built scripting around that.
It’s an iterative process. You build out your SCA integration, confirm and review it, and ensure developers agree with what’s being collected in the SBOM—no leftover dev dependencies, no junk. Developers can leave little nuggets around the repos, so you have to filter those out. Then we build scripting on top of that to produce Cyclone DX or a custom open-source dependency document. So we use the power of scripting and SCA tooling to build that.
Dan: Yeah, it’s cool, right? We even produce a customer-facing, branded PDF that shows up on schedule in a Slack channel—again, using existing tools so developers don’t have to go elsewhere. Talk to them natively, and they’ll actually pay attention and read your stuff.
Which is a good segue into what I wanted to ask you next—developers, developers, developers, developers. For those of you who get that reference, great. For those who haven’t, check it out and add “techno remix” to see some of the early days of the industry.
When talking about developers, there’s sometimes an adversarial relationship between AppSec and developers. I gave a talk for our marketing team a while ago where we used tarot cards to assign identities to different roles. One persona was the AppSec engineer, and another was the developer. I used the Tower for the developer—everything is fine, everything’s on fire.
Developers get a lot of people going to them with a lot of issues, whether those are requests from product to build faster, whether that is requests from customers to keep things running, or, you know, requests from the field, like SREs asking them to handle operational challenges. How do you, as an AppSec engineer, avoid being just another voice bugging them? What strategies do you use?
Frank: That’s the question our listeners really want to hear, Paul. No pressure, but this is a topic that will resonate with a lot of folks.
Paul: Yeah, as you say, devs just want to get their job done. They want to move fast, push code, see it in the product, and move on with their day. They don’t want to manually add CI/CD jobs or implement extra steps. They just want to focus on their work. So, automating where you can is key. If you’re implementing CI/CD, make sure the tools run fast, align with developer expectations, and provide useful feedback without adding noise or friction.
So I think that’s one really important area. Another is integrating seamlessly into their workflows. If they use a Jira board or a Kanban flow, push vulnerabilities straight into their workflow so they can easily find and address them. They shouldn’t have to go to another tool just to search for a vulnerability. And then another part is checking stuff ourselves before pushing it to a dev.
Dan: Oh, that is huge. This is absolutely huge.
Paul: Doing a little bit of triage because they they love that. If you push junk to developers, they’ll lose interest and respect for what AppSec is trying to do. They’re already busy—you don’t want to waste their time.
And then just having those contact points, like we’ve also built a security champions program at Invicti. We talk directly to devs but also have key contacts in each team. That way, we understand context and ensure we’re not just adding pain points for developers as they’re trying to do their job. A mix of all these strategies works well.
Dan: That aligns with the strategic view we discussed in a previous episode—people, process, and technology at the team level.
And that pre-filtering of issues is huge. Just off the cuff, when security issues are synced to Jira, does everything from a tool go straight in, or is there filtering? What percentage of issues tend to make it through?
Paul: One interesting example is how we implemented SAST. SAST is notorious for false positives and junk and lots of random stuff that pops up. We’ve used logical project grouping and AI-driven proprietary tools to auto-triage issues, marking some as potential false positives or saying this is not in an area that’s important.
Another strategy is talking to devs. If we get a lot of SAST alerts about a project, we ask them and they might say, “oh this folder is all tests,” or “this folder might not be called tests, but it is,” or “this folder isn’t used in production.” So we can filter those out. Only real issues get pushed to devs.
Dan: So, it’s not a direct pipeline from tool output to Jira—there’s curation. You’re like a sommelier for vulnerabilities.
Paul: Exactly.
Frank: It’s a fine vintage! Love that.
Let’s talk a little more about the tools themselves and bugs. If we’re looking at things from a bug level, Paul, and I know you’ve instrumented things like this in the past, especially here, we talk about keeping devs in their native toolsets, reducing friction, and having that curated list of a fine vintage, if you will.
We also talk about the integration of tools devs use—Jira, GitLab, GitHub, etc.—versus asking them to log into some vendor portal or have direct access to a specific vendor tool. I think I know the answer from my perspective, but from yours, which approach provides the most benefit? What are some of those benefits, and how do you make it as effective as possible for developers when resolving bugs? Let’s dig a little deeper into that.
Paul: Yeah, I think there are three key areas here. First, providing feedback directly on a merge request. That’s the first place you can do it. A dev pushes code, an alert pops up saying there’s a vulnerability in their code, and they can fix it before merging. It doesn’t even disrupt their workflow. It’s really nice.
Second, integrating into their workflow. Integrations are really important. You can integrate into their sprint planning, dev reports, or prioritization mechanisms. That could mean pushing issues to a specific Jira board where those devs work so they can pull them in themselves. Devs don’t like going outside their workflow. They don’t want extra noise or extra places to check. They just want the information and context they need right there.
Dan: Yeah, honestly, as more SAST tools get adopted, it’s just one more thing to remember, right? The cognitive friction of having so many random accounts just to get work done—it’s a lot. If that number can be reduced by one, that’s always better.
Paul: Exactly. And then, as you build out those integrations, devs are naturally curious. You don’t want to block them from going into the tool you’ve adopted. If you have a platform, give them the link. If they want to dig deeper or tinker with it, let them. But make sure all the necessary information is in their workflow. Yeah, they’re just going there to have a bit of fun.
Dan: Right, don’t force it. Another nice thing about having security champions is that you’ve almost pre-selected the eager ones who might want to check things out. Some will want to dig deeper, explore the whole wine cellar instead of just the bits you’ve brought to the table.
Paul: Exactly. And in an AppSec company, a lot of our devs have great security knowledge and insights. They can really dig in and help us improve the AppSec program.
Dan: Yeah, and they can spot patterns—whole classes of things that are either good or not worth wasting time on.
Frank: Absolutely. Well, with that, Paul, I think that’s all the time we have for today. Looking forward to having you back on at a later date. This has been a really excellent and informative discussion.
Dan: Big shoutout, Paul. It’s always a pleasure working with you on AppSec. We appreciate it, and we love what you’re doing. Keep it up.
Frank: Absolutely. Thank you.
Paul: Thanks so much, Dan. Thank you, Frank.
Credits
- Fiction story and voiceover: Dan Murphy
- Discussion: Frank Catucci & Dan Murphy
- Special guest: Paul Good
- Voice of Alice: Alexa Rogers
- Production, music, editing, processing, voice of Bob: Zbigniew Banach
- Marketing support and promotion: Meaghan McBee
Bonus content
Carefully selecting your background music is crucial for both attackers and defenders, so here’s Mallory’s track from this episode to help your AppSec efforts:
