What is Red Teaming and Pen-testing?
2025.08 - Imagine spending weeks, months, or even years working on a product, feature, or service, only to botch the launch by missing major security issues. Learn how not to do that.
Hi, I am Aadil and I write this newsletter The Art of Doing Technical Program Management. You are reading my free weekly post where I share practical insights and advice on a specific topic or subject to help may readers elevate their Technical Program Management game.
Summary: How do you make sure your next product launch doesn’t end up in a security nightmare.
Penetration Testing (Pentesting): A focused inspection to find security weaknesses in your product before launch, often done by internal teams or third-party vendors.
Red Teaming: A creative, real-world attack simulation where skilled testers try unconventional methods to breach your product’s defenses.
Best Timing: Run tests when the product is nearly complete, after core features are functional but before code freeze, allowing time for fixes.
TPM’s Role: Make security testing a mandatory launch requirement, plan for remediation, and ensure all critical vulnerabilities are addressed before release.
Imagine spending weeks, months, or even years working on a product, feature, or service, only to launch and discover major security issues. It’s every team’s worst nightmare.
A smooth and successful product launch isn’t just about rolling out great features—it also demands dedicated time to test how secure your product really is. Security isn’t a box to check at the last minute; it’s an integral part of building a reliable, trustworthy product.
📰 Some real world examples:
If you're a Technical Program Manager (TPM), Product Manager (PM), or Team Lead (TL), understanding how to leverage red teaming and penetration testing (pentesting) can make the difference between a successful launch and a disastrous one.
What is Penetration Testing and Red Teaming?
Let’s imagine your product as a castle. You’ve spent months designing it, building strong walls, adding layers of protection, and ensuring that life inside the castle runs smoothly. You want this fortress to be impenetrable.
But how do you test if your defenses really work?
Penetration Testing (Pentesting)
Pentesting is like a focused security inspection to ensure your castle (product) holds up under pressure. It’s about finding weak spots before attackers do. Here’s what pentesting typically looks for:
Is the moat functional? Are there strong defenses guarding the perimeter?
Are all doors locked? No hidden or secret entrances left open?
Is the interior secure? Are there solid security protocols within the castle?
How does data flow? Are the pathways inside the castle safe and well-guarded?
Pentesters methodically probe your product for vulnerabilities, ensuring it’s as secure as possible before launch.
Red Teaming
Red teaming takes it a step further. It’s less about methodical checks and more about creativity and real-world attack simulation. Think of it as inviting skilled individuals or our your primary user to break into your castle—using any means necessary.
Bypass the moat. Find a way around your primary defenses.
Ram through the front door. Test for obvious security gaps.
Exploit weaknesses in the walls. Search for overlooked flaws.
Fly over the defenses. Think outside the box and find creative ways in.
In red teaming, no idea is off the table. It’s about simulating real-world attacks to see how your product holds up against determined, creative attackers.
How Are These Tests Run?
Penetration Testing (Pentesting)
In Big Tech:
Larger companies usually have dedicated security engineering teams. These teams often work alongside product teams, conducting security architecture reviews and running pentests regularly—especially before launches.
Pentesting in these environments is part of the standard development lifecycle, ensuring that security isn't an afterthought.
In Startups:
Startups often rely on third-party vendors and self-serve tools for pentesting. Vendors conduct deep or shallow tests, depending on the scope, and provide detailed reports that outline:
What was tested? (e.g., APIs, web apps, hardware)
Testing methods used.
Findings. (e.g., vulnerabilities, risks)
A prioritized list of issues:
Must-fix vulnerabilities: Critical issues that could lead to breaches.
Nice-to-fix issues: Less severe but still worth addressing.
These reports help teams focus on the most significant security concerns before launch.
Red Teaming
Real-World Simulations:
Red teaming involves real users, beta testers, or dedicated red teams attempting to break into your product. Why involve real users? Because sometimes the simplest, most obvious approaches reveal the biggest security holes.
Big Tech vs. Startups:
In Big Tech, red teams are often well-established and run dedicated simulations.
In startups, the approach varies. Some leverage internal resources or beta programs, while others hire external red team experts to run more sophisticated tests.
No matter the setup, creativity is key in red teaming. Attackers are encouraged to think like real-world hackers—using unconventional methods to breach defenses.
When is the Right Time to Use Penetration Testing and Red Teaming?
Timing is crucial. Running security tests too early or too late can waste time and resources.
The Best Time:
Run these tests when your product is nearly complete—with core use cases working and initial testing done.
Think of it as testing your castle after all the walls are up, the moat is filled, and the gates are installed—but before the grand opening.
Why Not Earlier?
Testing too soon—when there are obvious gaps or missing features—is like testing a castle without walls. It doesn’t provide meaningful results and wastes effort.
Why Not Too Late?
Waiting until the last minute to test can leave you scrambling to fix critical issues, delaying your launch.
Pro Tip:
For hardware products, conduct testing during the Production Validation Test (PVT) stage. This is when the hardware is close to its final version, complete with security mechanisms like burned fuses in place.
Provide debug-capable devices for in-depth testing if needed, but ensure most tests happen on near-final hardware.
How Should TPMs Incorporate These Into Project Plans?
As a TPM, it’s your job to ensure that security testing is baked into the project plan—not bolted on at the end.
1. Make Security Testing a Launch Requirement
Mandatory Sign-Offs:
Treat pentesting and red teaming as critical gates in the project plan.
Make them mandatory sign-offs before launch.
At the Go/No-Go meeting, be clear about which security issues were fixed and which (if any) were deferred—with clear justifications.
Milestones:
Track security testing as milestones between feature complete and code complete.
Ensure there's buffer time for remediation after tests are done.
2. Plan for Adequate Testing and Fixes
Red Teaming:
Leverage internal resources or set up beta programs to run red team exercises.
The more diverse the group of testers, the more unique attack vectors you'll uncover.
Pentesting:
Schedule at least one round of pentesting before launch.
Build in time for fixing issues, and plan for a retest—especially for critical vulnerabilities.
Pro Tip: Always rerun tests on areas where vulnerabilities were found to ensure fixes worked.
3. Hardware-Specific Considerations
For hardware products, pentesting is all about ensuring the software on the device (firmware + OS) is secure.
Testing should focus on PVT hardware, which closely mirrors the final product.
Debug-enabled devices can be provided for deeper testing, but the primary focus should be on production-like units to reflect real-world conditions.
Final Words
Red teaming and penetration testing aren’t just "nice-to-have" extras—they’re essential parts of launching a secure product. Whether you're building software, hardware, or a combination of both, these tests help ensure that your product can withstand real-world threats.
As a TPM, PM, or TL, it's your responsibility to champion security, bake it into the project plan, and ensure that your product doesn’t just function—it protects users and your company’s reputation.
So before your next launch, ask yourself: "Is my castle secure?" 🏰🔐
Until next time!
-Aadil
Here is a sneak peak at the upcoming “TPM Microguide on Release Management” shipping March 11th to my paid subscribers only.
TPM Microguides are concise, actionable guidebooks designed to help TPMs level up in specific areas of their craft — whether it’s mastering technical concepts, building program structures, or learning how Apple builds software. If you want full access, please consider becoming a paid subscriber.
Roll back Or Fix Forward
When an engineer lands a code change that causes a failure in production, you have two main options for addressing the issue: roll back or fix forward. These approaches are driven by different philosophies and have their own advantages and disadvantages.
1. Rollback
Rollback involves undoing the change that caused the failure and restoring the system to its previous stable state. This is typically a quick fix, especially when the failure is critical and must be addressed immediately.
Advantages:
Quick resolution: Restores the system to a stable state without much downtime.
Minimized risk: By reverting the change, you avoid introducing additional issues in the short term.
Disadvantages:
Lost benefits: Any benefits from the new code are lost, and you may miss out on progress.
Doesn’t address the root cause: The failure is resolved temporarily, but the underlying issue still needs to be identified and fixed.
2. Fix Forward
Fix forward means diagnosing the issue without rolling back and submitting a new fix to address the failure directly. This approach can help keep progress moving forward without needing to revert code.
Advantages:
Less disruption: The system stays up with minimal changes, avoiding the need to return to earlier versions.
Potentially faster resolution: If the problem is small or localized, a fix can be quicker than a rollback.
Disadvantages:
Risk of introducing more problems: The fix might not fully resolve the issue or could introduce new complications.
Can lead to technical debt: Quick fixes may mask deeper problems if not addressed properly.
What is better and when?
The choice between rollback and fix forward largely depends on the type of system you're working with, as well as the severity and frequency of failures.
Rollback tends to work well in high-risk, critical systems where stability is paramount and any issues need to be resolved immediately. For example, in financial services or healthcare systems, where even a brief failure could have significant consequences, rolling back ensures that you restore the system to a known, safe state as quickly as possible.
Fix forward is often better suited for fast-moving environments like SaaS products or consumer-facing applications, where frequent releases and updates are common. These systems often prioritize speed and continuous delivery, and fixing forward allows you to resolve issues without interrupting user experience or delaying progress. It’s also ideal when the problem is localized or can be addressed with a targeted, low-risk fix.