Are All Pen Tests Created Equal?

Share this:

Are All Pen Tests Created Equal


A penetration test is a penetration test, right? Not exactly. As these types of scans have become popular requests for compliance initiatives and other requirements, almost every cybersecurity company offers them.

However, the proverbial devil is in the details. As an important tool in advancing organizations’ cybersecurity posture, it’s worth the time and effort to ask the right questions and understand exactly what you’re getting with your next pen test.

Recently someone asked me, “Aren’t pen tests pretty much a commodity?” I responded that one would think so based on how I see them used in some cyber programs. If we don’t ask the right questions, getting a basic, automated scan of your external IP addresses with a nicely formatted report as the primary deliverable can be easy. It’s not to say these tests/tools are useless (although, for some environments, they might be), but they are only scratching the (attack) surface.

Before we dig into the right questions to ask, let’s start with the basics. 

What is a Pen Test?

A penetration (pen) test is a technique used to try to penetrate a network boundary using offensive attack scenarios. They are usually conducted—at least partially—with an automated tool but can also be conducted manually. A lot of different services can be wrapped around the pen test to improve the effectiveness of the test. Pen tests are common and are meant to simulate an offensive attack to reveal vulnerabilities.

Pen tests are generally performed by individuals with extensive experience in offensive security techniques and are often called “ethical hackers.”

Buying a Pen Test

As organizations across every industry spend millions of dollars to shore up their cybersecurity postures, pen tests have become extremely popular. Insurance carriers and regulators commonly require pen tests. Consequently, their lead times have reached 4-6 weeks, even longer for some of the more highly sought-after companies. If you’re in the market for one, keep these lead times in mind.

It’s also common to have recommendations to rotate the company with which you request your annual pen tests, so that you always get a “fresh set of eyes.” In some cases, it may be suitable to stick with a company for longer than usual if there are continuity benefits (such as ongoing vulnerability tracking and remediation), but at least make sure they have pen tester diversity, and the same individual is not leading the pen test each year.

How Are Pen Tests Conducted?

Pen tests are commonly conducted with a software package that systematically scans the target systems for vulnerabilities. The attack types could be brute-force attacks, SQL injections, and others.

In addition to software, attackers with physical access to the premises may plug a hardware device into the network to facilitate remote access.

Finally, social engineering may also be used to obtain access through phishing or physical means, such as impersonating a help-desk representative. 

Types of Pen Tests

Despite the common assumption, there is more than one type of pen test. As you can imagine, the different types run the continuum of how much information is given to the attacker as part of the simulation and the awareness of the defensive team.

When no one in the target organization is made aware of the test, except for a few that coordinate the effort, and the attacker is given no information about the organization except the name, this is referred to as a “covert test.” A covert test is theoretically one of the more in-depth and real-world tests that can occur because it most truly emulates the scenario whereby an external attacker targets an organization.  Covert tests are also referred to as “double-blind” tests.

A closed box test is often referred to as a “single blind” test where the stakeholders on the organizational side are all aware of the test, but the attacker is only given the company name. 

With open-box tests, the attacker is given more information about the organization being attacked. Perhaps they are given the physical location and potentially even specific information about assets to be tested.

Another important distinction in the type of pen test is where the attacker is located. With an external pen test, traditionally the most common type, the attacker is outside the organization’s network attempting to penetrate the network perimeter and gain access from a remote location. Beyond that, the attacker may be given added instructions for internal testing. With a strictly internal test, the attacker is given physical and logical access to the internal network and tasked with testing the internal security, sometimes referred to as lateral movement. In other words, how far can an attacker go within the network once they have breached the perimeter?

Scope of the Pen Test

Now, let the fun begin. There are many options to consider when selecting a pen test. Assuming a “standard” pen test is scanning externally visible IP addresses, we’ll cover many additional options or features you might consider, depending on your situation.

Options to Consider Include: Why It’s Important:
Internal, lateral movement testingStatistics say that bad actors may already be on your network. Do they have free reign once inside?   
Human-driven Automation is great. Scanners are great. But they are not humans. They are not as creative and don’t think like the creative bad actors. 
Wi-Fi infrastructure If you allow Wi-Fi, it’s part of the network and often overlooked and vulnerable. 
Web applications Consider including all your business-critical web applications. They are yet another avenue into your network. 
Mobile devices Do you allow mobile devices to connect to the corporate network? If so, they are endpoints that need to be tested. 
Act on vulnerabilities as they surface You can have the pen tester address vulnerabilities as they surface instead of just receiving a report at the end. 
Retesting to confirm vulnerability fixesIf you choose to have vulnerabilities addressed during the test, make sure a retest is included to confirm those fixes. 
Integration with other internal tracking tools  Integration with internal tools (like Jira, GitHub, Zendesk, etc.) allows you to track remediation of vulnerabilities yourself. 
SDLC integration Consider including your software development process in the test if you develop in-house software. If you use an Agile methodology, that means your apps are updated as often as daily. 

Other types of tests

Several other types of tests and offensive approaches can be taken, so make sure you are matching your specific needs and not just asking for a “pen test” when you might need a different one altogether.

Vulnerability tests scan your environment for existing vulnerabilities and provide the output for future consideration. Vulnerability scans are more detective in nature, while pen tests take a more offensive approach.

Bug-bounty programs also exist whereby you can compensate individuals for finding vulnerabilities in your environment through ongoing pen testing. This is a different approach to hiring one company to perform a snapshot-in-time pen test.

A tabletop exercise is a live breach simulation that tests the organization’s response. These are a good addition to annual pen tests as they address the actual response. And, since bad actors are likely already in our networks, we must exercise those response muscles as it is likely not a matter of if but when a breach will occur.


Well, we’ve made the point in good detail; a pen test is not just a pen test.  There are many questions to ask in advance of seeking out this important tool in one’s cyber hygiene program. I’m sure there are options, features, and other considerations out there that I haven’t covered.

Share this:

Keep Up with Us!

Talk to an ATC technology advisor today!

Keep Up with Us!

Keep Up with Us!