Security: Pen testing; IDS/IPS; network scanning


Start with an uncomfortable question: what can an organization actually say about whether it has been compromised? Only that it has not detected a compromise. You can sometimes prove you have been breached; you can never prove you haven't. Today is about living with that — deliberately attacking your own network to find problems first, watching traffic for signs of someone else doing the same, and planning for the day detection succeeds. The tools are the ones from the reconnaissance discussion last class, now used in earnest, and you'll build a detector for them yourself.

Slides from class

Files:

Additional information:

Agenda

These notes cover penetration testing and how it's structured, network scanning with nmap step by step, intrusion detection and prevention, incident response planning, the economics of security spending, and certifications.

Notes

Why penetration testing

Three related activities, often confused:

Note the scope in the middle item. A pen test should cover people, process, and technology, not just technology — which is why social engineering and physical entry are in scope for a serious test.

How much the testers are told

A question worth thinking through: can a company's own IT staff perform a closed box test? No — they already know the network. A closed box test has to come from outside.

And the more important question: if the pen test can't get in, you're safe, right? No. All you learn is that those testers, with that much time and that scope, didn't get in. Pen testers commonly report never having encountered a network they couldn't penetrate.

How a pen test is structured

The Penetration Testing Execution Standard (PTES) recommends seven stages:

  1. Pre-engagement interactions — define scope and expectations (some systems may be excluded), create a clear contract, agree on what will be tested and when. The contract is your get out of jail free card: without it in writing, the work you are about to do is a crime. Note also the scheduling question — testing production systems may need to happen outside business hours.
  2. Intelligence gathering — OSINT (open source intelligence) plus other intel, some of which the client may provide.
  3. Threat modeling — contemplate which threats matter (an APT? script kiddies?) and model their likely approach. Defending against a foreign intelligence service is a different exercise from defending against a bored teenager.
  4. Vulnerability analysis — run scanners to find targets: Metasploit, nmap, Nessus, ZAP, Burp Suite, custom tools.
  5. Exploitation — attempt to breach the target, and watch for network responses and defensive actions.
  6. Post exploitation — attempt to escalate privileges, install backdoors.
  7. Reporting — report findings and recommend remediation.

NIST 800-115 groups this more coarsely into three high-level phases: planning (establish goals, determine what is and isn't in scope, note any standards that must be met such as PCI, review recent audits and breaches, model threats, set expectations), execution (intel gathering, network discovery, port and service identification, vulnerability scanning, wireless scanning), and post-execution (close any breaches left open during execution, report findings, remediate).

That "close breaches left open" item deserves emphasis. A test that leaves backdoors installed has made the organization less secure, not more.

The exploitation phase in four steps

  1. Passive scanning — gather as much data on the target as possible via sniffing, OSINT, and possibly social engineering. Risk of detection is low.
  2. Active scanning — scan ports on all available IP addresses, run vulnerability scanners. Risk of detection increases.
  3. Breaching — manually conduct exploits (buffer overflows, SQL injection, XSS), phishing, Metasploit, harmless viruses.
  4. Completion — fix the holes.

The team doing this is the red team, or OPFOR (opposing force), and its remit commonly includes physical covert entry and social engineering as well as network attacks.

External then internal

Tests are often run in two phases with different goals.

External — goal: find what's vulnerable from the Internet. Port scan all public-facing IP addresses (web server, gateway router), identify OS and service versions, try default passwords, run vulnerability scanners on what you find (nmap, ZAP, Burp Suite), manually attempt common attacks (XSS, SQL injection), try Metasploit, attempt wireless access (try to get admin on the Wi-Fi router), and send phishing emails.

Internal — goal: find what's vulnerable to insiders. Perform the external steps but from inside, use a sniffer to look for passwords and unencrypted data, and attempt covert entry.

A natural pairing from experience: run the external test as closed box (the outside attacker knows nothing) and the internal test as open box (the insider knows everything).

Reporting and prioritizing findings

A good pen tester will find issues — some large, some small. Which do you fix first, and how do you explain any of it to people who don't work in technology?

A cautionary story. Before presenting the results of a first pen test to a non-technical board of directors, the instructions to the testers were: give the high-level picture of where we stand, and unless asked, don't get into specific bugs. The testers opened with "let me tell you about the Unix R-login bug, because it's really interesting." The board wanted to know three things: what does this mean, how are we doing, and what do we do next. What they really wanted was confidence that there is a plan to address the issues and that someone will report back when it's done.

For prioritizing the findings themselves, DREAD rates each vulnerability on five dimensions, typically 1–5, then sums:

Some practitioners omit discoverability — "DREAD minus D" — on the grounds that counting it rewards security through obscurity. A practical refinement: rate DREAD, separately rate ease of fix, and do the high-DREAD, easy-to-fix items first.

Passive reconnaissance

You can learn a surprising amount about a target without ever sending it a packet, which means without any chance of being detected. Several services scan the Internet continuously and let you query their results:

Then two facts that make the results actionable. Devices are frequently left with default usernames and passwords, and the defaults are published — in the device's own manual, and in lists online. Recall Mirai, which compromised huge numbers of webcams this way and took down a DNS provider. And services run on well-known ports, so an open port tells you what's probably listening.

Combine those: a search engine tells you which devices exist and what banners they return, the banner tells you the make and model, and the manual tells you the default password. No packets sent by you, and no IDS alert anywhere.

Active scanning with nmap

Active scanning means sending packets, which means risking detection. The standard methodology — from the EC-Council's Certified Ethical Hacker curriculum — has five steps plus diagramming, and nmap can do all of it.

You could do step one by hand. ping 10.0.2.5 tells you a device is up, but you must know its address, and a 10.0.2.0/24 network has 254 possible hosts. Pinging each in turn is tedious.

Step 1: check for live systems

$ nmap -sP 10.0.2.0/24

Nmap scan report for 10.0.2.1
Host is up (0.0015s latency).
Nmap scan report for 10.0.2.2
Host is up (0.0013s latency).
Nmap scan report for 10.0.2.5
Host is up (0.00067s latency).
Nmap scan report for 10.0.2.15
Host is up (0.00057s latency).
Nmap done: 256 IP addresses (4 hosts up) scanned in 2.55 seconds

256 addresses in under three seconds. We now know which addresses are in use — we're 10.0.2.5, and 10.0.2.15 looks worth a closer look.

Step 2: check for open ports

$ nmap -sT 10.0.2.15

Nmap scan report for 10.0.2.15
Host is up (0.00098s latency).
Not shown: 994 closed ports
PORT     STATE SERVICE
21/tcp   open  ftp
22/tcp   open  ssh
23/tcp   open  telnet
53/tcp   open  domain
80/tcp   open  http
3128/tcp open  squid-http

-sT uses TCP to check for open ports. With no ports specified, nmap scans the 1,000 most popular ones. The result is a service inventory: an FTP server, SSH, telnet, DNS, a web server, and Squid (a web proxy).

But this is a full-open scan — it completes the entire three-way handshake (SYN, SYN/ACK, ACK) and then closes each connection with a reset. That's noisy, and an IDS may well notice this much activity. In Wireshark you can watch it happen: SYN from 10.0.2.5 to 10.0.2.15 on port 80, SYN/ACK back, ACK, then RST to close — while the scan of the next port has already started.

Step 3: scan around the IDS

$ sudo nmap -sS -D 10.0.2.6 -T2 10.0.2.15

Three evasion techniques in one command:

Note that this is the mirror image of P > D + R from last class. The defender wants protection to outlast detection plus reaction; the adversary wants to avoid the D entirely, and slowing down is the cheapest way to do it. Also note that -sS requires sudo, since crafting half-open connections means bypassing the normal socket API.

Step 4: banner grabbing

$ sudo nmap -O 10.0.2.15

...
MAC Address: 08:00:27:F2:D2:08 (Oracle VirtualBox virtual NIC)
Device type: general purpose
Running: Linux 3.X|4.X
OS CPE: cpe:/o:linux:linux_kernel:3 cpe:/o:linux:linux_kernel:4
OS details: Linux 3.2 - 4.0
Network Distance: 1 hop

-O identifies the operating system, correctly detecting Linux here, and also reports the MAC address — note the OUI identifying it as a VirtualBox virtual NIC, which is the OUI lookup from Day 1 being used for fingerprinting.

-A goes further, adding service versions: 

$ sudo nmap -A 10.0.2.15

PORT     STATE SERVICE    VERSION
21/tcp   open  ftp        vsftpd 3.0.3
22/tcp   open  ssh        OpenSSH 7.2p2 Ubuntu 4ubuntu2.2 (Ubuntu Linux; protocol 2.0)
| ssh-hostkey:
|   2048 4b:6f:e9:4a:8e:5b:2b:4d:12:34:94:48:9b:fc:05:1a (RSA)
|_  256 37:61:b8:e9:07:af:1c:f1:6a:49:94:ea:de:19:cf:b4 (ECDSA)
23/tcp   open  telnet     Linux telnetd
53/tcp   open  domain     ISC BIND 9.10.3-P4-Ubuntu
80/tcp   open  http       Apache httpd 2.4.18 ((Ubuntu))
|_http-title: Apache2 Ubuntu Default Page: It works

Every one of those version strings is a gift to the attacker, because a version number is a CVE lookup away from a list of known vulnerabilities. This is why banner suppression is a hardening step. The scan also grabs the SSH host key and throws in a traceroute for good measure, takes about 25 seconds, and will almost certainly anger the IDS.

Step 5: scan for vulnerabilities

$ sudo nmap --script vuln 10.0.2.15

80/tcp   open  http
|_http-csrf: Couldn't find any CSRF vulnerabilities.
|_http-dombased-xss: Couldn't find any DOM based XSS.
| http-slowloris-check:
|   VULNERABLE:
|   Slowloris DOS attack
|     State: LIKELY VULNERABLE
|     IDs:  CVE:CVE-2007-6750
|       Slowloris tries to keep many connections to the target web server open and
|       hold them open as long as possible... starves the http server's resources
|       causing Denial Of Service.
|_http-stored-xss: Couldn't find any stored XSS vulnerabilities.

Nmap done: 1 IP address (1 host up) scanned in 324.85 seconds

Note the double dash on --script. This took 6.5 minutes for a single host, and found a likely Slowloris vulnerability with its CVE identifier attached. It will also anger the IDS — there's a real tension here between thoroughness and stealth, and you generally can't have both.

Which suggests building a detector

Is scanning hosts and ports normal activity? No — it suggests a potential intruder. So how would you detect it? Pseudocode a port scan detector:

That's the end-of-class exercise, and it's also exactly what a real IDS does — which is why -T2 works as an evasion.

IDS and IPS

An Intrusion Detection System watches traffic looking for exploits against systems and applications — buffer overflows, XSS, SQL injection, and so on — and raises an alarm when it detects one, after which a human must investigate. It receives a copy of network traffic and monitors passively.

An Intrusion Prevention System does the same detection but sits inline, so it can drop packets and stop an attack in progress.

That placement difference is the whole distinction, and it's a real tradeoff: an IPS can stop an attack, but a false positive now blocks legitimate traffic, and being inline makes it a potential bottleneck and point of failure. (An IDS can take limited action — sending TCP resets to both ends to tear down a connection — but malicious software can simply reconnect, and it doesn't work for UDP at all.)

Four detection approaches

None is perfect, and the hard part isn't the detection algorithm — it's the operational consequence. Balancing false alarms against false negatives is genuinely difficult; investigating alerts is time-consuming; false negatives can be worse than false positives because you never learn about them; and too many false positives produce alarm fatigue, at which point your analysts stop reading alerts and the system provides nothing.

Snort and Suricata

Snort is a popular free and open-source IDS/IPS, originally developed in 1998 and now part of Cisco Talos. It's rule-based: you can write custom rules or subscribe to Cisco's. You can install it on the VM: 

$ sudo apt install snort -y
# (use ifconfig to find your interface)

$ sudo snort -A console -q -u snort -g snort -c /etc/snort/snort.conf -i eth0

The flags: -A console sends alerts to the console, -q is quiet mode (suppressing the startup banner), -u snort -g snort runs as that user and group, -c names the rules file, and -i the interface. Have a look at /etc/snort/snort.conf and at a rules file such as /etc/snort/rules/mysql.rules to see what a rule looks like, then start Snort and ping the machine from elsewhere to watch it alert. Be warned: the default install is very sensitive and reports nearly everything — alarm fatigue, demonstrated.

Suricata is increasingly the free IPS of choice. The comparison: 

Feature              Snort                        Suricata
-----------------------------------------------------------------------------
Architecture         traditionally single-        natively multi-threaded,
                     threaded (Snort 3 adds       better on high-speed
                     multi-threading)             networks
Rule compatibility   standard Snort rules         compatible with Snort rules,
                                                  plus its own
Logging              basic                        Network Security Monitoring
                                                  in JSON, easier SIEM
                                                  integration
Popularity           widely deployed,             growing
                     industry standard

Honeypots

A honeypot is a system that looks like a production machine and is intentionally left vulnerable, to bait intruders into exploring it. The elegance is in the detection logic: nobody would normally have any reason to be on that system, so any access at all is evidence of an intruder. No signatures, no baselines, essentially no false positives. Many honeypots together are a honeynet.

A discussion question worth sitting with: your IDS reports someone on the network running John the Ripper, a password cracker, trying to gain root on a server. What do you do? Note that the answer involves the incident response plan you should already have written, not improvisation.

Incident response planning

An incident response plan is a set of instructions letting an organization respond efficiently to a security event. It's distinct from a disaster recovery plan, which is usually aimed at physical disasters like floods.

Steps to create one:

  1. Identify an incident recovery team.
  2. Determine the critical components of the network.
  3. Identify single points of failure and address them.
  4. Create a disaster recovery plan.
  5. Create the incident response plan itself: list roles and responsibilities; summarize the tools, technologies, and physical resources that must be in place; list critical network and data recovery processes; and plan for communications, both internal and external.
  6. Train staff and practice the plan.

Make a usable plan

The recurring theme from these notes: a consulting firm hired to help develop a disaster recovery plan wanted to write an enormous tome covering every scenario the organization might face. The problem is that whatever actually happens probably won't match any scenario in the plan.

A better structure is a small decision tree over the questions that actually change your response. Can we use our office, yes or no? Is it during business hours, yes or no? That's four plans. And each plan answers the same handful of questions: how will we communicate with employees, in what order will we bring systems up, who does what, what credentials do we need, and how will employees get their jobs done?

Then test it — quarterly, by actually working from the offsite facility, so you find out whether employees know where to go and whether the systems work as expected before you need them to.

The economics of security

Every dollar spent on security is a dollar not spent elsewhere, and someone will eventually ask why you're spending so much. You need an answer.

Annualized Loss Expectancy

You shouldn't spend $5,000 on a safe to protect a $2 item. So how much should you spend?

ALE = damage × probability

A worked version, from a hedge fund: fund size $10B, long-term return 15%, so the fund needs to make $1.5B/year, which over 252 trading days is about $6M every day. What are the odds of something taking us down for a day this year? Say 10%. Then the expected annual loss is $6M × 0.10 = $600K — and we're currently spending under $200K/year on security. Now consider the odds of a two-day outage. You're getting a bargain.

Honest caveats from practice: ALE is easy to say and hard to use. Some things are hard to value — customer confidence after an incident, staff morale. Estimating the probability of an adverse event is also difficult. And be wary of a vendor's estimate of the threat, since the vendor has an agenda.

Benchmarking

The other approach is to compare against peers. Firms survey industry participants and sell reports summarizing security spending, usually as a percentage of sales, so you can see what your industry spends. The industry overall spends more than $120B on security hardware, software, and services.

Typical figures: companies spend around 5% of their IT budget on security, ranging from about 1% to 13%, varying by industry, with financial firms spending more. A quote worth remembering about how that budget often materializes: "We never had a cybersecurity budget until recently when I said we need a dedicated budget for it. So they took a chunk of the IT budget and told me to be grateful. We have no more money overall, but some for cybersecurity."

What breaches cost

IBM's annual report put the global average cost of a data breach at $4.88M in 2024, and the average has been rising steadily. The costs aren't limited to detecting and escalating the incident; they also include notification, post-breach response, and — typically the largest component — lost business. By root cause, phishing and lost or compromised credentials were both the most common and the most costly.

Note what that implies: the most expensive failures are people problems, not protocol problems.

Prioritizing projects

Every department has projects it considers vital, and there are always more projects than the technology group can work on at once. So some wait, and the department heads whose projects are waiting get angry — at you. Worse, they may turn to shadow IT, standing up their own systems outside your control, which is a security problem created directly by prioritization failure. Security projects have to compete in this queue like everything else.

One method: hold a quarterly meeting with all department heads, each bringing their most important projects. You rate each project on ease of implementation; the group rates each on business impact. Plot them on a scatter graph and split into quadrants. Do the easy, high-impact ones first, and order the rest by the resources they need, trading off easy-but-unimpactful against hard-but-impactful.

Two practical notes. Let the business leaders decide impact together, so they own the prioritization and have a justification for it. And a pro tip: don't put numbers on the axes while discussing, or the conversation becomes about the numbers. The known danger: long-term strategic projects, which are hard by nature, always risk being bumped.

Certifications

If you're interested in this professionally:

Exercises

Detecting port scans with Scapy. Extend a basic packet sniffer into a tool that detects a possible TCP SYN scan — one of the most common reconnaissance techniques. Your program should:

Deliverables: a Python script (portscan_sniffer.py), plus a short report of one to two pages explaining what a port scan is and why attackers use it, how your detection logic works, and what happened when you tested it with nmap. Grading: basic sniffer works (30%), correct SYN identification (20%), detection logic (30%), report and test results (20%).

Starter code is linked at the top of this page. The skeleton uses sniff(prn=process_packet, store=False) from Scapy, checks flags == "S" to identify SYN packets, and keys a scan_tracker dictionary on source IP with a ports set and a first_seen timestamp.

Then test your detector against the scans from these notes, and answer: does it catch nmap -sT? Does it catch nmap -sS? Does it catch nmap -sS -T2? What would you change to catch a slow scan, and what would that change cost you in false positives?

Terms

Not a comprehensive reference — the textbook has much more — but these are the terms from Day 15 that will keep coming back.

vulnerability scan vs. penetration test vs. audit
A scan looks for known vulnerabilities without exploiting them; a pen test attempts actual exploitation and privilege escalation across people, process, and technology; an audit reviews documentation, policies, and plans.
privilege escalation
Turning limited access into greater access, typically the goal immediately after an initial breach.
open / closed / grey box
Testers given full, no, or partial knowledge of the network. A closed box test cannot be performed by internal IT staff, who already know the network.
PTES
Penetration Testing Execution Standard, with seven stages: pre-engagement interactions, intelligence gathering, threat modeling, vulnerability analysis, exploitation, post exploitation, and reporting.
get out of jail free card
The written contract defining scope and authorization. Without it, the testing is indistinguishable from a crime.
OSINT
Open source intelligence: information about a target gathered from publicly available sources.
threat modeling
Deciding who you're defending against — APT, organized crime, script kiddies — and modeling their likely approach, since the defenses differ.
red team / OPFOR
The team simulating the adversary, commonly including physical covert entry and social engineering alongside network attacks.
DREAD
A prioritization scheme rating Damage, Reproducibility, Exploitability, Affected users, and Discoverability, typically 1–5 and summed. Sometimes used without Discoverability, so as not to reward security through obscurity.
banner grabbing
Reading the identifying strings services return in order to learn the OS and exact service versions — which then map directly to CVE entries.
full-open vs. half-open scan
nmap -sT completes the full three-way handshake and then resets, which is noisy; nmap -sS sends only the SYN, learning the port state from the SYN/ACK without completing a connection, so it may leave no trace in connection logs.
decoy scanning
nmap -D: sending additional packets spoofed from other addresses so the defender cannot tell which source is the real scanner.
timing templates
nmap -T0 through -T4, from slowest to fast parallel. Slow scans stay under IDS rate thresholds — the adversary's way of defeating detection rather than protection.
Slowloris
A denial of service attack (CVE-2007-6750) that opens many connections to a web server and sends partial requests, holding them open to starve the server's resources.
IDS vs. IPS
An intrusion detection system receives a copy of traffic and raises alarms for humans to investigate; an intrusion prevention system sits inline and can drop packets, at the cost of blocking legitimate traffic on a false positive and becoming a potential bottleneck.
detection approaches
Signature/rule-based (exact match against known attacks, needs constant updates), anomaly-based (deviation from a learned baseline), behavior-based (specific prohibited actions), and heuristic (AI over attack characteristics).
false positives / false negatives / alarm fatigue
False alarms waste analyst time; missed detections are worse because you never learn of them; and too many false positives lead analysts to stop reading alerts, which makes the system worthless.
Snort / Suricata
Snort is the long-standing open source rule-based IDS/IPS, now part of Cisco Talos. Suricata is natively multi-threaded, Snort-rule compatible, and logs in JSON for easier SIEM integration.
honeypot / honeynet
A deliberately vulnerable system that looks like production but has no legitimate users, so any access at all indicates an intruder. Several together form a honeynet.
incident response plan
Instructions for responding to a security event: recovery team, critical components, single points of failure, roles, tools, recovery processes, and internal and external communications — then trained and practiced. Distinct from a disaster recovery plan, which targets physical disasters.
ALE (Annualized Loss Expectancy)
Damage × probability, used to justify security spending. Easy to state, hard to apply: some losses resist valuation and probabilities are hard to estimate.
shadow IT
Systems a department stands up outside the technology group's control, often because its projects were deprioritized. A security problem produced by a prioritization problem.
certifications
CEH (EC-Council, oldest), GPEN (SANS), OSCP (hands-on: you must actually breach a test system), CISSP (requires four to five years of experience).

Credit: Most information on this page is from Computer Networking: A Top-Down Approach, 9th edition by Kurose and Ross, or from Internet Security: A Hands-on Approach" by Du.
This page was generated by Claude from in-class slides.