Tracing packet drops in Florida and sniffing traffic from 35K feet

I recently traveled to Hollywood, Florida for a customer on-site network troubleshooting. We usually do things over the phone and remote access if needed, but this customer insists that its our VoIP application acting up, and not their network. So the next day I went to their facilities, met with one of their IT Staff and immediately started mapping out their wired network (for fun and profit).



After a couple of minutes of tracing un-labeled RJ45 cables and network devices in general, I was able to trace the root cause. The bottleneck is originating from a commercial firewall installed on their network. I am not going to identify the brand and model, but its one of those firewall not meant to handle tremendous amount of traffic. In short, its a small-office-home office firewall/router. Their facility generates around 12 to 20mbps of outbound traffic on a daily basis.

This firewall goes gaga when hit by too much traffic; it simply drops all concurrent connections and resets as evident on the firewall and router logs. Good thing their Network Admin made the right choice and decided to get hold of a Cisco ASA 5505 Security Appliance and replace their current firewall. The problem is this guy does not know how to configure the ASA and needs to outsource the configuration and installation, so the ASA needs to wait while the problem still persists on their converged network.

To add salt to the wound, they are using old-school workstations; running Celeron 2.0ghz processors with a measly 256mb of SDRAM. Understand that these workstations handle a softphone-based VoIP client, a web-based CRM, Instant Messaging client, and Agent productivity apps. I say good luck with that. As suspected, Agents usually encounter the white screen of death where everything halts and freezes, hitting the Reset button is their usual routine.

Add the workstation hardware issues and misconfiguration on the network and you get a very painful and regretful VoIP experience.

Knowing how painful this experience is to their Agents, what I did was strip down Windows XP Pro to the bare minimum to free additional memory and overall system resources. What I meant with a stripped-down version is by disabling all Local Services that are not needed, adjust the workstation to Best Performance, disable tons of start up and running applications via msconfig, and finally, lock down the Agent login to Limited Rights so they can't install those nasty shopping IE add-on toolbars, lol. Things you must do when no Domain Controller is not present on a large network.

On my way back home to the Bay Area, I had some fun on-flight thanks to Gogo In-flight Internet without actually signing up for their service.

Thanks to Wireshark, ZenMAP GUI, and my laptop's Intel(R) Wireless WiFi Link 5100 card I was able to take a glimpse of the WiFi activity on-board the plane.

Intense Scan plus UDP output on NMAP:

nmap -sS -sU -T4 -A -v -PE -PA21,23,80,3389 172.19.131.2

Starting Nmap 5.00 ( http://nmap.org ) at 2009-09-03 16:40 Pacific Daylight Time
NSE: Loaded 30 scripts for scanning.
Initiating ARP Ping Scan at 16:40
Scanning 172.19.131.2 [1 port]
Completed ARP Ping Scan at 16:40, 0.22s elapsed (1 total hosts)
Initiating Parallel DNS resolution of 1 host. at 16:40
Completed Parallel DNS resolution of 1 host. at 16:40, 11.39s elapsed
Initiating SYN Stealth Scan at 16:40
Scanning 172.19.131.2 [1000 ports]
Discovered open port 80/tcp on 172.19.131.2
Completed SYN Stealth Scan at 16:40, 5.05s elapsed (1000 total ports)
Initiating UDP Scan at 16:40
Scanning 172.19.131.2 [1000 ports]
Completed UDP Scan at 16:40, 4.26s elapsed (1000 total ports)
Initiating Service scan at 16:40
Scanning 1001 services on 172.19.131.2
Service scan Timing: About 0.40% done
Service scan Timing: About 1.50% done; ETC: 18:43 (2:00:31 remaining)
Service scan Timing: About 3.00% done; ETC: 18:13 (1:29:33 remaining)
Service scan Timing: About 4.50% done; ETC: 18:02 (1:18:15 remaining)
Service scan Timing: About 5.99% done; ETC: 17:57 (1:12:09 remaining)
Service scan Timing: About 7.49% done; ETC: 17:54 (1:08:07 remaining)
Service scan Timing: About 10.39% done; ETC: 17:43 (0:56:12 remaining)
Service scan Timing: About 10.49% done; ETC: 17:50 (1:02:43 remaining)
Service scan Timing: About 13.39% done; ETC: 17:43 (0:54:02 remaining)
Service scan Timing: About 13.49% done; ETC: 17:48 (0:58:55 remaining)
Service scan Timing: About 16.38% done; ETC: 17:43 (0:52:03 remaining)
Service scan Timing: About 16.48% done; ETC: 17:47 (0:55:49 remaining)
Service scan Timing: About 19.38% done; ETC: 17:42 (0:50:03 remaining)
Service scan Timing: About 19.48% done; ETC: 17:46 (0:53:11 remaining)
Service scan Timing: About 22.38% done; ETC: 17:42 (0:48:06 remaining)
Service scan Timing: About 28.37% done; ETC: 17:42 (0:44:16 remaining)
Service scan Timing: About 34.37% done; ETC: 17:42 (0:40:29 remaining)
Service scan Timing: About 40.36% done; ETC: 17:42 (0:36:45 remaining)
Service scan Timing: About 46.35% done; ETC: 17:42 (0:33:01 remaining)
Service scan Timing: About 52.35% done; ETC: 17:42 (0:29:19 remaining)
Service scan Timing: About 58.34% done; ETC: 17:42 (0:25:37 remaining)
Service scan Timing: About 64.34% done; ETC: 17:42 (0:21:55 remaining)
Service scan Timing: About 70.33% done; ETC: 17:42 (0:18:13 remaining)
Service scan Timing: About 76.32% done; ETC: 17:42 (0:14:32 remaining)
Service scan Timing: About 82.32% done; ETC: 17:42 (0:10:51 remaining)
Service scan Timing: About 88.31% done; ETC: 17:42 (0:07:10 remaining)
Service scan Timing: About 94.31% done; ETC: 17:42 (0:03:30 remaining)
Service scan Timing: About 98.90% done; ETC: 17:43 (0:00:41 remaining)
Completed Service scan at 17:42, 3688.53s elapsed (1001 services on 1 host)
Initiating OS detection (try #1) against 172.19.131.2
NSE: Script scanning 172.19.131.2.
NSE: Starting runlevel 1 scan
Initiating NSE at 17:42
Completed NSE at 17:43, 36.19s elapsed
NSE: Starting runlevel 2 scan
Initiating NSE at 17:43
Completed NSE at 17:43, 5.02s elapsed
NSE: Script Scanning completed.
Host 172.19.131.2 is up (0.0014s latency).
Interesting ports on 172.19.131.2:
Not shown: 1000 open|filtered ports, 999 filtered ports
PORT STATE SERVICE VERSION
80/tcp open http?
| html-title: Site doesn't have a title.
|_ Did not follow redirect to http://airborne.gogoinflight.com/abp/page/abpDefault.do?REP=127.0.0.1&AUTH=127.0.0.1&CLI=172.19.131.153&PORT=54273&RPORT=54272&acpu_redirect=true
MAC Address: 00:E0:4B:22:96:D9 (Jump Industrielle Computertechnik Gmbh)
Warning: OSScan results may be unreliable because we could not find at least 1 open and 1 closed port
Device type: general purpose
Running: Linux 2.6.X
OS details: Linux 2.6.18 - 2.6.27, Linux 2.6.26
Uptime guess: 0.405 days (since Thu Sep 03 07:59:45 2009)
Network Distance: 1 hop
TCP Sequence Prediction: Difficulty=197 (Good luck!)
IP ID Sequence Generation: All zeros

Host script results:
|_ nbstat: ERROR: Name query failed: TIMEOUT

Read data files from: C:\Program Files\Nmap
OS and Service detection performed. Please report any incorrect results at http://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 3757.66 seconds
Raw packets sent: 4045 (148.498KB) | Rcvd: 31 (1502B)

Wireshark Capture Screenshot:



Noteworthy discovered Protocols and Services gathered from the Wireshark .pcap capture:

- Cisco IP-SLA
- TACACS and XTACACS
- BOOTP
- TFTP
- CLDAP (Connectionless Lightweight Directory Access Protocol)
- Cisco Wireless LAN Context Control Protocol
- Mobile IP Protocol (RFC 3344)
- RIP (Routing Information Protocol)
- OpenVPN
- OCSP (Online Certificate Status Protocol)
- Slimp3 Communication Protocol (Device ID: 101) (Firmware Revision: 6:12 (0x6c)
- Base Station Subsystem GPRS Protocol (BSSGP)
- CFLOW (Cisco NetFlow/IPFIX)
- CUPS (Common Unix Printing System)
- GPRS Tunneling Protocol (GTP)
- H.225.0 RAS

Discovered Network Device Signatures/MAC OUI's:

- JUMP INDUSTRIELLE COMPUTERTECHNIK GmbH (00:e0:4b)
- Hon Hai Precision Ind. Co., Ltd. (00:22:69)

You can easily Google those two identified manufacturers and you will have and idea what type of devices they produce.

As always, hit me up on E-mail if you want a copy of the complete .pcap capture and I will be glad to send you a copy, for research and analysis of course. Let me know if you guys need additional information as well about my recent 35K feet packet-sniffing adventure.

On my next flight, I am bringing an external USB antenna with packet-injection capability :-) attached to my future 1000HE netbook.

Happy packet-sniffing everyone and try not to break any law in the process!
Ron