Hello

Reverse-Engineering and Fuzzing Custom Network Protocol

Overview

The communication protocol defines the format and semantics of message exchange between applications. In modern times there are a myriad of proprietary application protocols like Skype Protocol, Dropbox Protocol, etc which applications use to achieve various goals like bandwidth efficiency, custom encryption/compression, etc. These protocols could have security vulnerabilities. Protocol Reverse Engineering (PRE) is not only useful for offensive purposes but also used by modern Intrusion Detection Systems(IDS), they use the knowledge of protocol specification to do Deep Packet Inspection(DPI) which can enhance its capabilities, where it earlier relied just based on pattern matching which may produce lots of false positives. Custom protocols are not only used by legitimate applications but also by malware and botnets like Zeus, Emotet, etc. By reversing malware protocol you can connect to malware servers and track their campaigns.

Protocol Reverse Engineering(PRE) is an art and science of recovering the protocol specification of the obscure/proprietary protocol whose documentation is unavailable or poorly documented. There are efforts to develop automated PRE tools but they are largely academic and are not mature enough to be usable, and can’t give the accuracy a human analyst can offer. Automated tools face the challenges of heterogeneous protocol data which is often a mixture of text and binary, and it has different data types and variable-length fields and this is the reason I have created this training to help you understand these challenges and learn to recover protocol specification.

This training is divided into three parts, in the first part we will learn about Protocol Reverse Engineering principles. We will look at some of the common data formats and other protocol structures and with that understanding we will write a protocol dissector using Scapy framework for a target Desktop game Minetest (a Minecraft clone). Minetest has a multiplayer feature in which different players can connect to the server and play with other players, there are also many public servers which you can connect and play. Once we have written the dissector for the communication we will sniff the connection and look at the communication flow between the client and the server which we will capture and re-analyze the traffic to improve the dissector further, using this newly improved dissector we will implement a custom game client/bot which will connect to the server and play as a Bot player.

In the second part, with a decent understanding of the Minetest protocol we will move on to the offensive side of the training and try to fuzz the game server to find some security vulnerabilities, we will start with basic fuzzer and try to do incremental improvement such that we have good code coverage. A good protocol fuzzer has a semantic understanding of the protocol, since we have that understanding we will do generation fuzzing in which we will define protocol specification in Boofuzz fuzzing framework and aging fuzzing the application. We will also try the approach of mutation fuzzing and see if we get lucky with a crash or two.

Why should you take this course?

1. Understand the structure of the requests and response especially useful for a malware analyst.
2. Construction of protocol decoders useful for writing gaming clients or to add support for a third-party proprietary product.
3. Reverse engineer communication of online games like MMORPG which can help you do security testing of multi-player online games.
4. Create Network signatures for malware communication that can be integrated with IDS and IPS, understanding the protocol specification can help you to do deep packet inspection.
5. Write a protocol fuzzer to feed the remote server with crafted randomness in the data to crash the data processing part of the application with the intent of finding security vulnerabilities.
6. Identify security vulnerabilities in protocol implementation like authentication bypass, replay attack, information disclosure, DOS, RCE etc. PRE can also help you to do deeper black-box testing of the application.
7. Build a protocol specification for a vaguely/undocumented protocol.
8. Audit the privacy and security of an application running on your phone/computer by looking at what data is it exporting.

Who should take this course?

1. DFIR practitioner - to investigation malicious activity in the network
2. Reverse Engineer - write a custom client that fully replicates the existing client software/game.
3. Bug Hunter - Write protocol fuzzer for Black Box testing for application processing remote data, for example, lots of IoT Devices use custom protocol for efficient communication.
4. Malware Analyst - To decode C&C server commands and the data which is exfiltrated
5. Threat Hunting - write network signatures for new emerging APT threats or it could be an intruder in your network, this course will help you decode network and analyze network traffic.
6. Developers
   1. who don’t have access to source code or protocol documentation, it usually happens when you are dealing with a legacy system which is too old and the company cannot find any documentation and you intend to migrate the system to new technology.
   2. While debugging software over the network, writing a protocol dissector can help you to get a deeper understanding of network communication done by your software.
   3. Helps you to do network debugging/diagnostics of application layer data.
   4. It helps you understand what is really transmitted over the network.
7. RED Team - take advantage of what the Security Operation Center (SOC) doesn’t know. Look for data leaks, do attacks like inject, replay and spoofing.
8. Vulnerability Researcher/Exploit Developer - this will also help exploit developer and vulnerability research to reproduce remote vulnerability and find zero-day bugs.

Course Outline

Day 1 - Basics

1. Networking Basics
2. Capturing Network Traffic
   1. Passive analysis
      1. Network Sniffing
      2. Syscall hooking (strace)
   2. Active analysis
      1. Network Proxies
3. Protocol Reversing
   1. Protocol Structure
      1. Common data format
      2. Data Encoding
      3. Binary Protocol Structure
      4. Text Protocol Structure
   2. Protocol Flow
4. Protocol Dissector (targeting Minetest game)
   1. Scapy 101
   2. Implementing protocol dissector in scapy for Minetent game. This section will have Labs on
      1. Protocol decoding TLV format
      2. Packet decompression
      3. Packet Reassembly

Day 2 - Fuzzing

1. Custom Client (Bot Player for Minetest Game)
   1. Brief Understanding of Application
   2. Authenticate the client
   3. Establish a valid session
   4. Some game hacks like making the player fly
   5. Create A Bot Army (if time permits)
2. Protocol Fuzzing (targeting Minetest game)
   1. What is fuzzing?
   2. Implement Mutation fuzzer
   3. Implement Dumb fuzzer
   4. Implement Generation Fuzzing (Protocol Aware Fuzzing)
   5. Vulnerability Triaging (Harness)

Note: each of the above section will have labs

Tools of the Trade

Below are some of the tools that you will learn in this training that will make you Protocol Reversing experience more fun.

1. Protocol Reversing tools
   1. Wireshark
   2. tshark
   3. tcpdump
   4. Scapy
2. Protocol Fuzzing Tool
   1. boofuzz(Sulley) fuzzing framework
3. Intrusion Detection System (for Defensive purpose)
   1. Snort IDS/IPS

Prerequisite

1. Knowledge of security concepts
2. Basic understanding of networking concepts
3. Knowledge of Linux OS
4. Basic Python programming language
5. What attendees should bring
6. Laptop with at least 50 GB free space
7. 8+ GB minimum RAM (4+GB for the VM)
8. External USB access (min. 2 USB ports)
9. Administrative privileges on the system
10. Virtualization software – Latest VirtualBox (5.2.X) (including Virtualbox extension pack)
11. Linux host machines should have exfat-utils and exfat-fuse installed (ex: sudo apt-get install exfat-utils exfat-fuse)
12. Virtualization (Vx-t) option enabled in the BIOS settings for VirtualBox to work

What attendees will be provided with

1. Virtual Machine with all the needed software pre-installed.
2. Training Material/slides.
3. Lab Manual
4. What to expect
5. Hands-on Labs
6. The joy of Reverse Engineer (looking under the hood)
7. Getting familiar with Network Protocol Analysis
8. Unlimited Email Support.

What not to expect

1. Become a Protocol Reversing Ninja.
2. Use the knowledge gained in this training to start exploring some Open and Close Protocol to improve your understanding of this topic. That will help you to get a deeper understanding of some underlying issues more closely.