Diploma thesis - Intrusion detection system based on process behavior rating

Introduction

The basic idea of this thesis is to implement an intrusion detection system (IDS) based on an approach similiar to that used by the SpamAssassin software to detect spam (unsolicited commercial e-mail, nowadays considered a severe problem on the Internet).

SpamAssassin is using a rule-based approach to spam detection. It provides a set of tests (773 at the time of writing) that combine different techniques of e-mail header and body analysis. Each e-mail message is processed by these tests and given point values, according to passed tests. Final status of a message (spam or non-spam, here called "ham") is determined by counting all its points, where the user (or system administrator) can specify a point threshold necessary to consider message a spam.

We would like to introduce an IDS that will provide a set of tests applicable to operating system processes. Each passed test will be rated by a configured score. By counting these point values, the IDS will try to determine whether a particular process is harmless, or is performing a suspicious or dangerous activity. In such case, the IDS will report the process to system administrator.

The system is currently working as a prototype. In case anybody would be interested in this topic, or in future development of this system, please feel free to contact me.

Thesis abstract

The goal of the work is to implement a kernel module for the FreeBSD operating system, performing a defined set of tests on every running process in the OS and rating results of these tests according to whether they correspond to a common benign process or to a process performing a dangerous or suspicious activity. An occurence of a process with a score exceeding a predefined threshold will be reported as a potential intrusion. Part of the work is to prepare a basic set of tests analyzing instantaneous properties of a process as well as its behaviour and unexpected changes in this behaviour in particular, and to determine suitable parameters used to rate results of these tests. The implemented tests focus on 3 types of common security problems: buffer overflow, symlink attack and denial of service. Parameter assignment is realized using a genetic algorithm. At the end, a possible future approach to developping such a detection system is proposed.

Download

License

All files are provided under BSD license. Feel free to use, modify, distribute them or whatever else is possible.

Contact

See homepage.