Analysis of a Compromised Honeypot

...where people throw ducks at balloons and nothing is as it seems.

Christopher J. Reining
creining@packetfu.org

Honeypot Operating System: Redhat 6.2 x86 Vanilla

The honeypot was internet accessible for roughly 3 weeks before compromise. Pretty respectable TTL with the amount of vulnerable daemons awaiting exploitation. I was alerted to the compromise as soon as I started investigating the "RPC portmap request status" alert from Snort. This means a query was sent to the portmap daemon, requesting port information for the status service. This query usually precedes attempts to access status.

Refer: SID 587
Refer: arachNIDS

Sure enough, shortly after the initial alert the exploitation takes place. The shellcode is an exact match for a published statd exploit by ron1n called statdx.c. It is non-ripped linux IA32 portbinding shellcode, port 39168 and 133 bytes.

Refer: SID 600
Refer: arachNIDS
Refer: Bugtraq

The signs of the compromise point to an autorooter, which is an entire process that is automated, from the scan, identification of running service, to exploitation. Here is what takes place, ending with the payload of the exploit. Time in hh:mm:ss format. 

18:58:07 - 66.206.21.1 sends SYN to honeypot on port 111
18:58:07 - honeypot responds SYN/ACK to 66.206.21.1
18:58:07 - 66.206.21.1 sends ACK to honeypot
18:58:08 - 66.206.21.1 sends "RPC GETPORT Call" to honeypot
18:58:08 - honeypot sends "RPC GETPORT Reply" to 66.206.21.1 indicating its RPC port is listening on UDP 933
18:58:08 - 66.206.21.1 sends "STAT" to UDP 933 of honeypot attempting to buffer overflow rpc.statd
18:58:10 - 66.206.21.1 sends "STAT" to UDP 933 of honeypot attempting to buffer overflow rpc.statd
18:58:12 - 66.206.21.1 sends "STAT" to UDP 933 of honeypot attempting to buffer overflow rpc.statd

I say it's likely an autorooter because I don't believe an attacker would try and exploit portmapper three times in a row with a seperation of attempts at exactly two second intervals.

After the portmapper exploitation, the attacker had a root shell bound to port 39168. They added two user accounts, kernel and httpd. The kernel account with an uid/gid of 0, meaning it has the same system level authority of the root account. Here is the session that transpired: Now that they have two user accounts they end their root shell session and will use telnet to log in to the honeypot. It is interesting to note that different source ip's were used to perform the exploit and to log in via telnet. Let's take a quick detour and see what p0f can tell us about the two source ip's operating systems. p0f is a neat tool which does passive operating system identification using various TCP/IP flag settings. WHOIS output is also provided.

Machine A - performed the exploit
p0f output: 66.206.21.1 [15 hops]: Linux 2.4.2 - 2.4.14 (1)

Machine B - used to telnet to the honeypot
p0f output: 80.97.35.83 [24 hops]: Windows 9x (1) *

If I had to make an educated guess, I would say that Machine A is a compromised machine that is being used to scan and exploit other vulnerable machines via an autorooter. Machine B is possibly a compromised machine but may be the hacker's personal machine in Romania.

On to the telnet session with the newly created accounts:

At this point the attacker has tried unsuccessfully to retrieve an IRC proxy, psybnc, which is commonly used as a bouncer. An ssh backdoor with rootkit was installed, logs cleaned, system binaries replaced. The ssh server listens for connections on port 54321, and the attacker connects to the honeypot utilizing this encrypted communication channel. They download an another rootkit and scanning tools. The second rootkit is installed. The scanning of /8 networks for ports 22 (ssh) and port 111 (portmapper) begins. A /8 network is nothing to sneeze at, it's 16,777,214 hosts, targets in this case. The transparent bridge that performs packet filtering between the internet and the honeypot blocked these scans before they reached the internet.

On a comical sidenote, the second rootkit by default sends via email various informational tidbits to the attacker. These include ifconfig output, sniffed usernames/passwords from inbound or outbound sessions, and login credentials from the console. Well, the default yahoo.com address the attacker used as their email address apparently wasn't working (because that traffic is not allowed outbound from the bridge) so the attacker decided to send test messages to an email account from a domain I can only believe is their place of professional employment.

Files retrieved to honeypot follow, with a brief explanation. I was able to obtain these files via tcpdump log files. I used tcpslice to combine multiple tcpdump style log files into one. Then I used ethereal to load the capture and specify the filter "tcp.port eq 20 and not tcp.checksum_bad", chose relevant ftp session, chose Follow TCP Stream option, and saved the file.

r.tgz SSH Backdoor Server By Akamai-Team

srk3.tar Sin Rootkit 3.0

nebunu.tgz+ 
          |-apache    Bash wrapper to exploit Apache using httpdtype and openssl 
          |-httpdtype determines type of web server running on a host
          |-open      OpenSSL remote apache exploit for BSD 
          |-openssl   OpenSSL remote exploit
          |-pinky     variant of apache
          |-root      FTPD MassRooter
          |-scanssl   OpenSSL vulnerability scanner
          |-script    Bash wrapper for apache taking ip as argument
          |-start     wrapper for root but substitutes port 443 for 21 (!) 

haos.tgz+              
        |-libpcap-0.6.2 libpcap  
        |-.i.c          scans arbitrary port(s) of all users on a specified IRC channel 
	|-.ii.c         same as .i.c but with different #define of IRC server 
	|-.iii.c        same as .i.c but with different #define of IRC server 
	|-.iiii.c       same as .i.c but with different #define of IRC server 
        |-.ip           reads .sv and performs SSH exploit using Denyed(2) 
        |-.s.c          arbitrary TCP port scanner 
        |-.sv           0-length file 
        |-.v            SSH version mapper 
	|-.x            SSH autorooter utilizing Denyed(2)
        |-h=            one domain name pointer (possible vulnerable host?) 
        |-hh=           one domain name pointer (possible vulnerable host?) 
        |-targets.txt   SSH offsets 
        |-targets       SSH offsets 
	|-Denyed(1)     autoroot wrapper for 7350wurm
        |-Denyed(2)     SSH exploit 
	|-dat1          wrapper for dat2
        |-dat2          rpc.statd exploit
        |-haosv         HTTP version identifier
	|-haosx         wrapper for dat1
        |-haosp         arbitrary TCP port scanner 
        |-FTP           FTPD autorooter
        |-7350wurm      wu-ftpd exploit 

flood.tgz+ 
         |-broadcast.txt      list of smurf amplifiers
         |-juno               DoS tool 
         |-madscan.c          finds smurf amplifiers
         |-slice2             DoS tool
         |-smurf6-linux+LPG.c DoS tool utilizing smurf amplifiers
         |-vadimI             DoS tool
         |-vadimI.c           truncated binary
         |-vadimII.c          DoS tool
There seems to be a common theme among the programs that the attacker downloaded to the honeypot - scanning and exploiting other systems. The purpose of doing this is not certain, perhaps building up a fleet of DDoS zombies. There is quite a nice little arsenal of DoS tools.

The Sin Rootkit is a rather simple rootkit and it's install script is in shell. Let's look further at what it performs by dissection of this script:

- unset HISTFILE
- chattr -iau of files to be modified whichs removes immutable, append and undeletable attribute flags  
- remove /var/lock/subsys/atd and kill atd
- replace syslogd with trojaned copy
- stop syslog and restart it
- copy list of processes to not show to /dev/ttyop
- copy list of ports and ip addresses to not show to /dev/ttyoa
- copy list of filenames to not list to /dev/ttyof
- copy list of lognames to not log to /dev/ttyos
- touch -acmr trojaned binaries to system binaries which sets correct access and modification time
- chmod +s chsh which sets user or group ID on execution and replace chsh
- replace system binaries with trojaned binaries
- chkconfig --add atd and link trojaned atd SysV init to system atd SysV init
- install DoS tools in /usr/bin/
- install linsniffer, which sniffs usernames and passwords for popular protocols
- install sshd backdoor
- replace either xinetd or inet with trojaned copy, chkconfig --add inet and restart inet
- add crontab entry to email ifconfig output, hostname and sniffed traffic to author 
  (attacker is supposed to change this email address, author gets a treat if not done)
- check for other rootkits
- send email with sysinfo to the author
- start syslog
- clean text in current logs
- chattr +i of modified files which adds immutable flag
The /dev/ttyo{p,a,f,s} files contain the things to not display when the trojanized binaries are run. This is a typical rootkit footprint. By running strings on the trojaned binaries, for instance netstat, /dev/ttyoa is shown.

Now on to some light investigation of the still running honeypot. One tool that attempts to locate rootkits on a machine is chkrootkit. This tool operates under a simple logic, looking for known signatures of the most popular rootkits. For instance it runs strings on binaries looking for rootkit signatures, checks for network interfaces in promiscuous mode, and attempts to find sniffer logs and rootkit config files. It is important to note that chkrootkit will use a handful of system binaries to do its analysis, so it is recommended to use known good binaries. Here is the ouput of chkrootkit in quiet mode:

chkrootkit identified chsh and ifconfig as being trojaned binaries. It identified a file existing in /dev (/dev/.haos/haos1/.f/Denyed), which is not normal. It also found the following suspicious directories and files (excluding false positives): 
/usr/lib/.lib 
/usr/lib/sn/.X
/usr/lib/sn/.sys
/usr/lib/ld/.X
/usr/man/man1/...
/usr/man/man1/.../.m
/usr/man/man1/.../.w
Below is a look through modified files in the past week which reveals what was modified on the honeypot. Remember that I have the luxury of getting a concise output of modified files whereas a compromised and heavily used production system would have a lot of legitimately modified files. Further investigation of the honeypot can be done on a seperate machine, by creating an image of the disk. A tool to create the image is the unix utility dd which will make a bit-by-bit copy. I would like to point out that one may be inclined to use tar or rsync to make a backup of a disk for forensic analysis. This is never a good idea for forensics because information like slack space at the end of files, unused inode information and empty space in the filesystem will be lost. Therefore, an attacker who uses techniques like utilizing slack space (difference between the logical file and the physical file) or writing to a set of blocks and marking the blocks as bad (refer phrack 59) to hide data will go undetected. After the image is created it should be mounted on the forensic workstation read-only,nodev,noexec.

In conclusion, I have to say that there was not anything within the compromise that was groundbreaking in the world of security. This type of compromise happens every single day to machines on the internet. I would like to add that the attacker really did make a mess of the filesystem with files and directories strewn about. I would never feel comfortable, in the case the honeypot was a production system, in simply trying to remove the attackers files and replacing binaries. There is no option except wiping the disk and reinstalling the operating system. I can only recommend keeping up with the latest security patches, deploying intrusion detection technologies like Snort in order to detect possible breaches, and installing file integrity or host intrusion detection systems like Samhain or AIDE.