Snort & OpenAppID on ESXi virtual ubuntu (2 NICs)

A Network Intrusion Detection & Prevention with Snort and OpenAppID (application identification) on a ESXi ubuntu virtual machine (2 NICs) with PF_RING.

 

An open source network intrusion prevention system (NIPS) and network intrusion detection system (NIDS) created by Martin Roesch in 1998 and owned by Cisco since 2013.

 

For our test we need to add a masquerade in iptables between the 2 NICs to make NAT. Take a look on the post about bro framework for the HowTo

 

Prepare & Install

 

OpenAppID

 

Install the pre-requisites from the Ubuntu repositories

 

$ sudo apt-get install -y build-essential libpcap-dev libpcre3-dev libdumbnet-dev bison flex zlib1g-dev liblzma-dev libnghttp2-dev libluajit-5.1-dev pkg-config openssl libssl-dev checkinstall autoconf automake libtool

Create the source folder for our install

 

$ mkdir ~/snort_src
$ cd ~/snort_src


Download and install Data Acquisition library (DAQ)

 

$ wget https://www.snort.org/downloads/snort/daq-2.0.6.tar.gz
$ tar -xvzf daq-2.0.6.tar.gz
$ cd daq-2.0.6
$ ./configure
$ make
$ sudo checkinstall

**********************************************************************

 Done. The new package has been installed and saved to

 /home/sysadmin/snort_src/daq-2.0.6/daq_2.0.6-1_amd64.deb

 You can remove it from your system anytime using:

 dpkg -r daq

**********************************************************************

Use checkinstall, better if you want to remove after. For remove do dpkg -r daq

 

Snort

 

You’re ready to install it, make the install with –enable-open-appid and –enable-sourcefire option to enable OpenAppID support.

 

$ cd ..
$ wget https://snort.org/downloads/snort/snort-2.9.9.0.tar.gz
$ tar -xvzf snort-2.9.9.0.tar.gz
$ cd snort-2.9.9.0
$ ./configure --enable-sourcefire --enable-open-appid
$ make
$ sudo checkinstall

**********************************************************************

 Done. The new package has been installed and saved to

 /home/sysadmin/snort_src/snort-2.9.9.0/snort_2.9.9.0-1_amd64.deb

 You can remove it from your system anytime using:

 dpkg -r snort

**********************************************************************

Same before, use checkinstall to have a debian package (.deb) in the directory.

 

Update the shared libraries

 

sudo ldconfig

Specifics

 

Time to create few directories and files

 

# Create the Snort directories:
sudo mkdir /etc/snort
sudo mkdir /etc/snort/rules
sudo mkdir /etc/snort/rules/iplists
sudo mkdir /etc/snort/preproc_rules
sudo mkdir /usr/local/lib/snort_dynamicrules
sudo mkdir /etc/snort/so_rules
 
# Create some files that stores rules and ip lists
sudo touch /etc/snort/rules/iplists/black_list.rules
sudo touch /etc/snort/rules/iplists/white_list.rules
sudo touch /etc/snort/rules/local.rules
sudo touch /etc/snort/sid-msg.map
 
# Create our logging directories:
sudo mkdir /var/log/snort
sudo mkdir /var/log/snort/archived_logs
 
# Adjust permissions:
sudo chmod -R 5775 /etc/snort
sudo chmod -R 5775 /var/log/snort
sudo chmod -R 5775 /var/log/snort/archived_logs
sudo chmod -R 5775 /etc/snort/so_rules
sudo chmod -R 5775 /usr/local/lib/snort_dynamicrules

Next, copy few files

 

cd ~/snort_src/snort-2.9.9.0/etc/
sudo cp *.conf* /etc/snort
sudo cp *.map /etc/snort
sudo cp *.dtd /etc/snort
cd ~/snort_src/snort-2.9.9.0/src/dynamic-preprocessors/build/usr/local/lib/snort_dynamicpreprocessor/
sudo cp * /usr/local/lib/snort_dynamicpreprocessor/

Finish this part by running the following command

 

sudo sed -i 's/include \$RULE\_PATH/#include \$RULE\_PATH/' /etc/snort/snort.conf

 

Edit the /etc/snort/snort.conf

 

On line 45 put your local network (to protect)

 

###################################################
# Step #1: Set the network variables. For more information, see README.variables
###################################################

# Setup the network addresses you are protecting
ipvar HOME_NET 172.16.0.0/20

# Set up the external network addresses. Leave as "any" in most situations
ipvar EXTERNAL_NET any

Starting line 104 modify few lines like this

 

# Path to your rules files (this can be a relative path)
# Note for Windows users:  You are advised to make this an absolute path,
# such as:  c:\snort\rules
var RULE_PATH /etc/snort/rules
var SO_RULE_PATH /etc/snort/so_rules
var PREPROC_RULE_PATH /etc/snort/preproc_rules

# If you are using reputation preprocessor set these
# Currently there is a bug with relative paths, they are relative to where snort is
# not relative to snort.conf like the above variables
# This is completely inconsistent with how other vars work, BUG 89986
# Set the absolute path appropriately
var WHITE_LIST_PATH /etc/snort/rules/iplists
var BLACK_LIST_PATH /etc/snort/rules/iplists

Uncomment line 546 to enable the local.rules file

 

###################################################
# Step #7: Customize your rule set
# For more information, see Snort Manual, Writing Snort Rules
#
# NOTE: All categories are enabled in this conf file
###################################################

# site specific rules
include $RULE_PATH/local.rules

It’s time to make a first test

 

$ sudo snort -T -i ens160 -c /etc/snort/snort.conf

 --== Initialization Complete ==--

 ,,_ -*> Snort! <*-
 o" )~ Version 2.9.9.0 GRE (Build 56)
 '''' By Martin Roesch & The Snort Team: https://www.snort.org/contact#team
 Copyright (C) 2014-2016 Cisco and/or its affiliates. All rights reserved.
 Copyright (C) 1998-2013 Sourcefire, Inc., et al.
 Using libpcap version 1.7.4
 Using PCRE version: 8.38 2015-11-23
 Using ZLIB version: 1.2.8

 Rules Engine: SF_SNORT_DETECTION_ENGINE Version 3.0 <Build 1>
 Preprocessor Object: SF_POP Version 1.0 <Build 1>
 Preprocessor Object: SF_DCERPC2 Version 1.0 <Build 3>
 Preprocessor Object: appid Version 1.1 <Build 5>
 Preprocessor Object: SF_SMTP Version 1.1 <Build 9>
 Preprocessor Object: SF_REPUTATION Version 1.1 <Build 1>
 Preprocessor Object: SF_FTPTELNET Version 1.2 <Build 13>
 Preprocessor Object: SF_MODBUS Version 1.1 <Build 1>
 Preprocessor Object: SF_SSLPP Version 1.1 <Build 4>
 Preprocessor Object: SF_GTP Version 1.1 <Build 1>
 Preprocessor Object: SF_SDF Version 1.1 <Build 1>
 Preprocessor Object: SF_SSH Version 1.1 <Build 3>
 Preprocessor Object: SF_IMAP Version 1.0 <Build 1>
 Preprocessor Object: SF_DNP3 Version 1.1 <Build 1>
 Preprocessor Object: SF_DNS Version 1.1 <Build 4>
 Preprocessor Object: SF_SIP Version 1.1 <Build 1>

Snort successfully validated the configuration!
Snort exiting

Application Detector Package

 

Download the Application Detector Package, which contains the rules for detecting types of traffic, and copy the extracted files. Create also a folder for thirdparty application detectors.

 

$ cd ~/snort_src
$ wget https://snort.org/downloads/openappid/5048 -O snort-openappid.tar.gz
$ tar -xvzf snort-openappid.tar.gz
$ sudo cp -r ~/snort_src/odp/ /etc/snort/rules/
$ sudo mkdir /usr/local/lib/thirdparty

Enable OpenAppID

 

Edit the file /etc/snort/snort.conf, add pre-processor before Step 6 (after preprocessor reputation) and configure output plugins of Step 6

 

# Reputation preprocessor. For more information see README.reputation
preprocessor reputation: \
 memcap 500, \
 priority whitelist, \
 nested_ip inner, \
 whitelist $WHITE_LIST_PATH/white_list.rules, \
 blacklist $BLACK_LIST_PATH/black_list.rules

preprocessor appid: app_stats_filename appstats-u2.log, \
 app_stats_period 60, \
 app_detector_dir /etc/snort/rules

###################################################
# Step #6: Configure output plugins
# For more information, see Snort Manual, Configuring Snort - Output Modules
###################################################

# unified2
# Recommended for most installs
output unified2: filename snort.u2, limit 128, appid_event_types

# Additional configuration for specific types of installs
# output alert_unified2: filename snort.alert, limit 128, nostamp
# output log_unified2: filename snort.log, limit 128, nostamp

# syslog
# output alert_syslog: LOG_AUTH LOG_ALERT

# pcap
# output log_tcpdump: tcpdump.log

Enable HTTP uri & hostname logging

 

We want logging the URI and the hostname (only logged in Unified2 mode), edit the /etc/snort/snort.conf file and add log_uri and log_hostname

 

# HTTP normalization and anomaly detection. For more information, see README.http_inspect
preprocessor http_inspect: global iis_unicode_map unicode.map 1252 compress_depth 65535 decompress_depth 65535
preprocessor http_inspect_server: server default \
 http_methods { GET POST PUT SEARCH MKCOL COPY MOVE LOCK UNLOCK NOTIFY POLL BCOPY BDELETE BMOVE LINK UNLINK OPTIONS HEAD DELETE TRACE TRACK CONNECT SOURCE SUBSCRIBE UNSUBSCRIBE PROPFIND PROPPATCH BPROPFIND BPROPPATCH RPC_CONNECT PROXY_SUCCESS BITS_POST CCM_POST SMS_POST RPC_IN_DATA RPC_OUT_DATA RPC_ECHO_DATA } \
 chunk_length 500000 \
 server_flow_depth 0 \
 client_flow_depth 0 \
 post_depth 65495 \
 oversize_dir_length 500 \
 max_header_length 750 \
 max_headers 100 \
 max_spaces 200 \
 small_chunk_length { 10 5 } \
 ports { 80 81 311 383 591 593 901 1220 1414 1741 1830 2301 2381 2809 3037 3128 3702 4343 4848 5250 6988 7000 7001 7144 7145 7510 7777 7779 8000 8008 8014 8028 8080 8085 8088 8090 8118 8123 8180 8181 8243 8280 8300 8800 8888 8899 9000 9060 9080 9090 9091 9443 9999 11371 34443 34444 41080 50002 55555 } \
 non_rfc_char { 0x00 0x01 0x02 0x03 0x04 0x05 0x06 0x07 } \
 enable_cookie \
 extended_response_inspection \
 inspect_gzip \
 normalize_utf \
 unlimited_decompress \
 normalize_javascript \
 apache_whitespace no \
 ascii no \
 bare_byte no \
 directory no \
 double_decode no \
 iis_backslash no \
 iis_delimiter no \
 iis_unicode no \
 multi_slash no \
 utf_8 no \
 u_encode yes \
 webroot no \
 log_uri \
 log_hostname

Launch at startup (as systemd service)

Create systemd service file at /lib/systemd/system/snort.service with the following contents:

 

[Unit]
Description=Snort: Network Intrusion Detection & Prevention
After=syslog.target network.target
Documentation=https://www.snort.org/

[Service]
Type=simple
#User=root
#Group=
WorkingDirectory=/var/log/snort
ExecStart=/usr/local/bin/snort -i ens160 -c /etc/snort/snort.conf -k none

[Install]
WantedBy=multi-user.target

Make a symbolic link, activate the service.

 

$ sudo ln -s /lib/systemd/system/snort.service /etc/systemd/system/multi-user.target.wants/snort.service
$ sudo systemctl daemon-reload
$ sudo systemctl enable snort.service

Start and Check the status

 

$ sudo service snort start
$ sudo service snort status

Barnyard2-extra

 

This version contains functionality and features not found (yet) in the original Barnyard2. Support of database storage of “ExtraData” and New database schema to store Unified2 “ExtraData”

 

Pre-requisites

 

$ sudo apt-get install -y mysql-server libmysqlclient-dev mysql-client autoconf libtool

Install

 

$ cd ~/snort_src
$ git clone https://github.com/beave/barnyard2-extra.git
$ cd barnyard2-extra
$ autoreconf -fvi -I ./m4
$ sudo ln -s /usr/include/dumbnet.h /usr/include/dnet.h
$ sudo ldconfig
$ ./configure --with-mysql --with-mysql-libraries=/usr/lib/x86_64-linux-gnu --enable-dns --enable-healthcheck
$ make
$ sudo checkinstall

**********************************************************************

 Done. The new package has been installed and saved to

 /home/sysadmin/snort_src/barnyard2-extra/barnyard2_2.1.14-1_amd64.deb

 You can remove it from your system anytime using:

 dpkg -r barnyard2

**********************************************************************

Make a check

 

$ /usr/local/bin/barnyard2 -V


 ______ -*> Barnyard2 <*-
 / ,,_ \ Version 2.1.14 (Build 336-Quadrant-7)
 |o" )~| By Ian Firns (SecurixLive): https://www.securixlive.com/
 + '''' + (C) Copyright 2008-2013 Ian Firns <firnsy@securixlive.com>
 Modified by Champ Clark <clark@quadrantsec.com> & Adam Hall
 <ahall@quadrantsec.com> - DNS, Health and ExtraData.

Configure Snort to use Barnyard2

 

sudo cp ~/snort_src/barnyard2-extra/etc/barnyard2.conf /etc/snort/
 
# the /var/log/barnyard2 folder is never used or referenced
# but barnyard2 will error without it existing
sudo mkdir /var/log/barnyard2
sudo chown snort.snort /var/log/barnyard2
 
sudo touch /var/log/snort/barnyard2.waldo
sudo chown snort.snort /var/log/snort/barnyard2.waldo

The  MySQL database

 

$ mysql -u root -p
mysql> create database radius;
mysql> use radius;
mysql> source ~/snort_src/barnyard2-extra/schemas/create_mysql
mysql> CREATE USER 'snort'@'localhost' IDENTIFIED BY 'MYSQLSNORTPASSWORD';
mysql> grant create, insert, select, delete, update on radius.* to 'snort'@'localhost';
mysql> exit

Edit the /etc/snort/barnyard2.conf file

 

# database: log to a variety of databases
# ----------------------------------------------------------------------------
#
# Purpose: This output module provides logging ability to a variety of databases
# See doc/README.database for additional information.
#
# Examples:
output database: log, mysql, user=snort password=MYSQLSNORTPASSWORD dbname=radius host=localhost sensor name=sensor01
#   output database: alert, postgresql, user=snort dbname=snort
#   output database: log, odbc, user=snort dbname=snort
#   output database: log, mssql, dbname=snort user=snort password=test
#   output database: log, oracle, dbname=snort user=snort password=test
#

 

Protect the barnyard2.conf file

$ sudo chmod o-r /etc/snort/barnyard2.conf

 

Disable Strict SQL Mode

 

Due to incompatibility we need to desactivate the STRICT_TRANS_TABLES modes, to do this create the file /etc/mysql/conf.d/disable_strict_mode.cnf and put inside

 

[mysqld]
sql_mode=IGNORE_SPACE,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION

Then restart the service mysql 

 

 

Bro on ESXi virtual ubuntu (2 NICs)

A try of Bro on an ESXi ubuntu virtual machine (2 NICs) with GeoIP and PF_RING dependencies.

 

A powerful network analysis framework Originally written by Vern Paxson (Professor of Computer Science at Berkeley), It works with scripts and has support for clustering for high throughput environments. It is actually a very powerful complement to Snort.

 

For our test we need to add a masquerade in iptables between the 2 NICs to make NAT. To do this you need to be logged in root (sudo su).

 

Tell the kernel that you want to allow IP forwarding.

 

$ echo 1 > /proc/sys/net/ipv4/ip_forward

Iptables commands

 

$ /sbin/iptables -t nat -A POSTROUTING -o ens192 -j MASQUERADE
$ /sbin/iptables -A FORWARD -i ens192 -o ens160 -m state --state RELATED,ESTABLISHED -j ACCEPT
$ /sbin/iptables -A FORWARD -i ens160 -o ens192 -j ACCEPT

Add these two lines to allow  ssh

 

$ /sbin/iptables -A INPUT -i ens192 -p tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
$ /sbin/iptables -A OUTPUT -o ens192 -p tcp --sport 22 -m state --state ESTABLISHED -j ACCEPT

Install the ubuntu package iptables-persistent to keep rules after reboot

 

$ apt-get install iptables-persistent

If you make any changes to the rules, run the following commands so you don’t lose them during a system reboot.

 

$ netfilter-persistent save
$ netfilter-persistent reload

Last step, edit the /etc/sysctl.conf file

 

# Uncomment the next line to enable packet forwarding for IPv4
net.ipv4.ip_forward=1

To show masquerade rule in iptable you can use the command

 

$ iptables -L -v -n | more

Bro required Dependencies

 

$ sudo apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev checkinstall lnav

 

Optional Dependencies

 

$ sudo apt-get install sendmail libgeoip-dev curl libgoogle-perftools-dev

Information: For the install I suggest you to create a bro_install folder and put every download inside.

 

GeoIPLite Database Installation

 

For the full install documention, it’s here GeoIP

 

$ wget https://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
$ gunzip GeoLiteCity.dat.gz
$ sudo mv GeoLiteCity.dat /usr/share/GeoIP/GeoIPCity.dat
$ wget https://geolite.maxmind.com/download/geoip/database/GeoLiteCityv6-beta/GeoLiteCityv6.dat.gz
$ gunzip GeoLiteCityv6.dat.gz
$ sudo mv GeoLiteCityv6.dat /usr/share/GeoIP/GeoIPCityv6.dat

PF_RING

 

$ git clone https://github.com/ntop/PF_RING.git
$ cd PF_RING/userland/lib
$ ./configure --prefix=/opt/pfring 
$ make
$ sudo checkinstall

**********************************************************************

 Done. The new package has been installed and saved to

 /home/sysadmin/bro_install/PF_RING/userland/lib/lib_20170328-1_amd64.deb

 You can remove it from your system anytime using:

      dpkg -r lib

**********************************************************************

$ cd ../libpcap
$ ./configure --prefix=/opt/pfring
$ make
$ sudo checkinstall

**********************************************************************

 Done. The new package has been installed and saved to

 /home/sysadmin/bro_install/PF_RING/userland/libpcap-1.7.4/libpcap_1.7.4-1_amd64.deb

 You can remove it from your system anytime using:

      dpkg -r libpcap

**********************************************************************

$ cd ../tcpdump
$ ./configure --prefix=/opt/pfring
$ make
$ sudo checkinstall

**********************************************************************

 Done. The new package has been installed and saved to

 /home/sysadmin/bro_install/PF_RING/userland/tcpdump-4.9.0/tcpdump_4.9.0-1_amd64.deb

 You can remove it from your system anytime using:

      dpkg -r tcpdump

**********************************************************************

$ cd ../../kernel
$ ./configure --prefix=/opt/pfring
$ make
$ sudo checkinstall

Enter a number to change any of them or press ENTER to continue: 3
Enter new version:
>> 6.5.0

**********************************************************************

 Done. The new package has been installed and saved to

 /home/sysadmin/bro_install/PF_RING/kernel/kernel_20170328-1_amd64.deb

 You can remove it from your system anytime using:

      dpkg -r kernel

**********************************************************************

$ sudo modprobe pf_ring enable_tx_capture=0 min_num_slots=32768

Install

 

full install documentation

 

$ cd ~/bro_install
$ git clone --recursive git://git.bro.org/bro
$ cd bro
$ ./configure --with-pcap=/opt/pfring
$ make
$ sudo checkinstall

**********************************************************************

 Done. The new package has been installed and saved to

 /home/sysadmin/bro_install/PF_RING/userland/lib/lib_20170328-1_amd64.deb

 You can remove it from your system anytime using:

      dpkg -r bro

**********************************************************************

Configure the environment variable

 

Edit the /etc/environment file to add the value /usr/local/bro to the path. do the same thing in ~/.profile file.

 

Warning! This will not work for sudo command.

 

$ sudo vim /etc/environment

PATH="/usr/local/sbin:/usr/local/bin:.................:/usr/local/bro/bin

$ vim ~/.profile

PATH="$HOME/bin:$HOME/.local/bin:/usr/local/bro/bin:$PATH"

export also like this (or reboot to take effect)

 

$ export PATH=$PATH:/usr/local/bro/bin

Testing GeoIPLite

 

Check if the GeoIP functionality works by running this command

 

 

$ bro -e "print lookup_location(8.8.8.8);"

Configure PF_RING

 

Print shared library dependencies to see if correctly linked to pf_ring-aware libpcap

 

$ ldd /usr/local/bro/bin/bro | grep pcap
        libpcap.so.1 => /usr/local/lib/libpcap.so.1 (0x00007fa371e33000)
Show information about the Linux Kernel module pf_ring to see if present

 

$ modinfo pf_ring

filename: /lib/modules/4.4.0-62-generic/kernel/net/pf_ring/pf_ring.ko
alias: net-pf-27
version: 6.5.0
description: Packet capture acceleration and analysis
author: ntop.org
license: GPL
srcversion: 414F094C8FD5E8D55A89517
depends:
vermagic: 4.4.0-62-generic SMP mod_unload modversions
parm: min_num_slots:Min number of ring slots (uint)
parm: perfect_rules_hash_size:Perfect rules hash size (uint)
parm: enable_tx_capture:Set to 1 to capture outgoing packets (uint)
parm: enable_frag_coherence:Set to 1 to handle fragments (flow coherence) in clusters (uint)
parm: enable_ip_defrag:Set to 1 to enable IP defragmentation(only rx traffic is defragmentead) (uint)
parm: quick_mode:Set to 1 to run at full speed but with upto one socket per interface (uint)
parm: force_ring_lock:Set to 1 to force ring locking (automatically enable with rss) (uint)
parm: enable_debug:Set to 1 to enable PF_RING debug tracing into the syslog, 2 for more verbosity (uint)
parm: transparent_mode:(deprecated) (uint)

Edit the file node.cfg in /usr/local/bro/etc/ with

 

[worker-1]
type=worker
host=localhost
interface=ens160
lb_method=pf_ring
lb_procs=4
pin_cpus=0,1,2,3

 

 

BroControl

 

An interactive shell for easily operating/managing installations. Warning! BroControl work only in standalone mode, for this part you can comment the pf_ring part of node.cfg

 

Basic configuration

 

Go in /usr/local/bro/etc/ directory. Set the right interface to monitor (node.cfg). Comment out the default settings, add the networks that Bro will consider local to the monitored environment (networks.cfg). Change the email address and the LogRotationInterval if needed (broctl.cfg).

 

Start the BroControl (perform an initial installation for the first time)

 

$ broctl

[BroControl] > install
[BroControl] > start

To stop this instance use the stop command and exit to close the instance.

 

Log files

 

The logs are stored in /usr/local/bro/logs/ directory. current contain the live logs (suggest to open a new ssh connection to the server to take a look to the different files) and the history are in the other directories.

 

Monitoring Live Traffic

 

Edit /usr/local/bro/share/bro/site/local.bro and made change/add value to look like this:

 

##! Local site policy. Customize as appropriate.
##!
##! This file will not be overwritten when upgrading or reinstalling!

# This script logs which scripts were loaded during each run.
@load misc/loaded-scripts

# Apply the default tuning scripts for common tuning settings.
@load tuning/defaults

# Estimate and log capture loss.
@load misc/capture-loss

# Enable logging of memory, packet and lag statistics.
@load misc/stats

# Load the scan detection script.
@load misc/scan

# Detect traceroute being run on the network. This could possibly cause
# performance trouble when there are a lot of traceroutes on your network.
# Enable cautiously.
#@load misc/detect-traceroute

# Generate notices when vulnerable versions of software are discovered.
# The default is to only monitor software found in the address space defined
# as "local". Refer to the software framework's documentation for more
# information.
@load frameworks/software/vulnerable

# Detect software changing (e.g. attacker installing hacked SSHD).
@load frameworks/software/version-changes

# This adds signatures to detect cleartext forward and reverse windows shells.
@load-sigs frameworks/signatures/detect-windows-shells

# log the version of Windows.
@load frameworks/software/windows-version-detection

# Load all of the scripts that detect software in various protocols.
@load protocols/ftp/software
@load protocols/smtp/software
@load protocols/ssh/software
@load protocols/ssh/geo-data
@load protocols/http/software
@load protocols/http/software-browser-plugins
@load protocols/mysql/software
# The detect-webapps script could possibly cause performance trouble when
# running on live traffic. Enable it cautiously.
@load protocols/http/detect-webapps

# This script detects DNS results pointing toward your Site::local_nets
# where the name is not part of your local DNS zone and is being hosted
# externally. Requires that the Site::local_zones variable is defined.
@load protocols/dns/detect-external-names

# Script to detect various activity in FTP sessions.
@load protocols/ftp/detect

# Scripts that do asset tracking.
@load protocols/conn/known-hosts
@load protocols/conn/known-services
@load protocols/ssl/known-certs

# This script enables SSL/TLS certificate validation.
@load protocols/ssl/validate-certs

# This script prevents the logging of SSL CA certificates in x509.log
@load protocols/ssl/log-hostcerts-only

# Uncomment the following line to check each SSL certificate hash against the ICSI
# certificate notary service; see https://notary.icsi.berkeley.edu .
# @load protocols/ssl/notary

# If you have libGeoIP support built in, do some geographic detections and
# logging for SSH traffic.
@load protocols/ssh/geo-data
# Detect hosts doing SSH bruteforce attacks.
@load protocols/ssh/detect-bruteforcing
# Detect logins using "interesting" hostnames.
@load protocols/ssh/interesting-hostnames

# Detect SQL injection attacks.
@load protocols/http/detect-sqli

#### Network File Handling ####

# Enable MD5 and SHA1 hashing for all files.
@load frameworks/files/hash-all-files

# Detect SHA1 sums in Team Cymru's Malware Hash Registry.
@load frameworks/files/detect-MHR

# Uncomment the following line to enable detection of the heartbleed attack. Enabling
# this might impact performance a bit.
# @load policy/protocols/ssl/heartbleed

# Uncomment the following line to enable logging of connection VLANs. Enabling
# this adds two VLAN fields to the conn.log file.
# @load policy/protocols/conn/vlan-logging

# Uncomment the following line to enable logging of link-layer addresses. Enabling
# this adds the link-layer address for each connection endpoint to the conn.log file.
@load policy/protocols/conn/mac-logging

# Uncomment the following line to enable the SMB analyzer. The analyzer
# is currently considered a preview and therefore not loaded by default.
# @load policy/protocols/smb

# Finds connections with protocols on non-standard ports with DPD.
@load frameworks/dpd/detect-protocols

# Logs in JSON by default.
@load tuning/json-logs

 

Analyzing live traffic from an interface (basic with json answer):

 

$ bro -i ens160 /usr/local/bro/share/bro/site/local.bro "Site::local_nets += { 172.16.0.0/20 }"

Warning! all log files will be created in the directory where you launch the command.

 

Launch at startup (as systemd service)

 

It’s time to create a systemd service and control it with systemctl management tools. Create systemd service file at /lib/systemd/system/bro.service with the following contents:

 

[Unit]
Description=Bro: a powerful network analysis framework
After=syslog.target network.target
Documentation=https://www.bro.org/sphinx/index.html

[Service]
Type=simple
#User=root
#Group=
WorkingDirectory=/usr/local/bro/logs/current
ExecStart=/usr/local/bro/bin/bro -i ens160 /usr/local/bro/share/bro/site/local.bro "Site::local_nets += { 172.16.0.0/20 }"

[Install]
WantedBy=multi-user.target

We use the WorkingDirectory variable to specify the /usr/local/bro/logs/ current for our log files

 

Create a symbolic link and enable the service with systemctl

$ sudo ln -s /lib/systemd/system/bro.service /etc/systemd/system/multi-user.target.wants/bro.service
$ sudo systemctl daemon-reload
$ sudo systemctl enable bro.service

Start and Check the status

 

$ sudo service bro start
$ sudo service bro status

See you next time to realize a monitoring application of our live log datas in Angular, NodeJS and socket.io

 

How-To: Secure Ubuntu server (fail2ban)

Fail2ban scan the log files of the server and bans IPs that show the malicious signs. Like for exemple too many password failures, seeking for exploits, etc.. It work as a service and create rules that automatically alter iptables configuration. All based on a predefined number of unsuccessful login attempts. This will allow the server to respond to illegitimate access attempts without manual intervention.

 

Install

 

Fail2ban is in package list of Ubuntu. To install it from a command prompt do like this (update first).

 

xxxx@xxxx:~$ sudo apt-get update
xxxx@xxxx:~$ sudo apt-get install fail2ban

 

Configure

 

Configuration files are in the /etc/fail2ban directory.

 

First stop the service

 

xxxx@xxxx:~$ sudo service fail2ban stop

Duplicate the config file jail.conf to keep default options inside (this file can be overwriten when update applied). Put all the specific settings in jail.local

 

xx:~$ sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Now we can modify the jail.local file to adjust to our server case

 

The base

 

Setup the base consist to add one or more source @ip to ignore, the bann time and the number of retry allowed. To do this we need to modify the variables:

 

  1. ignoreip = @ip to ignore (separated by a space).
  2. bantime (in second) = parameter for banned client (default 10 minutes).
  3. findtime (in second) =  a window of time to find a specific number of tries (see below).
  4. maxretry = number of tries before being banned. By default ban a client after 5 tries in 10 minutes (findtime variable).

 

In our case we add the @ip  192.168.0.15 to the list to ignore, change the bantime to 1800 sec (1/2 hour). To do this we edit the /etc/fail2ban/jail.local file like this

 

# The DEFAULT allows a global definition of the options. They can be overridden
# in each jail afterwards.

[DEFAULT]

#
# MISCELLANEOUS OPTIONS
#

# "ignoreip" can be an IP address, a CIDR mask or a DNS host. Fail2ban will not
# ban a host which matches an address in this list. Several addresses can be
# defined using space separator.
ignoreip = 127.0.0.1/8 192.168.0.15

# External command that will take an tagged arguments to ignore, e.g. <ip>,
# and return true if the IP is to be ignored. False otherwise.
#
# ignorecommand = /path/to/command <ip>
ignorecommand =

# "bantime" is the number of seconds that a host is banned.
bantime  = 1800

# A host is banned if it has generated "maxretry" during the last "findtime"
# seconds.
findtime  = 600

# "maxretry" is the number of failures before a host get banned.
maxretry = 5

 

Email alerts

 

Configure email alerts with the variable destemail, sendername, and mta. To use it you need the Ubuntu package sendmail!

 

#
# ACTIONS
#

# Some options used for actions

# Destination email address used solely for the interpolations in
# jail.{conf,local,d/*} configuration files.
destemail = sysadmin@imtase.com

# Sender email address used solely for some actions
sender = sysadmin@imtase.com

# E-mail action. Since 0.8.1 Fail2Ban uses sendmail MTA for the
# mailing. Change mta configuration parameter to mail if you want to
# revert to conventional 'mail'.
mta = sendmail

we need to adjust the action parameter too.

 

# Choose default action.  To change, just override value of 'action' with the
# interpolation to the chosen action shortcut (e.g.  action_mw, action_mwl, etc) in jail.local
# globally (section [DEFAULT]) or per specific section
action = %(action_mwl)s

 

SSH

 

To activate a service juste need a line enabled = true in the appropriate section. Exemple with [ssh]

 

#
# SSH servers
#

[sshd]
enable  = true
port    = ssh
logpath = %(sshd_log)s

We can no restart fail2ban

 

xxxx@xxxx:~$ sudo service fail2ban start

If you go want deeper in fail2ban I suggest this excellent post from Linode

 

Fail2ban Client

 

use the command fail2ban-client with one of these command to action/check information:

 

  1. start: Starts the server and jails.
  2. reload: Reloads configuration files.
  3. reload JAIL: Replaces JAIL with the name of a Fail2ban jail; this will reload the jail.
  4. stop: Stop the server.
  5. status: Show the status of the server, and enable jails.
  6. status JAIL: Show the status of the jail, including any currently-banned IPs.

 

Replace JAIL by the service you want to check, exemple with ssh

 

xxxx@xxxx:~$ sudo fail2ban-client status
Status
|- Number of jail: 1
`- Jail list: sshd
xxxx@xxxx:~$ sudo fail2ban-client status sshd
Status for the jail: sshd
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /var/log/auth.log
`- Actions
 |- Currently banned: 0
 |- Total banned: 0
 `- Banned IP list:

 

NMAP

 

Install and use the port scanning NMAP

 

xxxx@xxxx:~$ sudo apt-get install nmap

To scan the port of the server you can use the command

 

xxxx@xxxx:~$ nmap -sV -p 1-65535 localhost

-sV   = service identification
-p     = list of port to scan (range separate by -)

 

Can pass hostnames, IP addresses, networks, etc. (localhost, www.my.com, range like 192.168.1.1/24 or 192.168.1.1-50, host like 192.168.1.1/32)

 

 

You have now made your Ubuntu Server more secure, in part 3 we will see the usage of DPI (Deep Packet Inspection)

 

How-To: Secure Ubuntu server (part 1)

Increase the security and usability of your Ubuntu server is very important and do at the same times you install it is the best way. There are few configuration/install that you should take early on as part of the basic setup.

 

Install

 

The install of Ubuntu Server is easy and not need a detailed how-to for this, the only point you need to take care is the manual partitioning of your hard drive (depend of your case). Suggest to follow ubuntuserverguide documentation or this one from ubuntu. The second thing to take care is to use a strong password scheme (Upper/Lower/Number/Special) who stay easy to remember (not have to write), to write (your fingers will say thanks)

 

Ubuntu Root Login

 

Never use directly the user root and prefer to create a new user (in our case we will use an account named admin with sudo power) and use a strong password scheme (Upper,  Lower, Number, Special) who stay easy to remember for not have to write it.

As root, run this command to add your new user to the sudo group

 

xxxx@xxxx:~$ usermod -aG sudo admin

 

 

Set up public key authentication for your new user. Setting this up will increase the security of your server by requiring a private SSH key to log in.

 

Generate The Key Pair for SSH

 

If you haven’t an SSH key pair already you can create it by following this process. To generate a new key pair, enter the following command (use the option -b 4096 for higher security) in your terminal.

 

xxxx@xxx:~$ ssh-keygen -b 4096

Assuming your local user is “admin”, you will see the following output:

 

admin@xxx:~$ ssh-keygen -b 4096
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/admin/.ssh/id_rsa):

Hit return to accept

 

Created directory '/Users/admin/.ssh'
Enter passphrase (empty for no passphrase):
Enter same passphrase again:

Securing your keys with passphrases is more secure, but in this case you need to use it each time you connect. The choice depend of the level of security you want.

 

At the end you will have an output like this

 

Your identification has been saved in /home/admin/.ssh/id_rsa.
Your public key has been saved in /home/admin/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:XXXXXXXXXXXXXXXXXXX5GcBMBXXXXXXXXXXM admin@xxx
The key's randomart image is:
+---[RSA 4096]----+
|XXXX             |
|XX               |
|X X XX XXX       |
|XXX X XXX        |
|X XXXX X  XX     |
| XX XX           |
|   X             |
| X               |
|                 |
+----[SHA256]-----+

You have now 2 files in the directory /home/admin/.ssh/ a private key id_rsa and a public key id_rsa.pub

 

Remember that the private key id_rsa should not be given to anyone who should not have the right to access to your server!

 

Rename the Public Key

 

if you generate the keys directly on the server rename the public key id_rsa.pub in authorized_keys like this

 

admin@xxx:~/.ssh$ sudo mv id_rsa.pub authorized_keys

And retrieve the private key id_rsa on your computer (not let it on the server!)

 

Putty case

 

Putty users, you need to load the private key id_rsa in PuTTYgen then save the private key for have it in .ppk format

 

Disabling Password Authentication

 

If you were able to login to your account using SSH with the private key then you have successfully configured SSH key-based authentication to your account. We can now remove the authentication with password only in the ssh config file (not hesitate to change also the port number 22 if you want).

 

admin@xxx:~/.ssh$ sudo vim /etc/ssh/sshd_config

Search for a directive called PasswordAuthentication. This may be commented out. Uncomment the line and set the value to “no“. This will disable your ability to log in through SSH using account passwords.

 

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Change to no to disable tunnelled clear text passwords
PasswordAuthentication no

Save and close the file when you are finished. To actually implement the changes we just made, you must restart the service.

 

admin@xxx:~/.ssh$ sudo service ssh restart

 

Basic Firewall

 

The default firewall configuration tool for Ubuntu is ufw. It’s an interface to iptables.

 

Allow an application or a TCP/UDP port

 

To add an application you can use the command below to list all application

 

admin@xxx:~/$ sudo ufw app list

You will have this in Output

 

Available applications:
 OpenSSH

To allow OpenSSH (ssh) use this command

 

admin@xxx:~/$ sudo ufw allow OpenSSH

You can also use directly the TCP or UDP port number

 

in standard case of port 22

 

admin@xxx:~/$ sudo ufw allow ssh

in case you change the port use this (where XX is the port number).  keep always the port number below 1024 as these are privileged ports that can only be opened by root or processes running as root. A good link to find an available tcp port under 1024 : wikipedia

 

admin@xxx:~/$ sudo ufw allow XX

when you want specify a specific protocole add the proto tcp (exemple for TCP only)

 

Activate ufw

 

To enable ufw, use this command:

 

 admin@xxx:~/$ sudo ufw enable
 Command may disrupt existing ssh connections. Proceed with operation (y|n)?

Answer “y” to the question for proceed

 

To list the active rules you can use the command

admin@xxx:~/$ sudo ufw status

The output will be something like this

 

Status: active

To Action From
-- ------ ----
OpenSSH ALLOW Anywhere
OpenSSH (v6) ALLOW Anywhere (v6)

More commands on ufw in this excellent post of DigitalOcean : UFW Setup

 

You have now the base for a Secure Ubuntu Server, in part 2 we will see the usage of Fail2Ban to scan logs and ban suspicious hosts and scan open Ports with Nmap

 

vmware ESXi 6.5 with custom NIC

Create a custom wmware ESXi 6.5 with package for NIC drivers.

 

When you want to make a test of virtualization, with the standard ISO to install vmware ESXi, on a non server machine. Then what happen in the middle of the install? The system tell you bye, go home, no want to play with you because I not found you NIC!

 

KEEP HOPE! Nothing is over and the game continue!

 

I want to use it, it’s a very good bare-metal hypervisor and the version 6.5 with the webgui  is really nice and easy to use. No application to install, you manage everything with the webgui, have a good community and a lot of documentation.

 

After few search I found a solution working for me and I hope for you too.

 

Thank you Andreas Peetz for your help with your ESXi-Customizer-PS tool!

 

source for more information:
https://www.v-front.de/2016/11/esxi-65-release-notes-for-free-license.html
https://vibsdepot.v-front.de/wiki/index.php/Net55-r8168

 

HowTo proceed

 

Follow these steps to proceed of the creation of your personalized vmware hypervisor installation ISO.

 

Prerequise

 

Download and install everything you need and if you not have yet register on my vmware (it’s FREE).
ESXi-Customizer-PS the Powershell script  who do everything for you.
VMware PowerCLI – installer the Powershell client of vmware (can’t use the Windows PowerShell to do it)

 

To proceed you need to authorize the Powershell script to execute on your computer (where you want to create your personalized ISO).

 

Open the Windows PowerShell in administrator right (right click and select “run as administrator”). From the PowerShell prompt enter: Set-ExecutionPolicy RemoteSigned.

You can refer here https://blogs.vmware.com/PowerCLI/ for a most detailed process.

 

After done you’re ready to build your personalized ISO.

 

build the ISO

 

Launch your VMware PowerCLI, go in directory of the ESXi-Customizer-PS script (in my case ESXi-Customizer-PS-v2.5.ps1) and type the command line:

 

[qodef_blockquote text=” .\ESXi-Customizer-PS-v2.5.ps1 -v65 -vft -load net55-r8168″ title_tag=”h6″ width=”75″]
I use the -v65 to create a 6.5 version of ESXi, the -vft to connect the V-Front Online Depot and other depots and in my case -load for the package Net55-r8168 in need for Realtek 8168/8111/8411/8118 based NICs drivers.

 

When finished you can burn the ISO, put it in the machine to install, start and quietly drink a cup of coffee.

 

 

Tips: to have a higher resolution for the screen size of the esxi terminal you need to edit /etc/default/grub and change:

 

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
#GRUB_HIDDEN_TIMEOUT=0
GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=2
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX=""

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
GRUB_GFXMODE=1024x768
GRUB_GFXPAYLOAD_LINUX=keep

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

Not forgot to do a sudo update-grub and sudo reboot after.

 

Next time, we will see how to make a nvidia gpu card working in passthrough and a comparaison with Unraid 6.3 from Limetch, who is more that only an hypervisor.