Graylog Extractors for pfSense 2.2 filter logs

Hi all,

I’m trying out Graylog for log collection, aggregation, and analysis. It’s free and pretty damn easy to deploy, available in OVA format.

The first thing I noticed is there seemed to be no extractors for pfSense 2.2’s new log format. Extractors allow you to parse a syslog message and place certain values into ‘fields’ for analysis or use in graphs.

Here’s one I prepared relatively quickly. You can import by:

  1. Click System -> Inputs in the Graylog UI
  2. Click ‘Manage extractors’ next to the relevant input
  3. Click ‘Import extractors’ in the ‘Actions’ menu at the top right of the page
  4. Paste the below script into the window and then click ‘Add extractors to input’

The extractors will parse the following fields out of the pfSense 2.2 filterlog messages:

  • Rule number into pfsense_filter_rulenum
  • Direction into pfsense_filter_direction
  • Ingress interface into pfsense_filter_ingress
  • Action into pfsense_filter_action
  • Protocol into pfsense_filter_proto
  • Source IP into pfsense_filter_sourceip
  • Source Port into pfsense_filter_sourceport
  • Destination IP into pfsense_filter_destip
  • Destination Port into pfsense_filter_destport

Right now they only interpret IPv4 logs, IPv6 log entries don’t get parsed (thanks to the condition regex) as they are formatted differently.

The script is available here, or click ‘Continue Reading’.

Hope this helps!

(more…)

Ruby 1.9, Rails 2.3, and MySQL on Ubuntu 8.10 (Intrepid Ibex)

I haven’t the time to write out a big fleshy post atm, so I’ll post the short story.

I was trying to set my Ubuntu 8.10 VPS up with a fresh Ruby/Rails install to host some webapps I’m working on. Rather than apt-getting my way to glory, I decided to build Ruby 1.9.1 from source, the main reason being that the version available through apt is only 1.9.0 and I’m working with 1.9.1 on my dev machine.

After building/installing Ruby, I started installing rails, rake, rack, etc… via RubyGems. I knew that I couldn’t install the standard ‘mysql’ gem, as it hasn’t yet been updated for Ruby 1.9, so I added http://gems.github.com/ to my gem sources – if you don’t know how to do this, the command is:

gem sources -a http://gems.github.com

…and proceeded to install kwatch’s mysql-ruby gem.

To my surprise (as it had worked under OSX on my dev box), I got the following message:

plasma@syn-app01:~$ sudo gem install kwatch-mysql-ruby
Building native extensions. This could take a while...
ERROR: Error installing kwatch-mysql-ruby:
ERROR: Failed to build gem native extension.

/usr/local/bin/ruby extconf.rb
Trying to detect MySQL configuration with mysql_config command...
Succeeded to detect MySQL configuration with mysql_config command.
checking for mysql_ssl_set()... yes
checking for rb_str_set_len()... yes
checking for rb_thread_start_timer()... no
checking for mysql.h... yes
creating Makefile

make
gcc -I. -I/usr/local/include/ruby-1.9.1/i686-linux -I/usr/local/include/ruby-1.9.1/ruby/backward -I/usr/local/include/ruby-1.9.1 -I. -DHAVE_MYSQL_SSL_SET -DHAVE_RB_STR_SET_LEN -DHAVE_MYSQL_H -D_FILE_OFFSET_BITS=64 -I/usr/include/mysql -DBIG_JOINS=1 -fPIC -fPIC -O2 -g -Wall -Wno-parentheses -fPIC -o mysql.o -c mysql.c
In file included from /usr/include/stdlib.h:320,
from /usr/local/include/ruby-1.9.1/ruby/ruby.h:50,
from /usr/local/include/ruby-1.9.1/ruby.h:32,
from mysql.c:6:
/usr/include/sys/types.h:151: error: duplicate 'unsigned'
make: *** [mysql.o] Error 1

Gem files will remain installed in /usr/local/lib/ruby/gems/1.9.1/gems/kwatch-mysql-ruby-2.8.1 for inspection.
Results logged to /usr/local/lib/ruby/gems/1.9.1/gems/kwatch-mysql-ruby-2.8.1/ext/gem_make.out

Instructions on fixing the issue and installing the gem after the jump.

(more…)