cipherdyne.org

12 January, 2009

The 1.9.10 release of fwknop is ready for download. This release adds a few new features such as the ability to send SPA packets over HTTP requests, and the ability to sniff interfaces without requiring any IP address to be assigned. Several minor bug fixes were made as well such as restoring the ability to send SPA packets over ICMP and properly decode them on the fwknopd server system.

Here is the complete ChangeLog:

• Added the ability to send SPA packet over HTTP to a webserver. This requires that the same running fwknopd is also running a webserver, or that ENABLE_TCP_SERVER is enabled so that fwknopd spawns fwknop_serv to listen on a real TCP socket. Sending SPA packets over HTTP is accomplished with a new command line argument --HTTP on the fwknop client command line, and via a new configuration variable ENABLE_SPA_OVER_HTTP in the fwknop.conf file.
• Added ENABLE_EXTERNAL_CMDS for fwknopd to control whether the EXTERNAL_CMD_OPEN and EXTERNAL_CMD_CLOSE directives are used (instead of just checking whether they are set to __NONE__).
• Bug fix to make sure to properly construct hash reference for the "include" command list for the check_commands() function when checking for the mail command.
• Bug fix for fwknopd to not require Net::Pcap::lookupnet() to succeed on interfaces with no IPv4 address assigned. This function sets the IP and netmask of the local interface, but if fwknopd sniffs an interface without any IP assigned, then such information will not necessarily exist.
• Bug fix to add --Override config support to knopwatchd (Franck Joncourt).
• Bug fix to add client timeout (--fw-timeout) support to both forward NAT and local NAT modes (Damien Stuart). This required increasing the number of expected fields in decrypted SPA packets in fwknopd.
• Bug fix in the install.pl script for Cygwin systems (or others where a client-mode only install is done) to take into account the newer perl library path handling code.
• Updated minimum ICMP header length to 8 bytes in fwknopd to accept spoofed SPA packets over ICMP echo requests.
• Added config dumping support to knopwatchd with -D (Franck Joncourt).
• Minor code cleanups and updates to knopwatched (such as the usage of isspace() to allow tab chars between variable names and values in the fwknop.conf file (Franck Joncourt).
• Added ENABLE_COOKED_INTF to force fwknopd to always treat the sniffing interface as the "cooked" interface type found on Linux.
• Updated knopwatchd to allow more than one overwrite file (Franck Joncourt).
• Added --Single-mod-install to the perl installer so that individual module dependencies can be installed piecemeal.
• (Test suite): Bug fix for the proper usage of the 'ps' command on FreeBSD and Mac OS X systems. The test suite now runs successfully on these systems after this fix.
• (Test suite): Added the ability to test sending SPA packets over established TCP connections with the fwknop_serv daemon.
• (Test suite): Added support for collecting *.warn and *.die output for each test as it is executed and appending this data to each test output file.

13 November, 2008

The 1.9.9 release of fwknop is ready for download. This is a fairly major update to allow integration with third party applications with the ability to execute external programs on SPA open and SPA close. There were also updates made to the fwknop signal handling code, the IPTables::ChainMgr and IPTables::Parse modules for Linux systems, and a few bug fixes as well.

There will probably be a 1.9.10 release before the 2.0 release, and it will most likely include support for the PF firewall, and more advanced support for using SPA in the Amazon EC2 cloud computing service.

Here is the complete ChangeLog:

• Added support to fwknop for the Linux 'any' interface which allows SPA packets to be received on multiple interfaces on a Linux system. This is useful for running fwknop on a dual-homed Linux host, and then accepting SPA packets on either the internal or external interface so that SPA packets can influence the packet filter from either network.
• Added support for interfacing fwknop with third party software through the addition of three new variables in the access.conf file (or set globally in the fwknop.conf file): EXTERNAL_CMD_OPEN, EXTERNAL_CMD_CLOSE, and EXTERNAL_CMD_ALARM. The "open" and "close" commands might be manually supplied firewall commands, and both support variable substitution of any of the variables in the access.conf file with "$VAR". Also, three special variables are supported: $SRC, $PORT, and $PROTO, which are derived from actual values from within valid SPA packets (as opposed to $SOURCE from access.conf which may contain a list of networks instead of a single IP address). Here are some examples:
  Execute a specific iptables command on behalf of the source IP in a valid SPA packet to add a new ACCEPT rule, and execute another command (to delete the same rule after a timeout):
  EXTERNAL_CMD_OPEN iptables -A INPUT -s $SRC -j ACCEPT
  EXTERNAL_CMD_CLOSE iptables -D INPUT -s $SRC -j ACCEPT
• Execute a custom binary with the SOURCE and OPEN_PORTS variables from the access.conf file as input on the command line, and after a timeout execute a different program but use the real SPA source IP:
  EXTERNAL_CMD_OPEN /path/someprog $SOURCE $OPEN_PORTS
  EXTERNAL_CMD_OPEN /path/otherprog $SRC
• Added IPT_CMD_ALARM to control the number of seconds that the IPTables::ChainMgr module uses to wrap alarm() calls around iptables commands (for IPTables::ChainMgr 0.8 and later, although this does not interfere with earlier versions of the module).
• Added IPT_EXEC_STYLE to control the execution method used for iptables commands in the IPTables::ChainMgr module. The default is "waitpid", but "system", and "popen" are also supported.
• Added IPT_EXEC_SLEEP to control the number of seconds that the IPTables::ChainMgr module uses to delay between each iptables command. The default is zero (no delay), but this can be increased to ensure that iptables commands are issued at a slower pace.
• Added IPT_EXEC_TRIES to allow critical iptables commands to be tried multiple times (with a default of 1) in case there are any errors from iptables execution.
• Added --Override-config to fwknopd (suggested by Franck Joncourt) to allow config variables in the normal /etc/fwknop/fwknop.conf file to be superseded by values from other specified files. The --Override-config command line argument accepts a comma-separated list of multiple files from which to import configuration variables from.
• Added code to prefer the usage of the /usr/sbin/sendmail binary to send email alerts before falling back to the mail binary (suggested by Alexander Perlis).
• Added --Dump-config to fwknopd (suggested by Franck Joncourt).
• Added execution of --Dump-config to the test suite to collect the installed version of the fwknop.conf and access.conf files (personal information is anonymized).
• Updated fwknopd to use the POSIX sys_wait_h API for SIGCHLD handling in order to be more consistent with an example from the perlipc man page.
• Updated fwknopd to pass in a reference to the SIGCHLD signal handler to the IPTables::ChainMgr module so that all command executions via fork() and exec() are associated with the same signal handler.
• Updated to IPTables::ChainMgr version 0.8.
• Updated to IPTables::Parse version 0.7.
• (Test suite): Added time stamps to MSG and TEST lines for each test (useful to see the relative time if an alarm expires).
• (Test suite): Added tests for fwknopd --Override-config, --Dump-config, and writing SPA packets to disk with the --Save-packet functionality (in the fwknop client).
• (Test suite): Added tests for IPT_EXEC_SLEEP delays for executing iptables commands.
• (Test suite): Added tests for Linux 'any' interface capture of SPA packets on all interfaces.
• (Test suite): Added the ability to collect output from knoptm to see when requests are received from fwknopd and when rules are added and removed.
• Added version information for fwknopd to syslog startup message.
• Bug fix for the fwknop client in symmetric key mode where the terminal would not be taken out of 'noecho' mode if a password less than 8 characters long is provided. Previous to this fix, it was necessary to blindly type 'reset'. (Reported by Alexander Perlis.)

18 October, 2008

New versions (0.7 and 0.8) of the IPTables::Parse and IPTables::ChainMgr modules are available for download, and Trac is now setup to visualize the module sources. Franck Joncourt has also packaged both modules for the Debian Linux distro: libiptables-parse-perl_0.7-1_all.deb and libiptables-chainmgr-perl_0.8-1_all.deb. The IPTables::Parse and IPTables::ChainMgr modules provide important functionality to all three of the psad, fwsnort, and fwknop projects.

In psad, the IPTables::Parse module is primarily used to look for default logging rules in the INPUT and FORWARD chains in the local iptables policy. Such rules are necessary for psad to detect port scans and other malicious traffic because it uses iptables log information as its data source. Traffic that does not hit an ACCEPT rule should normally be logged and dropped anyway, and psad tries to programmatically determine whether the iptables policy is indeed configured in this way. The IPTables::ChainMgr module is also used by psad to add and delete new blocking rules against malicious IP addresses whenever automatic responses are configured (see the ENABLE_AUTO_IDS variable in the /etc/psad/psad.conf file).

fwknop uses the IPTables::ChainMgr module to add ACCEPT rules for IP addresses that have been properly authenticated either by generating a proper port knocking sequence (which is parsed out of iptables log messages), or by sending a valid SPA message. The knoptm daemon then uses the IPTables::ChainMgr to delete any rules that were added by the fwknopd daemon after a configurable time delay.

fwsnort uses the IPTables::Parse module to look for iptables rules that block traffic that matches the transport layer criteria in translated Snort rules. That is, if the iptables policy blocks all attempts to communicate with a web server, then it is pointless to translate Snort rules that are designed to detect attacks against web servers. While fwsnort uses the iptables state tracking modules to ensure that application layer attacks are only searched for in established TCP connections (so spoofing a TCP packet with a malicious payload does not cause an event to be generated), if the iptables policy blocks all traffic anyway for some particular TCP port then it is more efficient to exclude signatures that detect attacks on this port.

The updates to the IPTables::ChainMgr module are as follows (and similar changes have been made to the IPTables::Parse module as well):

• Added the ability to control the iptables execution model. The default is to use waitpid(), but other options are to use system() or popen().
• Added the ability to introduce a configurable time delay between each iptables command.
• Added the ability to use a function reference for the SIGCHLD signal handler.
• Added the ability to configure the number of seconds used as the alarm timeout for iptables command execution in the waitpid() execution model.
• Passed IPTables::ChainMgr options for the execution model, configurable alarm timeouts, the SIGCHLD signal handler reference, and the configurable number of seconds for additional sleeps between iptables commands to the IPTables::Parse module.
• Bugfix for SIGALRM handling to be more consistent with an example from the perlipc man page.
• Added append_ip_rule() so that new iptables rules can be appended to the end of a chain instead of just inserted at a particular rule number.

This software release moves the state of both modules closer to being ready for submission to CPAN, and I've applied for a CPAN developer account to get this process rolling.

01 October, 2008

The 1.9.8 release of fwknop is ready for download. This release introduces support for SPA packets encrypted with gpg2, fixes issues around the usage of GnuPG options files (fwknop now does not reference them at all by default, but there are new --gpg-use-options and GPG_USE_OPTIONS directives to override this), and adds configurable base64 encoded prefixes. Normally the fwknop client strips the 'hQ' prefix (base64 encoded 0x8502) before sending an SPA packet encrypted with GnuPG out on the wire, and the fwknopd server adds it back in before base64 decoding. This is to make it more difficult to write Snort rules to detect fwknop communications.

Here is the complete ChangeLog:

• Made the updated UI from Sean Greven available on cipherdyne.org. This update fixes the timezone problem so that SPA packets generated by the UI will be properly handled by an fwknopd server.
• Added GPG_NO_REQUIRE_PREFIX to access.conf to control whether the GnuPG 'hQ' prefix is added before base64 decoding and decrypting. Normally this is not needed, but if there appear to be communications issues between the fwknop client and the fwknopd server then this option can be useful to ensure that encrypted SPA data is sent through the GnuPG decryption routine. The 'hQ' prefix is a heuristic derived from the file 'magic' database for describing data encrypted with GnuPG, and the fwknop client normally strips this data from outgoing SPA packets (unless the --Include-gpg-prefix option is used).
• Added 'GPG_PATH <path> to fwknopd (via access.conf) so that different paths to the gpg binary can be specified on a per-SOURCE basis. This allows one SOURCE stanza to apply one gpg binary to decrypt incoming SPA packets (say /usr/bin/gpg), and another SOURCE stanza to apply to another gpg binary (say /usr/bin/gpg2). In this way, fwknop/fwknopd now supports gpg2 in addition to gpg.
• Bugfix to make sure that neither fwknop nor fwknopd reference any options file in GnuPG mode, and this is now the default (which overrides the now unnecessary --gpg-no-options arg). There is a new option --gpg-use-options and GPG_USE_OPTIONS to restore the usage of an options file by GnuPG by fwknop and fwknopd (not normally needed).
• Added '--gpg-prefix <bytes> to the fwknop client so that the predictable prefix for GnuPG encrypted data can be changed. Normally this prefix is 'hQ' (base64 encoded), or the raw bytes 0x8502.
• Added the ability to control the path used for the gpg binary on the client side with a new argument '--gpg-path <path> and on the server side with gpgCmd in the fwknop.conf file. The GnuPG::Interface module normally just takes the first instance of gpg that is the current path, but this new feature allows the path to the binary to be explicitly set.
• Added --Save-packet-append to allow SPA packets to be appended to the --Save-packet-file in --Save-packet mode. This allows multiple SPA packets to more easily be stored for closer examination (i.e. to make sure randomness is high or to test encryption properties over large sets of SPA packets).
• Updated fwknopd to enforce the DIGEST_TYPE variable more strictly by not accepting SPA packets that do not include digest of the specified type. The DIGEST_TYPE default is 'ALL', so normally fwknopd accepts any supported digest.
• Bugfix to make sure to apply BLACKLIST checks to IP addresses specified with -a (or derived via -R) in addition to the source IP in the IP header (which can be modified via --Spoof-src). (Franck Joncourt submitted a patch for this.)
• Bugfix to ensure that the permissions for the /var/run/fwknop/knopwatchd.pid file are set to 0600 (noticed by Franck Joncourt).
• Bugfix to remove the Net::IPv4Addr dependency in the fwknop client and knoptm daemon (Franck Joncourt).
• (Test suite) Added the base64_byte_frequency.pl script to the test/ directory. This script parses files that contain base64 encoded data (one record per line), and produces data files that can be graphed with Gnuplot in order to visualize SPA packets. The new --Save-packet-append argument makes it easy to generate large collections of SPA packets with the fwknop client, and this data can then be parsed by base64_byte_frequency.pl to look for features that are common across SPA packets (this should be minimized because every fwknop SPA packet contains 16 bytes of random data).
• (Test suite) Added tests for GPG_NO_REQUIRE_PREFIX functionality and for the expected GnuPG prefix.
• (Test suite) Added tests for GnuPG version 2 (a check is made to see if it is installed before these tests are run).

31 August, 2008

The 1.9.2 release of gpgdir is ready for download. This release introduces new functionality to support the recursive signing and verification of files within a directory in addition to the usual recursive encryption/decryption cycle. Each signature is created as a detached .asc file, which GnuPG normally creates with the '-b -a' arguments. As an illustration of this, suppose that you want to recursively sign all files with a .tar.bz2 extension within a directory "/home/user/data" and all of its sub-directories. This could be useful if you need to switch GnuPG keys after one key expires, and update all signatures to be generated from a new key. The following command will accomplish this: $ gpgdir --Include .tar.bz2 --sign data
[+] Executing: gpgdir --Include .tar.bz2 --sign data
Using GnuPG key: 1234ABCD
Enter signing password.
Password:

[+] Signing files in directory: /home/user/data
[+] Building file list...
[+] Signing: /home/user/data/file1.tar.bz2
[+] Signing: /home/user/data/file2.tar.bz2
[+] Signing: /home/user/data/dir1/file3.tar.bz2
[+] Signing: /home/user/data/dir1/file4.tar.bz2
[+] Signing: /home/user/data/dir2/file5.tar.bz2
[+] Signing: /home/user/data/dir2/file6.tar.bz2

[+] Total number of files signed: 6

And now to recursively verify all GnuPG signatures in the /home/user/data/ directory: $ ./gpgdir --verify /home/user/data
[+] Verifying signatures in directory: /home/user/data
[+] Building file list...
[+] Verifying: /home/user/data/file1.tar.bz2.asc
[+] Verifying: /home/user/data/file2.tar.bz2.asc
[+] Verifying: /home/user/data/dir1/file3.tar.bz2.asc
[+] Verifying: /home/user/data/dir1/file4.tar.bz2.asc
[+] Verifying: /home/user/data/dir2/file5.tar.bz2.asc
[+] Verifying: /home/user/data/dir2/file6.tar.bz2.asc

[+] Total number of files verified: 6

Here is the complete ChangeLog: for the 1.9.2 release:

• Added new modes '--sign <dir>' and '--verify <dir>' to allow all files in the specified directory to be signed or verified instead of encrypted or decrypted. All GnuPG signatures are created as "<file>.asc", and the original file is not removed in --sign mode. In --verify mode, if any file does not match the expected .asc signature, then a warning like the following will be generated: [+] Verifying: /home/mbr/src/gpgdir/test/data-dir/multi-line-ascii.asc [GNUPG:] BADSIG 9EDEEEEBA742EEEF Some User <someuser@domain.org>
• Bugfix to not die() when files that are encrypted with a different GnuPG key are encountered in a directory that is being decrypted. A warning message (see below) is now generated and the file is skipped: [+] Decrypting: /home/mbr/tmp/gpgdir/a.gpg [GNUPG:] BAD_PASSPHRASE CF16F0FCFFF3FF4F [-] Skipping file encrypted with different GnuPG key: a.gpg
• Updated to use the status output from GnuPG::Interface to detect a bad passphrase and whether a file is encrypted with the expected GnuPG key.
• Moved the GnuPG::Interface, Class::MethodMaker, and Term::ReadKey modules to the deps/ directory, and updated the installer and RPM spec file to account for the path change. This change was suggested by Franck Joncourt for the other cipherdyne.org projects.
• Updated the test suite to generate files in the output/ directory according to test number and append the result of each test within each file. This makes it easy to tell which tests have failed with a simple 'grep fail output/*test'.
• Added the gpgdir-nodeps.spec file to allow an RPM to be built that does not contain any perl modules dependencies.
• Updated gpgdir to import perl modules via 'require' statements instead of 'use' statements so that the path to the modules directory can be changed via the --Lib-dir command line argument. Also updated to use the 'auto' heuristic (first implemented in the fwknop project) to detect perl module directories that should be used in the --Lib-dir directory to import perl modules from.

24 August, 2008

The 1.9.7 release of fwknop is available for download. This release changes several areas of fwknop related to packaging and packet decoding. First, in order to better support the work of Franck Joncourt to package the cipherdyne.org projects for Debian, all perl module dependencies have been moved into the deps/ directory. Second, it is looking like fwknop will eventually be integrated with Fedora thanks to the work of Mirek Trmac who contributed significant patches (including the removal of the NetPacket dependency). Also, as mentioned in the latest releases of fwsnort and psad, every project release is now signed with a new GnuPG key that is dedicated just for this purpose, and this key can be downloaded here.

The complete ChangeLog for fwknop-1.9.7 appears below:

• Mirek Trmac from Red Hat contributed several patches so that fwknop can be bundled within the Fedora Linux distribution. These patches implemented the following changes:
• Updates to fwknopd to remove the NetPacket module as a dependency (this is a particularly important update since it assists with getting fwknop bundled with Debian as well). The patch manually decodes the network and transport layer headers.
• A patch to make the fwknop init script not start fwknopd by default on Red Hat systems. This patch also supports Fedora init script conventions better (i.e. fwknop instead of the fwknopd name for the lock file in /var/lock/subsys).
• Updated the fwknop Makefile to respect the OPTS variable which is used in the RPM spec file.
• Bugfix in fwknop_serv to support the variable expansion code from fwknopd. This was important for the TCPSERV_PID_FILE file which is defined as $FWKNOP_RUN_DIR/fwknop_serv.pid.
• Updated fwknopd to use the Net::Pcap API valid in Net::Pcap-0.14 for the datalink() function (used to detect the datalink layer type).
• Updated fwknop, fwknopd, and knoptm to import perl modules out of the /usr/lib/fwknop/ directory if it exists. This allows the perl module path to be manipulated via the --Lib-dir command line argument and 'require' statements instead of the old 'use module' strategy.
• Added module version output for each non-core perl module used by fwknop and fwknopd in --debug mode. This is mostly useful for the test suite to see which versions of the modules are being used.
• Added the ability to ignore any local GnuPG 'options' file with a new command line argument --gpg-no-options (for the fwknop client) and a new access.conf config variable GPG_NO_OPTIONS (for the fwknopd daemon). This fixes a problem reported by Mike Holzmann where the 'encrypt-to' option in the default options file was causing SPA packets to exceed 1500 bytes when encrypted with a 2048-bit GnuPG key. Also added the MAX_SNIFF_BYTES to the fwknop.conf file and --Max-packet-size to the fwknop command line to alter the default of 1500 bytes if needed (but this shouldn't really be necessary).
• Bugfix for 'Premature end of base64 data' and 'Premature padding of base64 data' warning messages from MIME::Base64 errors. Now fwknopd applies more rigorous checks for base64 encoded characters, and either of these two messages above will result in the packet data being discarded before it is sent through any decryption function. Mike Holzmann reported this issue.
• (Test suite) Added --test-system-fwknop to allow any installed version of fwknop to be installed instead of the scripts bundled within the local source distribution.

22 August, 2008

The 2.1.4 release of psad is available for download. This release moves all dependencies into a new deps/ directory - including all perl modules, Snort rules files, and the whois client (from Marco d'Itri). This makes for a cleaner integration of psad with the Debian Linux distribution, thanks to Franck Joncourt. There are also a couple of minor bugfixes and feature enhancements according to the ChangeLog entries below. Finally, all cipherdyne.org projects are now signed with a new GnuPG key available here.

• Restructured perl module paths to make it easy to introduce a "nodeps" distribution of psad that does not contain any perl modules. This allows better integration with systems that already have all necessary modules installed (including the IPTables::ChainMgr and IPTables::Parse modules). The main driver for this work is to make all cipherdyne.org projects easily integrated with distributions based on Debian, and Franck Joncourt has been instrumental in making this process a reality. All perl modules are now placed within the "deps" directory, and the install.pl script checks to see if this directory exists - a separate psad-nodeps-<ver> tarball will be distributed without this directory. The Debian package for psad can then reference the -nodeps tarball, and a new "psad-nodeps.spec" file has been added to build an RPM from the psad sources that does not install any perl modules.
• Updated to use the normal system whois client if the /usr/bin/whois_psad path does not exist, and moved the whois/ directory into the deps/ directory. This removes /usr/bin/whois_psad as a strict dependency.
• Bugfix to honor the IPT_SYSLOG_FILE variable in --Analyze-msgs mode.
• Switched from the deprecated bleeding-all.rules file to the new emerging-all.rules available from Matt Jonkman at Emerging Threats (http://www.emergingthreats.net).

21 August, 2008

The 1.0.5 release of fwsnort is available for download. For this release all Snort rules and perl modules packaged within the fwsnort sources have been moved into the deps/ directory so that it is clear which components are dependencies, and this change makes it easy to build "nodeps" tar archives and RPMs of fwsnort that do not include any dependencies. This change was suggested by Franck Joncourt in support of his efforts to create Debian packages of fwsnort, and this will also mean that fwsnort will integrate easily into Linux distributions such as Ubuntu). Also, from now on, all cipherdyne.org project will be signed with a new GnuPG key that is dedicated just for this purpose, and this key can be downloaded here.

The complete ChangeLog for fwsnort-1.0.5 appears below:

• Replaced the bleeding-all.rules file with the emerging-all.rules file. This is because Matt Jonkman now releases his rule sets at http://www.emergingthreats.net/
• Restructured perl module paths to make it easy to introduce a "nodeps" distribution of fwsnort that does not contain any perl modules. This allows better integration with systems that already have all necessary modules installed (including the IPTables::ChainMgr and IPTables::Parse modules). The main driver for this work is to make all cipherdyne.org projects easily integrated with distributions based on Debian, and Franck Joncourt has been instrumental in making this process a reality. All perl modules are now placed within the "deps" directory, and the install.pl script checks to see if this directory exists - a separate fwsnort-nodeps-<ver> tarball will be distributed without this directory. The Debian package for fwsnort can then reference the -nodeps tarball, and a new "fwsnort-nodeps.spec" file has been added to build an RPM from the fwsnort sources that does not install any perl modules.
• Updated to import perl modules from /usr/lib/fwsnort, but only if this path actually exists in the filesystem. This is similar to the strategy implemented by psad. A new variable FWSNORT_LIBS_DIR was added to the fwsnort.conf to support this.
• Added support for multiple Snort rule directories as a comma-separated list for the argument to --snort-rdir.
• Moved 'threshold' to the unsupported list since there will be several signatures that use this feature to detect the Dan Kaminsky DNS attack, and fwsnort does not yet support the usage of the iptables --limit match.

22 July, 2008

The 1.9.6 release of fwknop is ready for download. This release was made at The Last HOPE conference over the weekend in NYC, and introduces several new features that make SPA traffic more difficult to detect on a network even when an IDS is watching. Previous to the 1.9.6 version of fwknop, it was possible to write Snort rules to detect SPA traffic by looking for invariant artifacts from base-64 encoding and also from encryption operations (both Rijndael and GnuPG). These artifacts are now stripped out by the fwknop client before being transmitted on the wire, and the result is that SPA packets are now more highly randomized. This implies that SPA packets generated by the 1.9.6 release are not compatible with older fwknop deployments by default, but the client does offer command line arguments to maintain compatibility if necessary.

Here is an excerpt from the ChangeLog:

• SPA packets are base64-encoded by the fwknop client, and this encoding pads data with '=' chars until the total length of the encoded data is a multiple of four. This characteristic can be used within a Snort rule to assist in the detection of SPA communications. The 1.9.6 release of fwknop strips out these padding characters before the client sends an SPA packet, and the fwknopd server adds them back in (to form a multiple of four) before base64 decoding the packet data. This reduces the level of identifying information in SPA packets and therefore makes it more difficult to detect the usage of SPA for service access. For reference, a Snort rule that would detect SPA packets via the trailing '=' chars (previous to this release) would be:
  
  alert udp any any -> any 62201 (msg:"fwknop SPA traffic"; dsize:>150; pcre:"/==$/"; sid:20080001; rev:1;)
  
  
• According to the 'file' command (via it's 'magic') database, files that are encrypted with GnuPG begin with 0x8502, and this is true for SPA packets generated by fwknop (previous to this release). In fwknop-1.9.6, the "hQ" prefix is removed by the fwknop client and added back in by the fwknopd server if it doesn't exist. This measure is another effort to make SPA packets more difficult to detect on the wire, such as with the following Snort rule:
  
  alert udp any any -> any 62201 (msg:"fwknop GnuPG encrypted SPA traffic"; content:"hQ"; depth:2; dsize:>1000; sid:20080003; rev:1;)
  
  
• Updated the fwknop client to randomize the UDP source port for default SPA packet generation. There is also a new command line argument --Source-port <port> to allow the user to manually set the source port on the fwknop client command line. A lot more attention is given now to source ports after the Dan Kaminsky DNS caching exploit, and it turns out that even on Linux that the kernel did not randomize UDP source ports until the 2.6.24 kernel. Of course, any userspace process is free to request a random port itself, but if a userspace application did not build this in then it would be up to the kernel to assign a source port. In the case of Linux, here are two links that show the change to the kernel code as well as the ChangeLog entry for UDP source port randomization:
  
  Kernel commit
  
  ChangeLog-2.6.24

19 June, 2008

Gootrude-0.2 is available for download. This release changes the default search engine from Google to Yahoo because it appears that Yahoo's Terms of Service do not prohibit automated queries (although I'm not a lawyer). Gootrude retains the ability to query Google at your own risk if you so desire. As long as you use Gootrude as designed to collect search engine results only once per day with a limited number of search terms (say, 20 or less), it is unlikely to provoke a negative reaction from search engines. Over time, support for many additional search engines will be added to Gootrude so that search results can be trended from all sorts of different data sources.

Also, this post should really have announced Gootrude as a "search results trender" instead of a "search trender". The goal of the project is made clear in the README, and here is the complete ChangeLog, for the Gootrude-0.2 release:

• Added support for querying Yahoo for search results and made this the default because Google's TOS does not technially allow automated queries. However, Gootrude continues to support querying Google for search results, and responsible use of Gootrude once per day with a limited set of search terms should most likely not be a cause for concern for Google. A new syntax of the searchterms.conf file allows search terms to express which search engine should be queried on an individual basis according to the following syntax (note the middle element): [Linux "highspeed firewalls"] [yahoo:count] [Linux_highspeed_firewalls.dat]
• Changed the default USER_AGENT variable to "Gootrude <version>".
• Added a percentage-based offset for the plot minimum and maximum ranges on the y-axis (with a default of 10% controlled by two new variables MIN_PERCENT_DIFF and MAX_PERCENT_DIFF). This value can also be set by the search terms themselves via an additional config element as follows (for a 20% offset instead for example): [yrange:20%/20%]
• Increased SEARCH_TERM_DELAY to two seconds to be nicer to search engines.