Day Two

Highwinds

Application Programming Interfaces

Highwinds APIs

• Authentication API

• Post Filter API

• Spam Filter API

• Twister Web API

We support everything

• By creating APIs for Authentication, Spam Filtering, and Post filtering, we can support Oracle, SQL, Perl, C, Python, and so on.

• Most APIs speak through stdio

Authentication API

• To activate it, turn on <Feed>AuthenticationProgram /news/typhoon/progs/authentication.exe</Feed>

• Typhoon forks off the program and sends data to it via stdin

Authentication API

• Client connects to Tornado FE

• TornadoFE sends authinfo to auth program

• Auth Program responds with either a request for username/password, allow, or deny

• If Auth Program responded with – Allow) client is allowed to read– Deny) client is disconnected– request for u/p) Tornado requests u/p from client

Auth Program responds with either an allow or deny

• Upon client disconnection, Tornado sends statistics to the Auth Program

Authentication API

• What Twister sends on connect Action: connect\r\n

Version: Twister v2.1.0.360 \r\n Protocol: NNTP \r\n IncomingFeedName: feed1 \r\n Subscription: * \r\n FilterSubscription: !* \r\n AllowFeeding: true \r\n AllowReading: true \r\n AllowPosting: true \r\n AllowNewNews: true \r\n SendXrefInOverviews: true \r\n WelcomeMessage: Connected Bad Monkey \r\n XComplaintsTo: abuse@company.com \r\n Organization: Company News Server \r\n ForceOrganization: false \r\n TimeOut: 900 \r\n MaxConnectTime: 0 \r\n

Authentication API

HostConnectionLimit: 5 \r\n

MaxIncomingNumberOfStreams: 5 \r\n

FeedMaxBytesPerSecond: 0 \r\n

MaxBytesPerSecond: 0 \r\n

BandwidthTrackingWindow: 0 \r\n

BandwidthLimitTrigger: 0 \r\n

TriggeredMaxBytesPerSecond: 0 \r\n

TriggeredBandwidthTrackingWindow: 0 \r\n

HTTPDirectory: ../htdocs/frontpage.html \r\n

TemplateDirectory: ../xml \r\n

ErrorTemplate: errmsg.tpt \r\n

AllowProtocols: NNTP \r\n

HttpCategorization: ../templates/lib/category/cat.cat \r\n

SpecialSubscription: \r\n

AllowThreading: true \r\n

Interface: 10.0.0.1 \r\n

HostName: 10.0.0.10 \r\n

IpAddress: 10.0.0.10 \r\n

SessionID: Rhc-115-3T2-3@news.company.com \r\n

Cookie: empty \r\n

ConnectionTag: - \r\n

AuthID: 4939 \r\n

Authentication API

• Responses to Connect:– 200\r\n - Allows the host to connect

without authentication.

– 480\r\n - Allows the host to connect but REQUIRES that the host authenticate with a username/password.

– 502\r\n - Does NOT allow the host to connect.

Authentication API

• What twister sends on authenticate:Action: authenticate \r\n Version: Twister v2.1.0.360 \r\n Protocol: NNTP \r\n IncomingFeedName: feed1 \r\n Subscription: * \r\n FilterSubscription: !* \r\n AllowFeeding: true \r\n AllowReading: true \r\n AllowPosting: true \r\n AllowNewNews: true \r\n SendXrefInOverviews: true \r\n WelcomeMessage: Connected Bad Monkey \r\n XComplaintsTo: abuse@company.com \r\n Organization: Company News Server \r\n ForceOrganization: false \r\n TimeOut: 900 \r\n MaxConnectTime: 0 \r\n

Authentication API

HostConnectionLimit: 5 \r\nMaxIncomingNumberOfStreams: 5 \r\nFeedMaxBytesPerSecond: 0 \r\nMaxBytesPerSecond: 0 \r\nBandwidthTrackingWindow: 0 \r\nBandwidthLimitTrigger: 0 \r\nTriggeredMaxBytesPerSecond: 0 \r\nTriggeredBandwidthTrackingWindow: 0 \r\nHTTPDirectory: ../htdocs/frontpage.html \r\nTemplateDirectory: ../xml \r\nErrorTemplate: errmsg.tpt \r\nAllowProtocols: NNTP \r\nHttpCategorization: ../templates/lib/category/cat.cat \r\nSpecialSubscription: \r\nAllowThreading: true \r\nInterface: 10.0.0.1 \r\nHostName: 10.0.0.10 \r\nIpAddress: 10.0.0.10 \r\nSessionID: Rhc-115-3T2-3@news.company.com \r\nCookie: empty \r\nConnectionTag: - \r\nUsername: newsuser \r\nPassword: mypassword \r\nAuthID: 4939 \r\n

Authentication API

• Responses to Action: Authenticate– 281\r\n - Indicates authentication was

successful and can be followed by additional fields.

– 502\r\n - Indicates authentication failed and any additional fields that follow will be ignored.

– 503\r\n - Indicates authentication failed and the connection will be IMMEDIATELY forced closed.

Authentication API

• What Twister sends on disconnect:Action: disconnect \r\n Version: Twister v2.1.0.360 \r\n Protocol: NNTP \r\n IncomingFeedName: feed1 \r\n Subscription: * \r\n FilterSubscription: !* \r\n AllowFeeding: true \r\n AllowReading: true \r\n AllowPosting: true \r\n AllowNewNews: true \r\n SendXrefInOverviews: true \r\n WelcomeMessage: Connected Bad Monkey \r\n XComplaintsTo: abuse@company.com \r\n Organization: Company News Server \r\n ForceOrganization: false \r\n TimeOut: 900 \r\n MaxConnectTime: 0 \r\n

Authentication API

HostConnectionLimit: 5 \r\nMaxIncomingNumberOfStreams: 5 \r\nFeedMaxBytesPerSecond: 0 \r\nMaxBytesPerSecond: 0 \r\nBandwidthTrackingWindow: 0 \r\nBandwidthLimitTrigger: 0 \r\nTriggeredMaxBytesPerSecond: 0 \r\nTriggeredBandwidthTrackingWindow: 0 \r\nHTTPDirectory: ../htdocs/frontpage.html \r\nTemplateDirectory: ../xml \r\nErrorTemplate: errmsg.tpt \r\nAllowProtocols: NNTP \r\nHttpCategorization: ../templates/lib/category/cat.cat \r\nSpecialSubscription: \r\nAllowThreading: true \r\nInterface: 10.0.0.1 \r\nHostName: 10.0.0.10 \r\nIpAddress: 10.0.0.10 \r\nSessionID: Rhc-115-3T2-3@news.company.com \r\nCookie: empty \r\nConnectionTag: - \r\nUsername: newsuser \r\nPassword: mypassword \r\nStatistics: 108264 9635 11 0 1082649635 0 \r\nAuthID: 4939 \r\n

Authentication API

• Authentication Program can override many values of assigned to the connection

• The response should be in the form of:<numeric code>\r\n[<Directive to Override>: <Value>][<Directive to Override>: <Value>][<Directive to Override>: <Value>].\r\n

Authentication API

• Examples:502\r\n WelcomeMessage: Permission Denied.\r\n .\r\n

281\r\n .\r\n

281\r\n AllowReading: True\r\n AllowPosting: True\r\n AllowNewNews: True\r\n .\r\n

Authentication API

• Re-authentication – in future versions of tornado

– Effective bandwidth limiting

– Drop abusive connections

Authentication API

• Other uses for the Auth API– Tracking of individual users

– Virtual Servers with a single feed object

– Adjust access based on external influences:

• RADIUS/LDAP

• Time of Day

• Server Load

• Weather

• Fed Rates

• ???

Sample

#!/usr/local/bin/perl

# File: sample_auth.pl

# Created: 8-Mar-1998

# $Header: /opt/nr/rcs/lib/authinfo/sample_auth.pl,v 1.17 1999/11/03 18:36:06 driley Exp $

# COPYRIGHT 1998-1999

# bCandid Corporation.

# All Rights Reserved.

#

# software@bcandid.com

# 303-545-5550

# Simple illustration of an authentication program

#

# Per the documentation, authentication programs need to respond to

# three types of commands: "connect", "authenticate", and "disconnect".

# This program simply "syslog's" information and allows everyone to

# connect. However, it is a useful program to study if you are writing

# your own authentication program.

Sample

use Sys::Syslog;

$| = 1; # Flush STDOUT

$/ = "\r\n"; # Authentication data lines are terminated by "\r\n"

openlog($0,'ndelay,pid','news'); # Open the syslog

# Valid Responses to "connect"

$allow_connect = "200\r\n";

$require_authentication = "480\r\n";

$deny_connect = "502\r\n";

# Valid Initial Responses to "authenticate"

$auth_succeeded = "281\r\n";

$auth_failed = "502\r\n";

Sample

# Loop "forever" on inputwhile(defined($line = <>)) { # Remove the trailing \r\n $line =~ s/\r\n$//; # # Deal with each type of command # if ($line eq "Action: connect") {

# Read arguments to the connect, saving a few fieldswhile(defined($line = <>)) { # Remove the trailing \r\n $line =~ s/\r\n$//;

last if ($line eq "."); # Grab arguments if ($line =~ /^Hostname: /) {

($hostname = $line) =~ s/^Hostname: //; } elsif ($line =~ /^IPAddress: /) {

($ipaddress = $line) =~ s/^IPAddress: //; } elsif ($line =~ /^Interface: /) {

($interface = $line) =~ s/^Interface: //; } elsif ($line =~ /^IncomingFeedName: /) {

($feed = $line) =~ s/^IncomingFeedName: //; } elsif ($line =~ /^SessionID: /) {

($sessionid = $line) =~ s/^SessionID: //;

Sample

} elsif ($line =~ /^AuthID: /) {

($authid = $line) =~ s/^AuthID: //;

}

}

# Syslog the Connect Information

syslog('info', "Connect received for $feed on $interface from $hostname/$ipaddress as $sessionid");

# Let everyone connect, but, make them authenticate and override

# a few directives

print $require_authentication;

if ($authid ne "") {

print "AuthID: $authid\r\n";

}

print "AllowReading: True\r\n";

print "AllowPosting: True\r\n";

print "AllowFeeding: True\r\n";

print ".\r\n";

$authid = "";

Sample

} elsif ($line eq "Action: disconnect") {# Read arguments to the disconnect, saving a few fieldswhile(defined($line = <>)) { # Remove the trailing \r\n $line =~ s/\r\n$//;

last if ($line eq "."); # Grab arguments if ($line =~ /^Hostname: /) {

($hostname = $line) =~ s/^Hostname: //; } elsif ($line =~ /^IPAddress: /) {

($ipaddress = $line) =~ s/^IPAddress: //; } elsif ($line =~ /^Interface: /) {

($interface = $line) =~ s/^Interface: //; } elsif ($line =~ /^IncomingFeedName: /) {

($feed = $line) =~ s/^IncomingFeedName: //; } elsif ($line =~ /^SessionID: /) {

($sessionid = $line) =~ s/^SessionID: //; } elsif ($line =~ /^Username: /) {

($user = $line) =~ s/^Username: //; } elsif ($line =~ /^Password: /) {

($pass = $line) =~ s/^Password: //; } elsif ($line =~ /^AuthID: /) {

($authid = $line) =~ s/^AuthID: //; }}

Sample

# Syslog the Disonnect Information

syslog('info', "Disconnect received for $user/$pass using $feed on $interface from $hostname/$ipaddress as $sessionid");

# No response to disconnect

$authid = "";

} elsif ($line eq "Action: authenticate") {

# Read arguments to the authenticate, saving a few fields

while(defined($line = <>)) {

# Remove the trailing \r\n

$line =~ s/\r\n$//;

last if ($line eq ".");

# Grab arguments

if ($line =~ /^Hostname: /) {

($hostname = $line) =~ s/^Hostname: //;

} elsif ($line =~ /^IPAddress: /) {

($ipaddress = $line) =~ s/^IPAddress: //;

} elsif ($line =~ /^Interface: /) {

($interface = $line) =~ s/^Interface: //;

} elsif ($line =~ /^IncomingFeedName: /) {

($feed = $line) =~ s/^IncomingFeedName: //;

} elsif ($line =~ /^SessionID: /) {

Sample

($sessionid = $line) =~ s/^SessionID: //; } elsif ($line =~ /^Username: /) {

($user = $line) =~ s/^Username: //; } elsif ($line =~ /^Password: /) {

($pass = $line) =~ s/^Password: //; } elsif ($line =~ /^AuthID: /) {

($authid = $line) =~ s/^AuthID: //; }}# Syslog the Authentication Informationsyslog('info', "Authenticate received for $user:$pass with $feed on $interface from $hostname/$ipaddress as $sessionid");# Let EVERYONE in, and let them do ANYTHING to non-"alt" groups# Be sure to re-override everything. :-)print $auth_succeeded;if ($authid ne "") { print "AuthID: $authid\r\n";}print "AllowReading: True\r\n";print "AllowPosting: True\r\n";print "AllowFeeding: True\r\n";print "AllowNewNews: True\r\n";print "SendXrefInOverviews: True\r\n";print "WelcomeMessage: Typhoon Authentication\r\n";print "Organization: Typhoon Authentication Testing Club\r\n";

Sample

print "XComplaintsTo: newsmaster\@company.com\r\n";

print "ForceOrganization: True\r\n";

print "TimeOut: 3600\r\n";

print "Cookie: I love cookies!\r\n";

print "ConnectionTag: consider-yourself-it\r\n";

print "Subscription: *, !alt.*, !comp.*\r\n";

print "FilterSubscription: alt.*\r\n";

print "SpecialSubscription: comp.*\r\n";

print "MaxBytesPerSecond 0\r\n";

print ".\r\n";

$authid = "";

} else {

syslog('info', "Unexpected authentication line ($line). Discarding.");

}

}

closelog();

Authentication API

• Questions:

Post Filter

• Post Filter is turned on via commandline flag –postprog /path/to/program

• Tornado forks off the program and sends data to it via stdin

Post Filter

• Client Posts to Twister

• Data about connection and entire article are sent to the post filter

• Postfilter accepts, denies, or modifies message

Post Filter

• When a message is posted, the post filter is sent the following data:IncomingFeedName: value\r\n Subscription: value\r\n FilterSubscription: value\r\n AllowReading: value\r\n AllowFeeding: value\r\n AllowPosting: value\r\n AllowNewNews: value\r\n SendXrefInOverviews: value\r\n WelcomeMessage: value\r\n Organization: value\r\n TimeOut: value\r\n

Post Filter

HostConnectionLimit: value\r\n MaxIncomingNumberOfStreams: value\r\n Interface: value\r\n Cookie: value\r\n ConnectionTag: value\r\n IPAddress: value\r\n SessionID: session\r\n Hostname: value\r\n Username: value\r\n \r\n

From: value\r\n Subject: value\r\n ... additional header fields (lines terminated with "\r\n") ...\r\n ... body of the article (lines terminated with "\r\n")... .\r\n

Post Filter

• Responses Post Filter– 235 – Tells Twister to accept article.

– 236 <delay-time> – Tells Twister to accept article and delay acknowledgement <delay-time> seconds.

– 435 – Tells Twister to reject article.

– 435 <reason> – Tells Twister to reject article and give <reason> for a reason

– 436 <delay-time> – Tells Twister to reject article and delay acknowledgement <delay-time> seconds.

Post Filter

• Post Filter has the ability to modify message by placing the new version of the message (header and body) immediately following the response code.

Post Filter

• Sample Post Filter Responses:

435\r\n.\r\n

435 rejection reason\r\n .\r\n

235\r\n From: value\r\n Subject: value\r\n ... additional header fields (lines terminated with "\r\n") ... \r\n ... body of the article (lines terminated with "\r\n")... .\r\n

Post Filter

Sample Post Filter Program#!/usr/local/bin/perl -w# File: sample_postfilter_backoff.pl# Created: 24-Jul-1999

# COPYRIGHT 1999# bCandid Corporation.# All Rights Reserved.## software@bcandid.com# 303-545-5550

# This sample filter for posts implements a "sliding window" scheme

# preventing an authenticated user from posting more than $postlimit

# articles during any $interval (in seconds) period. Posts from

# unauthenticated users are rejected (since this would totally

# circumvent the back-off scheme).

Post Filter

# --- Begin user-configurable constants ---

# Allow $postlimit POSTs every $interval seconds.

my $interval = 20;

my $postlimit = 5;

# --- End user-configurable constants ---

# An associative array mapping a user's name to a list of the times at which

# the user's last $postlimit POSTs occurred (sorted ascending).

my %posts = ();

# $delay = access_delay($client, $time, $interval, $postlimit)

#

# Returns $delay, the minimum amount of time to delay so that the user

# ($client) will not have been able to post more than $postlimit messages

# in a "sliding window" of $interval seconds. $time is the current time.

# $time most be nondecreasing across sequential calls.

Post Filter

sub access_delay

{

my ($client, $time, $interval, $postlimit) = @_;

# Get the appropriate access-time list.

my $times = $posts{$client};

if (!defined($times)) {

# Never seen $client before - add to the assoc. array.

$times = $posts{$client} = [];

}

# Forget all posts which took place at least $interval ago.

shift @$times while (@$times > 0 && $time - $$times[0] >= $interval);

if (@$times == $postlimit) {

# No more posts permitted for a while! Remove the first (oldest)

# time, but back off until $interval past that time. Record the

# post as actually occurring after the back-off period.

Post Filter

push(@$times, $$times[0] + $interval);my $delay = $$times[0] + $interval - $time;shift @$times;return $delay;

} else {# Record the post.push(@$times, $time);return 0;

}}

# Flush STDOUT$| = 1;$previous_line = "\r\n";

# Enter a simple state machinewhile (defined($line = <>)) { # If we just completed the article send a response if ($previous_line =~ /\r\n$/ && $line eq ".\r\n") {

if ($username) { # The user authenticated; apply the back-off scheme.

Post Filter

my $delay = access_delay($username, time(), $interval, $postlimit);

if ($delay == 0) {

print "235\r\n";

} else {

print "236 $delay\r\n";

}

} else {

# Do not accept unauthenticated POSTs!

print "435\r\n";

}

# Terminate the response

print ".\r\n";

$username = $in_header = $in_body = undef;

}

Post Filter

# We are fed: # FeedInformationLines\r\n # \r\n # HeaderLines\r\n # \r\n # BodyLines\r\n # .\r\n # # So, set up simple state to save the right lines #

# If we are in the FeedInformation area, grab Authentication

# information for later use if (!$in_header && !$in_body) {

if ($line =~ m/^Username: (.*)\r\n$/) { $username = $1;}

}

Post Filter

if ($line eq "\r\n") {

if ($in_header) {

$in_body = 1;

} else {

$in_header = 1;

}

}

# Save last line for use in end of transaction detection

$previous_line = $line;

}

Post Filter

Questions

Spam Filter

• The Filter is activated in the –program flag for cyclone, typhoon, twister, and tornado_be.

• When active, cyclone forks the filter program and sends data through it’s stdin and reads data from its stdout.

Spam Filter

• The spam filter expects the header of the article followed by a \r\n.\r\n

• The spam filter responds with either a 335\r\n indicating acceptance, or 435\r\n indicating rejection.

• -body flag tells the cyclone to provide the entire message to the spam filter

• - multifilter <n> tells cyclone to fork of n instances of the spam filter

Spam Filter

Sample Spam Filter Program#!/usr/local/bin/perl## File: sample_filter.pl# Created: 13-Feb-1998

# $Header: /opt/nr/rcs/cmd/typhoond/sample_filter.pl,v 1.5 1999/09/29 09:17:41 driley Exp $

# COPYRIGHT 1998,1999# bCandid Corporation.# All Rights Reserved.

## Simple illustration of a "MAKE MONEY FAST" Typhoon/Breeze filter#

$| = 1; # Flush STDOUT$good_article = "335\r\n";$bad_article = "435\r\n";

Spam Filter

# Initialize

$result = $good_article;

$previous_line = "\r\n";

# Loop on input

while(defined($line = <>)) {

# If the Subject contains "MAKE MONEY FAST", reject the article.

if ($line =~ /^Subject:/) {

if ($line =~ /MAKE MONEY FAST/) {

$result = $bad_article;

} else {

$result = $good_article;

}

}

if ($previous_line =~ /\r\n$/ && $line eq ".\r\n") { print $result; }

$previous_line = $line;

}

Spam Filter

Questions

Fast Filter

• Fast filter is an option for the spam filter

• Fast filter uses shared memory to rather than stdio for making articles visible to filter

• Fast filter is activated by turning on –fastfilter <n> on the command line where n is the shared memory key

• Fast filter is compatible with –multifilter

Fast Filter

• Stdio is still used for to negotiate protocol and receive responses from filter program

• Response codes are the same 335 and 435, but with fastfilter followed with <messageid>

• Perl module is included with our software to support Fast Filter

Twister/Tornado Web API

• Twister and Tornado Web support web interface.

• -httpport and –httplistenq command line arguments for the http port

• Feed directives AllowProtocol, HTTPHostName, TemplateDirectory, HTTPDirectory, and ErrorTemplate are specific for web interface

Twister/Tornado Web API

• Bandwidth Throttling is compatible with the web interface.

• Templates are writing in xml files and compiled to tpt files

Twister/Tornado Web API

Excerpt of cf file<!--====================================myOverviewLine -- the text which should be output

for each overview line. You will probably want to make use of the "ovCurrLeftMarkers"

variable to set up the thread indentation & reply/top-level-thread markers, as well as the "selPtr" variable, which gives you

either your "selectedText" or "unselectedText" as appropriate.

Example:<tr> <td width="24"><Insert var="selPtr"/></td> <td><Insert

var="ovCurrLeftMarkers"/>&nbsp;<Insert var="plusMinus"/><Insert var="subjectLink"/></td>

<td>&nbsp;</td> <td><Insert var="author"/></td> <td>&nbsp;</td> <td><Insert var="date"/></td></tr>End of Example -->

Twister/Tornado Web API

Twister/Tornado Web API

Questions

• Questions