To install qmail-relayreject, you'll need finished build directories for

qmail
serialmail

and the qmail and uscpi-tcp installed (but you're running a qmail server, so
you had those anyway, right?). Note by 'finished' I mean make has been run so
all the .o and .a files are sat there; we'll need some of those.

First, build qmail-relayreject:

- Copy qmail-relayreject.patch into your qmail source directory
- Now run the following commands

cp qmail-smtpd.c qmail-relayreject.c
patch qmail-relayreject.c <qmail-relayreject.patch
./compile qmail-relayreject.c
./load qmail-relayreject rcpthosts.o commands.o timeoutread.o \
    timeoutwrite.o ip.o ipme.o ipalloc.o control.o constmap.o \
    received.o date822fmt.o now.o qmail.o cdb.a fd.a wait.a \
    datetime.a getln.a open.a sig.a case.a env.a stralloc.a \
    alloc.a substdio.a error.a str.a fs.a auto_qmail.o `cat \
    socket.lib`

(this list could easily be trimmed, I suspect - feel free to try)

Next, build relaytest:

- Copy relaytest.c into your serialmail source directory
- Now run the following commands

./compile relaytest.c
./load relaytest auto_qmail.o qmail.o maildir.o timeoutread.o \
    timeoutwrite.o config.o quote.o prioq.o now.o env.a \
    libtai.a getln.a strerr.a substdio.a stralloc.a alloc.a \
    error.a str.a open.a wait.a sig.a fd.a fs.a getopt.a mess822.a

(again, could and probably should be trimmed)

No, copy relaytest and qmail-relayreject into your /var/qmail/bin and locate
your service directory for incoming smtpd (probably /service/smtpd). In it,
open the run script and find the point at which qmail-smtpd is invoked from
tcpserver. It'll probably look similar to

tcpserver -u 7770 -g 2108 0 smtp /var/qmail/bin/qmail-smtpd

Here, we want to replace '/var/qmail/bin/qmail-smtpd' with a shell script which
calls tcpclient and then relaytest, then qmail-smtpd or qmail-relayreject as
appropriate. For example -

tcpserver -u 7770 -g 2108 0 smtp sh -c '
if tcpclient -T2 -RH -l0 $TCPREMOTEIP 25 /var/qmail/bin/relaytest
then
  /var/qmail/bin/qmail-relayreject
else
  /var/qmail/bin/qmail-smtpd
fi'

This will cause tcpclient to wait a maximum of two seconds for an SMTP
connection. and not to retry; if it fails to obtain a connection in that time
it will return a failure and the host is assumed not to have port 25 open and
is allowed through. If it connects, it will call relaytest which returns a zero
(true) exit status if it finds an open relay and a non-zero (false) exit status
if the remote host correctly denies its request.

If the assembly returns true (open relay), qmail-relayreject is invoked, if
false (no SMTP server or correctly configured), qmail-smtpd is invoked as
normal.

Now restart your smtpd service (probably 'svc -h /service/smtpd') and
qmail-relayreject should be functioning.