relay-enforcer v3.9.4 [11 Sep 2024] by Dominic

Description

relay-enforcer enables a postfix-based mail server, which is relaying incoming emails on to Gmail ('gsmtp'), to act on reports which it has received back from Gmail about blocked emails. Depending on the reason given by Gmail (in the text of its report) relay-enforcer will either block the offending incoming client's domain_name/ip_address, or forward the email, encapsulated as an attachment, to its intended recipient.

The intended use case is where Gmail is acting - without Google Workspace - as the backend for a domain's emails (including sending of emails from Gmail so that they appear to come from that domain). Setting up such a scenario, requiring your postfix mail server to be specially configured for relaying to Gmail, is not covered here. But once it is set up then, as your server relays emails into Gmail, some of these will be blocked by Gmail as being in some way 'bad'; worse, if there are too many of these then Gmail may block your mail server entirely, so that emails you relay into Gmail can no longer reach recipients' mailboxes. relay-enforcer provides workarounds/mitigations for these blockages so that emails blocked mistakenly by Gmail can still be delivered, whilst - for emails blocked correctly by Gmail - relay-enforcer will block the sender domain_name or ip_address on your own server, preventing a flood of such emails from being relayed to Gmail and damaging your server's reputation.

Note that relay-enforcer is not a substitute for strong anti-spam and anti-virus defences on your mail server; indeed these are pre-requisites, so that as far as possible neither 'real' spam nor virus-laden emails are ever forwarded by your server to Gmail. relay-enforcer is intended to catch the inevitable 'edge case' emails that, although 'bad', have been missed by your own defences, as well as 'edge case' emails that are 'good' but are mistakenly identified as 'bad' by Gmail.

Usage

relay-enforcer is intended to be run at startup e.g. by adding it as a root cron job thus:

@reboot nohup /usr/local/bin/relay-enforcer -v </dev/null >>/var/log/relay-enforcer.log 2>&1 &

If required, if can be (re)started from command line (with superuser permissions) thus:

/usr/local/bin/relay-enforcer -r -v </dev/null >>/var/log/relay-enforcer.log 2>&1 &

By default relay-enforcer is silent until a matching message is found; use verbose, test or debug options to see more. Monitor relay-enforcer's output to see what it is doing. If run with '-v' it outputs a dot for each line of the mail log that it has inspected, and starts a new line with a datetime hourly; this allows you to be confident that it is running and processing the mail log. Actions generate output unless it is run with '-q'.

relay-enforcer uses tail -f to inspect lines as they are added to the mail log. If it takes action this is reported on standard output (unless quiet option is chosen) and, if -m option is used, by email as well.

If a block (rather than a forward) action is required, relay-enforcer normally (but see 'Sender_d actions' below) bans the IP of the offending sender. It does this by adding an entry in the system log which ends with the ip of the incoming client whose message gave rise to the bad report; this line is to be read by the matching fail2ban jail which then effects the ip ban from incoming mail ports for some period (see 'Ban actions' below).

relay-enforcer runs under bash, and has been tested on Ubuntu 22.04, 20.04, 18.04, 16.04.

Amavis compatibility

relay-enforcer is compatible with (but does not require) re-injection by amavis acting as a content filter (where the queue-id of the email rejected by the onward server is not the same as the queue-id of the mail from the incoming client). relay-enforcer assesses whether to run with this compatibility automatically, however command line options can be used to force the desired behaviour - for instance if amavis is run as a milter. Warning: usage of relay-enforcer without amavis re-injection is untested.

Note that relay-enforcer is not currently compatible with re-injection (content filtering or post-queue filtering) by utilities other than amavis.

Welcomelist

A file named relay-enforcer.welcomelist in the same location as relay-enforcer is used to ensure that certain ips or hostnames cannot generate a report by relay-enforcer and thus a potential ban; instead of a blocking or banning action, emails from a welcomelisted IP or hostname will be encapsulated and sent on (see DMARC forward actions below). If missing, relay-enforcer.welcomelist is created at startup containing local ips and all possible local-network ips. relay-enforcer.welcomelist uses plain, range or CIDR formats as understood by grepcidr; each pattern should start on a newline. You can also add hostname patterns using egrep (extended regex) syntax, so dots etc must be escaped. Comments (starting with #) and any space immediately preceding them are ignored, as are blank lines and any whitespace at the start of a non-blank line (so that indenting of valid lines is possible). relay-enforcer.welcomelist can be updated while relay-enforcer is running.

Sender_d actions

Where the rejection message from Gmail specifies that the problem was with the sending domain, then instead of adding a log entry which might trigger a source IP ban by fail2ban, relay-enforcer will add this domain to $SENDER_D_BANFILE (normally: '/etc/postfix/sender_access') and hash it; provided this file is specified with a postfix restriction list for check_sender_access then this ensures that no more emails from this domain will (ever) be accepted.

Ban actions

When an IP is identified for banning, the actual banning action is carried out by fail2ban jail 'relay-enforcer' when it finds a 'bannable' log entry added by relay-enforcer. This jail needs to be created, for instance:
(a) create /etc/fail2ban/filter.d/relay-enforcer.local:

[INCLUDES]
before = common.conf
[Definition]
_daemon = relay-enforcer
failregex = ^%(__prefix_line)sbannable.* <HOST>$
ignoreregex =

(b) add entries in, say, /etc/fail2ban/jail.local like this:

[relay-enforcer]
enabled = true
maxretry = 1
bantime = 14700
findtime = 14700
port = smtp,465,submission
logpath = %(syslog_ftp)s

DMARC forward actions

If the local mailserver is enforcing DMARC (e.g. with opendmarc), a rejection from an onward server should only occur if the offending email's 'From:' sender domain publishes a DMARC p=reject policy but the email lacks valid DKIM and relies for successful delivery on SPF (which is inevitably broken by forwarding). In such a case (which is rare) relay-enforcer will extract the failed email from a local backup (e.g. /var/mail/backup - postfix settings 'always_bcc = backup@localhost' and 'mail_spool_directory = /var/mail' can be used to save all mails here), and send it on as an attachment to the original recipient, with a covering message. To enable this DMARC forwarding action, first ensure that you have an effective DMARC enforcement policy on your local mailserver, and have all sent emails saved (via always_bcc as shown above) in a single mbox/mailbox-type file, then use relay-enforcer's -f option to define the local backup file when starting relay-enforcer.

Authentication checks failures

Gmail rejects some emails on the grounds of 'authentication checks|not a valid RFC-5321|fails to pass SPF', these are usually round-robin emails from a Microsoft server, or from a sender with a 'hard fail' SPF policy. Experience has shown that they are always or nearly always good emails, so relay-enforcer will resend them encapsulated (same as for DMARC forward actions, and with the same requirement that recent emails are saved in a local mbox/mailbox-type file).

Log rotation

To prevent relay-enforcer from hanging upon log rotation of the mail log file, add 'copytruncate' to the relevant section of the relevant logrotate file (e.g. /etc/logrotate.d/rsyslog).

DSN code rewriting

relay-enforcer does not rewrite the code responses that are received from Gmail and then relayed by the local mailserver to the original sender; this must be done separately as part of the postfix configuration, but only for messages to Gmail. 550 responses upon failure because of DMARC or authentication checks need to be re-written to 250 so that the original sender considers the message was delivered ok (as, after forwarding by relay-enforcer, it should be), and temporary failure codes should be rewritten as permanent (i.e. 4xx to 5xx).

Example file extracts for postfix configuration are shown here -
/etc/postfix/master.cf -

...
smtp_dsn_rw unix - - n - - smtp -o smtp_reply_filter=pcre:/etc/postfix/smtp_dsn_rw_filter.pcre
...

/etc/postfix/main.cf -

...
transport_maps = hash:/etc/postfix/transport
...

/etc/postfix/transport -

...
gmail.com smtp_dsn_rw
...

/etc/postfix/smtp_dsn_rw_filter.pcre -

# modify response codes to original sender
# no change if there is a temporary problems with Gmail servers
/^(421 4\.[0-9]\.[0-9] Temporary )/ ${1}
# DMARC rejection by Gmail - change to 250 OK; we will forward it encapsulated
/^550 5\.7\.26( DMARC.+gsmtp)/ 250 2.0.0${1}-fmod_dmarc
# Authentication checks rejection by Gmail - change to 250 OK; we forward it encapsulated
/^421 4\.7\.0( information.+gsmtp)/ 250 2.0.0${1}-fmod_lackauth
/^550 5\.7\.26( information.+gsmtp)/ 250 2.0.0${1}-fmod_lackauth
# Other temporary rejection by Gmail - make rejection permanent
/^421 4\.7\.28( .+gmstp)/ 554 5.7.28${1}-fmod_unsol
/^421 4\.([0-9]*\.[0-9]* review our Bulk.+gsmtp)/ 554 5.${1}-fmod_bulk
/^421 4\.([0-9]*\.[0-9]* .+gsmtp)/ 554 5.${1}-fmod_generic

Options

-a - force compatibility with amavis as content-filter (see -n)
-d - debug mode - lots of extra text output, implies verbose
-f /var/local/backup - specify a local backup mbox/mailbox-style file which will contain copies of recently sent mails (e.g. by postfix's always_bcc), so that it can be used to retrieve blocked mails for encapsulating and forwarding
-h - show this help
-l - show changelog
-m mail@address - send report on any action to mail@address (default: root)
-n - force non-compatibility with amavis as content-filter (see -a)
-q - be quiet even if reportable actions are found
-r - kill another instance of relay-enforcer (if any), then continue running
-s - stop another instance of relay-enforcer (if any)
-t - test mode (don't submit reports to system log, don't update count of lines in mail log)
-v - be verbose
-w columns - set terminal width e.g. 80 (for help/changelog word-wrapping only, normally determined automagically)

Options must be specified individually not combined i.e. -v -r not -vr

Philosophy

relay-enforcer does not follow RFC5321 which says of the reply code+text sent by an onward smtp receiver (such as Gmail), that 'an SMTP client MUST determine its actions only by the reply code, not by the text' (https://tools.ietf.org/html/rfc5321#section-4.2). If this troubles you, don't use relay-enforcer.

Dependencies

bash
fail2ban with relay-enforcer jail - (relay-enforcer cannot ban without this)
grepcidr
a mail user agent (MUA) which supports options -t [use message headers] -a [add attachment] -r [envelope_from = From: header] - currently only s-nail (tested up to v14.9.23) and GNU Mailutils mail(x) v3.14
opendkim
opendmarc
postfix
python3

License

Copyright © 2024 Dominic Raferd. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at https://www.apache.org/licenses/LICENSE-2.0. Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Changelog

Note - small changes and bugfixes may occur between releases listed here, these may be indicated by a sub-minor version change (3.4.0->3.4.1).
3.9 [26 Jan 2024] - improved postsrsd compatibility (v2.0+)
3.8 [17 Apr 2022] - bugfixes, make bash-dependent (compatibility break)
3.7 [21 Mar 2021] - many small changes to achieve close compliance with shellcheck, use tail -f instead of inotifywait
3.6 [06 Jan 2021] - opendkim and opendmarc are now dependencies (compatibility break)
3.5 [10 Jul 2020] - all references to whitelist (except in this changelog) changed to welcomelist (compatibility break)
3.4 [26 Jun 2020] - only one fail2ban ban type (relay-enforcer, not relay-enforcer-long and relay-enforcer-short; compatibility break)
3.3 [09 Jun 2020] - alter whitelist ip format (compatibility break)
3.2 [17 Apr 2020] - add preceding comment when adding entry to /etc/postfix/sender_access
3.1 [01 Apr 2020] - work with logrotate copytruncate
3.0 [13 Dec 2019] - bugfix paths for programs run by relay-enforcer, tidy up some old code
2.9 [07 Dec 2019] - bugfix for postfix QueueIDs of >10 characters
2.8 [02 Dec 2019] - don't copy encapsulated emails to administrator
2.7 [22 Jul 2019] - add an extra check to each candidate line from log to ensure that we didn't already process it (otherwise duplication occasionally happens)
2.6 [27 Jun 2019] - forward (by encapsulating) on any 'authentication checks|not a valid RFC-5321|fails to pass SPF' Gmail response (previous behaviour was to do this only when the From header was matched in $LACK_AUTH_WHITELIST)
2.5 [02 Jan 2019] - tweak output in non-verbose non-quiet mode
2.4 [18 Oct 2018] - first published version
2.3 [05 Apr 2018] - bugfix ' sending .*domain' blocking
2.2 [29 Mar 2018] - add a 5s [now 10s] delay before triggering a fail2ban ban, to allow time for any outgoing email to original sender (also helps ensure that any subsequent recorded blocks by fail2ban represent *new* connection attempts by the now-banned host at that ip), also bugfix code for unexpected gsmtp responses
2.1 [12 Feb 2018] - introduce ' sending .*domain' blocking via postfix if cause of Gmail unhappiness is stated to be sender domain - permanent block by sender domain name
2.0 [04 Nov 2017] - introduce 'authentication checks|not a valid RFC-5321|fails to pass SPF' forwarding if email 'From:' is in $LACK_AUTH_WHITELIST
1.9 [16 Jul 2017] - help info updated/bugfixed, log entry upon DMARC forwarding no longer says 'bannable'
1.8 [13 Jul 2017] - email any unrecognised rejection messages to mail address (if defined by option -m)
1.7 [08 Jul 2017] - output any unrecognised rejection messages
1.6 [01 Jul 2017] - wait up to 20m (not 60s) for mail log to appear when starting
1.5 [01 Jun 2017] - fix when restarting after mail log rotation
1.4 [12 May 2017] - bugfix (grep -a option added throughout)
1.3 [01 May 2017] - added instructions to avoid hang upon rotation of monitored log file
1.2 [18 Mar 2017] - set Reply-To for forwarded emails
1.1 [11 Mar 2017] - many bugfixes
1.0 [22 Feb 2017] - add special DMARC treatment, option -f
0.9 [18 Jan 2017] - various bugfixes
0.8 [09 Jan 2017] - don't delete existing queued message
0.7 [01 Jan 2017] - delete existing queued message, bugfixes
0.6 [30 Dec 2016] - bugfixes
0.5 [22 Dec 2016] - skip attempt to ban if ip is already banned
0.4 [14 Dec 2016] - use inotifywait, run continuously
0.3 [14 Dec 2016] - work without amavis (untested)
0.2 [09 Dec 2016] - allow 'short' and 'long' ban specifications
0.1 [06 Dec 2016] - first version

Download relay-enforcer

Donation

I have provided this software free gratis and for nothing. If you would like to thank me with a contribution, please let me know and I will send you a link. Thank you!

My Other Sites

TimeDicer - Onsite/offsite data backup for Windows (uses rdiff-backup)
Finding a 4D Backup Solution
Web Scraping How To - extracting data from web sites

My Programs

Here is a selection of some (other) programs I have written, most of which run under GNU/Linux from the command line (CLI), are freely available and can be obtained by clicking on the links. Dependencies are shown and while in most cases written and tested on an x86-based Linux server, they should run on a Raspberry Pi, and many can run under Windows using Windows Subsystem for Linux (WSL) or Cygwin. Email me if you have problems or questions, or if you think I could help with a programming requirement.

Backup Utilities

rdiff-backup-regress - GNU/Linux script to regress an rdiff-backup archive. [ GNU/Linux: 2012-24 ]

Debian/Ubuntu kernel and LVM Utilities

kernel-remove - GNU/Linux script to list the installed GNU/Linux kernels in a Debian-based distro (e.g. Ubuntu), and can be used to remove an unwanted kernel and related packages, updating grub appropriately. [ GNU/Linux-Debian/Ubuntu: 2010-24 ]
lvm-usage - GNU/Linux script to show available disk space and how it is used; run as cron job to warn if usage is above a set percentage. Provides additional information if LVM is in use. [ GNU/Linux: 2012-24 ]
lvm-delete-snapshot - GNU/Linux script to remove LVM snapshot that has been left over by another process. [ GNU/Linux: 2012-21 ]
netnames - GNU/Linux script shows current name, biosdevname and 'predictable name' of network device - helps with network device name scheme migration. [ GNU/Linux-Debian/Ubuntu: 2020-20 ]
lv-convert2cache - GNU/Linux script to convert an existing LV into a cache LV using a smaller faster device as a cache. [ GNU/Linux: 2022-24 ]

Miscellaneous Programs

sleepwalker - Windows© program which can be run from a remote machine to 'wake up' a Windows© machine behind a router, wait for it to start and then initiate Remote Desktop session. [MS Windows©: 2008-22]
numliststat - GNU/Linux program giving statistical value(s) for a piped-in list of numbers. [ GNU/Linux: 2022-24 ]
pdf-compress - GNU/Linux program to create smaller b/w pdf file from an original large pdf file, especially when original resulted from scanning. [ GNU/Linux: 2016-23 ]
tiny-device-monitor - GNU/Linux program to test webpages (including password-protected) or machines to check they are live; use as a cron job for your own websites, for hardware presenting a webpage, or for any machines with a presence on your local LAN or on the internet. [ GNU/Linux: 2009-24 ]
form-extractor - GNU/Linux program to extract form tags from a web page or downloaded file. [ GNU/Linux: 2012-24 ]
mythic-dns-sync - GNU/Linux program to update DNS record at mythic-beasts.com to match local external ip. [ GNU/Linux: 2016-23 ]
saynoto0870 - For UK, a GNU/Linux script which performs automated lookup of the www.saynoto0870.com database, finding cheap or free geographic number replacements for expensive non-geographic (087* or 084*) numbers. [ GNU/Linux: 2012-12 ]
bind9-resolved-switch - GNU/Linux program for switching permanently between using bind9 or systemd-resolved as the system DNS resolver. [ GNU/Linux: 2016-24 ]
unlock - GNU/Linux remote program for easy entering of decrypt passphrase on a remote machine which has root dm-crypt+LUKS. [ GNU/Linux: 2017-18 ]
wifi-updown - GNU/Linux program to take down wifi interface if there is a working wired interface (or restore wifi if not). [ GNU/Linux: 2018-23 ]
routefix - GNU/Linux program to restore a default ip traffic route if there is none such (e.g. after running wifi-updown). [ GNU/Linux: 2018-23 ]
dutree - GNU/Linux program to show a tree-style list of files and directories at the specified location which are greater than the specified size (default 1GB). [ GNU/Linux: 2012-24 ]
Rents Program - Residential lettings/landlord front office program, with many special features for UK market [MS Windows©: 1991-2024]

Comments

This section is closed. If you have a question, please submit it by email, thank you.

No comments yet for '/programs/help/relay-enforcer.sh.php'