-- Fuglu 1.0-WIP xxxx-xx-xx


-- Fuglu 0.10.8 2021-02-15
 * Python 3.9 compatibility
 * DKIM plugin: check if signed in author/sender domain
 * Fuzor plugin: Kafka reporting backend
 * FProt plugin: use bytes buffer directly
 * Fixes preventing unmodified msg modification
   - Fprot plugin adding Content-Type header
   - smtpconnector/esmtpconnector detecting end-of-data too early
 * Attachment manager limits for max num files extracted
 * Attachment plugin: rules for protected archives
 * fixed "fuglu_conf" fixes and "-d" option to print defaults
 * "--logconfig" option to define a specific log-config file foreground mode

Developers:
 * smtpconnector: read SIZE in MAILFROM state if available,
   defer if buffer creating suspect is much smaller
 * URIExtract:
   * refined href link extraction, imags src uri extraction
   * detect uris split on different lines
   * check mime parts with parsing errors too
   * extract redirects
   * use <base> html-tag for uris
 * clientinfo:
   * regex input possible to trust received lines
   * input to skip received with same domain (rcv from/by)
 * Async milter implementation (enabled by milter + asyncprocess as backend)
 * Async milter process pool  (milter-mode only)
   "backend=asyncprocess"


-- Fuglu 0.10.7 2020-07-06
 * dropped support for Python 2 - Python 3.6 is new minimum requirement
 * use Python EmailMessage (rfc5322) object
 * minor improvements and fixes to many plugins (e.g. dkim, spf, sa)
 * new plugin: imapcopy (from fuglu-extra-plugins)
 * new plugin: uriextract (from fuglu-extra-plugins)
 * new plugin: DMARC check
 * new tool 'fuglu_healthcheck' allows monitor fuglu health (docker, zabbix)


-- Fuglu 0.10.6 2019-12-03
 * improvements to DKIM and SPF plugins
 * pass envelope sender to spamassassin
 * contain suspect.id in reject message
 * THIS WILL BE THE LAST VERSION WITH OFFICIAL SUPPORT FOR PYTHON2!

-- Fuglu 0.10.3 2019-11-13
 * ClamAV fix for infinite loop
 * Improvements for multiline attachment filenames
 * Include suspect id in reject response

Developers:
 * sssp refactor and unit tests
 * more tests for 7z-archives
 * on bounce, save postfix queue-id as tag
 * improvements decoding from-type headers

-- Fuglu 0.10.1 and 0.10.2, 2019-07-31
 * switch python3 testing from 3.4 to 3.6
 * allows to define custom conf.d (in fuglu config file or as runtime parameter)
 * disable bounces for addresses given by file
 * in reinject, use SMTPUTF8 for unicode addresses even if not given
 * new plugin SALearn

Developers:
 * improvements string encoding/decoding
 * improvements get_client_info
 * fix in SpearPhishPlugin for empty sender
 * virusscanner - skip on previous virusscanner hit
 * Fuzor: Prevent checksum recalculation, always add tag, improved exception handling
 * RSpamdPlugin: fix for non-ascii chars in rspamd answer
 * strip_attachments moved from SAPlugin to Suspect
 * Fuzor: Strip oversize option
 * Py3 fix for fuglu_debug

-- Fuglu 0.10.0, 2019-04-11
 * require setuptools
 * performance improvements (smtpconnector)
 * improved header parsing and decoding
 * Milter mode:
   * handle multiple suspects in one session
   * significant stability improvements
 * F-Prot plugin: lint file mode
 * RSpamd: new plugin (imported from fuglu-extra-plugins)
 * Fuzor: new plugin (imported from fuglu-extra-plugins)
 * AppenderSkipper: new plugin
 * Threading backend: force configurable number of free worker threads
 * FileTypePlugin: process only message parts marked as "attachment", "inline" or with filename present

Developers:
 * More Python 3 fixes
 * Use objgraph for debugging (fuglu_control)
 * Attachment manager: more attachment properties
 * New redis extension: provides keepalive redis connections
 * Milter: access to queue id and SASL info
 * Docker dev environment
 * static prepend_header_to_source routine in Suspect (conversion, encoding, prepend)
 * static decode_msg_header routine in Suspect (decoding and conversion to string)
 * static parse_from_type_header routine in Suspect (returns list of (display name, mail))
 * improved archive error handling in mailattach
 * improved getting payload for corrupted mail
 * MailAttachment
   * location for attachment (in archive tree if file is archive)
   * md5 and sha1 checksum
   * Content-Disposition: inline/attachment
   * in_archive property useful for recursive extraction
   * parts with decode errors get contenttype: application/unknown

-- Fuglu 0.9.0, 2018-10-25
 * ClamAV plugin: Fix sending large files to ClamAV which caused timeouts
 * Attachment plugin: Improve file type detection of unnamed attachments
 * Spamassassin plugin: strip attachment function now generates proper mail format
 * F-Prot plugin: fix missing content-type header
 * New Plugin: Sender Rewrite Scheme in module domainauth (imported from fuglu-extra-plugins)
 * Fix for CentOS-6 system script to correctly stop all processes
 * Configurable check for mail address validity, fail action and message
 * Configuration option to remove (default) or keep temporary message files on internal errors (receive,address,unknown)
 * new input "att_mgr_cachesize" to define attachment manager cache size
 * new input "archiveextractlevel" in FiletypePlugin to define attachment archive extract level
 * allow binding incoming ports to different addresses
 * gzip archive handler
 * new milter connector allowing changes to the message
 * support for unicode addresses (SMTPUTF8) and 8BITMIME in smtpconnctor
 * checksums for all attachments on request
 * fuglu with "--foreground" option ignores logging.conf and prints to stdout

Developers:
 * extended signature of scan_stream function in AV plugins / new subclass shared.AVScannerPlugin
 * tighter integration of dnsquery extension
 * Mail attachment manager - "Suspect.att_mgr" - storing all message attachments and message parts
 * Attachment objects are created as needed, cached as long as withing the cache limit
 * caching framework for member variables and methods
 * New implementation for address validity check
 * Consistent types in SuspectFilter/Suspect getters/setters (bytes/unicode)
 * nonzero fuglu return values on error
 * fix encoding problems in bounce messages
 * fixes and exception handling in string encoding/decoding for Python 3
 * mail scantime details (to be written to log)


-- Fuglu 0.8.0, 2018-04-23
 * Significant improvements of Python 3 support fixing many issues.
 * Logging rewrite, log level can now be changed at runtime. Logging will spawn a dedicated process.
 * Caching rewrite, caching will spawn a dedicated process.
 * Multiprocessing improvements improving performance and fixing many issues.
 * Toggle between multithreading and multiprocessing run modes at runtime
 * IPv6 support for connections to Postfix and SpamAssassin.
 * Allow multiple plugin directories
 * SpamAssassin plugin: option to strip attachments from big mails
 * ClamAV plugin: option to fallback to clamscan if clamd fails

Developers:
 * New dnsquery extension to faciliate dns lookups
 * convenience function is_ham in shared.Suspect
 * gitlab CI tests


-- Fuglu 0.7.0, 2018-01-14
 * Attachment Plugin:
  * support more archive types ( tar.gz, tgz, tar.bz2, .7z)
  * extract archives based on detected magic mime type if possible (instead of file ending)
  * the plugin now supports scan-time configuration options
 * New Plugin: SpearPhish detection
 * Improve body text extraction by also extracting multipart/mixed and the message's epilogue
 * various refactorings / remove lots of code smell
 * suspect: new "blocked" default tag to indicate a message contains unwanted content (blocked attachents etc), but is not strictly malicious like spam/virus
 * new tool 'fuglu_client' allows scanning messages from the commandline by either injecting them into a running fuglu
   with the Netcat plugin or by starting its own temporary fuglu instance. This is helpful for debugging or running fuglu
   in fetchmail like environments
 * multi processing mode: instead of running multithreaded the scanning tasks can now run in their own process
 * prequeue mode can now be configured to only deliver to one recipient per message - useful in spamtrap setups
 * Add IPv6 support in various plugins connecting to other daemons (clamav, spamasassin, drweb, fprot, icap, sssp)
 * domainauth plugin: remove hard dependency of pkg_resources

Developers:
 * suspect.to_addr is no longer a writeable member, it has been converted to a property which always returns the first recipient of a message


-- Fuglu 0.6.6, 2016-05-17
 * Minimum python version 2.6
 * Many changes to start making fuglu python3 compatible
 * core: Keep additional groups when dropping privileges
 * ScriptFilter : stop() now requires action and message arguments for any action other than DUNNO
 * Attachment Plugin: treat .z archives like .zip
 * improve DKIMVerify plugin - it is now slightly less experimental and actually works in most cases ;-)
 * new experimental Plugin "DomainAuth" ( poor man's DMARC: check if message is either DKIM or SPF authenticated)
 * make writing the SpamStatus header configurable
 * fix crash when reading null values from db config overrides
 * support float values for highspamlevel
 * start the fuglu-extra-plugins repository to reduce the amount of mostly unused plugins in the core distro
 * new log handlers: Group(Readable|Writable)(Timed)?RotatingFilehandler
 * Extracting the client ip address from received headers now supports IPv6

Developers:
 * new helper class fuglu.shared.FileList : maps lines in a file to a list which refreshes automatically

-- Fuglu 0.6.5, 2015-11-19
 * core: Fix unquote lines with leading dots in pre-queue mode
 * core: Specifiy stricter default permissions for daemon's pidfile
 * SuspectFilters: support new fields clienthostname, clientip, clienthelo
 * smtp connector: don't crash if a plugin tries to backscatter using REJECT
 * Attachment Plugin: improve detection of installed filemagic lib

-- Fuglu 0.6.4, 2015-07-30
 * core: warn about known issues/security risks of current version in startup/lint.
   Note: This performs a DNS lookup containing *only* the current version number
   This check can be disabled by setting [main]versioncheck=0
 * core: optimized thread handling to reduce CPU usage
 * core: lint now also checks the logging configuration
 * F-Prot plugin: support additional scan options
 * Attachment plugin: Support rarfile extraction (requires additional lib) and improve zipfile extraction
 * Clamav plugin: new experimental option "pipelining" to scan multiple messages over the same connection
 * new plugin: DrWeb Antivirus (experimental)
 * new appender plugin: statsd sender (experimental) : Send various stats to a statsd/graphite system
   * Plugin execution time
   * Message decisions
   * Per recipient message stats

Developers:
 * plugdummy tool: now supports prepender plugins

-- Fuglu 0.6.3, 2015-03-20
 * lint now also checks SuspectFilter files in some plugins
 * SuspectFilters: improved HTML stripping
 * Attachment plugin: can now check file names in zip archives and extract files from zips to check their content type
 * bugfix: Spamassassin plugin no longer treats ham as spam if forwardoriginal=0 and spamheader contains bayes test result
 * bugfix: Attachment plugin should no longer cause fuglu to crash under load if python-file is installed
 * bugfix: Attachment plugin should now evaluate rules within the same file in the correct order

Developers:
 * suspects now have an additional built-in tag 'scantimes' which contains a list of tuples (<plugin section name>, scan time) - you no longer have to add your custom Myplugin.time tag
   Existing plugins no longer write the .time tag themselves.
 * the built-in "decisions" tag now uses the plugin's section name instead of the  human readable name

-- Fuglu 0.6.2, 2014-09-05
 * new Prepender: PluginFraction (basically the opposite of PluginSkipper:include instead of exclude list)
 * new tool: fuglu_suspectfilter . Use this to find out which rules in a suspect filter match a sample message
 * Spamassassin Plugin: unix domain socket support
 * Improved bounce message formatting
 * new config: outgoing_host for reinjecting messages into a remote MTA
 * the HeaderPlugin has been removed, the plugin was buggy and the functionality is redundant
 * Clamav Plugin: removed the deprecated STREAM scan method with INSTREAM
 * tuned the internal thread pool which should slightly improve scan performance
 * reorganized script directory, added init/systemd scripts for the common linux distributions
 * added action DISCARD as alias to DELETE to be more consistent with postfix

Developers:
 * plugdummy tool: can run run multiple plugins in one go, autodetects plugin type, can read messages from stdin
 * suspect.get_tag() now takes a default argument which will be returned if the requested tag is not found

-- Fuglu 0.6.1, 2013-11-15
 * new Plugin: ScriptFilter (dynamically loaded filter scripts)
 * new (experimental) domain authentication plugins: DKIM Sign / Verify, SPF Check
 * Archive Plugin:
    * new options chown/chgrp/chmod to change owner/permissions of archived file
    * storage directory and filename are now configurable using templates. "makedomainsubdir" option is deprecated
 * fuglu start script has more common options like --foreground, --pidfile, -c <configfile>
 * milter protocol: some improvements. It "should work" but has not received much testing
 * Antivirus plugins: Increased default timeout to 30 sec
 * Prequeue Bugfix: correctly close connection to 2nd postfix instance when message is deferred/rejected/discarded
 * Fuglu doesn't completely shut down anymore if one connector fails to start
 * SQL extension: (experimental) Some configuration values can now be dynamically loaded from a database
   see: https://fumail.gitlab.io/fuglu/configuration-index.html#fetching-scan-time-configuration-values-from-a-database
 * Reduce log and header noise: The log template is now configurable, removed tag logging by default (use ${tags} in logtemplate to get the old behaviour)
   The suspect-id header can now be disabled

Developers:
 * plugdummy.py has new options (show default plugin config, lint plugin)
 * Spamassassin Plugin: now can pass "temporary headers" to SA. read from tag `SAPlugin.tempheader`. These headers are only visible to Spamassassin.
 * apply_template now takes a callbackfunction which allows modifying the built-in values before the template is applied
 * to add support for configuration domain/user overrides from a database use a fuglu.extension.sql.DBConfig object instead of self.config
 * suspectfilter get_args can now optionally return extended info about a match (matched field, used regex, matched value)
 * suspect has a new method get_client_info which tries to extract helo, ip and reverse dns information from received headers
 * fuglu_control has a new command 'netconsole' which starts a python interactive shell on a network socket. This allows runtime debugging of plugins or fuglu itself.

-- Fuglu 0.6.0, 2012-07-04
 * various bugfixes and improvements in before-queue support
 * new Plugin: ActionOverride (Custom action based on filter)
 * new Plugin: F-prot Antivirus
 * new Plugin: SSSP (Sophos Antivirus scan over SSSP Prodocol / SAVDI)
 * new Plugin: ICAP (Antivirus scan over ICAP - Sophos/Symantec/...)
 * new Plugin: Killer (Deletes All messages) - used in some special setups
 * Vacation Plugin: Now supports domain wide ooo messages
 * SuspectFilter Files now have support advanced regex format /<perl style regular expression>/<modifiers>
   see:  https://fumail.gitlab.io/fuglu/plugins-index.html#suspect-filters
 * Archive Plugin: Improved Logging
 * Attachment Plugin: bugfix, sendbounce option could not be disabled
 * Spamassassin Plugin: bugfix, better detection of spamflag header
 * Clamav Plugin / Spamassassin Plugin / Attachment Plugin: new option 'rejectmessage' for before queue rejects
 * improved core support for multi recipient messages. it is no longer absoultely required to set fuglu_*_destination_recipient_limit=1
   (but some plugins still only act on the first recipient)
 * outgoinghelo is no longer mandatory, fuglu auto-detects the local hostname if the config option is empty
 * new tool: fuglu_conf to export current config, show differences from the default (fuglu_conf -n) etc
 * the message returned to the connecting client is now configurable

Developers:
 * suspect.addheader() has a new 'immediate' option to make headers visible for consecutive plugins. by default, headers are only added before re-injecting
 * Spamassassin Plugin: writes new tag "SAPlugin.skipreason' if the message was not scanned
 * new way of defining required config variables. plugins can now provide defaults and config option descriptions.
   see: https://fumail.gitlab.io/fuglu/plugins-index.html#define-configuration-options-for-your-plugin
   important: for default values to work, make sure your plugin doesn't read it's own config values in __init__ !
 * Suspect.getSource() now has an optional maxsize parameter to limit the amount of body data read into memory
 * new development tool: 'plugdummy.py', dry-run and debug your plugins without a running fuglu
   see: https://fumail.gitlab.io/fuglu/plugins-index.html#debug-a-plugin-without-running-fuglu

-- Fuglu 0.5.0, Released 2011-10-28
 * started changelog ;-)
 * bugfix: Milter protocol errors
 * bugfix: Incorrect unquoting of two leading dots in smtp transactions could lead to broken s/mime sigs
 * A few plugins now have a 'PROBLEMACTION' so it is now configurable if a message should be deferred or accepted if for example the clamav daemon is not reachable
 * new option: [main] trashlog (boolean), default False, if True, Fuglu writes a 00-fuglutrash.log file in the trashdir listing all deleted messages
 * SuspectFilters (previously 'HeaderFilters'):
 	* now supports header wildcard (Character '*')
 	* accepts colon at the end
 	* alias "from_address" for envelope_from and "to_address" for envelope_to
 	* new type: 'mime:<headername>' to check mime headers in all attachments
 	* new type: 'body:stripped'  (or just 'body') to match all text parts with html tags removed and newlines replaced by space
 	* new type: 'body:raw' to match the decoded body in all text/* parts
 	* new type: 'body:full' to match the full undecoded body
 	* bugfix: regexes now match everywhere in the field, not just at the beginning
 * Attachment Plugin:
 	* Configuration of attachment rules can now be read from a database table (sql extension)
 	* Support rule configuration from sql
 	* Support both 'python-magic' and 'python-file' libraries
 	* Improved log messages
 	* "BOUNCE" action has been removed and replaced with the config option 'sendbounce'
 * Archive Plugin:
 	* Allow archive exception rules (append "no" to the regex)
 	* Messages are now stored as '<suspect-id>.eml' instead of a random string
 * Clamav Plugin:
 	* Virusaction is now configurable
 * Spamassassin Plugin:
 	* now Supports lowspam / highspam configuration
 	* new option "scanoriginal" -> Scans the unmodified original message (not the probably modified version from other plugins)
 	* new option "forwardoriginal" -> Does forward the original message source to the next plugin, without modified SA-Headers

Developers:
 * Suspects now also have a "highspam" tag which allows different behavior based on whether a message is "probably" or "definitely" spam
 * if a plugin cannot be loaded, a full stack trace is displayed
 * Appenderplugins can now read the tag "injectanswer" which contains the final smtp answer from SMTP reinjects
 * (sql extension) DBFiles allow reading sql tables and treat them like config files
 * HeaderFilter is now called SuspectFilter

-- Fuglu 0.4.5, Released 2010-05-20
