PDMSC live test service for rspamd
==================================

*** READ THIS FIRST ***********************************************************************
This is a LIVE TEST service for individuals and for evaluation and testing, at low volume.
It is not for production mail flow. It works by SENDING EACH MESSAGE YOU SCAN, IN FULL
(headers and body), OFF YOUR SYSTEM to live.pdmsc.com, which PDMSC operates. Only enable it for
mail you are allowed to share and that may leave your network. See TERMS.txt.
*******************************************************************************************

live.pdmsc.com scores a message with three PDMSC engines in ONE secure request:

    PDMSC     the SVM classifier cascade  -> PDMSC_SPAM_0..5 / PDMSC_HAM_0..5 / PDMSC_UNSURE ...
    S1        INTENT classifier           -> PDMSC_S1_<CLASS>
    S2        INTENT, second path         -> PDMSC_S2_<CLASS>
              <CLASS> = FRAUD SPAM BULK TRANSACTIONAL LIST PERSONAL SYSTEM

This package adds a module to your rspamd that makes the request and puts the results in
your normal rspamd results (X-Spamd-Result). It starts in MIRROR MODE: every symbol has
score 0, so you can see what PDMSC would say WITHOUT changing any verdict.

You need
--------
  * rspamd 3.x or newer (tested on rspamd 4.1.0)
  * a PDMSC API key tied to an active licence (see LICENSING.txt: it takes about 5 minutes)
  * outbound HTTPS to live.pdmsc.com

Quick start
-----------
1. Install the package:

       Debian / Ubuntu :  sudo apt install ./rspamd-pdmsc-live_VERSION-1_all.deb
       RHEL / Rocky / Fedora / SUSE :  sudo rpm -i rspamd-pdmsc-live-VERSION-1.noarch.rpm
       FreeBSD :  sudo pkg add ./rspamd-pdmsc-live-VERSION.pkg
                  (rspamd's config is /usr/local/etc/rspamd; curl is needed for --test)
       anything else (BSD, Solaris, ...) :  tar xzf rspamd-pdmsc-live-VERSION.tar.gz
                                            cd rspamd-pdmsc-live-VERSION && sudo ./install.sh

2. Get your API key (LICENSING.txt), then:

       sudo pdmsc-live-setup --key pdl_YOUR_KEY --reload
       sudo pdmsc-live-setup --test

   The tool stores the key in one file only rspamd can read, writes the rspamd configuration,
   checks it with rspamadm, and reloads rspamd. It only touches its own files and a clearly
   marked block that it can remove again (--remove-config). --test sends a harmless sample
   message and tells you in plain words whether your key and licence are accepted.

3. Look at some real mail:

       rspamc < some-message.eml | grep PDMSC

   Example (mirror mode):

       PDMSC_SPAM_3 (0.00) [english x0.30 LL0|-1.117 LL1|-1.170 (would +9)]
       PDMSC_S1_FRAUD (0.00) [p=0.85 147ms (would +10)]
       PDMSC_S2_FRAUD (0.00) [p=0.85 51ms (would +10)]

   "(would +N)" is the score you will get once you turn scoring on:

       sudo pdmsc-live-setup --scored --reload

pdmsc-live-setup options
------------------------
    --key KEY | --key -        the API key (or - to read it from standard input); with no
                               --key you are asked for it, and it is not shown on screen
    --engines pdmsc,s1,s2      which engines to enable (default: all three)
    --mode external|standalone external = three external_services rules (default)
    --scored                   use the production scores (default: mirror mode, all 0)
    --reload                   reload rspamd after the configuration tests clean
    --test                     send a sample message to live.pdmsc.com and report
    --status                   show what is configured (never prints the key)
    --remove-config [--purge]  take the configuration out again (--purge also deletes the key)
    --confdir DIR, --group G   non-standard rspamd config directory / key-file group

Your own scoring changes go in  <rspamd dir>/local.d/pdmsc_live_extra.inc , which the tool never
overwrites (see "Scoring" below).

How it runs
-----------
Mode A, external_services (default): three rules, one per engine (types "pdmsc_live",
"pdmsc_live_s1", "pdmsc_live_s2"). Each can be configured, scored or left out on its own, but
they SHARE ONE REQUEST per message. Mode B, standalone: the same symbols from one
"pdmsc_live { }" section, no external_services needed. See the files in examples/.

Scoring: everything is configurable
-----------------------------------
Put these in pdmsc_live_extra.inc (they apply to all rules), or in a rule, or in the standalone
section / sub-blocks s1 {} s2 {} pdmsc {}:

    scored = true;                 use the production scores instead of 0
    scores {                       explicit score for any symbol; always wins
        PDMSC_SPAM_5    = 20.0;
        PDMSC_S1_FRAUD  = 12.0;
        PDMSC_FAIL      = 1.0;
    }

  PDMSC rule      spam_mult = 3.0;  ham_mult = 3.0;     band n scores +/- n x mult
                  spam_band0 = 1.0; ham_band0 = -1.0;   band 0 = "only just"
                  authfail_cap = true;      HAM never above band 0 if SPF/DKIM/DMARC failed
                  subject_only_cap = true;  HAM from a lone subject model goes to band 0
  S1 / S2 rules   min_conf = 0.5;           under this the answer is *_UNSURE (scores 0)
                  class_scores { FRAUD = 10.0; SPAM = 5.0; BULK = 0.0; TRANSACTIONAL = -2.0;
                                 LIST = -2.0; PERSONAL = -2.0; SYSTEM = -2.0; }

  Production defaults: PDMSC_SPAM_n = +3n, PDMSC_HAM_n = -3n, band 0 = +/-1; S1 and S2 as in
  class_scores. Any symbol can also be re-scored with rspamd's normal local.d/groups.conf.

Connection options: servers = "live.pdmsc.com"; timeout = 2.5; max_size = 2097152;
url = "/v1/scan"; use_https = true;   (the API key comes from api_key_file or api_key)

What the symbols mean
---------------------
  PDMSC_SPAM_n / PDMSC_HAM_n   the classifier's verdict; n = confidence band 0 (barely) .. 5 (very)
  PDMSC_UNSURE / NOLANG / SPARSE   no verdict: unsure / language we have no model for / too little text
  PDMSC_S1_x / PDMSC_S2_x      INTENT's best class when    it is at least min_conf (50%) sure; the
                               option shows its confidence and time
  PDMSC_S1_UNSURE / S2_UNSURE  answered but under min_conf; the option lists every class
  PDMSC_FAIL / PDMSC_S1_FAIL / PDMSC_S2_FAIL
                               that engine did not answer, or the service could not be used. The option
                               says why (standalone mode shows service problems as PDMSC_LIVE_FAIL):
                                 unauthorised: check the API key   -> key wrong or revoked
                                 licence expired / suspended ...   -> licence not valid
                                 rate-limited retry-after=N        -> too fast, see Limits
                                 busy / timeout / http 5xx         -> try again, mail is unaffected
                                 too-large                         -> over max_size

Results for other components
----------------------------
The reply is fetched once per message and left for any other Lua code on that message:

    rspamd_plugins.pdmsc_live.get(task, function(res)
        -- res.state = "ok" | "fail", res.reason, res.authfail (0/1/2)
        -- res.data.pdmsc / res.data.s1 / res.data.s2  (the parsed JSON, see below)
    end)
    task:get_mempool():get_variable('pdmsc_live_json')      -- the raw JSON reply

Limits
------
  * 30 messages per minute per key (a short burst of 10 is allowed), ONE scan at a time per key.
    Intended for low and test volumes; ask if you need more.
  * Messages up to 2 MiB.
  * No uptime guarantee. If anything fails, your mail flows exactly as it did before.
  * The module waits at most 2.5 seconds per message (timeout), well inside rspamd's own limit.

Privacy and terms
-----------------
YOUR MAIL LEAVES YOUR SYSTEM. The WHOLE message (headers, body and attachments, up to 2 MiB) is
sent to live.pdmsc.com over HTTPS, together with your rspamd's SPF/DKIM/DMARC result. It
contains other people's personal data, and you are the one deciding to send it.

  * We score it in memory and do not keep it, copy it, add it to any corpus, or share it.
  * No temporary files: messages (2 MiB at most) are received and scored in memory only.
  * Our staff do not intercept or read your messages. If you want that in writing, we are happy
    to sign a general-purpose NDA covering your test data: enquiries@misullivan.com.
  * Our gateway log records only your key id, the message size, the timings and the verdict
    classes (no Message-ID, subject or addresses); the scoring engines' own logs hold only
    scores and technical MIME-parse warnings, no message text.
  * Do not send mail that must not leave your network, or that you have no right to share.
    Leave it out with an rspamd settings rule, or run the module on a separate rspamd.
  * For individuals and for evaluation and testing only; not production mail flow; no uptime
    guarantee; the scores are an opinion, not a promise.
  * To stop at any time:  pdmsc-live-setup --remove-config --reload  (or revoke the key).

The full terms of the service are in TERMS.txt. Installing and configuring the module means
you accept them.

The module itself is free software (BSD 2-Clause licence, see COPYRIGHT): you may copy, modify
and redistribute it. That licence covers the code only; using live.pdmsc.com still needs a
valid key and the terms above.

Not using rspamd? Call it directly
----------------------------------
    curl -sS -X POST https://live.pdmsc.com/v1/scan \
         -H "Authorization: Bearer pdl_YOUR_KEY_HERE" \
         -H "Content-Type: message/rfc822" \
         --data-binary @message.eml

Reply (JSON):
    {"ok": true, "ms": 150,
     "pdmsc": {"ok": true, "side": "spam", "level": 3, "confidence": 0.30, "lang": "english", ...},
     "s1":    {"ok": true, "class": "fraud", "verdict": "malicious", "conf": 0.85, "probs": {...}, "ms": 147},
     "s2":    {"ok": true, "class": "fraud", "verdict": "malicious", "conf": 0.85, "probs": {...}, "ms": 51}}

If one engine is down you still get the others; the failed one shows {"ok": false, "reason": "..."}.
Status codes: 200 ok | 401 bad key | 403 licence not valid | 413 too large | 429 slow down
(see Retry-After) | 503 busy or every engine failed (see Retry-After).

Optional header X-PDMSC-Authfail: 0, 1 or 2 (none / soft / hard SPF-DKIM-DMARC failure) lets PDMSC
use your own authentication result, as the rspamd module does.

Liveness check without a key:   curl https://live.pdmsc.com/v1/ping

Removing it
-----------
    sudo pdmsc-live-setup --remove-config --reload      (add --purge to delete the key file too)
    then remove the package (apt remove / rpm -e) or run  sudo ./install.sh --uninstall
