"bad message from server!" and 144 msgs truncated

First the logs of the event, then the confs of my system, then the event in the traces.


Here's what happened, in logging terms.

My getmail4 STDOUT log (gotten with this command line that I'm too lazy to make a script out of), .getmail/log/getmail_171213_113429_gdOv.log, both the command and the log are included in the download below, short excerpt of the STDOUT log here:

getmail version 4.54.0
Copyright (C) 1998-2012 Charles Cazabon.  Licensed under the GNU GPL version 2.
SimpleIMAPSSLRetriever:miro.rovis@croatiafidelis.hr@lin22.mojsite.com:993:
IMAP SSL connection  established with fingerprint c9c83d6ecf350841c16e32ca990dccc66b7beafcebe2c6dc6853a279e59c8ed3 using cipher ECDHE-RSA-AES256-GCM-SHA384:TLSv1.2:256
  msg   1/244 (4907 bytes) from  delivered to MDA_external command maildrop (), deleted
bad message from server!  msg   2/244 (6761 bytes) from  delivered to MDA_external command maildrop (), deleted
bad message from server!  msg   3/244 (4720 bytes) from  delivered to MDA_external command maildrop (), deleted
bad message from server!  msg   4/244 (23199 bytes) from  delivered to MDA_external command maildrop (), deleted

 [136 lines cut here] 

bad message from server!  msg 141/244 (7840 bytes) from  delivered to MDA_external command maildrop (), deleted
bad message from server!  msg 142/244 (5520 bytes) from  delivered to MDA_external command maildrop (), deleted
bad message from server!  msg 143/244 (4459 bytes) from  delivered to MDA_external command maildrop (), deleted
bad message from server!  msg 144/244 (7940 bytes) from  delivered to MDA_external command maildrop (), deleted
  msg 145/244 (5983 bytes) from  delivered to MDA_external command maildrop (), deleted
  msg 146/244 (4035 bytes) from  delivered to MDA_external command maildrop (), deleted
  msg 147/244 (8521 bytes) from  delivered to MDA_external command maildrop (), deleted
  
 [30 lines cut here] 

  msg 178/244 (3296 bytes) from  delivered to MDA_external command maildrop (), deleted
  msg 179/244 (3261 bytes) from  delivered to MDA_external command maildrop (), deleted
  msg 180/244 (5121 bytes) from  delivered to MDA_external command maildrop (), deleted

 [60 lines cut here] 

  msg 241/244 (6259 bytes) from  delivered to MDA_external command maildrop (), deleted
  msg 242/244 (7377 bytes) from  delivered to MDA_external command maildrop (), deleted
  msg 243/244 (6760 bytes) from  delivered to MDA_external command maildrop (), deleted
  msg 244/244 (6068 bytes) from  delivered to MDA_external command maildrop (), deleted
  244 messages (4934324 bytes) retrieved, 0 skipped
Summary:
Retrieved 244 messages (4934324 bytes) from SimpleIMAPSSLRetriever:miro.rovis@croatiafidelis.hr@lin22.mojsite.com:993

Again, pls. notice there are 144 lines like this one:

bad message from server!  msg 141/244 (7840 bytes) from  delivered to MDA_external command maildrop (), deleted

And in the log (posting only the today's part) .getmail/log/mirorovis@croatiafidelishr.log --included in the download below--, there is (even more shortened here):

2017-12-13 11:34:34 msg   1/244 (4907 bytes) msgid 448395258/79559 from  delivered to MDA_external command maildrop (), deleted
2017-12-13 11:34:35 msg   2/244 (6761 bytes) msgid 448395258/79560 from  delivered to MDA_external command maildrop (), deleted
2017-12-13 11:34:35 msg   3/244 (4720 bytes) msgid 448395258/79561 from  delivered to MDA_external command maildrop (), deleted
2017-12-13 11:34:35 msg   4/244 (23199 bytes) msgid 448395258/79562 from  delivered to MDA_external command maildrop (), deleted

 [138 lines cut here]

2017-12-13 11:34:51 msg 143/244 (4459 bytes) msgid 448395258/79701 from  delivered to MDA_external command maildrop (), deleted
2017-12-13 11:34:51 msg 144/244 (7940 bytes) msgid 448395258/79702 from  delivered to MDA_external command maildrop (), deleted
2017-12-13 11:34:51 msg 145/244 (5983 bytes) msgid 448395258/79703 from  delivered to MDA_external command maildrop (), deleted
2017-12-13 11:34:51 msg 146/244 (4035 bytes) msgid 448395258/79704 from  delivered to MDA_external command maildrop (), deleted
2017-12-13 11:34:51 msg 147/244 (8521 bytes) msgid 448395258/79705 from  delivered to MDA_external command maildrop (), deleted
2017-12-13 11:34:51 msg 148/244 (5266 bytes) msgid 448395258/79706 from  delivered to MDA_external command maildrop (), deleted

 [90 lines cut here]

2017-12-13 11:35:07 msg 239/244 (8535 bytes) msgid 448395258/79797 from  delivered to MDA_external command maildrop (), deleted
2017-12-13 11:35:07 msg 240/244 (6065 bytes) msgid 448395258/79798 from  delivered to MDA_external command maildrop (), deleted
2017-12-13 11:35:07 msg 241/244 (6259 bytes) msgid 448395258/79799 from  delivered to MDA_external command maildrop (), deleted
2017-12-13 11:35:07 msg 242/244 (7377 bytes) msgid 448395258/79800 from  delivered to MDA_external command maildrop (), deleted
2017-12-13 11:35:07 msg 243/244 (6760 bytes) msgid 448395258/79801 from  delivered to MDA_external command maildrop (), deleted
2017-12-13 11:35:07 msg 244/244 (6068 bytes) msgid 448395258/79802 from  delivered to MDA_external command maildrop (), deleted

Here's how it looks like in my ~/.mailfilter.log (in the download below), shortened here

Date: Wed Dec 13 11:34:34 2017
From: 
Subj: 
File: /home/mr/Maildir                                                   (221)

...[cut a LOT]...

Date: Wed Dec 13 11:34:35 2017
From: 
Subj: 
File: /home/mr/Maildir                                                   (228)

Date: Wed Dec 13 11:34:51 2017
From: 
Subj: 
File: /home/mr/Maildir                                                   (228)

...[cut a LOT]...

Message-ID: <20171213110403.6c5f2217@digimed.co.uk>
Date: Wed Dec 13 11:35:07 2017
From: Neil Bothwick <neil@xxxxx>
Subj: Re: [gentoo-user] Re: Is gnome becoming obligatory?
File: /home/mr/Maildir/.mirorovis@croatiafidelishr.gentoo-usergentooorg (7403)

Message-ID: <20171213110850.11380-1-cbosdonnat@xxxxx>
Date: Wed Dec 13 11:35:07 2017
From: =?UTF-8?q?C=C3=A9dric=20Bosdonnat?= <cbosdonnat@xxxxx>
Subj: [virt-tools-list] [PATCH] python3: fix bytes/string mess in serial
File: /home/mr/Maildir/.mirorovis@croatiafidelishr.virt-tools-listredha (6793)

Message-ID: <4c541345-5efc-d3e5-25bf-9411066f08d8@xxxxx>
Date: Wed Dec 13 11:35:07 2017
From: Martin Brampton <martin@xxxxx>
Subj: Sending only local mail to another mail server
File: /home/mr/Maildir/.mirorovis@croatiafidelishr.postfix-users        (6131)

And it's not just in the logs! These are the 144 messages that were delivered truncated into my maildir (it's included in the download below):

171213-server-bad-mail.mbox

First of all, in case the gentle reader is entertaining suspicion about my claim that this is very likely an attack (if he/she does not, let her/him pls. go to the event in the trace) straight), here's my packages and setup, I don't think they hold any fault in this in the least.


( First, actually, I will have to check with my hoster. Of course, they could have been attacked, but it does look from the traces that the attack was on my side, not on the server side. Well, it looks so to me... )

This is my distro, Devuan, the Debian fork, and it's becoming a very powerful and trustworthy distro --so even unstable need not be a concern:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Devuan
Description:    Devuan GNU/Linux unstable (ceres)
Release:        unstable
Codename:       ceres

These are installed and were in operation at/during/in the event:

# apt-cache policy getmail4
getmail4:
  Installed: 4.53.0-2
  Candidate: 4.53.0-2
  Version table:
 *** 4.53.0-2 500
        500 tor+https://pkgmaster.devuan.org/merged ceres/main amd64 Packages
        500 tor+https://packages.devuan.org/merged ceres/main amd64 Packages
        100 /var/lib/dpkg/status
     4.53.0-1+deb9u1 100
        100 tor+https://pkgmaster.devuan.org/merged ascii-proposed-updates/main amd64 Packages
        100 tor+https://pkgmaster.devuan.org/merged testing-proposed-updates/main amd64 Packages
     4.53.0-1 500
        500 tor+https://pkgmaster.devuan.org/merged ascii/main amd64 Packages
        500 tor+https://pkgmaster.devuan.org/merged testing/main amd64 Packages
#

and:

# apt-cache policy maildrop
maildrop:
  Installed: 2.8.4-2
  Candidate: 2.8.4-2
  Version table:
 *** 2.8.4-2 500
        500 tor+https://pkgmaster.devuan.org/merged ascii/main amd64 Packages
        500 tor+https://pkgmaster.devuan.org/merged ceres/main amd64 Packages
        500 tor+https://pkgmaster.devuan.org/merged testing/main amd64 Packages
        500 tor+https://packages.devuan.org/merged ceres/main amd64 Packages
        100 /var/lib/dpkg/status
#

Now pls. have a look at how long I haven't touched (most of) the config files they had at/during/in the event:

# ls -ABRgo .mailfilter .maildirmake.incL* .getmail/config/mirorovis@croatiafidelishr
-rw------- 1   580 2017-12-06 23:12 .getmail/config/mirorovis@croatiafidelishr
-r-------- 1   480 2013-10-30 15:20 .maildirmake.incL1
-r-------- 1   516 2017-05-24 13:41 .maildirmake.incL2
-r-------- 1   549 2013-10-30 15:20 .maildirmake.incL3
-r-------- 1 10818 2017-05-24 13:36 .mailfilter
#

NOTE: and that configuration I actually only minimally edited and brought in from my previous --but I will be back some day-- distro Gentoo, newbies should find useful fragments of info how to set up Mutt with Getmail and Maildrop starting from

Postfix smtp/TLS, Bkp/Cloning Mthd, Censorship/Intrusion

By the fact that I've had this setup without, previously, anything, anything at all to ever happen in my Devuan similar to this event, I dare conclude, that my confs and my setup are all correct.

Still, just in case, the sole very marginally possible suspect, just to dispell the doubts, the mirorovis@croatiafidelishr getmail4's conf file is included in the download below.

The sole edit of one week ago was my commenting out the clamdscan functionality (maybe later/elsewhere about why).


Now the event in the traces.

Plain and simple, the minimally anonymized PCAPs of the (kind of) session of three short intervals online, late this morning UTC. The accompanying screencasts?... Maybe, or maybe only parts of them, too much work to anonymize them at the intervals where Ethers need to be blanked out...

---

dump_171213_1102_gdO.pcap

I ran this command:

$ torsocks wget "https://bugs.devuan.org//cgi/bugreport.cgi?bug=145&mbox=yes" \
	|& tee ~mr/LOG_/torsocks_wget_$(date +%y%m%d_%H%M%S)_$(hostname)0

And indeed I have a few of the logs, the first few, in the span of some 10 minutes, i.e. all from that trace --but all with hidden content in the traces from even the user; it's Tor--, are all like with the content that would --if it were decryptable-- correspond to this one:

--2017-12-13 11:07:57--  https://bugs.devuan.org//cgi/bugreport.cgi?bug=145&mbox=yes
Resolving bugs.devuan.org (bugs.devuan.org)... 1513163277 PERROR torsocks[22291]: socks5 libc connect: Connection refused (in socks5_connect() at socks5.c:202)
failed: Non-recoverable failure in name resolution.
wget: unable to resolve host address ‘bugs.devuan.org’

However, none of that is decryptable for me...

During this time --I can see in the screencast, I was fixing perms in /etc/grsec/policy. I run the grsec-unoff kernel that is not easily broken in. Pls. read about the (likely) tries at these:

https://github.com/minipli/linux-unofficial_grsec/issues: 13 17 18 19 20 21

Also absolutely worth noting is the martian source that I got:

Dec 13 11:02:58 gdOv kernel: [71548.200537] sky2 0000:06:00.0 eth1: Link is up at 100 Mbps, full duplex, flow control both
Dec 13 11:03:09 gdOv kernel: [71558.779709] grsec: (root:U:/bin/ip) exec of /bin/ip (ip l show dev eth1 ) by /bin/ip[bash:22257] uid/euid:0/0 gid/egid:0/0, parent /bin/bash[bash:3672] uid/euid:0/0 gid/egid:0/0
Dec 13 11:03:09 gdOv kernel: [71558.783913] grsec: (root:U:/bin/ip) exec of /bin/ip (ip a show dev eth1 ) by /bin/ip[bash:22258] uid/euid:0/0 gid/egid:0/0, parent /bin/bash[bash:3672] uid/euid:0/0 gid/egid:0/0
Dec 13 11:03:19 gdOv kernel: [71569.461555] IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev eth1
Dec 13 11:03:19 gdOv kernel: [71569.461564] ll header: 00000000: 00 30 4f 47 37 17 2c 95 7f xx xx xx 08 00        .0OG7.,...N...
Dec 13 11:03:19 gdOv kernel: [71569.476559] IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev eth1
Dec 13 11:03:19 gdOv kernel: [71569.476566] ll header: 00000000: 00 30 4f 47 37 17 2c 95 7f xx xx xx 08 00        .0OG7.,...N...
Dec 13 11:03:19 gdOv kernel: [71569.477390] grsec: (root:U:/sbin/dhclient-script) exec of /sbin/dhclient-script (/sbin/dhclient-script ) by /sbin/dhclient-script[dhclient:22259] uid/euid:0/0 gid/egid:0/0, parent /sbin/dhclient[dhclient:4947] uid/euid:0/0 gid/egid:0/0

Of course, it's real hex ciphers, not xx xx xx in those places. The previously manually macchanger'ed local Ether I don't mind to expose.

Pls. note that the attacker has programmed the waiting for my coming online. Likely has somebody at the provider's, or belongs to the special shadows. Or has programmed his exploit --which does not have to be anything very special, just a little MiTM'ing to ruin my mail-- to act right when I connect. Or, in theory, is so knowledgeable and so fast to be able to get himself an alert and act so promptly.

Because as soon as the link was up, 21 seconds later and there you go! Argh!...

dump_171213_1133_gdO.pcap (there's also dump_171213_1133_gdO_SSLKEYLOGFILE.txt but it doesn't decrypt anything for me)

Finally having arranged the perms in the /etc/grsec/policy correctly (grsec RBAC is marvelous, if interested, go to the RBAC policy section on grsecurity forums, such as: Libvirt virtualization policies and around)...

But I was saying, having finally arranged the perms in the /etc/grsec/policy correctly here I finally downloaded the Devuan bug 145 mbox:

--2017-12-13 11:33:20--  https://bugs.devuan.org//cgi/bugreport.cgi?bug=145&mbox=yes
Resolving bugs.devuan.org (bugs.devuan.org)... 193.170.194.207
Connecting to bugs.devuan.org (bugs.devuan.org)|193.170.194.207|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
Saving to: ‘bugreport.cgi?bug=145&mbox=yes’

     0K ......                                                 72.2M=0s

2017-12-13 11:33:39 (72.2 MB/s) - ‘bugreport.cgi?bug=145&mbox=yes’ saved [6426]

Because I was interested in helping with that bug... But how can I do anything else now... I have to deal with this nuissance, and ascertain, or try to get people --the Getmail and Maildir maintainers, Plus.hr the hoster, T-com the "Croatian" Telecom, others, such as people at Wireshark, get them to ascertain how come maildrop deleted messages that it really didn't deliver --just stumps of those it did-- what is the exploit in the wild that made my (presumed and unknown) attackers capable of doing so?

In this trace:

dump_171213_1146_gdO_noPW.pcap

I mentioned I needed to ask others for help, such as people at Wireshark. Hi guys! I see in my Wireshark:

$ wireshark -v
Wireshark 2.2.6 (Git Rev Unknown from unknown)

Copyright 1998-2017 Gerald Combs  and contributors.
License GPLv2+: GNU GPL version 2 or later 
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Compiled (64-bit) with Qt 5.7.1, with libpcap, with POSIX capabilities (Linux),
with libnl 3, with GLib 2.50.3, with zlib 1.2.8, with SMI 0.4.8, with c-ares
1.12.0, with Lua 5.2.4, with GnuTLS 3.5.8, with Gcrypt 1.7.6-beta, with MIT
Kerberos, with GeoIP, with nghttp2 1.22.0, with QtMultimedia, without AirPcap.

Running on Linux 4.9.68-unofficial+grsec171212-03, with locale en_GB.UTF-8, with
libpcap version 1.8.1, with GnuTLS 3.5.8, with Gcrypt 1.8.1, with zlib 1.2.8.
AMD Phenom(tm) II X4 965 Processor

Built using gcc 6.3.0 20170516.
$

...the bug that I reported in wireshark-users is fixed! I see it really was a bug, because the reason that I couldn't do what I tried in: Filtering on (negated) frame.time_relative filters out wrong frame.number or locally in

The Test Sample for the (Imaginary or Not) Bug was because there was a bug! I am perfectly able to do now just what I tried then but couldn't.

Namely in this trace, I try and see what's wrong with the router... And find nothing that I can identify as misconfigured... But I had to log in, and that particular packet needed to be deleted for posting, and I did it like I explain in that other local topic linked. Now that recipe from that topic works!

---

Enough for now. Lot's of info is there. It is pretty clearly and almost completely documented, as far as I can post it. Let's try and get help. Because I'm not (yet) an expert (or won't ever be an expert).

---


The files necessary for this entire study (so far) are listed in:

ls-1

171213-server-bad-mail.mbox
dump_171213_1102_gdO.pcap
dump_171213_1133_gdO.pcap
dump_171213_1133_gdO_SSLKEYLOGFILE.txt
dump_171213_1146_gdO_noPW.pcap
getmail_171213_113429_gdOv.log
getmail.CMD
mailfilter.log
mirorovis@croatiafidelishr
mirorovis@croatiafidelishr.log

and verify to: ls-1.sum signed by: ls-1.sum.asc

You might find dump_dLo.sh script from my uncenz program more useful then downloading each file separately.

---