A bulk of work for improving the detector accuracy and quality based on the logs gathered and provided so far.
- detection happens recursively now on all data available in the request (headers, path params, body) - deep search. The script will convert the data to a nested structure if it knows how to read it, to provide better accuracy of matches, and better context for where it was found.
- introduced LocationDetector which supports several common representations of coordinates, and can generate different severity incidents depending whether or not the match seems to be confident.
- detecting constants respects word boundaries in order to decrease the amount of false positives (no match for 'lat' in 'platform')
- added support for some more of the most common content-types (based on all the logs so far): application/x-gzip, application/x-protobuf. For the latter one I integrated a library for extracting information from common binary protocols (Kaitai struct) - we should be able to use it for some other binary protocols too. It proven to extract some protobuf payload in the dating app log I received from Hannah, and there were some matches (for example IMEI) that we would not have detect until now. The next content-type to handle is application/grpc, which seems rather common. It's payload is protobuf though, so it shouldn't be too hard.
- conversations are now grouped by destination and also by taking the KeepAlive flag into account.
- better TLS related incidents. Report plaintext (unencrypted) requests, and report requests that should be HTTP, but where TLS negotiation failed. We are limited in terms of what mitmproxy provides here.
- rework the incidents output structure as discussed with Bernd (we are still iterating on it)
- email detector checks MX records
Note: the output of the script is now more verbose. For example, all connections are reported as low-severity incidents. When browsing incidents manually, I recommend to pass '-g' option flag. It will group incidents by trigger, which is way easier to read for human.
Note: new dependency, pip3 install kaitaistruct