Receiving email via IMAP in Totum

Question

Instruction

Answer 1

From root, install the necessary software:

apt install fetchmail procmail uudeview

Switch to the totum user:

su totum

Create the fetchmailrc configuration file:

nano ~/.fetchmailrc

poll imap.host.com protocol IMAP port 993
  user "email@host.com" password "some_pass_here" is "totum" here
  mda "/usr/bin/procmail"
  ssl

Check the port and settings parameters with your IMAP server. If it doesn't have SSL, ask GPT how to configure it.

Set permissions on fetchmailrc:

chmod 700 ~/.fetchmailrc

Create the procmail config:

nano ~/.procmailrc

MAILDIR=$HOME/Mail
DEFAULT=$MAILDIR/mbox
LOGFILE=$MAILDIR/procmail.log

:0
* ^From.*
| $MAILDIR/extract.sh

Set permissions on procmailrc:

chmod 600 ~/.procmailrc

Create the Mail folder and inside it data:

mkdir -p ~/Mail/data && cd ~/Mail

Emails will be stored in data!

Create the email processing script (towards the end, find MY_KEY and replace it with a random set of characters and save it, also replace YOUR_HOST with your totum host or ip and if ip, change the protocol from https to http):

nano extract.sh

#!/bin/bash

EMAIL=$(cat)
DATE=`date +%s`
RANDOM_NUM=$(echo "$RANDOM" | head -c 3)
UNIQ_NAME="email_${DATE}_${RANDOM_NUM}"
DIR="$HOME/Mail/data/$UNIQ_NAME"
JSON="$DIR/$UNIQ_NAME.json"

mkdir -p $DIR

echo "$EMAIL" > "$HOME/Mail/data/$UNIQ_NAME.mail"

echo "$EMAIL" | uudeview -i +a +o -t -p $DIR -

decode_and_concatenate() {
  local input_string="$1"
  local result=""

  for part in $input_string; do
    decoded_part=$(echo $part | awk -F '[?]' '{print $4}' | openssl enc -base64 -d -A)
    result+="$decoded_part"
  done

  echo "$result"
}


FROM=$(echo "$EMAIL" | formail -c -x From: | awk -F'[<>]' '{print $2}')

FROM_NAME=$(echo "$EMAIL" | formail -c -x From: | awk -F'<|>' '{print $1}' | sed 's/^[ \t]*//;s/[ \t]*$//' | while read -r name_line; do
    if [[ $name_line == =?* ]]; then
      name_clear=$(echo "$name_line" | sed 's/  / /g')
      decode_and_concatenate "$name_clear"
    else
      echo "$name_line"
    fi
  done | tr -d '\n' | base64 -w 0)

TO=$(echo "$EMAIL" | formail -x To: | sed 's/^[ \t]*//;s/[ \t]*$//;s/"//g' | sed 's/.*<\([^>]*\)>.*/\1/')

SUBJECT=$(echo "$EMAIL" | formail -c -x Subject | sed 's/^[ \t]*//;s/[ \t]*$//' | while read -r sub_line; do
    if [[ $sub_line == =?* ]]; then
      sub_clear=$(echo "$sub_line" | sed 's/  / /g')
      decode_and_concatenate "$sub_clear"
    else
      echo "$sub_line"
    fi
  done | tr -d '\n' | base64 -w 0)

if [ -f "$DIR/UNKNOWN.001" ]; then
  BODY=$(base64 -w 0 "$DIR/UNKNOWN.001")
else
  if formail -c -x Content-Type: < "$DIR/0001.txt" | grep -qv "multipart"; then
    BODY=$(formail -I "" < "$DIR/0001.txt" | base64 -w 0)
  else
    BODY=$(base64 -w 0 "$DIR/0001.txt")
  fi
fi

FILES=()

for FILE_PATH in "$DIR"/*; do
    FILE_NAME=$(basename "$FILE_PATH")
    FILE_NAME_BASE=$(echo "$FILE_NAME" | tr -d '\n' | base64 -w 0)
    if [[ ! "$FILE_NAME" =~ ^UNKNOWN\.[0-9]+$ ]] && [[ ! "$FILE_NAME" =~ ^000[0-9]+\.txt$ ]]; then
        FILE_CONTENT=$(base64 -w 0 "$FILE_PATH")
        FILES+=("{\"name\":\"$FILE_NAME_BASE\",\"filestringbase64\":\"$FILE_CONTENT\"}")
    fi
done

FILES_JSON=$(printf ",%s" "${FILES[@]}")
FILES_JSON="[${FILES_JSON:1}]"

echo "{\"email_id\":\"$UNIQ_NAME\",\"from\":\"$FROM\",\"from_name\":\"$FROM_NAME\",\"to\":\"$TO\",\"subject\":\"$SUBJECT\",\"body\":\"$BODY\",\"files\":$FILES_JSON}" > $JSON

KEY="my_key"

curl -d @"$JSON" -H "Content-Type: application/json" -H "Authorization: $KEY" https://YOUR_HOST/Remotes/get_email

rm -r $DIR

Make the script executable:

chmod +x extract.sh

Go to Totum in ttm__remotes and create a remote get_email (assign a user and add the code, replacing MY_KEY with the previously saved key):

=: if(condition: $#headers[Authorization][0] = "MY_KEY"; then: $insert)

insert: insert(table: 'email_table'; field: 'raw' = $#input; field: 'email_id' = $email_data[email_id]; field: 'email_data' = $email_data_decode; field: 'email_files' = $email_files_decode; log: true)

email_data_decode: rowAdd(row: $email_data; field: "from_name" = $from_name_decode; field: "subject" = $subject_decode; field: "body" = $body_decode)
    from_name_decode: strBaseDecode(str: $email_data[from_name])
    subject_decode: strBaseDecode(str: $email_data[subject])
    body_decode: strBaseDecode(str: $email_data[body])

        ~email_data: rowKeysRemove(row: $extract; key: "files")

email_files_decode: listReplace(list: $extract[files]; action: "name" = $name; value: "value")
    name: strBaseDecode(str: $#value[name])

~extract: jsonExtract(text: $#input)

Create a table in Totum with name=email_table and add fields:

email_id — {string}
raw — {string}
email_data — {data}
email_files — {file} + [multiple] + [hashes at the end of the file] or [protected file]

Set the table to [pagination] and optionally [reverse order]

First, check from the server console:

fetchmail

The log is recorded in:

tail -100 ~/Mail/procmail.log

Add a cron job (can be from the system or from totum). If from totum, execSSH must be enabled! Choose the check period as desired.

From Totum, add a cron job to check and clean the technical folder of emails older than 3 days and rotate the log:

= : execSSH(ssh: "fetchmail -s && find ~/Mail/data/ -type f -mtime +3 -exec rm {} \;) && tail -n 1000 ~/Mail/procmail.log > ~/Mail/procmail.log.tmp && mv ~/Mail/procmail.log.tmp ~/Mail/procmail.log

Enable execSSH (only in PRO, as well as file type fields):

nano /home/totum/totum-mit/Conf.php

Ctrl + W type 'inner', enter and replace 'inner' with true (true without quotes!)

OR! From the system as the totum user:

su totum

crontab -e

* * * * * fetchmail -s && find ~/Mail/data/ -type f -mtime +3 -exec rm {} \; && tail -n 1000 ~/Mail/procmail.log > ~/Mail/procmail.log.tmp && mv ~/Mail/procmail.log.tmp ~/Mail/procmail.log

Remember, the last line of crontab must always remain empty

It is not recommended to connect to the main email that receives emails from the outside world. Use sorting and forwarding rules on your email server and forward emails that meet the conditions to a technical email. And from there, retrieve the data into the database!