Send notification at the end of a deployment

Thu Mar 24 15:32:11 CET 2022

I am sending a compressed file with errors via mail in case some appear
(but nothing that is sent in case of successful deployment) with a modified
version of "savelog.LAST.sh" hook.
Find attached.

As gmail allows to sent via smtps to other gmail addresses I use the
following variables for configuration:

FAI_INFOMAIL_SENDERR='true'
FAI_INFOMAIL_SMTPURL='smtps://<GMAILUSER>:<GMAILPASS>@smtp.gmail.com'
FAI_INFOMAIL_TOADDRESS="<GMAILUSER>@gmail.com"

Obviously you need to have "mutt" application installed and you might need
to adjust the following variables to match your needed patterns..
[ -z "${FAI_ERR_PATTERNS}" ] && FAI_ERR_PATTERNS="X_X-X_XX"
[ -z "${FAI_ERR_WARN}" ] && FAI_ERR_WARN="X_X-X_XX"
[ -z "${FAI_ERR_IGNORE}" ] && FAI_ERR_IGNORE="X_X-X_XX"

Am Do., 24. März 2022 um 14:46 Uhr schrieb Romain Pelissier <
romain.pelissier at gmail.com>:

> Hello everybody,
> Do you know a way to execute a script on the server (FAI) side after a
> successful or unsuccessful client's deployment?
> In case of an unsuccessful attempt, is there a way to get some logs and
> attach them to an email for example?
> Thanks for any useful info you can provide about this :)
> Romain
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.uni-koeln.de/pipermail/linux-fai/attachments/20220324/27dd6e18/attachment.html>
-------------- next part --------------
#! /bin/bash
# version: 2019.11.25

# parse all log files for error messages
# print errors and warnings found to error.log
# WARNING: This will only work with english error messages!

errfile=$LOGDIR/error.log

# Define grep patterns. Do not start or end with an empty line!
globalerrorpatterns="error
fail
warn
 bad
bad 
no space
syntax
Couldn't stat
Cannot access
 conflict
is bigger than the limit
did not exist
non existent
not found
couldn't
can't
E: Sorry, broken packages
^E: 
operator expected
ambiguous redirect
No previous regular expression
No such
Device or resource busy
unknown option
[a-z]\+\.log:E:
No candidate version found
segfault
Couldn't find any package whose name or description matched
cannot create
The following packages have unmet dependencies"

globalignorepatterns="[a-z]\+\.log:#
Error: Driver 'pcspkr' is already registered, aborting
: bytes  packets  errors  dropped
:+ error=0
:+ trap error=
task_error_func=
STOP_ON_ERROR=
courier-webadmin
plugins-bad
Enabling conf localized-error-pages
ibwebadmin
kernel-patch-badram
kolab-webadmin
kolabadmin
gstreamer.\+-plugins-really-bad
liberrors.so
liberrors-samba
libsamba-errors
gsambad
libad
libtest-nowarnings-perl
libtest-warn-perl
libclass-errorhandler-perl
zope-ploneerrorreporting
libroxen-errormessage
liberror-perl
libgpg-error-dev
libgpg-error0
Opts:.\+errors=remount
[RT]X packets:
WARNING: unexpected IO-APIC
warned about = ( )
daemon.warn
kern.warn
rw,errors=
Expect some cache
no error
failmsg
RPC call returned error 101
deverror.out
(floppy), sector 0
mount version older than kernel
Can't locate module
Warning only .\+MB will be used.
hostname: Host name lookup failure
I can't tell the difference.
warning, not much extra random data, consider using the -rand option
confC._FILE
Warning: 3 database(s) sources
were not found, (but were created)
removing exim
The home dir you specified already exists.
No Rule for /usr/lib/ispell/default.hash.
/usr/sbin/update-fonts-.\+: warning: absolute path
hostname: Unknown server error
EXT2-fs warning: checktime reached
RPC: sendmsg returned error 101
can't print them to stdout. Define these classes
warning: downgrading
suppress emacs errors
echo Error:
Can't open dependencies file
documents in /usr/doc are no longer supported
if you have both a SCSI and an IDE CD-ROM
Warning: /proc/ide/hd?/settings interface is obsolete, and will be removed soon
Monitoring disabled
Error: only one processor found.
Error Recovery Strategy:
sector 0 does not have an
syslogin_perform_logout: logout() returned an error
grub is not in an XFS filesystem.
grub-install: line 374:
grub-probe: error: Cannot open \`/boot/grub/device.map'
is harmless
not updating .\+ font directory data.
register_serial(): autoconfig failed
Fontconfig error: Cannot load default config file
asking for cache data failed
However, I can not read the target:
Warning: The partition table looks like it was made
task_error=0
task_local_error=0
^info: Trying to set
warning: /usr/lib/X11/fonts
can't read /etc/udev/rules.d/z25_persistent-net.rules
/cow': No such file or directory
Dummy start-stop-daemon called
X: bytes  packets  errors
ACPI Error
ACPI Warning
AE_NOT_FOUND
conflicts with ACPI region
cannot stat \`/etc/modprobe.d/\*.conf'
cdrom: open failed.
libgpg-error
process \`kudzu' used the deprecated sysctl system call
PM: Resume from disk failed
JBD: barrier-based sync failed
aufs: module is from the staging directory, the quality is unknown
warning: linuxlogo stop runlevel arguments (none) do not match
insserv: warning: script .\+ missing LSB tags and overrides
live-premount.\+ If this fails
cannot read table of mounted file systems
error: no alternatives for
ERST: Error Record Serialization Table (ERST) support is initialized
ERST: Table is not found
HEST: Table not found
failed to stat /dev/pts
Failed to connect to socket /var/run/dbus/system_bus_socket
fail to add MMCONFIG information
can't initialize iptables table
can't initialize ip6tables table
Authentication warning overridden
41-warning.sh
PCCT header not found
Download is performed unsandboxed as root as file
update-alternatives: warning: skip creation of
loop: module verification failed: signature
Warning: apt-key output should not be parsed
WARNING: Failed to connect to lvmetad. Falling back to device scanning
Warning: The home dir /var/lib/usbmux you specified
diff: /var/lib/apparmor/profiles/.apparmor.md5sums: No such file or directory
error reporting disabled
Enabling Firmware First mode for corrected errors
errors: 0
 0 errors
Memory Error Correction:
Memory Controller 0 - Channel . Error
IIO RAS/Control Status/Global Errors
update-rc.d: warning: start and stop actions are no longer supported"

# add pattern on some conditions
if [ -n $FAI_ALLOW_UNSIGNED ] ; then
   globalignorepatterns="$globalignorepatterns
WARNING: untrusted versions
WARNING: The following packages cannot be authenticated
Ignoring these trust violations"
fi
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# Here you can define your own patterns. Put one pattern in a line,
# do not create empty lines.
myerrorpatterns="X_X-X_XX"
myignorepatterns="X_X-X_XX"

# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# The main routine

[ -z "${FAI_ERR_PATTERNS}" ] && FAI_ERR_PATTERNS="X_X-X_XX"
[ -z "${FAI_ERR_WARN}" ] && FAI_ERR_WARN="X_X-X_XX"
[ -z "${FAI_ERR_IGNORE}" ] && FAI_ERR_IGNORE="X_X-X_XX"

NORMAL='\033[0m'
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[0;33m'
LIGHTRED='\033[0;91m'

errorpatterns="$globalerrorpatterns
$myerrorpatterns
$FAI_ERR_PATTERNS"
ignorepatterns="$globalignorepatterns
$myignorepatterns
$FAI_ERR_IGNORE"
FAI_ERR_WARN="X_X-X_XX
$FAI_ERR_WARN"

cd "$LOGDIR" || exit 3
##--remove values from <wharever>PW= variables in logfile
#if [ -f "$LOGDIR/variables.log" ];then
#	 sed -i "s#\(^.*PW=\)\(.*$\)#\1XXXX#" "$LOGDIR/variables.log";fi
sed -i 's#\(^.*PW *=\)\(.*$\)#\1XXXX#' "$LOGDIR/"*".log"
sed -i 's#\(^.*Password *= *\)\([^ ]\+\)\(.*$\)#\1XXXX\3#' "$LOGDIR/"*".log"
sed -i 's#\(^.*password *= *\)\([^ ]\+\)\(.*$\)#\1XXXX\3#' "$LOGDIR/"*".log"
sed -i 's#\(^ *Password *: *\)\([^ ]\+\)\(.*$\)#\1XXXX\3#' "$LOGDIR/"*".log"
sed -i 's#\(^ *password *: *\)\([^ ]\+\)\(.*$\)#\1XXXX\3#' "$LOGDIR/"*".log"

if [ -s "$errfile" ]; then
    echo -e "${YELLOW} Errorfile already exists. Aborting. ${NORMAL}" >&2
    exit;fi

grep -i "$errorpatterns" ./*.log | grep -vi "$ignorepatterns" > "$errfile"
if [ X$verbose = X1 ]; then
    egrep -v '^software.log:' $errfile > $LOGDIR/tempfile
    mv "$LOGDIR/tempfile" "$errfile";fi

if [ -s "$errfile" ]; then
	errlines=$(cat "$errfile" | grep -vi "${FAI_ERR_WARN}") #filter warn entries - they are shown in file but not in output	
	if [ -n "$errlines" ];then	
		echo -e "${RED} ERRORS found in log files. See $errfile ${NORMAL}" >&2  
		if [ "${FAI_ACTION}" = "install" ] && [ "$show_errorlog" = "true" ];then echo -en "$LIGHTRED" && echo "$errlines";echo -en "$NORMAL";fi
		[ -z "${IPADDR}" ] && IPADDR=$(ip addr | grep "inet" | grep -v "inet6" | grep "$(ls --ignore=lo /sys/class/net/ )" | sed 's#\(^.*inet \)\(.*\)\(/[0-9]\+ brd.*$\)#\2#')
		[ -z "${MAC}" ] && MAC=$(ip addr | grep "ether" | sed 's#^.*ether\s\+\([0-9a-fA-F:]\+\)\s\+brd.*$#\1#g')
		echo "device info: $HOSTNAME $IPADDR $MAC" #some information for easier troubleshooting

		if [ "${FAI_INFOMAIL_SENDERR}" == "true" ] && [ -n "${FAI_INFOMAIL_SMTPURL}" ] && [ -n "${FAI_INFOMAIL_TOADDRESS}" ];then
			echo "I: send fai error information via mail to: ${FAI_INFOMAIL_TOADDRESS}"
			mutt=$(which mutt) #; [ -z "${mutt}" ] && mutt="$($ROOTCMD which mutt)" && mutt="$ROOTCMD $mutt"
			if [ -z "${mutt}" ];then echo -e "${RED} Cannot sent errologs via mail - missing mutt application! ${NORMAL}" >&2
			else
			logxz="${target}/tmp/$(basename ${LOGDIR}).tar.xz" && cd "$(dirname ${LOGDIR})" && tar -cJf "${logxz}" "./$(basename ${LOGDIR})/"* &>/dev/null
			[ -z "${FAI_INFOMAIL_FROM}" ] && FAI_INFOMAIL_FROM="FAI ${SERVER} <noreply at mail.com>"
			export FAI_INFOMAIL_FROM FAI_INFOMAIL_SMTPURL FAI_INFOMAIL_TOADDRESS
			echo "${mutt} -F /dev/null -e 'set from=$FAI_INFOMAIL_FROM' -e 'set smtp_url=$FAI_INFOMAIL_SMTPURL' -s \"FAI ${FAI_ACTION} ${HOSTNAME%%.*}\" -a \"${logxz#$target}\" -- \"${FAI_INFOMAIL_TOADDRESS}\""
			${mutt} -F /dev/null -e 'set from=$FAI_INFOMAIL_FROM' -e 'set smtp_url=$FAI_INFOMAIL_SMTPURL' -s "FAI ${FAI_ACTION} ${HOSTNAME%%.*}" -a "${logxz#$target}" -- "${FAI_INFOMAIL_TOADDRESS}" << EOM
FAI ${FAI_ACTION} failed at $(date)

--- system infomation --
device: ${HOSTNAME} | $(hostname -f)
network: ${IPADDR} ${MAC}
users: $(users)
os: $(uname -a)

--- error messages ---
${errlines}

--- log details ---
see attached: $(basename ${logxz})

EOM
			r="${?}";[ "${r}" -ne 0 ] && echo -e "${RED} Cannot sent errologs via mail: ${r} ${NORMAL}" >&2		
			[ -n "${logxz}" ] && rm -f "${logxz}";fi;fi

		if [ "${FAI_ACTION}" = "install" ] && [[ ! "$root" =~ "live:CDLABEL" ]];then #do not reboot/halt device on error within network installation even if specified!
			if  [[ "$FAI_FLAGS" =~ reboot ]];then # (nobody knows that an error appeared if we simply reboot/halt)
				read -r -p "Press <RETURN> to reboot (do not remove your removable medium yet).$(echo $'\n ')";
			elif [[ "$FAI_FLAGS" =~ halt ]];then
				read -r -p "Press <RETURN> to shutdown (do not remove your removable medium yet).$(echo $'\n ')";fi
			fi
	else echo -e "${YELLOW} Good enough! Found only warnings in log files.${NORMAL}";fi
else echo -e "${GREEN} Congratulations! No errors found in log files. ${NORMAL}";fi