Monit and Go Applications

For those not aware of it Monit (http://mmonit.com/monit/) is an open source application for process monitoring on unix, well actually monit says it better....

"Monit is a free open source utility for managing and monitoring, processes, programs, files, directories and filesystems on a UNIX system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations."

Why use monit

One of the benefits of using a server like IIS or Apache is that even if your application crashes then the chances are that Apache or IIS is protected and when the next request comes in then your hosted application will re-spawn.  This also means that you do not generally have to wire up any startup code for when the server itself starts.

When you create a go application which is directly serving web requests then you are responsible for starting and keeping the application up.  If for some reason the process crashes or your server reboots then you have to log on the server and restart it or script this to happen automatically.

Monit manages just this behaviour, it also does a lot more such as alerting you at various configurable states.  In this tutorial I am just going to describe how to use monit to manage the startup and stayup for your go application but once you have the basics running you can experiment with additional configuration.

Wrapping your Go Application

Monit works by monitoring the pid (process identification file) file for your application, because Go does not automatically create these you either have to explicitly add this to your startup code.  In my humble opinion this is not so desirable as we want to keep our code cross platform capable without having to write extra nastiness just for the unix environment and also we can never be sure that the pid file directory is going to be the same.  Monit also asks you to declare startup and stop commands, for this reason I created a simple batch file.

 

#!/bin/bash

 case $1 in
    start)
       exec $(cd /var/go/go_application_root; ./application_executable &> log.txt &);
       sleep 5;
       echo $(pgrep application_executable) > /var/run/application_executable.pid
       ;;
     stop)
       kill -9 $(cat /var/run/application_executable.pid) ;;
     *)
       echo "usage: application_executable {start|stop}" ;;
 esac
 exit 0

So what is going on in this file? 

It's a basic bash script which expects one parameter with a value of either start or stop, we are staring the go application application_executable which resides in the same directory as our starup script and piping the output to the file log.txt, note the use of & this tells bash to run the command in the background.  The next line (echo $(pgrep documents_prod) > /var/run/documentserver.pid) we are echoing pid of the documents_prod process to the lock file which monit will monitor.  

I know you can get the pid by using the bash command $$ and $! but I did not have much success with this I was constantly getting a pid which was +/- 1 incorrect.  

The line sleep 5 is also important as before we can pgrep the process id we need to ensure that it has started.

To stop the process we are calling the standard kill comand and using the contents of the pid file we created earlier which contains the pid of the running process.

Note: the file needs to be executable so chmod 755 ./filename.sh

Setting up our environment

I use ubuntu but please feel free to substitute these commands with your favourite package manager.

sudo apt-get install monit

This command will install and start monit running as a daemon, monit has various options that can be enabled such as a http server for checking status, most are disabled as default.  We are only going to configure our mail server for receiving alerts and also set up the config for our go application.  For any other options please see the monit website.

Configuring Monit

First we need to add our mail server to the monit config, we are going to use a local smtp account but you can use anything you choose to

Edit the file /etc/monit/monitrc using your favourite editor, i'm going to use Pico because I could never figure Vi (let the flaming begin)...

sudo pico /etc/monit/monitrc

Find the section which starts with

## Set the list of mail servers for alert delivery. Multiple servers may be

## specified using a comma separator. If the first mail server fails, Monit

# will use the second mail server in the list and so on. By default Monit uses

# port 25 - it is possible to override this with the PORT option.

#

# set mailserver mail.bar.baz, 

and change set mailserver mail.bar.baz to ...

set mailserver localhost

This will send error messages via the local mail server, if you dont have a mail server you can install one by sudo apt-get install sendmail, be sure that you have inbound port 25 blocked by your firewall or you are opening a spam server.

Now lets set the address to send alerts to, add a line beneath the below section in the monit config.

## You can set alert recipients whom will receive alerts if/when a

## service defined in this file has errors. Alerts may be restricted on

## events by using a filter as in the second example below.

#

# set alert sysadm@foo.bar                       # receive all alerts

set alert nic.jackson@myaddress.com { nonexist } # receive only when service is stopped or restarted

Now set the mail from this may be required dependant upon your sendmail setup, edit the below section.

## You can override this message format or parts of it, such as subject

## or sender using the MAIL-FORMAT statement. Macros such as $DATE, etc.

## are expanded at runtime. For example, to override the sender, use:

#

set mail-format { from: nic.jackson@myaddress.com }

save this file "ctrl x"

Now we can create our monitoring config, for convenience separate config files can be placed in /etc/monit/conf.d/ so lets create our file here

check process documents_prod with pidfile /var/run/application_executable.pid

       start program = "/var/go/go_application_root/application_executable.sh start"

       stop program = "/var/go/go_application_root/application_executable.sh stop"

if 5 restarts within 5 cycles then alert

Reload Monit

You can now tell monit to reload your config with monit reload, next time monit runs its checks your process will be started if not already running.  For debugging check out the monit log in /var/log/monit.log

Good luck,

Nic