Icinga2

From philcrump.co.uk
Jump to: navigation, search

I use Icinga2 for monitoring my own services, as well as those I maintain at Telemetricor.

Using Pushover

/etc/icinga2/conf.d/notifications.conf

apply Notification "pushover-notify" to Service {
  command = "pushover-service-notify"
 
  user_groups = host.vars.notification.mail.groups
  types = [ Problem ]
 
  assign where service.vars.pushover-notify
}
apply Notification "pushover-warn" to Service {
  command = "pushover-service-warn"
 
  user_groups = host.vars.notification.mail.groups
  types = [ Problem ]
 
  assign where service.vars.pushover-warn
}
apply Notification "pushover-alert" to Service {
  command = "pushover-service-alert"
 
  user_groups = host.vars.notification.mail.groups
  types = [ Problem ]
 
  assign where service.vars.pushover-alert
}

/etc/icinga2/conf.d/commands.conf

object NotificationCommand "pushover-service-notify" {
  import "plugin-notification-command"
 
  env = {
    PUSHOVER_PRIORITY = 0
    PUSHOVER_USER = "$user.vars.pushover-api-user$"
    PUSHOVER_TOKEN = "$user.vars.pushover-api-key$"
    PUSHOVER_TITLE = "$service.display_name$: $service.state$"
    PUSHOVER_MESSAGE = "$service.output$"
  }
 
  command = [ "/usr/bin/notify_by_pushover.sh" ]
}
object NotificationCommand "pushover-service-warn" {
  import "plugin-notification-command"
 
  env = {
    PUSHOVER_PRIORITY = 1
    PUSHOVER_USER = "$user.vars.pushover-api-user$"
    PUSHOVER_TOKEN = "$user.vars.pushover-api-key$"
    PUSHOVER_TITLE = "$service.display_name$: $service.state$"
    PUSHOVER_MESSAGE = "$service.output$"
  }
 
  command = [ "/usr/bin/notify_by_pushover.sh" ]
}
object NotificationCommand "pushover-service-alert" {
  import "plugin-notification-command"
 
  env = {
    PUSHOVER_PRIORITY = 2
    PUSHOVER_USER = "$user.vars.pushover-api-user$"
    PUSHOVER_TOKEN = "$user.vars.pushover-api-key$"
    PUSHOVER_TITLE = "$service.display_name$: $service.state$"
    PUSHOVER_MESSAGE = "$service.output$"
  }
 
  command = [ "/usr/bin/notify_by_pushover.sh" ]
}

Example Service:

object Service "http-philcrump" {
  host_name = "philcrump-crowther"
  check_command = "http"
  vars.http_vhost = "www.philcrump.co.uk"
  vars.pushover-warn = true
}

Issues I've faced

Outgoing Notifications are not received by some Mail hosts

Notifications were not being received on Telemetricor email accounts (hosted by UK2), however they were received correctly and reliably by gmail accounts, and my own mail server.

The icinga box is configured to send it's own email, so that it can get notifications out if my Primary SMTP goes down. (Helpful given this is one of the services it monitors.)

In /var/log/mail.log:

Jan  9 12:34:06 baran postfix/smtp[5592]: 0906B1A649: to=<xxxxx@telemetricor.com>, relay=mx.telemetricor.com.cust.a.hostedemail.com[216.40.42.4]:25, delay=0.7, delays=0.01/0.01/0.68/0, dsn=5.5.2, status=bounced (host mx.telemetricor.com.cust.a.hostedemail.com[216.40.42.4] refused to talk to me: 504 5.5.2 <localhost>: Helo command rejected: need fully-qualified hostname)

The issue here was that at the start of an SMTP Handshake, the client sends HELO [client hostname]. Some servers (although UK2 is so far the only I've come across), will reject the Handshake if the hostname is not an FQDN.

The offending line was in /etc/postfix/main.cf:

myhostname = localhost

This required changing to the FQDN of the Icinga server:

myhostname = icinga.philcrump.co.uk

Thereafter notifications were successfully received in the UK2 Accounts.

Icinga Web 2 - Hosts without IP Addresses crash Web Interface

I monitor several Hosts/Services that are not IP Services, and therefore don't have an IP Address. Not putting the address line in the Host definition however causes an error to be displayed when trying to view a child Service on Icinga Web 2, preventing access to the Service information page.

Filed as Icinga Bug #7918

I've got around this by giving a false IP (0.0.0.0), and then switching off the ping and hostalive checks.

Adding a condition on host.vars.unpingable in /etc/icinga2/conf.d/services.conf for the ping4 Service rule:

apply Service "ping4" {
  import "generic-service"
 
  check_command = "ping4"
 
  assign where host.address && host.vars.unpingable != true
}

Then by adding this variable to the relevant Host definition, we disable the ping check, allowing us to use a fake IP, I've also used this where the remote network is not friendly to ICMP and I can rely on Service checks to inform me of Host status:

object Host "example-host" {
  address = "0.0.0.0"
  check_command = "dummy"
  vars.unpingable = true
  ...
}

The web interface is now happy, as it has an IP address to put in the SQL query.