Here’s a few Splunk queries I’ve used to supply some data for a dashboard I used to manage a MongoDB Cluster.

Election events

If any MongoDB elections happen at 3AM on a Wednesday night I want to know about it. This query, added to a single value panel allows me to do this easily…

host=mongo* source=/var/log/mongo*.log "Starting an election" | stats count

Rollbacks

I also want to know about any rollbacks than happen during an election…

host=mongo* source=/var/log/mongo*.log "beginning rollback" | stats count

Log message with severity ERROR

Count log messages with ERROR severity…

host=mongo* source="/var/log/mongodb/*.log" | rex "(?<timestamp>^\d\d\d\d-\d\d-\d\dT\d\d:\d\d:\d\d.\d\d\d\+\d\d\d\d) (?<severity>.) (?<component>\S*) "| where severity=E | stats count

Chunk moves initiated

Have any chunks moved…

host=mongo* source="/var/log/mongodb/*.log" "moving chunk" | stats count

State changes

How many states changes, i.e. PRIMARY -> SECONDARY in period…

host=mongo* "is now in state" | stats count