Jamie Thingelstad

It was a beautiful morning to walk over to Minnehaha Creek and watch the Twin Cities Marathon.

I’m a big fan of graphs, and as such am a big fan of Cacti. I’ve used it at work and at home. It’s a wonderfully powerful, and ridiculously complicated front-end to RRDTool, which is also wonderfully powerful and ridiculously complicated. I’ve used Cacti to graph hundreds of servers, the temperature in my house, heat collected from solar panels and Twitter followers.

By default Cacti runs a poller every 5 minutes to collect data. Cacti gets very unhappy if the time required to run the poller exceeds the 5 minute interval. You will get blank data gaps and there aren’t any alarms that go off when this happens. Polling times can also vary without any changes in Cacti. If you are polling an external service and it gets slow, that could spike your polling times up in a terrible way.

I decided to write a quick script to catch this. Rather than integrate it into Cacti’s poller I made it completely separate. Save this script to a shell script and add it to /etc/cron.hourly and you can rest easily knowing your Cacti poller is healthy.

#!/bin/bash
#
# by Jamie Thingelstad

# Where is the log?
CACTI_LOG=/usr/local/cacti/log/cacti.log

# Sample up how many lines?
CACTI_SAMPLE=24

# Warning level for 5 minute poll interval, in seconds.
# Nonsensical to set this higher than 300.
WARN_AT=240

# Where is mail?
MAIL=/usr/bin/mail

# Where to send warning? What subject?
ERR_TO="bad-things-happening@youremail.com"
ERR_SUBJECT="[Cacti] Cacti Poller time is too high"

# Scan the cacti log and see how things are
# Grep for STATS \
# Get the last CACTI_SAMPLE samples \
# Find the duration field (cut) \
# Remove the label (cut) \
# Strip the decimal points, make it an integer (cut) \
# Add the total of all the samples and count the number of samples (awk) \
#   - this step is needed because the file may have < CACTI_SAMPLES in it
POLL_DATA=$(grep "SYSTEM STATS" $CACTI_LOG | \
            tail -$CACTI_SAMPLE | \
            cut -d " " -f 7 | \
            cut -d ":" -f 2 | \
            cut -d "." -f 1 | \
            awk '{ sum += $1; count++ } END { print sum, count }')

# Get the two variables we want
TOTAL_TIME=$(echo $POLL_DATA | awk '{print $1}')
TOTAL_COUNT=$(echo $POLL_DATA | awk '{print $2}')

# Figure out the allowed time as a multiple of WARN_AT
AVERAGE_TIME=$(($TOTAL_TIME / $TOTAL_COUNT))

# Check how long it took and warn if too long, in seconds
if [ "$AVERAGE_TIME" -gt "$WARN_AT" ]; then
    # Polling is exceeded limit, send warning
    ERR_MESSAGE="Polling time for cacti is dangerously high. Last $TOTAL_COUNT polling periods averaged $AVERAGE_TIME seconds, higher than $WARN_AT second threshold."
    echo $ERR_MESSAGE | $MAIL -s "$ERR_SUBJECT" "$ERR_TO" >/dev/null 2>/dev/null
fi

Some notes on what this script is doing:

• Sampling 24 lines means 24 five-minute polling periods, in other words, the last 24 * 5 minutes, or 2 hours.

• The script will calculate the average of the last 24 (or fewer) samples and alarm if it is over threshold. One sample over the threshold won’t matter since it uses the average.

• The awk complexity is used to deal with a new log file that has fewer than CACTI_SAMPLE events.

Good CaringBridge board meeting this morning with Sona Mehring Lars Leafblad and more. Proud to be able to work with such a great org.

Tomorrow Tammy and I start our 30 day vegan challenge! I’m a little concerned I have no idea what to eat, but I think it will be fun!

I had my iPad stolen out of my hands tonight while waiting for the 8:50pm 4K bus at Lake and Lyndale.

We had a fun 8thBridge outing this afternoon at Bryant Lake Bowl. After that a bunch of us went over to moto-i and had some dinner and then a final beer at Muddy Waters. Everyone else left by cab and I walked over to the bus stop to catch the 4K home. I was about 15 minutes early so I got my iPad out and started reading The Selfish Gene, my book club book.

After a few minutes a guy walked up next to me and asked me for the time. I didn’t hear him at first and then he asked again so I told him the time. I didn’t think anything of it, but it seems after the fact that I was the mark at that point. Two other people that were at the bus stop mentioned that he looked suspicious and after he talked to me he had gone back and talked to another guy.

The 4K was still a few minutes away when the 4F started to approach. That wasn’t my bus so I just kept reading, and right as the bus approached a different person approached quickly behind me from my right, snatched my iPad with both hands and forcefully pulled it from me. He then ran very fast northbound on Lyndale. It happened in about 1 second. I didn’t get any look at him, other than to know he was shorter then me, likely 5 foot 6 inches. He was a black male in his twenties.

Two people waiting at the stop highlighted at this time they thought something was up. I was completely clueless. The driver on the 4F was super nice and called the transit police for me. I immediately sent a remote wipe command via MobileMe. About 15 minutes later MetroTransit Police called me and took all the details including the serial number of the iPad.

I seriously doubt that the iPad will be recovered. I have no idea what they will even do with it since I have a passcode and the device will certainly be locked. It is a WiFi device so it won’t be online, and if and when it does it will be wiped. I’m bummed to have lost the device, bummed to have been robbed and also just annoyed by the whole situation. On the very good side I didn’t even get so much as pushed or shoved. The guy snatched it and ran. I thought about pursuing but he was clearly faster than I, and even if I did catch him what would I do then.

The MetroTransit police officer I talked to was nice. I asked him if they ever found these things and he said the fact the 4F was approaching was a good thing. The buses, he said, have cameras all over them and they are going to pull the relevant video and see if they can find anything.

First time I’ve ever been robbed. Happily my wallet was still in my pocket, I still had my iPhone in my pocket and nobody threatened me. Doesn’t make me feel different about taking the bus, but I probably need to be a bit more “street wise” at night on a local bus as opposed to an express bus with commuters.

Update Oct. 5

I got a call from the MetroTransit Police and they were able to pull the video from the 4F bus that was approaching when the iPad was taken from me. The investigator confirmed that they saw a person matching my description grab the device from me and run away. They were not able to see his face or any other clues to help with identifying the person so it is mostly a dead end.

She said they were going to call around to local pawn shops but beyond that there wasn’t too much to go on. I thanked her for following up, but I don’t hold much hope for recovery.

The fancy built-in cabinetry was put in today for the family room. It looks really great. There is still one part to add, the floating granite shelf that goes a few inches above the white drawers on the bottom (the dark line in the picture is the inset for it). The TV will get mounted in place along with the in-wall speakers for surround sound very soon.

Hoping that the materials for SMWCon are shared for those that can’t attend.

Trying to update Final Cut X but App Store can’t seem to authenticate me. 🙁

Landscapers blew me away with how fast they finished. 2 days and done.

Alright, no more pistachios. Ate about 20 and feel notably “wrong”. Must be a little allergic.

Cube: Time Series Visualization ← This looks incredibly interesting. Cool possibilities.