Move along, nothing to read

Munin is commonly used to graph lots of systems stuff, however it lacks a common piece of functionality: 95th percentile.

The Munin bug tracker has ticket #443 sitting open for 7 years now, asking for this, and proving a not-great patch for it.

I really wanted to add 95th percentile to one of my complicated graphs (4 base variables, and 3 derived variables deep), but I didn't like the above patch either. Reading the Munin source to consider implementing VDEF properly, I noticed an undocumented setting: graph_args_after. It was introduced by ticket #1032, as a way of passing things directly to rrdtool-graph.

Clever use of this variable can pass in ANYTHING else to rrdtool-graph, including VDEF! So without further ado, here's how to put 95th percentile into individual Munin graphs, relatively easily.

# GRAPHNAME is the name of the graph you want to render on.
# VARNAME is the name of the new variable to call the Percentile line.
# DEF_VAR is the name of the CDEF or DEF variable from earlier in your graph definition.
# LEGEND is whatever legend you want to display on the graph for the line.
#   FYI Normal rrdtool escaping rules apply for legend (spaces, pound, slash).
${GRAPHNAME}.graph_args_after \
  VDEF:${VARNAME}=gcdef${DEF_VAR},95,PERCENT \
  LINE1:${VARNAME}\#999999:${LEGEND}:dashes \
  GPRINT:${VARNAME}:\%6.2lf\%s\\j

# Example of the above I'm using
bandwidth1.graph_args_after \
  VDEF:totalperc=gcdeftotal,95,PERCENT \
  LINE1:totalperc\#999999:95th\ Percentile\ (billable\):dashes \
  GPRINT:totalperc:\%6.2lf\%s\\j

Many thanks to logik for this work of brilliance. Posted with permission, and slightly reformatted here.

A stoner, takes a puff of his joint and says, "Hi, I'm a mac!".
The poorly dressed wannabe bank teller beside him says, "... and I'm a PC."

The door nearby blows in and a heavily armed tactical team storms the room,
throwing both of them to the floor, barrels of MP5k's against their skulls.

Someone yells, "AREA CLEAR!"
The lieutenant comes in after them, smoking a cigar, surveying the area.
"I'm Solaris,
the sergeant over there is BSD (You remember your daddy mac?),
the pretty boy with the M14, he's Linux,
and the guy toting the M60... That there is HPUX.
Now, shut the fuck up, both of you.
We've had about enough of your 'Bill and Ted Get a Computer' bullshit.
Keep it up, and we're gonna do the same thing to you that we did to OS2, got it?"

Setting up the storage on my new machine, I just ran into something really interesting, what seems to be deliberate usable and useful, but completely undocumented functionality in the MD RAID layer.

It's possible to create RAID devices with the initial array having 'missing' slots, and then add the devices for those missing slots later. RAID1 lets you have one or more, RAID5 only one, RAID6 one or two, RAID10 up to half of the total. That functionality is documented in both the Documentation/md.txt of the kernel, as well as the manpage for mdadm.

What isn't documented is when you later add devices, how to get them to take up the 'missing' slots, rather than remain as spares. Nothing in md(7), mdadm(8), or Documentation/md.txt. Nothing I tried with mdadm could do it either, leaving only the sysfs interface for the RAID device.

Documentation/md.txt does describe the sysfs interface in detail, but seems to have some omissions and outdated material - the code has moved on, but the documentation hasn't caught up yet.

So, below the jump, I present my small HOWTO on creating a RAID10 with missing devices and how to later add them properly.

( MD with missing devices HOWTO )

For the geeks and security-conscious amongst us, there will be a PGP keysigning event here in Vancouver. September 1st, 7pm. I will be attending, as will a notary on the Gossamer Spider Web of Trust (GSWoT) and an OpenCA Assurer.

Think Coffee Lounge & Bistro
4512 10th Avenue West,
Vancouver, BC V6R2J1
(604) 228-9510

Please see http://lunar.ca/keysigning.htm for further details and instructions of things that must be done before attending.

Please circulate this announcement amongst anybody you know that uses PGP.