Welcome to NBlog, the NoticeBored blog

I may meander but I'm 'exploring', not lost

Jan 10, 2013

PRAGMATIC Security Metric of the Quarter #3

PRAGMATIC Security Metric of the Third Quarter




These are the example information security metrics we have discussed and scored over the past three months, ranked in descending order of their PRAGMATIC scores: 





Example metric P R A G M A T I C Score

Metametrics
96 91 99 92 88 94 89 79 95 91%

Access alert message rate
87 88 94 93 93 94 97 89 79 90%
Asset management maturity 90 95 70 80 90 85 90 85 90 86%
Compliance maturity 90 95 70 80 90 85 90 85 90 86%
Physical security maturity 90 95 70 80 90 85 90 85 90 86%

Thud factor
82 80 60 60 70 45 85 86 84 72%
Business continuity spend 75 92 20 82 95 70 70 70 70 72%
Benford's law 84 30 53 95 11 98 62 98 23 62%
Controls coverage  87 89 65 40 74 35 46 40 30 56%
Homogeneity 67 70 40 59 67 50 33 65 45 55%
Access control matrix status 70 50 60 60 88 25 40 20 40 50%

Unaccounted software licenses
1 1 90 84 1 70 50 81 30 45%

Unauthorized/invalid access count
61 78 33 16 33 0 44 35 33 37%

On that basis, we are happy to announce that the Information Security Metric of the Quarter is <cue drum roll> metametrics meaning a systematic measure of the quality (utility, value, fitness for purpose etc.) of the organization's information security metrics.  

Clearly we have in mind here the PRAGMATIC approach, but in fact other/variant approaches are possible, so long as there is a sensible, rational way of assessing the metrics that makes sense to management.  The key point of the metric, and the reason it scores so highly, is that measuring the quality of the organization's security metrics is an important step enabling management to improve the suite of metrics in a systematic manner, rather than the much more common ad hoc approach.  Better information security metrics, in turn, allows management to get a firm grip on the organization's information security arrangements, bring them under control, and improve them systematically too.  There is in fact a positive feedback loop at play, since better, more reliable and suitable information security arrangements generate better, more reliable and suitable data concerning information security risks and controls, in other words better metrics. 

That said, we fully accept our own obvious bias in this matter.  Having invented and written about the PRAGMATIC approach, we inevitably see metametrics through rose-tinted glasses.  Things may not be quite so rosy from your perspective, and that's fair enough.  But when you have a moment to yourself, take another look at the 13 metrics on the summary table above, plus those covered in the previous 2 quarters (browse back through the blog, or visit the Security Metric of the Quarter #1 and Security Metric of the Quarter #2), and draw your own conclusions.  You probably disagree with us on the scoring of some of the metrics (even in the hypothetical context of an imaginary company).  But, overall, do you accept that this is a reasonably straightforward, sensible way to consider, compare and contrast metrics?  Would you agree that, on the whole, the metrics that score well on the PRAGMATIC scale are better than those that score badly?  Is this discussion about the pros and cons of security metrics, using metametrics, something that you might use back at the ranch?

Bottom line: if we have persuaded you that the PRAGMATIC approach has merit, perhaps even that it might be a valuable addition to your arsenal of security management techniques, read the book for the full nine yards.  This blog is just a taster of what's to come.

Gary & Krag