Welcome to NBlog, the NoticeBored blog

Like the finer things in life, quality trumps quantity.

Jan 18, 2017

Nuts to risk management

In all sorts of contexts, we often focus or obsess about the wrong things. We fear flying more than the road trip to/from the airport. We are terrified of cancer, but tolerate obesity and do our level best to ignore heart disease. We are petrified of driverless vehicles, while downplaying their safety, economic and social advantages relative to human-driven vehicles. "Foreigners" (especially Russians and Chinese, it seems) and terrorists are clearly out to get us, more so than our own governments, our friends and relatives, and just about everyone else for that matter, including ourselves! Gun-ownership is fine but guns are dangerous. Cellphone masts are patently evil (especially in/near schools), whereas the cellphones clasped to our heads or pocketed next to our crown jewels are good ...

Seems to me this is a widespread issue with risk management in general, including but going well beyond information risk management. It’s a sad indictment of our profession, too. We’re often wrong when we identify and prioritize the risks, or the constituent threats, vulnerabilities and impacts. Hence we’re often pushing the wrong controls, or ‘playing it safe’ by pushing and wasting effort & money on unnecessary controls when we perhaps ought to be focusing more on the truly necessary/critical ones. We promote IDS/IPS and SIEM and a million other shiny high-tech controls while neglecting backups, policies, awareness and training - the basics, fundamentals really.

So, on that premise, what is the alternative, the antidote? What if anything can we do about it? Here are nine non-exclusive approaches that occur to me, with a few cynical comments as I mull this over:
  1. Data-driven risk management, using data on actual incidents to assess risks and prioritize them more rationally (hmmm, this implies a historical bias, and a bias towards recognized/known incidents. And we probably lack sufficient data anyway. Oh and the Global Financial Crisis demonstrated how easy it is to get carried away by our own presumed competence)
  2. Find/invent better risk management methods … (good luck!)
  3.  … and apply the methods more rigorously (doing a fundamentally broken process well doesn’t get us anywhere much though, except that it perhaps encourages us to improve systematically and acknowledge the limitations)
  4. Baseline or standardized security: put in place the basic controls that are generally accepted as being necessary (but if we are all deluded about the risks, what help is that? I guess it relieves us of thinking about the basics, leaving more head space for the remainder)
  5. Just do the best we can, but be more realistic about our limitations. Acknowledge that our controls – including risk management as a whole - are fallible, so emphasize incident management, resilience, recovery and contingency approaches (like, errr, reliable off-line backups to recover from ransomware and Windows updates [much the same thing!]. Oh oh.)
  6. Worry or obsess about everything. Control everything. Make security so tight the business squeaks. Be genuinely shocked when something big breaks spectacularly. (Snowden!)
  7. Downplay or ignore this issue. Put head in sand. Close eyes and stick fingers in ears, chanting la-la-la. Pretend that we’re on top of things, hoping that someone else will fix this (nonexistent) issue before we get hit hard. (Prepare excuses and maintain Curriculum Vitae)
  8. Take ourselves out of the loop. Hand this over to the robots. Hope that AI trumps wetware. (Hinson tip: this is the way of the future, like it or not)
  9. Make a serious, conscious effort to identify and counteract human biases, prejudices and blind-spots, including our own. Become more self-aware. (Think on!)

And before anyone comments, I appreciate that, as an infosec pro, I too am part of the problem but at least I'm contemplating solutions.


PS This piece was prompted by a BBC article about squirrels being more of a threat to the US critical national infrastructure than terrorists and a troll on the security metrics mailing list who plaintively insists he has a cunning new risk management method but refuses to tell us anything substantive about it.

Jan 13, 2017

Earthquake alerting app

Quake map courtesy of  GeoNet.co.nz
An idea came to me in bed this morning: smartphones have the capability to identify and measure phone movements in three dimensions, thanks to their built-in positional sensors. So, with the appropriate app, my smartphone sat on my bedside table ought to be able to recognise when it is being shaken about violently in the middle of the night, and alert me to an earthquake in progress.

I distinctly remember waking up a few minutes past midnight during the big Kaikoura earthquake last November, and being utterly befuddled about what was happening. I was still fast asleep when the inital jolt came, and it took me a little while to stir by which time we were into a period of gentler but strong rolling movements, enough to set the bedroom light swinging. I wasn't awake enough to react, as I should have done, doing the 'drop, cover, hold' thing. Instead, I just laid there in a bit of a daze, blinking up at the pendulous light and wondering what was going on.

If my phone had noticed the quake and started chanting "Drop!  Cover!  Hold" at me, maybe that would have had the right effect? Perhaps a message along the lines of "Head for the hills!" would be a worthwhile prompt for those in the tsunami zones? 

I wonder if anyone has already produced such an app. If not, I'd love to see someone take this idea forward, perhaps do some trials to clarify the specification and make it happen. Clearly, the app would need to be able to distinguish quakes from conventional phone use, perhaps using clues such as time of day, long periods of resting, no incoming call or alarm message, and maybe a "False alarm?" button on the screen.

Since it is just an alert, the conventional alert/alarm sounds could be used, or better still we could record our own messages, picking up on the cocktail party effect by having a significant other call our name at the start of the message. For bonus points, the 'head for the hills' thing could even tell me what way to go, using the phone's built-in GPS and mapping capabilities. [And, yes, this is scope creep in action!]

It occurs to me that if such an app were used widely across the country, that would effectively comprise a distributed network of semi-intelligent earthquake sensors, measuring the intensity of the shakes in real time.  It might prompt phone owners to submit subjective earthquake reports, supplementing the scientific sensors and existing web reporting system. [More creeping scope - I could go on ...].

The trouble is, I have no idea how to take this idea forward. I'm not even sure if it's novel or lame, perhaps blindingly obvious to anyone who makes the conceptual connection between smartphones and earthquakes. I'm neither a smartphone programmer nor seismologist, just someone with a brainwave that got me out of bed this morning. I don't know who might be both interested and capable of exploring and developing the idea, perhaps taking it further.

The main reason I'm blabbering on about it here is that I think it might be a patentable invention but I'm not looking to make any money out of this, and I'd much rather see it exploited for the public good than line some entrepreneur's pockets. This is a 'spoiler', deliberately disclosing the invention to prevent it being patented.


Jan 10, 2017

Surveillance: awareness challenge or opportunity?

We're busy preparing February's NoticeBored security awareness module on the topic of surveillance.

As often happens, what we anticipated would be a fairly narrow and specific issue has mushroomed before our very eyes as we've delved into the writing. We're now looking at surveillance on the population by the authorities, by the organization on workers and third parties, by third party organizations on workers, and by individuals on each other ...

The awareness module covers a fascinatingly diverse patchwork of information risks e.g. industrial espionage and intelligence, health and safety, network and physical security monitoring, oversight and supervision, privacy and confidentiality, office security, things, portable devices, artificial intelligence, hacking, malware and more. We've covered all of them separately but this is our chance to bring them together - an awareness story with with a novel perspective.

The news story about a TV presenter's verbal comment allegedly triggering Alexa to order dolls houses through Amazon's Echo thing fell into our lap just a few days ago: if people haven't even bothered to configure their gizmos so they don't order stuff without their confirmation, what's the betting that they don't realize they are being silently monitored 24x7? And it's not just Alexa listening-in, not by a long chalk. Look around you right now: how many microphones and cameras can you spot? And how do we know there aren't more surveillance devices, tucked away inside the high-tech stuff that increasingly surrounds us?

If your security awareness program is tedious and ineffective, perhaps you need to get out more! If that's not an option, get in touch: we've got the boring bits covered (researching and preparing creative copy) leaving you the fun of interacting with your fellow employees, delivering and emphasizing the key messages. Surveillance is one of ~60 topics in our bulging portfolio, all designed to intrigue, inform and motivate workers to value and protect information.

Gary (Gary@isect.com)

Dec 31, 2016

Internet security awareness

We've just delivered our first awareness module for 2017 with a few brief hours left until the new year.  

Updating the awareness module on Internet security turned out to be a mammoth task: we've basically rewritten it from scratch, such is the pace of change in this area. We could probably have continued writing for another month, in which time I'm quite sure further issues would have emerged ... so we had to call a halt to the writing in order to hit our self-imposed delivery deadline. We can always come back later for another bite at the cherry and, to be fair, most security awareness topics touch on the Internet in some fashion.

"Fake news" is a recurring theme in the materials, picking up on media reports following the US presidential election. Today, we completed the final piece for the module, the awareness newsletter, drawing on a US CERT - DHS - FBI alert about GRIZZLY STEPPE published yesterday. Two Russian hacking groups used Remote Access Trojans to compromise systems belonging to US political parties and perhaps other targets. Russian interference in the US elections, through circulating propaganda and directly attacking political systems, marks a new phase in cyberwarfare, in effect using information as an offensive military weapon against a superpower. As noted in the newsletter, however, the US, UK and other governments have conducted Internet surveillance for years, and espionage predates the WWW by millennia. Whether you see sinister implications for civil liberties, or a legitimate use of modern technology to fight terrorism and foreign interference in the domestic economy, is just a matter of perspective. The fact remains that securing information on the Internet is an arduous, costly task with serious implications for privacy and commercial confidentiality, as well as politics and the economy.

Against that backdrop, awareness advice to patch systems, beware phishing links, use firewalls and so forth seems trivial but the truth is that from the outside the average corporation looks more like a colander than a shield. Without a decent level of security awareness throughout the organization, the fanciest of high-tech security technologies are worse than useless in the sense that they give a misleading impression of protection. They help, sure, but they are not sufficient.

Talking of technology, I am dismayed that so few organizations make any effort to train their technical staff on information security matters. How exactly are they supposed to pick up this stuff - Vulcan mind-meld perhaps, or some sort of virtual osmosis? If you think employing IT graduates magically lets you off the hook, take a look at the curricula or talk to your IT people about how much information security their courses actually covered. Go on, I dare you, ask them about the lectures on propaganda/fake news and surveillance! There's a good chance some of your IT pros left uni before the Internet was even invented.


Dec 26, 2016

What worries CEOs?

According to KPMG, the answer is ...

I've picked out just one of a plethora of gaudily-colored horizontal bar charts from  KPMG's 2016 Global CEO Outlook, which I encourage you to download and read. The Now or Never report blends insightful analysis with survey data and anecdotal comments. It's well-written, glossy and eminently readable.

The reason I chose to illustrate this blog piece with that particular chart is that it confirms what I already thought about risks of concern - in other words, I openly admit my 'confirmation bias'. The same could be said for most of the rest of the charts, and indeed the report as a whole, which at least partially explains why I'm blabbering on about it here. It was on-topic for me, interesting enough to read, consider ... and then pick holes in.

Even if you disagree with the report's findings, or have little to no interest whatsoever in the subject matter, it's still worth reading in respect of the manner in which the survey was designed, conducted and reported.

The survey methodology - or rather the surveyed population - is briefly outlined thus in the report:
"The survey data published in this report is based on a survey of 1,268 chief executives from Australia, China, France, Germany, India, Italy, Japan, Spain, the UK and the US. The findings from CEOs in an 18 additional regions and countries can be found in the Appendix section of this report. Eleven key industries are represented, including automotive, banking, infrastructure, insurance, investment management, life sciences, manufacturing, retail/ consumer markets, technology, and energy/utilities and telecom. Two hundred and seventy-five CEOs came from companies with revenues between US$500 million and US$999 million, 595 from companies with revenues from US$1 billion to US$9.9 billion, and 398 from companies with revenues of US$10 billion or more. Eight hundred and ninety-three CEOs came from public companies and 375 from private companies. The survey was conducted during 15 March and 29 April of 2016."  [page 39]
Hmmm, that one paragraph is about all they have to say about the survey, aside from calling out various KPMG people and respondents by name throughout the report (call me a cynic, but I suspect the names were not drawn at random out of a hat!). 1,268 may be a reasonably large size sample for this kind of survey but nevertheless it is an almost infinitesimal proportion of all CEOs in the world. 

Bearing that in mind, take a fresh look at, say, the colorful bar graph above. We can see what it tells us, but what is missing?

For a start, we aren't told how many respondents answered the particular question. Did all 1,268 of the CEO's surveyed answer, I wonder? More contentiously, did any of them answer it? The actual number of responses to that question probably lies between those extremes, quite likely towards the top end, but we simply don't know from the report.

That issue, in turn, has implications on how we interpret the numbers. Are those differences statistically significant or within the margin of error for this survey question? 24-30% is actually quite a narrow range, even for a fairly large scientifically-designed survey. The 1 or 2% differences between the bars, and hence the priority/sequence of their presentation, could easily be an artifact of the survey. We just don't know.

Furthermore, there is no report appendix listing the actual survey questions and (I guess) the multiple-choice answers permitted. Were respondents asked to choose between a list of those 5 particular risks, or were there others listed that didn't even make it into the report? How were those options selected and worded? What, precisely, is "cyber security risk" for instance? Was there a 'something else' option, allowing them to elaborate on whatever burning issues keep them up at night - and if so, how were those free-text responses handled? Were respondents able to select multiple answers, or required to rank them in some way? Were respondents induced, fooled or coerced into answering in a particular way, for example by dint of the context or preamble to the survey or this question, or by the fact that KPMG was asking? Did the bar chart even arise from 'a question', as such, or was it a synthesis of selected data plus some analyst's fertile imagination? Again, we simply don't know from the report.

On a broader note, there is no explanation of how those 1,268 survey respondents were selected (perhaps KPMG clients, or maybe a pool of Forbes Insight contacts?) nor how the survey was conducted (not even the fundamental approach such as on-line or in-person). Were any respondents de-selected i.e. disqualified outright, or their responses to this hypothetical question discounted for some reason? 

Talking of Forbes Insight (whose pretentious name I only spotted when I re-read the report while blabbering on about it here), this is what I've found out about them so far, a byline from the Forbes website:

That's hardly a detailed or convincing statement as to their competence and suitability for such a survey. They might as well have said, lamely. "We do this kind of stuff". Digging just a bit deeper, I find that Forbes is heavily into public relations and overtly self-promotional cobblers like this:

I suspect KPMG paid Forbes Insight either to conduct the entire survey, or maybe to access their mailing list of 'pre-qualified' (pre-duped, softened-up) potential respondents. Note, yet again, that I'm speculating due to the dearth of information in the report itself. There's a lot more omitted than stated ... which perhaps explains the 8 month delay between completing the survey in April and issuing the report in December. The statistical analysis and report-writing would have taken, I guess, about 1-4 months whereas it takes longer to spin a line. 

So much for the forward-looking 2017-2020 focus of the report: the CEO's were surveyed months ago, prior to Brexit (June) and Trump (December). Hey, KPMG, haven't you noiticed? Big Things have happened this year.

Oh and another thing. Take a closer look at the survey sample. Why did they only apparently survey respondents from "Australia, China, France, Germany, India, Italy, Japan, Spain, the UK and the US"? What about the other 200 odd countries of the world? And how come the report discusses locations such as Korea, Ireland and East Africa if there were not (according to the report) any respondents from those areas?  Odd, that, very odd. [Cue faint whiff of rodent].

Clearly, I am a cynic. Take my inane comments with a pinch of salt. Argue with or berate me, if you will. But please at least think about what you are reading, both here on the blog and in surveys of the kind I've critiqued. FWIW this is one of the better ones.


Dec 23, 2016

Tenable Global Cybersecurity Assurance Report Card critique

The "2017 Global Cybersecurity Assurance Report Card" reports on a web-based survey of 700 IT security people from mid to large organizations. The survey was sponsored by Tenable and conducted by CyberEdge.  It makes a reasonable job of measuring concerns and opinions within the narrow constraints of the specific questions posed. For example here's one of the survey questions:

The 5 categories of response (1-5) and the 7 listed items (a-g) constitutes a Likert scale, a commonplace opinion survey approach with both strengths (primarily low cost) and weaknesses. In respect of just that question, methodological concerns include:
  • The question stem is a little ambiguous, referring to 5 being "highest" when it presumably means "of greatest concern" or whatever.
  • The 1-5 values are ordinals and do not indicate absolute value or relative proportions. A value of 4 does not literally mean "twice as concerned" as a value of 2.  A value of 1 is not "one fifth of the concern" indicated by a 5.
  • The use of just a few discrete categories prevents respondents indicating intermediate or marginal responses (e.g. "Right at the top edge" of any category is indistinguishable from "Right at the lower edge" of the same category).
  • There is a natural bias against selecting the 'extremes' (or rather, the  1 and 5 categories), on top of which the center category is often chosen when the respondent has no opinion or knowledge, especially if the survey is configured to demand a response to every item. This bias leads to a statistically-significant preponderance of the middle category.
  • The question concerns "challenges facing IT security professionals" i.e. not the respondents' own challenges, nor those facing their organizations, but their opinions about an ill-defined/unspecified generic group. Who are these "IT security professionals" anyway? I suspect each survey respondent has their own interpretation.
  • The specific a-g sequence of Likert items can influence the result of this kind of question. The sequence could simply have been randomized for each respondent to eliminate the bias, but the methodology statement does not say so. 
  • Calling the items "challenges" immediately frames the problem space in a particular way: these are problems, things of concern, difficulties, troublesome areas. With slight re-wording, the question could have referred to "interests" or "investments", or even "opportunities [for improvement]" which would probably have influenced the result.
  • The Likert items - the particular "challenges" listed - narrowly constrains the responses, and they too are ambiguously worded. There appears to be no way for survey respondents to identify other "challenges", or or to question the meaning of the items listed, nor to provide explanatory comments amplifying or qualifyng their responses. Item e, for instance, begs questions about the precise meaning of every word: what is 'low' (e.g. is that below some other thing, low on some sort of measurement scale, or lower now than at a previous or later time?), 'security' (IT security, information security, physical security, safety, protection, national security, or something else?), 'awareness' (a general appreciation, alertness, the motivation to behave differently, completion of a training course, self-study, or what?), and yes even 'employees' (meaning everyone on the payroll or just staff or IT users maybe?).  
  • The methodology was (partially) explained, and the survey questions were included in the report along with (some) other basic parameters ... but there is virtually no detail on how the respondents engaged with the study, aside from it being a web survey. Was it on the Tenable website, maybe?Did Tenable send out emails with links to their mailing list, their customers, or some other pre-selected audience? Why did respondents participate? Were they expecting to get something in return? If they self-selected, was it because they have an unusual level of concern about the subject matter, or because they are outspoken critics, or for some other reason?

The report states the results of that question thus:

Although it is not actually stated in the report, I presume the numbers are means of the values selected by respondents, which immediately raises statistical concerns since (as stated above) the values are ordinal numbers. Are means even valid in this case? (I'm no statistician, but I suspect not).

Notice that all the reported values are above 3.3, indicating an overall preponderance of 4's and 5's - in other words, respondents are generally concerned about the 'challenges' listed. That is not unexpected since respondents are described as "IT security professionals" who are generally risk-averse by nature, but nevertheless it is a systematic bias to the method.

Notice also that the scale on figure 10 ranges from 3.00 to 3.90, not 1 to 5. Without further information, we have no way of determining whether the differences between the items are significant. There are no confidence limits or other statistics concerning the range of responses to each item. We aren't even told how many responses there were to each item in the question. Giving two decimal places implies a degree of precision that I suspect may be misleading, but it does rather conveniently allow them to rank the "challenges" in numerical order with no ties.

Oh and notice the legend to figure 10: it and the following text refers glibly to "top challenges" whereas in fact the challenges were chosen by the surveyors, not by respondents. Speaking personally, my "top challenges" are not even listed as options. I am barely even interested in the items listed, except for security awareness of course, and in that I willingly admit extreme bias since I make my living from addressing that "challenge" (except I consider it an opportunity!!).


PS  Despite several mentions of "2017" in the report, I believe we are still in 2016. Or have I been in a coma all year?

Dec 21, 2016

You know you're a geek if ...

... your favourite hot drink is URL grey tea

... Chewbacca, Ytterbium and Hal are the names of just three of your many servers

... your Myers Briggs Personality Type starts with INT - as in integer

... you know what a Myers Briggs Personality Type is, even without Googling it

... assorted acquaintances and relatives genuinely expect you to be fully expert in whatever quaint technologies are currently giving them problems

... you've had the same PC for well over a decade, with 3 different cases, 4 different PSUs, 5 different motherboards, 6 different CPUs and at least a dozen new hard drives, now all solid state (naturally)

... you welcome the moniker 'geek', whereas other lesser beings may think it perjorative

... you spot technical errors in TV portrayals of geeks

.. you care about stuff that others don't even notice

... friends and family ask you to explain your Christmas wish-list in words of one syllable

... you are so well-stocked with batteries that you could open a battery shop

... you read (or indeed write) blog pieces titled "You know you're a geek if ..."

Merry Christmas all.  Have fun playing with your new toys.


ISO/IEC 27004 revised and much improved

A substantially improved version of the security metrics standard ISO/IEC 27004 has just been published.

The standard covers "Information security management ― Monitoring, measurement, analysis and evaluation", a direct reference to clause 9.1. of ISO/IEC 27001 ... in other words, it is primarily about the metrics needed to make management decisions about, and systematically improve, an ISO27k-style Information Security Management System.

These are the main sections:
  1. Rationale - explains the value of measuring stuff e.g. to increase accountability and performance;
  2. Characteristics - what to measure, monitor, analyze and evaluate, when to do it, and who to do it;
  3. Types of measures - performance (efficiency) and effectiveness measures;
  4. Processes - how to develop, implement and use metrics.
Annex B catalogs 35 metrics examples using a typical metrics definition form. These are not exactly shining demonstrations of the art, in fact some of the examples are of poor quality. I'm sure we can come up with a better set of example metrics, and in fact I plan to do so over the coming months, free time permitting. I have in mind documenting a suite of metrics relating to the whole of 27001, including both the management system aspects in the main body of the standard and the information security controls listed in Annex A. Watch this space.

I am pleased, relieved in fact, that the 2009 version of this standard is now consigned to history. It was an academic piece, full of theory and an obsessive focus on the calculation part of measurement - strange, really, given that it is such a simple and inconsequential part of metrics (essentially just 'collect data, run statistical analysis, generate result') compared to the far more important issues of what to measure and why. I honestly feel that its publication retarded rather than advanced the field of security metrics. The new release is much more pragmatic and helpful for those designing, implementing, using and improving ISO27k ISMSs. I commend it to the house.

The new standard is available to purchase from ISO, from ANSI, and no doubt from other official sales outlets too. It costs about $200 (don't shoot the messenger: I wish all the ISO27k standards were available free of charge in order to encourage widespread global adoption and improve the general state of information security but it's ISO that sets the price, not me).


Dec 19, 2016

Online infosec dictionary

ComplianceDictionary.com is an online dictionary of terms defined in various standards, laws, regulations etc., maintained by UCF, the Unified Compliance Framework.

I have a lot of respect for the UCF and have blogged about them before. They systematically collate and analyze a wide variety of laws, regulations and standards, helping clients identify the areas of commonality that equate to both savings and good practice. If a given security control satisfies numerous compliance obligations or expectations, it make business sense to implement it properly, once. It may even qualify as a critical control.

Just in case you are wondering, I have no financial interest in UCF and don't earn any commission from them. I do however admit to being envious of the idea underpinning UCF!

The Compliance Dictionary is essentially a search engine that spews out both informal and formally-defined explanations for information security-related terms. The first term I entered to check it out gave a disappointing but not altogether surprising result: my search on "acountable" led to the following:

That's one informal/uncited reference to a generic definition ("The expectations or requirement to justify actions or decisions") with links from the 'Relationships' diagram to further entries including definitions of "accountability":

Notice that only the last definition has a cited source ... but (at least as far as I'm concerned) 'accountability' is a fundamental concept underpinning information security. It makes the difference between someone simply saying that information is a valuable asset worth protecting, and adequately protecting it in practice in order to avoid being held to account for incidents.

'Responsibility' is another fundamental concept, one that is also formally undefined according to the Compliance Dictionary. 

More surprisingly still, Compliance Dictionary identifies no formal definitions of 'control' ... but I know of at least one definition within the compliance documents that UCF claims to track, namely ISO/IEC 27000:

So, based on this small sample, the Compliance Dictionary is a nice idea that fails in practice. I'm disappointed but not surprised.


Dec 13, 2016

IT security spend as a % of IT budget

According to an article in The Register, Gartner has pointed out that 'proportion of IT budget spent on IT security' is not a good metric.

One can determine any metric's strengths and weaknesses systematically and objectively using the PRAGMATIC method, so here goes:
  • Predictiveness: at a superficial level of analysis, the budget obviously affects the amount that can be spent on, or invested in, anything, hence there is bound to be some relationship between the money spent and the amount achieved ... but that is not a direct, linear relationship (in practice a somewhat vague correlation I suspect). Organizations with tight budget constraints have to spend more carefully, and naturally focus their efforts on optimizing the value they obtain. Furthermore, many would acknowledge the preponderance of snake oil salesmen in the IT security field, hence spending more might even, in some cases, be counterproductive. Score: 50%
  • Relevance: the metric may be relevant to IT security, and to financial management for the organization, but is that enough to score well on this criterion? What of its relevance to information risk and information security, compliance, governance and business continuity? Score: 75%
  • Actionability: at first glance, increasing or decreasing the proportion of the IT budget allocated to IT security would be the obvious response to low or high values of this metric. However, that's not how budgets are normally determined. Conceivably the metric might be one of the factors taken into account in the budget proposal. More likely, management would expect to see a reasoned, rational business case to spend money, not something as crude as a proportion of spend (even if it was presented as a benchmarking comparison relative to other similar organizations - assuming that could be done). Score: 55%
  • Genuineness: does the numeric value of this metric genuinely and straightforwardly reflect the object of measurement? Could it be manipulated, perhaps by someone with a hidden agenda? The metric is generated very simply by dividing one number by another, so there's not much leeway for manipulation ... but can those two numbers - the base data - be trusted? I'm not so sure (more notes below). Furthermore, issues with the true meaning of the metric (the next PRAGMATIC criterion) may be explained away by creative interpretation when presenting the metric, especially if the audience is unfamiliar with broader, more mature concepts such as information risk. Score: 65%
  • Meaning: the meaning of any metric depends on the intended audience. It's a matter of perspective. So who is or are the prime audiences for this metric? For financially- and/or IT-aware managers, the metric seems self-evident. To other general managers, it may also appear meaningful, at face value, but to anyone who digs a little deeper, and most likely to the CISO, ISM or other experts in risk, security, governance, compliance etc. (including, reportedly, the analysts responsible for Gartner's report), the metric is distinctly misleading. There is more to managing information risk than IT security, and anyway the amount of investment in IT security is not necessarily reflected in the results. It is depressingly easy to come up with examples of IT security investments that have not paid off, including some that have failed spectacularly: consider Target, Sony and others for instance. Score: 30%
  • Accuracy: Gartner acknowledged that the metric varies widely between organizations, in the range 1-13%. Does IT security status even vary by such an amount between organizations, let alone vary in accordance with the value of this metric? Possibly, yes, but personally I doubt it. There are other significant concerns over the accuracy (see below). Score: 10% 
  • Timeliness: it takes hardly a moment to calculate this metric provided the base data are available - simply divide the two figures. Obtaining the base figures is straightforward too, assuming the organization captures or reports them as part of the budgetary/financial management. Score: 95% 
  • Independence: could someone (such as an auditor or manager) validate the metric? Yes, checking the calculation is trivial but there is some question about the base data, and how they are determined (see below). Score: 65% 
  • Cost-effectiveness: although the cost of generating this metric is negligible, its value is not strong. Consequently, and especially if compared to many other similar metrics, this one does not generate much net value for the business. Score: 30%
  • Overall PRAGMATIC score = 53% (a simple arithmetic mean of the individual scores)
By all means challenge the thinking, adjust the individual scores and even weight the individual criteria if you feel so inclined. According to my PRAGMATIC analysis, inaccuracy and low net value are the most significant issues with this metric, along with its potential to mislead naive recipients. There are questions about the base values from which the metric is calculated, and about the relationship between the metric and the organization's IT security status, plus still bigger questions about its relevance to information risk and the organization's business objectives. The overall score of 53% is near the bottom of the 'acceptable' scoring range (just above the 50% cutoff point), making this metric barely acceptable - hardly worth considering unless there is no better metric (which I am sure there is - 'IT security maturity' for a single example). 

That said, the metric's PRAGMATIC score could be improved if those and other issues were addressed by redefining it, which I leave as an exercise for the reader.

Gartner clients can obtain the report (cited as "Identifying the Real Information Security Budget" in The Register but identified as "How to Manage and Defend Your Security Budget" on the Gartner page to which they linked) for free. Others must part with $195 for the pleasure of reading the 10-page report if this blog piece and a moment's quiet reflection was insufficient. As to whether the $195 represents good value for money, and whether it legitimately qualifies as 'IT security expenditure', I leave to your discretion. I'll simply point out that it prompted the journalist to comment on, and then me to scratch beneath the surface of, what turns out to be a lame metric.