A recent consulting assignment had me creating a comprehensive, 300 slide course on IT risk management. It was an interesting exercise, and gave me the chance to learn a lot about risk management. I am also exceedingly glad it’s done!

During the development of the course, I worked with a number of the vendors of IT risk management software tools, many of whom graciously allowed me to use screen shots to illustrate various aspects of the risk management process.

One of the things that struck me as I was building the course, and looking for screen shots to illustrate things like threat modeling, was how little functionality there is generally in this area. The IT risk management tools as a category provide a lot of functionality in the area of information gathering (modeling the organization structure, gathering information about controls in place via assessments or via API’s or direct connections). In some cases the tools will calculate risks purely on the basis of the existence and strength of controls (as evaluated). The tools then apply proprietary formulas to decide how much risk exists if a best practices control is lacking on a key asset. In essence, ignoring things like threat modeling, threat capability, loss event frequency, etc.. Many also ignore probable loss magnitude (or impact), and loss frequency.

If you think about some of the formulas and approaches to risk analysis that are drilled into every CISSP candidate…

Annual rate of occurrence (ARO) = number of times the threat can be expected to occur, affecting the asset, annually

Probable loss magnitude (PLM) = expected loss per occ

urrence

Annual loss expectancy (ALE) = ARO x PLM

… it seems like there’s a disconnect between the functionality in popular IT risk management tools, and the way that IT security practitioners are taught to measure risk. In looking at ten or so different IT risk management tools, I saw just a couple that had any functionality in the area of threat identification and modeling. And the functionality was pretty limited.

Risk analysis still seems to be more art than science, with qualitative versus quantitative debates, and multiple methodologies and frameworks (FIRM, FAIR, Octave, FRAP, and so on) available. Most of these frameworks assess and analyze risk differently, and there is lots of variability on basic things like terms and definitions. Few of the IT-GRC products map to these frameworks, and some of the products use a model that assesses the presence or absence of controls, and the maturity of controls, to determine the level of risk…a sort of best practices approach to gauging risk. The algorithms used to calculate risk are almost always proprietary. Relatively few of the IT-GRC products provide any real threat modeling capability.

My guess is that practitioners using IT risk management tools still end up doing a whiteboard exercise to quantify risk (or to assign a qualitative high-medium-low measurement).

I would love to hear from any users who struggling with this, and who use an ITRM tool. Understanding how you gather data for risk analysis, how much of the data you gather can be done by automated means, and how you then use the information gathered to conduct a risk analysis would be very interesting.

Also, I will post a link and blog entry on the risk management course when it is made commercially available, likely in 6-8 weeks.