There is a wide agreement that the benefits of Communities of Practice are very hard to measure. Therefore CoP sponsoring is a difficult sell in corporate environments, on indeed anywhere that resources are managed. Intuitive evidence and personal experience of users suggest that a good CoP creates enormous value, but pinning it down in a financial report is another matter.

 

There have been attempts at it, mainly along three lines:

 

-         Anecdote-driven. Best described theoretically by Richard McDermott [1], it has been extensively used in large industrial CoPs. It relies on capturing and documenting individual instances of value-adding actions directly related to the activity of the CoP, and translating those anecdotes into hard, conservative estimates of gained (or saved) revenue.

-         Indirect measurement. Most popular in development agencies and government bodies [India. Canada]. It relies on estimates of the effects of CoPs through the evolution of the issues they were created to solve: how productivity has increased, how literacy or vaccination have risen since the CoP operates. Documentation may be story-driven or based on polls of users.

-         Activity indexes. As exemplified by the Macuarium set of measurements [4] (aimed mostly at CoP health), it is possible to portray the relevance of a CoP through some of its metrics. Member participation, answers to issues raised, frequency of visit and even direct relevance qualification by users, all give a view of the CoP’s role in processes.

 

None of them is perfect. Anecdote-driven methods miss most of the impact, indirect measurement risks masking the effect of other factors (positive or negative), and activity indexes can conceal non-productivity-related activity.

 

Still, it is hard to improve on them for a number of reasons. Most of them are derived from the way resources have been traditionally managed.

 

-         Human productivity management. The first and foremost is that human productivity in a knowledge-intensive job is extremely hard to measure. When measuring a process we can know how many people we’ve dedicated to it, and how many other resources are involved. But we can’t easily know what is the level of use of the people’s capacity. In most situations we don’t know just how hard that person is working, how intensive is the use of his/her faculties. Is that engineer taking more time to finish the project than he could? Is that doctor operating at maximum quality? How can we tell that a receptionist is more efficient than last week? How do you measure that copy-writer’s latest output?We can’t, because most of our measurement systems treat people as inputs, not work or quality of results. Most of our cost structures are still built (estimated, in truth) on people hired, not on time actually employed.

-         Lack of a baseline. In the same vein, most processes are not described or measured in sufficient detail to appreciate a change in the time used to complete a task within them. In other words, the initial process structure and productivity levels are (usually) not known. Most companies can’t truthfully say they know what their real use of resources and costs are: they rely on estimates (“standard cost”, it is often called). Therefore it is ridiculous to ask for an identification of improvements: we can’t possibly tell the initial situation from the CoP-derived evolution, at least not with objective indicators, because we don’t know the initial situation.

 

This situation has driven many practitioners and theorists to avoid measurement like the plague, and even to ask for its need to be exorcised from the mind of “modern”, “aware”, “abundance-paradigm”- abiding managers. Unsurprisingly they have cut little ice with budget-aware organisations; and what they have gained with these arguments, was soon lost as the sponsor has been replaced with harder-headed types.

 

Still, the fact that something is impossible doesn’t mean that it shouldn’t be attempted. And anyway most purse-holding managers would refuse to be moved by that kind of arguments. So let’s look at ways to overcome that situation and find ourselves some objective indicators that can prove (or disprove) a CoP’s worth in a financial statement.

 

In these last years we’re witnessing the business world’s reaction to the above problems. And they include a couple of things that can be of very serious use:

 

-         Quality management. In particular, Six Sigma methodologies... but also practically every other quality methodology, are useful in this context. They strive to shed light on to what is actually done in the organization, and by whom: they map the processes, the activities and the roles. In fact, it is plain that many of those efforts are no more that wishful thinking pasted onto chaotic realities... but, where they are really practised and management insists on their implementation, they provide a very close proxy for a baseline’s process structure.

-         Workflow or BPM. In other words, the definition and management of the organisation’s processes through automated routines that link each point in the process (role) and notes the execution of each task, passing it on to the next point. For instance, in credit assessment circuits or expense approvals, but also in simpler tasks... and in more complex ones (support issues, letter-writing, customer proposal generation), a “workflow” provides an underlying automation that routes the necessary information from person to person, and meanwhile manages to capture real, objective times of execution: a close proxy for a baseline’s productivity levels.

 

In other words: armed with information on the processes and tasks in which a CoP user is involved, and a real, objective measurement of the user’s productivity, we can start to gather some systematic and objective measurements on productivity evolution.

 

Of course, this will never give us complete information on CoP returns. For three main reasons and a big externality:

 

-         Time-based measurements don’t cover quality. The evolution in a worker’s output quality can’t be mapped this way; it will need a different measurement. Some times (credit assessment; sales proposals) it can be inferred from information available to the company (rates of default; closed sales and profit margin); sometimes it will have to depend on fuzzier customer-satisfaction measurements. Not all of them have easy translation into financial reports.

-         Most CoPs deal on exceptions. In other words, CoP members consult each other on issues that represent deviations from standard procedure. While a worker can be relied upon to carry through the most usual situations, any number of reasons (technical issues, information issues, resource or supplier issues, customer relationship, internal channeling, criteria to be applied...) can cause doubts. These doubts are the food of CoPs, and their solution is their main product. In other words, CoP influence on productivity depends on the frequency of exceptions and their type: they need to be recorded, and baselined (we need to know the standard pre-CoP cost of dealing with them), if we are to measure CoP financial impact. 

-         There are other influences on productivity. The organisation will continually strive to improve; department leaders will work on this, policies will evolve, marketing programmes and training initiatives will roll through; not to mention new IT systems and changes in the market (changes in our customers’ perception of our quality can be due to changes in our competitors’ quality). What part of a worker’s increase in productivity (both efficiency and quality) is due to the CoP and which is due to other factors? Again, we are reduced to estimates and very complicated calculations, or to subjective, poll-based user measurement.

 

And last but not least:

 

-         Not all benefits are aimed internally. Many organizations sponsor CoPs to benefit people or stakeholders outside its boundaries, where they can’t rely on direct objective measurements. Every advocacy group is an example, but also most open CoPs (where members belong to several organisations). Those CoPs are also frequently oriented at producing effects that cannot be sensibly measured in any economic terms (except maybe as substitutes for better-known sources of them), such as literacy increases or health (well-being of chronic patients, innovation in depressed areas). Even when they can, we must rely on external sources or subjective data.

 

Overall, on modern corporations we can get something significant out of a well-built system based on the above mentioned techniques. They can give us very concrete numbers on parts of the CoP impact; and at least we can point at the change in figures, however many caveats about “cross influences” by other factors are added. Another matter is the cost of developing an information system able to extract, process and present this. CoP support is usually cheap; CoP measurement less so.

 

Of course, most organisations won’t be able to access this hard objective data: because they have no defined processes, because they don’t have a good information system, or because they don’t wish to invest in this kind of measurement structure. The roots of any of those can be many, and not all inherently bad. In those cases, indirect or subjective measurements will need to take front stage.

 

As a conclusion, we can find some guidelines that will be useful for both large, IT-intensive organisations and smaller, nimbler or poorer ones:

 

-         Anecdotes highlight changes in methodologies. These changes are not easily tracked in any other way; when due to CoP influence, they should be reflected in the impact report. Therefore, strict, conservative reporting (along the lines suggested above) on changes in methodologies, is a very good idea. It could be extended to exception handling if possible.

-         Exception handling is the core of support CoPs. Thus some sort of measurement of its efficacy at this is essential. When detailed reporting is available (rates of occurrence, time and resources used) it should be measured. When it is not, a direct estimation by users of time saved by the CoP is a handy method, since it can be translated into costs (translation worker hours into the amount paid for them). Other relevant data can be found looking into the number of calls on the formal support facilities or channels (knowing the average time spent per solved call, and knowing the difference in call numbers, we can work out the “saved” cost).

-         Knowing what you do tells you where you’re saving. A CoP about support issues will produce measurable support gains. A CoP about spreading best practices will produce measurable improvements in quality assurance. A CoP on methodology or innovation will produce measurable changes in processes. Knowing what you’re looking for tears away most of the CoP’s “unmeasurability” veil. True, most CoPs do a bit of everything; but they are funded for a reason, and it is that reason that they must be measured against.

 

 

Fig. 1 – CoP types by core activity.

 

And finally:

 

-         CoPs don’t manage. Since human work is not purchased “by the activity”, total organizational productivity (i.e. results for its costs) need not vary just because its members are more productive. It may perfectly well translate into more time freed and wasted, as people stand idle. Productivity gains need to be harnessed into production increases. Quality gains need to be exploited into commercial advantage. In other words, translating CoP-derived potential gains in efficiency into actual profit is the work of an aware, proactive management.

 

With all this in mind, meaningful reporting systems on the benefits of CoPs can succesfully be built.

 

 

The author:

 

Miguel Cornejo Castro is an economist who works as management and IT consultant. As a partner of Macuarium Network SL, he has done extensive work as CoP manager, moderator and “fosterer”, especially at the Macuarium.com CoP system for creative professionals and Mac users, but also at other venues such as the EU’s Knowledgeboard.com, and in projects for several third parties. He has also written a number of white papers about the nature of Cops and CoP management that can be found here:

http://macuarium.net/joomla/index.php?option=com_content&task=view&id=12&Itemid=33

 

Macuarium Network SL is a research-driven initiative in the Knowledge Management realm that provides community and CoP management and consulting services.

 

References and bibliography:

 

[1] Richard McDermott, 2002; Measuring the impact of communities; Knowledge Management Review. Can also be downloaded from:

http://www.knowledgeboard.com/item/506/23/5/3

 

 [India]: Intercooperation in India, 2005; Experiences with communities of practice in India Working Paper 1. Can be downloaded from:

http://www.intercooperation.ch/offers/download/ic-india/wp-1.pdf/view

 

[Canada]: Vicky Tam Carpio, 2004; Assessing the value of CoPs: summary of findings; Simon Fraser University. Can be downloaded from:

http://www.km4dev.org/index.php/articles/downloads/241

 

[4] Miguel Cornejo Castro, 2006; The Macuarium set of community of practice measurements; Knowledgeboard.com. Can also be downloaded from:

http://www.macuarium.com/cms/index.php?option=com_remository&itemid=0&func=fileinfo&id=87