Matrix Assessment Tool Introduction Matrix Example

Introduction to the Matrix Assessment Tool
En Español

The Committee on Monitoring International Labor Standards (CMILS) developed a matrix framework as one possible way to analyze the data (See Figure 1 below).

A matrix framework, with variations in levels of compliance on one axis and the direction of change on the other might be a useful way of capturing these two dimensions simultaneously. In addition to avoiding the superficial precision implied by using specific numbers and weights to create a detailed index or ranking of countries, the matrix approach would have the advantage of giving a dynamic picture of compliance. Using this framework, assessors would carefully analyze each of the indicators in all three categories (legal framework, government performance, overall outcomes) for a given standard and come to a judgment about the placement of each indicator along both axes.

In general, "some problems" with compliance are those that are not so frequent, broad, or severe as to seriously undermine compliance with a given standard. "Severe problems" are such as to make compliance difficult or impossible. CMILS expects, however, that many indicators for many countries will fall into the middle category: problems that are serious enough to raise questions about compliance in important areas or for particular groups, but not across the board. For indicators in this middle category, careful scrutiny of whether compliance is improving or deteriorating will be of particular importance.

Clearly, there are other ways to design a matrix framework, and other ways to label the cells. For example, a 2 x 3 matrix that retains the three-part dynamic element but collapses the compliance decision to a binary one (in or out) is possible, but the Committee believes that it would fail to reflect the complexity of the situation in most countries. As assessors work with the indicators and the matrix, they may find that four categories, such as those used by the Organization for Economic Cooperation and Development (OECD, 2000), or perhaps subcategories, such as those used by Freedom House within the broad categories of free, partly free, or not free, are more appropriate (Freedom House, 2003). But as a starting point, the 3 x 3 matrix may be a useful way of highlighting the fact that, while some countries will fairly clearly fall into either the some or severe problems categories for some standards, most countries will fall into the middle category with respect to some indicators.

CMILS did not attempt to delineate the thresholds along either axis that would place a country above or below the line separating the categories because we do not believe that it is possible to set thresholds that are appropriate for every country and every situation. Judgments about government performance have to reflect the resources that a government has to work with: the question is not solely whether a government is spending more or less than a given percent of its gross domestic product, but whether it is doing the best it can with what it has. And, as noted above, many of the outcome indicators can mean different things under different circumstances. The burden of demonstration, argumentation, and persuasion necessarily falls on assessors and their critics to defend or challenge any given assessment.

Under this or any other monitoring framework, it is impossible to ensure that individual assessors are not hasty, biased, or ill-informed in making judgments about compliance. Given this reality, the matrix framework may be useful in identifying weaknesses or political biases, as well as divergences, in alternative assessments. A hypothetical example of how this might work is presented here.