Japan is No.1 for human rights and democracy in Asia and Myanmar the worst, according to the Asia Democracy Index 2005 (ADI). 

ADI has attempted to “more accurately” survey and evaluate the scope of political freedoms in Asia by using statistical analysis of civil and political rights in 16 Asian Countries. The result is a league table, some might say ‘rogues’ gallery’, of democracy, justice and good governance in Asia. 

ADI based their findings around six categories of democratic structure: civil rights, elections and political processes, governance and corruption, media, rule of law, and participation and representation. An individual or civil society organisation in each of the countries was chosen to survey “politically aware” people from each country’s civil society. The purpose being to “provide a snapshot of which governments are making progress as far as democracy is concerned”.  

But what do such surveys and rankings actually tell us? Are they reliable, or do they just give the semblance of scientific accuracy? As Benjamin Disraeli famously said: “There are three kinds of lies: lies, damn lies, and statistics.” 

As criticism of such methodologies has noted, such indices are based on the ‘perceptions’ of those surveyed thus introducing subjectivity. ADI concedes that its survey samples are not entirely representational. The recommendations section on Japan reads, “if this survey were taken by non-Japanese residents, it is beyond doubt that those respondents would mark that there [sic] rights are limited and they suffer from a variety of discriminatory practices and behaviour.” Moreover, the results of the ADI survey do not make clear exactly who they polled and why they were chosen. 

Comparative rankings according to survey results should be treated with caution. The perception of civil society is an important aspect of gauging a national situation. But on its own it cannot be an accurate means of comparing countries with one another, as different populations have differing expectations, national narratives and beliefs about their own governments, which will reflect the picture given of each country. 

Moreover, simply asking whether an individual believes their judicial system to be good or bad does not tell one anything without knowing the exact criteria by which this opinion has been formed. Is there data to back up the opinion and was it measured against internationally recognised criteria? The benefit of specific and objective data, as underpins such surveys as the UNDP’s Human Development Index, is wholly absent from the ADI index. 

The practice of ranking countries also opens itself to charges of bias. A certain number of indicators are chosen and then weighted by the index compilers. This often reflects the interests and concerns of the compilers. The oft-cited Freedom House Democracy Index is a good example. Similarly, small changes to the system of calculation can produce large differences in the results, leaving numerical data vulnerable to either manipulation or significant, if inadvertent, inaccuracies. 

ADI does not explain how or why its chose its indicators. From a human rights perspective, ADI at minimum has failed to capture key qualitative factors such as whether or not a country is a signatory to major international human rights treaties, whether the provisions of such treaties have been implemented at the national level, and whether such provisions are being applied justly in practice.  

By merely gathering civil society perceptions of practices and institutions ADI creates the perception of accuracy, when in fact the numbers are not supported by any underlying research data or analysis.  To be of any use at all, such surveys must be based on transparent, specific and objective data, systematically collected, and subject to careful and detailed analysis against broad and sophisticated indicators and benchmarks derived from internationally recognised standards or norms. Without such safeguards, the accuracy of such surveys is more illusory than real. 

Numerical rankings in the context of human rights also stand accused of dubious relevance and false precision. As Robert Justin Goldstein has put it, “I doubt that it would be worth the cost to know, with a kind of false precision, that, for example, Chile has a ‘human rights score’ of 144 and East Germany has one of 150 when it is much easier and just as useful to know (through a variety of qualitative and quantitative indicators, including journalistic evidence) that they both are highly repressive regimes.” 

The danger is that rather than fully and accurately reporting on unique situations, numerical scores and rankings reduce the reality of complex situations into a series of categorised numbers. Numbers that are good only in comparison to worse numbers may easily be taken for tacit approval of certain governments, when in reality there may be significant problems hiding under the surface. The picture is distorted, with those at the bottom of the table being given disproportionate attention to those at the top. 

In the context of the rule of law in Japan, ADI asked the participants whether they agreed with the statement, “[t]he judicial system effectively protects human rights and democratic principles”: 68.85% agreed. But what help does such a bold statement really give us? Impliedly over 30% of participants did not feel that the judicial system effectively protects human rights or democratic principles, but ADI do not say what those participants thought or why they held those views. A statement which on the face of it compliments the Japanese judiciary does not in fact begin to reflect the complexities and nuances of the Japanese judicial system.  

Rankings also cannot adequately explain changing situations. Human rights conditions are constantly in flux. Legislation that fundamentally undermines human rights may be introduced or repealed in a matter of days or weeks. How is this to be reflected in a statistical analysis? Is the repeal of a flawed piece of counter-terrorism legislation worth an additional 10, 20, 30 points in the index?  

What is important is the precise nature of change, which is not illustrated within an index compiled as an average of several disparate indicators. What is more useful in practice is the presentation of complete information about situations as they happen analysed against international criteria, rather than seeking to reduce the adoption or appeal of a piece of legislation to an averaged points shift in an index. 

Some of the written analyses that follow ADI’s statistical results are insightful, but more often they highlight the flaws of the index approach. 

ADI’s written material on Japan briefly highlights the multiple and serious flaws of the Japanese system. The report acknowledges that the election system is top-down, that single-party domination is entrenched and government lacks transparency, that media access to information is limited, that the justice system is draconian and dissent may be easily stifled, that minorities go unrepresented, and that political processes and civil institutions are distinctly non-participatory. Each of these is a serious deficit in democratic governance. However, the statistical model used gives Japan first place. 

By contrast, the US Department of State Human Rights Report on Japan (2004) details the efficacy of constitutional and legislative provisions, how government agents fare in putting those provisions into practice, and all specific matters of concern such as allegations of criminal beating, sex trafficking and social discriminations within the country. This information is of far more use to governments and human rights actors than to know that Japan has an overall average ADI score of 62.41. 

Another example is South Korea, which is ranked tenth. The report notes that there are problems with the country’s National Security Law, weak political parties, restrictions on media, and migrant workers’ rights. However, the report does not capture the complex shifts and liberalisations that South Korea has gone through in recent years and in recent months. 

The US State Department Report on South Korea (2004), on the other hand, details the areas in which the system continues to fall short of fully functioning democratic institutions. It notes where circumstances have changed, for the better or for the worse, from past practices, and at the same time gives an exhaustive accounting of all issues and illustrative cases available. In contrast, the ADI does not adequately examine the protections afforded citizens in the constitution, or laws more recently passed that have greatly democratised the state. It does not detail the strengthening civilian control over police and military, the increase in women’s legal and social rights, or the government’s steps toward ending corruption. The indicators used give no account to these factors. Perhaps the individual participants in the survey did, but we simply do not know. 

Even if indices, like ADI, are conducted in accordance with best practice, they are inherently open to charges of inefficacy and irrelevance. At best they may be used to ‘name and shame’ those States with the worst records, or as an advocacy tool in some form of international ‘keeping up with the Jones’. But at worst they present an overly simplistic and incomplete picture, open to distortion for political ends. 

Human rights and democracy should not be treated as a political football league. There are better and more meaningful ways to scrutinise the performance of States, such as reports like those of the US State Department. Such approaches have substance and analysis, are based on objective criteria, and are ultimately more likely to reflect an accurate picture of human rights and political freedoms in individual States.

 Human Rights Features