The False Science of the Times Higher Education World University Rankings

Anthropologists complain about rationalization and bad quantification, but we generally moan about it to each other and are often not very specific. I want to try to explain what is wrong with the Times Higher Education (THE) World University Rankings. I want to argue that the survey is based on bad data and thus gives spurious results. All the results do is confirm loosely held biases, but by presenting them in quantitative form, the league tables become more real and powerful than they deserve.

This year, I was again asked to participate in the survey. I received the following email:

Dear Colleague,

Thomson Reuters is pleased to invite you to participate in the annual Academic Reputation Survey, which will support the Global Institutional Profiles Project and Times Higher Education World University Rankings.  You have been statistically selected to complete this survey and will represent thousands of your peers. The scholarly community, university administrators, and students worldwide depend on the survey results, as they provide the most reliable access to the voice of scholars like you.  

A sample survey is available online. I have to commend ThomsonReuters for their transparency on this; most pollsters try to protect themselves from criticism by keeping their questions secret. 

The heart of the questionnaire are four questions, in which we are asked to list the 15 best research universities 1) in our region and 2) in the world, and the 15 best teaching institutions 3) in our region and 4) in the world. This seems straightforward enough, and I am a great fan of "freelisting" (see in Bernard’s Research Methods in Anthropology or in ANTHROPAC). But people can only list things they know something about.

I do not normally think in terms of “best research universities”; I know of some good scholars in different universities whose work I admire, but in each case, I do not know most of their colleagues unless they also publish in areas I am interested in. I may notice that “State University” appears more often than “Podunk U” in authors’ affiliations, but if State is a large school, I may not actually attribute its frequent appearance to “excellence” in research. It may just be because they are big. It may also be because many of their graduating PhDs still have not found jobs, and are using their more prestigious school affiliation rather than the schools where they teach as adjuncts.

Ranking the “best research universities” may be more of a problem in anthropology, where we do not work in “research teams”, and where there is an emphasis on departments covering the diversity of cultures and approaches to culture, rather than concentrating on one area. I am actually more likely to know which university has a good Asian Studies program than which one has a good anthropology program. If I were asked to list the best universities for the anthropology of China, that would be easier. Recently the Society for Economic Anthropology weblist collectively listed the top graduate programs for economic anthropology. That is a reasonable list: where to go to study a particular subfield. Anthropology PhD students work with a small number of scholars, so it is more important that there be scholars in their primary area of interest rather than a "good program."

The problems with ranking "teaching universities" are even worse. First of all, teaching at the undergraduate and graduate levels are totally different. Many large state research universities have huge classes, and undergrads are often taught by TAs and instructors rather than the famous teachers who appear in the catalog. Those famous professors are freed up for research and to work with graduate students. So my recommendation for student is totally different depending on whether they are an undergrad or looking for a PhD program.

In addition, the truth is that most of us have NO IDEA what the teaching environment is like at other universities. (And here I’m not even going into the issue of whether students prefer more experiential participatory learning, or seminar style, or more traditional lecture formats, among many other variables. The idea that there is one teaching scale, good to bad, that we can all agree on is laughable.) Even for two elite universities where I have three data points--students who attended their graduate programs--I do not know how to generalize, as some thrived, and others struggled or were not that happy. I cannot understand how anyone can confidently list the best 15 schools for teaching.

Obviously, the problems with ranking universities for teaching is a point of contention, because it was addressed in the justification of the methodology this year. 

ThomsonReuters say that respondents who at the start of the survey indicate that their work is primarily teaching “are later asked to identify the one institution they would recommend that a student attend ‘to experience the best undergraduate and/or graduate teaching environment’ in their subject area.”  

They thus assume that those who teach primarily are in the best position to know where the best teaching programs are. This is ridiculous. Most academics who are primarily teachers are overburdened with teaching and struggling hard to try to get a research position. Just because they primarily teach in their current position does not mean they are aware of which universities offer excellent teaching.

Thus, since most people cannot really answer which are the best universities for research or teaching, they fall back on vague images of “prestige” and image. And when one adds in all the universities of the world, the results get even more confused. Do people filling out the survey really know whether Tokyo University or Beijing University or the Sorbonne is good? Aren’t people just filling out the survey based on vague impressions, many of them decades old?

I have noticed that many, indeed most, China scholars in the USA cannot remember the difference between Hong Kong University and The Chinese University of Hong Kong. I know because I am often introduced as coming from Hong Kong University (not correct). I am convinced that a small but significant portion of HKU’s lead over CUHK in THE tables and in other such reputational rankings is due to similar mistakes by the people who fill in such surveys. In medicine and some other areas, HKU may well be better than CUHK. But since HKU does not have an anthropology department, and their history department has gone through serious difficulties, I can confidently say that in a number of the social sciences and humanities, CUHK is stronger than HKU.

Another problem with the survey is more prosaic: the software did not work. On one page, I had not finished filling in the names of universities but hit the “Enter” key instead of the  Tab key and it went on to the next question. There is no way to go back on the survey. Then, when I was asked to select non-university research centers, the page would not let me add any because an error message popped up claiming that center had already been selected. I had to claim “I don’t know of any NON-UNIVERSITY research only institutions in this subject area” in order to go on. I also had to run the survey three times before I could get to the end, because it hung on me twice.

I hope we can pull the screen back and show that there is really just vague bias behind these league tables, and they should not have the outsized importance that they have achieved. I for one refuse to submit my survey. I think it is unconscionable to participate in this fraud. And I urge other scholars to publicize the tricks their deans promote to game the system. The more people realize the fraudulent and unscientific nature of the surveys, and the perverse incentives they engender, the less they will treat them as serious measures of quality.

The Press, China, and Cross-Cultural Communication

Many Americans would be surprised to find out how many Chinese students in the US are not very happy with their host country. I don't know the percentage, and maybe my sample is biased, but I do think most Americans assume the rest of the worlds wants to be American, which is at best a gross exaggeration and over-generalization. One thing I find many Chinese students complain about is the anti-China bias in US media. At first this was hard for me to understand. In part, students who complain that the US focuses too much on Tiananmen Square are a bit biased themselves, since they don't know anything about it so of course think if it as a minor issue. But they do have some points. Here is an article entitled "Westerners are so convinced China is a dystopian hellscape they'll share anything that confirms it" that points out that stories published in Time, CBS News, and Huffington Post that were totally wrong (even absurd) were re-posted as though they were true. Such stories, to a Chinese reader, suggest a hostile intent on the poster, and stupidity on the reader for believing such stories are possible. Who could believe that giant screens show the sunrise to compensate for the heavy pollution in Beijing?!

Unfortunately, cross-cultural understanding is not easy. There is a heart-breaking column by Helen Gao in the NY Times where she describes the rejection she faces from family and friends because she has studied seven years overseas. Even when she says the same criticism about pollution or politics as others, it is seen as inappropriate coming from her who has spent time abroad. At the same time, these same people have unrealistic images of the US:

Like someone gazing out a window but staring only at his own reflection, many Chinese look at America in ways that are colored by their feelings toward China. The United States they see — a nation with a pristine environment, perfect schools, generous welfare and complete political transparency — is a figment of their imagination, custom-made in stark contrast to the reality we live in.

 It has long been the fate of bicultural persons not to be trusted by either side. When the English arrived in North America, they exchanged young boys with the natives so each side would have native speakers of each other's language. This was an early type of AFS intercultural exchange! The boys would then go back to their own cultures to act as language and cultural interpreters. They were not trusted, however; . They would often try to make their countrymen see the situation from "both sides", which, under tense situations, would be perceived as disloyalty. It is the dilemma of cultural exchange and cross-cultural competence: we need to foster it to avoid international conflict, yet those who have these essential skills are often the ones who suffer the most for their competence. They have greater understanding (of both sides), but are often ignored or even rejected for their "deviant" ideas. When Helen Gao notes that she cannot repeat a friend's statement because it will be interpreted differently due to her time in the US, we see that there is really no central, unbiased point or position from which we can look at the world. There is no neutral "objective" space,  though we all try to stand there. But crazy stories about China do not help create that space, or the trust that is necessary for communication.

Cathay Pacific "Flash Mob"

Going through some old newspapers from early December, I came across an intriguing picture of several hundred Cathay Pacific attendants dancing in the arrival lobby of the Hong Kong airport. The online article has an SCMP video embedded, and there are a number of videos of the event on Youtube (see here, and here for a longer version showing the choir, and here for a more professional version). It is fun, and features my favorite airline. It was also part of "Operation Santa Claus," which is a good fundraiser run by our public radio and the SCMP (I'm not clear how this raised funds, however).

I do have a quibble with the term "flash mob," however, and it reflects something about Hong Kong. A flash mob is not supposed to be so well-planned, or be so purposeful. The Wiki entry for "flash mob" says it "is a group of people who assemble suddenly in a public place, perform an unusual and seemingly pointless act for a brief time, then quickly disperse, often for the purposes of entertainment, satire, and artistic expression."  It notes that for planned social activities and PR stunts, the term used should be "smart mob."

This stunt was well coordinated: the videos show guards keeping the space empty, the music is played over the airport PA system, and the dancing is choreographed. (Hmm, I wonder if they paid royalties for the music they played; might be a violation of HK's strict copyright laws!).

I think the term "flash mob" is being applied here for two reasons (in addition to the fact that worldwide, companies have used the term for stunts that then become viral online commercials). First, last year (2012) a much smaller less formal such event was held that was probably  more like a flash  mob. This year, more professionals got involved, including the fundraising aspect, so perhaps what was more like a flash mob has evolved into something more choreographed, but the name has not changed. But in addition, there is a tendency in Hong Kong for little to be truly spontaneous. Like elections (even student elections on campus), the idea of it being spontaneous is attractive, but everyone likes to know the result ahead of time (in fact, on campus elections, there is always only one ticket running for all the offices). In this case, you can actually see dozens of spectators anticipating the event, waiting for it to start (probably friends of family of the participants!)

A true flash mob is actually a bit like vandalism; it can cause congestion, not to mention accidents and injuries to bystanders who do not expect dancers to be in their way as they walk through an airport or a train station. Yet we seem to like the incongruity of a show (and a well choreographed one at that) in a public place. That is part of what makes it fun, even if it is, after all, a PR event. At least this is for a fund raiser and not just to try to convince us that Cathay Pacific staff are one big happy family who just love to take care of clients. Their staff is well trained and does offer a high level of service, but they are, after all, employees. It will be interesting to see how this event evolves in the next few years.

Academic Freedom and Hong Kong

I recently posted on a listserv an announcement about the Hong Kong PhD Fellowship, a great scholarship that pays for PhD studies in Hong Kong. I was surprised to receive a query by an anthropologist who does research on minorities in China, asking “How much academic freedom is really possible in Hong Kong as an anthropologist? Is it that much different from the mainland?” I was stunned. Hong Kong universities are funded and administered totally separately from the mainland. I think anyone who knows Hong Kong would agree that Hong Kong has a high degree of academic freedom, with no restrictions on teaching or research, a free Internet and plenty of books on Taiwan and Tibet in the libraries. We are quite vigilant about academic freedom. But obviously the perception out there is different.

As I thought more about this, I realized that there are periodic stories that will pop up in the international news (e.g. The New York Times) that might make Hong Kong seem less open than it is. In fact, we are living through one such story right now. This is the case of Hong Kong Television (HKTV). Hong Kong has two free-to-air TV companies, TVB and ATV, which broadcast one Cantonese and one English channel each. There has long been talk of expanding the market, and a process of adding more stations has been dragging on inexplicably for over four years. A couple of weeks ago, the HK government announced that only two of the three applicants for new licenses would be granted, and that HKTV would not get one. The astonishing part of the story is that the government has refused to explain its decision. It says that Executive Council decisions are confidential. This is true, but it is bad politics not to give any explanation. It has emerged that consultants’ reports said that all three licenses could be issued, but it seems that there is some worry that the Hong Kong market, with only a bit over 7 million people, is not big enough to support five TV companies (and ATV is considered the weakest of all; TVB currently has an astonishing 80 percent market share).

Protests have erupted, confidence in the Chief Executive has declined, public opinion ratings of the government have hit an all-time low, and even some pro-government politicians such as Michael Tien and former ExCo members have said this decision needs to be explained or perhaps reversed.

It is not clear why the government is being so silent about the reasons. In the silence, speculation fills the gap. Some suggest that the tycoons who control the existing stations pressured the government, since the winning bidders are companies owned by tycoons who run cable TV companies in the SAR. Many were quick to suspect interference from China, but HKTV owner Ricky Wong said he doubted that, noting that he was not involved in any politics and that his proposed station was all entertainment and no news or current affairs. On the other hand, there has been speculation that his mistake was not including any mainland officials on his board to make Beijing comfortable with his company. It  has been suspected that because he is truly independent and has no investments in the mainland, that Beijing put pressure to have his license application denied. This is all speculation, however. It would explain why the government cannot reveal the reasons for denying him a license, but so would the tycoon conspiracy theory. Part of the reasons for the protests and speculation, however, is that people in Hong Kong are very vigilant about their freedoms.

Someone reading about this story from overseas might rightly worry about Hong Kong’s freedom. The Edward Snowden story also raised questions about Hong Kong’s autonomy. But the Snowden case also affected international relations (as well as legal process), so was at least partly under Beijing’s mandate. (As an aside, I find it hilarious that Hong Kong claimed they could not hold Snowden because of discrepancies in his middle name on the documents provided by the US. This is the kind of bureaucratic idiocy and procedural rigidity we often face in Hong Kong, both with the government, in the university, and even in banks. It made for a believable excuse in the Hong Kong context, though sounded ridiculous to Americans--as it indeed was. If he had been wanted by the HK police or PRC authorities, there is no doubt they would have detained him).

But when it comes to things like political and academic freedom, we in Hong Kong definitely do not think of ourselves as part of China. Indeed, I was surprised to read in Slate the sentence that said “... Sarah Harrison, a WikiLeaks advisor who has reportedly been by Snowden's side since he was in China.” I wanted to say, “Hey, he was never in China, he was in Hong Kong!” This is not to deny that Hong Kong is part of China, but that there is a big difference between his being in Hong Kong and being in mainland China. The tragedy of the Snowden affair for Hong Kong is that in the contradiction between following legal procedure and the need to follow Beijing on international relations, the law may have been bent.

Academic freedom everywhere is very important, and needs to be constantly protected and even expanded. This is why MarshallSahlins’ article in The Nation on the Confucius Institutes is important (see also an earlier NYTimes story here). These Institutes are established within leading universities abroad but have teachers hired by the Chinese government. Of course, every donor has an agenda when they donate money to a university. But universities who accept such funds need to take more care to assure they maintain control and academic freedom. In the meantime, I can assure readers that we are quite free to research controversial topics in Hong Kong, from religious groups to the Hong Kong Autonomy Movement to environmental pollution. Whether we get government grants for some of these projects is another matter, but then the problem there seems to be more that our government is most concerned with promoting research that supports the HK economy. But that is another problem. When it comes to freedom, we in Hong Kong even have professors who are a thorn in the side of the government and are organizing "Occupy Central," and no one is talking of them being punished or fired. There are costs to activism (see Chan Kin Man's case here), but not in the form of retribution from our administration. So far, at least, we have maintained a high degree of autonomy, and freedom.

Daniel Ng, 1937-2013

Last week I attended a memorial for Daniel Ng, the entrepreneur who brought McDonald's to Hong Kong. He had a PhD in chemical engineering and worked first in the US (and I understand he had a patent for a fuel cell for pacemakers), and then at 35 years of age in 1972 moved back to Hong Kong to work with a venture capital fund. In 1974 he went back to Chicago for a year to learn the McDonald’s system, before opening his first restaurant in Hong Kong in 1975. He had a 20 year franchise and was extraordinarily successful, so that when McDonald’s exercised its option to buy back the franchise, he was very wealthy. He ran the company so well, and kept McDonald’s prices so low, that no other hamburger chain has ever had much success in HK. It was he who suggested the "Big Mac Index" to the Economist.

I knew him because he was the primary supporter of AFS Hong Kong, and was the chairman of the board, on which I am a member. The success of McDonald’s in HK was due to many factors, but top among them were: 1) his deep understanding of branding (he also created a logo and brand image for AFS worldwide); 2) his sense of humor and love of fun (he liked McD because it sold fun); and 3) his belief in persistence and determination. Apparently he loved a quotation from Calvin Coolidge to this effect, but while Coolidge said talent, genius and education were not enough, Daniel had all those characteristics PLUS persistence!

Daniel was quirky. One friend described him as “weird,” even. But he was genuine. He was gentle, and not abrasive, but decisive. Often he got his point across with humor. Whether due to intelligence, experience, or a well developed “executive function” in the brain, he was very decisive. He was completely bi-lingual and bi-cultural, with native English and Cantonese. He was also apparently notorious among his friends for only buying second-hand cars, and not spending a lot of money on clothes (those who know me will see why I feel a connection with him).

He was passionate about many things. He was a strong supporter of music, and commissioned a number of works to promote young composers. He was a strong supporter of the Ronald McDonald House and had been working in the past few years to make sure that a second house would be opened with the new specialty pediatric hospital on Kai Tak. And he was a strong believer in intercultural understanding. He wanted Hong Kong youth to be more cosmopolitan by going overseas on exchanges, and by meeting exchange students who come to Hong Kong for a year. He did not suffer fools, and he thought a particular horse-related charity in Hong Kong was being very narrow-minded when they decided only to sponsor outgoing Hong Kong students and not incoming foreign students (they claimed they only wanted to support Hong Kong students, as if Hong Kong students don’t benefit from having an exchange student in their midst).

Daniel died on August 23rd of cancer at age 76, which is still young by today’s standards. As his children’s statement put it, He had been ill for some time, but never wanted to let anyone know, choosing as always to focus on living. he had a formidable spirit, insatiable curiosity and zest for life that served him well even at the end.” His memorial was attended by probably over 400 persons, and it was by invitation only, or it would have been many, many more. Everyone had humorous stories to tell. Former employees and colleagues were very grateful for his mentoring and guidance.

There is one image of Daniel that captures his youthful spirit for me. Years ago, shortly after the iPhone first came out, he already had one. He also had several apps, and he had a blast showing them off. He had the “lighter app” that would allow you to hold up a “lighter” at a rock concert, and he had the “beer app,” which appears to fill the iPhone screen with beer, and then allows you to “drink” the beer as you tilt the iPhone to your mouth. He loved to laugh. Almost every picture you will find of Daniel has him smiling broadly or laughing. He was truly an impressive figure.

The lessons of North Korea

Juche Monument, Pyongyang, North Korea (aka DPRK) with a human standing in front, for scale. There should be an index that measures the size of national monuments. I can't help but speculate that the larger a nation's monuments, the more authoritarian is the state. I think of the pyramids, and the monument to Vittorio Emanuele II in Rome. Perhaps there are other, stronger, correlations. Maybe the larger monuments indicate a stronger state elite (elites who build things to impress the masses with their power, and to get the masses to be "proud" of the state--i.e., the elite). And maybe this is all too crude. After all, the US seems to have had its peak of monument building well before its recent peak of centralized power. The biggest monuments in DC arethe Washington Monument (built between 1848-1885) and the Lincoln Memorial (1914-1922).

The DPRK is sociologically fascinating. Social scientists spend much of their time showing that people's choices are shaped by the culture and society (well, except for economists, who still believe in the autonomous individual). When we face a society where choices are really constrained, like the DPRK, then we are forced to recognize that choice is, in fact, important in most modern societies. Yes, all societies limit our choices and mold us in certain ways. But the DPRK is at an extreme. The task of combining structure and agency is never easy, and new terms such as "structuration" don't help much, except to highlight the challenge. Every society is weird in some way. The DPRK's weirdness forces anyone who dismisses choice and markets as not important to recognize that freedom to choose actually does matter. It is an exotic culture of our time that shows the limits of what humans can bear and do, a fascinating example, and one in the end that highlights why some level of individual freedom is in fact necessary and desirable.

Hong Kong Has "The Most Efficient Health Care System In The World"

The Huffington Post has an interesting article that shows that according to a recent Bloomberg ranking, Hong Kong has the most efficient health care system in the world. The real point of the story is to show that the US is 46th of 48 countries: 

"In other words, the world's richest country spends more of its money on health care while getting less than almost every other nation in return."

Of course, all such rankings depend on how variables are measured. In this case, though I'm happy to have Hong Kong's success recognized, I think the measure is a bit too primitive.

Each country was ranked on three criteria: life expectancy (weighted 60%), relative per capita cost of health care (30%); and absolute per capita cost of health care (10%). Countries were scored on each criterion and the scores were weighted and summed to obtain their efficiency scores.

So life expectancy is the only measure of health outcome.  A bit primitive, but not invalid.

The main message from the story is that all the top (most efficient) systems are heavily state controlled, and are universal.

Despite being considered by some as having the freest economy in the world, Hong Kong's universal health care system involves heavy government participation; its own health secretary calls public medicine the "cornerstone" of the system. Public hospitals account for 90 percent of in-patient procedures, while the numerous private options are mostly used by the wealthy.

Many Americans will just dismiss this. Nothing will shake their faith in the "free" market.