Analysts are very important in the investment process. Investors pay hundreds of millions of dollars for their recommendations and forecast data so there is understandably keen interest in identifying the better performing analysts. Two measures with a high profile are the annual rankings by Institutional Investor (II) and the Wall Street Journal (WSJ).

Both publications rank the "star" analysts and the results can give a significant boost to those who are named. II does not provide much information on its criteria for selection and has been criticized as being a popularity contest, while WSJ claims its rankings are determined solely by performance. But are they so different? And do they offer anything of value to investors?

Xi Li and co-author Douglas R. Emery investigated the performance of 5,941 analysts between 1993 and 2005 and found that both rankings were essentially popularity contests. Worse, neither was a great indicator of analyst quality.

"We do not find the rankings to have any significant investment value. When compared to other analysts in the year after becoming stars, WSJ stars actually perform significantly worse and II stars no differently with respect to investment recommendations. Similarly, neither II nor WSJ stars are significantly different from non-stars with respect to earnings forecast accuracy," they say.

The authors applied a range of criteria to their sample and found those selected by II scored highly on factors relating to recognition. Accuracy in recommendations had the lowest effect (3.83 per cent). For existing stars hoping to repeat in II's annual rankings, recognition remained the dominant factor.

The situation was a rather different for WSJ stars. Recommendation performance was the most important determinant of whether someone was named a star and also whether they repeated in the rankings the following year. However, the selection criteria just to be considered in the rankings were biased towards recognition.

"Analysts in the WSJ rankings must cover five or more qualified stocks in the industry and normally at least two of these must be among the 10 largest stocks. In 2001, only 1,370 of more than 4,000 analysts were eligible. Like II's process, analysts working for smaller houses were at a disadvantage because small- and medium-sized houses typically focus on coverage of small- and mi-cap stocks, which are much less likely to be among the 10 largest stocks in an industry," they say.

The authors say while the two rankings do indeed differ - II is more like a club, WSJ like a tournament - they both have eligibility requirements that are significant barriers to entry and that put an emphasis on recognition.

And the fact that they do not provide value to investors indicates there is much room for improvement.

"Our results suggest the rankings could be improved by minimizing the influence of explicit and implicit eligibility requirements. Even though some eligibility requirements are inevitable, the influence of the current set is excessive. It also appears that risk-adjusted rather than industry-adjusted recommendation performance would be a better basis for the rankings," they say.