peer-review | James Colliander

A Report on the 2012 NSERC Discovery Grants Results for Toronto Math

Fri, 06 Apr 2012 18:51:57 +0000

Fifteen¹ faculty members from the Department of Mathematics at the University of Toronto submitted proposals to the 2012 NSERC Discovery Grants competition. Of these, one was a first time applicant (En), two (Ga, Ia) applied after a successful appeal of 2011 results, and one (Cd) was an appellant whose appeal was denied but could reapply because the 2011 award was for zero dollars. The first table below shows the 2012 results (in thousands of dollars per year) with 2010, 2011 award amounts for those researchers. The second table shows similar data for Toronto mathematicians in the 2011 competition, including the amounts for researchers (1d, 2d, 3d, 4d) whose appeals of 2011 results were rejected. The average for 2012 awards to Toronto mathematicians was 153% the average for 2011.

Average Grant Amount (Toronto Math)

2006: \$27k/y
2007: \$26.3k/y
2008: \$26.5k/y
2009: \$25k/y
2010: \$25.3k/y
2011: \$19.3k/y
2012: \$29.5k/y

Instability Visualized

Viewing the results from the perspective of researchers in these competitions reveals instability in the Discovery Grants evaluation and appeals processes:

Imagine the experience of researcher 2d. This person had five years at \$42k/y, was cut to \$30k/y in 2009, and successfully appealed that outcome. The result of the appeal was a one year reinstatement of the previous grant level at \$42k/y and permission to reapply to the 2011 competition. In 2011, this researcher’s grant was hacked to \$18k/y so this person files another appeal. The 2011 appeal is rejected.
Contrast the experience of professors 4d and Ga. Both launched their Canadian research careers and entered the Discovery Grants competition for the first time in 2011. Ga’s appeal of the \$13k/y result from 2011 was successful and the 2012 competition led to a new result of \$30k/y for the next five years. 4d’s 2011 appeal was denied so this researcher is locked in for five years at \$11k/y. Which of these researchers is likely to have better HQP numbers at renewal time five years from now?
The experience of 2011 appellant Ia is also a bit strange. After a long run of celebrated research funded at the \$40k/y level, this researcher’s funding level was dropped in 2011 to \$15k/y. That outcome was successfully appealed and the 2012 outcome was \$35k/y.
The 2011 appeals by 1d, 2d, 3d, and 4d were all turned down so these researchers are locked in at relatively low funding levels for the next five years.
NSERC deviated from standard policy (a “pilot program”) in their handling of the 2011 appeals of Toronto mathematicians. Toronto’s appeals were evaluated by multiple appeals advisers while appeals from other universities were evaluated by one. There is evidence² showing that Toronto appeals were denied even when one of the appeals advisers advocated for granting the appeal.

Consistent Results on Appeals Cases show 2011 was Anomalous

Proposals by three Toronto researchers were evaluated in both the 2011 and 2012 competitions. The outcomes for these proposals provide a comparison³ between the accuracy of the merit evaluations by the 2011 and 2012 Evaluation Groups and bin-to-funding assignment by the Executive Committee and NSERC staff. Here is the data, including the percentage adjustment from 2011 to 2012:

These three cases provide further evidence, consistent with the message in the public statement signed by over 300 Canadian researchers, that the 2011 evaluations were anomalous. Despite the consensus opinion from the Canadian math/stats community, a public message from a majority of the 2011 Evaluation Group, and advice from top administrators that the 2011 anomalies required an altered appeals process, NSERC chose not to reevaluate the scientific merit of proposals when considering whether to grant or deny an appeal. The grounds for a successful appeal required evidence of administrative errors; evidence of error in merit evaluation was not considered germane.

The Math-NSERC Liaison Committee is collecting data from department chairs about the 2012 competition for Section 1508. It might turn out that 2012 will be viewed as more consistent with expectations, a more accurate evaluation compared to 2011. This would be encouraging. However, the unsuccessful 2011 Toronto “pilot program” appellants (researchers 1d, 2d, 3d, 4d) face the next five years with inadequate funding to support their research programs.

Footnotes:

The names of the faculty members are suppressed. The 2012 applicants will be referenced with codes A, B, C, …, H; 2011 applicants (without successful appeal) will be referenced using 1, 2, …, 7. The appended small case letters indicate whether the researcher had an appeal granted (a), an appeal denied (d), or was a new applicant (n).
This evidence is contained in the reports of the appeals advisers provided to the appellants by NSERC and also in documents obtained by some of the appellants through formal requests under the Access to Information Act.
It would be useful to know the data (number, success rate, basis for granting) for Discovery Grant appeals submitted to NSERC over the past few years. As far as I can tell, NSERC does not provide this data.

Anticipating the 2012 NSERC Discovery Grants Competition Results

Mon, 26 Mar 2012 18:50:38 +0000

2011 Discovery Grants Competition Aftermath

Anomalies in the results of the 2011 NSERC Discovery Grants competition provoked a flurry of activity nearly one year ago. My blog post from April last year reported on surprising results for several of my colleagues at Toronto. An email flurry among Canadian mathematicians culminated in a late April public statement which was eventually signed by 336 Canadian researchers, including 35 Fellows of the Royal Society and 27 Canada Research Chairs. In late May, a majority of the Evaluation Group for Section 1508 released a public letter to NSERC President Suzanne Fortier reporting on a “lack of fairness”, a “lack of transparency” and that their “confidence in the program, as currently administered, is regrettably shaken”. President Fortier gave a presentation to the Canadian Mathematical Community at the Summer CMS Meeting. President Fortier’s slides have not been released but there are notes and rebuttals by Walter Craig. Claims of grade inflation have also been analyzed.

Toronto Math Appeals of 2011 Results

Last May, seven Toronto mathematicians submitted appeals of their 2011 results. As far as I can determine, NSERC does not provide statistics on the number of appeals but seven math appeals was unprecedented from Toronto. Appeals are normally resolved within two to three months. As of late September, the decisions on the appeals were not yet announced. Toronto Math made email and telephone inquiries asking that decisions be made soon since successful appellants would need time to prepare new proposals in advance of the November 1 submission deadline.

Email correspondence revealed that Toronto’s appeals were handled differently than those from other universities. In particular, NSERC’s Isabelle Blain informed me that Toronto’s appeals were processed through a “pilot” program involving two or more appeals advisers. Three of the appeals decisions were announced in late September and the remaining four appeals were announced on October 19, 2011: 2 of 7 appeals were granted. The good news for the successful applicants was bittersweet since it set in motion a rushed effort to prepare proposals for the 2012 competition. Unfortunately, the reports from the appeals advisers for one of the successful appellants could not be consulted while preparing the 2012 proposals since they were not provided by NSERC until after the submission deadline. Toronto appeals were rejected in circumstances when one or more of the appeals advisers advocated for granting the appeal. Some Toronto appellants perceive the “pilot” appeals process to have been unfair. Conversations along these lines continue….

Repercussions: Uncertainty, Frugality and HQP

In the wake of the 2011 anomalies, many Toronto mathematicians who face renewal of their NSERC Grant in 2012, 2013 and even in 2014 are hesitant to commit funds toward hiring postdocs or toward training graduate students. Uncertainty has provoked frugality. Running a successful, even world leading, research program is no longer sufficient to justify an NSERC Discovery Grant. The new evaluation metrics, introduced by NSERC in 2009, require evidence that the principal investigator is successful at training highly qualified personnel (HQP). Taking frugal actions now will limit HQP production over the next few years resulting in a lower or zero grant next time: a vicious cycle which obstructs research advancement and training of young investigators. With the goals of breaking this cycle and maintaining the research vibrancy postdocs bring, the Department of Mathematics at Toronto has raised its funding support level (by 33%) for postdoctoral hires.

Looking Forward: Continued Instability

The 2012 Discovery Grants Competition results will be announced in the next week or two. Hearsay and anecdotal reports about the evaluation process have been reassuring. The Evaluation Group received recommendations jointly authored by the Long Range Plan Committee Chair Nancy Reid and the Math-NSERC Liaison Committee Chair Walter Craig. Similar recommendations were made in a December letter by a group of physicists to President Fortier. The Long Range Plan should help heal the rifts between the mathematics and statistics communities of Canada when it is released later this year. There are reasons to be optimistic in the short term.

In the intermediate term, there is likely to be continued instability with the peer review system at NSERC. Despite the 2011 anomalies for math and the ensuing kerfuffle, receiving similar complaints and suggestions for improvements by physicists, and faced with public criticisms by astronomers, chemists and engineers, NSERC staff insists that the new system is working well. The new evaluation system, the “conference model” with its “bins” and HQP touchstone, was introduced in 2009 and will be the subject of a five year review in 2014. Whether led by current or future NSERC staff, or by an energized scientific community, I anticipate substantial changes, and therefore instabilities, to the peer review process at NSERC in 2014 or 2015.

Shrinking Funds for Basic Research

Beyond the fairness of the evaluation process, there is the issue of shrinking federal investment in basic research. According to NSERC's 2010-2011 tables, expenditures on Discovery have been flat to decreasing when viewed in constant 2000 dollars over the past decade (data taken from Table 1). In contrast, the total NSERC budget has grown with the expansion occurring in programs aimed at commercialization rather than basic research.

Meanwhile, strategic recruitment leveraged by the CRC and CERC programs has increased the number of faculty submitting Discovery Grant applications (data taken from Table 27). Over the same time period, the number of successful applications has decreased (data taken from Table 27).

Table 50 shows that mathematicians do not really benefit from programs outside of the Discovery Grants. Mathematics, the “poetry of logical ideas”, is profoundly relevant to various industries. Canada’s research mathematicians need to find ways to share their expertise and leverage their Discovery Grant funds the way other disciplines do.

NSERC Peer Review System is Broken for Mathematics

Tue, 12 Apr 2011 05:49:49 +0000

Anomalous results of the 2011 NSERC Discovery Grants competition in mathematics have provoked a loss of confidence in the NSERC peer review system. To avoid a substantial loss of Canada’s scientific talent, which has been enhanced through the Canada Research Chairs program and other spectacular hiring over the past ten years, scientific policymakers need to quickly fix the broken peer review system. In the absence of an effective peer review process setting the strategy for research investment, Canada will miss out on the rewards made over the past decade’s recruitment of scientific talent.

What is happening in other sciences? Anecdotal reports from the following sources suggest the anomalies are not restricted to the math department at Toronto:

Toronto: MATH, CHM, EEB, PHY, STA, Engineering
UBC: CS, MATH
Queens: MATH

What happened to other disciplines? I’d like to know but NSERC won’t reveal the 2011 data until after the federal election. I’d like to hear from other scientific disciplines about their confidence in the recently transformed NSERC peer review system.

In 2007, NSERC commissioned a review by an international committee culminating in this report. (Please find my annotated version here and a marked up version of the NSERC Management response to the 2007 International Review Committee Report.) This report made recommendations leading to fundamental changes in the peer review process for all disciplines starting in 2009. The implementation of these changes (involving the so-called conference model and binning system) and other forces have provoked a loss of confidence in the peer review process at NSERC among mathematicians at Toronto, and across Canada.

Toronto Math Results are Anomalous

The results (names omitted) of the 2011 NSERC Discovery Grants Competition for the Department of Mathematics at the University of Toronto are anomalous:

Professor A. \$29k/y to \$18k/y
Professor B. 40 to 15
Professor C. 42 to 30 to 42 to 18
Professor D. 26 to 18
Professor E. 40 to 40
Professor F. 38 to 47
Professor G. 0 to 0
Professor H. 15 to 13

(The numbers represent annual research grant amount in dollars for the past 5 years and the new number for the next 5 years. For a description about how mathematician’s use these funds, go here.)

About Professor C.

Consider the case of CMS award winning Professor C. In 2010, this researcher’s grant was cut from 42 down to 30. After an appeal, the grant was reinstated for one year back to 42. In this year’s competition, one year after the appeal, NSERC drops it to 18, a 57% cut. Meanwhile, Professor C’s frequent collaborator (each had more than 50% overlap of their research with the other during 2006-2011), Professor K., received 45 staying at 100% of the previous level in this competition. Will the real opinion of NSERC on Professor C’s research please stand up? Professor C’s story is quite similar to Don Fraser’s personal account. (Within the conference model, I understand that a different group of only 5 experts may have reviewed the proposals of Professors C and K. This remark can account for the inconsistency but reveals aspects of larger issues that need to be fixed.)

About Professor B.

Imagine running a successful research operation (success, former students get awards, 13 major pubs in 2006-2011) for the past five years like Professor B using 45K/y. Students are in the pipeline; postdoc candidates have been scouted; Professor B has new ideas. NSERC rewards this person with a drop from 45 down to 18, a 60% cut. This researcher is confused with the outcome: “What did I change? What should I have done differently?”

About Professor G.

This person is a (perhaps the) world leading expert on a substantial research area. It seems this person, despite spectacular research success, is unworthy of a Discovery Grant because they don’t produce enough students. It is as though Canada has a sports car and they don’t put tires on it.

Secondary Effects Scenarios

Mathematicians, of international calibre, with a steady research production stream and surrounded by young researchers have had their grants slashed by nearly 60% during the 2011 NSERC Discovery Grants Competition.

Now, imagine you are an assistant professor in Canada. You might have nice support right now, like a Sloan or an ERA. You are building a research group, spending money on HQP, scouting talent. But your funding has a finite time horizon and the Discovery Grants look unstable, unpredictable. So, would you leave Canada if you could? Fix NSERC or the young talent will leave Canada.
Now, imagine you are a recently recruited Canada Research Chair: if you just concluded that your junior faculty member might be wise to leave Canada, how do you see your department in 10 years? Would you leave Canada if you could? Fix NSERC or the CRCs will leave Canada.

The system is broken and needs to be fixed.

There are (at least) two main problems:

Math in Canada is treated unfairly compared to other disciplines
The Peer Review system is broken

Math in Canada is treated unfairly

The main problem with mathematics funding in Canada is the amount invested is too low. I’ve written about this before. Consider the data from 2009 of NSERC Discovery Grants (2010 is similar, 2011 is not available) over the disciplines:

The average math and stats grant is $20K/y while the average over all disciplines is $41K/y. Why is it that the average scientist in Canada can expect more than double the amount a Canadian mathematician can expect? Keep in mind that Discovery Grants are primarily used to fund research personnel not expensive labs.

David Wehlau’s data (posted and discussed here) reveals the trend: over the past twenty years, mathematics investment as a percentage of the total amount in Discovery Grants funding has declined from nearly 4% down to 2%. Math has received less and less funding compared to other disciplines. This subtle reallocation needs to be abruptly reversed.

The unfairness then multiplies. Consider the following snippet from the 2007 international review report: Other disciplines are benefitting more from other industrially targeted NSERC programs and other sources compared to pure mathematicians. NSERC views Discovery Grants as grant-in-aid: a precursor grant leading to other sources of funds. The international review committee reports that is not the case for mathematics AND mathematicians receive on average less than half the funds received by other scientists. This is an implicit funding reallocation away from mathematics toward other disciplines and is unfair.

Broken Peer Review System

The outcome of the 2011 competition, and consistent reports (like Don Fraser’s) from the past two years, have provoked a loss of confidence in the peer review system at NSERC. To rebuild trust and avoid the departure of talented scientists, the mathematics community of Canada needs to know what happened in 2011. We need to understand why the peer review system produced the 2011 funding allocations.

There has been considerable chatter in the mathematics community about the 2011 competition. However, we need people with official roles to speak officially at this time. In addition to the forthcoming data from NSERC, I hope that Section 1508, the Mathematics and Statistics Evaluation Committee will explain the 2011 evaluation process and actively participate in discussions leading to an improved system that regains the confidence of mathematicians and statisticians working in Canada. What happened? How can we fix it?

The Lucky Few of Waterloo: Does the Perimeter Institute deserve $50M Times Two

Thu, 31 Mar 2011 05:44:16 +0000

There is chatter (here is more) suggesting that the $50M from the Conservative federal government (over 5 years) and the additional $50M (also over 5 years) from the Ontario Liberal Government to the Perimeter Institute is based more on politics than on scientific merit. These funding announcements emerge just a few weeks after the news that Neil Turok, Director of the Perimeter Institute, joined the Science, Technology and Innovation Council which advises the government on science policy. The chatter resonates with other statements that scientists from Western Canada get more than their fair share, that scientists from Toronto are discriminated against in funding decisions, and that Quebec scientists get funding just to allay separatist agitations. The key difference is that scientists outside of Perimeter must compete for funds through the Tri-council granting process which, at least in principle (but not in practice for mathematics), provides accountability and selects for scientific success.

The peer review system leverages the expertise of leading scientists to assess proposals for research investment by the government. When it works well, the process is trusted by the community of scientists to be based on scientific merit. A trusted peer review process is the ecosystem in which scientific creativity and excellence flourish. When the selection of scientific investments is perceived to be based upon factors other than scientific merit, the entire system is destabilized. The goals of research funding will not be met if the distribution system provokes poisonous comments against fellow scientists. The politicization of research funding and the ensuing unscientific conversations inside the community of scientists distract us from our agenda to advance the basic understanding of everything.

Scientific research investments selected by scientists through effective peer review are strategically superior and less risky than research investments made by politicians.

Consider the numbers:

The annual amount for 2009-10 Discovery Grants funding for biology, chemistry, physics, mathematics, computer science, all types of engineering,....was 64 Million dollars distributed over around 2000 researchers. The average individual grant amount was $33K.

In 2009-10, the Perimeter Institute had 12 Faculty, 12 Associate Faculty, 47 postdoctoral fellows, and 25 PhD students. The planned 100 million dollar investment over five years yields 20 million dollars per year. Let's imagine PI has 50 principal investigators. This averages to an individual grant amount of $ 400K ~ 12 X $ 33K. (Keep in mind that Perimeter Institute faculty also have Discovery Grants, that we are making underestimates, and that PI has other substantial recent investments.)

It is not obvious that the average Canadian scientist merits 12 times less research funding than the average scientist at the Perimeter Institute.

Canadians should insist on an accountable distribution system of government investments in research. We should demand an effective peer review process in the distribution of all research and development investments made by the government, even those funds distributed outside the Tri-council umbrella like the allocations for Perimeter and the $4.7 Billion in SR&ED tax breaks. The Perimeter Institute might merit these investments but the scientific innovation system in Canada is threatened by earmarked research investments chosen without peer review.