How Cathedral Music Trust used adaptive comparative judgement to assess grant applications
Case Study | Cathedral Music Trust
Background
Cathedral Music Trust is a charity that supports the valuable work of musicians who enrich lives through the power of cathedral music.
It seeks innovative and sector-leading ways to support high-quality musicianship in cathedrals and churches through its funding and delivery programmes, training, advocacy and research.
Its vision is to create a vibrant choral music scene that champions excellence and provides opportunities for people from all walks of life to thrive and develop.
Introduction
Cathedral Music Trust runs two annual grant programmes, collectively disbursing approximately £500,000 each year. As a major funder in the sector, the Trust regularly receives applications amounting to roughly three times the funds at its disposal, sometimes more. The Trust, therefore, has to make considered decisions over which applications to support. As these decisions represent a significant investment in the cathedral music sector, it is important to employ a robust, fair and transparent method of assessing applications.
The challenge
Recommendations for grant awards are agreed by the Trust’s Grants Assessment Panel and presented to the Board for approval.
The previous assessment method involved Panel members, all of whom are volunteers, scoring each application against a series of criteria. The scores awarded by an assessor for each criterion were totalled and the totals used to calculate a mean score for each application. The Trust’s staff team, which supports and advises the volunteer assessors, used the mean scores to rank the applications. This ranking was presented at the Panel’s assessment meeting, where it formed the basis for discussion of applications considered for funding.
Members of the Grants Assessment Panel raised a number of issues with this method of assessment:
- Judging applications against multiple criteria was very time consuming
- Assessors found it difficult to make absolute judgements
- Assessors did not have confidence in the consistency of their own judgements
- Because interpretation of the criteria and approach to scoring differed across assessors, there was limited confidence that the ranking accurately reflected the combined views of the Panel
The ranking was time consuming to produce but was nevertheless of limited use to the Panel. This meant the Panel was required to discuss a large number of applications in order to make robust recommendations for funding, rather than relying on the assessment exercise to identify the applications that most closely aligned with the Trust’s funding priorities.
The alternative
Rather than assess against a range of criteria, members of the Assessment Panel were asked to compare pairs of applications. The carefully chosen holistic statement asked the assessor to select the application that had the greater potential for impact, in accordance with the Trust’s three key priorities of excellence, participation and sustainability. Getting the holistic statement right was crucial to the outcome of the exercise.
The Trust first tested the platform using a small set of fictitious applications and used these applications to deliver training to members of the Assessment Panel. Having established that the platform met requirements and was straightforward to use, the Trust employed it to replace the scoring of its grant rounds in 2024 and 2025.
To its knowledge, Cathedral Music Trust is the first grant-maker to apply Adaptive Comparative Judgement (ACJ) to grant assessment. Grant assessment differs in some important features from other forms of assessment for which ACJ is suited, since an application’s position in the ranking does not equate to a ‘mark’, a place in a competition, or even the value of a financial award, but rather provides an indication as to whether assessors believe it warrants consideration for funding. For this reason, ACJ replaced only the first stage in the assessment process, and all applications were presented for discussion at the assessment meeting, informed by the outcome of the ACJ exercise.
The outcome
Using RM Compare, each application was judged 20 times by multiple judges. The time taken by the assessors over this exercise represented about half the time it took to assess the applications using the previous method.
Because the RM Compare platform automatically produces a ranking of items, it greatly reduced the administration time required between the completion of the assessment exercise and the assessment meeting. As moderation is built into ACJ, there were no concerns about consistency of scoring, and the assessors were more confident in using the ranking as a basis for discussion. This meant assessors requested discussion of fewer lower-ranking applications, and found decision-making at the assessment meeting easier.
ACJ produced a flatter ranking with a lower reliability than expected. This revealed that assessors sometimes found it difficult to make judgements between applications. Further training for assessors and adaptations to the application form to allow the differences between applications to stand out may help assessors when making judgements.
Minimising conscious and unconscious bias is important in the assessment process. Unlike academic assessment, it is virtually impossible to anonymise applications, as the location of the proposed work is a consideration in the assessment. Assessors, therefore, occasionally found they were presented with applications over which they had a conflict of interest. If the RM Compare platform allowed the administrator of the session to prevent particular judges from seeing items over which they had a conflict of interest, this would increase its potential for use within the grant assessment sector.
The future
Cathedral Music Trust intends to use ACJ to assess grant applications in future years. The Trust will continue to reflect on and adapt its application process to make it better suited to this method of assessment. Once the Trust feels it has developed a robust model for assessment using ACJ, it intends to share the process as an alternative to traditional grant assessment methods with others within the Trusts and Foundations sector.
