Sensibility and understandability
After grading all 12 examples, the judges were asked 16 questions regarding the sensibility and understandability of the approach. Each question consisted of a statement and five response options: strongly disagree, disagree, not sure, agree, and strongly agree. Eleven people completed this questionnaire. The questionnaire was adapted from Feinstein [15] and the 16 statements were:
1. The approach is applicable to different types of interventions, including drugs, surgery, counselling, and community-based interventions.
2. The approach is clear and simple to apply
3. The information that is needed is generally available.
4. Subjective decisions are generally not needed.
5. All of the components included in each of the five types of judgements should be included
6. There are not important components that are missing for any of the five types of judgements.
7. The ways in which the components are aggregated for each of the five types of judgements are clear and simple.
8. The ways in which the included components are aggregated are appropriate for each of the five types of judgements.
9. The categories are sufficient to discriminate between different grades for each of the five types of judgements.
10. The approach successfully discriminates between different grades of evidence.
11. The approach successfully discriminates between different grades of recommendations.
12. The overall quality of evidence is clear and understandable.
13. The balance between the benefits and harms is clear and understandable.
14. The recommendation is clear and understandable.
15. The way in which the overall quality of evidence was graded is better than other ways of doing this with which I am familiar.
16. The way in which the recommendation was graded is better than other ways of doing this with which I am familiar.