Formally Assessing the Quality of a Single Study

Assessment 2: Report — Formally Assessing the Quality of a Single Study

Purpose

This report will require you to assess the quality of an intervention trial based on formal Cochrane criteria covered in Module 2. The criteria are available to any researcher evaluating trials with a control group as part of an evidence synthesis. The Cochrane criteria are the quality assessment standard of choice across the health sciences, so familiarity with the criteria will enable you to become a more discerning consumer of reviews across a number of fields. You will draw conclusions about the quality of various features (‘domains’) of the study’s methodology as specified in the criteria.

Scenario and directions

Working for an NGO promoting workplace diversity, you are collaborating with a university on a systematic review of randomised controlled trials of workplace interventions to promote gender equality. The systematic review is up to the quality assessment stage, and you are one of two assessors. Your research team is using the Cochrane standards, and you need to use the standards for cluster-randomised controlled trials to assess the trial.

The assessment task is to complete an assessor form addressing the following domains, and justifying a judgement of risk of bias in each.

  • Domain 1a: Risk of bias arising from the randomisation process
  • Domain 1b: Risk of bias arising from the timing of identification or recruitment of participants in a cluster-randomised trial
  • Domain 2: Risk of bias due to deviations from the intended interventions (effect of assignment to intervention)
  • Domain 3: Risk of bias due to missing outcome data
  • Domain 4: Risk of bias in measurement of the outcome
  • Domain 5: Risk of bias in selection of the reported result
  • Overall risk of bias.

This assessment relates to the content presented in Module 2.

You should consult the Full Assessment 2 Brief for comprehensive instructions. Ensure that you consult the assessment rubric when preparing your submission.

Resources for interpreting the signalling questions in the form


Journal article

Carnes, M., Devine, P. G., Baier Manwell, L., Byars-Winston, A., Fine, E., Ford, C. E., Forscher, P., Isaac, C., Kaatz, A., Magua, W., Palta, M., & Sheridan, J. (2015). The effect of an intervention to break the gender bias habit for faculty at one institution: A cluster randomized, controlled trial. (Links to an external site.) Academic Medicine, 90, 221–230.

Submission format

You will submit the assessor form as a text document (doc, docx, pdf) via the link at the top of this page.

Academic Integrity Policy

By submitting an assignment via MyUni/Canvas, you are agreeing to the following statement:

I declare that all material in this assessment is my own work, except where there is clear acknowledgement and reference to the work of others. I have read the University of Adelaide’s Academic Integrity PolicyLinks to an external site. .

I give permission for any assessed assignments to be reproduced and submitted to other academic staff for the purposes of assessment and to be copied, submitted and retained in a form suitable for electronic checking of plagiarism.

Applicable policy linksLinks to an external site.

Questions?

Please post your questions on Assessment 2: Discussion.

 

Rubric

Assessment 2: Report – Formally Assessing the Quality of a Single Study

Assessment 2: Report – Formally Assessing the Quality of a Single Study
Criteria Ratings Pts
This criterion is linked to a learning outcomeDomain 1a: Interpretation and integration of supporting documents
3 Pts

Full marks

You interpret all signalling questions in line with the relevant supporting documentation—the crib sheet(s) and guidance document(s).

2 Pts

Partial marks

You interpret the majority (more than half) of the signalling questions in line with the relevant supporting documentation.

1 Pts

Partial marks

You interpret only a minority of the signalling questions in line with the relevant supporting documentation.

0 Pts

No marks

You do not interpret any of the signalling questions in line with the relevant supporting documentation.

3 pts
This criterion is linked to a learning outcomeDomain 1a: Argument and referencing of examples
3 Pts

Full marks

You use examples from the article to develop a clear argument justifying your choice of response option for each signalling question.

2 Pts

Partial marks

You justify your response options for all signalling questions, but, for a few questions, your argument is hard to follow—possibly, due to the absence of references to specific parts of the article.

1 Pts

Partial marks

You justify your response options for all signalling questions, but, for most questions, your argument is hard to follow.

0 Pts

No marks

Your justification of response option is unclear for all signalling questions.

3 pts
This criterion is linked to a learning outcomeDomain 1a: ‘Risk of bias‘ judgement and response options
1 Pts

Full marks

You correctly apply the algorithm for this domain to arrive at a domain-level ‘risk of bias’ judgement. You specify all response options in the rightmost column.

0 Pts

No marks

You make an error in applying domain-level algorithm and some response options in the rightmost column are missing.

1 pts
This criterion is linked to a learning outcomeDomain 1b: Interpretation and integration of supporting documents
3 Pts

Full marks

You interpret all signalling questions in line with the relevant supporting documentation—the crib sheet(s) and guidance document(s).

2 Pts

Partial marks

You interpret the majority (more than half) of the signalling questions in line with the relevant supporting documentation.

1 Pts

Partial marks

You interpret only a minority of the signalling questions in line with the relevant supporting documentation.

0 Pts

No marks

You do not interpret any of the signalling questions in line with the relevant supporting documentation.

3 pts
This criterion is linked to a learning outcomeDomain 1b: Argument and referencing of examples
3 Pts

Full marks

You use examples from the article to develop a clear argument justifying your choice of response option for each signalling question.

2 Pts

Partial marks

You justify your response options for all signalling questions, but, for a few questions, your argument is hard to follow—possibly, due to the absence of references to specific parts of the article.

1 Pts

Partial marks

You justify your response options for all signalling questions, but, for most questions, your argument is hard to follow.

0 Pts

No marks

Your justification of response option is unclear for all signalling questions.

3 pts
This criterion is linked to a learning outcomeDomain 1b: ‘Risk of bias‘ judgement and response options
1 Pts

Full marks

You correctly apply the algorithm for this domain to arrive at a domain-level ‘risk of bias’ judgement. You specify all response options in the rightmost column.

0 Pts

No marks

You make an error in applying domain-level algorithm and some response options in the rightmost column are missing.

1 pts
This criterion is linked to a learning outcomeDomain 2: Interpretation and integration of supporting documents
3 Pts

Full marks

You interpret all signalling questions in line with the relevant supporting documentation—the crib sheet(s) and guidance document(s).

2 Pts

Partial marks

You interpret the majority (more than half) of the signalling questions in line with the relevant supporting documentation.

1 Pts

Partial marks

You interpret only a minority of the signalling questions in line with the relevant supporting documentation.

0 Pts

No marks

You do not interpret any of the signalling questions in line with the relevant supporting documentation.

3 pts
This criterion is linked to a learning outcomeDomain 2: Argument and referencing of examples
3 Pts

Full marks

You use examples from the article to develop a clear argument justifying your choice of response option for each signalling question.

2 Pts

Partial marks

You justify your response options for all signalling questions, but, for a few questions, your argument is hard to follow—possibly, due to the absence of references to specific parts of the article.

1 Pts

Partial marks

You justify your response options for all signalling questions, but, for most questions, your argument is hard to follow.

0 Pts

No marks

Your justification of response option is unclear for all signalling questions.

3 pts
This criterion is linked to a learning outcomeDomain 2: ‘Risk of bias‘ judgement and response options
1 Pts

Full marks

You correctly apply the algorithm for this domain to arrive at a domain-level ‘risk of bias’ judgement. You specify all response options in the rightmost column.

0 Pts

No marks

You make an error in applying domain-level algorithm and some response options in the rightmost column are missing.

1 pts
This criterion is linked to a learning outcomeDomain 3: Interpretation and integration of supporting documents
3 Pts

Full marks

You interpret all signalling questions in line with the relevant supporting documentation—the crib sheet(s) and guidance document(s).

2 Pts

Partial marks

You interpret the majority (more than half) of the signalling questions in line with the relevant supporting documentation.

1 Pts

Partial marks

You interpret only a minority of the signalling questions in line with the relevant supporting documentation.

0 Pts

No marks

You do not interpret any of the signalling questions in line with the relevant supporting documentation.

3 pts
This criterion is linked to a learning outcomeDomain 3: Argument and referencing of examples
3 Pts

Full marks

You use examples from the article to develop a clear argument justifying your choice of response option for each signalling question.

2 Pts

Partial marks

You justify your response options for all signalling questions, but, for a few questions, your argument is hard to follow—possibly, due to the absence of references to specific parts of the article.

1 Pts

Partial marks

You justify your response options for all signalling questions, but, for most questions, your argument is hard to follow.

0 Pts

No marks

Your justification of response option is unclear for all signalling questions.

3 pts
This criterion is linked to a learning outcomeDomain 3: ‘Risk of bias‘ judgement and response options
1 Pts

Full marks

You correctly apply the algorithm for this domain to arrive at a domain-level ‘risk of bias’ judgement. You specify all response options in the rightmost column.

0 Pts

No marks

You make an error in applying domain-level algorithm and some response options in the rightmost column are missing.

1 pts
This criterion is linked to a learning outcomeDomain 4: Interpretation and integration of supporting documents
3 Pts

Full marks

You interpret all signalling questions in line with the relevant supporting documentation—the crib sheet(s) and guidance document(s).

2 Pts

Partial marks

You interpret the majority (more than half) of the signalling questions in line with the relevant supporting documentation.

1 Pts

Partial marks

You interpret only a minority of the signalling questions in line with the relevant supporting documentation.

0 Pts

No marks

You do not interpret any of the signalling questions in line with the relevant supporting documentation.

3 pts
This criterion is linked to a learning outcomeDomain 4: Argument and referencing of examples
3 Pts

Full marks

You use examples from the article to develop a clear argument justifying your choice of response option for each signalling question.

2 Pts

Partial marks

You justify your response options for all signalling questions, but, for a few questions, your argument is hard to follow—possibly, due to the absence of references to specific parts of the article.

1 Pts

Partial marks

You justify your response options for all signalling questions, but, for most questions, your argument is hard to follow.

0 Pts

No marks

Your justification of response option is unclear for all signalling questions.

3 pts
This criterion is linked to a learning outcomeDomain 4: ‘Risk of bias‘ judgement and response options
1 Pts

Full marks

You correctly apply the algorithm for this domain to arrive at a domain-level ‘risk of bias’ judgement. You specify all response options in the rightmost column.

0 Pts

No marks

You make an error in applying domain-level algorithm and some response options in the rightmost column are missing.

1 pts
This criterion is linked to a learning outcomeDomain 5: Interpretation and integration of supporting documents
3 Pts

Full marks

You interpret all signalling questions in line with the relevant supporting documentation—the crib sheet(s) and guidance document(s).

2 Pts

Partial marks

You interpret the majority (more than half) of the signalling questions in line with the relevant supporting documentation.

1 Pts

Partial marks

You interpret only a minority of the signalling questions in line with the relevant supporting documentation.

0 Pts

No marks

You do not interpret any of the signalling questions in line with the relevant supporting documentation.

3 pts
This criterion is linked to a learning outcomeDomain 5: Argument and referencing of examples
3 Pts

Full marks

You use examples from the article to develop a clear argument justifying your choice of response option for each signalling question.

2 Pts

Partial marks

You justify your response options for all signalling questions, but, for a few questions, your argument is hard to follow—possibly, due to the absence of references to specific parts of the article.

1 Pts

Partial marks

You justify your response options for all signalling questions, but, for most questions, your argument is hard to follow.

0 Pts

No marks

Your justification of response option is unclear for all signalling questions.

3 pts
This criterion is linked to a learning outcomeDomain 5: ‘Risk of bias‘ judgement and response options
1 Pts

Full marks

You correctly apply the algorithm for this domain to arrive at a domain-level ‘risk of bias’ judgement. You specify all response options in the rightmost column.

0 Pts

No marks

You make an error in applying domain-level algorithm and some response options in the rightmost column are missing.

1 pts
This criterion is linked to a learning outcomeOverall ‘risk of bias‘ judgement
1 Pts

Full marks

Your judgement for overall ‘risk of bias’ follows from the domain-level judgements in line with the rules in the crib sheet or guidance document for cluster-randomised controlled trials.

0 Pts

No marks

Your judgement for overall ‘risk of bias’ does not follow from the domain-level judgements.

1 pts
Total points: 43