When is it justified to claim that a practice or policy is evidence-based? Reflections on evidence and preferences

First condition

I hold the position that an individual or organisation is justified in claiming that a specific practice or policy is evidence-based only if they possess comparative evidence about the effects of the specific practice or policy in comparison to the effects of at least one alternative practice or policy. This implies that it would be unjustified for the individual to make a claim such as: ‘This practice/policy is evidence-based, but I do not possess any evidence about its effects in comparison with the effects of any other practices/policies’. Similarly, it would be unjustified for the organisation, via someone who has the authority to speak on its behalf (such as an organisational leader or a spokesperson), to make a claim like the following: ‘This practice/policy is evidence-based, but the organisation does not possess any evidence about its effects in comparison with the effects of any other practices/policies’.

Let us consider a scenario where an individual or organisation claims that utilising restorative justice conferences – a specific type of facilitated dialogues between victims, offenders, and their supporters (Zinsstag and Vanfraechem, 2012) – is an evidence-based practice or policy for addressing crime. For this claim to be justified, the individual or organisation must possess evidence that compares the effects of restorative justice conferences to those of at least one alternative crime management practice or policy, for instance generated through a randomised controlled trial or another method used within the fields of evidence-based practices and policies. It could, for example, consist of comparative evidence on the relative effects of restorative justice conferences and prosecution in court, which are two different approaches to addressing crime (for such evidence, see Sherman et al, 2015). The comparative evidence could also pertain to the relative effects of a specific treatment compared to a ‘no treatment’ alternative, such as when a specific medicine is compared to a placebo. In summary, justified claims about practices or policies being evidence-based always rely on comparative evidence, whether it is comparing different treatments or comparing treatment against a no-treatment alternative. Consequently, individuals and organisations are never justified in claiming that a specific practice or policy is evidence-based if they do not possess evidence about any alternatives. However, it is unlikely that they will have comparative evidence about all possible alternatives. This suggests that it would be more accurate for them to claim that ‘this practice/policy is evidence-based in comparison with this (or these) specific alternative(s)’, rather than simply stating ‘this practice/policy is evidence-based’.

In connection with the first condition, it should be noted that some may argue that the requirements concerning evidence should be increased by raising the threshold for which kind of evidence counts. For example, some may argue that only scientific evidence should be considered, and perhaps only specific types of scientific evidence. Instead of adopting such an approach, which would exclude specific epistemic communities (Haas, 1992), such as those that adhere to indigenous knowledge (Kirmayer, 2012; Macfarlane and Macfarlane, 2013), from the realm of evidence-based practices and policies, I rather recommend a transparency approach, as elaborated on later.

Second condition

I hold the position that an individual or organisation is justified in claiming that a specific practice or policy is evidence-based only if this practice or policy is supported by comparative evidence, according to at least one of the individual’s or organisation’s preferences in the given practice or policy area. To argue for this position, I will begin by asserting that the individual or organisation is justified in making such a claim only if the evidence, at least to some extent, supports the specific practice or policy according to the individual’s or organisation’s perspective. Thus, it would be unjustified for the individual to make a claim like: ‘This practice/policy is evidence-based, but I do not believe that the evidence supports this practice/policy’. Similarly, it would be unjustified for the organisation, via someone who has the authority to speak on its behalf, to make a claim stating: ‘This practice/policy is evidence-based, but the organisation does not believe that the evidence supports this practice/policy’.

It is crucial to recognise that determining whether the evidence supports a specific practice or policy according to the individual or organisation is fundamentally a matter of their preferences in the given practice or policy area. For example, let us consider a scenario where the specific practice or policy is the adoption of a particular model of social work, and where the individual’s or organisation’s sole preference regarding social work is low economic cost. In this case, if the evidence demonstrates that the specific practice or policy is cheaper than an alternative practice or policy, the individual or organisation would consider this evidence as supporting the specific practice or policy rather than the alternative. However, if the individual’s or organisation’s preference for some reason is that social work should be expensive, then the same evidence would be viewed by the individual or organisation as supporting the alternative practice or policy that is more expensive. In the latter case, according to the view of the individual or organisation, the evidence would not provide any support for the cheaper practice or policy. This example serves to highlight that whether it is justified for the individual or organisation to claim that the specific practice or policy being evidence-based is always contingent upon both evidence and preferences. For the individual or organisation to make a justified claim, it is necessary that the evidence supports the specific practice or policy according to at least one of the individual’s or organisation’s preferences. Or, put differently, for the individual or organisation to make a justified claim, the specific practice or policy must be better in some aspect, as determined by the evidence, and interpreted through the lens of the individual’s or organisation’s preferences.

In connection with the second condition, it should be noted that some may argue in favour of raising the requirements regarding preferences. For example, some may argue for the introduction of an objectivity requirement, according to which only preferences that are objectively appropriate should count. Instead of adopting such an approach, which will introduce challenges in determining objective appropriateness and potentially exclude specific communities from the realm of evidence-based practices and policies, I suggest the aforementioned transparency approach, as further explained later.

Third condition

Finally, I hold the position that an individual or organisation is justified in claiming that a specific practice or policy is evidence-based only if the individual or organisation can provide a sound account for the earlier discussed support, by explaining the evidence and preferences that lay the foundation for the claim. If this is true, then it implies that it would be unjustified for the individual to make a claim such as: ‘This practice/policy is evidence-based, but I cannot explain which evidence I possess or what my preferences are regarding this practice/policy’. Similarly, it would be unjustified for the organisation, via someone who has the authority to speak on its behalf, to claim that: ‘This practice/policy is evidence-based, but the organisation cannot explain which evidence it possesses or which preferences it has regarding this practice/policy’.

Let us consider a scenario where someone makes a claim only because they believe it is true (the belief in a claim’s truth can, of course, be based on many different things, such as empirical evidence, personal experience, logical reasoning, and intuition or gut feeling). As I cannot imagine any counterexamples, I believe it holds true as a general principle that if such a claimant cannot provide any sound account for their belief in the claim’s truth, then it is not justified for them to make the claim. For example, if the claim is that ‘there are black swans’, the claimant must be able to provide a sound account for why they believe black swans exist in order for it to be justified for them to make the claim. In the concrete example, they could provide the account by explaining that they have personally seen several black swans in Australia and have also heard several experts affirming the existence of black swans. In connection with the general principle, it should be noted that someone may be justified in making a specific claim, even though someone else is not justified in making the same claim.

It follows from the general principle that it is only justified for the individual or organisation to claim that a specific practice or policy is evidence-based if they can provide a sound account for their belief in this claim. For example, let us consider a scenario where a government department claims that a new educational policy is evidence-based. If someone asks why the department makes this claim, a spokesperson for the department may provide an explanation on the department’s behalf. They might explain that the department claims the new educational policy is evidence-based because the available scientific evidence shows that this policy leads to better reading skills among pupils compared to a previous educational policy. Additionally, the spokesperson may elaborate that officials from the department have diligently gathered and thoroughly analysed the available scientific evidence pertaining to both policies. Someone may then ask a pertinent follow-up question: ‘But why does this make the new educational policy evidence-based, that is, supported by evidence?’ The spokesperson may then respond as follows: ‘Better reading skills are preferred over poorer ones, according to the department’s stance. The department claims that the new policy is evidence-based because it is the better policy according to the department’s careful evaluation of the scientific evidence and viewed through the lens of the department’s preference for strong reading skills’. This account is sound because it explains both the evidence and the preferences that lay the foundation for the claim. If the spokesperson had responded to the follow-up question by saying ‘just because’ or ‘because I said so’, or had declined to offer a response, reflecting that the department was unable to provide a sound account, then the department would be unjustified in claiming that the policy was evidence-based.

In general, to be sound, the account that is required by the third condition must always address both the evidence supporting the practice or policy and the preferences that form the basis for evaluating this support. This is because the claim ‘this practice/policy is evidence-based’, unlike the claim ‘there are black swans’, is not purely empirical. It possesses a normative dimension as it implies that, according to the claimant, the practice/policy is supported by the evidence, meaning that the claimant perceives it as better than the alternative, at least in some regard. The normative assessment of support is contingent upon the claimant’s preferences, emphasising the subjective nature of such assessments. Below, I will delve further into the matter of subjectivity and objectivity.

First implication

The first implication is that others can be justified in claiming that a practice or policy is evidence-based, even if you are not justified in making the same claim. If you are not justified in making the same claim, then it is because the same evidence is not available to you, and/or because you do not share the same preferences, and/or because you cannot provide a sound account for the claim that the practice or policy is evidence-based. Indeed, there may be instances where others are justified in claiming that practices or policies, which you find utterly repulsive, are evidence-based. For example, one can imagine a scenario where an organisation claims that its practice of imprisoning victims is evidence-based because the organisation prefers victims to suffer as much as possible for their offenders’ crimes, and because the organisation possesses evidence showing that victims suffer more when subjected to imprisonment compared to other forms of punishment.

It should be noted that if the threshold for preferences is raised by incorporating an objectivity requirement, as mentioned in the section on the second condition, it may be possible to mitigate scenarios where others are justified in claiming that practices or policies, which you find repulsive, are evidence-based. However, adding an objectivity requirement can pose a challenge since determining the objective appropriateness of preferences can be difficult, if not impossible. In fact, I personally do not know who holds the authority to decide on objective appropriateness, nor how this is decided. Thus, instead of an objective appropriateness approach, I argue in favour of adopting a transparency approach that emphasises transparency regarding both evidence and preferences. If the organisation provides a clear explanation of their evidence and preferences in the abovementioned scenario, it enables you to understand their claim that incarcerating victims is evidence-based. It also allows for open discussion of differing positions based on individual preferences. You can express your position by stating that you do not share the organisation’s preferences, and you can engage in a dialogue about potential benefits and problems regarding different preferences. This transparency approach aligns well with the fulfilment of the third condition, which emphasises the need for providing explanations of both evidence and preferences.

Second implication

The second implication is that being justified in claiming that a practice or policy is evidence-based does not imply that this practice or policy ought to be implemented. Clearly, you may hold the position that practices and policies, which others are justified in claiming are evidence-based, ought not to be implemented. This position may stem from the fact that you do not share the preferences that underpin their claim, and/or it can be related to the evidence that forms the basis for the claim. For example, you may have more trust in other studies that demonstrate different results, or you may completely dismiss the results because you do not have any trust in the kind of evidence that is presented. For example, the evidence may stem from a cultural community that has ‘ways of knowing’ that do not rely on the kind of observational and experimental measures that are predominant in the fields of evidence-based practices and policies (Kirmayer, 2012). You might lack trust in such forms of evidence, or conversely, if you belong to such a community, you may have limited trust in the more conventional forms of evidence. However, instead of excluding any forms of comparative evidence from the realm of evidence-based practices and policies, where the process of exclusion may be based on subjective and debatable preferences, I rather argue in favour of the transparency approach presented above. Establishing transparency about the specific nature of the evidence that forms the foundation of the claim may potentially lead to valuable dialogues about the potential value of different types of evidence within the fields of evidence-based practices and policies. It also provides a solid basis for saying: ‘I find it insignificant that you are justified in claiming that the practice or policy is evidence-based because I do not believe in the validity of the kind of evidence you possess’.

It can also be the case that you do not believe a specific practice or policy ought to be implemented, even if you yourself are justified in claiming that this practice or policy is evidence-based. Let us imagine a scenario where, within the context of crime management, you have a preference for policies that result in low recidivism, high victim satisfaction, and low economic cost. Furthermore, let us assume that you possess evidence from a study showing that Policy A leads to a 60% lower recidivism rate than Policy B, while Policy B yields a 20% higher victim satisfaction rate and 30% lower economic cost than Policy A. In this scenario, assuming you also meet the third condition about providing a sound account, it would be justified for you to claim that both Policy A and Policy B are evidence-based. This is because both policies are supported by the evidence according to at least one of your preferences. Specifically, you are justified in claiming that Policy A is evidence-based in relation to recidivism, while Policy B is evidence-based in relation to victim satisfaction and economic cost. However, you may hold the view that only Policy A should be implemented because you assign greater weight to the 60% lower recidivism rate compared to the 20% higher victim satisfaction and 30% lower economic cost. Thus, in this scenario, you are justified in claiming that Policy B is evidence-based, even though you do not believe it should be implemented.

Third implication

The third implication is that the individual’s or organisation’s preferences ought to guide the collection of evidence to assist them in identifying what the best practices and policies are based on their normative stance. For example, if a research team collaborates with an organisation that wants to implement either Policy A or Policy B, depending on which one it finds superior, there should be a dialogue between the team and the organisation for the team to understand the organisation’s preferences in the given policy area. If the organisation holds preferences across multiple dimensions, such as recidivism, victim satisfaction, and economic cost, then the research team should ideally collect comparative evidence specific to all these dimensions. However, if there are resource restraints on the collection of evidence, which is often the case in practice, the research team should engage in a dialogue with the organisation to determine which preferences are most important for its decision about implementation. If the organisation places significantly greater importance on potential differences in recidivism compared to potential differences in victim satisfaction and economic cost, then it would be reasonable to prioritise the collection of evidence regarding recidivism if it is only possible to collect evidence in one dimension. In short, the collection of evidence should be guided by the organisation’s preferences, including how much relative weight it ascribes to different preferences. This guidance requires dialogue, and the call for dialogue aligns well with the transparency approach advocated in this article. I hope that this approach, along with the outlined conditions for being justified in claiming that a practice or policy is evidence-based, will be the subject of fruitful future debates.