Abstract and specific approaches to justify your alpha

9 minutes read
statistics • science • ethics

Following JYA, of which i am a co-author, the question how such a justification should look like is still open. We suggested in JYA to use a utility function. This seems reasonable, as long as this function is convex and can be optimized. Yet, consider the case that the function can not be optimized (i.e. the function is not convex), the cost and benefits are not tangible, or are themselves considered to need justificiation. See e.g.:

Introduction

I tried to adress this question from a more philosophical point of view in an older post, discussing the limits of justification. To paraphrase: Ultimate justificiation is not possible, and ends either in infinite regress, dogma or circular reasoning. The question is therefore: how much justification is sufficient? In this post, i will try to give some examples of "sufficient" justification. Whether these examples are sufficient enough for your taste, or whether you find them convincing enough to act upon, i can not decide. This is a question that can inevitably not be answered.(See #1)

Analysis of justification

Justification usually exhibits the following analytical form. You state a proposition, e.g. "We should use an alpha of 0.005". Subsequently, you might want this proposition to be coherent with your principles, and possibly convincing others. You justify this proposition by adding arguments. These arguments are empirical-normative bridging principles. They bridge an empirical fact (e.g. "studies passing a lower alpha exhibit higher evidence") with a normative claim (e.g. "more evidence is better"). Subsequently, this additional arguments are evaluated for consistency with your original propostion (e.g. "does it flow from requiring higher evidence to use this treshold?"), with their veracity (e.g. "do such studies really exhibit higher evidence?"), and for their coherency (e.g. "we have a public mandate to use resources only for studies with a chance for good evidence") or dissonance with other ethical principles (e.g. "using such a low threshold conflicts with our curiosity").

Consider that the process of justification usually follows this form:

An original proposition, its justification with an empirical-normative bridge, and the subsequent check of consistency, veracity and coherency. Some parts of the justification are therefore empirical (checking the veracity), others are ethical (citing principles in conflict or supporting).

Intangibility and consensus

Especially the ethical principles, and how they are weighted against each other feels often not "tangible". Which must be the case. Otherwise they would be the empirical, and not the normative, part of the justification process. In that regard, there is sadly no unique solution to the coherency-dissonance problem of ethical principles. You might feel comfortable with a solution, while others do not (See #1).

This does not mean that a justification lacks coherency. But it means there might not be enough consensus. Consensus is important for the credibility of a profession. The conensus is maintained and established in guidelines, journals, peer review, reputation, proper citation, science twitter, and many more similar venues and traditions.

Specific examples of justification

Traditional justification

We therefore start with the most obvious one: "We decided to use an alpha of 0.05, as such a threshold is the tradition in our field." Formally, what could be wrong with this justification? First, it is consistent with the original proposition. Yet, consider the veracity and coherency: Is the traditional consensus within your field strong, or is is under debate? Following JYA, i'd say it is no longer certain, whether there is a consensus. You might want to find out for your field by checking its current discussions. Furthermore, a tradition itself is only sufficiently justified if it is itself coherent with its own ethical principles. Unless you have full trust in your field, check the principles the tradition calls upon. Are they in agreement with your own principles?

Justification by citation

A similar one, if traditional consensus is under debate is citation: "We decided to use an alpha of 0.005, as such a threshold exhibits higher evidential value (see Benjamin et al. 2017)" Independent from this specific reference (See #2), justification by citation requires to look up the cited reference. Without access to the reference, consistency, veracity, and coherency can not be evaluated. This is especially the case if only the citation without repetition of the key arguments is used (e.g. "We used an alpha of 0.005 (see Benjamin et al. 2017)."

In that regard, it is similar to justification by tradition. Yet, the requirements are less strict, as partial consenus is already sufficient. Many traditions shortcut a full justification by allowing the reference to one of many justificational sub-traditions. Each of these branches within the field might be considered sufficiently justified, even if they are mutually exclusive.

Justification by listing normative principles

The minimal criteria for a justification by citation are that the cited justification is a sufficient justification. As discussed above, a sufficient justification needs to be consistent, veracious and coherent.

An inconsistent justification, where the proposition is not a necessary consequence of the empirical-normative bridging principle, can often be discarded a priori. Therefore, the main debate within a tradition focusses on the veracity and coherency of the bridging principles.

Science has a long experience with mutually excluse empirical theories, as long as they fulfill minimal criteria. I personally can not see how this should not be possible with mutually exclusive justifications of alpha (or beta) -thresholds.

Scientists are usually more acquainted with conflicting theories about empirical aspects, but not about ethical aspects. Furthermore, scientists are often comfortable to make decisions based on uncertain veracity of an empirical proposition. After all, it is their job to ascertain the veracity of such a claim.

Therefore, it can reasonably expected that the confusion about the justification of alpha will be mainly about the ethical principles. Scientists are often not trained or equipped with the rhetorical and analytical tools for evaluating the coherency of ethical principles. Additionally, because justification requires an empirical-normative briding principles, creating your own justification requires an additional empirical claim.

Consider a justification like "We decided to use an alpha-beta ratio of 0.05/0.8, as we consider this to be reasonable trade-off between curiosity, evidence and economical constraints." Such a justification claims that you can operationalize what constitutes a reasonable trade-off, that this is true, and that curiosity, evidence, and economical aspects are the ethical principles that justify your decision.

To many scientists trained to be sceptical, this can feel like a too broad and at the same time too narrow perspective. Doesn't the same proposition follow from more than these listed principles? And can you validate the veracity of your empirical claims? Often, this is not possible. The utility of justification by normative principles is therefore often limited to few applications.

Justification by adding bridging principles

Extensive, principled justifications can appear unnecessarily complicatet. It can open up additional issues which might need further justification, but which you do not really oppose. Such a justification is rhetorically difficult.

A simpler approach to justification is claiming something which is not under dispute: "We used an alpha-beta ratio of 0.05/0.8, considering this returns a likelihood ratio of 16." This is sadly not really a justification - while it is consistent, it does not contain any briding principles.

The better alternative is to state more than one bridging principle, under which a justification can then become sufficient. Some examples: "We used the traditional alpha threshold of 0.05, as we could not define a sufficiently convex cost-benefit function." or "We used an alpha threshold of 0.01, which can be considered stricter as traditional, as we were performing an explorative data analysis and wanted to reduce our risk of false positives."

As you can see, there is no longer a clear connection to highly specific normative principles. Instead, you establish your own "tradition", under which you evaluate coherency. If you use once an alpha of 0.01 for the given reason, you should follow this rule again, if the same conditions apply. Therefore, veracity, consistency and coherency can be sufficiently fulfilled.

Objective justification

Objective justification is not possible, unless you are a foundationalist. In that branch, a justification is sufficient, if it successfully references one specific normative principle, e.g. truth. For several reasons, i do not believe that this is actually possible. First, these abstract terms are not uniquely linked to a specific proposition. For example, truth can justify more than one specific alpha-threshold. If it is not unique, it is not objective. Second, the abstractness of the term often only mirrors a traditional consensus or masks a dissonance. Consider freedom, and what diferent people think it means. Therefore, i consider going for objective justification not reasonable.

Conclusion

I hope you found this discussion of several appraoches towards justification helpful. I personally favour justification by additional bridging principles.

Consider also that justification is linked to motivation. Making the motivation behind your decision explicit is already half the justification. Don't overdo it. Justifications are not better by listing more reasons and principles. Why? Inevitably, all attempts for ultimate justification will fail. As long a you practice what you preach with a minimum of coherency, and create some links to your tradition, this makes your justification likely already sufficent. Good luck with JYA!

FN1: Sufficient and necessary justification

A connected issue is that justification is often considered to have some intrinsic motiviation value. That is, once a proposition is sufficiently justified, it should motivate somebody to act according to that justified proposition. That is, a sufficient justification should be convincing. This is not necessarily the case. There are philosophical differences between justification and the motivation to act accordingly. Besides philosophical differences between justification and motivation, the two are different but related social and psychological processes. You usually justify your decision only if you experience some kind of dissonance. Otherwise, justification would not be necessary.

Therefore, it is also an open question whether justification is necessary. Consider that, often at least one person was motivated to act based on a proposition. You might have published before using an alpha at 0.05. Afterwards, you or others might claim that this choice was not sufficiently justified. Do you then have to justify your decision, or do you have to also convince them with your justification? I would say that justification does not necessarily require a motivational influence. It does not have to convince, but it must be convincable. A possible critera, rooted in rationality, but also in motivational psychology, is as follows: A justification can be considered sufficient if it is coherent, and further justification is necessary, if its coherency is found to be flawed. Please note that this is different from a consensus, as again, not all people need to consider a coherent argumentation convincing (i.e. are motivated to act upon it), if they exhibit different preferences.

FN2: justification

I discussed some aspects of this specific justification above, and more can be found in detail in JYA