Decision Criteria

The Decision Criteria area contains 2 sections:

Decision Criteria

Decision Criteria specify how the results of the analyses should be handled to define what success looks like. E.g., is it based on a p-value or the upper limit of a confidence interval or does it depend on the joint significance of two analyses? At least one decision criterion must be specified for a project. The Decision Criteria Description field in the Decision Criteria Details section can be used to add a short description of the decision criterion if desired.

The different types of decision criteria are explained below.

Single Decision Criteria

To evaluate study outcomes, a KerusCloud project must contain at least one Single Decision Criterion.

To create a Single Decision Criterion, click on Single in the Decision Criteria Details section, type an identifier into the Decision Criteria Identifier field and use the drop-down menu to select the analysis on which the decision criterion is to be based. In KerusCloud, a Single Decision Criterion compares an analysis metric (specified in the left-hand side of the Decision Criteria Definition) to a fixed value or another metric (specified in the right-hand side of the Decision Criteria Definition).

Depending on the selected analysis type, a list of analysis metrics are available for comparison including p-value, odds ratio, difference in means, posterior probabilities, and the lower/upper confidence interval limit. Some metrics will require specification of options such as sidedness of the test and the specific subgroups for comparison. Further information regarding using specific analysis types in KerusCloud can be found under the analysis section of the user guide.

Use the Type menu on the right-hand side to indicate whether the left-hand side metric should be compared to a fixed value (“Value”) or to another metric (“Metric”). In the case of the “Metric” type, the comparative metric should then be defined as for the left-hand side.

Select the logic used to compare the left-hand side metric to the value or metric defined on the right-hand side from the options in the central column of the Decision Criteria Definition section. Available logic options are: less than (<), less than or equal to (<=), greater than (>), greater than or equal to (>=), equal to (==) and not equal to (!=).

Subgroups

If applicable, subgroup options will depend on the variables used in the analysis. For example, when creating a decision criteria based on an ANOVA analysis where the grouping variable has 3 levels, you will be required to choose 2 of the 3 subgroups. If you are performing a linear regression with two categorical covariates, one with 2 levels and one with 3 levels, then you will have 6 possible combinations of categories to choose from.

Metrics which have a direction will be calculated based on the order in which the subgroups are specified, i.e. whether they are considered the Target or Reference Subgroup. For example, a difference in means will be calculated as the mean of the Target Subgroup minus the mean of the Reference Subgroup, while an odds ratio will be calculated such that the odds of the Target Subgroup is in the numerator and the odds of the Reference Subgroup is in the denominator.

Bayesian Hypothesis Testing

For metrics such as P(Difference in Means), the Cutoff Value and Comparison Logic are used to calculate the posterior probability. If Cutoff Value equals “2” and Comparison Logic equals “>”, the metric calculated is P(Difference in Means > 2).

Regression Analyses

When creating a Decision Criteria based on a Regression analysis, after selecting the metric of interest from the Metric drop-down, one of the model covariates must be selected from the Covariate drop-down list.

For example, where a linear regression analysis has been defined based on a list of covariates including “Treatment” (a group variable), selecting the “Treatment” covariate allows you to investigate the difference between two subgroups of the covariate selected within the context of the linear regression model you have defined. For example, if you wish to estimate the selected metric, such as the p-value, for the difference between the Placebo and Active treatment groups, adjusting for other covariates.

You may also wish to investigate the difference between the two subgroups, when the categorical variable of interest, such as “Treatment”, is included as part of an interaction with a continuous covariate. For example, you may want to use Placebo and Active treatment groups at timepoint 10, where time has been included as a continuous covariate and in an interaction term with Treatment variable.

Where the specified covariate is an interaction between at least one continuous variable and a level-based variable, a Covariate Term Value must be specified for the continuous variable (in the numeric input box labelled “At Variable Identifier”). This will be used to calculate the metric value.

The Covariate Term Value is potentially applicable to all regression types and to all metrics on those regressions except for the Number in Analysis regression metric in mixed model regressions. Mixed Models where “Time” is included as continuous is a special case, where the expectation is that there are multiple measurements on the same participant at different times. When “Time” is included in the model as part of an interaction with “Treatment”, it allows the user to investigate the difference between two treatment groups at a specified point in time, as the difference between groups is expected to change with time.

Combined Decision Criteria

Combined Decision Criteria allow success to be defined based on the outcome of multiple decision criteria. At least two Single Decision Criteria must be created before a Combined Decision Criterion can be created.

To create a Combined Decision Criterion, click on Combined in the Decision Criterion Type section and enter an identifier into the Decision Criterion Identifier field.

Use the drop-down menus to select the identifiers of existing decision criteria and use the +/- buttons to add or remove up to eight decision criteria.

Choose a logical operator (either AND or OR) at the start to determine how the outcomes of all selected decision criteria will be combined.

Next to each of the decision criteria selection menus is an “Is/Not” toggle button. The default selection is the “Is” button, click the “Not” button to activate the “Not” condition for the criterion. When the “Not” condition is activated, the button will change to navy. This will indicate that in order for the combined criterion to be met, the single criterion marked with “Not” should not be met. Conversely, clicking the “Is” button, will revert the condition for the criterion, resetting the button back to the navy “Is” state.

The Combined Decision Criterion allows users to implement a fixed sequence testing strategy by enabling the comparison of multiple combinations of individual Decision Criteria. In fixed sequence testing, the significance of a specific Decision Criterion at a given alpha is evaluated in the context of its position within a predefined sequence.

For example, consider five Decision Criteria: A, B, C, D, and E. To test whether Decision Criteria C is significant when it appears third in a sequence, users would create and compare the Combined Decision Criteria:

“A & B & C” – tests whether A, B, and C are all significant
“E & D & C” – tests whether E, D, and C are all significant

By comparing such Combined Decision Criteria, users can determine whether a particular Decision Criterion (e.g., C) retains significance across different sequences.

Multiplicity Decision Criteria

A Multiplicity Decision Criterion can be created when at least two Single Decision Criteria with p- value metrics have been defined.

To create a Multiplicity Decision Criterion, click on the Multiplicity type and enter an identifier. Then choose at least two existing decision criteria from the Decision Criteria Included menu. This menu contains all existing Single Decision Criteria with p-value metrics. In the Decision Criteria Definition section, choose a Multiplicity Adjustment Method and Logic from the drop-down menus and enter the Error Rate/False Discovery Rate.

The set of p-values for the selected Single Decision Criteria will be adjusted for multiple testing using the selected multiplicity adjustment method. The available options are the Holm and Hochberg, Benjamini-Hochberg, Benjamini-Yekutieli and Bonferroni adjustment methods.

The success of the Multiplicity Decision Criterion can be defined as requiring all adjusted p-values to be less that the target error rate by selecting the AND logic or defined as requiring at least one adjusted p-value to be less than the target error rate by selecting the OR logic option. When the SPECIFIC logic option is selected, the user will be required to select one of the previously selected Single Decision Criteria from the Specific Decision Criterion drop-down, for which the adjusted p-value will be returned as the Multiplicity Decision Criteria result. When evaluating this type of Decision Criteria, adjusted p-values are compared to the specified Error Rate.

Behaviour Under Adaptive Designs

If a Group Sequential Design has been defined for a project this can affect the outcome of a Single Decision Criterion. If the Efficacy Rule for the Group Sequential Design and the decision criterion left-hand side metric are both based on the same analysis, analysis metric, metric covariates (if applicable), subgroups (if applicable, and with the subgroups in the same order) and have identical metric options then the analysis metric on the left-hand side of the decision criterion will be compared to the adjusted alpha value at instead of the right-hand side value entered in the decision criterion. The adjusted alpha value for each interim time point is calculated by the group sequential alpha spending specified in the Group Sequential Design. If any details of the analysis and metric are not identical for the Group Sequential Design and Single Decision Criterion, then the right-hand side value or metric entered in the decision criterion will be used to evaluate success.

Selecting Futility or Sample Size Re-Estimation will only affect the final sample size for each simulated trial. The decision criteria will be based on the metric as calculated at the end of each trial, which will contain the number of participants as determined by the futility rule or the updated sample size.

Decision Criteria Review

This provides an overview of the number of decision criteria defined and whether they are Single, Combined or Multiplicity criteria and displays the number of Kerus Credits which will be used.

You can choose from three different speed options to run a task. After selecting a speed, the credit usage for the task will be updated, allowing you to select the option which best balances speed and cost for your needs. Find out more.

Once happy with the set up check the number of Kerus Credits which will be used and click on the GO button to generate the results and obtain the probability of success for each of the criteria. The success of all criteria will be evaluated for all scenarios, treatment allocations, designs and sample

sizes requested.