How do you calculate the sample size for a study to Validate the surgical risk calculator?

Hello Raunak,

The answer depends on what you mean by "validate."

Using this calculator as an example: https://riskcalculator.facs.org/RiskCalculator/, the model for which apparently was built via logistic regression:

0. Note that there are risks estimated for a variety of surgery outcomes. Thus, the "final" risk model for one specific outcome may well have been different from that for any other outcome.

1. If you simply want to verify that a given IV is related to a DV (some specific surgical outcome), then ordinary calculators such as the freely available g*power. Here's a link that explains how to do it: https://statisticseasily.com/2022/05/19/sample-size-logistic-regression/ Note that the result will differ based on the attributes of the IV you enter. Therefore, you should consider all the candidate IVs (and all of the candidate DVs), and base your result on the specific IV-DV combination which yields the largest requisite N.

2. If you want to verify that all of the current model IVs are significant contributors to the model and that the overall model fares better than chance, then the N value you found for #1 above would be a lower bound. Note that 15 IVs (for example) implies that you're aiming to evaluate 16 hypothesis tests: One for significance of the overall model and 15 for the respective IVs. Hence, you'd be wise to control type I error risk by setting your significance level lower and recompute as in #1.

3. If you want to know whether the current model functions differently than a competing model, then you can compare the classification accuracy of the two models. You could do this as: (a) OR of correct classification by model, or identically as a simple logistic regression of model (IV) and correct classification (DV); or (b) compare Cohen kappa values for the two models.

As you might expect, in order for small differences to be statistically significant, you will need larger N.

4. If you mean to investigate something different than the above list, perhaps you could explain further your research goals.

Good luck with your work.

Isaac Oluwasegun Adeniyi

Validating the applicability of a surgical risk calculator in a specific context, may require a slightly different approach to determining the sample size. You can use any of these below:

Define the objectives: Clearly define the objectives of your study, including the specific aspects of the surgical risk calculator you want to validate and the performance metrics you will assess. For example, you might want to validate the accuracy of predicted surgical outcomes, such as mortality or complications, in the Indian hospital setting.

Consider the validation study design: Determine the most appropriate study design for your validation study. This could involve prospective data collection, retrospective analysis of existing data, or a combination of both. Consider factors such as data availability, feasibility, and ethical considerations.

Identify the validation metrics: Determine the performance metrics that will be used to assess the applicability of the surgical risk calculator in the Indian hospital context. This could include measures such as sensitivity, specificity, positive predictive value, negative predictive value, or area under the receiver operating characteristic curve (AUC-ROC). Select the metrics most relevant to your study objectives.

Assess the existing literature: Conduct a thorough literature search to identify any existing studies or publications that have validated similar surgical risk calculators or prediction models in similar settings or patient populations. These studies can provide valuable insights into sample sizes used and statistical methods employed in similar validation efforts.

Calculate the sample size: Once you have defined the performance metrics and consulted with a statistical expert, you can proceed with calculating the sample size. The calculation will depend on factors such as the chosen statistical test, desired level of precision, expected effect size, confidence level, and power. The statistical expert can assist you in selecting the appropriate formula or software to calculate the sample size based on your study design and objectives.

However, calculating the sample size for a study to validate a surgical risk calculator typically involves statistical considerations. The specific approach may vary depending on the type of validation study and the statistical methods employed.

General steps involved are:

Determine the desired level of precision: The desired level of precision refers to the acceptable margin of error in the validation study. It is typically expressed as a confidence interval or a maximum allowable difference between the estimated and true values. This is often based on clinical or practical considerations.

Choose the desired level of confidence: The desired level of confidence represents the probability that the estimated results from the study will fall within the chosen confidence interval. Commonly used confidence levels are 95% or 99%.

Estimate the expected performance of the surgical risk calculator: Based on previous research or pilot data, an estimate of the performance of the surgical risk calculator can be determined. This may include parameters such as sensitivity, specificity, or predictive values.

Select the appropriate statistical test: The choice of statistical test depends on the validation study design and the performance metrics being assessed. Commonly used tests include the chi-square test, t-test, or regression analysis.

Conduct a power analysis: Power analysis helps determine the required sample size to achieve adequate statistical power. Power refers to the ability of the study to detect a true effect if it exists. The power analysis considers factors such as the expected effect size, significance level, and desired power level.

Calculate the sample size: Using the information gathered from the previous steps, you can use statistical formulas or power analysis software to calculate the sample size needed for the validation study. The calculation may involve equations specific to the chosen statistical test and the desired level of precision and confidence.

Example of calculating sample size for validating a surgical risk calculator in a specific context:

Let's assume the following parameters for our example:

Desired level of precision: We want our estimate of the AUC-ROC (performance metric) to have a margin of error (half-width) of 0.05.
Desired level of confidence: We choose a 95% confidence level.
Expected AUC-ROC: Based on preliminary data or previous studies, we estimate that the expected AUC-ROC of the surgical risk calculator is 0.80.
Power: We aim for a power level of 80%, which corresponds to an 80% chance of detecting a true effect if it exists.

To calculate the sample size, we can use the following formula for estimating sample size for an AUC-ROC:

n = [(Z_α/2 + Z_β) / (AUC_0.5 - AUC_1)^2] + 1

Where:

n: Required sample size
Z_α/2: The critical value of the standard normal distribution at the desired level of confidence (e.g., 1.96 for a 95% confidence level)
Z_β: The critical value of the standard normal distribution at the desired power level (e.g., 0.84 for an 80% power level)
AUC_0.5: The null hypothesis AUC value (e.g., 0.5 for no discrimination power)
AUC_1: The alternative hypothesis AUC value (e.g., 0.80 for the expected AUC of the surgical risk calculator)

Using these values in the formula, we can calculate the sample size:

n = [(1.96 + 0.84) / (0.5 - 0.80)^2] + 1

n = [2.80 / (-0.30)^2] + 1

n = [2.80 / 0.09] + 1

n = 31.11 + 1 ≈ 32

So, based on this calculation, you would need a sample size of approximately 32 patients to validate the surgical risk calculator for predicting postoperative complications in the given context.

Is receiving Invitation to submit a article from the journal is big deal ???

Can any tell me or suggest any research article or review about the concentration of peptide dissolve in system water and given to Adult Zebrafish?

Electroactive surface area of bare and the synthesized material?

Is it reliable to use toluene as a solvent in measuring density by liquid displacement method?

Are hydrogels really important for food processing sector ?

Fluorescence spectra problem during protein ligand interaction??

How can i select suitable gene sequence at Ring stage of Plasmodium Falciparum for primer design?

How to identify the compounds present in the methanolic extract of Artemisia leaves using LC-MS?

How to calculate sample size?

Which are the post harvesting processes of turmeric which results in maximum loss of curcumin content of Turmeric?

How optimize the Size Exclusion Chromatography?

No title

What test should I use to compare two or three means, if data is not normally distributed but sample size is > 30 in each group?

Is it necessary to report effect size (ES) in statistics of biomedical research given that we already report p values routinely?

Why there are two peaks while using the mie plot of Silver nanoparticle?

How to calculate Z (the number of molecules in a unit cell) for calculating theoretical density?

How to handle studies which report perfect effect sizes in a meta-analysis using Pearson's r?

Sample size determination for non-parametric tests?

How to determine a priori sample size for t-test with unequal means?

What are alternatives to factor analysis for a small sample?