Permutation Test Simulator

Overview

The Permutation Test Simulator demonstrates a powerful non-parametric alternative to traditional parametric tests like t-tests. By permuting group labels, you can test hypotheses without assuming any particular distribution. Watch as thousands of permutations create a null distribution under the assumption of no group difference, then compare your observed test statistic to see if the groups truly differ.

Open in new tab

Tips

  • Start by generating sample data with different group means to see how permutation tests detect real differences
  • The p-value is the proportion of permuted test statistics that are as extreme or more extreme than the observed value
  • Use at least 1,000 permutations for stable p-value estimates; 10,000 is even better
  • Try different test statistics (mean difference, median difference, t-statistic) to see which is most powerful
  • Permutation tests work under minimal assumptions - just exchangeability under the null hypothesis
  • Compare results when groups have equal vs unequal variances to see permutation tests’ robustness
  • The histogram shows the null distribution - if the observed value is in the tail, that’s evidence against the null
  • Permutation tests are exact for small samples and approximate for large samples
  • Unlike t-tests, permutation tests don’t require normal distributions or equal variances