Describing Distributions

Type one data set, or two to compare, or pick one below. Each is drawn as a dotplot over its boxplot on one shared scale, with the mean marked under the box. The quartile switch changes how Q₁ and Q₃ are found, and with them the fences.

Mean xˉ=1n∑xi\bar x=\frac{1}{n}\sum x_i
Standard deviation s=∑(xi−xˉ)2n−1s=\sqrt{\frac{\sum(x_i-\bar x)^2}{n-1}}
Outlier fences Q1−1.5 IQRQ3+1.5 IQR\begin{aligned}&Q_1-1.5\,\mathrm{IQR}\\&Q_3+1.5\,\mathrm{IQR}\end{aligned}
Loading interactive simulation...

Lesson

The theory — Describing Distributions

To describe a distribution is to give its shape, its centre, its spread and any unusual values, in the context of the data. To compare two is to say how each of those differs, with numbers.

Where the formula comes from

  1. Shape. Look at a dotplot or histogram: one peak or more, roughly symmetric or with a longer tail to the right (skewed right) or to the left, and any gaps or clusters. The skewness g₁ puts a number on the tail: positive for a long right tail.
    g1=1n∑(xi−xˉ)3(1n∑(xi−xˉ)2)3/2g_1=\frac{\frac1n\sum(x_i-\bar x)^3}{\left(\frac1n\sum(x_i-\bar x)^2\right)^{3/2}}
  2. Centre. The mean is the balance point of the data; the median has at least half the values at or below it and at least half at or above it. One extreme value can move the mean a long way and the median hardly at all, so the median is called resistant.
    xˉ=1n∑xi,M=x(n+12)\bar x=\frac{1}{n}\sum x_i,\qquad M=x_{\left(\frac{n+1}{2}\right)}
  3. Spread. The standard deviation s measures typical distance from the mean, dividing by n − 1 because the deviations from x̄ must add to zero. The IQR is the spread of the middle half and, like the median, resists extreme values. The range uses only the two most extreme.
    s=∑(xi−xˉ)2n−1,∑(xi−xˉ)=0s=\sqrt{\frac{\sum(x_i-\bar x)^2}{n-1}},\qquad\sum(x_i-\bar x)=0
  4. Unusual values. A value more than 1.5 IQR below Q₁ or above Q₃ is an outlier. A value exactly on a fence is not.
    x<Q1−1.5 IQR  ∨  x>Q3+1.5 IQRx<Q_1-1.5\,\mathrm{IQR}\ \ \lor\ \ x>Q_3+1.5\,\mathrm{IQR}
  5. Choose the pair that suits the shape. For a roughly symmetric distribution with no outliers, report the mean and s; when it is skewed or has outliers, report the median and the IQR. To compare two groups, use the same pair for both, on the same scale, and say which is larger and by how much.
Assumes
Quartiles depend on the rule. The switch offers three: the halves of the TI-84, and types 7 and 6 of Hyndman and Fan, the defaults in R and in Minitab and SPSS. They can disagree on a data set of any size.

Problems solved in full

  1. Describing one distribution 3 steps

    Describe the distribution of Old Faithful’s 175 waits after an eruption of 3 minutes or more: n = 175, mean 79.99, s = 5.99, minimum 64, Q₁ = 76, median 80, Q₃ = 84, maximum 96, skewness −0.04.

    Open this problem in the tool

    1. Shape: the dotplot has a single peak and the tails are about equally long; the skewness is close to zero.

      g1≈−0.04g_1\approx-0.04
    2. Centre and spread: the shape is roughly symmetric, so the mean and standard deviation are fair summaries. Check for outliers with the fences:

      76−1.5×8=64,84+1.5×8=9676-1.5\times8=64,\qquad 84+1.5\times8=96
    3. 64 and 96 sit exactly on the fences, and a value on a fence is not an outlier.

    Answer

    Single-peaked and roughly symmetric, centred near 80 minutes (mean 79.99, median 80), with a standard deviation of about 6 minutes and waits from 64 to 96; no outliers.

    xˉ≈80,M=80,s≈6,64≤x≤96\bar x\approx80,\quad M=80,\quad s\approx6,\quad 64\le x\le96
  2. Mean and standard deviation by hand 3 steps

    Nine students spent 10, 15, 19, 20, 23, 29, 29, 31 and 49 minutes on homework. Find the mean and the standard deviation.

    Open this problem in the tool

    Show solution
    1. Add the values and divide by n:

      xˉ=10+15+19+20+23+29+29+31+499=2259=25\bar x=\frac{10+15+19+20+23+29+29+31+49}{9}=\frac{225}{9}=25
    2. Square each deviation from the mean and add the squares:

      225+100+36+25+4+16+16+36+576=1034225+100+36+25+4+16+16+36+576=1034
    3. Divide by n − 1 = 8 and take the square root:

      s=10348=129.25≈11.37s=\sqrt{\frac{1034}{8}}=\sqrt{129.25}\approx11.37

    Answer

    x̄ = 25 minutes, s ≈ 11.37 minutes.

    xˉ=25,s≈11.37\bar x=25,\qquad s\approx11.37
  3. Which summaries resist an outlier 2 steps

    Newcomb’s 66 timings have mean 26.21, standard deviation 10.75, median 27 and IQR 7. Without the two outliers, −44 and −2, the 64 that remain have mean 27.75, standard deviation 5.08, median 27.5 and IQR 6.5. Which summaries changed most, and which pair should describe the full data?

    Open this problem in the tool

    Show solution
    1. Compare each summary with and without the two values:

      xˉ: 26.21→27.75,s: 10.75→5.08,M: 27→27.5,IQR: 7→6.5\bar x:\ 26.21\to27.75,\qquad s:\ 10.75\to5.08,\qquad M:\ 27\to27.5,\qquad\mathrm{IQR}:\ 7\to6.5
    2. The standard deviation more than halves and the mean moves by 1.54, while the median and the IQR move by 0.5 each. The outliers pull hard on the non-resistant measures.

    Answer

    The mean and especially the standard deviation changed most. The full data, with its two outliers, are best described by the median, 27, and the IQR, 7.

    M=27,IQR=7M=27,\qquad\mathrm{IQR}=7
  4. Comparing two distributions 2 steps

    Compare the waits after short eruptions (n = 97: median 54, Q₁ 50, Q₃ 59, range 43 to 71, skewness 0.30) with the waits after long ones (n = 175: median 80, Q₁ 76, Q₃ 84, range 64 to 96, skewness −0.04).

    Open this problem in the tool

    Show solution
    1. Centre: the medians differ by:

      80−54=2680-54=26
    2. Spread: the IQRs are 9 and 8 minutes, nearly the same. Shape: both single-peaked; the short-eruption waits have a slightly longer right tail. Unusual values: none in either group by the fences. Overlap:

      Q1 long=76>59=Q3 shortQ_1^{\,\mathrm{long}}=76>59=Q_3^{\,\mathrm{short}}

    Answer

    Waits after a long eruption are typically 26 minutes longer (medians 80 against 54), with about the same spread (IQRs 8 and 9) and no outliers in either group. The middle half of the long-eruption waits, 76 to 84, lies wholly above the middle half of the short-eruption ones, 50 to 59.

    Mlong−Mshort=26,IQR: 8, 9M_{\mathrm{long}}-M_{\mathrm{short}}=26,\qquad\mathrm{IQR}:\ 8,\ 9
  5. The same data, two quartile rules 2 steps

    For 10, 15, 19, 20, 23, 29, 29, 31, 49, find Q₁ and Q₃ by the halves method and by type 7, h = (n − 1)p + 1, and decide each time whether 49 is an outlier.

    Open this problem in the tool

    Show solution
    1. Halves: leave out the median, 23, and take the median of each four:

      Q1=15+192=17,Q3=29+312=30,30+1.5×13=49.5>49Q_1=\frac{15+19}{2}=17,\qquad Q_3=\frac{29+31}{2}=30,\qquad 30+1.5\times13=49.5>49
    2. Type 7: h = 8 × 0.25 + 1 = 3 and h = 8 × 0.75 + 1 = 7, so Q₁ and Q₃ are the 3rd and 7th values:

      Q1=x(3)=19,Q3=x(7)=29,29+1.5×10=44<49Q_1=x_{(3)}=19,\qquad Q_3=x_{(7)}=29,\qquad 29+1.5\times10=44<49

    Answer

    Halves: Q₁ = 17, Q₃ = 30, fence 49.5, so 49 is not an outlier. Type 7: Q₁ = 19, Q₃ = 29, fence 44, so 49 is an outlier.

    49.5>49,44<4949.5>49,\qquad 44<49
  6. Changing the units 2 steps

    Newcomb recorded each timing t as a code x, where t = 24.8 + x/1000 millionths of a second. The codes have mean 26.21, standard deviation 10.75, median 27 and IQR 7. Find the same four summaries for t.

    Show solution
    1. Adding a constant moves every measure of centre and leaves spread alone; multiplying by a positive constant scales both. Centre:

      24.8+26.211000=24.82621,24.8+271000=24.82724.8+\frac{26.21}{1000}=24.82621,\qquad 24.8+\frac{27}{1000}=24.827
    2. Spread, multiplied by 1/1000 only:

      10.751000=0.01075,71000=0.007\frac{10.75}{1000}=0.01075,\qquad\frac{7}{1000}=0.007

    Answer

    Mean 24.82621 and median 24.827 millionths of a second; standard deviation 0.01075 and IQR 0.007.

    tˉ=24.82621,Mt=24.827,st=0.01075,IQRt=0.007\bar t=24.82621,\quad M_t=24.827,\quad s_t=0.01075,\quad\mathrm{IQR}_t=0.007
  7. When the mean and median disagree with the tail 3 steps

    In 100 households, 38 have one adult, 49 have two, 9 have three, 3 have four and 1 has five. Find the mean and the median, and say which way the distribution is skewed.

    Open this problem in the tool

    Show solution
    1. The mean:

      xˉ=38⋅1+49⋅2+9⋅3+3⋅4+1⋅5100=180100=1.8\bar x=\frac{38\cdot1+49\cdot2+9\cdot3+3\cdot4+1\cdot5}{100}=\frac{180}{100}=1.8
    2. The median is the mean of the 50th and 51st values. The first 38 are 1 and values 39 to 87 are 2:

      x(50)=x(51)=2 ⇒ M=2x_{(50)}=x_{(51)}=2\ \Rightarrow\ M=2
    3. The tail runs to the right, to 4 and 5 adults, and the skewness is positive:

      g1≈1.2>0,xˉ=1.8<2=Mg_1\approx1.2>0,\qquad\bar x=1.8<2=M

    Answer

    Mean 1.8, median 2, skewed right (g₁ ≈ 1.2): the mean is below the median even though the long tail is on the right.

    xˉ=1.8<M=2,g1≈1.2\bar x=1.8<M=2,\qquad g_1\approx1.2

Supplementary problems, answers only

  1. Find the mean and standard deviation of 2, 4, 4, 4, 5, 5, 7, 9.

    Show answer

    x̄ = 5, s ≈ 2.14

    xˉ=5,s=327≈2.14\bar x=5,\qquad s=\sqrt{\tfrac{32}{7}}\approx2.14
  2. A data set has Q₁ = 12 and Q₃ = 20. Which of 0, 8, 32 and 33 are outliers?

    Show answer

    Only 33. The fences are 0 and 32, and a value on a fence is not an outlier.

    12−1.5×8=0,20+1.5×8=32<3312-1.5\times8=0,\qquad 20+1.5\times8=32<33
  3. Every value in a data set is multiplied by 3 and then 5 is added. What happens to the mean, the median, the standard deviation and the IQR?

    Show answer

    The mean and median become 3 × (old) + 5; the standard deviation and IQR are multiplied by 3.

    xˉ→3xˉ+5,M→3M+5,s→3s,IQR→3 IQR\bar x\to3\bar x+5,\quad M\to3M+5,\quad s\to3s,\quad\mathrm{IQR}\to3\,\mathrm{IQR}
  4. Test scores are strongly skewed left with two very low scores. Which measures of centre and spread should a report use?

    Show answer

    The median and the IQR.

  5. For 10, 15, 19, 20, 23, 29, 29, 31, 49, find Q₁ and Q₃ by type 6, h = (n + 1)p.

    Show answer

    Q₁ = 17, Q₃ = 30

    h=2.5: 15+0.5 (19−15)=17,h=7.5: 29+0.5 (31−29)=30h=2.5:\ 15+0.5\,(19-15)=17,\qquad h=7.5:\ 29+0.5\,(31-29)=30

Old Faithful's two humps are two kinds of eruption

Press Old Faithful, short and long. The histogram of all 272 waits had two humps; here the same waits are split by the eruption that came before them. After an eruption shorter than 3 minutes, the middle half of the waits lies between 50 and 59 minutes. After a longer one it lies between 76 and 84. The two boxes do not overlap at all, and a histogram of either group alone, in 5-minute bins, has a single hump. So the second hump was a second group, poured into one histogram with the first. When a distribution has two peaks, the useful question is what separates them, and here a stopwatch on the eruption itself answers it.

Two calculators, two verdicts on the same 49

Press Homework, halves. A TI-84 finds Q₁ = 17 and Q₃ = 30, the upper fence lands at 49.5, and the longest time, 49, is just inside it. Press Homework, type 7, the rule R uses unless told otherwise: Q₁ = 19 and Q₃ = 29, the fence drops to 44, and the same 49 is an outlier. Nine numbers, nothing changed but the software. In 1996 Hyndman and Fan compared nine definitions of sample quantiles in use in statistical software and found packages that use one rule for a quartile and another for their own boxplots. With nine values the rule decides the verdict, so write down the quartiles you used, not only the conclusion.

Skewed right, and the mean is below the median

The textbook rule says the mean sits on the side of the long tail. Press Adults per household: the tail runs right, out to households of four and five adults, the skewness is +1.2, and yet the mean, 1.8, is below the median, 2. Paul von Hippel found the rule in 14 of 18 introductory textbooks in 2005, and found it failing in the 2002 General Social Survey, which these made-up households copy in shape: 38% below the median, 49% exactly at it and 13% above. The mean balances the heavy left side, while the few long values on the right decide the skewness, because their distances are cubed. With discrete data and many values tied at the median, read the shape from the picture rather than from the order of the mean and the median.

Learning path

Statistics: from data to proportions

References (6)