Exploratory Data Analysis #2

Bea Taylor - beatrice.taylor@ucl.ac.uk

13th October 2026

Last week

Exploratory data analysis, part 1

• Our dataset — DfE school performance in England, 2022/23
• Data types
• Descriptive statistics: distributions, histograms, boxplots
• Descriptive statistics: mean, median, variance, standard deviation
• Making things comparable: counts, rates and standard scores
• Missing data and outliers

4 data types: nominal, ordinal, interval, ratio Numerical vs categorical

Key data metrics - the middle: mean, median, mode and the spread: variance and standard deviation

different types of data outliers: caused by measurement error, or irregular pattern or influential outliers

This week

Last week was largely descriptive, this week we go a bit deeper!

• Last week we talked about histogram’s, this week we’ll look at probability distributions. These are functions with parameters, that you can compute and predict with.
• We’ll look at how data can be processed to make it easier to analyse.
• Last week you considered what the population of interest was, this week we will consider different sources of data bias.

Learning objectives

By the end of this lecture you should be able to:

1. Describe the characteristic features of common probability distributions
2. Apply exponentials and logarithms to transform data
3. Evaluate whether a dataset is representative

When descriptive statistics aren’t enough

import pandas as pd 

# read data
df_4datasets = pd.read_csv("../data/anscombe_quartet.csv")
# print descriptive statistics
df_4datasets.groupby("dataset").describe()

	x								y
	count	mean	std	min	25%	50%	75%	max	count	mean	std	min	25%	50%	75%	max
dataset
1	11.0	9.0	3.316625	4.0	6.5	9.0	11.5	14.0	11.0	7.500909	2.031568	4.26	6.315	7.58	8.57	10.84
2	11.0	9.0	3.316625	4.0	6.5	9.0	11.5	14.0	11.0	7.500909	2.031657	3.10	6.695	8.14	8.95	9.26
3	11.0	9.0	3.316625	4.0	6.5	9.0	11.5	14.0	11.0	7.500000	2.030424	5.39	6.250	7.11	7.98	12.74
4	11.0	9.0	3.316625	8.0	8.0	8.0	8.0	19.0	11.0	7.500909	2.030579	5.25	6.170	7.04	8.19	12.50

all 4 datasets have very similar descriptive statistics of both the x, and y variables.

Same same but different

These all have mean x = 9, variance x = 11. mean y = 7.5, variance y =4.1.

But we can see from the diagrams that the distribution of the data is very different, some is linear, some is elliptic etc… So we need more advanced ways of describing the data.

Probability distributions

A mathematical function that describes the likelihood of different events occuring in a random process.

More sophisticated than descriptive statistics, probability distributions allow us to mathematically approximate the processes which generated the data.

Continuous vs. Discrete

How we measure data is important for how we model and analyse it.

Continuous

Measurable data which can take any value within a given range.

Discrete

Measurable data which can take seperate, countable values.

Quiz

Is this data continuous or discrete?

1. The air temperature on the platform at Warren Street
2. The number of tubes arriving at the platform in the next 10 minutes
3. The distance commuters travel to work
4. How late a train is, as recorded in the TfL delay log (whole minutes)

1. Continuous - temperature can take any value within a range
2. Discrete - number of tubes is a countable value
3. Continuous - distance can take any value within a range
4. Discrete - delay is a countable value (whole minutes), however the underlying distribution before it gets rounded to whole minutes is continuous.

The probability function

\[\begin{align} p(x) \end{align}\]

Having a function for the distribution allows us to evaluate the probability of events, and hence (spoiler alert for lecture 3) evaluate hypotheses.

• continuous distributions -> probability density functions (PDFs)
• discrete distributions -> probability mass functions (PMFs)

Key distributions

	Normal	Binomial	Poisson	Exponential
Type	Continuous	Discrete	Discrete	Continuous
Shape	Symmetric, one peak	Depends on \(p\); symmetric if \(p=0.5\)	Right-skewed (more symmetric as \(\lambda\) grows)	Right-skewed, highest at zero
Parameters	\(\mu\), \(\sigma\)	\(n\), \(p\)	\(\lambda\)	\(\lambda\)
Probability function	PDF	PMF	PMF	PDF

Normal distribution

The most fundamental distribution

Talked about this last week - in terms of things being approxiamtely normal.

Also known as Gaussian distribution after Gauss. The shape is known as the bell curve.

• Human height
• Petal size (specifically length of iris petals)
• blood pressure
• birth weight of babies
• measurement errors in science experiments

Key features

• Data is continuous
  • it is something you measure not something you count
• It is symmetric
  • data is equally likely to be larger or smaller than average
• It has a characteristic size
  • there is a single peak, with all data points close to the mean
• There are smooth tails
  • there is less data further away from the mean

See it in histograms

Same data as last week: Attainment 8 scores, restricted to mainstream state-funded secondaries so we’re comparing like with like

Approximately normal

Uniquely described by two variables…

…and a probability density function

The probability density function (PDF) describes the likelihood of different outcomes for a continuous random variable

A note on samples

When we talk about a distribution, we are usually referring to the population distribution. However, in practice, we often only have a sample from the population.

The standard deviation of the whole population is \(\sigma\):

\[\begin{align} \sigma = \frac{\sum_{i=1}^{n}(x_i - \mu)^2}{n} \end{align}\]

But when we only have a sample from the population, the standard deviation is:

\[\begin{align} \sigma = \frac{\sum_{i=1}^{n}(x_i - \mu)^2}{n-1} \end{align}\]

Bessel’s correction

Bessel’s correction refers to the adjustment made when calculating the standard deviation of a sample.

Intuiiviely, we expect the sample standard deviation to be a bit smaller than the whole population standard deviation, as we have less information about the population. We correct for this by dividing by n-1 instead of n. This correction makes the estimate a bit larger, and hence more accurate.

The difference is that we divide by n-1 instead of n. This is because we are estimating the population standard deviation from a sample, and we want to account for the fact that we have less information. This is known as Bessel’s correction.

Intuitively we expect the sample standard deviation to be a bit smaller than the population standard deviation, as we have less information about the population. So we correct for this by dividing by n-1 instead of n. Which makes the estimate a bit larger, and hence more accurate.

Random variable

A random variable is a way to map the outcome of a random process to a probability.

In mathematical notation, if a random variable \(X\) is approximately normally distributed about a mean of \(\mu\) with a standard deviation of \(\sigma\) we write:

\[\begin{align} X \sim N(\mu,\sigma) \end{align}\]

Calculating probabilities

Can use the PDF to evaluate the probability at a specific point.

\[\begin{align} X \sim N(0,1) \end{align}\]

\[\begin{align} p(x) = \frac{1}{\sqrt{2 \pi \sigma^2}} e^{-\frac{(x-\mu)^2}{2\sigma^2}} \end{align}\]

Or the probability of a range of values by integrating the PDF:

import scipy.stats as sps

mu = 0
std = 1

x_dash = 0.5

# probability x>0.5
prob = 1 - sps.norm.cdf(x_dash, loc=mu, scale=std)
print(f"Probability x > {x_dash}: {prob:.4f}")

Probability x > 0.5: 0.3085

Binomial distribution

Coin toss

• I flip a coin 10 times
• How often can I expect to get at least 7 heads?

Discrete outcomes Describes the frequency of successes in a test with 2 outcomes. Coin flip is the classic example - have exactly two outcomes, either heads or tails.

Probability mass function

\[\begin{align} P(X = k) = \binom{n}{k} p^k (1-p)^{n-k} \end{align}\]

Where \(n\) is the number of trials, and \(p\) is the probability of success for each trial.

The probability mass function (PMF) describes the likelihood of different outcomes for a discrete random variable

Note the combination here.

Plotting the distribution

Example 1: a council survey

• Southwark council asks 200 randomly chosen residents whether they support a new cycle lane by Waterloo station.
• Suppose we know that residents are really split 60/40 in favour of the new cycle lane.
  • \(n=200\), \(p=0.6\)
• How likely is it that 120 or more say yes, just by chance?
  • \(k \geq 120\)

from scipy.stats import binom

# P(X >= 120) = P(X > 119)
p = binom.sf(119, n=200, p=0.6)
print("Probability X >= 120: {}".format(round(p, 4)))

Probability X >= 120: 0.5307

Will return to the idea of hypothesis testing in week 3.

Poisson distribution

Death by horse kicks

• It’s 1894.
• You’re the statistician Ladislaus Bortkiewicz.
• And you’re wondering,
• How many soldiers in the Prussian army have been killed by horse kicks?

Hussars trying not to get trampled by horses. Image credit: https://en.wikipedia.org/wiki/Prussian_Army

Measuring rare events

• Imagine a situation where certain rare events can occur in an independent fashion.
• The Poisson distribution estimates how many such events are expected within a time interval
  • Fixed time interval (e.g. one minute)
  • Fixed rate of events (\(\lambda\))
• Poisson distribution gives the probability of \(k\) events.

Typically used to measure rare events like mail arriving or death by horse kicks.

Probability mass function

\[\begin{align} P(X = k) = \frac{\lambda^k e^{- \lambda}}{k!} \end{align}\]

Where \(\lambda\) is the expected number of events in a given interval.

Plotting the distribution

Example 1: bike-share arrivals

• A docking station receives on average 0.5 Lime bikes per minute
• How likely is it that no bikes arrive in the next 10 minutes?
• How likely is it that 8 or more bikes arrive?

from scipy.stats import poisson

lambda_10min = 0.5 * 10  # expected arrivals in 10 minutes

p_no_arrivals = poisson.pmf(0, mu=lambda_10min)
p_8_or_more_arrivals = poisson.sf(7, mu=lambda_10min)

print("Probability of no arrivals: {}".format(round(p_no_arrivals, 4)))
print("Probability of 8 or more arrivals: {}".format(round(p_8_or_more_arrivals, 4)))

Probability of no arrivals: 0.0067
Probability of 8 or more arrivals: 0.1334

Timmy C found a bike in time.

Hopefully a slightly more relatable example than getting kicked to death by a horse.

Note that lambda scales with the interval: 0.5 per minute is 5 per ten minutes. The assumptions box is a seed for spatial dependence later in the course (events in space are often not independent).

Example 2: trees in a forest

• London Plane trees are randomly distirbuted in London’s parks, with an average of 2 trees per 100m².
• What is the probability of finding exactly 3 trees in a 100m² area of Hyde Park?

from scipy.stats import poisson

lambda_trees = 2  # expected trees in 100m²

p_3_trees = poisson.pmf(3, mu=lambda_trees)

print("Probability of finding exactly 3 trees: {}".format(round(p_3_trees, 4)))

Probability of finding exactly 3 trees: 0.1804

A London Plane tree in Wandsworth Park.

Exponentials and Logarithms

Exponentials

If the Poisson measures the probability of \(x\) events within a time period, then the exponential measures how long we are likely to wait between events.

So what is the exponential function?

“The greatest shortcoming of the human race is our inability to understand the exponential function” – Albert Bartlett (physicist)

He’s a bit dramatic

A game of chess…

• A scholar has invented chess
• The emperor is really grateful and asks the scholar to choose a gift in recompense
• The scholar asks for grains of rice
  
• Specifically, rice to fill the chessboard, such that the number of grains double on each square

Krishna and Radha playing chaturanga - the precursor to chess. Image credit: https://en.wikipedia.org/wiki/Chaturanga

Famous story, of the man who invented chess. Emperor was so grateful, he said what do you want in return The man asked for rice, such that on each square of the chess board, the number of grains of rice doubled. The Emperor thought this was a really small ask for such a great invention. He asked if he didn’t want a better gift. But the man insisted that what he wanted was rice.

Image is of Chatarunga - which is the earliest known form of chess dating to 6th century AD northern India. Not to be confused with the yoga pose.

…and rice

A bowl of rice is around 4,000 grains.

By the final square have 10^18 grains - which is more than the global production of rice.

What does this look like on a graph?

Generally we have

The exponential function

The (natural) exponential function is:

\[\begin{align} y=e^x \end{align}\]

• \(e \approx 2.718...\) is eulers number - a mathematical constant.

Its a mathematical constant similar to pi - it’s just a fixed number which we have a name for. Like pi comes from geometry (circles etc), e comes from the limit of the equation of compound interest.

The exponential distribution

We’ve talked about the exp function - but now want to talk about the exp distribution.

This is a continous probability distribution.

Probability density function

The PDF of the exponential distribution is:

\[\begin{align} P(x) = \lambda e^{-\lambda x} \end{align}\]

where \(\lambda\) is the rate parameter.

As in the poisson distribution \(\lambda\) is the fixed rates of events for a predetermined time interval.

Note

This is sometimes referred to as the negative exponential distribution - as it is a negative exponent.

Traditionally used to model time between rare events.

• time between volcanic eruptions
• time between customers entering a shop
• time of radioactive decay

Example 1: wait time for a bike-share arrival

• One Lime bike arrives at the docking station every minute.
• What is the probability that it will be ten minutes until the next bike arrives?
  • \(\lambda=1\)
  • \(p(10) = 1 \times e^{-1 \times 10}\)

from scipy.stats import expon

lambda_1min = 1  # rate per minute
x = 10  # time in minutes

p_10min = expon.pdf(x, scale=1/lambda_1min)

print("Probability of 10 minutes until next bike arrives: {}".format(round(p_10min, 4)))

Probability of 10 minutes until next bike arrives: 0.0

Timmy C found a bike in time.

Example 2: wait time for a bike-share arrival

• One Lime bike arrives at the docking station every minute.
• What is the time at which the probability of a bike arrival is 0.5?
  • \(\lambda=1\)
  • \(0.5 = p(x) = 1 \times e^{-1 \times x}\)
  • what is \(x\)?

This is easier said than done – the best way is to invert the equation.

Probability distributions

	Normal	Binomial	Poisson	Exponential
Type	Continuous	Discrete	Discrete	Continuous
Shape	Symmetric, one peak	Depends on \(p\); symmetric if \(p=0.5\)	Right-skewed (more symmetric as \(\lambda\) grows)	Right-skewed, highest at zero
Parameters	\(\mu\), \(\sigma\)	\(n\), \(p\)	\(\lambda\)	\(\lambda\)
Probability function	PDF	PMF	PMF	PDF

Quiz

The cross at Charing Cross - the geographic centre of London.

What probability distribution most likely describes the following data?

1. Heights of a sample of UCL students
2. The number of students arriving at the lecture in the next ten minutes
3. Commute distance to work in London
4. Public votes for a new block of flats in the borough of Camden
5. How far north or south people’s guesses of Charing Cross are from the actual location (signed: + north, − south)
6. The value of land parcels across London

1. Normal
2. Poisson - coutn of events in a fixed interval
3. Exponetial - although this is more debatable. It doesn’t go below zero, and it has a long tail, so exp is probably a bettter fit than normal.
4. Binomial - assuming it’s a yes/no vote, and that the votes are independent of one another.
5. Discrete - delay is a countable value (whole minutes), however the underlying distribution before it gets rounded to whole minutes is continuous.
6. Normal - the distribution of errors is often normal, and this is a signed error.
7. Exponential - land values are often right skewed, with a long tail of very high values. Although in rewlaity - probably closer to a log-normal distribution.

Transforming data

Inverse operations

The mathematical operation that reverses.

Subtract is the inverse of adding.

\[\begin{align} 2 + x=5 \implies 5-2=x \end{align}\]

Divide is the inverse of multiplying.

\[\begin{align} 2 \times x =6 \implies 6 \div 2=x \end{align}\]

Logarithms

Taking the logarithm is the inverse of taking the exponential.

\[\begin{align} 2^3 = 8 \implies \log_2(8) =3 \end{align}\]

More generally:

\[\begin{align} a^x = b \implies \log_a(b) =x \end{align}\]

For the natural logarithm:

\[\begin{align} e^x = b \implies \log_e(b) =ln(b) = x \end{align}\]

Multiply and divide are inverse operations of one another (they reverse the process).

Read the equation as log of 8 base 2 equals 3.

Why transform?

Because logarithms undo exponentials, they give us two very useful tools for data:

1. Straighten exponential growth
2. Tame long right tails

The most important rules:

\[\begin{align} log_a(a^x) = x \\ ln(e^x) = x \end{align}\]

Log it! to straighten growth

As the logarithm inverses the exponential, taking the logarithm of an exponential function will result in a linear relationship.

Here we take log base 2, which results in a line through \(y=x\), but any base will work.

Log it! to tame long tails

Long right tail, no typical size: most schools cluster at low absence, but a few sit far out — after the log it looks much closer to normal.

Here we take log base \(e\) (the natural logarithm). This is the most common choice in statistics and data analysis, but any base will work.

Same schools data as the rest of this deck: absence rate (PERCTOT), mainstream state-funded secondaries only.

Representative data

Before we try to describe the data, it’s important to know where the data came from.

The dream vs reality

Ideally, we would like all the relevant data.

… in reality we normally only have some.

It would be great if I knew the height of everyone at UCL, but unrealistic to collect – however maybe I could collect all students in this lecture theatre.

Approximating

Hence, we sample a subset of the data. We need to choose our sample carefully - we want what happens in the sample to approximate what happens in the whole population.

In practise we might try different sampling approaches such as:

• random sampling
• systematic sampling

I might assign everyone a number and then choose 100 random numbers and measure them. Or I might choose a specific subset, for example just everyone in this lecture theatre.

Bias

It’s important to understand if your dataset is unrepresentative or biased.

Bias is the idea that there is an unrepresentative pattern in your data.

Where does bias creep in?

Three places, three questions:

• In the sample: selection bias. Who ended up in the data, and who didn’t?
• In the world: historical bias. What existing inequalities does the data reflect?
• In the analyst: cognitive bias. What did we choose to collect, keep and report?

Particularly important for large datasets and AI models, where the model can learn patterns that aren’t obvious to us.

Already talked about the idea of cognitive bias. There are lots and lots of different types of biases - going to talk a little about different ways they can bias the dataset.

Dataset bias has become a more prevalent topic of concern on the era of large AI models. These models are really good at learnign patterns which aren’t obvious to the human eye - but the pattern might reflect a bias. For example generative AI models learn that scientists are white men, even though if I asked you you’d tell me anyone can be a scientist.

• Hungry judges

Selection bias

When the sample chosen doesn’t represent the whole population of interest

Examples:

• Roy Model
  • People who self select into certain groups, tend to have underlying characteristics.
• WEIRD people
  • WEIRD = western, educated, industrialised, rich, democratic
  • Commonly sampled in behavioural sciences, but reflects a very small proportion of global population.
• 2001 Westminster census
  • Unexpectedly small sample sizes, as high numbers of second homes.

Roy model - example from economics - basically there’s not a random selection of people in a specific job - thye have chosen that job based on underlying unobserved characterisitcs. For example suppose I’m interested in wages of workers in different occupations - but they have self-selected into that occupation based on their unique skill set - so it’s a biased representation.

WEIRD = western, educated, industrialised, rich, democratic

Historical bias

Reflects existing, real world, inequalities

Examples:

• Police profiling
  • Automated tools to detect ‘criminals’.
  • Trained on datasets which reflect current racist practises.
• Generative images
  • Studies have found AI-generated images perpetuate racist and sexist stereotypes.
    
  • Reflects who is represented in the training data.
• Hungry judges
  • Judges sentence differently based on how hungry they are.
  • Tend to be harsher just before lunch and more lenient after lunch.

Police profiling models - meant to make easy to spot criminals - but use exisitng crime data - which reflect racist prejudices in society - and the model learns to pick out people of certain ethnic backgrounds.

Tools also used to help the sentnecing of criminals in courts of law - observe the hungry judges effect, where they sentnece differently based on how hungry they are - tend to be harsher just before lunch and more lenient after lunch.

Cognitive bias

Systematic patterns in how we think about, and perceive, the world.

Our cognitive biases can impact:

• data collection
• data selection
• data processing
• modelling choices

We all have our own cognitive biases - shaped around our values and our backgrounds. Example of cognitive bias - confirmation bias, whereby I favour information that agrees with my existing beliefs. I’m writing a report on the spatial impact of a new railway line. Suppose I hate the idea of a new railway line, and then I see some data, fact a: 60% of residents living near the railline don’t want it due to noise. fact b: 75% of residents in the UK do want the railway line. i might only include fact a in my report as it confirms my existing beliefs.

Why is this important?

If we’re not careful we can propagate bias to the research, and hence results.

This can lead to incorrect conclusions.

As scientists we should try and present an accurate and fair balance of information. If we’re more aware of our biases it can be easier to try and actively avoid them.

Can data ever be truly representative?

Probably not.

Even when we think we have really good data, someone has made the choice to collect this data. Why that data over other data? Does that data reflect their underlying cognitive biases? For example the choice to collect information of peoples opnion about a new trainline might be driven by a political goal of understanding public support for a new government infrastructure project.

Not necessarilly a bad thing, but important to keep in mind. And if the bias is noticeable then it should be acknowledged and reported.

Failing that… .. we can acknowledge our biases!

Image credit: [xkcd](https://xkcd.com/2494/)

Overview

We’ve covered:

• Normal distribution
• Binomial distribution
• Poisson distribution
• Exponentials
• Logarithms
• Representative data

The practical will focus on exploratory data analysis for a variety of different datasets. Have questions prepared!