Why Exploratory Data Analysis Matters

Exploratory data analysis , or EDA, is the habit of looking at your data before you compute anything from it. This guide covers what to look at, the research showing why it is not optional, and what it looks like on real projects you can open and read.

It exists to prevent one specific failure, and that failure has nothing to do with math. An analyst computes an answer from data they never looked at. The numbers come out precise, confident and wrong, because the data had a quirk nobody checked for.

Detective work vs courtroom work

Data work has two distinct modes, and mixing them up causes most analytical failure:

A detective does not walk into a crime scene with a verdict. They look at everything, notice what is odd, and form theories. The courtroom tests one theory formally, later.

Exploratory analysis is the crime-scene phase. Skip it and you are prosecuting a case you never investigated. The comparison is not a teaching device invented for beginners. John Tukey, who founded the field, framed exploratory work as detective work, and the literature has used the analogy ever since (Behrens, 1997).

Where EDA came from (and why a famous statistician rebelled)

In 1962, John Tukey published a 67-page argument called The Future of Data Analysis that scandalized his own field. He was a Princeton mathematician, and he also coined the words "bit" and "software".

Statistics, he argued, had become obsessed with the courtroom. Elegant formal tests, applied to questions someone had already decided to ask. Almost nobody was studying the harder, earlier problem: how you work out what questions the data wants you to ask. He proposed treating data analysis as its own empirical science, one that starts by looking (Tukey, 1962).

Fifteen years later his book Exploratory Data Analysis (1977) turned that argument into a toolkit: plots, summaries and residual-checking habits, all built to make the unexpected visible. Its most quoted line is the mission statement of the whole field:

"The greatest value of a picture is when it forces us to notice what we never expected to see." (Tukey, 1977)

Note what that sentence actually says. The payoff of looking is not confirming what you expected. It is being ambushed by what you did not. Every technique in EDA is a machine for manufacturing useful ambushes.

The proof that summaries lie: a quartet and a dinosaur

Before the answer, put a number on your own confidence. Two datasets have the same average, the same spread and the same correlation. How similar do their shapes have to be? Very, somewhat, or not at all? Pick one.

Why is a table of averages not enough? In 1973, the statistician Francis Anscombe settled it with four small datasets, now called Anscombe's quartet. All four share nearly identical summary statistics. Same average of x, same average of y, same variance, same correlation of 0.816, same fitted trend line (Anscombe, 1973). By the numbers they are quadruplets. Graphed, they are strangers:

• One is a healthy, ordinary linear relationship, which is what the summary implies.
• One is a smooth curve , so the linear trend line is simply the wrong model.
• One is a perfect line except for a single outlier that dragged the trend off course.
• One is no relationship at all: a vertical stack of points, plus one extreme value that manufactured the entire correlation.

Four different realities, one identical scorecard. If your analysis stops at the scorecard, you cannot tell which reality you're in.

In 2017, Justin Matejka and George Fitzmaurice pushed the point to its comic extreme. They showed you can nudge a dataset's points into almost any picture, including a hand-drawn Tyrannosaurus rex. The mean, standard deviation and correlation stay identical to two decimal places.

Their "Datasaurus Dozen" is twelve wildly different plots, among them a dinosaur, a star and concentric circles, all with matching summary statistics (Matejka & Fitzmaurice, 2017). The lesson is Anscombe's, updated: never trust summary statistics alone. Always visualize.

What exploration looks like in a real project

The classics prove why ; here is what it looks like , from open portfolio projects you can read end to end. None of this is glamorous. All of it changed the results.

Peek before you trust: the phantom-column catch

Rule one of EDA is simple: look at actual rows before you compute anything. Picture the last dataset you loaded. Did you look at the rows, or did you go straight to a count? Most people go straight to the count, and the next paragraph is what that costs.

In a project on Billboard chart history, the first look at the imported table showed two mystery columns the source file did not have, and song titles chopped in half. The cause was a wrong import setting. It had split every title containing a comma, like "Let It Snow, Let It Snow, Let It Snow" , across columns.

Every count run on that import would have been silently wrong. Catching it cost one SELECT * ... LIMIT 20 and thirty seconds of reading. The Streaming Hidden Gems project hit the same class of problem from another direction: its "CSV" files turned out to be tab-separated with no quoting at all.

Check the data against arithmetic it must obey

Good exploration tests expectations the data has no excuse to fail. A chart that lists 100 songs a week must have a row count divisible by 100. Weeks times 100 must equal total rows, exactly.

When a check like that passes, the import earns some trust. When it fails, you found a problem at the cheapest possible moment. The Steam Hidden Gems and Streaming Hidden Gems projects both carry a "Validation" section showing these cross-checks, with totals reconciled two independent ways before any conclusion rested on them.

Interrogate the weird: why is a number stored as text?

Exploration means treating oddities as leads rather than annoyances. In the Billboard data, a chart-position column imported as text instead of numbers.

A detective does not shrug at that. The follow-up queries found the culprit: thousands of rows carrying the letters 'NA'. They also turned up something else. "Debut week" was marked two different ways in different eras of the archive, so every later query had to speak both dialects. Miss that during exploration and every debut-related count afterwards is quietly wrong.

That investigation pattern is hypothesis, targeted query, verdict, next hypothesis. It is walked through step by step in the SQL CASE expression guide, where exploring a messy artist-credit column ends up dictating the cleaning rule.

Let distributions choose your thresholds

Analyzes constantly need cutoffs. How many reviews make a rating trustworthy? How many votes make a movie "known"?

You can pick a round number from the gut, and the number will be indefensible. The exploratory move is to ask the data first. Plot or tabulate the distribution, find where the behavior actually changes, and pick a cutoff you can defend.

Both gems projects do this in the open. Steam Hidden Gems takes its review-count floor from the measured distribution of review volumes. Streaming Hidden Gems takes its vote-count band the same way, with the rejected alternatives written down. A threshold with a distribution behind it survives the question "why that number?"

Split the average before you report it

One number for a whole population is usually four numbers wearing a trench coat.

Take the headline number below and guess before you read the split. A telecom company loses 26.5% of its customers. Are the newest customers or the oldest ones doing most of the leaving, and by how much? Commit to a rough figure, then compare it to what the data says.

A telecom churn analysis opens on 26.5% churn. That is true, and useless for deciding anything. Splitting the same customers by how long they had been subscribed changed the story completely. Customers in their first year churn at 47%. The loyal core churns at 10%. And 91% of that at-risk group sits on month-to-month contracts.

So the company does not have a churn problem. It has a first-year problem, and that is a different budget line. The Telco Churn project shows the split-and-compare pass in pandas, from headline average to four personas to three recommendations.

The habit to keep: before an average leaves your notebook, cut it by the two or three variables most likely to be hiding a different population inside it.

The habits, distilled

The honest limit: exploration finds suspects, not verdicts

One warning keeps EDA honest. When you explore freely you will find patterns, and some of them are pure coincidence. Look at data from enough angles and something eventually looks interesting.

The detective and courtroom split exists for exactly this. A pattern found during exploration is a suspect , not a conviction. It earns a hypothesis, and that hypothesis needs testing on evidence that did not generate it: fresh data, a held-out sample, or a formal test specified in advance.

Treating an exploratory find as a proven fact is the analytical version of convicting the first person the detective interviewed. Explore boldly, conclude carefully. And whatever exploration turns up, the analysis has to account for it, which is covered in How Analysts Document Limitations.

References

1. Tukey, J. W. (1962). The future of data analysis. The Annals of Mathematical Statistics, 33(1), 1–67. doi:10.1214/aoms/1177704711
2. Tukey, J. W. (1977). Exploratory Data Analysis. Addison-Wesley.
3. Anscombe, F. J. (1973). Graphs in statistical analysis. The American Statistician, 27(1), 17–21. doi:10.2307/2682899
4. Behrens, J. T. (1997). Principles and procedures of exploratory data analysis. Psychological Methods, 2(2), 131–160. doi:10.1037/1082-989X.2.2.131
5. Matejka, J., & Fitzmaurice, G. (2017). Same stats, different graphs: Generating datasets with varied appearance and identical statistics through simulated annealing. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems , 1290–1294. ACM.

---

The full version of this guide lives on my site: Why Exploratory Data Analysis Matters.

If it was useful: Buy Me a Coffee.