The question
"Simpler" is four measurements, and they disagree
The usual version of this argument rests on a readability score. Readability
formulas need sentences, and this dataset has none: the lyrics come lowercased
with the punctuation stripped. So I counted four things that need no sentence
boundaries: how long a song is, how much it repeats, how varied its vocabulary
is, and how long its words are.
Words per song
length
+107%200 to 413, r = 0.92
Compression ratio
repetition; lower means the lyric squeezes down further
−19%0.357 to 0.290, r = −0.89
Unique words in the first 100
vocabulary, with length held fixed
+7%57.5 to 61.4, r = 0.58
Characters per word
word length
−0.3%3.84 to 3.83, r = −0.46
1965–69 average against 2010–15 average, from tables/trends.csv. r is the correlation of the yearly mean with year.
Length
Songs got much longer
The yearly mean rises from about 200 words to about 413, roughly 6 words a year,
with r = 0.92 against year. It is the cleanest trend in the data.
The peak came in the mid-2000s, when 2003's chart averaged 501 words; 2012 dropped
back to 369.
Repetition, with a caveat
More repetitive, partly because longer
Run a lyric through zlib and divide the compressed size by the original: text
that repeats itself compresses better. The ratio falls from 0.357 to 0.290
(r = −0.89), so later songs repeat more as whole documents.
A longer text also gives the compressor more history to match against, so part
of this trend is the length trend measured a second way. The data supports the
claim that whole songs repeat more. It says nothing about whether any single
line does.
Vocabulary
The vocabulary did not shrink
Here the popular claim fails. Counting only the first 100 words of each song, so
that length can't inflate the count, unique words rose from 57.5 to 61.4, about
7% (r = 0.58). That is a small move, in the opposite direction
from the dumbing-down story.
Mean word length went from 3.84 to 3.83 characters. With 51 yearly points the
slope is statistically detectable (r = −0.46), and it amounts to one hundredth of
a character over fifty years. Switch the chart above to the shared scale and the
line is flat.
Limitations
What this does not show
The Billboard Year-End Hot 100 is a sales ranking. A change in the chart's genre
mix moves every line here even if no songwriter changed how they write, and the
analysis does not break results out by genre.
Without punctuation there are no sentences, which rules out the readability
formulas this question is usually answered with. Songs under 50 words are dropped,
which removes instrumentals and fragments. Each yearly mean rests on 86 to 100
songs, so single years are noisy.
Method
Where the numbers come from
- Sample
- 4,816 songs across 51 years, after dropping anything under 50 words
- Length
- Whitespace tokens per song
- Vocabulary
- Unique tokens among the first 100, divided by 100
- Repetition
- zlib (level 6) compressed bytes over raw bytes for the whole lyric
- Trend
- Least-squares fit of the yearly mean on year (scipy
linregress); change compares the 1965–69 and 2010–15 averages
- Built with
- Python, pandas, SciPy, matplotlib; the chart on this page is hand-written SVG drawn from
tables/by_year.csv
