US Baby Names 1880-2010

US Baby Names 1880-2010
There are many things you might want to do with the data set:
• Visualize the proportion of babies given a particular name (your own, or another name) over time.
• Determine the relative rank of a name.
• Determine the most popular names in each year or the names with largest increases or decreases.
• Analyze trends in names: vowels, consonants, length, overall diversity, changes in spelling, first and last letters
• Analyze external sources of trends: biblical names, celebrities, demographic changes

I encourage you to download and explore the data yourself. If you find an interesting pattern in the data, I would love to hear about it.

As this is a nicely comma-separated form, it can be loaded into a DataFrame with pandas.read_csv:

import pandas as pd
names1880 = pd.read_csv('yob1880.txt', names=['name','sex', 'births'])
In [371]: names1880.groupby('sex').births.sum()
Out[371]:
sex
F 90993
M 110493
Name: births

Since the data set is split into files by year, one of the first things to do is to assemble all of the data into a single DataFrame and further to add a year field. This is easy to do using pandas.concat:

# 2010 is the last available year right now
years = range(1880, 2011)
pieces = []
columns = ['name', 'sex', 'births']
for year in years:
    path = 'names/yob%d.txt' % year
    frame = pd.read_csv(path, names=columns)
    frame['year'] = year
    pieces.append(frame)
# Concatenate everything into a single DataFrame
names = pd.concat(pieces, ignore_index=True)

With this data in hand, we can already start aggregating the data at the year and sex level using groupby or pivot_table

In [374]: total_births = names.pivot_table('births', rows='year',
.....: cols='sex', aggfunc=sum)
In [375]: total_births.tail()
Out[375]:
sex F M
year
2006 1896468 2050234
2007 1916888 2069242
2008 1883645 2032310
2009 1827643 1973359
2010 1759010 1898382
In [376]: total_births.plot(title='Total births by sex and year')

Next, let’s insert a column prop with the fraction of babies given each name relative to the total number of births.

def add_prop(group):
# Integer division floors
    births = group.births.astype(float)
    group['prop'] = births / births.sum()
    return group
names = names.groupby(['year', 'sex']).apply(add_prop)

Since this is floating point data, use np.allclose to check that the group sums are sufficiently close to (but perhaps not exactly equal to) 1:

In [379]: np.allclose(names.groupby(['year', 'sex']).prop.sum(), 1)
Out[379]: True

I’m going to extract a subset of the data to facilitate further
analysis: the top 1000 names for each sex/year combination. This is yet another group operation:

def get_top1000(group):
    return group.sort_index(by='births', ascending=False)[:1000]
grouped = names.groupby(['year', 'sex'])
top1000 = grouped.apply(get_top1000)

Analyzing Naming Trends

In [383]: boys = top1000[top1000.sex == 'M']
In [384]: girls = top1000[top1000.sex == 'F']

Simple time series, like the number of Johns or Marys for each year can be plotted but require a bit of munging to be a bit more useful. Let’s form a pivot table of the total number of births by year and name:

In [385]: total_births = top1000.pivot_table('births', rows='year', cols='name',
.....: aggfunc=sum)

In [387]: subset = total_births[['John', 'Harry', 'Mary', 'Marilyn']]
In [388]: subset.plot(subplots=True, figsize=(12, 10), grid=False,
.....: title="Number of births per year")

Measuring the increase in naming diversity