intro

A reddit user posted ~1.7 billion reddit comments. There is also a one month subset of this data that we can work with on our local machines.

Check out the code on github.

Download the one month subset of Reddit comments. We start by getting word counts of the comments and grouping them by subreddit. Then we will look for words that “define” each subreddit.

tokenization

The process of extracting words is called tokenization as a string is broken up into “tokens” that are used to characterize the string. In our case the tokens will, approximately, be each word of the comments. The tokenization specification is arbitrary. You could look at pairs of words (bigrams), triplets of words (trigrams), or even every other word. In our case we use every word individually.

What happens if we have different forms of the same word? We probably want to count “foot” and “feet” as the same token. We have two options: a stemmer which chops off the ends of words, or a lemmatizer which does a morphological analysis on each word to determine its lemma (a canonical version of the word).

In this project we can piggy back off of nltk’s implementation of a WordNet lemmatizer. We will also use nltk’s part of speech tagger to pass the part of speech into the lemmatizer. Finally, the word_tokenize function will do exactly what you think… it splits up the string into distinct words.

class LemmaTokenizer(object):

    def __init__(self):
        self.lemmatizer = WordNetLemmatizer()

    def __call__(self, doc):
        res = []
        for token, pos in pos_tag(word_tokenize(doc)):
            pos = pos.lower()

            if pos[0] in ['a', 'n', 'v']:
                res.append(self.lemmatizer.lemmatize(token, pos[0]))
            else:
                res.append(self.lemmatizer.lemmatize(token))

        return res

By passing text through an instance of this class we will get a list of words that approximate the original string.

vectorization

We can use this lemmatizer in our pipeline to vectorize our text. This means we transform our comments from a string into a vector of word counts (token counts). Scikit-learn provides an implementation of a vectorizer.

vect = CountVectorizer(tokenizer=tokenizer, preprocessor=preprocess, stop_words='english')

tfidf

We mentiond in the intro section that we were going to look for words that “define” each subreddit. So now that we have a count of each word for each subreddit we need to rank them. One way of doing this is using Term Frequency Inverse Document Frequency (tfidf). The basic idea is to look at how frequently a word appears in a subreddit and offset it by the number of subreddits that word appears in. So if the word “kumquat” appears a lot of times in /r/fruit but doesn’t appear in very many other subreddits it will be weighted strongly. If the word “banana” appears often in /r/fruit and it also appears in lots of other subreddits, then it will be weighted weakly. There are a number of variations on this but we will be using the default one from sklearn.

\[ idf(d, t) = \log{\dfrac{n_d + 1}{df(d, t) + 1} + 1} \]

where \(n_d\) is the number of documents and \(df(d, t)\) is the number of times term \(t\) appears in document \(d\). The resulting matrix is then normalized using the l2 norm.

TfidfTransformer().fit_transform(counts)

results

Let’s look at some of the top words for varying subreddits (words with corresponding tfidf scores):