Text Analysis

Getting the Data

First we need a data set of work from. I compiled a list of all strategic documents in Google Sheets using a web crawler, however there were items the algorithm missed due to a lack of implimnetation and testing time, so certain documents were added manually. Below is the working code for it done in Python.

Yes I still work in Python 2, but am getting there with 3!

#Python 2.7.1

import requests
import time
from bs4 import BeautifulSoup, SoupStrainer

r = requests.get([YOUR LINK])

soup = BeautifulSoup(r.content)

for link in BeautifulSoup(r.text, 
        parseOnlyThese=SoupStrainer('a')):
    if link.has_attr('href'):
    if 'pdf' in str(link):
        image_file = open(os.path.join([YOUR DIRECTORY]', 
        os.path.basename(link['href'])), 'wb')
        for chunk in r.iter_content(100000):
            image_file.write(chunk)
            image_file.close()

Because topic modelling is a somewhat subjective analysis based on the messy study of linguistics, I added columns for classification as per the client’s request, that would reflect areas the council were accustom to seeing. This will allow us to group documents together later. The client also helped cull unnecessary documents getting use down to the number we currently have.

Quick note, this data has been ammonized to remove references to any actual local government.

Getting the Text

Now we need text to analyse. This was the trickiest part of the analysis as we need two things; A script that will crawl the links to documents and a second one two extract and clean the text. I will be honest, I ended up doing a lot of this manually as the text crawler script I used could not cope with some of the PDF links. But what I was working from is below.

#Python 2.7.1

import urllib2
import BeautifulSoup
import re
import pandas as pd

text=[]
Newlines = re.compile(r'[\r\n]\s+')
data=pd.read_csv([YOUR FILE])
def getPageText(url):
    # given a url, get page content
    data = urllib2.urlopen(url).read()
    # parse as html structured document
    bs = BeautifulSoup.BeautifulSoup(data, 
      convertEntities=BeautifulSoup.BeautifulSoup.HTML_ENTITIES)
    # kill javascript content
    for s in bs.findAll('script'):
        s.replaceWith('')
    # find body and extract text
    txt = bs.find('body').getText('\n')
    # remove multiple linebreaks and whitespace
    return Newlines.sub('\n', txt)

def main():
    urls = [data$[URL COLUMN]
    txt = [getPageText(url) for url in urls]
    Text = txt.append(Text)

if __name__=="__main__":
    main()

This text was extracted raw with only minimal cleaning. The text was processed as UTF-8 with all non-conforming characters replaced with blanks. Because this is an English language analysis, most alphanumeric characters have an analogue in other encoding systems (Character such as @,£,-,’,: and so on, where more likely to be removed). The assumption made is that characters removed across documents would appear in similar settings as naming and wording conventions should be similar, thereby minimalizing the loss.

The Analysis

The output was saved in a .csv which was then was loaded into R for analysis. Strings were coerced again to ensure they were uniform. Some documents did not have text due to publishing timelines as well as difficulty in processing and those were also removed. As for the method, credit to Bodong Chen and this blog post about using Latent Sematic Analysis in R. I used much of his method to make my analysis possible. I will do my best to plain English annotate what is happening so that you can you dear reader can follow along.

# Required Packages
library(shiny)
library(tm)
library(ggplot2)
library(lsa)
library(readr)
library(stringr)
library(ggsn)
library(plotly)

# Pre-Pre Process Text
Result<-Result[!is.na(Result$Text),]
Result <- Result[ c("Classification","Document","Text") ]

The R package tm_map, a text mining and text processing package, was used for text preparation. The Corpus command converts documents (titles and text) into a collection of text within a singular object. A corpus is used in many text analysis projects as a standard and simple way to store collections such as this. To make the analysis more relevant we again subject the corpus to cleaning, removing punctuation, stop words (determiners, common words and those that have no inherent context such as ‘is’,‘the’,‘and’,etc.). This removed the ‘noise’ from the analysis so we are comparing substantive text (adverbs, verbs, nouns, etc.).

# Build a Corpus
corpus <- Corpus(VectorSource(Result$Text))
corpus <- tm_map(corpus, tolower)
corpus <- tm_map(corpus, removePunctuation)
corpus <- tm_map(corpus, function(x) removeWords(x, stopwords("english")))
corpus <- tm_map(corpus, stemDocument, language = "english")
corpus # check corpus

## <<SimpleCorpus>>
## Metadata:  corpus specific: 1, document level (indexed): 0
## Content:  documents: 117

Now the heavy lifting starts. In a process referred to as Multi-Dimensional Scaling (MDS) we covert our corpus into a matrix. Rows = words by columns = documents. In our case, 331,338 words by 49 documents. Each word is given a frequency for its appearance by document. Now that we have the raw term matrix we need a way to make them comparable.

Example: Imagine you have two documents. One is 100 pages and one is 10, both about similar subjects, and the word you are looking for is “growth.” We can assume that the 100 page document has a greater chance of having a high frequency of the word “growth.” So if we are comparing our documents on frequency alone we might say they are dissimilar. But that is unfair. We need to compare the relative appearences of words by documents in order to look at similarity. If both our documents have “growth” appear in 10% of all words, we are comparing apples to apples.

# Create a Distance Matrix
td.mat <- as.matrix(TermDocumentMatrix(corpus))

The R lsa package gives us many weighting options in two categories; local weighting and global weighting. We are in fact going to multiply by both to ensure a more unbiased result. Honestly, after much searching it was hard to find definitive answers on which to use. Below offer my humble opinions of each weighting method. Feel free to skip this next bit if it is to much detail.

# Multidimensional Scaling with Latent Semantic Analaysis
td.mat.lsa <- lw_bintf(td.mat) * gw_gfidf(td.mat) # weighting
lsaSpace <- lsa(td.mat.lsa) # create LSA space
dist.mat.lsa <- dist(t(as.textmatrix(lsaSpace))) # compute distance matrix
head(dist.mat.lsa,100) # check distance mantrix

##   [1] 32.57576 28.64380 50.63469 34.57704 37.48881 54.49891 41.66599 61.41389
##   [9] 33.76428 44.19097 42.53705 52.83014 46.14660 50.39423 61.41389 61.41389
##  [17] 42.46248 40.12651 36.13213 54.31993 51.13785 52.23854 51.75914 65.45626
##  [25] 58.28577 71.20452 56.47104 53.73415 61.41389 61.41389 61.41389 61.41389
##  [33] 61.41389 61.41389 61.41389 61.41389 61.41389 61.41389 61.41389 61.41389
##  [41] 61.41389 61.41389 61.41389 61.41389 61.41389 61.41389 61.41389 61.41389
##  [49] 61.41389 61.41389 61.41389 61.41389 61.41389 61.41389 61.41389 61.41389
##  [57] 61.41389 61.41389 61.41389 61.41389 61.41389 61.41389 61.41389 61.41389
##  [65] 61.41389 61.41389 61.41389 61.41389 61.41389 61.41389 61.41389 61.41389
##  [73] 61.41389 61.41389 61.41389 61.41389 61.41389 61.41389 61.41389 61.41389
##  [81] 61.41389 61.41389 61.41389 61.41389 61.41389 61.41389 61.41389 61.41389
##  [89] 61.41389 61.41389 61.41389 52.98277 71.13825 53.80037 57.27619 85.76956
##  [97] 78.82250 62.82307 46.69994 45.64090

If you’re still here we can explore the weight options. Local weights refer to the value itself, hence local. For local weights, we have:

lw_tf() -The tf stands for ‘term frequency’ and is basically what we strat with.

lw_logtf() -A logarithmic transformation in which ‘normalise’ each term by subjecting it to the same change making it comparable. In this case log( value+1). This might be more appropriate if you have dense and semantically rich text so you need to represent a wider range of words.

lw_bintf() - Simply changing the frequency matrix to 1 if the word is present and 0 if it is not. This might be appropriate if you have a small set of documents and you need them to be as comparable as possible. I would choose this options for simplicity as it is also the least stressful computationally.

Global weights refer to the entirety of the matrix. For global weights, we have:

gw_normalisation() - Every cell is equal to 1 divided by the square root of the document length. Essentially diving up the word weight based on the frequency. Not terribly robust as again the size of document might highlight disparity between long and short documents.

gw_idf() -Inverse document frequency looks at the ‘importance’ of a word to the overall corpus. For example, if we included words like ‘is’ it would appear so frequently that we would say it is less special. What we give higher weight too are those words and documents that have richer semantic differences. However, this is not appropriate in our circumstance because we are expecting documents to be different, bar a hand full of words that we are searching for. So if we choose this it might look like none of our documents are related.

gw_gfidf() -Inverse document frequency multiplied by global normalisation is pretty easy to grasp. This is the method chosen for our analysis. Again there is no standard way to evaluate these things. When comparing different outputs, I used my subjective knowledge of the documents contents to select this as it appears most appropriate.

entropy() -For anyone familiar with probability maths or the physical concept of entropy this is also straight forward. The concept behind an entropy model is they limit the amount of prior information required to populate a probability distribution. So if you know nothing about the given data entropy might give you an unbiased probabilistic account of the data. However it is not always appropriate given the outcome of the analysis.

gw_entropy() - returns one plus entropy.

If you skipped the detail start here, otherwise good on you for ploughing ahead! Now that our data is transformed we can FINALLY perform a LSA analysis. While the call is simple in R, the function returns three matrices equivalent to a Single Value Decomposition (SVD).

V- A transformation of columns by columns (m*m)

Σ- A diagonal matrix of intersections representing the single values (m*n)

U- A transformation of columns by columns (n*n)

Each of these matrices contribute to a LSA space. We will need these matrices to plot the distances between documents at the end.

We then call the dist() function to compute the similarity between documents by evaluating the words contained within them. This will give us a vector object which is a continuous list of values, nothing more.

<<<<<<< HEAD Then use the cmdscale (classic multi-dimensional scaling) to compute a five-piece object. I won’t go into detail on the whole five parts, only the piece that we will be using. The ‘points’ component is a tibble, or two value pairs, that can be associated with Cartesian coordinates, which means, we can plot them! We acheieve this by specifcying k=2, k referring to groups/dimensions and 2 meaning two of them. ======= Then use the cmdscale (classic multi-dimensional scaling) to compute a five-piece object. I won’t go into detail on the whole five parts, only the piece that we will be using. The ‘points’ component is a tibble, or two value pairs, that can be associated with Cartesian coordinates, which means, we can plot them! We acheieve this by specifcying k=2, k referring to groups/dimensions and 2 meaning two. >>>>>>> 3032d8e12e25dfbf73eebc5d7b650064da063c0a

# Classic Multi-Dimensional Scaling for Graphing
fit <- cmdscale(dist.mat.lsa, eig=TRUE, k=2)
points <- data.frame(x=fit$points[, 1], y=fit$points[, 2])

We change this component into a data frame and plot it. Here I have also eliminated outliers for ease of viewing. There were two documents that were mostly legal text that didn’t relate to others at all and ruined the graph scale.