• Treat – Text REtrieval and Annotation Toolkit, definitely the most comprehensive toolkit I’ve encountered so far for Ruby
  • Text extractors for various document formats
  • Chunkers, segmenters, tokenizers
  • LDA
  • much more – the list is big
• Ruby Linguistics – this is one of the more ambitious projects, but is not as mature as NLTK
  • interface for WordNet
  • Link grammar parser
  • some inflection stuff
• Stanford Core NLP – if you’ve gotten a headache trying to use the Java bridge, this is your answer
• Stanford Parser interface – uses a Java bridge to access the Stanford Parser library
• Mark Watson has a part of speech tagger [zip], a text categorizer [zip], and some text extraction utilities [zip], but I haven’t tried to use them yet
• LDA Ruby Gem- Ruby port of David Blei’s lda-c library by yours truly
  • Uses Blei’s c-code for the actual LDA but I include some wrappers to make using it a bit easier
• UEA Stemmer – Ruby port (again by yours truly) of a conservative stemmer based on Jenkins and Smith’s UEA Stemmer
• Stemmer gem – Porter stemmer
• Lingua Stemmer - another stemming library, Porter stemmer
• Ruby WordNet - basically what’s included in Ruby Linguistics
• Raspell – Ruby interface to Aspell spell checker

There are also a number of fledgling or orphaned projects out there purporting to be ports or interfaces for various other libraries like Stanford POS Tagger and Named Entity Recognizer.  Ruby (straight Ruby, not just JRuby) can interface just about any Java library using the Ruby Java Bridge (RJB).  RJB can be a pain, and I could only initialize it once per run (a second attempt never succeeds), so there are some limitations.  But using it, I was able to easily interface with the Stanford POS tagger.

So while there aren’t terribly many libraries for NLP tasks in Ruby, the availability of interfacing with Java directly widens the scope quite a bit.  You can also incorporate a c library using extensions.

Naturally, if I missed anything, no matter how small, please let me know.

Update: Here is a great list of AI-related ruby libraries from Dustin Smith.

Whenever something like this happens, you will see three groups of commenters emerge.  First are the I-don’t-care’s.  Actually, you don’t see them since they don’t give a crap.  The next are the fanboys.  They love the new language and are glad that MIT has discarded a dinosaur in favor of the language of Heaven.  And finally you have the sticks in the mud who lament the death of computer science because a whole generation will grow up retarded thanks to not learning programming just the way they did.  Obviously, these are exaggerated — I say it to shock the mind.

Cognitive psychology would have me believe that by drawing stark lines and exaggerating the situation, I will actually cause people to align themselves more closely with the stereotypes I laid out.  The logical alternative would be to view it as a joke, take a step back, and examine your own reaction.  Why do people get so worked up about this?  Why do I get so worked up about people getting so worked up?  :P

Maybe I’m getting crotchety in my old age.

We spent the day in a fairly hot conference room hacking away on a variety of projects.  We set up teams beforehand to work on projects that people proposed and I chose to work on a variation of a computing puzzles site, dubbed LangWar.  The idea is fairly simple in the early stages:  people submit programming puzzles and other people post their solutions in code form.  You can vote which questions you like and which answers you like (or dislike).  You can also leave comments on questions and answers.  The result of the ratings is that the best questions will be counted higher, in a method similar to Reddit, and the best answers will trickle to the top based on the votes of users.

This is very similar to Stack Overflow, but different in that it is intended to be more of a puzzle solving site that pits implementations in different programming languages against each other.  It’s sort of a battle royale of programming languages – thus the name, LangWar.  It’s more of an enhanced version of Project Euler, where people can vote on the questions and answers.

In any case, it was a great chance to get my hands dirty in Google App Engine, meet some Atlanta python coders, and have fun.  It’ll be interesting to see where LangWar goes from here, if it does go anywhere.

I was reading through Pang et al (2002), which deals with classifying movie reviews as positive or negative.  They look at three machine learning approaches:  Naive Bayes, Maximum Entropy classifier and Support Vector Machines.  This seemed like a good opportunity to try out my nascent ruby skills, since it’s the kind of crap I can roll together in python in short order (and do all the time).  So I downloaded the data for the paper (actually I downloaded the later data from the 2004 paper).  There are 1000 positive and 1000 negative movie reviews.  The task is to train a classifier to determine whether a review expresses a positive opinion (the author liked the movie) or a negative opinion (the author did not like the movie).  I chose to just use SVMs since they do best for this task according to the paper, they do really well for text categorization, and they are easy to use and download.

The results were quite nice.  Ruby turned out to be just as handy as python at manipulating text and dealing with crossfold validation:  the two main “challenges” in implementing this paper.  I used tf-idf for weighting the features and thresholded document frequency to discard words that didn’t appear in at least three reviews.  The result was that I achieved about 85.7% accuracy using the same cross validation setup described in their followup work (Pang and Lee, 2004).  In other words, the classifier could correctly guess the opinion orientation of reviews as positive or negative nearly 86% of the time.

Pang et al (2002) discussed some of their errors and hypothesized that discourse analysis might improve results, since reviewers often use sarcasm.  There’s also the case where authors use a “thwarted expectations” narrative.  This offered me one of the few chuckles I’ve ever had while reading a research paper:

“I hate the Spice Girls. … [3 things the author hates about them] …  Why I saw this movie is a really, really, really long story, but I did and one would think I’d despise every minute of it.  But… Okay, I’m really ashamed of it, but I enjoyed it.  I mean, I admit it’s a really awful movie …the ninth floor of hell… The plot is such a mess that it’s terrible.  But I loved it.”

References

Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan.  ”Thumbs Up?  Sentiment Classification Using Machine Learning Techniques.”  In Proceedings of the ACL 02 conference on Empirical Methods in Natural Language Processing – Volume 10, July 2002. [pdf]

Bo Pang and Lillian Lee.  ”A Sentimental Education:  Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts.”  In Proceedings of the ACL, 2004. [pdf]

This code is open source under the Creative Commons 3.0 Attribution license that this blog uses Creative Commons BSD license. Neville’s code appears to be under CC:Attribution 2.5 for South Africa, by what I could glean from his site. I have considered making this an open source project under Google code but have yet to take it all the way. Google sets a lifetime limit of 10 projects, so I will continue to hoard those against future need. If you make modifications to the code, please let me know and I will probably post them here and in the code for future releases, so we all win.

Note that the command line parameters have changed:

plurk.py <twitter username> <twitter password> <plurk username> <plurk password>

And of course, as with all software, use at your own risk.

The resulting tweet looks like this:

Download the script and set it up as a cron job (or you could execute it manually). It should work with python 2.4 and later. It stores a plurkdb.dat file (which you should probably assign an absolute path to, depending on the behavior of cron on your system). This file is checked every time it is run to make sure that duplicate plurks aren’t being tweeted. You should pass the following parameters on the command line (or modify the script so they are hardcoded, if you want): <twitter username> <twitter password> <plurk username> <plurk password>. Update: see later post on updated plurk script.  And like with all software, use at your own risk.

Please let me know if you have any problems with it or see room for improvement. I hacked this out in a hurry, so …

Calais is put out by Reuters which has a long history of helping out the NLP and IR research communities with data sets. Being Reuters, the data sets are all newswire stuff, and Calais is produced in that spirit. Currently the relationships and named entities available reflect that bias, but the list is expanding and it is probably flexible enough for most domains. Their claim is that with each new release, there will be additional entities and relationships available. Also, the software is completely open source free for commercial and private use. For this, I give Reuters props.

OpenCalais uses SOAP or HTTP post to issue requests and you can take a look at their tutorials for exactly how to use it. After some very shallow digging on the googles, I found an open source project called python-calais, which is basically just a script that wraps some text and sends it to the Calais service, then processes the output. The output is in RDF (resource description framework), which is a type of xml document that is not very friendly to the human eye but is nice and powerful otherwise. The python-calais script uses an rdf library for python, so you’ll need to download that if you don’t already have it.

Running it on my most popular post, you get the following output:

93B6642D-0D7C-37Ab-A92F-66Ebfef13C8D :: Recommender Systems (Industryterm)
0Dccb106-442A-3848-Bd0B-A388E73F4C8C :: Chris Sternal-Johnson (Person)
Aab0D16A-Ad5A-348A-A8Dc-58Cf59A1Bc15 :: Kristina Tikhova (Person)
42F476A0-2Fae-3F36-808D-803E4F620Ab0 :: Java (Technology)
6C4Cd5D9-5866-35B5-81Ab-B8A5C1751A44 :: Pre-Processing Phase (Industryterm)
4003D863-C7A6-3E6F-8E3C-0913Bf2F8242 :: National Aeronautics And Space Administration (Organization)
77D1Ceb3-9900-3Dd7-8351-F29408B21412 :: Carnegie Mellon University (Organization)
Ee58Ef4B-1C98-3F8B-Aff8-3Fd6E3D76A9E :: Wonderful Site (Industryterm)
8F12E551-A8F1-3705-866C-D44D1A6A54F4 :: Richard M. Hogg (Person)
Adee23De-B1B0-37Ad-9E20-1Fa8094F6D39 :: Steel (Industryterm)
0Ace00C6-2B9F-32C2-8949-82A0F6C6B444 :: Xml (Technology)
2Ed2F085-1C63-324E-B518-60332388E273 :: Norman French (Person)
136157D8-D62E-3C55-Ae67-3Ec182C2C703 :: Phil Barthram (Person)
B6A8Dbfa-Fd35-32Bb-9E05-A2811C480000 :: Mike Tan (Person)
Ed8B5Fe4-616A-36Ea-8C47-3Eea7C71Aee0 :: Ben Eastaugh (Person)
D3Bcba58-00Fc-33C5-9346-Dbf6A2441867 :: Machine Learning (Technology)
F17C3779-3810-3Ff9-A42D-75C3137F0F7F :: Modern English (Person)
38116E8D-F8B4-3D03-B0Ad-C9A24B888E61 :: Jason M. Adams (Person)
4386B07C-F6B8-3991-Af74-Ab11A951F0Ee :: David Petar Novakovic (Person)
Aa14303F-F9F0-31B8-Adff-3B9C68E0A9F1 :: Language Technologies Institute (Organization)
Ca1E4Eb7-7820-3862-8443-26E37B33E13F :: Machine Translation (Technology)

As it picks up everything on the page, there is a lot included there that isn’t related to the post about Old English translation. Also, it picks up some weird so-called industry terms like “steel.” If you filter out just the text (manually), the output is a little more sensible:

6C4Cd5D9-5866-35B5-81Ab-B8A5C1751A44 :: Pre-Processing Phase (Industryterm)
Ca1E4Eb7-7820-3862-8443-26E37B33E13F :: Machine Translation (Technology)
0Ace00C6-2B9F-32C2-8949-82A0F6C6B444 :: Xml (Technology)
2Ed2F085-1C63-324E-B518-60332388E273 :: Norman French (Person)
136157D8-D62E-3C55-Ae67-3Ec182C2C703 :: Phil Barthram (Person)
F17C3779-3810-3Ff9-A42D-75C3137F0F7F :: Modern English (Person)

(The codes are unique identifiers.) Unfortunately, some important terms are still missed, like Old English. So it appears Calais has some growing to do, but it’s off to a good start. Part of the problem might be that that blog post is out of domain. I imagine with time, it will continue to improve. We’ll see.

First let me clarify by saying that I’m not suggesting Matlab is copying Python (though they may be) or that Python has the market cornered on the similarities I’m about to talk about (it doesn’t). Also, I have no idea when this stuff was introduced to Matlab, so it might have all been there for years.

The first thing of interest to me was the discussion about memory management in Matlab. Suppose you create a matrix and store it in the variable X. Next, you assign X to Y: Y = X. Here, Y acts as a pointer to the data pointed to by X. If you clear X, Y still points to the data, so it continues to reside in memory until Y has been cleared also. Now if you modify Y, perhaps like so: Y(1,1) = 2, my thought was that both X and Y would be affected and the memory used would stay the same. However, that is not the case, thanks to the BLAS and LAPACK libraries that require matrices to be contiguous in memory. So when you modify Y as above, it creates a modified copy in memory that Y now points to, while X remains pointing to the same, unmodified data. This is decidedly not Pythonic, since in Python that modification would have updated both X and Y, and they still would be pointing to the same thing in memory.

The next set of interesting things were functions. Here is where things struck me as Pythonic. You can declare a simple function in an m-file like so:

function Y = square(x)
Y = x.^2
end

Here the end is optional. Like in Python, you can use function handles. This lets you do cool things like decide on the fly which operation you can apply to a matrix without having to worry about record-keeping and if checks (I’m not going into this any further). To demonstrate the function handle, here is one possibility:

fh = @sin
fh(pi)
fh = @cos
fh(pi)

We call the function handle fh on the same input, but get two different results: 0 and -1. Matlab also lets you create factory functions like in Python:

function Y = makeF(a,b)
Y = @makeFHelper;
function Z = makeFHelper(x)
Z = a * sin(x) + b;
end
end


This returns a function handle to the subfunction makeFHelper with the parameters you passed hard-coded in the function produced. So you could create two different functions:

Y1 = makeF(2,3)
Y2 = makeF(3,-1)

and when you execute them:

Y1(pi/3)
Y2(pi/3)

you get two different results: 4.7321 and 1.5981. If I had known about this before, I might’ve put more effort into learning Matlab programming, since it would have come in handy for a couple assignments this semester.

I felt the exact same way when I first picked up python.  It was like finding the holy grail of programming languages.  To be able to just throw things into a list and access them without having to worry about casting.  To throw around functions like they were variables.  To weave functions out of thin air and watch them vanish when their usefulness had expired.  It was magic.

Of course, the honeymoon faded.  I still use python as a first resort.  As a programming language for exploring new ideas, it can’t be beaten.  Development time is ridiculously fast.  There has been effort to get the runtime up to snuff as well, but with much reluctance I’m forced to admit it doesn’t compare to C or even Java, may God have mercy on my soul.  Granted, it all depends on the application, blah blah blah.

Despite all that, I still love it.  It’s definitely first in my heart as far as programming languages go.

So I’ve been reading A New Kind of Science by Stephen Wolfram, the creator of Mathematica. It was hyped up big time back when he first wrote it, since he had gone silent for a number of years, hinting that he was about to do something big. So my middle little sister got me the book for Christmas (cuz she rocks) and I cracked it open a few times. It’s about 846 pages of text (yipes!) and then another 351 pages of notes. Quite daunting. So I put it down and have meant to pick it back up a thousand times. Today I was needing a diversion because a particular C++ issue was giving me fits.

In Chapter 2, Wolfram introduces a fairly simple 2-dimensional cellular automata (one spatial dimension, one temporal dimension). The temporal dimension can be plotted as another spatial dimension producing a nice little spreadsheet style graph. Each cell of the graph can be considered a bit. Depending on whether the bit is set, the cell is either shaded or not. So the single line in the spatial dimension contains some initial setting. Let’s say there is one single bit set in the middle of the line, so it might look like this:

000000000010000000000

To move to the next time step, we must look at the neighborhood of each cell. A neighborhood for the middle cell of length 3 would be [010]. Another way of thinking about this neighborhood is as the bits of a number. Here we have three bits so we can represent the numbers 0-7. For each possible setting, we can create a rule. The rule simply says whether the bit is set or not. So let’s use the following rules:

• 0 => 0
• 1 => 1
• 2 => 0
• 3 => 1
• 4 => 1
• 5 => 0
• 6 => 1
• 7 => 0

As we step through our board, we look at the neighborhood of three for each cell. Let’s assume wrapping occurs at the edges, so for the first cell, the neighborhood is [000]. Most are zeros until we get to the 10th index (bolded, neighborhood underlined):

000000000010000000000

The neighborhood now is [001], which corresponds to 1 in our rules. So we output 1 in our new line. The next neighborhood is 010, which corresponds to rule 2. So we output 0. Next we have [100], which is rule 4, so we output 1. After we’re done processing this line, we have the new line at time step 2:

000000000101000000000

As we iterate through this ten more times, we begin to notice a pattern emerging:

000000000010000000000
000000000101000000000
000000001000100000000
000000010101010000000
000000100000001000000
000001010000010100000
000010001000100010000
000101010101010101000
001000000000000000100
010100000000000001010
100010000000000010001

Actually, it’s probably harder to notice in this crappy text form, so here is the graphical representation:

So with just those simple rules we could construct a fractal known as the Sierpinski triangle. Pretty frickin sweet. In Wolfram’s nomenclature, this is cellular automaton 90 (since the rule outputs form the binary number 01011010 = 90). I hacked together some quick python code to display such cellular automata given the size of the neighborhood and the Wolfram number for the automaton you want.

So to run the code below for the Sierpinksi triangle for 40 cycles (recommended), you’d use the format:

python ca.py 3 90 40

Download it here since this sourcecode plugin isn’t highly reliable and may have bugs in the output it shows you.

# cellular automaton

__version__ = "0.1"
__author__ = "Jason Michael Adams (jmadams@cs.cmu.edu)"

import math
import sys

class CellularAutomaton(object):
   """Usage: CellularAutomaton(boardSize, ruleSize, automatonNumber)

   """

   def __init__(self, size, rule_size, effects, initial_board=None,alpha=None):
      assert size % 2 == 1, "Size of board must be odd."
      self.size = size
      self.time = 0
      self.rule_size = rule_size
      rules = list()
      for x in xrange(2 ** rule_size):
         rules.append(make_binary_rep(x, rule_size))
      rules.reverse()
      ex = make_binary_rep(effects, 2 ** rule_size)
      self.rules = RuleSet(rules, ex)
      self.board = dict()
      if alpha is None:
         self.alpha = [0,1]
      else:
         self.alpha = alpha

      self.board[0] = self._init_board(initial_board)
      if initial_board is None:
         self.board[0][self.size / 2] = self.alpha[1]

   def increment(self):
      self.time += 1
      self.board[self.time] = self._init_board()

      for x in xrange(self.size):
         context = list()
         rn = self.rule_size / 2
         for y in xrange(self.rule_size):
            context.append(self.board[self.time - 1][(x + y - rn) % self.size])

         self.board[self.time][x] = self.rules * context

      return self.board[self.time]

   def run(self, cycles):
      for x in xrange(cycles):
         tmp = self.increment()

      print self

   def _init_board(self, initial_board=None):
      if initial_board is None:
         board = list()
         for x in xrange(self.size):
            board.append(self.alpha[0])
         board[self.size / 2] = self.alpha[1]
      else:
         assert len(initial_board) == size, "Size of initial board setting must be equal to size of board."
         board = initial_board

      return board

   def __repr__(self):
      keys = self.board.keys()
      keys.sort()
      tmps = list()
      for key in keys:
         tmps.append(self.__repb__(key))
      return '\n'.join(tmps)

   def __repb__(self, b):
      board = self.board[b]
      tmp = ""
      for x in xrange(len(self.board[b])):
         tmp += str(self.board[b][x])
      return tmp

class Rule(object):
   """Usage:  Rule(rule, effect)

   The rule parameter is a odd-length list indicating the
   prior pattern that will trigger the single value in
   effect.  The effect parameter should be anything that
   has a string representation.
   """

   def __init__(self, rule, effect):
      assert len(rule) % 2 == 1, "Size of rule must be odd."
      self.rule = rule
      self.effect = effect
      self.size = len(rule)

   def __mul__(self, context):
      assert len(context) == self.size, "Size of context must be equal to size of the rule (%d)." % (self.size)

      accept = True

      for x in xrange(len(context)):
         if self.rule[x] != context[x]:
            accept = False
            break

      if accept is True:
         return self.effect
      else:
         return None

   def __repr__(self):
      return "%s => %s" % (str(self.rule), self.effect)

class RuleSet(object):
   """A set of Rule objects that can be applied to
   a given context.
   """

   def __init__(self, rules, effects):
      self.rules = list()
      for rx in xrange(len(rules)):
         r = Rule(rules[rx], effects[rx])
         self.rules.append(r)

   def __mul__(self, context):
      middle = context[len(context) / 2]
      for rule in self.rules:
         outp = rule * context
         if outp is not None:
            return outp
      return middle

   def __repr__(self):
      return str(self.rules)

def make_binary_rep(num, size=0):
   outp = list()
   if size < = 0:
      size = int(round(math.log(num) / math.log(2)))
   for x in xrange(size - 1, -1, -1):
      if (num & (2 ** x)) > 0:
         outp.append(1)
      else:
         outp.append(0)
   return outp

if __name__ == '__main__':
   if len(sys.argv) < 4:
      print "Usage: %s   <# cycles>" % (sys.argv[0])
      sys.exit(0)
   nhsize  = int(sys.argv[1])
   autonum = int(sys.argv[2])
   cycles  = int(sys.argv[3])
   ca = CellularAutomaton(81, nhsize, autonum)
   ca.run(cycles)

This is some pretty simple stuff, but I figured I’d post it in case anyone wanted a starting point or just something to play with. There are also a number of things you can do to increase speed that I didn’t bother with. Please report bugs, etc.

ov umy rfgs f nmg
MY DOG EATS A LOT

There are many ways of going about solving substitution ciphers, but a common way is by counting frequencies of characters. As most people know, e tends to occur more than other letters in most written English. The rest of the letters typically follow a pattern, as well, but that pattern degenerates once you leave the most common letters. The domain of the text you are examining is fundamentally important here. By domain I mean whether this text is from a newspaper, an IM, transcribed speech, etc. You can also look at bigrams, two character sequences, to find the most commonly appearing sequences. In English, th appears much more ty, but ty still occurs. When trying to solve substitution ciphers this way, you are essentially matching the frequency distribution of the cipher text to the distribution of English and building a mapping from there. To put that a different way, you are matching up the most common letters or sequences in the garbled text with the most common real English letters or sequences.

Once the frequency counts have revealed the most common letters, many people proceed to deduction to eliminate the rest. Of course, this requires knowledge of English words directly, which has an impact on computational approaches to solving substitution ciphers automatically. I’m curious what approaches people have taken (if any) other than using a dictionary of English words and trying to find matches from there.

So, I’ll talk more about this later, after the homework assignment is due, but there is an interesting connection here with finite state transducers. Here is a little bit of python code to do some simple stuff with substitution ciphers in case you want to play around. I have found python a very handy tool in decrypting these ciphers. Anyone have any other tool they like and want to share?

def decrypt(words, charmap):
    # Given the ciphertext in lowercase (words) and a
    # mapping of ciphertext to upper case plaintext
    # (charmap), return the transformed string
    
    tmp = [" "] * len(words)
    for x in xrange(len(words)):
        if charmap.has_key(words[x]):
            tmp[x] = charmap[words[x]]
        else:
            tmp[x] = words[x]

    return tmp



def encrypt(words, charmap):
    # begin by making a reverse map
    rev_charmap = dict()
    for key in charmap:
        rev_charmap[charmap[key]] = key

    return decrypt(words, rev_charmap)



def make_random_charmap():
    letters = list()
    # make the list of uppercase letters
    for x in xrange(26):
        letters.append(chr(ord('A') + x))
    random.shuffle(letters)
    charmap = dict()
    for x in xrange(26):
        charmap[chr(ord('a') + x)] = letters[x]

    return charmap

And I’m only semi-fond of this plugin for source code. It does some weird crap and is really temperamental.

“Advancing text entry exponentially, kannuu’s powerful and precise Partial Word Completion® technology enables users with a fail-safe text entry solution. The kannuu application appears on device, as a four-point diamond shape, comprised of the most popular letters in the database it is indexing, with the center kannuu logo leading to the next set of choices.”

They registered a trademark on the phrase “partial word completion”?? Blerg. Not only do they have an über lame web 2.0 name in lowercase, they gotta stop people from marketing a similar technology under their oh-so-not-original name. Why does this make me so angry? Anyhow, I’m running off sideways on a rant that is pretty insignificant.

The real point here is the potential for coolness. So here is the technology: you enter a letter, it presents you with a “diamond” shape and the most common letters or group of letters that follow the letter(s) you just entered. In this way, most of your everyday phrases will be right up at the top of the list of things you’re presented so you could potentially be entering words with fewer keystrokes and all with very little thumb movement. This could really revolutionize key input and maybe bring pocket computers to reality [source].

So here is what I think the technology is based on. A very common technique in language technologies is the use of n-grams. So they use a character-based n-gram model to predict the most common letter or letters that you would type next based on some corpus. This isn’t anything new. Cell phones already have a T9 input method that guesses the most common word based on the single letters you choose. This isn’t all that different. If they have done the interface well, that could be a serious improvement.

If you’re interested in character-based n-gram models, I go into them in more depth after the jump.

Think about this. In English, the most common letter is e. If you don’t believe me, count the letters in the paragraph above that starts with “the real point”.

You can take these counts and get probabilities for the occurrences of each letter. These probabilities are useful because they give you a general idea of how often each letter occurs in English. Of course, the corpus I’m pulling them from is way too small to be representative of all of English, but it’s a start. Since we are looking at the probability of only one character, we call this a unigram model (1-gram). We can generate English text with this unigram model by sampling from the distribution. Some python code for that is below:

# code for generating from a toy unigram character-based language model for English
import random

letter_counts = [(1, 'j'), (1, 'z'), (4, 'k'), (4, 'v'),
(5, 'b'), (6, 'g'), (7, 'f'), (8, 'c'),
(8, 'w'), (10, 'm'), (11, 'd'), (13, 'p'),
(16, 'u'), (17, 'y'), (19, 'a'), (21, 'h'),
(22, 'n'), (24, 'i'), (25, 'l'), (27, 's'),
(30, 'r'), (41, 'o'), (53, 't'), (60, 'e'),
(93, ' ')]

d = [letter_counts[0]]      # initialize the sampling list to the first item in the list of counts
tmp = ""

for i in xrange(1, len(letter_counts), 1):
d.append((d[i-1][0] + letter_counts[i][0], letter_counts[i][1]))
for i in xrange(526):
r = random.randint(0,525)
for letter_pair in d:
if r < letter_pair[0]:
tmp += letter_pair[1]
break

print tmp

So we take the list containing the counts of the letters (and the space character) and then create a new list where each element consists of a tuple. The first element of the tuple is the sum of all counts of items in the list before it plus the count for the character, which is the second element in the tuple. This is just a sum of all the previous letter counts. Then we choose a random number and if the random number is less than the current sum, we add that letter to the output string. The idea is that if we have chosen our probabilities correctly, we should get something that looks like English:

h spiyoweytnt berkctpllittdulwa nhesh oensnuvuwgyysyeamcgt fir v oe obnstotitres ttelsdmt e hr liruorteevunaet uwyooe trewstpaayssers onyewiasne r oae yceu ruiz lorpse hg otderdretuk otlm odulloetnssaio wtygre ro gh eietvyrryeteetoat au y r vpte peascotog reiol cth enchl rpsn rtge saldi tngai iuentbdysbte eagrog nuee wetemsr srultnomisojhteolrtu t neyerhul io oe eeror dalts lr ib somunerno situlcihi e a j ec eotutnerrnu dpl io prntb t yli htsttiwmyop oydstwihn h loeebleeh ovebnrtoyldothdgo yto l bi

And of course, it looks nothing like English. The problem is that it doesn’t take the context of the letters into account. If you see a word that starts with t, what do you think the next letter will be? Probably h, right? And what’s the next letter after that? E. A lot of other letters co-occur in English, like ch and ou, to name a couple. So we can build a bigram model and this gets a bit closer to generating English-like words. The problem then becomes sparsity. We need a bigger corpus to accurately compute the probabilities for the bigram (2-gram) character based language model. But here is what a bigram model would generate using this data:

bey temm onowid eryobethigzee led hiorury tethp s nil sotr t iokd ngayle tt ndu s shtthefey ord tus co ere rsbemotemo cetntpoioouayinpuybeskeenlievngrsen yp e assfoan lhepud keleolerme ketlehuh or mene pndy ng ds keoc eerryt ttris hea thea tjug s anhee li h l ye sthillked ou aiz iheer mu aomheurt se be ct s ompulun urhrasdserrs t cth the rttrireese s has dtlo lurd t hnu luesetnptellamofery rstye hleoug hendisluupntwore sbethumouulf d r leitd ketib e zes io wu amerf rooua ketyomd al phesteaeyryntofe ioayalougyiti

So, it looks a little better, but it’s still clogged with junk. You’ll never get a model that produces all English words this way, but when you analyze enough English, you can reasonably predict the most common set of next letters you’ll use. There are plenty of other methods for doing this and improving the results.

One of the things the SciAm blog mentioned was having to enter scientific text. If they train the n-gram model using regular conversational English (or some corpus of email or something like that), scientific jargon is very unlikely to appear. The result is a much slower entry process. What they need to allow you to do is “create your own corpus.” You add the documents that are best representative of the kind of stuff you’ll be writing and it builds a model from that. Of course, it will need a lot of text to be really good, but it should start to work decently after a few hundred thousand words (or maybe even less). I know you can detect which language a text is written in decently well with less than a hundred thousand words of training text per language. If it allowed custom corpora for training, all it would need is the ability to switch language modes and you’re set. Also, it could learn which choices you are more likely to make over time and start suggesting things that you choose more often.

The idea is simple. Enter some keywords and get matching free items off of craigslist for your city sent to your cell phone. You can enter up to 5 sets of keywords and each set has exclusion terms. This makes it so that you can receive notices with the term computer but without the term desk. Israel just took the site out of private beta last week and it is currently in public beta mode.

It was an interesting project for me because it gave me the chance to work in python on some http and smtp protocol code. It also gave me the chance to work on processing xml and rss feeds. Definitely some cool stuff there and it has resulted in a spin-off that will probably be functioning fairly soon. Israel is one of those people with a lot of great ideas and he has the personality to inspire you with them. Plus he is also one of those rare people that actually care enough about the suffering of others to actually try to do something about it, which you just have to admire.

So, please, check it out and let us know how we can make it better.