Posts Tagged ‘ruby’

I just published the simple-random ruby gem, which is ported from C# code by John D. Cook.  You can view the source on github or install the gem via rubygems:

gem install simple-random

The gem allows you to sample from the following distributions:

  • Beta
  • Cauchy
  • Chi Square
  • Exponential
  • Gamma
  • Inverse Gamma
  • Laplace (double exponential)
  • Normal
  • Student t
  • Uniform
  • Weibull

Simple examples:

require 'rubygems'
require 'simple-random'

r = SimpleRandom.new
r.uniform # => 0.127064087195322
r.normal(5, 1) # => 5.71972152940515

Advertisements

Wordnik Gem

Posted: 12 March 2010 in Uncategorized
Tags: , , , , ,

Erin McKean

I’ve had my eye on Wordnik for a while, since finding out the excellent lexicographer Erin McKean co-founded it.  Wordnik is the most comprehensive dictionary in the known universe.  Srsly!

They released an API a few months ago and I quickly threw together a gem wrapping it, based on HTTParty.  Tonight I updated the gem for version 3 of the API and simplified it to just a single class with the bare essentials.  You can perform pretty much all of the API calls and get a hash of the results.  It’s nothing major, but will give you a chance to play around with the Wordnik API with almost no work on your part (aside from getting yourself a key).  This change breaks backwards compatibility completely, sorry.

Example usage:

w = Wordnik.new("YOUR_API_KEY")
w.define('gem') # => big hash with all the definitions
w.examples('gem') # => example sentences using "gem"

You can grab the gem off of RubyGems or you can take a look at the source on github.  As always, please let me know if you encounter any problems.

TunkRank Improvements

Posted: 17 February 2010 in Uncategorized
Tags: , , , , , , , , ,

Over the past few weeks, I’ve been working on a number of improvements to TunkRank that I will be rolling out tonight. First, I’ve secured a server to host it on, rather than my old Dell laptop, so reliability should improve and TunkRank is no longer a slave to dynamic DNS problems. Also, my cable company is less likely to hunt me down. TunkRank has gotten some increased attention over the past few weeks, including from Chris Dixon, CEO of the wonderful website hunch:

Twitter could fix the whole follower obsession by highlighting a more meaningful metric like TunkRank.

Awesome! So with this new version, there are a few changes that will immediately impact you, the end-user. I’ll go into the ones that affect you the most first, followed by some technical points of interest for those who care. Then I’ll conclude with a couple of hints at the future.

Changes to TunkRank

First and foremost, I have changed the main score that is reported. Previously I was using a percentile in the range (1-100). This got a lot of objections and created confusion. Partially because I consider the 100th percentile to be the “top-tier” of users, while standardized testing often reports the 99th percentile to mean you performed better than 99% of the population. Also, most people who actually care about their scores enough to use TunkRank are in the 95-100 percentile range, making more fine-grained comparisons difficult. Neal Richter even posted on his blog some suggestions for improving it (quite a while ago, now).

I took a page out of Neal’s book with the log scores, but I also put it in a range where the most influential twitter user (let’s call her MAX) will always have a score of 100. Your TunkRank Score™ is the ratio of the log of your raw score to the log of MAX’s score. So formulas aside, this means your TunkRank score is directly comparable to other users and is always in perspective of the maximum influence exerted by any user in the Twitterverse. Incidentally, comparing users with a difference of seven TunkRank score points means the user with the higher score is about twice as influential.

Accessing the API has also changed slightly, and I apologize to anyone actually using it at the moment. Basically, I am matching the API calls to more closely conform to the URLs used on the web side, and I’m returning more information with each call. TunkRank also supports XML responses in addition to JSON. You can find all of the documentation here.

Some Technical Notes

As part of the move, I’ve decided to transition from using Merb to Rails. My original decision to use Merb was partially as a learning exercise, but also because Merb appealed to me with its being lightweight. However, I often ran into roadblocks because some useful plugin wasn’t supported (or I couldn’t figure out how to make it work in the limited time I had). Sometimes the documentation for Merb was very good and sometimes it was absent altogether. Rails, on the other hand, has a substantial amount of documentation and people are always blogging about the best way to do things — which makes life as a developer much easier. Rails is my day job, so I knew I could transition quickly and easily.

I also migrated from MySQL to PostgreSQL. The main reason is that I love PostgreSQL — plain and simple. They both have their advantages, but MySQL gives me a sense of uneasiness I don’t have with PostgreSQL. I’ve managed to achieve some nice speed improvements as part of the redesign, though that is not to say that the same speed improvements wouldn’t have been possible with MySQL.

I’ve also adopted Resque as my background job-processing library. It is backed by Redis, an advanced key-value store that you can think of as a “data structures server.” The important thing for me is that Resque is fast, has a kick-ass web interface, and integrating with Rails is brain-dead easy.

The Road Ahead

I wrote before about the road ahead for TunkRank, and I have mostly held to it. I have many more ideas I want to expand on, including topic-sensitive influence rankings. I like the ideas in the recent WSDM paper (pdf) by Weng et al, but I have a few new ideas I’m eager to try out. TunkRank scores may also be integrated into Tickery in the near future, thanks to some discussions with Terry Jones of FluidDB.  I’m excited!

There are quite a few well-known libraries for doing various NLP tasks in Java and Python, such as the Stanford Parser (Java) and the Natural Language Toolkit (Python).  For Ruby, there are a few resources out there, but they are usually derivative or not as mature.  By derivative, I mean they are ports from other languages or extensions using code from another language.  And I’m responsible for two of them! :)

  • Treat – Text REtrieval and Annotation Toolkit, definitely the most comprehensive toolkit I’ve encountered so far for Ruby
    • Text extractors for various document formats
    • Chunkers, segmenters, tokenizers
    • LDA
    • much more – the list is big
  • Ruby Linguistics – this is one of the more ambitious projects, but is not as mature as NLTK
    • interface for WordNet
    • Link grammar parser
    • some inflection stuff
  • Stanford Core NLP – if you’ve gotten a headache trying to use the Java bridge, this is your answer
  • Stanford Parser interface – uses a Java bridge to access the Stanford Parser library
  • Mark Watson has a part of speech tagger [zip], a text categorizer [zip], and some text extraction utilities [zip], but I haven’t tried to use them yet
  • LDA Ruby Gem– Ruby port of David Blei’s lda-c library by yours truly
    • Uses Blei’s c-code for the actual LDA but I include some wrappers to make using it a bit easier
  • UEA Stemmer – Ruby port (again by yours truly) of a conservative stemmer based on Jenkins and Smith’s UEA Stemmer
  • Stemmer gemPorter stemmer
  • Lingua Stemmer – another stemming library, Porter stemmer
  • Ruby WordNet – basically what’s included in Ruby Linguistics
  • Raspell – Ruby interface to Aspell spell checker

There are also a number of fledgling or orphaned projects out there purporting to be ports or interfaces for various other libraries like Stanford POS Tagger and Named Entity Recognizer.  Ruby (straight Ruby, not just JRuby) can interface just about any Java library using the Ruby Java Bridge (RJB).  RJB can be a pain, and I could only initialize it once per run (a second attempt never succeeds), so there are some limitations.  But using it, I was able to easily interface with the Stanford POS tagger.

So while there aren’t terribly many libraries for NLP tasks in Ruby, the availability of interfacing with Java directly widens the scope quite a bit.  You can also incorporate a c library using extensions.

Naturally, if I missed anything, no matter how small, please let me know.

Update: Here is a great list of AI-related ruby libraries from Dustin Smith.

works-on-my-machine-starburstA while back I ported David Blei’s lda-c code for performing Latent Dirichlet Allocation to Ruby.  Basically I just wrapped the C methods in a Ruby class, turned it into a gem, and called it a day.  The result was a bit ugly and unwieldy, like most research code.  A few months later, Todd Fisher came along and discovered a couple bugs and memory leaks in the C code, for which I am very grateful.  I had been toying with the idea of improving the Ruby code, and embarked on a mission to do so.  The result is a hopefully much cleaner gem that can be used right out of the box with little screwing around.

Unfortunately, I did something I’m ashamed of.  Ruby gems are notorious for breaking backwards compatibility, and I have done just that.  The good news is, your code will almost work, assuming you didn’t start diving into the Document and Corpus classes too heavily.  If you did, then you will probably experience a lot of breakage.  The result, I hope is a more sensical implementation, however, so maybe you won’t hate me.  Of course, I could be wrong and my implementation is still crap.  If that’s the case, please let me know what needs to be improved.

To install the gem:

gem sources -a http://gems.github.com
sudo gem install ealdent-lda-ruby

Enjoy!

Reblog this post [with Zemanta]

A twitter friend (@communicating) tipped me off to the UEA-Lite Stemmer by Marie-Claire Jenkins and Dan J. Smith.  Stemmers are NLP tools that get rid of inflectional and derivational affixes from words.  In English, that usually means getting rid of the plural -s, progressive -ing, and preterite -ed.  Depending on the type of stemmer, that might also mean getting rid of derivational suffixes like -ful and -ness.  Sometimes it’s useful to be able to reduce words like consolation and console to the same root form: consol.  But sometimes that doesn’t make sense.  If you’re searching for video game consoles, you don’t want to find documents about consolation.  In this case, you need a conservative stemmer.

The UEA-Lite Stemmer is a rule-based, conservative stemmer that handles regular words, proper nouns and acronyms.  It was originally written in Perl, but had been ported to Java.  Since I usually code in Ruby these days, I thought it’d be nice to make it available to the Ruby community, so I ported it over last night.

The code is open source under the Apache 2 License and hosted on github.  So please check out the code and let me know what you think.  Heck, you can even fork the project and make some improvements yourself if you want.

One direction I’d like to be able to go is to turn all of the rules into finite state transducers, which can be composed into a single large deterministic finite state transducer.  That would be a lot more efficient (and even fun!), but Ruby lacks a decent FST implementation.

Reblog this post [with Zemanta]

Learning Scala

Posted: 11 July 2009 in Uncategorized
Tags: , , , , ,
Programming in Scala

Programming in Scala

Two weeks ago, I picked up my copy of Programming in Scala, which had been languishing on my shelf for months.  I pre-purchased it since I went to high school with one of the authors (Lex Spoon).  His mother, incidentally, was also my favorite math teacher.  When I started my new job back in September 2008, I was a total noob at Ruby, so learning that consumed my attention and other languages took a back burner.  Also, I’m always a little reluctant when it comes to learning new languages.  Not because I don’t like to learn them or because it’s difficult — but because it’s a serious investment of time that may be totally wasted.  Sure, Standard ML is an interesting language, but try finding a job doing it.  When I heard that Twitter was using Scala, I figured the time has come to pick up this book.  It also helped that a friend recently started an Atlanta Scala Meetup group.

Aside from being an update on my life, the point of this post is to say that this book is great.  Seldom have I encountered a programming book that achieves this level of depth while still being fun to read.  There are great examples with humor mixed in, the writing is clear and concise, and it’s thorough. What more could you want?

Has anyone else picked up Scala?  (I know there’s a few of you out there lurking!)  Are there any other good books you would recommend?

In the interest of full disclosure, though I know one of the authors, I haven’t actually talked to him in quite a long time (since high school, I think).  I also don’t make any extra money aside from the Amazon affiliate program commission if you happen to buy anything on their site after clicking the book link.

Reblog this post [with Zemanta]