web 2.0 « The Mendicant Bug

Twitrratr

27 October 2008 in computational linguistics, data, opinion mining, sentiment analysis, twitter, web 2.0 | 2 comments

Twitrratr is a new service that attempts to do sentiment analysis on Twitter (follow me while you’re at it). According to their about page, they started off by tracking opinions on Obama but have since expanded to any term. Enter a keyword and it searches twitter for occurrences. It then assigns a sentiment to each post and returns percentages of positive, neutral, and negative tweets for that word. You can also track your own sentiment by searching for @your-username. I come up neutral, but there’s not a lot of data to go on there.

Their method appears to be fairly simple. They have a collection of adjectives with sentiment values (negative, positive) and based on what appears in a given tweet, they can classify a sentence. Of course, this is probably low recall (meaning it misses a lot of tweets that do express sentiment) since sentiment can be expressed without using adjectives. I’m not sure if it tries to do anything with negation, but so far my scans of results look like it ignores it.

So even though it’s pretty ghetto, it’s a nice toy. If they care to extend the algorithm, they have some pretty cool data to work with. I think it would be cool to get some (possibly donated, probably not paid) human effort together to tag some of their data to release as a research dataset.

friendfeed

12 June 2008 in friendfeed, information diaspora, information overload, social networking, web 2.0 | 1 comment

After hearing about it for weeks, I caved and decided to check out friendfeed last night [and again, ht @dpn]. In previous posts I mentioned something I like to call the information diaspora. This is the phenomenon created by posting all sorts of personal information about your likes, dislikes, thoughts, opinions, etc all over the web and your subsequent loss of that information because it can’t be managed. I can see friendfeed coming in handy for removing some of this problem. You can attach a number of different social networking sites, flickr, youtube, etc all to your friendfeed account. Whenever you post something new in one of these sites, that information will be updated on friendfeed for all of your friends (and yourself) to be able to view. It’s not the perfect solution, but it is a very big step in the right direction.

Check it out. As usual, my username there is ealdent and feel free to friend me.

Dapper

3 June 2008 in acm, dapper, information trapping, rss feeds, tech news, web 2.0, website tracking | No comments

My friend Israel clued me in on Dapper a few weeks ago. I have played around with them a very small bit, but that was all it took to recognize their potential. The idea is simple, the implementation not so much. When you browse videos on YouTube, the layout of search results are all the same. So why can’t something recognize this and treat any search result as an rss feed, checking it periodically for changes? Enter Dapper. One thing that has bothered me for the past couple years is the fact that the ACM Technews does not have an RSS feed. WTF, ACM? Thanks to Dapper, now it does.

Unfortunately, Dapper is not perfect. It took me a few tries to get my first dapp working (what they call a single instance of the service). Granted, it was on fairly complicated output (not ACM Technews). If the service you are trying to create a dapp of uses sessions, your attempt will probably fail (and if it doesn’t, let me know how you did it). They are still improving the service, though, so perhaps that will change.

If you are into information trapping, though, Dapper is a must have in your arsenal of traps.

Plurk-or-Tweet

1 June 2008 in downtime, plurk, social networking, twitter, web 2.0 | 2 comments

Is it Hallowe’en already? A fellow nlp blogger (and twitterer) pointed me to Plurk just a few minutes ago. I have been messing with Twitter’s api over the past couple days, which hasn’t been as easy as you’d think since they are suffering from massive growing pains. Fetching the public timeline takes between 5-30 seconds. However, they just got like $15 million in funding, so maybe they’ll be able to address the issue. The even bigger question is can they turn this free advertising service (which is what it is partially becoming) into a revenue stream?

Plurk is basically Twitter with a makeover and some extra social features thrown in. It still has the 140 character status update style interface, but includes a function selection for each plurk (what they call qualifiers): you can say, think, ask, wish, etc. You can also add smileys. Rather than appearing as a series of boxes scrolling down the screen, your plurks appear as floating boxes on a side-scrolling timeline. Plurks of friends also appear on this timeline and the result is a more graphical and pleasing (to me) interface. You can reply directly to other plurks in the boxes and conversations are tracked very nicely. This is far superior to twitter, which requires you to visit the other person’s timeline and wade through their tweets to find previous tweets in a thread. With Twitter being slower than a drunken monkey with three broken legs, that’s even harder.

As my esteemed colleague pointed out, however, scaling is an issue for any service like this. Ultimately, you are bound by how fast you can access the database. If Plurk becomes as popular as Twitter (and I have every reason to believe it won’t), it will also become bogged down. Also, Plurk is just getting started and has no discernible API (unless I’m just missing it). Twitter already has quite a few third party apps.

I must say, though, I am sorely tempted to abandon Twitter in favor of Plurk just for the fact that Plurk is accessible. The massive lag of Twitter is getting to me. Of course, if no one is there to listen to my ramblings, what’s the point?

A Few Random Thoughts

19 March 2008 in arthur c clarke, authors, books, family, leadership, movies, ohio, sci-fi, social networking, spam, twitter, web 2.0 | No comments

Science fiction author Arthur C. Clarke died yesterday. He touched many lives through his writing and his ideas had an impact on me at an early age with short stories like “The Nine Billion Names of God” and movies based on his books like 2010 (which I saw in the theater) and later 2001 (which I saw as a young man). His novel Rendezvous with Rama is being made into a movie and IMDB is quoting 2009 as the release date. I thought it was interesting to find out he had been living in Sri Lanka for some time.

I visited my family in Ohio this past weekend and my uncle made a few interesting points. He’s an old-school spring engineer, meaning he learned coming up through the trade rather than by going to school, and he supervises a number of employees at a relatively small spring company. My grandfather used to own a spring company called, shockingly enough, Adams & Sons Spring Co. That was later bought out and a number of the employees were moved to a different plant, including my dad and uncle. So anyhow, my uncle was telling me a story, which I won’t go into, but the heart of it is that you should not wait for people to hand you “what you deserve.” If you are a leader, regardless of your job title, then lead. If you see someone who needs help, don’t wait for them to ask you. Help. Show that you have the initiative. That’s probably fairly obvious, I mean we’ve all heard it before, but it came at a particularly important time for me.

I’ve been on twitter for a while now, though I don’t update it super-regularly like some people. It’s fun and I hope more of my friends start using it, but I’ve noticed an interesting trend. Just about anything is open to potential spam. Friendster is sick with it. MySpace is abominable. LinkedIn seems fairly immune and I’ve gotten very few spam friend requests from Facebook. Twitter has so far been very good about it, but there is a new trend that I’ve found interesting. You can follow people and people can follow you on twitter. So your status updates are public and potentially seen by thousands of people. How do you increase the number of people who follow you? Follow them, of course! I’m having random people follow me left and right. It only helps me, since I don’t follow them back, but it’s interesting to note.

RateBeer 2.0

22 December 2007 in beer, collaborative filtering, craft beer, netflix prize, ratebeer, recommender systems, social networking, web 2.0 | 5 comments

I am a fan of good beer. In this post I am going to talk about my ideas for how to improve websites that offer ratings for different varieties of beer, and how I think recommender systems would improve their service.

Why I care

Whenever I’m asked what kind of beer I like, I experience a moment of awkwardness. Because I don’t just like good beer, I hate ~~bad~~ mediocre beer. Usually the person asking is a beer noob and I don’t want to sound too snobby by throwing beer jargon at them they probably won’t understand. So I say something along the lines of “I like the more expensive stuff, like from small breweries or imports.” The response is usually something about Sam Adams. I used to enjoy Sam Adams Boston Lager, but I can barely stomach it anymore. There are a couple Sam Adams brews that aren’t half bad, but the Boston Lager no longer cuts it for me.

Read the rest of this entry »

Tweets for Twits

18 December 2007 in facebook, information diaspora, social networking, twitter, web 2.0 | No comments

Well, after many frustrating months of waiting for Twitter to finally fix their gmail contacts import feature, I have finally done it! Surprise, only two contacts were signed up — and that’s two more than I expected. However, one of those is a professor who probably only checked them out because they’re using his technology and the other was a friend who had only one update:

“nothing.”

Social pressure from me caused him to add another update. That’s what I tell myself anyway.

What is Twitter, you ask? It’s basically Facebook status updates made global. Indeed, you can even add a Facebook app that allows Twitter to update your status. Of course, it means you get “is twittering: ” inserted at the beginning of any tweet (a single Twitter status update) as your status update.

While Twitter at first seems like status updates on steroids, it’s actually evolving into something else far more useful. I’ve talked before about the information diaspora and the difficulty of keeping up with all your personal information as it flies around the web. Twitter at first adds to that mess, but it does offer interesting ways of tracking small bits of information.

Erin McKean, the Dictionary Evangelist, uses it to keep track of new words she comes across. Twitter lets you text updates from your cell phone or IM client so it’s easy to update on the go. Robert Scoble uses it as a sort of mini-blog of things he comes across or finds out about that wouldn’t really make a full-fledged blog post. So Twitter has uses for logging your web surfing, hobby, life activities, etc., which is a useful information diaspora reducing measure in my book. The only question remains whether this would be of any use to you.

Check me out and follow my updates on Twitter. If you haven’t signed up, consider it. If you do, let me know so I can follow you.

Predictify!

11 December 2007 in crowd wisdom, futures markets, money, predictify, predictions, startups, web 2.0 | 2 comments

I wrote about Predictify a while back. It’s basically a website that pays users for predicting world events. When I first wrote about them, I presented the sample question: “How long will Michael Vick’s sentence be?” Well, the verdict came down and my prediction was very close. I predicted 24 months and the dirty bastard got 23. Total payout for me: $6.07. Not bad for 30 seconds effort.

The site appears to be doing well. Currently, there are 24 open polls with large cash payout potential. I was pretty skeptical it would succeed (and that still has yet to play out fully), but it would seem that the guys who predicted its success are going to be looking at nice payouts of their own.

Information Diaspora

9 December 2007 in facebook, information, information diaspora, information overload, mashable, neologisms, social networking, visual bookshelf, web 2.0 | 3 comments

I’m going to officially coin the term information diaspora to mean the dispersion of individual personal preference information throughout the web. Whenever you sign up for an account, you leave a part of your personal information somewhere. Whenever you enter an address to order a book, more information. When you look through digg comments and you thumbs-up or thumbs-down a comment, more information. Whenever you favorite a video on youtube, leave a wall post on facebook, rate a movie on netflix, more information. All of this information is accessible to you as long as you can recall where you have left it. If you forget about a website you signed up for, that information is now missing. It’s not dead or gone, just missing.

Your brain is no longer the homeland of all these orphaned data. Social networking is great, but with the current Web 2.0 bubble expanding the way it is, the inherent incompatibility in the global network is becoming more and more a problem.

Read the rest of this entry »

Predict this!

8 October 2007 in crowd wisdom, futures markets, futurism, predictify, predictions, startup, startups, web 2.0 | 1 comment

Predict the following:

How long will Michael Vick’s prison sentence be?
How many iPhones will Apple sell by year’s end?
How much will Hillary Clinton raise in funds during the fourth quarter of 2007?

How accurate is the crowd? Futures markets have done a good job at predicting how the market will go for certain commodities. This fact inspired DARPA (the Defense Advanced Research Projects Agency) to create a project that sought to build a futures market around terrorism. People would be able to bid on where and when terrorist events would occur (or be attempted). The public backlash from this idea was enormous. Senator Barbara Boxer raised quite the furor and demanded that the then Information Awareness Office director Admiral John Poindexter be fired. He did lose his job over the affair, but like most Washington insiders, soon found a job on a board of directors.

I actually don’t think the Policy Analysis Market (as it was called) was such a bad idea. It seemed that the three main critcisms against it were

it would encourage bidders to actually encourage terrorist acts in order to earn payoffs
it’s grotesque and immoral
pure vitriol

As for point (1), I can kinda see where they’re coming from. That would definitely be a bad side effect if it were to occur. However, I think the fact that the government knows exactly who is bidding on what, it’s also very dangerous for that person. Also, the thing had $8 million in funding. Not exactly winning the lottery considering that that money had to be spread around creating the system and as a payoff for each particular security to be traded. It was basically an advanced tip service that takes advantage of collective wisdom. Point (2) I’m also not so sure about, since we’re only talking speculation here. Is it immoral for me to predict that another soldier will die in Iraq tomorrow? I’d argue instead that the market, if it did anything at all, would save lives. And as for (3), the level of outrage coming from Senators Boxer and Wyden at the time (this all happened back in 2003) was disgusting. You’d think that DARPA had suggested we start using our babies as a food source.

I could see a worst case scenario cropping up if such a market existed, though. What would happen if this market suddenly predicted that Iran would launch a nuclear attack on American troops in Iraq (or a conventional attack for that matter)? If the market had experienced success before and gained some credibility, this could be considered a justification for war. That would be a very bad thing indeed.

So anyhow, after meandering way off point on a 4-year-old digression, let me come back around to my main point: Predictify. This is a new website that seeks to capitalize on crowd wisdom by predicting events ranging from the mundane to the presidential election. Users can predict outcomes for free and premium users can post polls that collect extra demographics from the users. Certain polls come with prize pots ranging in the hundreds of dollars.

After playing around a bit and making a few predictions, I’ve found it mildly interesting. There are slight problems with the interface. For example, if you’re browsing political predictions and submit a prediction, you’re presented with a link to go back to predicting. This link takes you back to the main page rather than the subtopic you were just at. I found this to be annoying since I had to go back to browsing the topic. Also, the top polls on the main page are starting to get slightly spammy. Predict the outcome of some motorcross race in Las Vegas this weekend for example. Predict the yearly earnings of some company no one has ever heard of. Predict how many speed dates SpeedDates.com will have by the end of the year.

I think Predictify will have to do something to prevent this sort of detritis from clogging up the site. The interesting polls that will bring in the visitors are getting lost in the clutter of these junk pedalers.

Twitter away

6 September 2007 in downtime, facebook, social networking, status updates, twitter, web 2.0 | 2 comments

So being generally interested in the whole social networking thing lately, I decided to try out Twitter a couple weeks ago. Twitter does 24/7 what Facebook status updates were meant to: allow you to keep your friends updated on the minutia of your life. Facebook status updates have degenerated to being a sort of contest of cleverness (perhaps to make interesting the boring crap of your life). There are a few classes of Facebook status updates (FBSU) I have categorized:

Life status — indicating tiredness, boredom, anxiety, desires, location, etc.
Temporary fandom — showing in some way that you’re a fan of a group, tv show, movie, politician, sports team, product, etc.
Pissed-offedness — invective-splattered status updates, often employing various symbols (%!$#@)
Pseudo-philosophical gibberish — for those who want their friends to think them deep

Twitter does all of that but by its nature also allows the minutia reports to be more natural. On Facebook, no one uses status updates to tell people they are at home or at school or at work. I tried, believe me. No one cared, oddly enough. So I stopped and have resorted mostly to #1, #2 and #4 on the list above.

So anyhow, for the past two weeks, I have tried to use my gmail contacts to find friends on Twitter, which are probably few. Over twenty failures. And now today, Twitter is down and has been down for a while. Supposedly it is being upgraded, and I’m hoping that will mean the problem is fixed. I emailed them yesterday about it after many, many failed importing attempts. Currently, I’m enjoying the thought that Twitter’s downtime is their way of saying, “this update’s for you.”

Update

As of 5pm EDT, Twitter is back up. But, I still can’t import my gmail contacts to find friends.

dot blerg

21 August 2007 in business, crowdsourcing, dotcom, entrepreneurship, myprogress, social networking, startup, startups, web 2.0 | No comments

I must be in some sort of funk today, because I’m seeing doom everywhere I look. Right now, I see the next dot com bust hiding just beyond the crest of the Web 2.0/social networking wave. VCs are dumping money into startups left and right and piece of crap companies like Club Penguin are being snatched up by major corporations for insane gobs of money. Or maybe what’s depressing me is the fact that once again I am seeing dozens of sites spring up that offer very little new except an interesting idea, but the product just isn’t there.

Read the rest of this entry »