Notes from From Understanding to Enabling Networks: Using Web Science to Enhance Recommender Systems

•October 24, 2011 • Leave a Comment
Noshir Contractor

Noshir Contractor

The keynote at #recsys2011 is by Noshir Contractor. He is the coauthor of “Theories of communication networks” which seems to be an interesting book from amazon reviews.

the presentation stack is available here (thanks to @barrysmyth for the link)

He started by presenting SNIF. SNIF is a device and social networks for dogs! Kind of social petworking. In contrast lovegety is the SNIF technology for people. Find love through random encounters.

Today we will talk about How we can take research in social sciences and bring it to recommender systems.

People have looked at citations and papers and found that people who write papers in teams have a high impact. Also articles by teams from different disciplines from different geographic locations have the highest impact. Fining the appropriate team from a diverse background and geography is much harder.

Thus we are looking at assembeling these type of teams. But how do we decide whom to bring to the team?

The exciting thing about our time is that we have theories, data and methods, additionally we have computation infrastructure to run these models

Why do people collaborate with each other?

MTML model:

  • self interest (from econ theories)
  • Social and resource exchange
  • Mutual interest and collective action
  • Theories of contagion
  • Theories of balance
  • Theories of homophily
  • Theories of proximity
My note: How about Robert Spolsky’s theory?
Exponential randome graphs can explain how these collaboration networks is formed (the shape of the graph)
They have looked at the structure of NSF proposals and they wanted to see if they can build a recommnder system that by using characteristics of the proposal make recommendations for acceptance
The likelihood of collaboration is highers if:
  • you have written an NSF proposal together
  • you have cited each other
Didn’t know about H-index. Interesting factor. Apparently those with higher H-index are less likely to collaborate.
Citing your collaborators actually reduces the likelihood of getting NSF funding (!)
Solving the link recommendation problem (recommending who should be on the team)
Link prediction approaches: node-wise similarity, network topology, or probabilistic modeling
P* for link prediction
Use p* models to calculate link probability
  • Estimate p*/ERGM
  • the rest I didn’t get to type (!)
I think the probabilistic model that he is refering to is the same as model fitting on Bayesian nets but I am not sure.
The talk ended by a demo of the implementation that is available here 
Noshir’s book is also available for free on his personal website


Notes from “Recommendations as a Conversation with the User” by Daniel Tunkelang

•October 24, 2011 • Leave a Comment
Daniel Tunkelang

Daniel Tunkelang

These are my unedited notes from Daniel Tunkelang’s presentation at #recsys2011. I am editing as you are reading this post.

“Recommendations as a Conversation with the User” by Daniel Tunkelang
Goal is to have a better relationship with the user

Three take aways from this talk:

  • Consider asking vs guessing
  • Ask good questions
  • It’s okay to make mistakes if you have a good explanation and adapt to feedback


The Man Who Lied to His Laptop”  is a great related read
Paul Grice’s maxims of conversations:

  1. Quality
  2. Quantity
  3. Relation
  4. Manner

**Do not lie

  • Don’t use “recommended” when you really mean “sponsored” or “excess inventory”. User’s loss of trust will cost you. but users do not have a model on how on how to trust a system
  • Optimize for the user’s utility
  • Apply a standard of evidence (quality, quantity) that you believe in

Right amount of information

  • Exchange small units of information
  • If recommendations supplement other content consider overall cognitive load
  • provide short meaningful explanations

Maxim 3: Relation. Relevant to the user

  • Offer value to the user
  • respect task context
  • don’t be obnoxious

Maxim 4: Manner

  • relevant to the user
  • Eschew obfuscation
  • Avoid ambiguity
  • be brief
  • be orderly

Another perspective

Another perspective is Gary Marchionini’s perspective on Human computer information retrieval

Empower people to explore large-scale information but demand that people also take responsibly for the control be expending cognitive and physical energy

principles of hcir

  1. do more than deliver information: facilitate sense-making
  2. require and reward effort
  3. adapt to increasingly knowledgeable users over time
  4. be engaging and fun to use

Adapt to user knowledge
Systems that don’t get better over time will frustrate users, because users DO get better over time

Personalized recommendations

  • be transparent about model so users gain insight
  • allow users to modify models to correct
  • solicit just enough information to provide value
  • Exemplars are interesting tools to communicate the recommender model to the user
  • Users should be able to modify the recommender system say you have a recommender system that uses location and user is using a proxy. He should be able to turn if off to make it noncreepy!

Social recommnedations

  • identify the right set of similar users
  • allow users to manipulate the social lens
  • accommodate users who break your model

When making item recs, explain your recommendations! Watch for non-sequiturs (diapers -> beer problem)

**Tell me about yourself is friendlier than “fill out 20 pages of survey”

Corpse bride is in the recommnded set and I have watched it, it is good. it gives me the feeling that recommender is working properly

Learning from netflix

  • Ask users for help upfront but not too much help
  • pay attention to what the user tells you
  • give users value often and early

75% of netflix views result from recommendation

Underpromissing and overdelivering is sometimes a good idea

Soe models more explainabel than others

  1. consider decision trees and rule based models
  2. avoid using latent, unlabled features
  3. if the model is opaque use exaples as surrougates

Make a good first impression
your user’s first experience is critical

See “Machine learning for large scale recommender systems” by Agrawal and Chen ICL 2011 Tutorial

“We Will All Be Jedi Masters Soon” or “Random But Coherent Thoughts on Modern Education”

•October 12, 2011 • 2 Comments

Three things have happened recently:

1- We are living in an exciting era. Stanford is offering their AI course online for free and I am telling you, it is not the crappy study material that MIT dumps on their open course ware website. These are serious, well curated videos with quizzes and assignments.  Basically what it means is that a brown kid in the desserts of south Oman can now learn what a wealthy full-of-himself Stanford kid in Palo Alto learns about AI .

2- For my PhD I worked on the intersection of human-computer interaction (HCI) and machine learning (ML). It took me a couple of years and I can assure you, you can find online and free educational material to become as bad-ass as I am in both of the fields. Ironically the number of educational videos on machine learning outnumbers the number of videos on CHI!

3- Steve Jobs died a couple of days ago. As an open source contributor I hated him while he was around. But I felt extremely sad when passed away. Let’s all face it, he might have been an ass when it comes to treating others or exploiting and abusing child labor. But he lead many amazing projects. Apple was a mecca for HCI people. They really set the standard for innovation in consumer products. I have recently found this video by Steve Jobs in which he emphasizes tool building and how computers are making us super humans. And it is very true. The guy is a visionary.  In his words we can quote him, disagree with him, glorify or vilify him, but the only thing we can’t do is to ignore him because he changed our lives forever.

4- In terms of computing we are living in an amazing time too. I work in a lab that has a pretty strong cluster of computers and 90% of the time the load on the cluster is not that much. Do you see it? It means we are entering an era that our computing power is way more than our computing needs! And if we utilize our computing power well we may actually have excess cycles. Things like Hadoop have allowed us to treat multi-computers like single computers and run massive jobs on them. Do things that were impossible before.

I guess what I am saying is that:

The other blog post is claiming that we need to be awesome otherwise we will loose our jobs. What I am saying is that the cost of becoming awesome is decreasing dramatically. With all these free courses, education is becoming cheap (while schooling becomes more and more expensive), our  tools are getting better and I also believe that seeing our friends on facebook/twitter/google+ has given us an incentive for self improvement and encourages us to learn and educate ourselves more. It has become much easier to push ourselves to become a Jedi master.

Methods for testing data-heavy applications – Part I

•July 14, 2011 • Leave a Comment
Software Testing

Software Testing

For the past two months, I have been building a piece of code for a very interesting company that is not on the Fortune 500 list yet but if it goes public it may very well be a Fortune 1 company! This application is very heavy on data. I am still developing, I feed it chunks of 100 megabyte datasets, it goes through labor for a couple of minutes and crunches the numbers, and eventually comes back with a table with about 100 rows that I could take and do some statistical analysis on and plot. That big chunk of code gets reduced to bite size human-readable numbers and graphs. We are still testing it and I honestly have no idea how it will perform once it gets connected to the data firehose of the company! and that drives me insane!

I have been testing this software. It works fine. It works fine on all of my test datasets and I have been scrolling through its algorithms that span over a thousand line of code to make sure they work as I expect. I am going insane over this. What if we ship it next week and there is a bug in it? It is not like a Google+ application that you write, ship and then notice that instead of the photo of your female friend it brings back the photo of your dad! there is no way to see the bug when the product is out. your users cannot report the bugs because they won’t see any of it.

I have been looking into the scientific literature and there is tons of methods in the field of software development with tons of different names but none is about the type of software that I am working on. So I asked on twitter how other people are testing their software and here is a number of responses that I have received.

1- Take the current data files and manually perform your algorithm on the data set (using something like Excel) and see if the final results match

2- Fabricate test data sets for which you know the final result. Run the code on those test files. For example say you have written a code for summation. You can just feed it an array of ones and see if it works on them. Test for boundary cases, test for data that produce intuitive outputs, if you know your algorithm should have a specific behavior, test for that too (like when you expect symmetry in your output)

3 – Hopefully you break your code into atomic blocks that only do one thing (inverse a matrix, svd calculations, etc) write unit tests for those functions individually. Then combine those individual functions into larger functions and components and test them too and so on.

I am more interested to know your thoughts. what are your suggestions for testing big data applications?





Mechanical Turk Market, Ethics and Milton Friedman

•April 7, 2011 • 6 Comments

Disclaimer: unlike what it may seem like, I am not a conservative. I am just another liberal researcher from Berkeley who values and celebrates weed and women

As an experimental economist I found the mechanical turk marketplace a very interesting example of macro economy. Yes, I know it is a zoo. But at the same time people get interesting results out of this market. Apparently Zappos is using it to correct the grammar and wording of the reviews on their website and Professor Franklin’s team has been building a database engine that can use the crowd to fill the missing values in their database. In some sense you can really get Shakespeare’s work out of this crowd of people who just tap on their keyboards. No complain on my part though, many turkers are honest workers and use it to get a legitimate salary but many (including myself) are just there for fun and they won’t blink if they can screw over your work.

Those researcher who piss me off the most are the ones who claim that they have found a groundbreaking theory on mechanical turk, I keep seeing people who claim they have observed a “new behavior” in the crowd while what they are really seeing is just another validation for old economics theories.

One of the issues that keeps coming up is “Is it ethical that there is no minimum wage for mechanical turk?” This is the question that was answered 60 years ago not on this stupid mechanical turk market but on large labour markets by our old friend, professor Milton Friedman (he has some other crazy ideas including geting rid of IRS and FDA but those are a little crazier than what I can really explain here 🙂 )

In essence, Friedman claims that when you set a minimum wage for people’s work, you are just causing part of your society to be out of work. It affects minorities, women and teenagers dramatically, increases unemployment and in these years it highly encourages businesses to hire people offshore. Watch Friedman’s video here. Or Google his books on Google books.

So an answer to those of my friends who question the ethics of having these labor markets like Mechanical Turk, here is my answer:

Not only it is ethical not to have a minimum wage in mechanical turk but it is also unethical to have a minimum wage in the US. In some scene mechanical turk was formed due to some nonsense minimum wage regulations in the US labor market.  If business owners could hire people in the US to clean up their messy databases for $3.00 they wouldn’t need to go on mechanical turk to hire the same person for $1.50 an hour.

I am interested in knowing what you think especially counter arguments are highly appreciated.

Everybody is spamming everybody else on Mechanical Turk

•March 9, 2011 • 5 Comments

Two months ago , an NYU professor who studies anthropological aspects of online slavery as well as doing nerdy computer science stuff, pointed out in his blog post that 41 percent of mechanical turk HITs are posted by spammers. These requesters are typically those who ask turkers to click on a link, friend somebody on facebook or trash somebody on facebook. These tasks may not look like spam to us as I personally friend and/or trash people on facebook on regular basis so no big deal, but I never get paid for it:)

Anyways as a curious researcher who  doesn’t want to accept anything until he has done his due diligence I posted a spam HIT with a generous 1 dollar reward asking people to write a review about interesting places to visit in the  “Shiraz City, France”. Shiraz is known for its ridiculously hot girls and great wine but the problem is that it is not city in France, surprisingly it is in Iran.

I exclusively asked for those turkers who have lived in “Shiraz City, France” and made sure to highlight that if the turker has never lived in that city the work is going to be rejected. (For those who would like to bring up ethical issues for  this experiment I would like to point out that turkers have screwed me over and over on countless number of occasions so I reserve the right for myself to mess with them occasionally, additionally you can look at this as the Milgram experiment of the crowdsourcing era. Who doesn’t want to be called Philip Zimbardo of the online slavery market? Now that I am thinking about it I actually kind of like it.

Antoine Dodson

The title of this post was inspired by a poem by Antoine Dodson, a frustrated community observer who once shouted out of frustration "Hide yo' keeds, hide yo' wife, and hide yo' husband cause they rapin'everybody out here" He was the first one to voice his frustration about the kind of society that we all have become, a community in which people don't care about each other, the society that leaves no choice for people but hiding!

Anyways if the market was fair I assume two things would have never happend

1) I would have never been allowed to submit such an obscene work

2) I would have never received any responses

To my surprise both happened.

Nobody really polices the mturk market so I was able to get my HIT through. but the more interesting thing was that I was actually receiving suggestions for visiting a city that didn’t even exist!

I receive three types of responses:

  • 1- Random excerpts from our old friend “the internet” about cities in france (mostly Paris)
  • 2- Email addresses of the turkers
  • 3- Random user names

We all know that Mechanical Turk challenges the whole “Junk-in, Junk-out” dilemma and makes it more like “Always junk-out, regardless of the input process” but I feel that users posted junk for my junk HIT assuming that there is no quality assurance process working behind the task and I would probably just accept them all (which I have done for many jobs before)

Anyways I just wanted to highlight that the spamming goes both ways. Turkers spam requesters, requesters spam turkers, everybody wins!

(On a side note, I really need some behind the scene data from Amazon, how can I convince amazon to give me some data for research? In return I am willing to give them a high quality travel booklet that I have curated about a city in France, the city is called “Shiraz”    : )

Distribution of Mechanical Turkers in the US

•February 16, 2011 • 1 Comment

A couple of weeks ago our friends at Techlist published one of their visualizations of the distribution mechanical turkers of the US.  They had chosen very large icons for each turker and it was hard to understand the general distribution. I took their XML file and read it in R and generated a csv file in R. The following visualization is produced in Tableau and might be more clear than their Google map visualization (h/t to techlist of their dataset)

Distribution of US Turkers

Distribution of US Turkers (16,000 Turkers)