Neel Sundaresan

He started by stating that he won’t have any greek symbols in the talk.

Arch West was the inventor of Doritos and David Pace we the inventor of Pace sauce. What they did was that they noticed the can sell more if they advertise the two products together. There is a lesson in cross-selling and recommender systems that we can learn from this story.

eBay started when Pierre Omidyar wanted to sell his broken laser pointer. He listed the laser pointer online for 99 cents and finally sold for 14 dollar. Was wondering if the person who bought it knew that it is broken. The guy responded “Yes I am a collector of broken laser pointers”

Why people buy something? it is hard to say. Some people buy stories! remember the toast that sold for 27K that was the cheapest marketing campaign that a casino had.  One man’s trash is another’s treasure.

The long tail in eBay’s context mean most people sell very few items and most of eBay’s revenue comes from these people. (i.e the mean is way larger than the median

The users constantly are running experiments to maximize their revenue. They are constantly testing to see if free shipping can sell more, different selling strategies are being tested by users at any time on eBay.

This causes an interesting behavior. If you promote a user’s product on the homepage they may increase their price! This is an interesting dynamics between the user and the seller (eBay)

One of the problems that locations like eBay have is the problem of big data. Complex algorithms are often impossible to work with in that scale. If you are looking for a job at eBay you need to know how to work with data in that scale.  A goal at eBay lab is that when a new scientist joins the lab on Monday he got access to all the data by Friday.

This amount of data has changed how economics is doing experiments. They can now run experiments on 400 million data points.

What are you optimizing for at eBay? is it profit maximization? do you want to increase the shopping cart size? are you looking for maximum customer satisfaction?

The other thing is how do you measure success?

Everything we do at eBay is a recommendation.

I KEEP six honest serving-men
(They taught me all I knew);
Their names are What and Why and When
And How and Where and Who.

– from The Elephant’s Child 

When we look at the tag cloud of eBay you see keywords like “used”, “vintage” and “antique” a lot more than “new”

The search is an interesting problem some people are looking for “ipod nano 4gb black new” and some are looking for the skin for their ipod. Our search engine should be able to differentiate between ”ipod nano 4gb black new” ”ipod nano 4gb black new skin”. This proposes a hard and challenging research questions.

Click trails can help us tremendously with building recommender systems that can capture these behavior and improve recommender systems.  At eBay a data cleanup is an important part of recommender. Specially when they use click trails.

eBay has a language like pig that allows them to do pattern recognition at scale. Sometimes a search is followed by some page views and another search. This pattern is useful to do recommendation to other users who have similar initial search queries. See two recent papers from Sundaresan for the results and model.

Fashion item buyer on eBay are very brand aware. Sometimes ebay does not have enough inventories and needs to recommend proper products from outside websites.

one of the challenges at eBay is that we do not have a catalogue of items (remember the laser pointer story?) Amazon does not have such a problem, you cannot sell anything on Amazon unless it is on the catalog.

eBay uses its own matrix factorization see their ICML paper. The sparsity in eBay’s data is fascinating it is 100 times the sparsity in the Netflix data.

eBay clusters items into pseudo products using LDA. He shows an example of a recommendation for a broken blackberry cellphone.

The most important thing is “why” are you recommending this to the user and “why” they should buy it. HCI is a useful tool here. reveal to the user why you are recommending. Something like “52% of the people who bought this item also bought …” are very effective.  Be very explicit on why certain recommendations are made.

Let’s look at “When”. things like reminders, post purchases, urgency, upgrades, seasonal sales fall into this. Reminders can be like “you have viewed this item” that reminds people that they can still go and buy. There is a temporal element to this problem too. They may not need the same item until after 30 days but need to buy it again after 30 days is passed.

See this wired article on persuasion based profiling and recommendation systems. (thanks to twitter).

There is a lot of seasonality on eBay. Mother and Father’s days, Christmas. There are other events that are we don’t know (so my question is how can we find them algorithmically)

We get more data from mobile devices than we get from online. It is a huge research opportunity.

Noshir Contractor

The keynote at #recsys2011 is by Noshir Contractor. He is the coauthor of “Theories of communication networks” which seems to be an interesting book from amazon reviews.

the presentation stack is available here (thanks to @barrysmyth for the link)

He started by presenting SNIF. SNIF is a device and social networks for dogs! Kind of social petworking. In contrast lovegety is the SNIF technology for people. Find love through random encounters.

Today we will talk about How we can take research in social sciences and bring it to recommender systems.

People have looked at citations and papers and found that people who write papers in teams have a high impact. Also articles by teams from different disciplines from different geographic locations have the highest impact. Fining the appropriate team from a diverse background and geography is much harder.

Thus we are looking at assembeling these type of teams. But how do we decide whom to bring to the team?

The exciting thing about our time is that we have theories, data and methods, additionally we have computation infrastructure to run these models

Why do people collaborate with each other?

MTML model:

• self interest (from econ theories)
• Social and resource exchange
• Mutual interest and collective action
• Theories of contagion
• Theories of balance
• Theories of homophily
• Theories of proximity

Daniel Tunkelang

These are my unedited notes from Daniel Tunkelang’s presentation at #recsys2011. I am editing as you are reading this post.

“Recommendations as a Conversation with the User” by Daniel Tunkelang
Goal is to have a better relationship with the user

Three take aways from this talk:

• Consider asking vs guessing
• Ask good questions
• It’s okay to make mistakes if you have a good explanation and adapt to feedback

Theory

“The Man Who Lied to His Laptop”  is a great related read
Paul Grice’s maxims of conversations:

1. Quality
2. Quantity
3. Relation
4. Manner

**Do not lie

• Don’t use “recommended” when you really mean “sponsored” or “excess inventory”. User’s loss of trust will cost you. but users do not have a model on how on how to trust a system
• Optimize for the user’s utility
• Apply a standard of evidence (quality, quantity) that you believe in

Right amount of information

• Exchange small units of information
• If recommendations supplement other content consider overall cognitive load
• provide short meaningful explanations

Maxim 3: Relation. Relevant to the user

• Offer value to the user
• respect task context
• don’t be obnoxious

Maxim 4: Manner

• relevant to the user
• Eschew obfuscation
• Avoid ambiguity
• be brief
• be orderly

Another perspective

Another perspective is Gary Marchionini’s perspective on Human computer information retrieval

Empower people to explore large-scale information but demand that people also take responsibly for the control be expending cognitive and physical energy

principles of hcir

1. do more than deliver information: facilitate sense-making
2. require and reward effort
3. adapt to increasingly knowledgeable users over time
4. be engaging and fun to use

Adapt to user knowledge
Systems that don’t get better over time will frustrate users, because users DO get better over time

Personalized recommendations

• be transparent about model so users gain insight
• allow users to modify models to correct
• solicit just enough information to provide value
• Exemplars are interesting tools to communicate the recommender model to the user
• Users should be able to modify the recommender system say you have a recommender system that uses location and user is using a proxy. He should be able to turn if off to make it noncreepy!

Social recommnedations

• identify the right set of similar users
• allow users to manipulate the social lens
• accommodate users who break your model

When making item recs, explain your recommendations! Watch for non-sequiturs (diapers -> beer problem)

**Tell me about yourself is friendlier than “fill out 20 pages of survey”

Corpse bride is in the recommnded set and I have watched it, it is good. it gives me the feeling that recommender is working properly

Learning from netflix

• Ask users for help upfront but not too much help
• pay attention to what the user tells you
• give users value often and early

75% of netflix views result from recommendation

Underpromissing and overdelivering is sometimes a good idea

Soe models more explainabel than others

1. consider decision trees and rule based models
2. avoid using latent, unlabled features
3. if the model is opaque use exaples as surrougates

Make a good first impression
your user’s first experience is critical

See “Machine learning for large scale recommender systems” by Agrawal and Chen ICL 2011 Tutorial

Three things have happened recently:

• I am going on the job market and looking for a research job
• Steve Jobs passed away
• I read a blog post entitled “Unless you are awesome, you will be outsourced“

1- We are living in an exciting era. Stanford is offering their AI course online for free and I am telling you, it is not the crappy study material that MIT dumps on their open course ware website. These are serious, well curated videos with quizzes and assignments.  Basically what it means is that a brown kid in the desserts of south Oman can now learn what a wealthy full-of-himself Stanford kid in Palo Alto learns about AI .

2- For my PhD I worked on the intersection of human-computer interaction (HCI) and machine learning (ML). It took me a couple of years and I can assure you, you can find online and free educational material to become as bad-ass as I am in both of the fields. Ironically the number of educational videos on machine learning outnumbers the number of videos on CHI!

3- Steve Jobs died a couple of days ago. As an open source contributor I hated him while he was around. But I felt extremely sad when passed away. Let’s all face it, he might have been an ass when it comes to treating others or exploiting and abusing child labor. But he lead many amazing projects. Apple was a mecca for HCI people. They really set the standard for innovation in consumer products. I have recently found this video by Steve Jobs in which he emphasizes tool building and how computers are making us super humans. And it is very true. The guy is a visionary.  In his words we can quote him, disagree with him, glorify or vilify him, but the only thing we can’t do is to ignore him because he changed our lives forever.

4- In terms of computing we are living in an amazing time too. I work in a lab that has a pretty strong cluster of computers and 90% of the time the load on the cluster is not that much. Do you see it? It means we are entering an era that our computing power is way more than our computing needs! And if we utilize our computing power well we may actually have excess cycles. Things like Hadoop have allowed us to treat multi-computers like single computers and run massive jobs on them. Do things that were impossible before.

I guess what I am saying is that:

The other blog post is claiming that we need to be awesome otherwise we will loose our jobs. What I am saying is that the cost of becoming awesome is decreasing dramatically. With all these free courses, education is becoming cheap (while schooling becomes more and more expensive), our  tools are getting better and I also believe that seeing our friends on facebook/twitter/google+ has given us an incentive for self improvement and encourages us to learn and educate ourselves more. It has become much easier to push ourselves to become a Jedi master.

Software Testing

For the past two months, I have been building a piece of code for a very interesting company that is not on the Fortune 500 list yet but if it goes public it may very well be a Fortune 1 company! This application is very heavy on data. I am still developing, I feed it chunks of 100 megabyte datasets, it goes through labor for a couple of minutes and crunches the numbers, and eventually comes back with a table with about 100 rows that I could take and do some statistical analysis on and plot. That big chunk of code gets reduced to bite size human-readable numbers and graphs. We are still testing it and I honestly have no idea how it will perform once it gets connected to the data firehose of the company! and that drives me insane!

I have been testing this software. It works fine. It works fine on all of my test datasets and I have been scrolling through its algorithms that span over a thousand line of code to make sure they work as I expect. I am going insane over this. What if we ship it next week and there is a bug in it? It is not like a Google+ application that you write, ship and then notice that instead of the photo of your female friend it brings back the photo of your dad! there is no way to see the bug when the product is out. your users cannot report the bugs because they won’t see any of it.

I have been looking into the scientific literature and there is tons of methods in the field of software development with tons of different names but none is about the type of software that I am working on. So I asked on twitter how other people are testing their software and here is a number of responses that I have received.

1- Take the current data files and manually perform your algorithm on the data set (using something like Excel) and see if the final results match

2- Fabricate test data sets for which you know the final result. Run the code on those test files. For example say you have written a code for summation. You can just feed it an array of ones and see if it works on them. Test for boundary cases, test for data that produce intuitive outputs, if you know your algorithm should have a specific behavior, test for that too (like when you expect symmetry in your output)

3 – Hopefully you break your code into atomic blocks that only do one thing (inverse a matrix, svd calculations, etc) write unit tests for those functions individually. Then combine those individual functions into larger functions and components and test them too and so on.

I am more interested to know your thoughts. what are your suggestions for testing big data applications?

Disclaimer: unlike what it may seem like, I am not a conservative. I am just another liberal researcher from Berkeley who values and celebrates weed and women

As an experimental economist I found the mechanical turk marketplace a very interesting example of macro economy. Yes, I know it is a zoo. But at the same time people get interesting results out of this market. Apparently Zappos is using it to correct the grammar and wording of the reviews on their website and Professor Franklin’s team has been building a database engine that can use the crowd to fill the missing values in their database. In some sense you can really get Shakespeare’s work out of this crowd of people who just tap on their keyboards. No complain on my part though, many turkers are honest workers and use it to get a legitimate salary but many (including myself) are just there for fun and they won’t blink if they can screw over your work.

Those researcher who piss me off the most are the ones who claim that they have found a groundbreaking theory on mechanical turk, I keep seeing people who claim they have observed a “new behavior” in the crowd while what they are really seeing is just another validation for old economics theories.

One of the issues that keeps coming up is “Is it ethical that there is no minimum wage for mechanical turk?” This is the question that was answered 60 years ago not on this stupid mechanical turk market but on large labour markets by our old friend, professor Milton Friedman (he has some other crazy ideas including geting rid of IRS and FDA but those are a little crazier than what I can really explain here )

In essence, Friedman claims that when you set a minimum wage for people’s work, you are just causing part of your society to be out of work. It affects minorities, women and teenagers dramatically, increases unemployment and in these years it highly encourages businesses to hire people offshore. Watch Friedman’s video here. Or Google his books on Google books.

So an answer to those of my friends who question the ethics of having these labor markets like Mechanical Turk, here is my answer:

Not only it is ethical not to have a minimum wage in mechanical turk but it is also unethical to have a minimum wage in the US. In some scene mechanical turk was formed due to some nonsense minimum wage regulations in the US labor market.  If business owners could hire people in the US to clean up their messy databases for $3.00 they wouldn’t need to go on mechanical turk to hire the same person for $1.50 an hour.

I am interested in knowing what you think especially counter arguments are highly appreciated.

Anyways as a curious researcher who  doesn’t want to accept anything until he has done his due diligence I posted a spam HIT with a generous 1 dollar reward asking people to write a review about interesting places to visit in the  ”Shiraz City, France”. Shiraz is known for its ridiculously hot girls and great wine but the problem is that it is not city in France, surprisingly it is in Iran.

I exclusively asked for those turkers who have lived in “Shiraz City, France” and made sure to highlight that if the turker has never lived in that city the work is going to be rejected. (For those who would like to bring up ethical issues for  this experiment I would like to point out that turkers have screwed me over and over on countless number of occasions so I reserve the right for myself to mess with them occasionally, additionally you can look at this as the Milgram experiment of the crowdsourcing era. Who doesn’t want to be called Philip Zimbardo of the online slavery market? Now that I am thinking about it I actually kind of like it.

The title of this post was inspired by a poem by Antoine Dodson, a frustrated community observer who once shouted out of frustration "Hide yo' keeds, hide yo' wife, and hide yo' husband cause they rapin'everybody out here" He was the first one to voice his frustration about the kind of society that we all have become, a community in which people don't care about each other, the society that leaves no choice for people but hiding!

Anyways if the market was fair I assume two things would have never happend

1) I would have never been allowed to submit such an obscene work

2) I would have never received any responses

To my surprise both happened.

Nobody really polices the mturk market so I was able to get my HIT through. but the more interesting thing was that I was actually receiving suggestions for visiting a city that didn’t even exist!

I receive three types of responses:

• 1- Random excerpts from our old friend “the internet” about cities in france (mostly Paris)
• 2- Email addresses of the turkers
• 3- Random user names

We all know that Mechanical Turk challenges the whole “Junk-in, Junk-out” dilemma and makes it more like “Always junk-out, regardless of the input process” but I feel that users posted junk for my junk HIT assuming that there is no quality assurance process working behind the task and I would probably just accept them all (which I have done for many jobs before)

Anyways I just wanted to highlight that the spamming goes both ways. Turkers spam requesters, requesters spam turkers, everybody wins!

(On a side note, I really need some behind the scene data from Amazon, how can I convince amazon to give me some data for research? In return I am willing to give them a high quality travel booklet that I have curated about a city in France, the city is called “Shiraz”    : )

Below are two of my initial drafts (produced by matplotlib library in python). They visualize 12,000 unique votes on 700 status updates in a social network.

The first one shows nodes on a circle and connects each note to its neighbor with a straight line. The other one connects nodes with a Bézier curve with four control points (starting node, end node, a point that is 2/3 of the way between the start point and the center, a point that is 2/3 of the way between the center and the end point). I still do not know what is the best way to draw arcs for a circle map and perhaps a circular arc might work better than a Bézier curve. but I’ve been happy with the Bézier curve so far. (make sure you take a look at the animations too [1], [2] they are large files so please wait a little for the animated gif to load completely)

Circle-map of a social network with straight lines connecting adjacent people

Circle-map of a social network with Bézier lines connecting adjacent people

I also produced two animations of these votes using Image Magick scripts. See the animation for straight lines and for Bézier curves.