Human vs. machine categorization

Published by marco on

Humans are currently better at detecting patterns than machines—this may change in the near future.[1]

However, machines are massively better at sorting detected patterns than humans. Humans can’t stick to a regime.

For example, a site like Reddit amounts to a vast sorting algorithm with posts, pictures, videos and articles as input and the subreddits as buckets.

Myriad users upvote and downvote these items to determine whether an item belongs in a given subreddit. If an item in one subreddit would be appropriate in another subreddit, a user can cross-post it to the other subreddit.

But humans suck at this.

As a specific example, look at a forum like /r/oddlysatisfying. The material you should expect to find there is in the name: images or videos of everyday objects that provide a soothing experience.

“This can be physical (like popping bubble wrap), visual (a perfectly looped GIF), or even aural (the crunching of leaves). […] subjective. Indefinable. Completely recognizable. It’s a wonderful feeling.”

For example:

 Coffee-cup Pyramid

Something like Casual Japanese Wrestling absolutely does not belong there. It just shows a neat backflip in a professional wrestling match.

Two challenges arise when using humans to categorize:

  1. In order to get humans to participate, there has to be a reward system. It’s either money or it’s something else. That something else is called karma on Reddit. Humans aren’t just happy to get the karma appropriate to their behavior; instead, humans modify their behavior to get more karma. That is, instead of sorting data in the system, they let the system train them to sort data. This leads to rampant cross-posting, where people take material that garnered a lot of upvotes in one subreddit and re-post that material in other subreddits to “farm karma”.
  2. Most people don’t really pay attention to which subreddit they’re browsing. They read the title, view the image or watch a few seconds of video and then decide it’s “cool” (upvote) or “stupid” (downvote). They don’t generally think about whether the item is appropriate to the subreddit. That is, you may think it’s cool, but is it oddly satisfying? If not, then you should downvote it in that subreddit.

Taken together, these behaviors lead to a tremendous amount of content being duplicated across a plethora of false categorizations.

Where a machine can be trusted to stick to its rules, applying them rigorously, humans are distracted by shiny objects and make for poor sorters.


[1]

I was inspired by the article How computers got shockingly good at recognizing images by Timothy B. Lee (Ars Technica). It’s a very good article, describing how image-detection works at the technical level. In it, Lee discusses the current limitations to recognition: the algorithms are good, but they work with brute force, by learning from a gigantic dataset with a tremendous amount of what would seem like repetitive data for a human.

“Convolutions allow neural networks to understand translations—they can tell if a pattern in the upper-right corner of one image is similar to a pattern in the upper-left corner of another.

“But beyond that, convolutional networks have no real understanding of geometry. They can’t recognize that one image is similar to another if it’s rotated by 45 degrees or scaled up by a factor of two. Convolutional networks make no attempt to understand three-dimensional object structures, and they can’t correct for varying lighting conditions.

“Yet deep neural networks can recognize pictures of dogs whether they’re taken from the front or the side, and whether they take up a small part of the image or a large one. How do they do that? It turns out that with enough data, a brute-force statistical approach is good enough to get the job done. […] with enough labeled examples, it can learn all possible permutations by sheer repetition. (Emphases added.)”