Will Big Data Kill Creativity?

Analytics has turned its attention toward ebook readers. Companies can now track the way you read; it may have implications for the way authors write. From the NY Times:

The longer a mystery novel is, the more likely readers are to jump to the end to see who done it. People are more likely to finish biographies than business titles, but a chapter of a yoga book is all they need. They speed through romances faster than religious titles, and erotica fastest of all…

At Oyster, a top book is “What Women Want,” promoted as a work that “brings you inside a woman’s head so you can learn how to blow her mind.” Everyone who starts it finishes it. On the other hand, Arthur M. Schlesinger Jr.’s “The Cycles of American History” blows no minds: fewer than 1 percent of the readers who start it get to the end…

He contrasted two romance novels. One had few Amazon reviews and little promotion, but Scribd’s data showed 6 out of 10 readers were finishing it — above average for the genre. Another romance had hundreds of reviews on Amazon, but only about 4 out of 10 readers bothered to finish it. They began closing the book, the data showed, when the writer plunged deeper into fantasy. Maybe this was not a good idea.

Some writers, of course, might not be receptive to hearing this.

“If you aren’t careful, you could narrow your creativity. You won’t take risks,” said Ms. Loftis, the young adult novelist. “But the bigger risk is not giving the reader what she wants. I’ll take all the data I can get.”


Data, Shmata

On my bad days, when I’m wavering in my faith that humans have any air of rationality, I often begin to have a creeping fear that we have become allergic to data and evidence. And I don’t just mean that we avoid science. I mean in our everyday lives. Here’s an example. As I was logging into my WordPress account today I noticed on the homepage a blog post about Jonathan Franzan’s top 10 writing tips. I read the post, smiled, and reflected for a few moments. Noticing the blog’s author had written a related article about Franzan’s The Corrections, and having thought about picking up the book for several months now, I clicked on the link.

Several paragraphs into the post was this claim:

“Second, reviews on The Corrections from non book critics (read: normal people like you and me) are mixed. Take one look at The Corrections on Amazon and you’ll see what I mean. It’s a three-star novel with more than 1,000 reviews. Why is it three stars? People either really love it or really hate it.”

I hear these sorts of declarations made quite often. “Either you’ll love his sense of humor, or you’ll hate it.”  “Either you love her beff stroganoff, or you hate it.” “Either you’ll read this book in a single night, or you’ll throw it down in disgust after the first five pages.”  I’ve never been one to buy into such extreme pronouncements so I decided to mozy on over to Amazon and take a look at this book’s customer reviews.

If we assign “really love it” a review of 5 stars and a “really hate it” a review of 1 star and do a little math we’ll learn that 55% of people either love this book or hate it, while 45% have some ambivalence toward it. Far from the claim that The Corrections is either loved or hated, only slightly more than half of readers harbor such strong convictions regarding the novel.

I don’t mean to pick on what seems like a well intentioned blog (the author is reading all 100 of Time Magazine‘s greatest English-speaking novels since 1923, a glorious and praiseworthy effort); and perhaps a few of you will write my complaint off as pedantic. But I still think it matters.

The example above is but one case of what has become (or maybe always was) an aversion to hard evidence. That we don’t calculate the percentage of people who feel a particular way toward a novel before writing a blog post is of no import. The larger point is that, far from staying neutral on topics we know little about, we transform into raconteurs—waxing lyrical, compelled to have an opinion one way or the other on every topic at hand—all the while ignoring the solid terra firma of the measurable and real as it sinks further and further away beneath our feet.

The question, “How do people on Amazon feel about The Corrections?”, like many other questions in life, has an answer (at least in part). Many of the riddles that confront our everyday lives do not. To treat that which is fact as merely a matter of opinion—or worse, to treat that which is unknowable as something that can be made real simply by will of conviction—is to infuse into the genuine a spurious nonsense. It is to give credence to intentions, hopes, and desires while discounting outcomes, history, and evidence.  It is to bake a pie made of lies—an American Lie Pie—and try to force others to eat it.

The more we make such unwarranted claims in our daily discourse, the lazier our brains become, the more susceptible we become to specious professions, and the more we view data and evidence as the banal details that should be relegated to science labs and courtrooms. I also believe it is partly at fault for our blind adherence to our own ideology.

We are creatures of habit, running in the same circles of friends week after week, watching the same news programs, reading the same websites. The claims we make and hear from others get batted around unchallenged, slipping into conversation as easily as laughter or talk of the weather. True or not, our allegations to one another become reality. So much so that we are jarred when we are confronted with anything different. So enraptured in what we know to be “true” we respond with vitriol and indignation; only then does our scientific mind suddenly jolt to life, demanding from those who have challenged us every scrap of data and evidence on the subject hand. And even then we are likely not to believe.

We may not be able to pause for mathematical calculations or deep research with every lackadaisical comment we make. But we can stop and think, the next time we say something, “I wonder if that is true”. If there is a computer nearby maybe we can look up the answer. We can choose not to speak about the many things for which we have no knowledge, or when we do, we can state them as a matter of opinion, noting that we could easily be wrong. We can understand that on many issues there are multiple sources of data, often conflicting, and in these cases we can talk about the relative merits of each rather than discounting completely the side which contradicts our sensibilities. On matters that are settled, we can follow the evidence, even if it disagrees with what we want to believe. Or at the very least we can say, “Sorry, I understand where you’re coming from and understand your evidence, but I’m biased on the subject. Even if you’re right my heart won’t let me agree.”

Amazon, Sales Tax, and Unintended Consequences

There is a rule in business: if you put a bucket of money on one side, a profit maximizing enterprise* on the other, and some sort of legislation in between, eventually that enterprise will find a way to get to that bucket of money. Economists call this an unintended consequence of the legislation because the cleverness with which rules are circumvented is rarely foreseeable…at least to politicians.


The latest display of this stalwart economic principal is the fight between Amazon and various state legislatures throughout the country who are now demanding that Amazon pay sales taxes in those states where affiliates or subsidiaries operate. The argument behind the proposals are two fold. First, if brick and mortar facilities have to pay sales tax it is only fair that online businesses do the same.  Second, as the current debate in Wisconsin indelibly demonstrates, states are in a pinch when it comes to their budgets.


Let’s take these one at a time.  The idea that online retailers paying sales tax is a matter of fairness is nonsense. At the heart of these arguments is the notion that somehow e-commerce is unfairly competitive with respect to businesses with physical retail spaces. These two types of business models are simply competitive along different dimensions.


While online sellers may enjoy a price and volume advantage in many cases (although this is by no means universal, think: Wal-Mart), they relinquish many other advantages.  First, customers must pay shipping expenses and even then must wait several days to receive their product. They risk having the product damaged during transit or missing the UPS driver if the item requires a signature. They don’t get to skim through the book, try on the skirt, or smell the candle. They don’t get to ask a sales person for advice or have him/her answer questions about the product. If the customer guesses wrong while shopping online, they don’t get the immediate feedback to know that the salesperson should bring them a smaller size of shoe. They don’t get to see colors and textures of fabrics in eye-poping REAL-D or in the unforgiving light of the real world. They don’t get to pick up the computer to see how light 5.1 Ibs really feels.  The customer won’t get to see the brightness of the HDTV, judge the size of the suitcase, or feel the firmness of the pillow. And trying to return something you bought online? If that online retailer doesn’t have a physical store nearby you’re off to the post office to wait in a 10-person line, destined to mail the item back and start the buying process all over again.


I could go on, but anyone who has been to an Apple store lately surely must know the value of a physical retail location. Daily, crowds of consumers pack between its walls to look, feel, hear and yes, purchase.


So how are online retailers stacking up against brick and mortar stores these days? I pulled this report from the US Census website:

E-commerce sales in 2010 accounted for 4.2 percent of total retail sales. In other words, less than $1 of every $20 a customer spends will be through an web-based vendor. Although online sellers are certainly increasing at a healthy pace, it looks like traditional retailer stores are still doing just fine.


Now what about those state budgets. The theory goes that forcing online retailers to pay sales tax will increase tax revenue thereby helping to close budget gaps. While that might work in a world where people’s feet are sealed in concrete, the real world is dynamic—companies and consumers constantly respond to the incentives laid out before them. So what do you do when someone slides a piece of legislation between you and a bucket of money?  You find a way to get to the bucket:

“Meanwhile, last fall, Texas officials sent Amazon a tax bill for $269 million, after determining that the retailer’s Dallas-area warehouse, owned by a subsidiary, qualified as a local address under state tax rules…In retaliation for Texas’s move, Amazon said last month that it would close the warehouse next month and cancel plans to build another.”


If this movement among states continues to gain steam, then Amazon will continue to respond, probably by consolidating its operations to minimize its tax exposure; all the while hurting the states that enact the legislation by reducing employment opportunities. At some unknown point in the future, I predict, some clever state will realize what is happening and invite online retailers by exempting them from sales tax—because states want to get to that bucket of money too—just as Delaware did with laws regulating incorporation of businesses. Laws which have since spread to other states, I might add. (Either that or Amazon will simply relocate to one of the four states with limited or no sales tax). And this certainly wouldn’t be the first time that companies have undergone geographic relocations in response to legislation.


*Note that humans are included as profit maximizing “enterprises”.