Zipf’s Principle of the Least Effort, Meta-Writers, or My Data is Bigger than Yours.

Innovation is driven by laziness. I was just thinking about how great it would be to have an algorithm write all my texts. Writing is governed by Zipf’s law, also called the Principle of the Least Effort, which basically states that we become less creative in our choice of words and tend to repeat ourselves. (Wikipedia states that given some corpus of natural language utterances, the frequency of any word is inversely proportional to its rank in the frequency table. Thus the most frequent word will occur approximately twice as often as the second most frequent word, three times as often as the third most frequent word, etc. For example, in the Brown Corpus of American English text, the word “the” is the most frequently occurring word, and by itself accounts for nearly 7% of all word occurrences (69,971 out of slightly over 1 million). True to Zipf’s Law, the second-place word “of” accounts for slightly over 3.5% of words (36,411 occurrences), followed by “and” (28,852). Only 135 vocabulary items are needed to account for half the Brown Corpus.)

Beginning with the Turing Test, followed by speech-generating programs like Joseph Weizenbaum’s Rogerian psychotherapist ELIZA (talk to her), Kenneth Colby’s PARRY (a simulation of a paranoid-schizophrenic company vice president) – you can read them talk to each other here – and finally my friend David Link’s Poetry Machine, creating language seemed to be the ultimate feat in artificial intelligence, the holy measure of what can be deemed human.

Good thing that every 30 seconds or so, an algorithm writes a news story these days. Companies like Forbes employ a technology implemented by a company called Narrative Science, whose CTO claims the Pulitzer for his software “Quill” is only a few years away. He sounds like PARRY when he speaks.

This is the age of Big Data and what Narrative Science calls “meta-writers”, trained journalists who have built a set of templates for the AI to use. “Quill” now writes monthly reports for fastfood companies and is beginning to write its first longform articles. It can identify and break news stories. Subject to the uncanny valley, some companies decide not to let the reader know that the story wasn’t written by a human. Some others show it off proudly. From now on you have to guess if I am an artificial intelligence or not. “My mother? I’ll tell you ’bout my mother!”