One of the major stories this election cycle has been “big data”: campaigns combining voter files, consumer records, and response data collected by their own volunteers to individually target voters. This practice is at once exciting because it allows campaigns greater precision than ever before in how they interact with individual voters, yet it also raises privacy concerns as citizens are often unaware of the amount of personal data available to third parties or how it is being used.

Right or wrong, big data is now a part of our political process. But how did it enter American politics in the first place? This history is recounted in Rasmus Kleis Nielsen’s new book, Ground Wars: Personalized Communication in Political Campaigns, which looks at how campaigns conduct field operations (door-to-door canvassing and phone-banking). Though these activities take place offline, computers are never far away, for it’s the analysis of digitized data that directs volunteers which doors to knock and which phones to call.

The Republicans Strike First

While consumer data has been used since the 1970’s to calculate credit scores and since the 1980�s for direct marketing, political campaigns didn’t get into the digital data game until 1995, when the Republicans created the Voter Vault, a shared and continuously-updated voter file hosted on a server available to Republican state parties and campaigns.

This was quite an improvement over the previous data system. “In the absence of a shared voter file,” writes Nielsen, “every new campaign would have to start from scratch, building their own voter files by collecting public records on registered voters, buying commercial data to enhance it, and making identification calls.” After each campaign, “the entire painstakingly constructed database typically simply disappeared”.

The Democrats Slowly Respond

As soon as the Republicans had a shared voter file, the Democrats had to have one too, though their effort to create one was far bumpier. It was not until 2002 that the Democratic National Committee (DNC), under chairman Terry McAuliffe finally invested in their own system. The result, Demzilla, was distinctly underwhelming. In a 2003 article in Roll Call, one anonymous Democratic consultant complained that “the quality of data is far from a level that would make it immediately useful…. [and] the system is overly cumbersome.” It was hard to use and not worth the effort.

Also, many of the state parties did not even donate their data to the project, afraid that the system would be used more for the 2004 presidential race that for their local campaigns. (This perception was not helped by the fact Demzilla was part of Project 5104, McAuliffe’s campaign to win at least 51% of the presidential vote in 2004.) Notes the Roll Call article, “Demzilla is an idea on paper makes a lots of sense… the problem is that [the DNC] took the idea and let the technology run ahead of the relationships….”

Howard Dean to the Rescue

Howard Dean did not win the presidency in 2004, but he developed enough of a grassroots following that he was able to take the DNC chairmanship in 2005 against the wishes of party leaders. His two big projects were the 50 State Strategy to put DNC-salaried organizers in every state to help the local parties and to hit reset (almost like an Etch-a-Sketch…) on the party’s voter database.

This time the party’s electoral and technological projects complemented rather than undermined one another. While Project 5104 had sown distrust in the state parties, the 50 State Strategy increased it, making the parties more likely to contribute their data. As a result, the new database, VoteBuilder, grew quickly, and VoteBuilder achieved the same data participation in one year that Demzilla achieved in three. Demzilla was abandoned and VoteBuilder rose from the ashes. A single online interface, known as the VAN (Voter Activation Network) was added in 2007. Also, while Demzilla allowed 300 hundred data points to be added for each voter, VoteBuilder allowed 900, a recognition that more data was both available and useful for voter targeting.

Business to Politics: Mitt Romney Shows the Way

This realization about the usefulness of data was largely due the example set by Mitt Romney. In 2001 Romney ran for governor of Massachusetts and his consultant, Tom Gage, used data to “supersegment” voters as never before. Using sophisticated statistical techniques, he created predictive models which determined the probability of future voting behavior based on information about their past political and consumer habits. (Gage’s term eventually loss ground to the term “microtargeting,” possibly because of the divisive connotation of “segmenting”).

In his book, Neilsen quotes Gage as saying that the businessmen who were Romney’s advisers “were flabbergasted when they learned that such techniques, mainstays in many parts of corporate America, were not already widespread in politics.” However, Romney himself likely played a role as well in the centrality of data in his campaign.

In a recent articlein the New Yorker, Louis Menand points out that at Boston Consulting Group and Bain Capital, Romney’s employers from 1975 to 1999, “data crunching seems to have been the main engine of analysis.” Menand draws the link between Romney’s business background in management consulting and his pioneering role in bringing big data into American political campaigning. “Virtually everyone agrees that Romney was extremely good at” data crunching, writes Menand, “and he runs his political campaign in the same way.”

Obama Steals the Data Crown

Though the Republicans clearly got a head start on big data, Nielsen notes that in 2008, many observers believed that the Democrats took the lead. VoteBuilder and the VAN “were built by experienced vendors… and were subject to repeated field testing… before they faced the ultimate test during the general election of 2008” notes Nielsen. “That year, many of my interviewees argue, the Democratic Party for the first time went one better than the Republican party in the targeting and data arms race.”

Coming Soon…. 2012: Clash of the Data Titans