Making sense of online chatter
by David Reed
July 23, 2010
Business Profile: Idea Works
Wrapping your head around what Idea Works founder Ed Brent does is a bit like trying to understand how a slight turn of the radio dial transforms static into a Mozart symphony.Brent, a professor in sociology and computer science at MU, has spent three decades refining the coding processes that enable computers to analyze language. He spent the first part of his academic career focused on sociology but then became interested in how computers could help with his research.
Idea Works is the company he formed in 1981 to bring his technology to the marketplace, including a computer program that can grade hundreds of student essays in minutes and provide feedback.
The company’s growth in the past five years, from a few employees to 10, correlates to the explosion of online conversations on websites, blogs and social networks such as Facebook and Twitter.
The latest technology from Idea Works is a product called Veyor that makes some sense of the millions of social media posts that flood into the online community every day. The computer program is taught to discover topics, themes and concepts found in text and analyze them for attitudes and opinions.
In a research paper for Social Science Computer Review, Brent used a musical composition analogy to explain how his tools help researchers analyze a vast database of coded data, typically segments of text: “The patterns of data that result have a clear structure, a ‘rhythm’ if you will, that provides the underlying ‘beat’ within which the ‘melody’ of codes is experienced.”
For a more practical explanation, the latest application of the Veyor technology measured and analyzed public sentiment related to several US Senate primary races in May. Insight, a public opinion research company based in Columbia, customized Veyor to analyze political races with a product it calls Globalpoint.
The metrics used by Insight, a subsidiary of Pure, were based solely on the analysis of text data from news and social media posts and included no public opinion polling data.
In the Senate primary race between Kentucky Democrats Jack Conway and Daniel Mongiardo, programmers using Globalpoint collected data the week before the election. A candidate’s name was one of the triggers that brought an article or website posting into the data collection system.
Candidates received positive or negative points based on what was being said about key issues in the race and were categorized under headings such as “government,” “economy,” “personal” and subsets such as “free market” and “tax issues.”
The computer system “read the documents much like a human would,” Wade Foster of Idea Works said.
Conway had been far behind in the race. But after analyzing the data, they came up with a report on sentiment trends, candidate mentions and campaign issues that predicted Conway would win by 2 percent.
Insight President Michael Urban said Globalpoint’s prediction was closer to the final margin — Conway won by 1 percentage point — than public opinion polls based on telephone calls to registered voters.
Pure President Brent Beshore said phone polling is becoming less accurate because more voters are using wireless phone services, and online data analysis is becoming more valuable as conversations over the Internet grow in volume.
“That conversation is growing louder and louder,” Beshore said. The sample size is now large enough to be considered representative of the entire population, he said.
Foster, Idea Works’ head of business development, said: “Politicians and media outlets and special interest groups are interested in what people have to say, especially during elections. Online discussion in the last few years has exploded, and we try to get a sense of what is being talked about.”
Foster said the performance of the technology during the Senate races “was a very interesting result, a validation.”
There are many other social media monitoring tools in the marketplace, such as Radian6, Scout Labs and Sysomos, but Foster said Veyor is better for customized applications such as Globalpoint.
Urban said, “We believe there is a big market demand for smarter and better intelligence in the political market, based on all the qualitative and text-based data that you find on social media networks, news sites, blogs and chat rooms that continue to grow in number.”
Idea Works formed its partnership with Insight last summer, and Foster said the joint projects are “where most of our development resources are going now.”
Idea Works had been focusing on educational and corporate applications of its technology, and Insight suggested they collaborate and expand into the political sphere.
“We’re looking forward to partnering with them on multiple products that come out of the baseline technology,” Beshore said.
Coincidentally, the offices of Pure and Insight on West Broadway are only half a mile or so away from Idea Works, which is run out of a large two-story house near the Broadway-Stadium Boulevard intersection.
Other products developed by Idea Works include the essay-grading software, tools for developing research proposals and qualitative data software for coding.
When not telecommuting from Kansas City or St. Louis, the programmers primarily work in a room formerly used as a den, surrounded by tall bookshelves, and meetings are sometimes held in the living room where workers sit on couches and watch a large monitor during teleconferences.
Brent said they’ve outgrown the house, and though they like the laid-back atmosphere it engenders, they’re looking for a new building with more office and parking space.