Handling Big Data in the New Year: Using Technology and Optimizing People
January 11, 2013-
Gartner has predicted that, by 2015, “big data” will generate 4.4 million jobs globally. The bad news? Only one-third of those jobs will be filled. Why? There is a shortage of skilled analysts to do the work.
The concept of “big data” entails collecting, structuring and using data effectively. Generally, says Bob Richardson, senior director, strategic insights, for Control Group, an innovation strategy firm based in New York, the skills needed by organizations related to big data include:
- Database engineering
- Statistical programming/analysis
- Design visualization
It is the second skill-statistical programming/analysis-that tends to get the most focus and companies are eager to find competent “data scientists.” But, he notes: “There is a well-documented shortage of data scientists and there will continue to be for the near future.”
Puneet Mehta is co-founder and CEO of MyCityWay, a mobile application powered by BMWi Ventures, based in New York. “Every subject has various sources of information, colored in various shades of opinion, just a mouse-click or finger-swipe away,” says Mehta. The core skill needed to manage all of this data, he says, is: “Knowing exactly where to cast your net into this ocean of data, then throwing away much of what you catch and looking only at a small portion to make sense out of it.”
Doing this effectively requires a combination of technology and people.
Using Technology Options
The old aphorism “garbage in, garbage out” is still applicable. It is not so much who is managing the data, as it is how the data is being managed.
“We think there is a powerful way that an organization can begin using data without hiring a data scientist,” says Richardson. How? By making sure the database is structured to deliver useful data and developing engaging, easy-to-use design visualizes that provide real-time data across the organization. Doing that, he says, will allow organizations to “outsource difficult data problems on an as-needed basis.”
A shortage of skilled staff demands strategic and technology-driven solutions, says Andres Reiner, CEO of PROS, a big data software company with world headquarters in Houston. “With a lack of professionals qualified or interested in big data jobs, company networks will have to fight fire with fire-using the same technology that created the data to overcome the challenge of organizing it into something meaningful.
“Customized platforms that analyze data sets and apply appropriate algorithms to kick out sales quotes, inventory predictions, etc., will become key tools for all members of the company.”
Technology can be great, but people are still an integral part of the mix. When it comes to big data, learning to use people effectively can pay big dividends.
Optimizing People
A high demand, and a shortage of talent, will require new perspectives about how to get the work done, say the experts.
First, data analysts’ time must be protected and used for analysis work-not routine tasks that could be performed by others. “Given the shortage of available data science resources, it’s critical that the data scientists that you do have are used as effectively as possible,” says David Smith, VP of marketing and community for Revolution Analytics, a software and services firm headquartered in Palo Alto. “The worst thing you can do with such a valuable resource is to have them spend most of their time responding to ad-hoc requests whose outputs-like a report or spreadsheet-will only ever be used once.”
Much of data analysts’ time is spent collecting, correcting and recollecting data, notes Kevin Lyons, SVP analytics, with eXelate, a provider of data-driven solutions for advertisers and agencies based in New York. “Making sure that the analyst’s time is spent on tasks which require an analyst will alleviate some of the constraints,” he says.
In addition, other employees will need to be trained on data skills, if demand continues as predicted, says Mehta: “It does not matter if you are in sales or business development or any other high-touch job-making data-driven decisions and using the tools to facilitate that would be a standard part of any job.”
“I think big data will become a job requirement for regular businesspeople, who will have to adapt and learn to extract value from data,” agrees Matt Fates, with Ascent Venture Partners in Boston. “I think business schools will increasingly incorporate data-crunching and statistics courses into the curriculum-more so than they do now,” he says.
Michele Chambers, chief strategy officer with Revolution Analytics says in an environment where talent is at a premium, there must be ways to leverage that constrained talent pool. She adds, “There will be traditional approaches-applications, productivity tools, outsourcing-but there will also be new business models. You see the early indicators of this in open source and crowd sourcing of big data analytics.” Necessity, she points out, is the mother of invention.