In 2012, Harvard Business Review crowned the data scientist occupation as the “sexiest job of the 21st century.” Six years later, the field continues to be red hot, with average data scientists earning $120,000 in salary, while a handful of firms pay twice that amount. There continues to be an acute shortage of such professionals while the need for them steadily expands, thanks in part to strong interest in AI and the burgeoning need for IoT data analytics in industrial and other business settings.
“All of the sudden, there’s this mass hysteria,” said Dave McCarthy, vice president, marketing at Bsquare. Many executives and IT professionals around the world are realizing, first, that they have a growing need for data scientists and, second, that it is difficult to find them.
The irony of the matter is, firms that already specialize in data analytics are more likely to recruit more data scientists than other firms. Some of the top employers of data science professionals are tech firms such as Amazon, Google, Facebook and Intel; oil and gas companies; consulting firms; along with data analytics vendors are scooping up data scientists. “That leaves the average company in dire straits. They want to be able to take advantage of the data their business generates. They’re looking for data science help, but they can’t necessarily find it,” McCarthy said.
As a result, companies like Bsquare are offering data science as a service for IoT data analytics projects. “Many companies don’t need to have a data scientist on their staff full-time,” McCarthy said. They may have a short-term project that needs considerable data architecting and analysis but lack the budget for an experienced data scientist.
A fair number of companies need help with basics such as articulating what their problem definition is. “There are cases where it is a little bit of the art of the possible. We will go into a company that says: ‘I don’t know what I can do with the data,’” McCarthy said.
More advanced industrial companies may have a firm sense of their problem, but need assistance creating the data inventory required for their project. “It’s great to decide: ‘Hey, I want to understand a predictive failure event on a piece of equipment,’” McCarthy said. “But in many cases, these companies aren’t sure whether or not they're even sampling the right data point. Are they doing it the right frequency? And do they have it in a format consumed by some other data science tools?”
Companies with the right kind of data may find the road to transforming data into insights can be winding. For one thing, the job of data science also entails a certain amount of grunt work. “There’s the joke that 80 percent of data science is cleaning the data and 20 percent is complaining about cleaning the data,” said Kaggle CEO Anthony Goldbloom in an interview with The Verge last year. The time-consuming and expensive process of cleansing data before analysis is leading to a hot market for data engineers and architects. The situation is also leading to growing interest in so-called augmented analytics, in which machine learning replaces some of the menial work of data preparation.
By 2020, Gartner expects technology such as augmented analytics to enable data scientists to focus the bulk of their time on solving specialized problems while “citizen data scientists” take on a central role converting raw data into actionable insights.
This shift could be empowering for industrial subject matter experts who may be data-savvy, but lack formal data science training. “There’s already this debate in the oil-and-gas field, for instance, of whether it is better to lure data scientists out of Silicon Valley down to Houston to try to teach them the oil and gas business, or to take petroleum engineers and train them to operate at the level of data scientists,” McCarthy said.
