Jobs Of The Future: How To Become A Big Data Expert?

There’s a stereotype that only IT departments, programmers, mathematicians and statisticians work with Big Data. The fact is this young industry involves a lot of other professions, from marketers and data storytellers to facility engineers and logistics specialists.

Software Focus has talked to Kate Goldberg, COO at Intersog, a Chicago-based provider of custom software solutions for Big Data, IoT and eHealth.Here’re key highlights of our interview.

Software Focus (SF): Who are Big Data specialists?

Kate Goldberg (KG): It is widely believed that a Big Data specialist is a rocket scientist with lots of different skills. This is true to some extent, because, apart from anything else, they need to have a good understanding of business. Of course, one person can hardly know everything, so we often work in teams — it’s much more productive. For example, one of my colleagues is a specialist in data visualization and data storytelling. She creates amazing infographics through which she can tell any story in numbers and correlations. The main thing is to have a 360-degree view which comes with experience. It took some of today’s leading and best-paid Big Data specialists more that 15 years to have this view.

SF: What’s the desired background for a person willing to pursue career as a Big Data specialist?

KG: There are many different roles in Big Data: for example, you can become a Big Data  engineer or analyst, and these are two entirely different functions. The basic thing pertaining to all Big Data roles is the knowledge of maths, statistics and computer science.

SF: Can you describe the main stages of work with Big Data specialist?

KG: We work with a wide variety of areas including finance, retail, legal industry and eHealthcare. One of the main roles on our Big Data teams is a data strategist: at first stage, most companies simply do not know how to get started with Big Data. Moreover, it can be very difficult sometimes to understand which problem in the company is related to that particular data set and how to use data for solving it.

Big Data in organizations doesn’t begin with purchasing expensive equipment, building robust custom software solutions or datasets analytics tools, but with defining the business objectives that can be achieved by means of data analytics, and with the right approach to their implementation.

For example, almost all of the world’s leading mobile operators create dedicated subdivisions for Big Data which have free access to the data within the company and support of top management and shareholders. This is one of the key success factors in Big Data projects, affecting many functions and entailing significant changes in the company processes.

Methodologically important factor is the so-called Lean Startup Approach, a flexible approach to solving business problems with the help of Big Data. Instead of a lengthy process of developing the final complex model or a product, based on Big Data, it’s necessary to progress with small iterations and quick victories, getting regular feedback from the key customers of the solution. For example, the Telefonica company, while designing their Smart Steps solution using the aggregated data on the location of subscribers, initially focused on retail sale companies. The operator planned to provide customers with data on the movement of people in certain streets. Thanks to regular feedback, Telefonica could make the decision about the necessary strategic turn, changing the product focus to analysing passenger traffic for the transport sector.

As I’ve said before, the most important thing to begin with is to define a problem faced by your company. We hold interactive workshops where we tell our clients and prospects about the possibilities of Big Data and help them strategize their Big Data journey. During this stage we need to transform the way your company works and enable you to take advantage of your data points, but our primary goal is to help you solve the problem. We talk with the customer asking lots of questions on all areas of activity. These conversations result in huge lists of points and tasks which we will take into account and based on which we will work. The main goal that we pursue when working with Big Data is the ability to better understand the consumer, the product, the employees, or the suppliers.

After collecting the information, we discuss all the problematic moments and understand whether they are related to Big Data. Some problems may be related to something else — for example, to the lack of employee motivation or poor employee morale. So we have to shorten the list and keep only the problems relating to our competence, i.e. Big Data. If you want to know more about your sales, you should be able to keep record of every tiniest transaction. Sometimes it’s quite difficult. For example, in stores you should be able to take account of each purchase. But this is not a Big Data problem. That being said, you may simply need to buy a system for tracking your accounting instead of spending money on custom data discovery and analysis. Sometimes a number of significant changes should occur in the company so that a Big Data specialist could start work.

The next step is compiling the list of recommendations. After that we discuss the company’s future strategy, and make sure there’s shared vision among all product owners and stakeholders.

Implementation of Big Data solutions normally envisions involving more than just one expert, it’s a change in all employees’ mindset. As such, it’s important for everyone in the company to understand how Big Data teams should be structured. It’s very important to bust the myth that Big Data is just a function within your IT department. It should be hardwired into your entire IT strategy to be successful.

Having determined the strategy and received corporate buy-in, we offer the way of its implementation, including algorithms, tools and platforms, UX design and other features and factors.

SF: And still, what are the key skills a Big Data specialist must have?

KG: What matters is the ability to work with huge amounts of information and possess good knowledge of modern technologies. At the same time, successful Big Data specialists have scientific thinking, and are very inquisitive. It’s very important to be able to think in business terms and out-of-the-box.

In addition, the key quality for both technical and managerial specialists is to be cross-functional. It’s almost impossible to have a full range of data analytics skills. However, technical specialists should have a basic understanding of the functioning of business and managers should have understanding of the basic principles of analytics. Therefore, educational programs in the field of Big Data, combining both the technical part and the business aspects with immersion into certain industries, have a good chance to train the personnel demanded by the market.

SF: What industries are in greatest need of Big Data specialists and solutions?

KG: Every single industry, I guess. Big Data is increasingly used in the banking sector, public administration and agriculture. Engaging a specialist in Big Data is an opportunity to look at the available data from different angles. Sometimes we work with very simple data sets such as spreadsheet tables with only three columns (e.g., date, customer number and amount of purchase). While this might seem primitive, it shows companies, especially the SMEs and startups, how much new information they can get out of it. Even if you do not have tons of data to analyze, you can make some basic predictions and conclusions and try to build your corporate BI based on them.

SF: What’s the most unusual area for which you’ve built a Big Data solution?

KG: Without doubt, it’s agriculture. There is a plethora of data analytics possibilities in this industry, but at the same time it is absolutely not ready for new technologies. One must learn to speak their language and understand what challenges are faced by farmers. For example, a very common task is to reduce the consumption of water used in agriculture each day in excess. Using Big Data solutions to help solve such problems is amazing. Agricultural organizations have to be pragmatic, aided by Big Data.

SF:What would you recommend to young professionals?

KG: Specialists in Big Data are a new type of professionals. You should understand that the most amazing thing in this job is the ability to greatly influence global processes. It’s something like a detective’s work. You find what’s happened, where and why. You can help companies understand why they’re losing money and customers and how to avoid this and increase future profits.