Data scientists reveal roadblocks to digital transformation

Digital transformation has accelerated significantly due to the Covid-19 pandemic, but the extra demands on data scientists have revealed significant barriers to effective working and high levels of job dissatisfaction in some areas, a statement said.

For example, around four in 10 are dissatisfied with their company’s use of analytics and model deployment, while more than 20 barriers to effective working emerged, according to a survey of data scientists commissioned by analytics company SAS.

However, the work of data scientists has grown in importance with many organisations accelerating digital transformation projects by using technology to improve business operations. More than 90 per cent of respondents indicated the importance of their work was the same or greater compared to before the pandemic.

To delve deeper into the state of data science, the report assesses the impact of the pandemic, challenges faced, overall satisfaction with the analytics environment and more. The research showed the pandemic upended standard business practices, shifting the assumptions and variables in models and predictive algorithms and causing a ripple effect of adaptations in processes, practices and operating parameters.

More than two-thirds of respondents were satisfied with the outcomes from analytical projects. However, 42 per cent of data scientists were dissatisfied with their company’s use of analytics and model deployment, suggesting a problem with how analytical insights are used by organisations to inform decision making. This was backed up by 42 per cent saying data science results were not used by business decision makers, making it one of the main barriers faced.

The survey also highlighted some specific skills gaps. Less than a third of the respondents reported having advanced or expert proficiency in programme-heavy skills, such as cloud management and database administration.

“There have clearly been more demands placed on data scientists and type of the demand started to show a lot of variations which in the end require different technical and business skills from them” said Celal Kavuklu, customer advisory director for Middle East and Africa at SAS.

“A major source of frustration for organisations is finding a way to have sustainable and agile AI processes with enabling technologies. Analytical insights are big catalysts for digital transformation initiatives and having a sustainable and agile analytics center brings a clear edge in the competitive market conditions.”

“Linked to this, we found concerns around retaining the data science talent and a lack of skills in teams for the emerging critical roles like Data Engineering, ModelOps and DevOps, which has been an issue for some time with demand outstripping supply. SAS is continuously helping organisations with its advanced cloud-native technology, bringing more than 40 years of experience and by providing fully managed services. Also, we are enabling talent building programs to make sure there are the right skills in place to create a generation whose insights, efficiency and agility will be the driving force behind the business transformation,” Kavuklu added.

Other challenges experienced were the amount of time spent on data preparation versus model creation. Respondents are spending more of their time (58 per cent) than they would prefer gathering, exploring, managing and cleaning data.

Overall, the data scientist has ample reason to feel empowered and optimistic about how the pandemic has shone a spotlight on the importance of their role within their organisation and how it might evolve over time. This holds especially true if data scientists can leverage the whole spectrum of available tools to manage the analytics life cycle, pursue data science training and skill development opportunities, and embrace data prep as the first step in modeling.