Syncsort, a global provider of Big Iron to Big Data solutions, announced the results from its third annual Hadoop survey, showing that as users gain more experience with Hadoop, they are building on their early success and expanding the size and scope of Hadoop projects.
The number one role of Hadoop continues to be increasing data warehouse capacity and reducing costs (cited by 62% of respondents) but the number of organisations using it for better, faster analytics is on the rise (49.4% of respondents, up from 45.5% last year). As the ways in which they are using Hadoop and the benefits they expect to achieve are evolving, there are several key trends to watch in 2017.
- Traditional data sources are more popular for filling the data lake, but newer sources are a sizable part of the mix. As Hadoop projects multiply, so too, will the sources and volume of data required to support them – from mainframes to NoSQL databases and beyond. Legacy systems remain essential for populating the data lake, with the top three sources including: enterprise data warehouses (EDW) (69%), relational database management systems (RDBMs) (61%) and mainframes (40%). Newer sources are gaining importance, especially those that generate streaming data, such as smart devices and sensors (53.5%). Therefore, having a tool that allows organisations to easily access and integrate all data – legacy and newer sources, batch and streaming – is vital.
- Big Data insights will become a mainstay in running all business operations. Respondents expect significant benefits from Hadoop, with increased business/IT agility (66.9%) and greater operational efficiency and reduced costs (66.5%) in a virtual tie at the top of the list. When looking at how respondents are turning data into business insights, the majority are interested in using Hadoop for advanced/predictive analytics (62%), followed by data discovery and visualisation (57%) and ETL (53%). As Big Data projects mature, businesses will become more confident in and dependent on their results. Big data won’t just be fodder for data scientists and analysts, but will feed the KPIs and other important metrics needed to run the business.
- Data governance grows in importance – mainframe data continues to be key. Over two-thirds of respondents from organisations with mainframes find accessing and integrating these critical data assets into Hadoop for real-time analytics valuable or very valuable (70%). Highly-regulated industries such as financial services, banking, insurance and healthcare have the challenge of making the critical legacy data assets available for real-time analytics while also maintaining cross-platform data lineage for compliance purposes. As Hadoop implementations spread across organisations, data governance, and the data quality needed to support it, will become more critical to meet the regulatory and compliance mandates.
- There will be a move toward simplicity. For the third year in a row, the need to learn a new set of skills and tools and finding affordable Hadoop programmers is among the top challenges (58%), followed by difficulty keeping up with evolving compute frameworks (53%) and integration with existing data sources and applications (48%). As more technology designed to support specific use-cases comes online, tools that remove the complexity and streamline processes will be in high demand.
- Organisations will increasingly implement strategies and tools that easily adapt to take advantage of a rapidly evolving technology stack. As the Hadoop market matures, MapReduce remains the more popular framework (62%) and half of respondents use at least one other compute framework along with MapReduce (51%). Spark adoption is on the rise – 55% currently use Spark, but that number is expected to grow to 76% in the future. 47% of those currently using MapReduce plan to move away from the framework. In addition, a third of respondents deploy Hadoop both on premise and in the cloud. To achieve the efficiencies enterprises seek with Hadoop, users will demand tools that handle different environments with ease, without burdening their teams with continuous application maintenance or requiring complex coding skills.
“Insights from Big Data can offer new revenue streams, new customer insights, improved decision-making, better quality products, enhanced customer experiences, and more, but organisations are still struggling to have easy access to all their enterprise data,” said Tendü Yoğurtçu, general manager of Syncsort’s Big Data business.
“From our survey results, it’s clear businesses are realising the importance of tapping into a full range of data sources, and the benefits of using products and solutions that can simplify and bridge the gap between legacy and emerging platforms, Big Iron to Big Data solutions to streamline and accelerate the ROI on their Big Data initiatives.”
Comment on this article below or via Twitter: @ VanillaPlus OR @jcvplus
