Big data has quickly turned into the biggest thing to hit information technology since the virtualization craze of the last decade. According to research firm Wikibon, the big data market is on the verge of a rapid growth spurt that will hit $50 billion worldwide within the next five years. The rate at which the importance and popularity of big data has grown can be directly attributed to open source. Most of the new big data frameworks and databases have their roots in the open source world, where developers routinely create new approaches to problems that haven't yet hit mainstream.
Hadoop Transitions from Experimental to Mainstream Enterprise IT
Big data is largely centered on leveraging the open source Apache Hadoop platform as well as innovation coming out of the companies supporting or extending it like Hortonworks or Cloudera. NoSQL databases are open source as well. This is where the center of IT innovation is and will continue to be, and the companies supporting this innovation will continue to disrupt incumbent vendors.
Not even five years after it first became available as an open source project, the Hadoop platform has become enterprise-ready and will gain significant acceptance by mainstream IT in 2013. This is due in part to certified and manageable Hadoop distributions such as Hortonworks’, Cloudera’s or MapR’s, but also to the availability of integration solutions that allow other applications, systems and databases to interact with Hadoop. No longer an experimental platform used only in labs, Hadoop will become mainstream technology in the overall IT environment.
A New Breed of Partnerships
Open source communities are fostering innovative new approaches and ecosystems, increasingly getting a jump on the traditional providers of proprietary offerings in advanced analytics, data warehousing and integration. Because Hadoop was born as an open source project and is governed by the Apache Software Foundation, it has created a unique development ecosystem. It’s remarkable that big data technology is actively being developed and maintained by several competing vendors. New technologies and partnerships are being announced almost every week.
While these companies are partners for the development and greater good of Hadoop – at the end of the day only one vendor will be selected as the technology partner for a given deployment for a given customer. Yet they’re all making contributions to the same Apache Hadoop stack for enterprises, to make it better. That’s the beauty of open source. The technical complexity of big data is so large that you need a body as large as a community, rather than a single vendor, to tackle it.
Going forward, we’ll start to see more “hybrid” platforms, and a symbiosis between established software companies – think Oracle Exadata & Cloudera, or Microsoft (News - Alert) & Hortonworks – and the open source community, leading to greater innovation and increased integration between open source and legacy systems.
Yves de Montcheuil is the Vice President of Marketing at Talend, the recognized leader in open source integration. Yves holds a master’s degree in electrical engineering and computer science and has 20 years of experience in software product management, product marketing and corporate marketing. He is also a presenter, author, blogger and social media enthusiast, and can be followed on Twitter (News - Alert): @ydemontcheuil.
Want to learn more about the latest in communications and technology? Then be sure to attend ITEXPO Miami 2013, Jan 29- Feb. 1 in Miami, Florida. Stay in touch with everything happening at ITEXPO (News - Alert). Follow us on Twitter.
Edited by Rachel Ramsey