Hadoop

Not so fast, VMware.

For the past two years, virtualization giant VMware has been pushing for virtualizing the Hadoop stack to make it run more easily for enterprise customers. To try and push users to virtualize Hadoop on its vSphere server virtualization stack, VMware created Project Serengeti, which virtualizes the control and compute nodes in a Hadoop cluster.

Now there’s an effort to virtualize Hadoop on the open-source OpenStack infrastructure-as-a-service platform, named Project Savanna, an effort that was announced last week at the #OpenStack Summit. Behind the project is Mirantis, the largest independent vendor of OpenStack services, as well as Hadoop distributor Hortonworks and open source giant, Red Hat (News - Alert).

Both setting up an OpenStack cloud and putting together a Hadoop cluster have been criticized for being too challenging, and the team behind Project Savanna have said that automating the process of installing both OpenStack and Hadoop is the answer.

"The infrastructure guy wants to deal with OpenStack and not deal with the complexities of Hadoop," said Jim Walker, director of product marketing at Hortonworks, as reported in The Register. "And the Hadoop guy doesn't want to have to deal with the underlying infrastructure."

Mirantis teamed up with Hortonworks for its Hadoop experience and Red Hat for its Enterprise Linux and KVM hypervisor. The Register is reporting that Rackspace Hosting (News - Alert) might contribute, too, although that is still just a rumor.

The basic cluster provisioning code from Mirantis is complete, according to The Register, and the first release of Savanna will include cluster operations. It will not be released until the end of June when the Hadoop Summit is held, however.

In phase two of the project, OpenStack will be tweaked to allow for the addition and removal of nodes in a manual mode from a Hadoop cluster, reported The Register (News - Alert), and Savanna will be able to configure the Hadoop cluster topology and integrate with the Swift object storage controller bit of OpenStack. This second phase will include integration with various third party management and monitoring tools, too.

The third phase is expected to be completed by the end of September, and will make the project start to resemble the Elastic MapReduce (EMR) Hadoop service over at Amazon Web Services (News - Alert), which lets customers submit a job and the cloud determines what hardware is needed. There will be an API in the OpenStack add-on that allows for Map/Reduce jobs to be kicked off without people knowing or caring what the underlying infrastructure looks like.

Also, Savanna will get a user interface for submitting ad-hoc queries to the Hive or Pig add-ons to Hadoop, according to The Register.

All this adds up to trouble for VMware’s efforts.

“There is a big difference between putting code out on GitHub, as VMware has done with Serengeti, and building a community around an integrated Hadoop/OpenStack setup, which is what the OpenStack community will now do,” noted The Register.