The folks at the Open Data Platform Initiative (ODPi) have heard the concerns and the criticisms of the Hadoop community, and today John Mertic, the standards organization’s Director of Program Management, took to Apache Big Data in Vancouver to clear the air.

Contrary to the Hadoop community’s concerns, ODPi does not want to take over the development of Hadoop, it does not want to fork Hadoop, Mertic said.

The ODPi wants the big data projects based at the Apache Software Foundation, including Hadoop, to continue to innovate, try new things and splash around in the code base’s pool – digitally speaking, of course.

What ODPi intends to do is keep the companies looking to use Hadoop in production downstream from getting wet while the ASF is making waves.

ASF for innovation, ODPi for standards

ODPi was formed last year by dozens of leading tech companies, including Hortonworks, IBM, and Pivotal, as a collaborative project at The Linux Foundation to develop a common reference platform called ODPi Core. The nonprofit has been developing standards around a few Apache projects, including Hadoop in its Runtime Spec and test suite, which includes HDFS, YARN, and MapReduce. The Operations Spec due out this summer will focus on Ambari.

To show the organization’s commitment to the ASF’s mission, Mertic announced that ODPi is now a Gold Sponsor of the open source foundation.

“We want the ASF to be the home of innovation,” Mertic said. “What we want to bring to the table is those use cases from industry; [to show] how are people using Hadoop.”

Mertic said that end users and independent software vendors (ISVs) have been frustrated by inconsistencies across the various Hadoop distributions, which is having a dampening effect on investment. He said while there are 15 generally-accepted common components to a Hadoop distribution, the versions of those components are often different from distro to distro.

“Both our organizations are laser focused – we want a stronger Hadoop ecosystem,” Mertic said, of the ASF and ODPi. “But let’s be honest, it hasn’t been easy.

“The Apache project really focuses on those raw components, the peanuts and the chocolate, if you will. But if you’re an end user, you’re looking for that Reese’s Peanut Butter Cup.

By developing a layer of standards for Hadoop distributions so end users and ISVs see a consistent level of performance and progression, to spur adoption and investment.

“From the ODPi perspective, we find this to be the place where we we need to provide a baseline group of standards so people know what to expect,” Mertic said.

Mertic gave the following quote by Robert W. Lane, CEO of Deere & Company, as the perfect encapsulation of the ODPi’s mission:

“A sustainable environment requires increased productivity; productivity comes about by innovation; innovation is the result of investment; and investment is only possible when a reasonable return is expected. The efficient use of money is more assured when there are known standards in which to operate.”

It only makes sense that as the community of Spark contributors got bigger the project would get even more ambitious. So when Spark 2.0 comes out in a matter of weeks it’s going to have at least three robust new features, according to Ion Stoica, the founder of Databricks and keynote speaker at Apache Big Data in Vancouver on Tuesday afternoon.

“Spark 2.0 is about taking what has worked and what we have learned from the users and making it even better,” Stoica said.

Queries will be more performant – the goal is 10x faster – through the success of Project Tungsten, an ongoing effort which set out to improve the efficiency of memory and CPU for applications. The three ways it’s succeeded is through cache-aware computation that uses algorithms and data structures to exploit memory hierarchy, code generation to exploit modern compilers and CPUs, and using application semantics to eliminate memory getting bogged down on garbage collection and the JVM object model.

“The more semantics you know the better you can optimize the applications,” Stoica said.

Spark 2.0 will ship with even more components from Project Tungsten, which has been rolling out in pieces, across multiple releases, since Spark 1.4 about a year ago. 

Spark 2.0 will also feature improved APIs to make it “even easier” to write applications for Spark, a feature for the influx of data scientists that are now using Spark who aren’t necessarily full-blown developers and database admins. Part of this feature is the introduction of the Dataset API. Datasets are static typed extensions that use Resilient Distributed Dataset (RDD)-like operations, and when added to Spark’s dataframes, it creates a best-of-both-worlds approach.

Stoica said each library in Spark – things like graphing libraries, machine learning libraries, and so on – will be rewritten to include datasets.

The third major improvement is increased support for streaming data by creating what Stoica called “infinite dataframes,” which is a table that’s constantly adding new entries.

“We’re going to integrate support for interactive and batch queries,” Stoica said. “It’s not just streaming, it’s what we’re going to call continuous applications.”

Stoica ran a demo of the streaming capabilities through his company Databricks’ Spark distribution by showing Twitter clusters on a map computed in real time. But the implications are much bigger for running analytics on streaming data with all the many systems and languages that connect standard to Spark. This will be huge, he said, for applications like fraud detection, or updating machine learning algorithms in real-time.

“Everyone knows that streaming is more and more important,” Stoica said. “You want to operate and do analytics on fresh data. You want the ability to do queries on the data that was just streamed.”