The financial industry is on the verge of an open source breakthrough, say three companies on the cutting edge of the trend. Traditionally very secretive about their technology, banks, hedge funds and other financial services companies have begun in the past few years to talk about how they use open source software in their infrastructure and product development. They have also been steadily increasing their contributions to upstream projects in the form of user feedback and code. And some companies have initiated their own open source projects or released portions of their own code to the open source community.

“The finance industry is just starting to catch onto open source and the open source way of developing software,” said Andrew Phillips, director of technical operations at LMAX Exchange, the world’s leading MTF for electronic foreign currency trading (FX) based in London. “In the next few years, there may be a bit of a sea change as people start pulling together and providing more of a common platform.”

Though many financial institutions have the same basic infrastructure requirements – largely based on Linux – they’ve long employed their own engineering teams to build these systems from the ground up, and at great expense. But stricter regulations on the finance industry after the 2007/08 financial crisis have caused IT departments to tighten their belts. So IT managers are starting to leverage open source tools and components to cut down on custom development costs and maintenance overhead.

This trend is most evident in the adoption of open source automation tools as institutions move to the cloud, says Vinod Kutty, senior director at CME Group. Enterprises are now looking to big web companies such as Google, Amazon Web Services, and Facebook as the model for their own cloud migration – adopting the same open source tools for devops and continuous integration such as Puppet, Chef, and Ansible.

“We’re starting to question what’s really necessary to keep proprietary and what’s not,” Kutty said. “If we don’t need to build it ourselves, then we might as well use open source tools.”

LMAX Exchange, Bloomberg, and CME Group are three companies in the financial industry that are innovating with open source tools and components, and moving past merely consuming open source software to becoming contributors. Below is more information on how they’re using open source today in vastly different markets and how they plan to increase their use and contributions in the future.

LMAX Exchange

Ranked as the UK’s fastest growing technology firm in 2014, LMAX Exchange is the first to introduce exchange-style execution, with complete pre and post-trade transparency, to the FX market. Servicing brokers, funds, corporations, asset managers and banks, LMAX Exchange is the emerging benchmark for global FX and is creating a level playing field for all market participants.

Launched in 2010, LMAX Exchange has been intentionally built on Linux and open source from the start, a departure in an industry ingrained with secrecy. The company has released the code for seven of its own components – including a Java library for “high performance inter-thread communication” that helps speed transaction times in massive, data-intensive applications. And its engineers contribute back upstream to the open source components they use, as well.

“We’re very open – we tell people what we’re running and how we’re running it. We speak at events and set up our own seminars,” said Phillips.

“There’s been some discussion among senior techies that we should open source the entire platform and we have various thought experiments about what would happen if we did that,” he said. “We don’t think the world would change much for us. It’s how we put the components together and our low latency magic that sets us apart. The source code is an incidental by product of what we built.”

LMAX Exchange has benefited greatly from its open source approach. It’s seen steadily increasing contributions to its projects from other financial services firms using them. The company has an easier time recruiting developers. And customers tend to come to them now, having heard about the company from their IT departments which use LMAX Exchange open source components.

“As time has gone by, just like the Linux kernel itself, a lot of people have contributed improvements that benefit them and the whole thing has gotten faster,” Phillips said. “Some core components are now running at tens of millions of transactions a second.”

It’s only a matter of time until the rest of the financial industry catches on to the benefits of open source, he says.

“The financial industry is realizing that if they don’t get on board, they’ll be left behind,” Phillips said. “They’re going to be producing buggy software and it will take them longer and get more expensive, when they could be applying the state of the art. And the state of the art is open source.”

Bloomberg

Bloomberg is a global software, data, and media company based in New York that delivers real-time financial data, news, and analytics through the Bloomberg Professional service (also known as The Terminal), as well as its media properties. Though the company is largely a consumer of open source technologies throughout its infrastructure and products, it is also increasingly contributing to open source projects like Hadoop and Solr, and is a member of the Linux Foundation and the Core Infrastructure Initiative. Bloomberg has even released some of its own technologies to the community.

“We’re not at the point where we have a formal open source program, but over the last three or four years we have begun to strongly encourage developers to use open source tools to solve problems, especially if we can contribute back to those projects as well,” said Kevin Fleming, an open source evangelist in the CTO office at Bloomberg.

Within the past year, for example, Bloomberg started using Hadoop to access its price history databases – essentially a spreadsheet that contains hundreds of billions of rows of security pricing data. Many Bloomberg applications query the database in order to generate charts of historical data on demand.

In the past, all of Bloomberg’s data crunching was done using databases and tools built internally. But in seeking open source alternatives to help lower development and support costs, their engineers found that Hadoop had great potential. The catch was that it didn’t quite meet the sub-second response times needed to produce the actionable information their customers needed to give market advice and make trades. The software could take minutes to respond if one of the data containers needed for the batch job was unavailable, Fleming said.

“We needed to figure out how to make the machine notice that a node isn’t responding and re-do that query on a different node so we could get a response back more quickly,” he said.

They found an open feature request for the same functionality from within the Hadoop community and decided they would tackle the problem. In collaboration with Hortonworks – which provided the development time while Bloomberg did the testing – they made the necessary changes and switched to Hadoop.

By working with the community they were able to make changes to the software that improved their products faster, and ultimately lowered their development costs.

“Our primary focus is to try to leverage as many of those open source projects as we can,” Fleming said.

CME Group

CME Group started using Linux more than ten years ago in order to scale out its IT infrastructure and lower costs at the same time. In the process the company learned that moving off of proprietary systems yielded performance benefits as well, said CME Group’s Kutty. So they became increasingly engaged in working with the Linux community to implement changes that further improved performance.

Open source technologies have since moved up the stack and CME Group has begun evaluating its open source options as it moves to the cloud, Kutty said. The focus has primarily been on adopting open source automation tools but the company is also interested in advancing common open source components in the financial industry focused on open, highly performant messaging, such as Aeron.

As a result, he said, “over the past few years our legal team has become more familiar with open source licenses, and we’ve gradually given more employees the ability to contribute to open source projects.”

The company plans to continue to develop software internally that’s core to its business, and divest itself of the commodity tools that it doesn’t need to build from scratch. Over time he expects they’ll begin to engage more with the open source communities surrounding the tools they’re using.

“Different open source communities have different maturity levels,” Kutty said. “What you find is that when some communities are young, if they’re not exposed to large customers they have a narrow perspective of how the operational concerns work.”

Developing a good working relationship with new open source communities takes time for companies, and CME Group is in the early stages of this now with cloud and automation tools. When the company first started working with the Linux community, for example, CME Group would request a feature and a kernel developer would ask why they needed it, he said. “Now they understand there are practical concerns in different environments. There’s usually a good reason why we ask for a feature. The conversation has gotten easier.”

Kutty expects CME Group’s relationship with open source communities further up the stack to continue to evolve, as well. If CME Group isn’t directly contributing code to an open source project at first, it’s often supporting the project financially. Contributions come as the relationship develops.

“Most people think contributing to open source means you actually write code and submit it but there’s an advantage as a customer to asking vendors to do that for you,” Kutty said, “and in many cases that’s what we’ve done.”