When people talk about cloud native applications you almost inevitably hear a reference to a success story using Apache Mesos as an application delivery framework at tremendous scale. With adoption at Twitter, Uber, Netflix, and other companies looking for scale and flexibility — Mesos provides a way to abstract resources (CPU, memory, storage, etc.) in a way that enables distributed applications to be run in fault-tolerant and elastic environments. The Mesos kernel provides access to these abstractions via APIs and scheduling capabilities in much the same way that the Linux kernel does but geared towards consumption at the application layer rather than the systems layer.
Benjamin Hindman (@benh), the co-creator of Apache Mesos, developed the open source powerhouse as a Ph.D. student at UC Berkeley before bringing it to Twitter. The software now runs on tens of thousands of machines powering Twitter’s data centers and is often credited for killing the fail whale and providing the scale Twitter needed to serve its growing base of over 300 million users. It’s also causing a huge ground swell in companies developing cloud native applications.
Ben, now founder of Mesosphere, will give the welcome address at MesosCon North America, the Apache Mesos conference going on in Denver on June 1-2. This event is a veritable who’s who from across the industry of those using Mesos as a framework to develop cloud native applications.
MesosCon is a great place to learn about how to design application clusters running on Apache Mesos from engineers who have done it like Craig Neth (@cneth), distinguished member of the technical staff at Verizon, who will walk attendees through how they got a 600 node Mesos cluster powered up and running tasks in 14 days.
Traditionally, machines were statically partitioned across the different services at Uber. In an effort to increase the machine utilization, Uber has recently started transitioning most of its services, including the storage services, to run on top of Apache Mesos.
At MesosCon, Uber engineers will describe the initial experience building and operating a framework for running Cassandra on top of Mesos across multiple data centers at Uber. This framework automates several Cassandra operations such as node repairs, the addition of new nodes, and backup/restore. It improves efficiency by co-locating CPU-intensive services as well as multiple Cassandra nodes on the same Mesos agent. And it handles failure and restart of Mesos agents by using persistent volumes and dynamic reservations.
Running Cassandra on Apache Mesos Across Multiple Datacenters at Uber at MesosCon
Netflix customers worldwide streamed more than forty-two billion hours of content last year. Service-style applications, batch jobs, and stream processing alike, from a variety of use cases across Netflix, rely on executing container-based applications in multi-tenant clusters powered by Apache Mesos and Fenzo, a scheduler Java library for Apache Mesos frameworks. These applications are consuming microservices that allows Netflix to build composable applications at massive scale.
Based on the experiences from Netflix projects Mantis and Titus, Netflix Software Engineer Sharma Podila (@podila) will share his experiences running Docker and Cgroups based containers in a cloud native environment.
Lessons from Netflix Mesos Clusters at Mesoscon.
Dragos Sccalita Haut is a solutions architect at Adobe’s API Platform, adobe.io, building a high scale distributed API Gateway running in the cloud. He realized that as the number of microservices increase and the communication between them becomes more complicated. This brings new questions to light:
How do microservices authenticate?
How do we monitor who’s using the APIs they expose?
How do we protect them from attacks?
How do we set throttling and rate limiting rules across a cluster of microservices?
How do we control which services allow public access and which ones we want to keep private?
How about Mesos APIs and frameworks, can they benefit from these features as well?
The answer to these questions was using the Mesos API management layer to expose microservices in a secure, managed and highly available way.
Let Dragos teach you to Be a Microservices Hero at MesosCon.
If you are interested in hearing how Apache Mesos is being developed and deployed by the world’s most interesting and progressive companies the place to see this is MesosCon on June 1-2, in Denver. The conference will feature two days of sessions to learn more about the Apache Mesos core, an ecosystem developed around the project, and related technologies. The program will include workshops to get started with Apache Mesos, keynote speakers from industry leaders, and sessions led by adopters and contributors.
With so many technologies, tools, and techniques to keep track of, it can be hard to know where to start learning new skills. This series on next-gen cloud technologies aims to help you get up to speed on the important projects and products in emerging and rapidly changing areas such as containers, container orchestration, software-defined networking, and more.
5 Next-Gen Cloud Technologies You Should Know
This article takes a brief look at five next-generation cloud technologies and some of the key open source projects in each space to help you get up to speed with the rapidly changing cloud landscape.
8 Container Orchestration Tools to Know
Container orchestration tools aim to simplify container management and provide a framework for defining and maintaining container deployments for improved availability, scaling, and networking.
5 Container as a Service Tools You Should Know
Container as a Service environments like Kubernetes and Google Container Engine sit between the IaaS and PaaS environments. CaaS tools provide a framework to manage container and application deployment.
4 Container Networking Tools to Know
Innovations in networking allow containers to connect with each other across hosts, and container networking tools can help accomplish the necessary scalability. This article looks at some of the tools available in this emerging area.
You can learn more about container management, software-defined networking, and other next-gen cloud technologies through The Linux Foundation’s free “Cloud Infrastructure Technologies” course — a massively open online course being offered through edX. Registration for this course is open now, and course content will be available in June.
At the recent Embedded Linux Conference and OpenIoT Summit, Mozilla Technical Evangelist Dietrich Ayala proposed a simple and affordable solution to home automation: A discarded smartphone can handle some of the most useful home automation tasks without requiring expensive hubs and sensors — or risking data security in the cloud.
“With a smartphone you can detect motion, sound, presence, and the absence of radio services,” said Ayala in his presentation, “Turning Sensors into Signals: Humanizing IoT with Old Smartphones and the Web.”
“Many phones have proximity or ambient light sensors, orientation, and battery,” continued Ayala. “Consumer devices have almost none of these. My phone knows if it’s being moved around, but my Nest doesn’t.”
Ayala introduced his “Context” JavaScript program for turning an old Firefox OS smartphone into a combination hub and sensor array for remote home monitoring. After Mozilla pivoted the Firefox OS team from phones to IoT, Ayala wondered how he might repurpose all the Firefox OS phones he had sitting around.
Ayala had also been contemplating the shortcomings of the first generation of home automation. “Devices aren’t actually connected today,” said Ayala. “You have to buy into a particular network of devices, and they’re not cheap. Then there are end of life issues, as with Revolv.”
Most commercial IoT products include cloud services for remote connectivity, storage, and in some cases, processing. “How much of your personal life is being exposed to a black box where you have no guarantee or visibility?” said Ayala. “There are no standards, legislation, or case law around what people can do with this data.”
Many hacker-oriented automation products avoid using the cloud, but at the price of greater complexity. You get more privacy and open source personalization, “but at a high cost in experimentation in time and learning,” said Ayala.
A greater challenge affecting commercial and DIY IoT systems alike is the lack of a compelling purpose. “You have to ask yourself, what problems am I addressing?” said Ayala. “Do I really need to have the light reflect my mood or do automated shopping? To me these aren’t solving day to day problems. People have problems like not having enough money or time, or worrying about sick relatives. Maybe they need to know if someone is in their house or whether basic services are working. What you need is physical awareness put in context — the presence or absence of things like noise, motion, or services.”
Most of these capabilities, Ayala realized, are already available on a smartphone even without hooking up additional sensors. You could even repurpose several old phones in a WiFi- or Bluetooth-based sensor network.
Ayala started by writing some JavaScript code to gain access to low level sensors. With Firefox OS, he found he could even avoid building a downloadable app. “With progressive web apps, you can distribute a web page, so users can load it and then receive push notifications forever without loading the page again,” said Ayala. “You don’t even need a UI. You can just reply to the body of channels and configure how much you want to know about a given topic.”
Notifications are currently sent to IFTTT’s maker channel. “From there, I can hook it up to wherever I want,” said Ayala.
Much of the functionality of Ayala’s Firefox OS script can work on other platforms. It would be fairly straightforward to do something similar in a mobile framework, or for more experienced developers, even a native app, Ayala said.
Ayala spent a lot of time studying the readouts from sensors, as well as from the phone’s microphone, camera, and, radios, that would enable a remote user to draw conclusions about what was happening at home. This contextual information could then be codified into more useful notifications.
With ambient light, for example, if it suddenly goes dark in the daytime, maybe someone is standing over a device, explained Ayala. Feedback from the accelerometer can be analyzed to determine the difference between footsteps, an earthquake, or someone picking up the device. Scripts can use radio APIs to determine if a person moving around is carrying a phone with a potentially revealing Bluetooth signature.
With the battery API, you can usually tell if the power went out. If the phone has some battery life and an SMS plan, you can have it send a text message alert.
When sensors hit certain levels, you can have the script use a media API to turn on a camera or mic to see what’s up. In one experiment, Ayala used the getUserMedia API to turn on the mic and record the average volume of ambient sound. “There are some signatures you can get from sound that might yield useful information around presence or absence,” he said.
Future enhancements might tap a mobile platform’s connectivity and discovery interfaces to hook up with other devices. On Firefox OS, these include TCP and UDP sockets, DLNA, and others. Ayala also sees possibilities using local speech recognition APIs.
“In the end, it’s about using the phone as an awareness tool,” said Ayala. “About learning about the environment and yourself.”
Watch the complete presentation below:
This article is sponsored by Mesosphere as a Diamond-level sponsor of MesosCon North America.
In April Mesosphere, along with 60 partners (including Accenture, Autodesk, Cisco, Hewlett-Packard, Yelp and Microsoft), announced the DC/OS project, what has been called the first open and comprehensive platform for building, running and scaling modern enterprise applications. Ahead of MesosCon in Denver next week, we got the chance to catch up with Keith Chambers, product manager at Mesosphere, to learn more about this important open source project that has Apache Mesos at its core.
Keith Chambers: The DC/OS project (dcos.io) is the open source version of our popular Datacenter Operating System technology, which is the simplest way to build, run, scale and manage modern enterprise applications. When we say modern enterprise applications, we’re talking about applications that utilize technologies such as containers, microservices, real-time data processing, distributed databases, and more. Our goal is to make DC/OS the datacenter-scale equivalent of Android, in that it has the potential to democratize the development of application architectures and operational techniques previously reserved to companies such as Google and Facebook.
The DC/OS project is a software platform that’s comprised entirely of open source technologies. It includes some existing technologies like Apache Mesos and Marathon, which were always open source, but also includes newer proprietary components developed by Mesosphere that we’ve donated to the community and which are fully open sourced under an Apache 2.0 license. Features include easy install of DC/OS itself (including all the components), plus push-button, app-store-like installation of complex distributed systems (including Apache Spark, Apache Kafka, Apache Cassandra and more) via our Universe “distributed services app store”. We’re also tightly integrating our popular Marathon container-orchestration technology right into DC/OS, as the default method for managing Docker containers and other long-running services (including traditional non-containerized web applications, as well stateful services such as databases).
Linux.com: How is it an operating system?
Chambers: We call DC/OS an operating system for a number of reasons, including how users go about installing the things they want to run on it. DC/OS abstracts a datacenter full of servers into a single logical computer (i.e., 1,000 dual-core servers become 1 computer with 2,000 cores), which means developers and operators don’t need to worry about individual servers or VMs and can simply tell DC/OS about their task’s or service’s resource requirements. DC/OS is very versatile and can manage a wide diversity of workloads, ranging from Ruby scripts to Microservices in Docker containers to entire database systems. DC/OS makes it easy to run these workloads in an operationally efficient way, maximizing resource utilization and providing automated high-availability. For example. DC/OS will keep all tasks running even when there is a significant hardware failure by restarting failed workloads on different machines, and gracefully failing over stateful applications.
Linux.com: Why did Mesosphere open source DC/OS?
Chambers: As a company, and as individuals, we believe in open source as the best way to drive innovation and adoption. We believe that DC/OS will create a revolution in the way developers build applications and the ways that organizations deploy them, at a scale and velocity that we’ve never seen before. We want to put the power of DC/OS into as many people’s hands as possible. The future of enterprise IT—whether running in the public cloud or a private datacenter—is going to look a lot different than what many of us have been used to. There will be new and exciting licensing models (i.e., open source rather than strictly proprietary) as well as exciting operational and architectural advancements. DC/OS provides an opportunity for savvy organizations to start on that journey today.
Organizations that adopt DC/OS will be more competitive in an increasingly software-driven world. DC/OS helps companies adopt the operational and architectural practices of companies such as Apple, Yelp and Netflix (and, at a broader level, Facebook and Google)—whose infrastructure, and the applications that run on it, are directly tied to their corporate success—without having to reinvent the wheel. The large pioneering companies spent millions of dollars in R&D budget and deployed thousands of highly skilled engineers to build their internal systems, but that’s an unreasonable expectation for most companies. With DC/OS, organizations don’t have to piece together open source technologies or build their own homemade technologies. DC/OS brings this type of advanced datacenter environment to anyone.
We also wanted to encourage our partners and open source contributors to build datacenter services for the DC/OS app ecosystem. The DC/OS app ecosystem allows organizations to adopt and operationalize complex technologies such as Spark, Kafka and Cassandra in minutes, including automating some of the operational tasks and best practices. Open sourcing DC/OS introduces our partners to the broader community of users, and allows them to build applications and services on a platform that is not locked to a single vendor or cloud.
And finally an open source foundation means real portability for workloads between clouds, racks and hybrids. This is dependable OSS. If organizations have to create their own stack and get it running in multiple environments, they’re wasting time they want to be spending on their apps (differentiating their business!). More time to write the apps, less time wasted making the datacenter work.
Linux.com: What do you think will be hot at MesosCon this year?
Chambers: You’re going to see a much broader base of users, in part because of enabling technologies like DC/OS, but you’re also going notice that Apache Mesos is the definitive kernel for distributed systems resource management. Ben Hindman,co-creator of Mesos, is going to show us how far we’ve come towards realizing his original vision of dual-level scheduler powering an operating system for the datacenter, which can be traced all the way back to the Berkeley AMPlab where Mesos was first created.
Whether you’re talking about Mesos as part of the DC/OS and the huge traction it’s receiving as the first operating system for distributed systems, or one-off cluster management solutions built for individual application frameworks — Mesos has become the de facto standard for managing the underlying infrastructure. I think you’re also going to see some major end users on display talking about how Mesos and DC/OS help them operate containers in production.
This article was sponsored by Mesosphere, creators of the world’s first Datacenter Operating System (DCOS). Learn more at: www.mesosphere.io
