Ahead of the official Mesa 12.0 debut later this month, Mesa release manager Emil Velikov of Collabora has announced 12.0 Release Candidate 2…
Read more at Phoronix
3 Emerging Open Source Data Analytics Tools Beyond Apache Spark
On the data analytics front, profound change is in the air, and open source tools are leading many of the changes. Sure, you are probably familiar with some of the open source stars in this space, such as Hadoop and Apache Spark, but there is now a strong need for new tools that can holistically round out the data analytics ecosystem. Notably, many of these tools are customized to process streaming data.
The Internet of Things (IoT), which is giving rise to sensors and other devices that produce continuous streams of data, is just one of the big trends driving the need for new analytics tools. Streaming data analytics are needed for improved drug discovery, and NASA and the SETI Institute are even collaborating to analyze terabytes of complex, streaming deep space radio signals.
While Apache Spark grabs many of the headlines in the data analytics space, given billions of development dollars thrown at it by IBM and other companies, several unsung open source projects are also on the rise. Here are three emerging data analytics tools worth exploring:
Grappa
Big organizations and small ones are working on new ways to cull meaningful insights from streaming data, and many of them are working with data generated on clusters and, increasingly, on commodity hardware. That puts a premium on affordable data-centric approaches that can improve on the performance and functionality of tools such as MapReduce and even Spark. Enter the open source Grappa project, which scales data-intensive applications on commodity clusters and offers a new type of abstraction that can beat classic distributed shared memory (DSM) systems.
As the developers note: “Grappa provides abstraction at a level high enough to subsume many performance optimizations common to data-intensive platforms. However, its relatively low-level interface provides a convenient abstraction for building data-intensive frameworks on top of. Prototype implementations of (simplified) MapReduce, GraphLab, and a relational query engine have been built on Grappa that out-perform the original systems.”
Grappa is freely available on GitHub under a BSD license. If you are interested in seeing Grappa at work, you can follow easy quick-start directions in the application’s README file to build and run it on a cluster. To learn how to write your own Grappa applications, check out the tutorial.
Apache Drill
The Apache Drill project is making such a difference in the Big Data space that companies such as MapR have even wrapped it into their Hadoop distributions. It is a Top-Level project at Apache and is being leveraged along with Apache Spark in many streaming data scenarios.
Drill is notable in streaming data applications because it is a distributed, schema-free SQL engine. DevOps and IT staff can use Drill to interactively explore data in Hadoop and other NoSQL databases, such as HBase and MongoDB. There is no need to explicitly define and maintain schemas, as Drill can automatically leverage the structure that’s embedded in the data. It is able to stream data in memory between operators, and minimizes the use of disks unless needed to complete a query.
Apache Kafka
The Apache Kafka project has emerged as a star for real-time data tracking capabilities. It provides unified, high-throughput, low-latency processing for real-time data. Confluent and other organizations have also produced custom tools for using Kafka with data streams.
The engineers who created Kafka at LinkedIn also created Confluent, which focuses on Kafka. Confluent University offers training courses for Kafka developers, and for operators/administrators. Both onsite and public courses are available.
Are you interested in more unsung open source data analytics projects on the rise? If so, you can find more in my recent post on the topic.
Quieting Scary Web Browser SSL Alerts
The entire Internet depends on OpenSSL to secure sensitive transactions, but until the Linux Foundation launched the Core Infrastructure Initiative to support crucial infrastructure projects, it was supported by a small underfunded team with only one paid developer. This is not good for an essential bit of infrastructure, and OpenSSL was hit by some high-profile bugs, such as Heartbleed. Now that OpenSSL has stable support, there should be fewer such incidents — and not a moment too soon, because we have no alternatives.
How OpenSSL Works
Asymmetric encryption is an ingenious mechanism for establishing encrypted sessions without first exchanging encryption keys. Asymmetric encryption relies on public-private key pairs. Public keys encrypt, private keys decrypt. Or, you can think of it as public keys lock, and private keys unlock. Anyone who has your public key can encrypt messages to send to you, and only you can decrypt them with your private key. It’s a brilliantly simple concept that greatly simplifies the process of establishing encrypted network connections.
Symmetric encryption uses the same key for encryption and decryption. You have to figure out how to safely distribute the key, and anyone with a copy of the key can decrypt your communications. The advantage of symmetric encryption is that it’s computationally less expensive than asymmetric encryption. OpenSSL takes advantage of this by establishing a session with asymmetric encryption, and then generating symmetric encryption keys to use for the duration of the session.
Certificate Authority
A Certificate Authority (CA) is the final stop in the public key infrastructure chain. The CA tells your website visitors that your site’s SSL certificate is legitimate. Obviously, this requires a high level of trust, which is why there are a number of commercial certificate authorities, such as Comodo, GlobalSign, GoDaddy, and many others.
All web browsers include a bundle of trusted root CAs. Mozilla publishes a list of included CAs. On Ubuntu systems, these are stored in /usr/share/ca-certificates/mozilla/ and symlinked to /etc/ssl/certs/. Any website that you visit that is trusted by these root CAs will display a happy little green padlock (Figure 1), and when you click the padlock, you’ll find all kinds of information about the site as well as a copy of the site certificate.
Figure 2 shows a product page on Amazon. The page is not SSL-protected. Why not encrypt all pages? On a site with all content hosted on the same server, it’s easy. There is a small performance hit, but it shouldn’t be noticeable. On a complex site that uses all kinds of external content and ad servers, the process becomes unmanageable. Various page elements are coming from many different domains, so it’s very difficult to set up a CA for all of them. Most sites don’t even try and focus instead on securing their login and checkout pages.
It’s worth using sitewide SSL even on a site that isn’t selling anything, such as a blog, because it assures your site visitors that they are visiting your site and not some fraudulent copycat.
Self-Signed Certificates
Self-Signed Certificates are the reliable old standby for LAN services. However, your web browsers are still going to pitch fits and report your internal sites as dangerous. I have mixed feelings about web browsers trying to protect us. On one hand, it’s a nice idea. It’s all the same to us: we’re staring at a screen and have no idea what’s going on behind the scenes. On the other hand, a steady diet of alarms doesn’t help — how are we supposed to judge if a warning is legitimate?
Figure 3 shows a typical Firefox freakout. You can look at the site certificate and make an educated guess. Most site visitors will do as they do with all computer warnings: Ignore them and forge ahead.
In the privacy of your own local network, you can shut up browsers permanently by importing your site certificates into your web browsers. In Firefox, you can do this by clicking the Advanced button and keep clicking through until Firefox gives up trying to scare you away and imports the certificate.
You can also import site certificates from the command line. Your certificate must be an X.509 .pem file. These are plain-text files that you can open and read; they start with —–BEGIN CERTIFICATE—–. On Ubuntu, copy your site certificate into /usr/local/share/ca-certificates/. Then run the CA updater:
$ sudo update-ca-certificates
Updating certificates in /etc/ssl/certs...
WARNING: Skipping duplicate certificate Go_Daddy_Class_2_CA.pem
WARNING: Skipping duplicate certificate Go_Daddy_Class_2_CA.pem
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
Done.
If you’re using a cool configuration management tool like Puppet, Chef, or Ansible, you can roll your certs out to everyone in your shop.
Let’s Encrypt
If you use a commercial certificate authority, you can avoid all this. There are many to choose from, so shop around for a good deal. There is also a good free option: Let’s Encrypt. Let’s Encrypt offers both production and test certificates, so you can test all you want to until you get the hang of it. Let’s Encrypt has broad industry support, including Mozilla, Cisco, Facebook, the Electronic Frontier Foundation, and many more.
Resources
Ubuntu how-to on creating SSL certificates