MariaDB has released into general availability ColumnStore 1.0, a storage engine allows users to run analytics and transactional processes simultaneously with a single front end on the MariaDB 10.1 database.
ColumnStore is a fork of InnoDB database engine, the default storage engine for MySQL. Despite the popularity of SQL alternatives, the company notes that every OLTP (online transaction processing) and analytics solution has been building a SQL layer.
If you run a mainstream distribution of Linux on a desktop computer, there’s a good chance security researcher Chris Evans can hijack it when you do nothing more than open or even browse a specially crafted music file. And in the event you’re running Chrome on the just-released Fedora 25, his code-execution attack works as a classic drive-by.
The zero-day exploits, which Evans published on Tuesday, are the latest to challenge the popular conceit that Linux, at least in its desktop form, is more immune to the types of attacks that have felled Windows computers for more than a decade and have increasingly snared Macs in recent years.
It’s been two years since The Linux Foundation forged a partnership with the Apache Software Foundation to become the producer of their official ASF events. This year, ApacheCon and Apache Big Data continued to grow and gain momentum as the place to share knowledge, ideas, best practices and creativity with the rest of the Apache open source community.
As 2016 draws to a close, we looked back at some of the highlights from ApacheCon and Apache Big Data and collected the 7 videos from our most-read articles about the events in 2016.
These videos help highlight the good work the open source community accomplished for and with Apache projects this year. We hope they inspire you to participate in the community and present your work again at ApacheCon and Apache Big Data, May 16-18, 2017 in Miami. The deadline to submit proposals is February 11!
When IBM got involved with the Linux open source project in 1998, they were betting that giving their code and time to the community would be a worthwhile investment. Now, 18 years later, IBM is more involved than ever, with more than 62,000 employees trained and expected to contribute to open source projects, according to Todd Moore, Vice President of Open Technology at IBM, speaking at ApacheCon in Vancouver in May.
“It became apparent that open source could be the de facto standards we needed to be the engine to go out and drive things,” Moore said in his keynote. “[The contributions] were bets; we didn’t know how this was going to come out, and we didn’t know if open source would grow, we knew there would be roadblocks and things we’d have to overcome along the way, but it had promise. We thought this would be the way of the future.”
Moore reiterated IBM’s commitment to open source, highlighting projects born at IBM’s developerWorks Open (dWOpen), such as SystemML, Toree, and Quarks, and now in the Apache Incubator.
2. Open Source is a Positive-Sum Game, Sam Ramji, Cloud Foundry
As open source software matures and is used by more and more major corporations, it is becoming clear that the enterprise software game has changed. Sam Ramji, CEO of the Cloud Foundry Foundation, believes that open source software is a positive sum game, as reflected in his ApacheCon keynote.
Invoking his love of game theory, Ramji stated emphatically that open source software is a positive-sum game, where the more contributors there are to the common good, the more good there is for everyone. This idea is the opposite of a zero-sum game, where if someone benefits or wins, then another person must suffer, or lose.
3. Apache Milagro: A New Security System for the Future of the Web
With 25 billion new devices set to hit the Internet by 2025, the need for a better worldwide cryptosystem for securing information is paramount. That’s why the Apache Milagro project is currently incubating at the Apache Software Foundation. It’s a collaboration between MIRACL and Nippon Telegram and Telegraph (NTT), and Brian Spector, MIRACL CEO and Co-Founder, discussed the project in his keynote at ApacheCon in May.
Spector said the project was born in a bar on the back of a napkin after a brainstorm about how one would rebuild Internet security from the ground up. That sounds like a lot of work, but Spector believes it’s absolutely necessary: the future of the Web is going to be very different from the past.
4. Netflix Uses Open Source Tools for Global Content Expansion
“We measured, we learned, we innovated, and we grew.”
Brian Sullivan, Director of Streaming Data Engineering & Analytics at Netflix, recited this recipe for the streaming video giant’s success several times during his keynote address at the Apache Big Data conference in Vancouver today. It was this mantra, combined with an open source toolkit, that took the stand-alone streaming product from a tiny test launch in Canada to making Netflix a global presence.
5. Spark 2.0 Is Faster, Easier for App Development, and Tackles Streaming Data
It only makes sense that as the community of Spark contributors got bigger the project would get even more ambitious. Spark 2.0 came out with three robust new features, according to Ion Stoica, the founder of Databricks.
“Spark 2.0 is about taking what has worked and what we have learned from the users and making it even better,” Stoica said.
6. IBM Uses Apache Spark Across Its Products to Help Enterprise Customers
IBM is invested in Spark’s machine-learning capabilities and is contributing back to the project with its work on SystemML, which helps create iterative machine-learning algorithms. The company offers Spark-as-a-service in the cloud, and it’s building it into the next iteration of the Watson analytics platform. Basically anywhere it can, IBM is harnessing the efficient power of Apache Spark.
“We at IBM … have noted the power of Spark, and the other big data technologies that are coming in [from the Apache Software Foundation],” said Luciano Resende, an architect at IBM’s Spark Technology Center.
7. How eBay Uses Apache Software to Reach Its Big Data Goals
eBay’s ecommerce platform creates a huge amount of data. It has more than 800 million active listings, with 8.8 million new listings each week. There are 162 million active buyers, and 25 million sellers.
“The data is the most important asset that we have,” said Seshu Adunuthula, eBay’s head of analytics infrastructure, during a keynote at Apache Big Data in Vancouver in May. “We don’t have inventory like other ecommerce platforms, what we’re doing is connecting buyers and sellers, and data plays an integral role into how we go about doing this.”
About five years ago, eBay made the conscious choice to go all-in with open source software to build its big data platform and to contribute back to the projects as the platform took shape.
Share your knowledge and best practices on the technologies and projects driving the future of open source. Submit a speaking proposal for ApacheCon and Apache Big Data today!
Not interested in speaking but want to attend? Linux.com readers can register now for ApacheCon or Apache: Big Data with the discount code, LINUXRD5, for 5% off the registration price.
The task of securing Linux systems is so mind-bogglingly complex and involves so many layers of technology that it can easily overwhelm developers. However, there are some fairly straightforward protections you can use at the very core: the kernel. These hardening techniques help developers guard against the bugs that haven’t yet been detected.
“Hardening is about making bugs more difficult to exploit,” explained Mark Rutland, a kernel developer at ARM Ltd, at the recent Embedded Linux Conference Europe 2016 in Berlin. There will always be dangerous bugs that manage to evade the notice of kernel developers, he added. “We do not yet know which particular bugs exist in the next kernel, and we probably won’t for five years,” he said, referring to Kees Cook’s recent analysis of kernel bug lifetimes.
“We see recurring classes of bugs involving things like dereferencing of null pointers or accessing memory controlled by user space, so we can assume that some of the bugs we don’t know about will fall into these buckets.”
Rutland, who earlier this year warned ELC North America attendees about the hidden dangers of unruly caches, noted that bugs are an unavoidable offshoot of programming. Most are relatively benign, but many cause problems, and some can open dangerous vulnerabilities.
“In the kernel 4.8 merge window we fixed over 500 bugs, many of which were in 4.7 or earlier,” said Rutland, noting that while varied techniques are used today to avoid bugs making it into the kernel, some will inevitably slip through and require later fix-ups. This is “slightly terrifying” given the long lifetime of bugs, which might not be discovered until affected devices are end-of-life.
Some of these bugs have significant security implications. Fortunately, kernel developers have in recent years begun to create hardening features that protect against many of the most common bugs. Rutland implored the audience to make use of these hardening features, noting that many are simple to enable, and their protections are “effectively free,” yet don’t see widespread use.
Rutland went on to discuss several of the main classes of hardening protections that pose the least amount of overhead. Here are some edited quotes about each:
Strict kernel memory permissions – “Historically, the kernel has mapped all memory as readable, writable, and executable…which leads to…being able to modify kernel code or const data, or executing data, all of which…are very useful primitives if you’re an attacker. We can get the MMU to enforce these permissions by…mapping that code as read only or mapping constant data as read only and non- executable. If it’s done in the MMU, it’s effectively free, as the hardware is handling it for us.” (For details, study up on CONFIG_DEBUG_RODATA and CONFIG_DEBUG_SET-MODULE_RONX.)
Stack smashing protection – “Stack smashing attacks work on the principle that stacks contain a return address and other data, as well as local variables. On most architectures, the stack grows downwards, and the buffers grow upwards. If you copy some data to a buffer on the stack, and the data is too large to fit in the buffer, you end up overwriting subsequent data, which happens to include the return address. So if an attacker knows what your stack frame layout will look like, they can control where you will return to, and…branch to any code of their choosing…to launch more advanced attacks. Stack smashing protection guards against this by having the compiler insert a secret value known as a canary between the data and the flow control information.” (For details, see CONFIG_STACKPROTECTOR_REGULAR and CONFIG_STACKPROTECTOR_STRONG.)
User/kernel memory segregation – “Typically the kernel shares an address space with user space in hardware. A pointer can encode an address to either space…using the same load and store instructions. If you accidentally dereference an address…controlled by user space, the hardware won’t notice and will happily give you the value, so if an attacker can convince you to dereference the address…it can be used as the basis for a number of attacks. If an attacker puts a buffer of code in a user space address and then uses a stack smashing exploit to branch to that, they can do whatever they want. The MMU can help by letting us change the page table dynamically, which we use to switch processes. Enabling access temporarily and then disabling access…will catch most of these unintentional user memory accesses or branches.”
Stricter permissions– “Some hardware can…automatically prevent arbitrary code execution from a user space buffer. On ARM we have a feature called privilege execute never (PXN), which…says I never want this page to be executed with kernel privileges. x86 has a similar thing called SMEP. An attacker can still branch to another piece of kernel code, so it doesn’t prevent arbitrary execution, but it limits one case. More recently, MMUs have become able to do this with data accesses as well.”
Rutland advised that it will be “years before we have a reasonable number of protections.” He also noted that the protections are not 100 percent effective, and that “we still have to find and fix bugs.”
In Rutland’s view, Linux systems would be more secure if more of these hardening features were turned on by default, which he said happened in kernel 4.9 with now mandatory, aforementioned DEBUG_RODATA. “Resistance is slowly going away for some of these protections,” he said. “Lots of the complaints about the features not looking like kernel code and doing things wrong are being solved quite quickly. People’s opinions about mainline are changing – there’s agreement that yes, we need to do something here.”
Watch the full video of Rutland’s presentation, “Thwarting Unknown Bugs: Hardening Features in the Mainline Linux Kernel” below:
Embedded Linux Conference + OpenIoT Summit North America will be held on February 21 – 23, 2017 in Portland, Oregon. Check out over 130 sessions on the Linux kernel, embedded development & systems, and the latest on the open Internet of Things.
Linux.com readers can register now with the discount code, LINUXRD5, for 5% off the attendee registration price. Register now>>
Welcome back, me hearty Linux syadmins! In part 1 and part 2 of this series, we learned to how to put Postfix and Dovecot together to make a nice IMAP and POP3 mail server. Now we will learn to make virtual users so that we can manage all of our users in Dovecot.
Sorry, No SSL. Yet.
I know I promised to show you how to set up a proper SSL-protected server. Unfortunately, I underestimated how large that topic is. So, I will realio trulio write a comprehensive how-to by next month.
For today, in this final part of this series, we’ll go into detail on how to set up virtual users and mailboxes in Dovecot and Postfix. It’s a bit weird to wrap your mind around, so the following examples are as simple as I can make them. We’ll use plain flat files and plain-text authentication. You have the options of using database back ends and nice strong forms of encrypted authentication; see the links at the end for more information on these.
Virtual Users
You want virtual users on your email server and not Linux system users. Using Linux system users does not scale, and it exposes their logins, and your Linux server, to unnecessary risk. Setting up virtual users requires editing configuration files in both Postfix and Dovecot. We’ll start with Postfix. First, we’ll start with a clean, simplified /etc/postfix/main.cf. Move your original main.cf out of the way and create a new clean one with these contents:
$ sudo postfix reload
[sudo] password for carla:
postfix/postfix-script: refreshing the Postfix mail system
Dovecot Virtual Users
We’ll use Dovecot’s lmtp protocol to connect it to Postfix. You probably need to install it:
$ sudo apt-get install dovecot-lmtpd
The last line in our example main.cf references lmtp. Copy this example /etc/dovecot/dovecot.conf, replacing your existing file. Again, we are using just this single file, rather than calling the files in /etc/dovecot/conf.d.
At last, you can create the file that holds your users and passwords, /etc/dovecot/passwd. For simple plain text authorization we need only our users’ full email addresses and passwords:
The Dovecot virtual users are independent of the Postfix virtual users, so you will manage your users in Dovecot. Save all of your changes and restart Postfix and Dovecot:
$ sudo service postfix restart
$ sudo service dovecot restart
Now let’s use good old telnet to see if Dovecot is set up correctly.
$ telnet studio 110
Trying 127.0.1.1...
Connected to studio.
Escape character is '^]'.
+OK Dovecot ready.
user molly@studio
+OK
pass password
+OK Logged in.
quit
+OK Logging out.
Connection closed by foreign host.
So far so good! Now let’s send some test messages to our users with the mail command. Make sure to use the whole user’s email address and not just the username.
$ mail benny@studio
Subject: hello and welcome!
Please enjoy your new mail account!
.
The period on the last line sends your message. Let’s see if it landed in the correct mailbox.
$ sudo ls -al /home/vmail/studio/benny@studio/.Mail/new
total 16
drwx------ 2 vmail vmail 4096 Dec 14 12:39 .
drwx------ 5 vmail vmail 4096 Dec 14 12:39 ..
-rw------- 1 vmail vmail 525 Dec 14 12:39 1481747995.M696591P5790.studio,S=525,W=540
And there it is. It is a plain text file that we can read:
$ less 1481747995.M696591P5790.studio,S=525,W=540
Return-Path: <carla@localhost>
Delivered-To: benny@studio
Received: from localhost
by studio (Dovecot) with LMTP id V01ZKRuuUVieFgAABiesew
for <benny@studio>; Wed, 14 Dec 2016 12:39:55 -0800
Received: by localhost (Postfix, from userid 1000)
id 9FD9CA1F58; Wed, 14 Dec 2016 12:39:55 -0800 (PST)
Date: Wed, 14 Dec 2016 12:39:55 -0800
To: benny@studio
Subject: hello and welcome!
User-Agent: s-nail v14.8.6
Message-Id: <20161214203955.9FD9CA1F58@localhost>
From: carla@localhost (carla)
Please enjoy your new mail account!
You could also use telnet for testing, as in the previous segments of this series, and set up accounts in your favorite mail client, such as Thunderbird, Claws-Mail, or KMail.
Troubleshooting
When things don’t work, check your logfiles (see the configuration examples), and run journalctl -xe. This should give you all the information you need to spot typos, uninstalled packages, and nice search terms for Google.
What Next?
Assuming your LAN name services are correctly configured, you now have a nice usable LAN mail server. Obviously, sending messages in plain text is not optimal, and an absolute no-no for Internet mail. See Dovecot SSL configuration and Postfix TLS Support. VirtualUserFlatFilesPostfix covers TLS and database back ends. And watch for my upcoming SSL how-to. Really.
Docker announced today that it was open sourcing containerd (pronounced Container D), making a key infrastructure piece of its container platform available for anyone to work on.
Containerd, which acts as the core container runtime engine, is a component within Docker that provides “users with an open, stable and extensible base for building non-Docker products and container solutions,” according to the company. Leading cloud providers have signed on to work on it including Alibaba, AWS, Google, IBM and Microsoft.
For a company like Slack that strives to be as data-driven as possible, understanding how our users use our product is essential.
The Data Engineering team at Slack works to provide an ecosystem to help people in the company quickly and easily answer questions about usage, so they can make better and data informed decisions: “Based on a team’s activity within its first week, what is the probability that it will upgrade to a paid team?” or “What is the performance impact of the newest release of the desktop app?”
The Dream
We knew when we started building this system that we would need flexibility in choosing the tools to process and analyze our data. Sometimes the questions being asked involve a small amount of data and we want a fast, interactive way to explore the results. Other times we are running large aggregations across longer time series and we need a system that can handle the sheer quantity of data and help distribute the computation across a cluster. Each of our tools would be optimized for a specific use case, and they all needed to work together as an integrated system.
Because the Linux kernel community celebrated a quarter-century of development in 2016, many people have asked us the secret to the project’s longevity and success. I usually laugh and joke that we really have no idea how we got here. The project has faced many disagreements and challenges along the way. But seriously, the reason we’ve made it this far has a lot to do with the community’s capacity for introspection and change.
About 16 years ago, most of the kernel developers had never met each other in person—we’d only ever interacted over email—and so Ted T’so came up with the idea of a Kernel Summit. Now every year kernel developers make a point to gather in person to work out technical issues and, crucially, to review what we did right and what we did wrong over the past year.
If you’ve encountered test-driven development (TDD), you may have encountered programmers who follow it with almost religious fervor. They will tell you that you must always write unit tests before you write code, no exceptions. If you don’t, your code will be condemned to everlasting brokenness, tortured by evil edge cases for all eternity.
This is an example of a common problem in programming: good advice by experts that gets turned into a counter-productive religion. Test-driven development is useful and worth doing… some of the time, but not always. And the experts who came up with it in the first place will be the first to tell you that.
Containers are a big deal, and they’re only going to get bigger. That’s my view after attending the latest KubeCon (and CloudNativeCon) in Seattle last week.
A year ago, I was confused about what containers mean for IT, because the name ‘container’ had me thinking it was about the little box that code was stored in: the container image. I’m here to tell you that the container image format itself (Docker, rkt, whatever you like) is not the point.
The most important thing about containers is the process of using them, not the things themselves. The process is heavily automated. No more installing software by sitting in front of a console and clicking ‘Next’ every five minutes. Unix people everywhere rejoice that Windows folk have discovered scripting is a good thing.
