Home Blog Page 418

CNCF to Host the Rook Project to Further Cloud-Native Storage Capabilities

Today, the Cloud Native Computing Foundation (CNCF) Technical Oversight Committee (TOC) voted to accept Rook as the 15th hosted project alongside Kubernetes, Prometheus, OpenTracing, Fluentd, Linkerd, gRPC, CoreDNS, containerd, rkt, CNI, Envoy, Jaeger, Notary and TUF.

Rook has been accepted as an inception-level project, under the CNCF Graduation Criteria v1.0. The CNCF provides every project an associated maturity level of either inception, incubating or graduated. At a minimum, an inception-level project is required to add value to cloud native computing and be aligned with the CNCF charter.

Rook brings File, Block and Object storage systems into the Kubernetes cluster, running them seamlessly alongside other applications and services that are consuming the storage. By doing so, the cloud-native cluster becomes self-sufficient and portable across public cloud and on-premise deployments. The project has been developed to enable organizations to modernize their data centers with dynamic application orchestration for distributed storage systems running in on-premise and public cloud environments.

“Storage is one of the most important components of cloud native computing, yet persistent storage systems typically run outside the cloud native environments today,” said Chris Aniszczyk, COO of Cloud Native Computing Foundation. “Rook was one of the early adopters of the Kubernetes operator pattern and we’re excited to bring in Rook as an inception level project to advance the state of cloud native storage.”

Instead of building an entirely new storage system which requires many years to mature, Rook focuses on turning existing battle-tested storage systems like Ceph into a set of cloud-native services that run seamlessly on-top of Kubernetes. Rook integrates deeply into Kubernetes providing a seamless experience for security, policies, quotas, lifecycle management, and resource management.

In this Software Engineering Daily podcast, Bassam Tabbara, CEO of Upbound and creator of Rook, said: “Rook is essentially using the operator pattern to extend Kubernetes to support storage systems. We’ve added a concept of a storage cluster, a storage pool, an object store and a file system. Those are all new abstractions that we’ve used to extend Kubernetes”

An alpha version of Rook (release 0.6) is available now, followed by a beta and production ready versions in the first half of 2018.

Main features:

  • Software-defined storage running on commodity hardware
  • File, block and object storage presentations integrated with Ceph
  • Hyper-scale or hyper-converged storage options
  • Elastic storage that can easily scale up or down
  • Zero-touch management
  • Integrated data protection with snapshot, cloning and versioning
  • Deployable on Kubernetes.

The latest release of Kubernetes 1.9 introduced a CSI alpha implementation that makes installing new volume plugins as easy as deploying a pod, and enables third-party storage providers to develop their solutions without adding to the core Kubernetes codebase. Rook will expose storage through CSI to Kubernetes.

“It’s a natural fit to run a storage cluster on Kubernetes. It makes perfect sense to bring it into the fold and keep the unified management interface,” said Dan Kerns, Senior Director at Quantum, the initial sponsor of the Rook project. “With Rook, we wanted to create a software-defined storage cluster that could run really well in modern cloud-native environments, and the storage cluster becomes even more resilient with an orchestrator like Kubernetes.”

Community support for Rook is growing rapidly as companies and users deploy Rook in their cloud-native environments (on-premise and public cloud). Companies and organizations like HBO, UCSD Nautilus Project, Norwegian Welfare, Verne Global, FlexShopper, and Acaleph have implemented Rook as part of their storage platforms.

Notable Milestones:

  • 47 contributors
  • 1,935 GitHub stars
  • 13 releases
  • 1,463 commits
  • 1.25M+ container downloads

“We used Rook underneath our Prometheus servers at HBO, running on Kubernetes and deployed on AWS,” said Illya Chekrygin, former senior staff engineer at HBO and founding member of Upbound. “Rook made a significant improvement on the Prometheus pod restart time, virtually eliminating downtime and metrics scrape gaps. We are looking forward to Rook being in a production ready state.”

As a CNCF hosted project, Rook will be part of a neutral foundation aligned with technical interests, receive help with project governance and be provided marketing support to reach a wider audience.

“Operating storage in cloud-native environments is a significantly more difficult task than stateless containers,” said Benjamin Hindman, co-founder of Mesosphere and CNCF TOC representative and project sponsor. “We’re thrilled to have Rook as the first CNCF inception project that begins to address the difficult problem of storage orchestration.”

For more read the Rook blog, Quantum’s recent announcement on the momentum of the project, Upbound’s blog, and listen to The New Stack’s Makers Podcast or Software Engineering Daily featuring Bassam Tabbara discussing Rook and Storage on Kubernetes.

This article originally appeared at Cloud Native Computing Foundation.

What Happens When You Want to Create a Special File with All Special Characters in Linux?

I recently joined Holberton School as a student, hoping to learn full-stack software development. What I did not expect was that in two weeks I would be pretty much proficient with creating shell scripts that would make my coding life easy and fast!

So what is the post about? It is about a novel problem that my peers and I faced when we were asked to create a file with no regular alphabets/ numbers but instead special characters!! Just to give you a look at what kind of file name we were dealing with —

*\’”Holberton School”’\*$?*****:)

What a novel file name! Of course, this question was met with the collective groaning and long drawn sighs of all 55 (batch #5) students!

1*Lf_XPhmgm-RB5ipX_lBjsQ.gif

Some proceeded to make their lives easier by breaking the file name into pieces on a doc file and adding in the “\” or “” in front of certain special character which kind of resulted in this format –

\*\\’”Holberton School”\’\\*$\?\*\*\*\*\*:)

1*p6s8WlysClalj0x2fQhGOg.gif

Everyone trying to get the \ right

bamboozled? me, too! I did not want to believe that this was the only way to solve this, as I was getting frustrated with every “\” or “” that was required to escape and print those special characters as normal characters!

If you’re new to shell scripting, here is a quick walk through on why so many “\” , “” were required and where.

In shell scripting “ ” and ‘ ’ have special usage and once you understand and remember when and where to use them it can make your life easier!

Double Quoting

The first type of quoting we will look at is double quotes. If you place text inside double quotes, all the special characters used by the shell lose their special meaning and are treated as ordinary characters. The exceptions are “$”, “” (backslash), and “`” (back- quote). This means that word-splitting, pathname expansion, tilde expansion, and brace expansion are suppressed, but parameter expansion, arithmetic expansion, and command substitution are still carried out. Using double quotes, we can cope with filenames containing embedded spaces.

So this means that you can create file with names that have spaces between words — if that is your thing, but I would suggest you to not do that as it is inconvenient and rather an unpleasant experience for you to try to find that file when you need !

Quoting “THE” guide for linux I follow and read like it is the Harry Potter of the linux coding world —

Say you were the unfortunate victim of a file called two words.txt. If you tried to use this on the command line, word-splitting would cause this to be treated as two separate arguments rather than the desired single argument:

[me@linuxbox me]$ ls -l two words.txt

ls: cannot access two: No such file or directory
ls: cannot access words.txt: No such file or directory

By using double quotes, you can stop the word-splitting and get the desired result; further, you can even repair the damage:

[me@linuxbox me]$ ls -l “two words.txt”
-rw-rw-r — 1 me me 18 2008–02–20 13:03 two words.txt
[me@linuxbox me]$ mv “two words.txt” two_words.t

There! Now we don’t have to keep typing those pesky double quotes.

Now, let us talk about single quotes and what is their significance in shell —

Single Quotes

Enclosing characters in single quotes (‘’) preserves the literal value of each character within the quotes. A single quote may not occur between single quotes, even when preceded by a backslash.

Yes! that got me and I was wondering how will I use it, apparently when I was googling to find and easier way to do it I stumbled across this piece of information on the internet —

Strong quoting

Strong quoting is very easy to explain:

Inside a single-quoted string nothing is interpreted, except the single-quote that closes the string.

echo 'Your PATH is: $PATH'

$PATH won’t be expanded, it’s interpreted as ordinary text because it’s surrounded by strong quotes.

In practice that means to produce a text like Here's my test… as a single-quoted string, you have to leave and re-enter the single quoting to get the character “'” as literal text:

# WRONG
echo 'Here's my test...'
# RIGHT
echo 'Here'''s my test...'
# ALTERNATIVE: It's also possible to mix-and-match quotes for readability:
echo "Here's my test"

Well now you’re wondering — “well that explains the quotes but what about the “”??”

So for certain characters we need a special way to escape those pesky “” we saw in that file name.

Escaping Characters

Sometimes you only want to quote a single character. To do this, you can precede a character with a backslash, which in this context is called the escape character. Often this is done inside double quotes to selectively prevent an expansion:

[me@linuxbox me]$ echo “The balance for user $USER is: $5.00”
The balance for user me is: $5.00

It is also common to use escaping to eliminate the special meaning of a character in a filename. For example, it is possible to use characters in filenames that normally have special meaning to the shell. These would include “$”, “!”, “&”, “ “, and others. To include a special character in a filename you can to this:

[me@linuxbox me]$ mv bad&filename good_filename

To allow a backslash character to appear, escape it by typing “\”. Note that within single quotes, the backslash loses its special meaning and is treated as an ordinary character.

Looking at the filename now we can understand better as to why the “\” were used in front of all those “”s.

So, to print the file name without losing “” and other special characters what others did was to suppress the “” with “\” and to print the single quotes there are a few ways you can do that.

1. echo $'It's Shell Programming'  # ksh, bash, and zsh only, does not expand variables
2. echo "It's Shell Programming"   # all shells, expands variables
3. echo 'It'''s Shell Programming' # all shells, single quote is outside the quotes
4. echo 'It'"'"'s Shell Programming' # all shells, single quote is inside double quotes
for further reading please follow this link

Looking at option 3, I realized this would mean that I would only need to use “” and single quotes at certain places to be able to write the whole file without getting frustrated with “\” placements.

So with the hope in mind and lesser trial and errors I was actually able to print out the file name like this:

‘*\’’’”Holberton School”’’’\*$?*****:)’

to understand better I have added an “a” instead of my single quotes so that the file name and process becomes more clearer. For a better understanding, I’ll break them down into modules:

1*hP1gmzbn7G7gUEhoynj1ew.gif

a*\a ’ a”Holberton School”a ’ a\*$?*****:)a

Module 1 — a*\a

Here the use of single quote (a) creates a safe suppression for *\ and as mentioned before in strong quoting, the only way we can print the ‘ is to leave and re-enter the single quoting to get the character.

Module 2 , 4— ’

The suppresses the single quote as a standalone module.

Module 3 — a”Holberton School”a

Here the use of single quote (a) creates a safe suppression for double quotes and along with regular text.

Module 5 — a\*$?*****:)a

Here the use of single quote (a) creates a safe suppression for all special characters being used such as *, , $, ?, : and ).

so in the end I was able to be lazy and maintain my sanity, and got away with only using single quotes to create small modules and “” in certain places.

1*rO34jp-bYSkCnHSdwoO3qQ.gif

And, that is how I was able to get the file to work right! After a few misses, it felt amazing and it was great to learn a new way to do things!

1*PE9_VtcfGGQjnYMwJ8YB1A.gif

Handled that curve-ball pretty well! Hope this helps you in the future when, someday you might need to create a special file for a special reason in shell!

Mitali Sengupta is a former digital marketing professional, currently enrolled as a full-stack engineering student at Holberton School. She is passionate about innovation in AI and Blockchain technologies.. You can contact Mitali on TwitterLinkedIn or GitHub.

7 Ways to Automate Kubernetes at Scale in Production

The Kubernetes open source container orchestration engine is not a management platform, nor should it be mistaken for one. The whole point of orchestration is to reliably enable an automated system to facilitate the deployment and management of applications at scale, without the need for human intervention at each and every step. If the tools you use with and for Kubernetes don’t enable automation, then you’re not truly taking advantage of the benefits of orchestration.

To that end, here are seven ways you can and should be automating your Kubernetes cluster in production.

1) Logging

Any Kubernetes production environment will rely heavily on logs.

Read more at The New Stack

A Step-by-Step Guide to Git

If you’ve never used Git, you may be nervous about it. There’s nothing to worry about—just follow along with this step-by-step getting-started guide, and you will soon have a new Git repository hosted on GitHub.

Before we dive in, let’s clear up a common misconception: Git isn’t the same thing as GitHub. Git is a version-control system (i.e., a piece of software) that helps you keep track of your computer programs and files and the changes that are made to them over time. It also allows you to collaborate with your peers on a program, code, or file. GitHub and similar services (including GitLab and BitBucket) are websites that host a Git server program to hold your code.

Read more at OpenSource.com

The Effects of the Spectre and Meltdown Vulnerabilities

Spectre and Meltdown aren’t anomalies. They represent a new area to look for vulnerabilities and a new avenue of attack. They’re the future of security­ — and it doesn’t look good for the defenders.

Modern computers do lots of things at the same time. Your computer and your phone simultaneously run several applications — ­or apps. Your browser has several windows open. A cloud computer runs applications for many different computers. All of those applications need to be isolated from each other. For security, one application isn’t supposed to be able to peek at what another one is doing, except in very controlled circumstances. Otherwise, a malicious advertisement on a website you’re visiting could eavesdrop on your banking details, or the cloud service purchased by some foreign intelligence organization could eavesdrop on every other cloud customer, and so on. The companies that write browsers, operating systems, and cloud infrastructure spend a lot of time making sure this isolation works.

Read more at Schneier on Security

Employers Want JavaScript, but Developers Want Python

When it comes to which programming languages are in demand by employers, JavaScript, Java, Python, C++, and C—in that order—came out on top in a recent developer survey. Developers, however, want to learn languages like PythonGo, and Kotlin.

A survey of developers by technical recruiter HackerRank, conducted in October, found no gap between languages employers want and what developers actually knowwith JavaScript barely edging out Java. But as far as which languages developers prefer, Python is the language developers most want to learn—and many already know it, HackerRank found.

Read more at InfoWorld

Introducing RLlib: A Composable and Scalable Reinforcement Learning Library

In a previous post, I outlined emerging applications of reinforcement learning (RL) in industry. I began by listing a few challenges facing anyone wanting to apply RL, including the need for large amounts of data, and the difficulty of reproducing research results and deriving the error estimates needed for mission-critical applications. Nevertheless, the success of RL in certain domains has been the subject of much media coverage. This has sparked interest, and companies are beginning to explore some of the use cases and applications I described in my earlier post. Many tasks and professions, including software development, are poised to incorporate some forms of AI-powered automation. In this post, I’ll describe how RISE Lab’s Ray platform continues to mature and evolve just as companies are examining use cases for RL.

Read more at O’Reilly

This Week in Open Source News: The Linux Foundation Launches Networking Fund

This week in open source/Linux news, The Linux Foundation announced a restructuring of their networking projects under one umbrella, Slack launches on Linux, and more!

1) The Linux Foundation consolidated its networking project under one umbrella this week.

Linux Foundation Re-Orgs to Simplify Open Source Networking– LightReading

Linux Foundation Seeks to Bring Rhyme and Reason to Open-Source Networking Projects– ZDNet

2) While the Ubuntu app is still in beta, Canonical has brought Slack to its Linux platform.

Slack Launches On Linux– ITPro

3) “CNCF will be key to ensuring inter-operability of services across different vendors’ platforms.”

What Does DevOps Do in 2018– infosecurity Magazine

4) Hyperledger has set in motion plans to give select startups access to some of the benefits accessed only by companies that are officially recognized.”

Hyperledger Creates Experimental Labs For Startups– Toinnov

Subgraph: This Security-Focused Distro Is Malware’s Worst Nightmare

By design, Linux is a very secure operating system. In fact, after 20 years of usage, I have personally experienced only one instance where a Linux machine was compromised. That instance was a server hit with a rootkit. On the desktop side, I’ve yet to experience an attack of any kind.
That doesn’t mean exploits and attacks on the Linux platform don’t exist. They do. One only need consider Heartbleed and Wannacry, to remember that Linux is not invincible.

See: Linux Malware on the Rise: A Look at Recent Threats

With the Linux desktop popularity on the rise, you can be sure desktop malware and ransomware attacks will also be on the increase. That means Linux users, who have for years ignored such threats, should begin considering that their platform of choice could get hit.

What do you do?

If you’re a Linux desktop user, you might think about adopting a distribution like Subgraph. Subgraph is a desktop computing and communication platform designed to be highly resistant to network-borne exploits and malware/ransomware attacks. But unlike other platforms that might attempt to achieve such lofty goals, Subgraph makes this all possible, while retaining a high-level of user-friendliness. Thanks to the GNOME desktop, Subgraph is incredibly easy to use.

What Subgraph does differently

It all begins at the core of the OS. Subgraph ships with a kernel built with grsecurity/PaX (a system-wide patch for exploit and privilege escalation mitigation), and RAP (designed to prevent code-reuse attacks on the kernel to mitigate against contemporary exploitation techniques). For more information about the Subgraph kernel, check out the Subgraph kernel configs on GitHub.

Subgraph also runs exposed and vulnerable applications within unique environments, known as Oz. Oz is designed to isolate applications from one another and only grant resources to applications that need them. The technologies that make up Oz include:

Other security features include:

  • Most of the custom Subgraph code is written in the memory-safe language, Golang.

  • AppArmor profiles that cover many system utilities and applications.

  • Security event monitor.

  • Desktop notifications (coming soon).

  • Roflcoptor tor control port filter service.

Installing Subgraph

It is important to remember that Subgraph is in alpha release, so you shouldn’t consider this platform as a daily driver. Because it’s in alpha, there are some interesting hiccups regarding the installation. The first oddity I experienced is that Subgraph cannot be installed as a VirtualBox virtual machine. No matter what you do, it will not work. This is a known bug and, hopefully, the developers will get it worked out.

The second issue is that installing Subgraph by way of a USB device is very tricky. You cannot use tools like Unetbootin or Multiboot USB to create a bootable flash drive. You can use GNOME Disks to create a USB drive, but your best bet is the dd command. Download the ISO image, insert your USB drive into the computer, open a terminal window, and locate the name of the newly inserted USB device (the command lsblk works fine for this. Finally, write the ISO image to the USB device with the command:

dd bs=4M if=subgraph-os-alpha_XXX.iso of=/dev/SDX status=progress && sync

where XXX is the Subgraph release number and SDX is the name of your USB device.

Once the above command completes, you can reboot your machine and install Subgraph. The installation process is fairly straightforward, with a few exceptions. The first is that the installation completely erases the entire drive, before it installs. This is a security measure and cannot be avoided. This process takes quite some time (Figure 1), so let it do its thing and go take care of another task.

Figure 1: The Subgraph installation includes erasing your drive.

Next, you must create a passphrase for the encryption of the drive (Figure 2).

Figure 2: Creating a disk encryption passphrase.

This passphrase is used when booting your device. If you lose (or forget) the passphrase, you won’t be able to boot into Subgraph. This passphrase is also the first line of defence against anyone who might try to get to your data, should they steal your device… so choose wisely.

The last difference between Subgraph and most other distributions, is that you aren’t given the opportunity to create a username. You do create a user password, which is used for the default user… named user. You can always create a new user (once the OS is installed), either by way of the command line or the GNOME Settings tool.

Once installed, your Subgraph system will reboot and you’ll be prompted for the disk encryption passphrase. Upon successful authentication, Subgraph will boot and land on the GNOME login screen. Login with username user and the password you created during installation.

Usage

There are two important things to remember when using Subgraph. First, as I mentioned earlier, this distribution is in alpha development, so things will go wrong. Second, all applications are run within sandboxes and networking is handled through Tor, so you’re going to experience slower application launches and network connections than you might be used to.

I was surprised to find that Tor Browser (the default—and only installed—browser) wasn’t installed out of the box. Instead, there’s a launcher on the GNOME Dash that will, upon first launch, download the latest version. That’s all fine and good, but the download and install failed on me twice. Had I been working through a regular network connection, this wouldn’t have been such a headache. However, as Subgraph was working through Tor, my network connection was painfully slow, so the download, verification, and install of Tor Browser (a 26.8 MB package) took about 20 minutes. That, of course, isn’t the fault of Subgraph but of the Tor network to which I was connected. Until Tor Browser was up and running, Subgraph was quite limited in what I could actually do. Eventually, Tor Browser downloaded and all worked as expected.

Application sandboxes

Not every application has to go through the process of downloading a new version upon first launch. In fact, Tor Browser was the only application I encountered that did. When you do open up a new application, it will first start its own sandbox and then open the application in question. Once the application is up and running, you will see a drop-down in the top panel that lists each current application sandbox (Figure 3).

Figure 3: The LibreOffice application sandbox is up and running, while Tor Browser continues to download.

From each application sub-menu, you can add files to that particular sandbox or you can shutdown the sandbox. Shutting down the sandbox effectively closes the application. This is not how you should close the application itself. Instead, close the application as you normally would and then, if you’re done working with the application, you can then manually close the sandbox (through the drop-down). If you have, say, LibreOffice open and you close it by way of closing the sandbox, you run the risk of losing information.

Because each application starts up in its own sandbox, applications don’t open as quickly as they would otherwise. This is the tradeoff you make for using Subgraph and sandboxes. For those looking to get the most out of desktop security, this is a worthwhile exchange.

A very promising distribution

For anyone hoping to gain the most security they can on a desktop computer, Subgraph is one seriously promising distribution. Although it does suffer from many an alpha woe, Subgraph looks like it could make some serious waves on the desktop—especially considering how prevalent malware and ransomware has become. Even better, Subgraph could easily become a security-focused desktop distribution that anyone (regardless of competency) could make use of. Once Subgraph is out of alpha, I predict big things from this unique flavor of Linux.

Learn more about Linux through the free “Introduction to Linux” course from The Linux Foundation and edX.

System Startup Gets a Boost with New LinuxBoot Project

The Linux Foundation is pleased to welcome LinuxBoot to our family of open source projects and to support the growth of the project community. LinuxBoot looks to improve system boot performance and reliability by replacing some firmware functionality with a Linux kernel and runtime…

LinuxBoot addresses the often slow, often error-prone, obscured code that executes these steps with a Linux kernel. The result is a system that boots in a fraction of the time of a typical system, and with greater reliability.

Read more at The Linux Foundation