Home Blog Page 282

A Hitchhiker’s Guide to the Blockchain Universe

Despite the significant potential of blockchain, it is also difficult to find a consistent description of what it really is. A Google search for “blockchain technical papers” returns nothing but white papers for the first three screens; not a single paper is peer-reviewed.10 One of the best discussions of the technology itself is from the National Institute of Standards and Technology, but at 50-plus pages, it is a bit much for a quick read.9

The purpose of this article is to look at the basics of blockchain: the individual components, how those components fit together, and what changes might be made to solve some of the problems with blockchain technology. This technology is far from monolithic; some of the techniques can be used (at surprising savings of resources and effort) if other parts are cut away.

Because there is no single set of technical specifications, some systems that claim to be blockchain instances will differ from the system described here. Much of this description is taken from the original blockchain paper.6 While details may differ, the main ideas stay the same. …

While there are lots of different ways to implement a blockchain, all have three major components. The first of these is the ledger, which is the series of blocks that are the public record of the transactions and the order of those transactions. Second is the consensus protocol, which allows all of the members of the community to agree on the values stored in the ledger. Finally, there is the digital currency, which acts as a reward for those willing to do the work of advancing the ledger. These components work together to provide a system that has the properties of stability, irrefutability, and distribution of trust that are the goals of the system.

Read more at ACM Queue

Docker and Kubernetes in High Security Environments

This is brief summary of parts of my master’s thesis and the conclusions to draw from it. This medium-story focuses on containerized application isolation. The thesis also covers segmentation of cluster networks in Kubernetes which is not discussed in this story.

Container orchestration and cloud-native computing has gained lots of traction the recent years. The adoption has increased to such level that even enterprises in finance, banking and the public sector are interested. Compared to other businesses they differ by having extensive requirements on information security and IT security.

One important aspect is how containers could be used in production environments while maintaining system separation between applications. As such enterprises uses private clouds powered by bare-metal virtualization, the separation loss upon migrating to a container orchestrated environment is not negligible. It is in this scope that my thesis is written –with the Swedish Police Authority as the target client.

The specific research question that the thesis explores is the following:

How can Docker and Kubernetes support the separation of applications for the Swedish Police Authority compared with virtual machines powered by the bare-metal hypervisor ESXi?

That question has a lot to unwrap. To break this down, let’s start by looking in to the common denominator — the applications.

Read more at Medium

Future-Proof Your Career with AI

AI is the fastest growing field in enterprise tech. Here’s how to get an AI job you will love.

AI job listings have become the fastest growing category on LinkedIn, and Indeed is packed with listings. But most job requisitions seek a computer scientist type with a PhD in neural networks or some other years-long study. The trick is to look past those, and you’ll find that what many companies need can’t be outsourced or given to a freshly minted college grad: an IT pro with enterprise-scale experience who also knows how to deliver on a machine learning project.

Machine learning is where the jobs are

Here’s the secret: There are plenty of AI-related jobs that aren’t advanced science but simply applying new machine learning features from cloud services giants to familiar IT environments. “Most ML jobs aren’t about advancing ML technology and algorithms,” says Ross Mead, founder and CEO of robotics software startup Semio and an industry consultant in AI with a PhD from the University of Southern California. “The money in AI for most companies is using ML for better business intelligence.” 

That means using turnkey ML packages to analyze internal data—customer behavior, sales, etc.—to look for patterns that indicate likely business success. Machine learning is different from deep learning, the more esoteric field of AI that it is often confused with. 

Read more at HPE

Remote Code Execution in apt/apt-get

tl;dr I found a vulnerability in apt that allows a network man-in-the-middle (or a malicious package mirror) to execute arbitrary code as root on a machine installing any package. The bug has been fixed in the latest versions of apt. If you’re worried about being exploited during the update process, you can protect yourself by disabling HTTP redirects while you update. To do that, run:

$ sudo apt update -o Acquire::http::AllowRedirect=false
$ sudo apt upgrade -o Acquire::http::AllowRedirect=false

If your current package mirrors redirect by default (meaning you can’t update apt when using that flag) you’ll need to pick different mirrors or download the package directly. Specific instructions for upgrading on Debian can be found hereUbuntu’s announcement can be found here.

As a proof of concept, below is a video of me exploiting the following Dockerfile:

FROM debian:latest

RUN apt-get update && apt-get install -y cowsay

Read more at Max Justicz

SAP: One of Open Source’s Best Kept Secrets

SAP has been working with open source for decades and has now established an open source program office (OSPO) to further formalize the coordination of its open source activities and expand its engagement with the open source communities. “SAP was one of the first industry players to formally define processes for open source consumption and contribution,” says Peter Giese, director of the Open Source Program Office.

Even so, many people do not yet consider SAP to be a company that embraces open source engagement and contributions.

“In the past, we may not have been active enough in sharing our open source activities,” says Giese.

Now, SAP is shining a spotlight on its work in open source. Transparency is an essential part of the new open source mandate, beginning with an explanation of what the company has been up to and where it is headed with open source.

How SAP came to adopt open source

“In 1998, SAP started to port the R/3 system, our market-leading ERP system, to Linux,” says Giese. “That was an important milestone for establishing Linux in the enterprise software market.”

Porting a system to Linux was just a first step, and a successful one. The action spurred an internal discussion and exploration of how and where to adopt Linux going forward.

Read more at The Linux Foundation

Container Storage Interface (CSI) for Kubernetes GA

The Kubernetes implementation of the Container Storage Interface (CSI) has been promoted to GA in the Kubernetes v1.13 release. Support for CSI was introduced as alpha in Kubernetes v1.9 release, and promoted to beta in the Kubernetes v1.10 release.

The GA milestone indicates that Kubernetes users may depend on the feature and its API without fear of backwards incompatible changes in future causing regressions. GA features are protected by the Kubernetes deprecation policy.

Why CSI?

Although prior to CSI Kubernetes provided a powerful volume plugin system, it was challenging to add support for new volume plugins to Kubernetes: volume plugins were “in-tree” meaning their code was part of the core Kubernetes code and shipped with the core Kubernetes binaries—vendors wanting to add support for their storage system to Kubernetes (or even fix a bug in an existing volume plugin) were forced to align with the Kubernetes release process. In addition, third-party storage code caused reliability and security issues in core Kubernetes binaries and the code was often difficult (and in some cases impossible) for Kubernetes maintainers to test and maintain.

CSI was developed as a standard for exposing arbitrary block and file storage storage systems to containerized workloads on Container Orchestration Systems (COs) like Kubernetes. With the adoption of the Container Storage Interface, the Kubernetes volume layer becomes truly extensible. Using CSI, third-party storage providers can write and deploy plugins exposing new storage systems in Kubernetes without ever having to touch the core Kubernetes code. This gives Kubernetes users more options for storage and makes the system more secure and reliable.

Read more at Kubernetes Blog

Using more to View Text Files at the Linux Command Line

There are a number of utilities that enable you to view text files when you’re at the command line. One of them is more.

more is similar to another tool I wrote about called less. The main difference is that more only allows you to move forward in a file.

While that may seem limiting, it has some useful features that are good to know about. Let’s take a quick look at what more can do and how to use it.

The basics

Let’s say you have a text file and want to read it at the command line. Just open the terminal, pop into the directory that contains the file, and type this command:

more <filename>

Read more at OpenSource.com

How Companies Are Building Sustainable AI and ML Initiatives

In 2017, we published “How Companies Are Putting AI to Work Through Deep Learning,” a report based on a survey we ran aiming to help leaders better understand how organizations are applying AI through deep learning. We found companies were planning to use deep learning over the next 12-18 months. In 2018, we decided to run a follow-up survey to determine whether companies’ machine learning (ML) and AI initiatives are sustainable—the results of which are in our recently published report, “Evolving Data Infrastructure.”

The current generation of AI and ML methods and technologies rely on large amounts of data—specifically, labeled training data. In order to have a longstanding AI and ML practice, companies need to have data infrastructure in place to collect, transform, store, and manage data. On one hand, we wanted to see whether companies were building out key components. On the other hand, we wanted to measure the sophistication of their use of these components. In other words, could we see a roadmap for transitioning from legacy cases (perhaps some business intelligence) toward data science practices, and from there into the tooling required for more substantial AI adoption?

Here are some notable findings from the survey:

  • Companies are serious about machine learning and AI. Fifty-eight percent of respondents indicated that they were either building or evaluating data science platform solutions. Data science (or machine learning) platforms are essential for companies that are keen on growing their data science teams and machine learning capabilities.

Read more at O’Reilly

More About Angle Brackets in Bash

In the previous article, we introduced the subject of angle brackets (< >) and demonstrated some of their uses. Here, we’ll look at the topic from a few more angles. Let’s dive right in.

You can use < to trick a tool into believing the output of a command is data from a file.

Let’s say you are not sure your backup is complete, and you want to check that a certain directory contains all the files copied over from the original. You can try this:

diff <(ls /original/dir/) <(ls /backup/dir/)

diff is a tool that typically compares two text files line by line, looking for differences. Here it gets the output from two ls commands and treats them as if coming from a file and compares them as such.

Note that there is no space between the < and the (...).

Running that on the original and backup of a directory where I save pretty pictures, I get:

diff <(ls /My/Pictures/) <(ls /My/backup/Pictures/) 5d4 < Dv7bIIeUUAAD1Fc.jpg:large.jpg

The < in the output is telling me that there is file (Dv7bIIeUUAAD1Fc.jpg:large.jpg) on the left side of the comparison (in /My/Pictures) that is not on the right side of the comparison (in /My/backup/Pictures), which means copying over has failed for some reason. If diff didn’t cough up any output, it would mean that the list of files were the same.

So, you may be wondering, if you can take the output of a command or command line, make it look like the contents of a file, and feed it to an instruction that is expecting a file, that means that in the sorting by favorite actor example from above, you could’ve done away with the intermediate file and just piped the output from the loop into sort.

In short, yep! The line:

sort -r <(while read -r name surname films;do echo $films $name $surname ; done < CBactors)

does the trick nicely.

Here string! Good string!

There is one more case for redirecting data using angle brackets (or arrows, or whatever you want to call them).

You may be familiar with the practice of passing variables to commands using echo and a pipe (|). Say you want to convert a variable containing a string to uppercase characters because… I don’t know… YOU LIKE SHOUTING A LOT. You could do this:

myvar="Hello World" echo $myvar | tr '[:lower:]' '[:upper:]' HELLO WORLD

The tr command translates strings to different formats. In the example above, you are telling tr to change all the lowercase characters that come along in the string to uppercase characters.

It is important to know that you are not passing on the variable, but only its contents, that is, the string “Hello World“. This is called the here string, as in “it is here, in this context, that we know what string we are dealing with“. But there is shorter, clearer, and all round better way of delivering here strings to commands. Using

tr '[:lower:]' '[:upper:]' <<< $myvar

does the same thing with no need to use echo or a pipe. It also uses angle brackets, which is the whole obsessive point of this article.

Conclusion

Again, Bash proves to give you lots of options with very little. I mean, who would’ve thunk that you could do so much with two simple characters like < and >?

The thing is we aren’t done. There are plenty of more characters that bring meaning to chains of Bash instructions. Without some background, they can make shell commands look like gibberish. Hopefully, post by post, we can help you decipher them. Until next time!

9 Trends to Watch in Systems Engineering and Operations

If your job or business relies on systems engineering and operations, be sure to keep an eye on the following trends in the months ahead.

AIOps

Artificial intelligence for IT operations (AIOps) will allow for improved software delivery pipelines in 2019. This practice incorporates machine learning in order to make sense of data and keep engineers informed about both patterns and problems so they can address them swiftly. Rather than replace current approaches, however, the goal of AIOps is to enhance these processes by consolidating, automating, and updating them. A related innovation, Robotic Process Automation (RPA), presents options for task automation and is expected to see rapid and substantial growth as well.

Knative vs. AWS Lambda vs. Microsoft Azure Functions vs. Google Cloud

The serverless craze is in full swing, and shows no signs of stopping—since December 2017 alone, the technology has grown 22%, and Gartner reports that by 2020, more than 20% of global enterprises will be deploying serverless. This is a huge projected increase from the mere 5% that are currently utilizing it. The advantages of serverless are numerous…

Read more at O’Reilly