The Cloud Native Architect

By

-

August 10, 2017

One of the biggest learning curves I ever endured in that time was working in an operations team building what I will call a virtualisation platform. They called it infrastructure as code, I called it automating previously manual processes using development techniques. It opened my mind again to a completely new way of looking at development teams outside of the DevOps culture. Development techniques were relatively new in that team but the real value was driven through collaborative knowledge share. System administrators were developing code and developers were gaining knowledge of automating and building up the infrastructure & networking of a cloud platform. At this point, this was the first time I had seen this degree of two-way communication outside of teams already building on a PaaS. This truly opened my eyes to the challenges these operations team face when the code is thrown over the fence and they are expected to agree to run it in production with confidence.

Historically the responsibilities of a production system were formed from an aggregated view of collective specialisms. For example: Infrastructure architects, network architects, security architects, QA, developers; technical and solution architects. What I am getting at here is that each role had a more narrow minded focus and set of key responsibilities.

Linux Load Averages: Solving the Mystery

By

Brendan Gregg's Blog

-

August 10, 2017

Load averages are an industry-critical metric – my company spends millions auto-scaling cloud instances based on them and other metrics – but on Linux there’s some mystery around them. Linux load averages track not just runnable tasks, but also tasks in the uninterruptible sleep state. Why? I’ve never seen an explanation. In this post I’ll solve this mystery, and summarize load averages as a reference for everyone trying to interpret them.

Linux load averages are “system load averages” that show the running thread (task) demand on the system as an average number of running plus waiting threads. This measures demand, which can be greater than what the system is currently processing. Most tools show three averages, for 1, 5, and 15 minutes:

$ uptime
 16:48:24 up  4:11,  1 user,  load average: 25.25, 23.40, 23.46

top - 16:48:42 up  4:12,  1 user,  load average: 25.25, 23.14, 23.37

$ cat /proc/loadavg 
25.72 23.19 23.35 42/3411 43603

Some interpretations:

If the averages are 0.0, then your system is idle.
If the 1 minute average is higher than the 5 or 15 minute averages, then load is increasing.
If the 1 minute average is lower than the 5 or 15 minute averages, then load is decreasing.
If they are higher than your CPU count, then you might have a performance problem (it depends).

How to Setup and Configure Hadoop CDH5 on Ubuntu 14.0.4

By

-

August 10, 2017

This document describes how to install and configure a Hadoop cluster on a single node on Ubuntu OS. Single machine Hadoop cluster is also called as Hadoop Pseudo-Distributed Mode. The steps and procedure given in this document to install Hadoop cluster are very simple and to the point, so that you can install Hadoop very easily and within some minutes of time. Once the installation is done you can play with Hadoop and its components like MapReduce for data processing and HDFS for data storage.

Install Hadoop Cluster on a Single Node on Ubuntu OS

1 Recommended Platform

I. Platform Requirements

Operating system: Ubuntu 14.04 or later, other Linux flavors like CentOS, Redhat, etc.
Hadoop: Cloudera Distribution for Apache Hadoop CDH5.x (you can use Apache Hadoop 2.x)

II. Configure & Setup Platform

If you are using Windows/Mac OS you can create virtual machine and install Ubuntu using VMWare Player, alternatively, you can create virtual machine and install Ubuntu using Oracle Virtual Box

2. Prerequisites

I. Install Java 8

a. Install Python Software Properties

To add the java repositories we need to download python-software-properties. To download and install python software properties run below command in terminal:

1	`$ sudo apt-get install python-software-properties`

NOTE: After you press “Enter”. It will ask for your password since we are using “sudo” command to provide root privileges for the installation. For any installation or configuration, we always need root privileges.

b. Add Repository

Now we will add a repository manually from where Ubuntu will install the Java. To add repository type the below command in terminal:

1	`$ sudo add-apt-repository ppa:webupd8team/java`

Now it will ask you to Press [Enter] to continue. Press “Enter”.

c. Update the source list

It is recommended to update the source list periodically. If you want to update, install a new package, always update the source list. The source list is a location from where Ubuntu can download and install the software. To update source list type the below command in terminal:

1	`$ sudo apt-get update`

When you run the above command Ubuntu updates its source list.

d. Install Java

Now we will download and install the Java. To download and install Java type the below command in terminal:

1	`$ sudo apt-get install oracle-java8-installer`

When you will press enter it will start downloading and installing Java.

To confirm Java installation has successfully completed or not and to check the version of your Java type the below command in terminal:

1	`$ java –version`

II. Configure SSH

SSH means secured shell which is used for the remote login. We can login to a remote machine using SSH. Now we need to configure password less SSH. Password-less SSH means without a password we can login to a remote machine. Password-less SSH setup is required for remote script invocation. Automatically remotely master will start the demons on slaves.

A Realistic Approach to Mixing Open Source Licenses

By

Swapnil Bhartiya

-

August 9, 2017

At the upcoming Open Source Summit in Los Angeles, Lars Kurth, director of Open Source Solutions at Citrix and chair of the Advisory Board of the Xen Project at The Linux Foundation, will be delivering a wealth of practical advice in two conference talks.

The first talk is “Mixed License FOSS Projects: Unintended Consequences, Worked Examples, Best Practices” and the second talk is “Live Patching, Virtual Machine Introspection and Vulnerability Management: A Primer and Practical Guide.”

Here, Kurth explains more about what he will be covering in these presentations.

There are thousands of open source licenses, and some are incompatible with each other. What are the issues about mixing licenses that you will be discussing in your talk?

Lars Kurth: License compatibility in general is fairly well understood. One of the areas I am focusing on in this talk, is that lots of open source projects start off as single license projects and over time additional licenses start to be incorporated into the codebase. This is true for many open source projects such as the Linux Kernel, QEMU, the Xen Project, the BSDs, and many others.

What most projects fail to do as new licenses are added is evolve best practice around mixing licenses as new licenses are added. This can lead to barriers for license and patent conscious adopters or contributors of a specific project.

For example, in the Xen Project, we came across the case where an organization’s IP lawyer wanted to know answers to questions like why certain licenses existed in the codebase, before they would approve code contributions by their employees. The whole process took more than half a year and was extremely painful, but it prompted us to introduce a set of Best Practices, so we would never have to go through such an exercise again.

Are there any examples of mixing licenses that can lead to disaster?

Kurth: Thanks to FOSS license compliance tools, such as FOSSology, we rarely see FOSS stacks that contain incompatible licenses. What does happen, occasionally though, is that licensing constraints can limit the ambition of open source projects. A high-profile example was Hyperledger’s attempt to include Ethereum’s C++ client, which ultimately failed because it would have required re-licensing some of Ethereum’s runtime from GPL-3 to Apache 2.0.

In projects that contain components with multiple licenses, choosing a component’s license without appropriate foresight can lead to situations that requires re-licensing a component to implement a new feature. The risk of this happening is high, when code refactoring is required. The only way to avoid this is to treat the license of a component like an architectural system property. Because re-licensing a component in projects with multiple licenses can’t be excluded, I will walk through a worked example on how to do this in my talk.

As Open Source is taking over the world, what risks do you see of mixing licenses? What else can people look forward to in your talk?

Kurth: This talk will highlight some risks and solutions to the problem of adding additional licenses as projects grow. One thing that I also think be really interesting is the risk of mixing GPLv2 code with GPLv2 or later code (GPLv2+). Linux for example contains 14 percent of code licensed under GPLv2+. Many projects unintentionally or even unknowingly do this without understanding potential consequences. To find out more, come and see the talk!

Let’s focus a bit on your other presentation. Why is live patching important?

Kurth: The importance of live patching of the Xen Project Hypervisor came to light when several cloud providers including AWS, Rackspace, and IBM SoftLayer had to reboot large numbers of servers in their fleets in late 2014 and again early in 2015 due to two Xen Project vulnerabilities. Applying security patches is not an easy task when thousands of servers running critical business applications require a reboot after a patch has been applied.

Hypervisor reboots inconvenience cloud customers, which is why the Xen Project developed Hypervisor Live Patching — first released in June 2016. This enables Xen users to apply security patches in real-time, without reboots, power cycles, or workload migrations.

What will you be talking about in regard to live patching?

Kurth: The process of patching a live hypervisor or kernel is not an easy task. What happens is a little bit like open heart surgery: The patient is the hypervisor and/or kernel and precision and care are needed to get things right. To do this safely requires expertise and appropriate build and test capabilities. In this talk, I will cover how Hypervisor Live Patching works, and how Live Patches are built and tested. I will also show how XenServer combines Linux and Xen Project Hypervisor Live Patching in a combined and easy-to use solution.

Besides Live Patching, we will also give a brief overview of the Xen Project Vulnerability Management process and how it was impacted by the introduction of Live Patching. In addition, we will briefly introduce Virtual Machine Introspection, which has been shown to detect and protect users from 0-day vulnerabilities such as EternalBlue.

Check out the full schedule for Open Source Summit here. Linux.com readers save on registration with discount code LINUXRD5. Register now!

Running LinuxKit on AWS Platform Made Easy

By

Collabnix

-

August 9, 2017

Soon after DockerCon 2017, I wrote a blog post on how to get started with LinuxKit for Google Cloud Platform. Since then I have been closely keeping eye on the latest features, enablements & releases of LinuxKit. Under this blog post, I bring up a simplified approach to get LinuxKit OS instance running on top of Amazon Web Services(AWS) Platform.

Here we go..

Steps:

Install AWS CLI on macOS(Using Homebrew)
Installing LinuxKit & Moby Tool(Using Homebrew)
Configuring AWS S3 bucket
Building a RAW image with Moby tool
Configuring VM Import Service Role
Upload the aws.raw Image to remote AWS S3 bucket using LinuxKit
Run the LinuxKit OS as EC2 Instance

MQTT for IoT Communication

By

D Zone

-

August 9, 2017

MQTT stands for Message Queue Telemetry Transport. As its name suggests, it’s a protocol for transporting messages between two points. Sure, we’ve got Messenger and Skype for that; but what makes MQTT so special is its super lightweight architecture, which is ideal for scenarios where bandwidth is not optimal.

The MQTT high-level architecture is primarily divided into two parts – a broker and a client.

Agile2017: What the Agile Development Model Needs To Do Next

By

Tech Target

-

August 9, 2017

It’s more than 16 years old now, but Agile still struggles to achieve broad enterprise adoption. Here’s what Agile2017 speakers and attendees are suggesting for the future. More than 16 years after the Agile Manifesto was written, “Agile is still hard,” admitted Tricia Broderick, the chair of Agile2017 in Orlando, Fla.

Just released data from a survey of more than 150 managers by CA Technologies underscores that fact — only 12% say their entire organization is on a path to achieving an Agile development model, even while 70% say they know it’s the process that can help them be organized and respond faster.

Container Networking Challenges the Focus of Tigera Calico Update

By

SDx Central

-

August 9, 2017

Tigera is adding new features to its Calico container networking product in an attempt to ease Kubernetes-based management and hit enterprise-grade needs.

The boldly named Essentials for Kubernetes product is the firm’s first commercial packaged platform. The product is specifically targeted at management of the container networking space, which includes a set of interfaces for adding and removing containers from a network.

Tigera is targeting a handful of connectivity platforms, including Container Networking Interface (CNI), its own Calico offer, Flannel, and Istio. CNI was initially proposed by CoreOS to define a common interface between network plugins and container execution. It has limited responsibility over network connectivity of containers, and it removes allocated resources when the container is deleted.

Future Proof Your SysAdmin Career: Locking Down Security

By

Sam Dean

-

August 8, 2017

For today’s system administrators, gaining competencies that move them up the technology stack and broaden their skillsets is increasingly important. However, core skills like networking remain just as crucial. Previously in this series, we’ve provided an overview of essentials and looked at evolving network skills. In this part, we focus on another core skill: security.

With ever more impactful security threats emerging, the demand for fluency with network security tools and practices is increasing for sysadmins. That means understanding everything from the Open Systems Interconnect (OSI) model to devices and protocols that facilitate communication across a network.

Locking down systems also means understanding the infrastructure of a network, which may or may not be Linux-based. In fact, many of today’s sysadmins serve heterogeneous technology environments where multiple operating systems are running. Securing a network requires competency with routers, firewalls, VPNs, end-user systems, server security, and virtual machines.

Securing systems and networks calls for varying skillsets depending on platform infrastructure, as is clear if you spend just a few minutes perusing, say, a Fedora security guide or the Securing Debian Manual. However, there are good resources that sysadmins can leverage to learn fundamental security skills.

For example, The Linux Foundation has published a Linux workstation security checklist that covers a lot of good ground. It’s aimed at sysadmins and includes discussion of tools that can thwart attacks. These include SecureBoot and Trusted Platform Module (TPM). For Linux sysadmins, the checklist is comprehensive.

The widespread use of cloud platforms such as OpenStack is also introducing new requirements for sysadmins. According to The Linux Foundation’s Guide to the Open Cloud: “Security is still a top concern among companies considering moving workloads to the public cloud, according to Gartner, despite a strong track record of security and increased transparency from cloud providers. Rather, security is still an issue largely due to companies’ inexperience and improper use of cloud services,” and a sysadmin with deeply entrenched cloud skills can be a valuable asset.

Most operating systems and widely used Linux distributions feature timely and trusted security updates, and part of a good sysadmin’s job is to keep up with these. Many organizations and administrators shun spin-off and “community rebuilt” platform infrastructure tools because they don’t have the same level of trusted updating.

Network challenges

Networks, of course, present their own security challenges. The smallest holes in implementation of routers, firewalls, VPNs, and virtual machines can leave room for big security problems. Most organizations are strategic about combating malware, viruses, denial-of-service attacks, and other types of hacks, and good sysadmins should study the tools deployed.

Freely available security and monitoring tools can also go a long way toward avoiding problems. Here are a few good tools for sysadmins to know about:

Wireshark, a packet analyzer for sysadmins
KeePass Password Safe, a free open source password manager
Malwarebytes, a free anti-malware and antivirus tool
NMAP, a powerful security scanner
NIKTO, an open source web server scanner
Ansible, a tool for automating secure IT provisioning
Metasploit, a tool for understanding attack vectors and doing penetration testing

For a lot of these tools, sysadmins can pick up skills by leveraging free online tutorials. For example, there is a whole tutorial series for Metasploit, and there are video tutorials for Wireshark.

Also on the topic of free resources, we’ve previously covered a free ebook from the editors at The New Stack called Networking, Security & Storage with Docker & Containers. It covers the latest approaches to secure container networking, as well as native efforts by Docker to create efficient and secure networking practices. The ebook is loaded with best practices for locking down security at scale.

Training and certification, of course, can make a huge difference for sysadmins as we discussed in “7 Steps to Start Your Linux Sysadmin Career.”

For Linux-focused sysadmins, The Linux Foundation’s Linux Security Fundamentals (LFS216) is a great online course for gaining well-rounded skills. The class starts with an overview of security and covers how security affects everyone in the chain of development, implementation, and administration, as well as end users. The self-paced course covers a wide range of Linux distributions, so you can apply the concepts across distributions. The Foundation offers other training and certification options, several of which cover security topics. For example, LFS201 Essentials of Linux System Administration includes security training.

Also note that CompTIA Linux+ incorporates security into training options, as does the Linux Professional Institute. Technology vendors offer some good choices as well; for example, Red Hat offers sysadmin training options that incorporate security fundamentals. Meanwhile, Mirantis offers three-day “bootcamp” training options that can help sysadmins keep an OpenStack deployment secure and optimized.

In the 2016 Linux Foundation/Dice Open Source Jobs Report, 48 percent of respondents reported that they are actively looking for sysadmins. Job postings abound on online recruitment sites, and online forums remain a good way for sysadmins to learn from each other and discover job prospects. So the market remains healthy, but the key for sysadmins is to gain differentiated types of skillsets. Mastering hardened security is surely a differentiator, and so is moving up the technology stack — which we will cover in upcoming articles.

Learn more about essential sysadmin skills: Download the Future Proof Your SysAdmin Career ebook now.

Future Proof Your SysAdmin Career: New Networking Essentials

Future Proof Your SysAdmin Career: Locking Down Security

Future Proof Your SysAdmin Career: Looking to the Cloud

Future Proof Your SysAdmin Career: Configuration and Automation

Future Proof Your SysAdmin Career: Embracing DevOps

Future Proof Your SysAdmin Career: Getting Certified

Future Proof Your SysAdmin Career: Communication and Collaboration

Future Proof Your SysAdmin Career: Advancing with Open Source

The Rise of Test Impact Analysis

By

Martin Fowler.com

-

August 8, 2017

Test Impact Analysis (TIA) is a modern way of speeding up the test automation phase of a build. It works by analyzing the call-graph of the source code to work out which tests should be run after a change to production code. Microsoft has done some extensive work on this approach, but it’s also possible for development teams to implement something useful quite cheaply.

One curse of modern software development is having “too many” tests to run all of them prior to check-in. When that becomes true, developers use a costly coping strategy of not running any tests on their local developer workstation. Instead they rely on tests running later on an integration server. And quite often even those fall into disrepair, which is inevitable when shift right becomes normal for a dev team.

Of course, everything that you test pre-integrate should immediately be tested post-integrate in the Continuous Integration (CI) infrastructure. Even the highest functioning development teams might experience breakages born from timing alone for commits landing in real time.