In a previous article I’ve announced my sponsorship project, where I offered to help a motivated young Linux Professional getting certified. I found an ideal candidate, and he has taken the RHCSA exam, and now we’re ready to take the next step.
Santos Chibenga from Zambia is so engaged in the local Linux community in Zambia that we decided to host an event together: https://www.vieo.tv/event/linux-event-lusaka-zambia. In this event we will have local speakers, and I will educate nearly 200 participants to become LFCS certified.
We do have some very cool sponsors, including Linux Foundation who has contributed 100 exam vouchers for the LFCS exam. I’m very excited about that, because that means that at the end of the event we’re able to leave something real in Zambia.
Kubernetes has received fixes for one of the most serious vulnerabilities ever found in the project to date. If left unpatched, the flaw could allow attackers to take over entire compute nodes.
“With a specially crafted request, users that are allowed to establish a connection through the Kubernetes API server to a backend server can then send arbitrary requests over the same connection directly to that backend, authenticated with the Kubernetes API server’s TLS credentials used to establish the backend connection,” the Kubernetes developers said in an advisory.
Furthermore, in default configurations, both authenticated and unauthenticated users are allowed to perform API discovery calls and could exploit this vulnerability to escalate their privileges. For example, attackers could list all pods on a node, could run arbitrary commands on those pods and could obtain the output of those commands.
The V4L2 (Video for Linux 2) API has long offered an open source alternative to proprietary camera/computer interfaces, but it’s beginning to show its age. At the Embedded Linux Conference Europe in October, the V4L2 project unveiled a successor called libcamera. V4L2 co-creator and prolific Linux kernel contributor Laurent Pinchart outlined the early-stage libcamera project in a presentation called “Why Embedded Cameras are Difficult, and How to Make Them Easy.”
V4l and V4L2 were developed when camera-enabled embedded systems were far simpler. “Maybe you had a camera sensor connected to a SoC, with maybe a scaler, and everything was exposed via the API,” said Pinchart, who runs an embedded Linux firms called Ideas on Board and is currently working for Renesas. “But when hardware became more complex, we disposed of the traditional model. Instead of exposing a camera as a single device with a single API, we let userspace dive into the device and expose the technology to offer more fine-grained control.”
These improvements were extensively documented, enabling experienced developers implement more use cases than before. Yet, the spec placed much of the burden of controlling the complex API on developers, with few resources available to ease the learning curve. In other words, “V4L2 became more complex for userspace,” explained Pinchart.
The project planned to add a layer called libv4l to address this. The libv4l userspace library was designed to mimic the V4L2 kernel API and expose it to apps “so it could be completely transparent in tracking the code to libc,” said Pinchart. “The plan was to have device specific plugins provided by the vendor and it would all be part of the libv4l file, but it never happened. Even if it had, it would not have been enough.”
Libcamera, which Pinchart describes as “not only a camera library but a full camera stack in user space,” aims to ease embedded camera application development, improving both on V4L2 and libv4l. The core piece is a libcamera framework, written in C++, that exposes kernel driver APIs to userspace. On top of the framework are optional language bindings for languages such as C.
The next layer up is a libcamera application layer that translates to existing camera APIs, including V4L2, Gstreamer, and the Android Camera Framework, which Pinchart said would not contain the usual vendor specific Android HAL code. As for V4L2, “we will attempt to maintain compatibility as a best effort, but we won’t implement every feature,” said Pinchart. There will also be a native libcamera app format, as well as plans to support Chrome OS.
Libcamera keeps the kernel level hidden from the upper layers. The framework is built around the concept of a camera device, “which is what you would expect from a camera as an end user,” said Pinchart. “We will want to implement each camera’s capabilities, and we’ll also have a concept of profiles, which is a higher view of features. For example, you could choose a video or point-and-shoot profile.”
Libcamera will support multiple video streams from a single camera. “In videoconferencing, for example, you might want a different resolution and stream than what you encode over the network,” said Pinchart. “You may want to display the live stream on the screen and, at the same time, capture stills or record video, perhaps at different resolutions.”
Per-frame controls and a 3A API
One major new feature is per-frame controls. “Cameras provide controls for things like video stabilization, flash, or exposure time which may change under different lighting conditions,” said Pinchart. “V4L2 supports most of these controls but with one big limitation. Because you’re capturing a video stream with one frame after another, if you want to increase exposure time you never know precisely at what frame that will take effect. If you want to take a still image capture with flash, you don’t want to activate a flash and receive an image that is either before or after the flash.”
With libcamera’s per-frame controls, you can be more precise. “If you want to ensure you always have the right brightness and exposure time, you need to control those features in a way that is tied to the video stream,” explained Pinchart. “With per-frame controls you can modify all the frames that are being captured in a way that is synchronized with the stream.”
Libcamera also offers a novel approach to a given camera’s 3A controls, such as auto exposure, autofocus, and auto white balance. To provide a 3A control loop, “you can have a simple implementation with 100 lines of code that will give you barely usable results or an implementation based on two or three years of development by device vendors where they really try to optimize the image quality,” said Pinchart. Because most SoC vendors refuse to release the 3A algorithms that run in their ISPs with an open source license, “we want to create a framework and ecosystem in which open source re-implementations of proprietary 3A algorithms will be possible,” said Pinchart.
Libcamera will provide a 3A API that will translate between standard camera code and a vendor specific component. “The camera needs to communicate with kernel drivers, which is a security risk if the image processing code is closed source,” said Pinchart. “You’re running untrusted 3A vendor code, and even if they’re not doing something behind your back, it can be hacked. So we want to be able to isolate the closed source component and make it operate within a sandbox. The API can be marshaled and unmarshaled over IPC. We can limit the system calls that are available and prevent the sandboxed component from directly accessing the kernel driver. Sandboxing will ensure that all the controls will have to go through our API.”
The 3A API combined with libcamera’s sandboxing approach, may encourage more SoC vendors to further expose their ISPs just as some are have begun to open up their GPUs. “We want the vendors to publish open source camera drivers that expose and document every control on the device,” he said. “When you are interacting with a camera, a large part of that code is device agnostic. Vendors implement a completely closed source camera HAL and supply their own buffer management and memory location and other tasks that don’t add any value. It’s a waste of resources. We want as much code as possible that can be reused and shared with vendors.”
Pinchart went on to describe libcamera’s cam device manager, which will support hot plugging and unplugging of cameras. He also explained libcamera’s pipeline handler, which controls memory buffering and communications between MIPI-CSI or other camera receiver interfaces and the camera’s ISP.
“Our pipeline handler takes care of the details so the application doesn’t have to,” said Pinchart. “It handles scheduling, configuration, signal routing, the number of streams, and locating and passing buffers.” The pipeline handler is flexible enough to support an ISP with an integrated CSI receiver (and without a buffer pool) or other complicated ISPs that can have a direct pipeline to memory.
Forget the job-stealing robot predictions. Let’s focus on artificial intelligence trends – around talent, security, data analytics, and more – that will matter to IT leaders.
We decided to focus on the trends that matter most urgently to IT leaders – you don’t need another “AI is taking over the world” story; you need concrete insights for your team and business. So, let’s dig into the key trends in AI – as well as overlapping fields such as machine learning – that IT leaders should keep tabs on in 2019.
Open source AI communities will thrive
Many of the cloud and cloud-native technologies in use today were born out of open source projects: Consider Kubernetes Exhibit A in that argument. Expect the development of AI, machine learning, and related technologies to follow a similar path.
“Today, more leading-edge software development occurs inside open source communities than ever before, and it is becoming increasingly difficult for proprietary projects to keep up with the rapid pace of development that open source offers,” says Ibrahim Haddad, director of research at The Linux Foundation, which includes the LF Deep Learning Foundation. “AI is no different and is becoming dominated by open source software that brings together multiple collaborators and organizations.”
In particular, Haddad expects more cutting-edge technology companies and early adopters to open source their internal AI work to catalyze the next waves of development, not unlike how Google spun Kubernetes out from an internal system to an open source project.
Linus Torvalds has stuck to his “no swearing” resolution with his regular Sunday night Linux kernel release candidate announcement.
Probably the most important aspect of the weekend’s release candidate is that it, in a way, improves the performance of STIBP, which is a mitigation that stops malware exploiting a Spectre security vulnerability variant in Intel processors.
In November, it emerged that STIBP (Single Thread Indirect Branch Predictors), which counters Spectre Variant 2 attacks, caused nightmare slowdowns in some cases. The mitigation didn’t play well with simultaneous multi-threading (SMT) aka Intel’s Hyper Threading, and software would take up to a 50 per cent performance hit when the security measure was enabled. Linux 4.20-rc5, emitted on Sunday, addresses this performance issue by making the security defense optional…
The Linux bash, or the command line, lets you perform both basic and complex arithmetic and boolean operations. The commands like expr, jot, bc and, factor etc, help you in finding optimal mathematical solutions to complex problems. In this article, we will describe these commands and present examples that will serve as a basis for you to move to more useful mathematical solutions.
We have run the commands and procedures mentioned in this article on a Ubuntu 18.04 LTS system.
We are using the Ubuntu command line, the Terminal, in order to perform all the mathematical operations. You can open the Terminal either through the system Dash or the Ctrl+Alt+T shortcut.
The expr command
The expr or the expression command in Linux is the most commonly used command that is used to perform mathematical calculations. You can use this command to perform functions like addition, subtraction, multiplication, division, incrementing a value and, even comparing two values.
You may have heard about the concept of custom Operators in Kubernetes. You may have even read about the CoreOS operator-sdk, or tried walking through the setup. The concept is cool: Operators can help you extend Kubernetes functionality to include managing any stateful applications your organization uses. At Kenzan, we see many possibilities for their use on stateful infrastructure apps, including upgrades, node recovery, and resizing a cluster. An ideal future for platform ops will inevitably include operators themselves maintaining stateful applications and keeping runtime intervention by a human to a bare minimum. We also admit the topic of operators and the operator-idk is a tad confusing. After reading a bit, you may secretly still be mystified as to what operators exactly do, and how all the components work together.
In this article, we will demystify what an operator is, and how the CoreOS operator-sdk translates its input to the code that is then run as an operator. In this step-by-step tutorial, we will create a general example of an operator, with a few bells and whistles beyond the functionality shown in the operator-sdk user guide. By the end, you will have a solid foundation for how to build a custom operator that can be applied to real-world use cases.
Hello Operator, could you tell me what an Operator is?
To describe what an operator does, let’s go back to Kubernetes architecture for a bit. Kubernetes is essentially a desired state manager. You give it a desired state for your application (number of instances, disk space, image to use, etc.) and it attempts to maintain that state should anything get out of wack. Kubernetes uses what’s called a control plane on it’s master node. The control plane includes a number of controllerswhose job is to reconcile against the desired state in the following way:
Monitor existing K8s objects (Pods, Deployments, etc.) to determine their state
Compare it to the K8s yaml spec for the object
If the state is not the same as the spec, the controller will attempt to remedy this
A common scenario where reconciling takes place is a Pod is defined with three replicas. One goes down, and with K8s’ controller watching, it recognizes that there should be three Pods running, not two. It then works to create a new instance of the Pod.
This simplified diagram shows the role of controllers in Kubernetes architecture as follows.
The Kubectl CLI sends an object spec (Pod, Deployment, etc.) to the API server on the Master Node to run on the cluster
The Master Node will schedule the object to run (not shown)
Once running, a Controller will continuously monitor the object and reconcile it against the spec
In this way, Kubernetes works great to take much of the manual work out of maintaining the runtime for stateless applications. Yet it is limited in the number of object types (Pods, Deployments, Namespaces, Services, DaemonSets, etc.) that it will natively maintain. Each of these object types has a predetermined behavior and way of reconciling against their spec should they break, without much deviation in how they are handled.
Now, what if your application has a bit more complexity and you need to perform a custom operation to bring it to a desired running state?
Think of a stateful application. You have a database application running on several nodes. If a majority of nodes go down, you’ll need to reload the database from a specific snapshot following specific steps. Using existing object types and controllers in Kubernetes, this would be impossible to achieve. Or think of scaling nodes up, or upgrading to a new version, or disaster recovery for our stateful application. These kinds of operations often need very specific steps, and typically require manual intervention.
Enter operators.
Operators extend Kubernetes by allowing you to define a Custom Controller to watch your application and perform custom tasks based on its state (a perfect fit to automate maintenance of the stateful application we described above). The application you want to watch is defined in Kubernetes as a new object: a Custom Resource (CR) that has its own yaml spec and object type (in K8s, a kind) that is understood by the API server. That way, you can define any specific criteria in the custom spec to watch out for, and reconcile the instance when it doesn’t match the spec. The way an operator’s controller reconciles against a spec is very similar to native Kubernetes’ controllers, though it is using mostly custom components.
Note the primary difference in our diagram from the previous one is that the Operator is now running the custom controller to reconcile the spec. While the API Server is aware of the custom controller, the Operator runs independently, and can run either inside or outside the cluster.
Because Operators are a powerful tool for stateful applications, we are seeing a number of pre-built operators from CoreOS and other contributors for things like ectd, Vault, and Prometheus. And while these are a great starting point, the value of your operator really depends on what you do with it: what your best practice is for failed states, and how the operator functionality may have to work alongside manual intervention.
Dial it in: Yes, I’d like to try Building an Operator
Based on the above diagram, in order to create our custom Operator, we’ll need the following:
A Custom Resource (CR) spec that defines the application we want to watch, as well as an API for the CR
A Custom Controller to watch our application
Custom code within the new controller that dictates how to reconcile our CR against the spec
An Operator to manage the Custom Controller
A deployment for the Operator and Custom Resource
All of the above could be created by writing Go code and specs by hand, or using a tool like kubebuilder to generate Kubernetes APIs. But the easiest route (and the method we’ll use here) is generating the boilerplate for these components using the CoreOS operator-sdk. It allows you to generate the skeleton for the spec, the controller, and the operator, all via a convenient CLI. Once generated, you define the custom fields in the spec and write the custom code to reconcile against the spec. We’ll walk through each of these steps in the next part of the tutorial.
Toye Idowu is a Platform Engineer at Kenzan Media.
The bash man page is close to 40K words. It’s not quite War and Peace, but it could hold its own in a rack of cheap novels. Given the size of bash’s documentation, missing a useful feature is easy to do when looking through the man page. For that reason, as well as to look for new features, revisiting the man page occasionally can be a useful thing to do.
The sub-section of interest today is Parameter Expansion—that is, $var in its many forms. Don’t be confused by the name though, it’s really about parameter and variable expansion.
I’m not going to cover all the different forms of parameter expansion here, just some that I think may not be as widely known as others. If you’re completely new to parameter expansion, check out my ancient post or one of the many articles elsewhere on the internet.
Has “serverless” surpassed containers? Will Kubernetes be the center of the universe for developers? Regardless of the technical benefits, your personal investment in the technologies impact your point of view.
Containers have the edge according to a survey conducted for our ebook about serverless. Overall, 53 percent of respondents would prefer containers as the platform to standardize how their organization abstracts IT infrastructure, with 33 percent choosing functions and 10 percent opting for virtual machines. Asked a different way, 55 percent lean towards container orchestrators like Kubernetes for new applications being deployed in the next year and a half.
We need to confront container documentation as the crucial, non-trivial problem that it is.
TL;DR — As far as I can tell, there’s currently no way of providing documentation for specific containers that we could fairly call canonical, “best practice,” or even all that widely used. This blog post suggests some currently available (but sadly not-great) workarounds but also points to what I think could be a fundamentally better path.
…The result is that questions like these virtually never have readily available, no-Google-involved answers:
How do I run this thing? Should I run a specific executable?
What’s the “blessed” way to run it?
Which ports should I use?
How do I debug it or run its in-container test suites?
Sometimes the problem is that no one has provided answers to these questions anywhere. More often, in my experience, the answers are out there but suffer from a really fundamental discoverability gap.
