The Estimators API in tf.contrib.learn (See tutorial here) is a very convenient way to get started using TensorFlow. The really cool thing from my perspective about the Estimators API is that using it is a very easy way to create distributed TensorFlow models. Many of the TensorFlow samples that you see floating around on the internets are not distributed — they assume that you will be running the code on a single machine. People start with such code and then are immeasurably saddened to learn that the low-level TensorFlow code doesn’t actually work on their complete dataset. They then have to do lots of work to add distributed training code around the original sample, and who wants to edit somebody else’s code?
So, please, please, please, if you see a TensorFlow sample that doesn’t use the Estimators API, ignore it. It will be a lot of work to make it work on your production (read: large) datasets — there will be monitors, coordinators, parameter servers, and all kinds of systems programming craziness that you don’t want to have to dive into. Start with the Estimator API and use the Experiment class.
Have you launched an open source project or are you considering doing so? Making a success of your project can involve everything from evaluating licenses to community outreach. The good news is that there are many free resources that can help you advance and protect your project.
A recent webinar called “Best Practices for Starting an Open Source Project” focused on this topic. Hosted by Capital One, the online event featured Mike Dolan, VP of Strategic Programs at The Linux Foundation, as well as Scott Nicholas, who is Senior Director in the same department and assists in the execution of The Linux Foundation’s annual Legal Summit and other legal programs.
Dolan and Nicholas noted that much of the work in launching a project involves community focus, and they provided advice on key considerations for a project launch including:
How you want your community of developers to build and evolve code
Setting up a decision-making model
Understanding licenses, charter documents, and policies
Creating a usage policy and trademark that fits your code and concept
Dolan and Nicholas both have extensive experience in launching and supporting open source projects. They noted during the webinar that The Linux Foundation has trained more than 800,000 people on open source technologies, and provided the following slide detailing open source milestones:
The Linux Foundation Best Practices
Dolan noted the importance of maintaining independent governance and neutral project assets for open source projects. At the same time, he emphasized that involvement from and dependencies with businesses can keep open source projects healthy and sustainable. For example, many leading businesses contribute to projects ranging from OpenStack to Linux.
Dolan also discussed The Linux Foundation’sCore Infrastructure Initiative (CII) Badge Program. It includes a Best Practices Badge that can showcase an open source project’s commitment to security. Project leaders should look into it.
Scott Nicholas provided a walkthrough of best practices that The Linux Foundation follows with its projects. “Project participants need expertise on how to collaborate with each other,” he said, encouraging meetings and group discussion. He added that governance models and scope for projects should be clearly defined.
“One of the things that is very important to us is separation of business governance and technical governance for projects,” he said. He added that The Linux Foundation builds out charter documents for the open source projects that it stewards. These documents include Inbound Contribution guidelines, Outbound Licensing guidelines, Documentation, and Trademarks.
“Many open source projects will have situations arise where code comes into the project from another project that has a different licensing setup,” he said. “We have an exception policy defined that can help make it easier for code like this to be accepted into a project.” He noted that Contributor Licensing Agreements can help streamline contribution processes and said that The Linux Foundation uses Creative Commons licensing for contributions to documentation for its projects. Additionally, he highly recommended that project leaders document their APIs.
Free Resources
As open source becomes more pervasive, individual contributors and tech and DevOps workers everywhere are building out and overseeing open source projects. And, companies from Google to Netflix to Facebook are releasing their open source creations to the community. You can learn more in the free webcast replay of Best Practices for Starting an Open Source Project, and you can check out the full presentation deck from the event,available as a PDF, which is a very handy reference guide.
As containers have become more important to businesses across the globe, it was necessary to create a system that would allow containers to scale out to meet the needs of enterprise-level deployments. That’s where Kubernetes comes into play.
Unlike Docker, Kubernetes is a very robust ecosystem. Instead of deploying a single container, Kubernetes enables you to deploy multiple containers to multiple hosts, making it ideal for larger deployments and load balancing.
This smart person’s guide is an easy way to get up to speed on Kubernetes. We’ll update this guide periodically when news about Kubernetes is released.
When last we met we reviewed some iptables fundamentals. Now you’ll have two example firewalls to study, one for a single PC and one for a LAN. They are commented all to heck to explain what they’re doing.
This is for IPv4 only, so I’ll write up some example firewalls for IPv6 in a future installment.
I leave as your homework how to configure these to start at boot. There is enough variation in how startup services are managed in the various Linux distributions that it makes me tired to think about it, so it’s up to you figure it out for your distro.
Lone PC Firewall
Use the lone PC firewall on laptop, desktop, or server system. It filters incoming and outgoing packets only for the host it is on.
#!/bin/bash
# iptables single-host firewall script
# Define your command variables
ipt="/sbin/iptables"
# Define multiple network interfaces
wifi="wlx9cefd5fe8f20"
eth0="enp0s25"
# Flush all rules and delete all chains
# because it is best to startup cleanly
$ipt -F
$ipt -X
$ipt -t nat -F
$ipt -t nat -X
$ipt -t mangle -F
$ipt -t mangle -X
# Zero out all counters, again for
# a clean start
$ipt -Z
$ipt -t nat -Z
$ipt -t mangle -Z
# Default policies: deny all incoming
# Unrestricted outgoing
$ipt -P INPUT DROP
$ipt -P FORWARD DROP
$ipt -P OUTPUT ACCEPT
$ipt -t nat -P OUTPUT ACCEPT
$ipt -t nat -P PREROUTING ACCEPT
$ipt -t nat -P POSTROUTING ACCEPT
$ipt -t mangle -P PREROUTING ACCEPT
$ipt -t mangle -P POSTROUTING ACCEPT
# Required for the loopback interface
$ipt -A INPUT -i lo -j ACCEPT
# Reject connection attempts not initiated from the host
$ipt -A INPUT -p tcp --syn -j DROP
# Allow return connections initiated from the host
$ipt -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
# If the above rule does not work because you
# have an ancient iptables version (e.g. on a
# hosting service)
# use this older variation instead
$ipt -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
# Accept important ICMP packets. It is not a good
# idea to completely disable ping; networking
# depends on ping
$ipt -A INPUT -p icmp --icmp-type echo-request -j ACCEPT
$ipt -A INPUT -p icmp --icmp-type time-exceeded -j ACCEPT
$ipt -A INPUT -p icmp --icmp-type destination-unreachable -j ACCEPT
# The previous lines define a simple firewall
# that does not restrict outgoing traffic, and
# allows incoming traffic only for established sessions
# The following rules are optional to allow external access
# to services. Adjust port numbers as needed for your setup
# Use this rule when you accept incoming connections
# to services, such as SSH and HTTP
# This ensures that only SYN-flagged packets are
# allowed in
# Then delete '$ipt -A INPUT -p tcp --syn -j DROP'
$ipt -A INPUT p tcp ! --syn -m state --state NEW -j DROP
# Allow logging in via SSH
$ipt -A INPUT -p tcp --dport 22 -j ACCEPT
# Restrict incoming SSH to a specific network interface
$ipt -A INPUT -i $eth0 -p tcp --dport 22 -j ACCEPT
# Restrict incoming SSH to the local network
$ipt -A INPUT -i $eth0 -p tcp -s 192.0.2.0/24 --dport 22 -j ACCEPT
# Allow external access to your HTTP server
# This allows access to three different ports, e.g. for
# testing.
$ipt -A INPUT -p tcp -m multiport --dport 80,443,8080 -j ACCEPT
# Allow external access to your unencrypted mail server, SMTP,
# IMAP, and POP3.
$ipt -A INPUT -p tcp -m multiport --dport 25,110,143 -j ACCEPT
# Local name server should be restricted to local network
$ipt -A INPUT -p udp -m udp -s 192.0.2.0/24 --dport 53 -j ACCEPT
$ipt -A INPUT -p tcp -m udp -s 192.0.2.0/24 --dport 53 -j ACCEPT
You see how it’s done; adapt these examples to open ports to your database server, rsync, and any other services you want available externally. One more useful restriction you can add is to limit the source port range. Incoming packets for services should be above port 1024, so you can allow in only packets from the high-numbered ports with --sport 1024:65535, like this:
It is sad that IPv4 still dominates US networking, because it’s a big fat pain. We need NAT, network address translation, to move traffic between external publicly routable IP addresses and internal private class addresses. This is an example of a simple Internet connection sharing firewall. It is on a device sitting between the big bad Internet and your LAN, and it has two network interfaces, one connecting to the Internet and one that connects to your LAN switch.
#!/bin/bash
# iptables Internet-connection sharing
# firewall script
# Define your command variables
ipt="/sbin/iptables"
# Define multiple network interfaces
wan="enp0s24"
lan="enp0s25"
# Flush all rules and delete all chains
# because it is best to startup cleanly
$ipt -F
$ipt -X
$ipt -t nat -F
$ipt -t nat -X
$ipt -t mangle -F
$ipt -t mangle -X
# Zero out all counters, again for
# a clean start
$ipt -Z
$ipt -t nat -Z
$ipt -t mangle -Z
# Default policies: deny all incoming
# Unrestricted outgoing
$ipt -P INPUT DROP
$ipt -P FORWARD DROP
$ipt -P OUTPUT ACCEPT
$ipt -t nat -P OUTPUT ACCEPT
$ipt -t nat -P PREROUTING ACCEPT
$ipt -t nat -P POSTROUTING ACCEPT
$ipt -t mangle -P PREROUTING ACCEPT
$ipt -t mangle -P POSTROUTING ACCEPT
# Required for the loopback interface
$ipt -A INPUT -i lo -j ACCEPT
# Set packet forwarding in the kernel
sysctl net.ipv4.ip_forward=1
# Enable IP masquerading, which necessary for NAT
$ipt -t nat -A POSTROUTING -j MASQUERADE
# Enable unrestricted outgoing traffic, incoming
# is restricted to locally-initiated sessions only
$ipt -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
$ipt -A FORWARD -i $wan -o $lan -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
$ipt -A FORWARD -i $lan -o $wan -j ACCEPT
# Accept important ICMP messages
$ipt -A INPUT -p icmp --icmp-type echo-request -j ACCEPT
$ipt -A INPUT -p icmp --icmp-type time-exceeded -j ACCEPT
$ipt -A INPUT -p icmp --icmp-type destination-unreachable -j ACCEPT
Stopping the Firewall
Take the first sections of the example firewall scripts, where it flushes, zeroes, and sets all default policies to ACCEPT, and put them in a separate script. Then use this script to turn your iptables firewall “off”.
The documentation on netfilter.org is ancient, so I recommend using man iptables. These examples are basic and should provide a good template for customizing your own firewalls. Allowing access to services on your LAN through your Internet firewall is good subject for another day, because thanks to NAT it is a huge pain and needs a lot of rules and explaining.
Learn more about Linux through the free “Introduction to Linux” course from The Linux Foundation and edX.
Developers looking to add GUIs to their embedded devices have a variety of open source and commercial options, with Qt generally leading the list. If you’re operating in severely constrained environments, however, especially for battery powered devices like wearables, the open source Enlightenment Foundation Libraries (EFL) should be given close consideration.
At the recent Embedded Linux Conference, Cedric Bail, a long-time contributor to the Enlightenment project who works on EFL integration with Tizen at Samsung Open Source Group, discussed some of the lessons learned in optimizing wearable apps for low battery, memory, and CPU usage. Bail summarized EFL and revealed an ongoing project to improve EFL’s scene graph. However, most of the lessons are relevant to anyone optimizing for wearables on any platform (see the ELC video below).
EFL has been under development for 10 years and was released in 2011. It was designed as a toolkit for the 20-year old Enlightenment, the first windows manager for GNOME. Today, however, it has evolved to feature its own rendering library and scene graph and can work with a variety of windowing environments. The EFL project is in the process of adding an optimized Wayland support.
Samsung was an early supporter of EFL and is still its biggest champion. The CE giant now uses EFL in its Tizen-based Samsung Galaxy Gear smartwatches, as well as its smart TVs, and several other Tizen-based devices.
Like Enlightenment, EFL was designed from the start for embedded GUIs. The toolkit is licensed with a mix of LGPL 2.1 and BSD, and written in C, with bindings to other languages. EFL is primarily designed for Linux, although it’s lean enough to run on RTOS- and MCU-driven devices such as the Coyote navigation device.
“EFL is optimized for reducing CPU, GPU, memory, and battery usage,” said Bail. “It takes up as little as 8MB, including all widgets. Some have pushed it lower than that, but it’s less functional. Arch Linux can run with EFL on 48MB RAM at 300MH, with a 1024×768 screen in full 32-bits display, and without a GPU. EFL does better than Android on battery consumption. That’s why Samsung is using Tizen on its smartwatches.”
Despite its minimalist nature, EFL supports accessibility (ATSPI) and international language requirements. It’s also “fully themable,” and can “change the scale of the UI based on the screen and input size,” said Bail. “We also account for the DPI and reading distance of the screen for better readability.”
Bail disagrees with the notion that improvements in power/performance will soon make minimalist graphics toolkits obsolete. “Moore’s Law is slowing down, and it doesn’t apply to battery life and memory bandwidth,” said Bail.
The UI is typically the most resource-intensive part of an application. “Most applications don’t do much,” said Bail. “They fetch stuff from the network, a database, and then change the UI, which does all the CPU and GPU intensive tasks. So optimizing the UI saves a lot of energy.”
The biggest energy savings can be found in the UI design itself. “If your designer gives you a 20-layer UI with all these bitmaps and transparency, there is very little our toolkit can do to reduce energy,” said Bail.
The most power-efficient designs stick to basic rectangles, lines, and vertical gradients, and avoid using the entire screen. “If you have full-screen animations where things slide from left to right or up and down, you can’t do partial updates, so it uses more energy,” said Bail. If you’re targeting AMOLED screens, consider using a black background, as the Enlightenment project does on its own website. “Reddit running in black consumes 41 percent less energy on Android AMOLED phone,” Bail added.
One other trick used on the Gear smartwatches is an integrated frame buffer within the display. “The system can completely suspend the SoC, while the display refreshes itself, and save a lot of battery life, which is fine if you’re just displaying a watch face.”
Memory optimization is another area where you can substantially reduce consumption. “All rendering operations are constrained by memory bandwidth,” said Bail. “And if you want to run a web browser, memory declines quickly.”
Memory consumption is primarily driven by screen size, which is not usually an issue for a smartwatch. There are other ways to optimize for memory usage, however, such as avoiding true multitasking and using CPU cache.
“Accessing main memory uses more energy than accessing the CPU cache,” said Bail. “You can optimize by improving cache locality and doing linear instead of random access. You can look for cache locality with Cachegrind and visualize it with Kcachegrind, and you can hunt for leaks and overuse with tools like massif and massif-visualizer.”
Power reduction in wearables can produce all-day battery life, as well as reduce dissipated heat for greater comfort, said Bail. Lower battery consumption also provides “more freedom for designers, who may want to go with a thinner device using a smaller battery.”
Wearables developers should also optimize for speed, which means “doing things more efficiently and avoiding unnecessary complications,” said Bail. This also applies to the GPU where you can optimize redrawing the screen if nothing has changed, or doing partial updates and reusing information from a past image. You should also trigger animations “only at speed the hardware is capable of,” he added.
The EFL project uses the Raspberry Pi as a target for speed optimization, and runs tests with Kcachegrind. For EFL, Bail recommends that Pi users adopt the more up-to-date Arch Linux over Raspbian.
Network optimization is also important. “When you send data, you are more likely to lose packets, which means you have to retransmit it, which takes energy,” said Bail. “You want to send as little data as possible over the network and download only what is needed.”
Optimizing downloads is a bit tricky because “this is usually a prefetch, and you don’t know if the user will need all of the download, so you may end up over-downloading something,” said Bail. “You should group your downloads together, and then switch to full network idle for as long as possible.” Otherwise, energy is consumed by wireless stacks time switching between energy states.
When optimizing specifically for battery use, developers rely on the Linux kernel. The kernel chooses the clock, voltage, and number of active cores “by trying to figure out what you are going to do even though it has no idea what you are doing,” said Bail. “For years, the kernel failed at this,” he added.
The problem comes from the kernel scheduler basing its decisions solely on past process activity whereas digital reality can be much more dynamic. Among other problems, the scheduler “forgets everything as soon as it migrates to another CPU core.” There are also complications such as the CPU frequency driver and CPU idle driver both looking at system load, but without coordinating their activities.
In the past, developers tried to overcome this problem with an energy aware userspace daemon equipped with a hard-coded list of applications and their corresponding behavior. ARM and Linaro are working on an a more dynamically aware solution called SCHED_DEADLINE as part of its Energy Aware Scheduling (EAS) project. The technology, which was covered in a separate ELC 2017 presentation by ARM’s Juri Lelli, promises to link CPU frequency and idle to the scheduler, while also retaining more information about past loads.
“SCHED_DEADLINE will let interactive tasks become properly scheduled by the system,” said Bail. “The userspace will break things apart in a thread that is dedicated to specific things.” With the new infrastructure, interactive tasks will be able to change behavior very quickly, even during the 16ms rendering of a frame.
With the increasing use of multi-core processors, the original EFL scene graph switched a few years ago to a dual-thread version. Now the project is developing a new version that will work in harmony with SCHED_DEADLINE.
The scene graph is “a bookkeeping of primitive graphical objects, of everything you draw on the screen,” said Bail. “It has a general view of the application from inside the toolkit so you can do global optimization.”
With the current dual-thread scene graph, “the kernel may have a hard time figuring out what is going on in the main loop,” explained Bail. “Our new version will help this by grouping computation — such as CPU-intensive tasks for generating spanline for shape, and decompressing image — into a specific thread for screen-intensive tasks, and grouping all the memory-bound computations in their own thread.”
The new scene graph is not without its drawbacks, however. “The main price we pay is increased memory usage because we need more threads,” said Bail. “Every thread has its own stack, which causes increasing complexity and new bugs. We are trying to patch all these risky things inside the toolkit.”
You can watch the complete video below:
Connect with the Linux community at Open Source Summit North America on September 11-13. Linux.com readers can register now with the discount code, LINUXRD5, for 5% off the all-access attendee registration price. Register now to save over $300!
The metric we all use for CPU utilization is deeply misleading, and getting worse every year. What is CPU utilization? How busy your processors are? No, that’s not what it measures. Yes, I’m talking about the “%CPU” metric used everywhere, by everyone. In every performance monitoring product. In top(1).
Gierad Laput wants to make homes smarter without forcing you to buy a bunch of sensor-laden, Internet-connected appliances. Instead, he’s come up with a way to combine a slew of sensors into a device about the size of a Saltine that plugs into a wall outlet and can monitor many things in the room, from a tea kettle to a paper towel dispenser.
Laput, a graduate student studying computer-human interaction at Carnegie Mellon University, built the gadget as part of a project he calls Synthetic Sensors. He says it could be used to do things like figure out how many paper towels you’ve got left, detect when someone enters or leaves a building, or keep an eye on an elderly family member (by tracking the person’s typical routine via appliances, for example). It’s being shown off this week in Denver at the CHI computer-human interaction conference.
Features are wonderful. When Node.js adds a new API, we can instantly do more with it. Wouldn’t a larger standard library be more useful for developers? Who could possibly object to Node.js getting better? And who, even more strangely, would actually remove APIs, making Node.js objectively worse? Turns out, lots of people…
A recent proposal to get HTTP proxy environment variable support into Node.js, https://github.com/nodejs/node/issues/8381, got a generally hesitant response from Node.js maintainers, and generally enthusiastic response from Node.js users.
What’s going on here? Why the disconnect? Is this an ideological stance?
No, for those who support a small core, its a matter of pragmatism. What is happening is that the instantaneous delight of new APIs in Node.js fades with time, even as the problems of maintaining those APIs grows. The longer you work with Node.js core, supporting, bug fixing, and documenting the existing APIs, the more you wish that Node.js had been more conservative in accepting new APIs in the past, and that those APIs had been implemented as npm modules that could be independently maintained and versioned.
Input Club’s mechanical keyboards aren’t just about producing exceptional products. They’re also proof that open source can solve any problem.
Open source software already powers most of the world, partially because it is free and mostly because it is so accessible. Under an open source system, the flaws and imperfections in every product can be observed, tracked, and fixed, much like the Japanese philosophy of “continuous improvement” known as kaizen, which is applied to every aspect of a process. By following these principles, we believe that the open hardware movement is poised to fundamentally change the global product economy.
At Input Club, we design and produce mechanical keyboards using this same philosophy and workflow, similar to how a person might develop a website or application. The design files for our keyboard frames and circuit boards are available via GitHub.
Google’s OSS-Fuzz bug-hunting robot has been hard at work, and in recent months, over 1,000 bugs have been exposed.
According to Chrome Security engineers Oliver Chang and Abhishek Arya, software engineer Kostya Serebryany, and Google Security program manager Josh Armour, the OSS-Fuzz bot has been scouring the web over the past five months in the pursuit of security vulnerabilities which can be exploited.
The OSS-Fuzz bot uses a technique called fuzzing to find bugs.
