Chapter 21
ThinkPenguin Wireless Router: Excellent CCS

Too often, case studies examine failure and mistakes. Indeed, most of the chapters that follow herein will consider the myriad difficulties discovered in community-oriented GPL enforcement for the last two decades. However, to begin, this is a case study in how copyleft compliance can indeed be done correctly.

This example is, in fact, more than ten years in the making. Since almost the inception of for-profit corporate adoption of Free Software, companies have requested a clear example of a model citizen to emulate. Sadly, while community-oriented enforcers have vetted uncounted thousands of “Complete, Corresponding Source” (CCS) candidates from hundreds of companies, this particular CCS release described herein is the first ever declared a “pristine example”.

Of course, most CCS examined for the last decade has (eventually) complied with the GPL, perhaps after many iterations of review by the enforcer. However, in the experience of the two primary community-oriented enforcers (Conservancy and the FSF), such CCS results routinely “barely comply with GPL’s requirements”. To use an academic analogy: while a “C” is certainly a passing grade, any instructor prefers to disseminate to the class an exemplar sample that earned an “A”.

Fortunately, thanks in large part to the FSF’s “Respects Your Freedom” (RYF) certification campaign1, a few electronics products on the market meet a higher standard of copyleft compliance. As such, for the first time in the history of copyleft, CCS experts have pristine examples to study and present as exemplars worthy of emulation.

This case study therefore examines the entire life-cycle of a GPL compliance investigation: from product purchase, to source request, to CCS review, and concluding in a final compliance determination. Specifically, this chapter discusses the purchase, CCS provision, and a step-by-step build and installation analysis of a specific, physical, embedded electronics product: the “TPE-NWIFIROUTER” wireless router by ThinkPenguin.2

21.1 Consumer Purchase and Unboxing

The process for copyleft compliance investigation, when properly conducted, determines whether users inclined to exercise their rights under a copyleft license will be successful in their attempt. Therefore, at every stage, the investigator seeks to take actions that reasonably technically knowledgeable users would during the ordinary course of their acquisition and use of copyleft-covered products. As such, the investigator typically purchases the device on the open market to verify that distribution of the copylefted software therein complies with binary distribution requirements (such as those discussed earlier in  5.2 and  9.9).

Therefore, the investigator first purchased the TPE-NWIFIROUTER through an online order, and when the package arrived, examined the contents of the box. The investigator immediately discovered that ThinkPenguin had taken advice from  15.1.2, and exercised GPLv23(a) and GPLv36, rather than using the problematic offer for source provisions. This choice not only accelerated the investigation (since there was no CCS offer to “test”), but also simplified the compliance requirements for ThinkPenguin.

21.2 Root Filesystem and Kernel Compilation

The CD found in the box was labeled “libreCMC v1.2.1 source code”, and contained 407 megabytes of data. The investigator copied this ISO to a desktop GNU/Linux system and examined its contents. Upon doing so, the investigator immediately found a file called “README” at the top-level directory:

$ dd if=/dev/cdrom of=libreCMC_v1.2.1_SRC.iso 
$ mkdir libCMC 
$ sudo mount o loop ./libreCMC_v1.2.1_SRC.iso libCMC 
mount: block device /path/to/libreCMC_v1.2.1_SRC.iso 
       is writeprotected, mounting readonly 
$ ls 1 libCMC 
bin 
librecmcuboot.tar.bz2 
librecmcv1.2.1.tar.bz2 
README 
uboot_reflash 
cat libCMC/README 
...

The investigator therefore knew immediately to begin the CCS check should begin with a study of the contents of “README”. Indeed, that file contained the appropriate details to start the build:

In order to build firmware images for your router, the following needs to be installed:

gcc, binutils, bzip2, flex, python, perl, make, find, grep, diff, unzip, gawk, getopt, libz-dev and libc headers.

Please use “make menuconfig” to configure your appreciated configuration for the toolchain and firmware. Please note that the default configuration is what was used to build the firmware image for your router. It is advised that you use this configuration.

Simply running “make” will build your firmware. The build system will download all sources, build the cross-compile toolchain, the kernel and all chosen applications.

To build your own firmware you need to have access to a GNU/Linux system (case-sensitive filesystem required).

In other words, the first “script” that investigator “ran” was the above. This was not a software script, rather the processor for the script was the investigator’s own brain — like a script of a play. Less glibly stated: instructions written in English are usually necessary for the build and installation operations that demand actual intelligence. In this case, the investigator ascertained the host system requirements for construction of this embedded firmware.

GPL does not give specific guidance on the form or location of “scripts used to control compilation and installation of the executable” and/or “Installation Information”. Community-oriented GPL enforcers apply a “reasonableness standard” to evaluate such instructions. If an investigator of average skill in embedded firmware construction can surmise the proper procedures to build and install a replacement firmware, the instructions are likely sufficient to meet GPL’s requirements. Fortunately, in this case, the instructions are more abundant and give extra detail.

Nevertheless, these instructions offer more options than the reader typically sees in other CCS candidates. More typically, top-level build instructions name an exact host distribution to use, such as “Debian 7 installed on an amd64 system with the following packages installed”. Of course, if the build will fail on any other system, instructions should include such details. However, this CCS builds on a wide range of distributions, and thus it was appropriate (and preferred) that the build instructions do not specify a specific distribution.

In this specific case, the developers of the libreCMC project (a Free Software project that forms the base system for the TPE-NWIFIROUTER) have clearly made an effort to ensure the CCS builds on a variety of host systems. The investigator was in fact dubious upon seeing these instructions, since finicky embedded build processes usually require a very specific host system. Fortunately, it seems such doubts were generally unfounded (although the investigator did find a minor annoyance that could be resolved with more detailed instructions).

Anyway, since these instructions did not specify a specific host system, the investigator simply used his own amd64 Debian GNU/Linux 6 desktop system. Before beginning, the investigator used the following command:

$ dpkg list | egrep ’ˆiii’ | less

to verify that the required packages listed in the README were installed3.

Next, the investigator then extracted the primary source package with the following command:

$ tar posix jxpf libCMC/librecmcv1.2.1.tar.bz2

The investigator did notice an additional source release, entitled “librecmc-u-boot.tar.bz2”. The investigator concluded upon simple inspection that the instructions found in “u-boot_reflash” were specific instructions for that part of the CCS. This was a minor annoyance, and ideally the “README” would so-state, but the CCS filesystem layout met the reasonableness standard; the skilled investigator determine the correct course of action with a few moments of study.

The investigator then noted the additional step offered by the “README”, which read:

Please use “make menuconfig” to configure your appreciated configuration for the toolchain and firmware. Please note that the default configuration is what was used to build the firmware image for your router. It is advised that you use this configuration.

This instruction actually exceeds GPL’s requirements. Specifically, the instruction guides users in their first step toward exercising the freedom to modify the software. While the GPL does contain requirements that facilitate the freedom to modify (such as ensuring the CCS is in the “preferred form …for making modifications to it”), GPL does not require specific instructions explaining how to undertake modifications. This specific instruction therefore exemplifies the exceptional quality of this particular CCS.

However, for purposes of the CCS verification process, typically the investigator avoids any unnecessary changes to the source code during the build process, lest the investigator err and cause the build to fail through his own modification, and thus incorrectly identify the CCS as inadequate. Therefore, the investigator proceeded to simply run:

cd libCMC 
$ make

and waited approximately 40 minutes for the build to complete4. The investigator kept a full log of the build, which is not included herein due its size (approximately 7.2K of text).

Upon completion of the “make” process, the investigator immediately found (almost to his surprise) several large firmware files in the “bin/ar71xx” directory. Typically, this step in the CCS verification process is harrowing. In most cases, the “make” step will fail due to a missing package or because toolchain paths are not setup correctly.

In light of such experiences, the investigator speculated that ThinkPenguin’s engineers did the most important step in self-CCS verification: test one’s own instructions on a clean system. Ideally, an employee with similar skills but unfamiliar with the specific product can most easily verify CCS and identify problems before a violation occurs.

However, upon completing the “make”, the investigator was unclear which filesystem and kernel images to install on the TPE-NWIFIROUTER hardware. Ideally, the original “README” would indicate which image is appropriate for the included hardware. However, this was ultimately an annoyance rather than a compliance issue. Fortunately, the web UI (see next section) on the TPE-NWIFIROUTER performs firmware image installation. Additionally, the router’s version number was specified on the bottom of the device, which indicated which of the differently-versioned images we should install. The investigator would prefer instructions such as those found at http://librecmc.org/librecmc/wiki?name=Tp+MR3020instructions similar to these in the README itself; however, application of the reasonableness standard here again indicates compliance, since a knowledgeable user can easily determine the proper course of action.

21.3 U-Boot Compilation

The investigator then turned his attention to the file, “u-boot_reflash”. These instructions explained how to build and install the bootloader for the device.

The investigator followed the instructions for compiling U-Boot, and found them quite straight-forward. The investigator discovered two minor annoyances, however, while building U-Boot:

The full log of the build is not included herein due its size (approximately 3.8K of text). After that, the investigator found a new U-Boot image in the “bin” directory.

21.4 Root Filesystem and Kernel Installation

The investigator next tested installation of the firmware. In particular, the investigator connected the TPE-NWIFIROUTER to a local network, and visited http://192.168.10.1/, logged in, and chose the option sequence: “System Backup / Flash Firmware”.

From there, the investigator chose the “Flash new firmware image” section and selected the “librecmc-ar71xx-generic-tl-wr841n-v8-squashfs-sysupgrade.bin” image from the “bin/ar71xx” directory. The investigator chose the “v8” image upon verifying the physical router read “v8.2” on its bottom. The investigator chose the “sysupgrade” version of the image because this was clearly a system upgrade (as a firmware already came preinstalled on the TPE-NWIFIROUTER).

Upon clicking “Flash image…”, the web interface prompted the investigator to confirm the MD5 hash of the image to flash. The investigator did so, and then clicked “Proceed” to flash the image. The process took about one minute, at which point the web page refreshed to the login screen. Upon logging in, the investigator was able to confirm in the “Kernel Log” section of the web interface that the newly built copy of Linux had indeed been installed.

The investigator confirmed that a new version of “busybox” had also been installed by using SSH to connect to the router and ran the command “busybox”, which showed the newly-compiled version (via its date of compilation).

21.5 U-Boot Installation

The U-Boot installation process is substantially more complicated than the firmware update. The investigator purchased the optional serial cable along with the TPE-NWIFIROUTER, in order to complete the U-Boot installation per the instructions in “u-boot_reflash” in its section “Installing u-boot to your router”, which reads:

1.
Install and configure any TFTP server on your PC (tftp-hpa). Set a fixed IP address on your PC …and connect it to the router, using RJ45 network cable …
2.
Connect USB to UART adapter to the router and start any application to communicate with it, like PuTTY. …
3.
Power on the router, wait for a line like one of the following and interrupt the process of loading a kernel:
 Autobooting in 1 seconds  
      (for most TP-Link routers, you should enter tpl at this point)  
Hit ESC key to stop autoboot: 1 (for 8devices Carambola 2, use ESC key)  
 Hit any key to stop autoboot: 1 (for D-Link DIR-505, use any key)

4.
Set ipaddr and serverip environment variables:
    hornet> setenv ipaddr 192.168.1.1 
    hornet> setenv serverip 192.168.1.2

The investigator used the purchased serial cable, which was a USB serial adapter with a male USB type A connector to 4 female jumper wires. The female jumper wires were red, black, white, and green.

The instructions here were slightly incomplete, since they did not specify how to connect the wires to the router. However, the investigator found general information available online at http://wiki.openwrt.org/toh/tp-link/tl-wr841nd#serial.console which described the proper procedure. While the “power” and “ground” cables were obvious, some trial and error was necessary to find the RX and TX cables, but this was easily determined since miswiring TX and RX yields no I/O and proper wiring yields the output as expected. Using the pin gender changer included with the adapter, the investigator was able to stably wire the pins for use once the proper RX and TX connections were determined.

The investigator then used the recommended 115200 8N1 for serial console settings, leaving all flow control off, and tested both with the minicom and screen commands. The investigator found that if all 4 wires were connected on the USB serial adapter that the router would start without additional power and the console would receive the startup messages. The investigator could replicate the same behavior by omitting the power cable from the USB serial adapter (red wire) and connecting the main power adapter to the router instead.

At this point, the on-screen messages as described in the installation instructions appeared, but the investigator found that no key events sent via the serial port appeared to reach the U-Boot console. In other words, while the investigator saw both U-Boot and kernel boot messages in the serial console, the investigator was unable to interrupt the boot process as instructed by “u-boot_reflash”. Hitting a key simply did not interrupt the boot process and yield the hornet> prompt.

After additional trial and error over a period of hours, the investigator had finally to consider this question for the first time during the process: “Has ThinkPenguin violated the GPL?” More specifically, the immediate question was: “Given this failure, has the distributor met the requirements for ‘scripts used to control …installation of the executable’ (GPLv2) and necessary ‘Installation Information’ (GPLv3)?”

The appropriate answer to the question (at this specific stage) is “possibly, but more information is needed”. Embedded installation and configuration is a tricky and complex technical process. While the GPL requires documentation and clear instructions for this process, the investigator did not immediately blame the distributor for what may be an honest, correctable mistake, or an issue legitimately explained by feasible alternative theories.

In this case, upon remembering the issues of wiring, the investigator wonder if perhaps the power pin was accidentally connected for a moment to the RX pin while live. Such action could easily fry the RX pin, and yield the observed behavior. Since the investigator could not rule out the possibility of accidental connection of power to the RX pin mentioned, the investigator had to assume the instructions would work properly if he had not made that error.

That conclusion, while correct, also left the investigator with only two option to complete the final verification of the CCS:

The investigator chose the latter and then contacted a libreCMC developer familiar with the product. That developer, who agreed the the RX pin was likely ruined, described an alternative method for console access using the netcat. The libreCMC developer described the process as follows:

Upon following this procedure, the investigator was able to confirm the (original) shipped version of U-Boot was still installed:

$ nc p 6666 192.168.1.1 6666 
uboot> version 
UBoot 1.1.4 (Jul 28 2014)

Thereafter, the investigator followed the instructions from “u-boot_reflash”. Specifically, the investigator configured a TFTP server and placed the newly built firmware into /srv/tftp. The investigator also followed the remaining instructions in “u-boot_reflash”, but used the netcat console rather than the serial console, and used U-Boot’s reset command to reboot the router.

Upon reboot, the serial console (still connect with working output) showed the message U-Boot 1.1.4 (Oct 17 2014), and thus confirmed a successful reflash of the U-Boot image built by the investigator.

21.6 Firmware Comparison

Next, to ensure the CCS did indeed correspond to the firmware original installed on the TPE-NWIFIROUTER, the investigator compared the built firmware image with the filesystem originally found on the device itself. The comparison steps were as follows:

1.
Extract the filesystem from the image we built by running find-firmware.pl on “bin/ar71xx/librecmc-ar71xx-generic-tl-wr841n-v8-squashfs-factory.bin” and then running bat-extratools’ “squashfs4.2/squashfs-tools/bat-unsquashfs42” on the resulting morx0.squash, using the filesystem in the new squashfs-root directory for comparison.
2.
Login to the router’s web interface (at http://192.168.10.1/) from a computer connected to the router.
3.
Set a password using the provided link at the top (since the router’s UI warns that no password is set and asks the user to change it).
4.
Logged into the router via SSH, using the root user with the aforementioned password.
5.
Compared representative directory listings and binaries to ensure the set of included files (on the router) is similar to those found in the firmware image that the investigator created (whose contents are now in the local squashfs-root directory). In particular, the investigator did the following comparisons:

(a)
Listed the /bin folder (“ls -l /bin”) and confirm the list of files is the same and that the file sizes are similar.
(b)
Checked the “strings” output of “/bin/busybox” to confirm it is similar in both places (similar number of lines and content of lines). (One cannot directly compare the binaries because the slight compilation variations will cause some bits to be different.)
(c)
Repeated the above two steps for “/lib/modules”, “/usr/bin”, and other directories with a significant number of binaries.
(d)
Checked that the kernel was sufficiently similar. The investigator compared the “dmesg” output both before and after flashing the new firmware. As the investigator expected, the kernel version string was similar, but had a different build date and user@host indicator. (The kernel binary itself is not easily accessible from an SSH login, but was retrievable using the U-Boot console (the start address of the kernel in flash appears to be 0x9F020000, based on the boot messages seen in the serial console).

21.7 Minor Annoyances

As discussed in detail above, there were a few minor annoyances, none of which were GPL violations. Rather, the annoyances briefly impeded the build and installation. However, the investigator, as a reasonably skilled build engineer for embedded devices, was able to complete the process with the instructions provided.

To summarize, no GPL compliance issues were found, and the CCS release was one of the best ever reviewed by any investigator at any community-oriented enforcement organization. However, the following annoyances were discovered:

Thus, no CCS is absolutely perfect, but GPL violation investigators always give the distributors the benefit of any doubts and seek to work with the vendors and improve their CCS for the betterment of their users, customers, and the entire software freedom community.

21.8 Lessons Learned

Companies that seek to redistribute copylefted software can benefit greatly from ThinkPenguin’s example. Here are just a few of the many lessons that can be learned here:

1.
Even though copyleft licenses have them, avoid the offer-for-source provisions. Not only does including the CCS alongside binary distribution make violation investigation and compliance confirmation substantially easier, but also (and more importantly) doing so completes the distributor’s CCS compliance obligations at the time of distribution (provided, of course, that the distributor is otherwise in compliance with the relevant copyleft license).
2.
Include top-level build instructions in a natural language (such as English) in a clear and conspicuous place. Copyleft licenses require that someone reasonably skilled in the art can reproduce the build and installation. Typically, instructions written in English are necessary, and often easier than writing programmed scripts. The “script” included can certainly be more like the script of a play and less like a Bash script.
3.
Write build/install instructions to the appropriate level of specificity. The upstream engineers in this case study clearly did additional work to ensure functionality on a wide variety of host build systems; this is quite rare. When in doubt, include the maximum level of detail build engineers can provide with the CCS instructions, but also double-check to investigate if a more generalized solution (such as other host systems) work just as well for the build.
4.
Seek to adhere to the spirit of copyleft, not just the letter of the license. Encouragement of users to improve and make their devices better is one of ThinkPenguin’s commercial differentiators. Copyleft advocates that other companies have undervalued the large and lucrative market of users who seek hackable devices. By going beyond the mere minimal requirements of GPL, companies can immediately reap the benefits in that target market.
5.
Community-oriented enforcement organizations do not play “gotcha”5 with distributors regarding GPL violations. The goal in the GPL enforcement process is to achieve compliance and correct mistakes and annoyances. Such organizations therefore take an “innocent until proven guilty assume guilty due to honest error rather than malicious action ” approach. The goal is compliance (in direct contrast with the discussion in  12.2 about the proprietary relicensing business model).

1RYF is a campaign by FSF to certify products that truly meet the principles of software freedom. Products must meet strict standards for RYF certification, and among them is a pristine example of CCS.

2The FSF of course performed a thorough CCS check as part of its certification process. The analysis discussed herein was independently performed by Software Freedom Conservancy without reviewing the FSF’s findings. Thus, this analysis is “true to form”, and explains the typical procedures Conservancy uses when investigating a potential GPL violation. In this case, obviously, no violation was uncovered.

3The “dpkg” command is a Debian-specific way of finding installed packages.

4Build times will likely vary widely on various host systems.

5For lack of a better phrase.