Anthony’s Mac Labs Blog

Posted 2018 October 29

For many people, AutoPkg could have easily been named AutoUpdate in recognition of its unparalleled ability to fetch updates for the apps you deploy. Yes, it can create packages as well, but perhaps the only packages you have seen being built by AutoPkg are those that are a simple wrapper around a downloaded app.^[1] With the AppPkgCreator processor (introduced with AutoPkg 1.0.0), creating such a recipe is even simpler than before.

But AutoPkg has always been about automating “the tasks one would normally perform manually to prepare third-party software for mass deployment.”^[2] What if you need to modify something or add to it before deploying? If it is the same process every time, why not have AutoPkg do it for you? Here are three examples of recipes that do just that, explained in detail.

FirefoxAutoconfig

The main model I used when learning about AutoPkg’s ability to build custom packages was the FirefoxAutoconfig.pkg recipe written by Greg Neagle. Firefox allows administrators to configure their deployment of Firefox, usually to preset or lock certain preferences. That ability can be very useful in certain deployments (in my case, a Lab deployment where we want to control updates, block password saving, and the like). While we will soon be able to establish such settings via a configuration profile,^[3] the only method supported until then requires making a configuration file and stuffing it in the application bundle. Historically, the most common tool used to create a supported configuration file has been the CCK (later, the CCK2) Firefox extension written by Mike Kaply. Deploying the configuration payload manually could be tedious; that’s where Greg’s FirefoxAutoconfig.pkg recipe comes in. It downloads a copy of Firefox, copies the app from the disk image to the cache, adds the autoconfig file (generated by CCK2) to the place within the application bundle that Firefox requires, and then creates an installer package of this modified app. All the recipe user needs to do is place the autoconfig.zip file they generated with CCK2 in the same directory as the recipe (usually, the RecipeOverrides directory within your AutoPkg setup); AutoPkg will do the rest. You can see how having a Firefox installer always available pre-configured with your settings could save a lot of work for an admin, especially considering Firefox’s regular release cycle. Let’s take a closer look at the recipe.

The recipe runs the Firefox.download recipe from the main AutoPkg recipes repo and then adds the following processor steps:

1. AppDmgVersioner [grabs the version number of the downloaded app for later use]
2. PkgRootCreator [creates a place in the cache where the files being deployed will be added]
3. Copier [copies the Firefox app from the disk image into the Applications folder inside the package root just created]
4. Unarchiver [copies the configuration files inside the autoconfig.zip archive to where Firefox requires them inside the app bundle]
5. PkgCreator [creates a package based on the content in the package root, named with a custom name and the app’s version number (acquired earlier)]

There are a few interesting details to consider here.

With PkgRootCreator, a directory for the installer payload is created inside the cache in a folder named with the value of the NAME input key; this is the normal convention. When you create the package root, you normally create all the directories you need for the payload you are going to deliver, since the package needs to mimic the system directory layout. Thus, the recipe creates an Applications directory at the root of the package with the same read-write-execute permissions (expressed as the octal number 0775) as you would usually see on that folder. You could create additional directories by specifying more key+string pairs within the pkgdirs dictionary.

The use of Unarchiver is both unusual and creative. In many cases, you would use a Copier processor here, but since the contents of the autoconfig.zip archive (once expanded) is what we need placed inside Contents/Resources within the application bundle, Greg chose to unarchive directly to that specific destination.

Finally, PkgCreator contains the instructions for AutoPkg to use when building the package (you can find out more about what each of the settings means in the documentation found within the source code of autopkgserver). The most interesting setting for us is the chown section. In general, any directory you created with the PkgRootCreator processor should have owner and group set within this processor.^[4] In this case, root and admin are the normal owner and group for the Applications folder, so they have been specified here.

Just a short sidebar on Firefox here: Mozilla created a new JSON-based configuration method that works with Firefox 60 or later (and is supported cross-platform). Neil Martin adapted Greg Neagle’s recipe to support this new method. If you examine the recipe (FirefoxPolicies.pkg in the neilmartin83-recipes repo), the only significant difference is the use of the Copier processor in the fourth processor step. It also requires the user to place the payload to add to Firefox within a particular location in your AutoPkg cache.

VueScanLicenced

The reason I wrote the VueScanLicenced.pkg recipe is not because I need to mass deploy VueScan—currently, I’m deploying it on one Mac. It’s because by doing so I can eliminate a manual process that is not going to change any time soon (due to the vendor’s free update policy for the Pro licence). Writing the recipe functions as both automation and documentation, as you will see in a moment.

Just as with the FirefoxAutoconfig.pkg recipe we just looked at, this recipe uses an existing download recipe for VueScan as the parent. It then adds the following processor steps, which are also very similar:

1. AppDmgVersioner [grabs the version number of the downloaded app for later use]
2. PkgRootCreator [creates a place in the cache where the files being deployed will be added, including the directories needed for the files]
3. Copier [copies the VueScan app from the disk image into the Applications folder inside the package root just created]
4. Copier [copies the VueScan licence information file from the user-specified location on the local disk to the place that VueScan requires it]
5. PkgCreator [creates a package based on the content in the package root, named with a custom name and the app’s version number (acquired earlier)]

The main difference between this recipe and FirefoxAutoconfig is that we create three directories when creating the package root, since we will have a file not just in /Applications, but also in /Users/Shared (we have to create /Users before we can create /Users/Shared). As I mentioned earlier, this is done by simply listing key+string pairs (path plus permissions) in the pkgdirs dictionary. This also means that when we build the package, we need to set the ownership for all three directories, which is done using an array in the PkgCreator processor step (pkg_request > chown > dict for each directory). Finally, we use the Copier processor to place the licence information file in the location required for VueScan to recognize it. In this case, I’ve documented in the recipe Description how to generate the necessary file, the easiest way being to temporarily authorize VueScan on the Mac (or VM) that runs AutoPkg such that the licence information file will be placed in the default location. But I have also documented what the contents of that licence file are, since it is a simple text file that you can construct and locate anywhere on the local system. Since the Copier processor requires that we specify the full path for the copied text file, this allows us the flexibility to let the user do this. In the FirefoxAutoconfig example, that recipe could just as easily been written to require that the user specify the path to the autoconfig.zip file as an input variable, but is probably cleaner as it was written. In the case of my recipe, since the filename of the licence information file that VueScan requires starts with a dot (and thus makes the file invisible in the Finder by default), I decided to allow the recipe user to make that a visible file if they wished, hence the RC_FILE input key. The (second) Copier processor simply renames the file to the required value when placing it in /Users/Shared within the package root.

To be clear, because this is a process I only need to do once a year, I could have suffered along doing it manually or I could have used a dedicated package creation tool (e.g., WhiteBox Packages, Jamf Composer, munkipkg, The Luggage) or even the pkgbuild tool in macOS (which all the other tools leverage to do their work). But since I was comfortable with how AutoPkg works, I took advantage of its ability to build arbitrary packages to make my future deployments easier.

AbletonLive

Recently, I needed to deploy Ableton Live 10 Lite. Conveniently, Tim Sutton had already written a download recipe for version 9 that could easily be adapted for the current version. But I noticed that the download recipe did not include code signature verification. So since Tim has changed jobs since he wrote that recipe and is unlikely to update (or want to maintain) it, I decided I would adapt his download recipe for version 10 and release it in my repo, adding further child recipes to support my deployment needs. Let’s examine the download and pkg recipes in my repo.

Here are the processor steps I use to create the package installer for Live:

1. URLTextSearcher [grabs the latest version number from a web page on the Ableton web site]
2. URLDownloader [downloads the latest installer]
3. EndOfCheckPhase [allows anyone running the recipe with the --check option to gracefully exit]
4. CodeSignatureVerifier [verifies that the app downloaded is codesigned]
5. FileFinder [grabs the name of the app from the disk image for use in subsequent steps]
6. Copier [copies the app from the disk image to a location in the cache]
7. PathDeleter [eliminates the SoundCloud extension from the app bundle – more on this in a moment]
8. AppPkgCreator [wraps the edited app in a pkg installer]
9. StopProcessingIf […we didn't need to build a new package]
10. PathDeleter [delete extra copies of the (rather large) app created as a part of the automation]

Let me highlight some of the details, both of Tim’s original recipe and my extensions thereof.

Ableton provides a page on their web site that lists the latest minor version of their two most recent versions. They also use a very uniform download URL scheme, such that, if you know the version number, you can determine the URL for the download. So Tim’s original recipe leveraged that and grabbed the version number from the web page.^[5] His recipe also gave the user the option to download the 32-bit or 64-bit version. In my update, I eliminated the 32-bit option (since it won’t be useful for very much longer) and added the ability to specify which major version you wanted. In many ways, I’m trying to future-proof the recipe, such that when Version 11 is released, you might only need to change your override to make it work.

The FileFinder-Copier-PathDeleter processor sequence was necessary because of an issue with the app bundle. When I simply used AppPkgCreator to create the package, it failed with the following error: pkgbuild failed with exit code -6: [/BuildRoot/Library/Caches/com.apple.xbs/Sources/Bom/Bom-194.2/FSObject/BOMFSOArchInfo.c:477] file OAuth2Client is corrupt: slice for <cputype 117440512, subtype 50331648> extends beyond length of file. (218168 > 218145)

The key in that message is that pkgbuild, the command line tool behind AutoPkg’s ability to build packages, found a problem building the package. I re-downloaded just in case, and it was not a case of a failed or corrupted download. Because the problem was with pkgbuild, that meant any other tool that would try to build a package from that app (like the ones I mentioned earlier) would have the same problem. In searching the #autopkg channel on the MacAdmins Slack, I saw that Stephen Warneford-Bygrave also had this problem back with Version 9 of Live and found that deleting the SoundCloud extension in the app bundle allowed the package to be built. This is what led to the steps before the AppPkgCreator processor.

The FileFinder processor was new to me. It helped solve a problem caused by the fact that the recipe can download any of the five variants of Live (and in one of two major versions). I want the app that we deploy to have the same name as the one that we downloaded, but the Copier processor requires that you supply the app name for the target; you can’t just do like you might using cp on the command line. In a recipe that had fewer options, I could have just hard coded the matching value. In this instance, FileFinder can determine that name for me. Since there is only one app in the disk image downloaded, I can use FileFinder to find the app name by specifying a simple *.app globbing match within the disk image; it will return the filename of the app as the variable dmg_found_filename, which I can then use in the Copier processor. Once I’ve got the app copied, I can remove the SoundCloud extension using the PathDeleter processor.

When I make the package installer from the modified app, I simplified things by using the AppPkgCreator processor. More commonly, you would use this processor on an app still contained in a disk image, but it works on any app you can point to, including ones extracted from an archive or already copied from a disk image (like we did in this case).

I would not normally include the last two processor steps, but Ableton Live is close to 2 GB in size, even in the “Lite” edition. The extra copies of the app generated by the recipe (particularly on a non-APFS drive) could start filling your storage quickly. So I chose to “clean up after myself” and delete both the app I copied (and then modified) and the scratch area that AppPkgCreator used to make another copy of the modified app for bundling into the package. The StopProcessingIf processor is the only processor in AutoPkg that is conditional. The reason I used it here is because I chose to be aggressive with what I cleaned up with PathDeleter; when the recipe is run and there is no new package to build, the payload directory is not created, so trying to delete it throws an error. Thus, StopProcessingIf causes this recipe to stop if there is no new package to build.

Long after I finished working on the pkg recipe, I took a close look at the MoofIT pkg recipe for Live 9. This takes an approach more like the first recipe we looked at, using the traditional package-building processors. It has the benefit of not using as much storage space (hence the lack of cleanup steps) but has to do the work that AppPkgCreator provides for free. You can make good arguments for either approach.

More Reading/Watching

If you decide that you want to learn more about using AutoPkg as a general purpose packaging tool, Greg Neagle wrote a blog post on the topic, complete with a few examples in a GitHub repo of his. Those examples are more of the type where you already have the files you need (because you probably created them) and want to use AutoPkg to turn them into an installer.

Also, since two of our examples messed with an application bundle, I feel it is my Mac Admins duty to remind you that you can do bad things with AutoPkg. Elliot Jordan has done a few talks on this very topic, most of which have been recorded. I recommend you check out his presentation from the 2017 Mac Admins Conference at Penn State (slides or video), as it includes information about the trust verification features added in AutoPkg 1.0.0.

I hope the examples I provided in this blog post help you leverage more of the power of AutoPkg to automate things you would otherwise need to do manually.

Note 1: An earlier version of this post suggested there was only one file inside the autoconfig.zip archive required for the FirefoxAutoconfig recipe. There are multiple files and directories. Note 2: A few minor cleanups were made for grammatical and formatting reasons on 2018-11-03.

[1] If you are a Munki user, you may not need to have it create a package for you, since Munki knows how to directly install drag-and-drop apps, configuration profiles, etc. [Return to main text]

[2] Source: AutoPkg ReadMe. [Return to main text]

[3] Support is scheduled for Version 64 (regular release)—in beta as I write this—and Version 68 Extended Support Release (ESR) in July 2019. [Return to main text]

[4] It is also possible to set the read-write-execute permissions at this point, but since you set them when using PkgRootCreator, it will maintain the existing permissions if you don’t specify anything here. [Return to main text]

[5] Note that Tim’s regular expression leveraged Python’s ability to assign a value to a variable based on a part of a regular expression. I discovered this first in another recipe and have been using it in my recipe writing ever since. In Python, the form is (?P<variable_name>what-you-want-to-capture) but since a recipe is an XML file, we need to “escape” the angle brackets, since they have a special meaning in XML. Hence, we use < and > in the recipe. In this instance, the version number is assigned to version, which we can then use in subsequent steps for any processor that wants a value for version or we can insert %version% in any string where we want that value. In this particular recipe, I used it in a lot of places. I also reference this technique in an earlier blog post. [Return to main text]