jazzace.ca

Anthony’s Mac Labs Blog

📦 Running an AutoPkg Recipe with the --pkg Option

Posted 2018 November 12

AutoPkg offers an option when you run a recipe from the command line that is mentioned exactly once in the Wiki: the --pkg (or -p) option.[1] Yet that option that can be very useful for recipe authors and users alike. This post is the missing documentation for that option.

What Does It Do? (TL;DR)

The --pkg option allows you to direct AutoPkg to use a package or disk image that you have on hand (which you probably manually downloaded) instead of downloading whatever the URLDownloader processor in your recipe would normally retrieve. URLDownloader will treat the supplied item as an updated version (i.e. it sets the output variable download_changed to True). If your recipe chain doesn’t use URLDownloader, your recipe can still access the path to the download using variable substitution (i.e. %PKG%).

Usage

The syntax is:

autopkg run --pkg /path/to/pkg_dmg_or_archive recipename.recipetype

or

autopkg run recipename.recipetype --pkg /path/to/pkg_dmg_or_archive

You may use -p instead of --pkg. You may only use this option when running a single recipe (not a list or multiple recipes).

Why Would You Use It?

There are three main reasons to use the --pkg option when running an AutoPkg recipe:

Let’s address each one of these in order.

Newer Version Available

“Why hasn’t AutoPkg found the latest version? I can see it on the vendor’s web site!” This is probably the most common usage of the --pkg option—it’s even described in the AutoPkg FAQ. When the vendor uses more than one mechanism to make new software available, sometimes these are not updated simultaneously. For example, there might be a new version available for download on their web site but their Sparkle feed hasn’t been updated yet, and the AutoPkg recipe uses the Sparkle feed to determine the current version. Or perhaps you have been provided with a beta version to help fix a bug and you would like to get that out to your testing group using the recipes that feed your software distribution system (e.g., Munki). Whatever the reason, you can manually download the item in the format that the recipe would normally expect (e.g., dmg, pkg, zip) and specify that path with the --pkg option. For example, MuseScore is provided as an app wrapped in a disk image. If I’ve downloaded the latest version of MuseScore to my Downloads folder, and I want to run the pkg recipe for it, I would use the following command:

autopkg run MuseScore.pkg --pkg ~/Downloads/MuseScore-2.3.2.dmg

AutoPkg will follow all the steps of the recipe, including code signature verification and possibly building a package, but when it gets to the URLDownloader processor, it will skip downloading and will return values as if the item specified by --pkg was just downloaded as a new item.

PackageRequired

Sometimes, you just can’t automate a particular download. Common reasons are: the vendor requires that you provide login credentials to obtain the software, there are too many redirects by the web site for AutoPkg to handle, the site’s Terms of Service do not allow automated downloads, and the recipe author (probably you) hasn’t figured out how to automate the download but still wants to use the other features of AutoPkg (like building a package installer or putting the software into your distribution system). Sometimes, you can automate a download, but the product is updated so infrequently, it’s not worth the effort. Regardless of your reason, the --pkg option works well in this case, in partnership with the PackageRequired processor.

PackageRequired is a processor that has no arguments. The code is so straightforward, you can understand it even if you can’t write a line of Python (see for yourself). It simply checks to see if you have used the --pkg option when running the recipe. If the user didn’t use it (or used it incorrectly), the recipe fails. So the main benefits of the PackageRequired processor are error trapping and self-documentation. Basically, if your recipe chain[2] doesn’t use URLDownloader, it probably needs PackageRequired.

Probably the most common product that requires this methodology is Audacity. The Terms of Service of the download provider prohibit automated downloads, so once the community found this out, the download recipe was voluntarily pulled and the other child recipes were changed to support using the --pkg option. The pkg recipe from Armin Briegel’s recipe repo is easy to understand. It has three processor steps: PackageRequired, CodeSignatureVerifier, and AppPkgCreator. The recipe confirms that the user supplied something with the --pkg option, verifies the code signature of the app within the disk image, and creates a pkg installer based on the app inside the disk image. Note that the path associated with the --pkg option is assigned to the variable PKG.[3] So, in both the CodeSignatureVerifier and AppPkgCreator processor steps, the recipe uses %PKG% to specify the path to the downloaded disk image.

New Recipe Testing

In the process of developing a new recipe, you may need to download the desired software a number of times, especially if you develop the download recipe first, then child recipes such as pkg, munki, or jss. (Each time you run a different recipe type, AutoPkg will build a new cache for it, thus initiating a new download.) You can avoid the time and bandwidth involved with re-downloading by using the --pkg option. Here’s a common workflow I use:

  1. Create a download recipe for the software product in the Recipes directory (usually ~/Library/AutoPkg/Recipes).
  2. Run it with trust verification off (or from an override you create) until you are satisfied that it downloads what you are intending. Run it one more time to confirm that it correctly handles the case where there is no new update available.
  3. Create a child recipe (e.g., pkg) in the same recipe directory. Run it with trust verification off (or from an override you create) with the --pkg option; since the download is in the cache from the download recipe you were just testing, you can specify that path as the parameter for the --pkg option. For example:
    autopkg run -vv SoftwareTitle.pkg --pkg ~/Library/AutoPkg/Cache/com.github.githubusername.softwaretitle.download/downloads/SoftwareTitle.dmg
    Once you are satisfied that your child recipe is doing what you are intending, run it one more time to confirm that it correctly handles the case where there is no new update available.
  4. Repeat the previous step for any further child recipes.

When I am done that testing, I generally trash all the caches created during testing and run the terminal child recipe without the --pkg option twice, just to confirm that it behaves correctly when it actually has to download something.

One More (Obscure) Use Case

For completeness, here is one obscure usage of the same functionality that does not require using --pkg at the command line. The URLDownloader processor has 8 input variables (only url is required). One of these variables is PKG. One could choose to specify a value for PKG, either hard-coded into a recipe you authored as an argument for URLDownloader or as an input variable in a recipe override. The value assigned will be honoured by URLDownloader as the path to the downloaded item. One advantage of this method is that you could include such a recipe in a list of recipes to be run (AutoPkg will throw an error if you use the --pkg option in a run command that includes more than one recipe). The major limitation of this method is that the path to the download (including the filename) needs to be the same every time you run it to make it as efficient as the --pkg option. Having said all this, no publicly-posted recipe uses this ability. So while it does work, the use case appears to be theoretical.

A Final Learning

I ended up rewriting one recipe chain I used an example here because of a (beneficial) property of the URLDownloader processor. When you download something with URLDownloader, it returns a number of output variables, including pathname. When you refer to the location of the download in subsequent processors, you want to leverage that variable, including when you specify a path using string substitution (i.e. %pathname%). For example, an earlier version of my MuseScore.download recipe told the CodeSignatureVerifier processor that the app to be verified was located at %RECIPE_CACHE_DIR%/downloads/%NAME%.dmg/MuseScore 2.app. That’s an entirely conventional path for the download, so it worked perfectly well when AutoPkg downloaded the item itself. However, when I used the --pkg option, AutoPkg would refuse to mount the disk image (or would mount a previously downloaded one) because the disk image was not in the expected location. Since URLDownloader correctly sets the value of pathname to the path of the downloaded or supplied item, using %pathname%/MuseScore 2.app works with both downloads and the --pkg run option. This is another reason why you might want to test the recipes you develop with the --pkg option—you can then be assured it will work when used with that option.

I hope that this better explains the utility of the --pkg option for running AutoPkg recipes. If you think some of this information should make its way into the AutoPkg wiki, please provide me direct feedback on Twitter (@AnthonyReimer) or on the MacAdmins Slack (@jazzace or #autopkg channel).

[1] Reference: AutoPkg Wiki FAQ [Return to main text]

[2] By “recipe chain” I mean all the parent-child relationships that need to be resolved to run the recipe you specify. For example, most jss recipes have a parent pkg recipe which in turn has a parent download recipe. Thus, the download-pkg-jss recipes for a particular product form a chain. I’m open to using a better term, but the community has not defined one yet. [Return to main text]

[3] Technically, this is what the --pkg option does every time. Since URLDownloader has an optional input variable named PKG, it gets assigned as the downloaded item that way. [Return to main text]