========================
== Experimental Emacs ==
========================
Corwin's Emacs Blog


Simple Emacs 29 Windows Build Script

Simple Emacs 29 Windows Build Script

ABSTRACT

TL;DR This "literate" post generates a shell script to make a new Emacs 29 and get it ready to publish.

The program the post generates, although simple, is actually kinda sweet.

It doesn't rebuild Emacs unless there have been changes upstream since the last successful build. It doesn't require any special caches or data-files. It also has some limited resume capabilities, in that it can skip completed steps along the way, to continue on a later failing bit without repeating earlier things that worked fine. It stores output in an "install" folder. Each sub-folder under install contains a ready-to-run version of Emacs 29, filed by git short revision ID. It also creates an "upload" folder. Sub-folders under upload have you need to upload Very Nearly Official™ versions of GNU Emacs (assuming you've got your authorization and credentials and such setup with FSF/GNU).

You also need, as may perhaps be obvious, a working MINGW64 (or MINGW32) environment with a bunch of stuff installed, plus a local check-out of the emacs-29 branch. (skip to: Build Environment)

Introduction

My "that looks reasonable to him?!?" one-liner pipeline command from the other day having worked, I was left staring at the remaining parts of the packaging and pre-release upload process. Having tried, by now, many manual and semi-automated means by now for some or all vs various branches, I had quite a lot of my own prior art to dig though. As I've said, I'm not looking for a large "script all the windows things" project right now (although "tiny" programs are always good if I think I can use them, especially if the uses are a few weeks apart and I don't really have to edit it much…) Ah hm, Right. As seen from ~/.bash_history, the prospect of manually fussing with what I'd recently been doing between build and upload (e.g those from ~) was unattractive.

In a surprising twist, it was straight-forward to dump some choice examples into a "trival" shell script.

While not hereby submitted for any beauty award, this not only "works on my system", it provides a nice straight-forward canvas for me to describe building and packaging Emacs for Windows (again).

But this time.. as a literate shell script! queue trumpets

Literate, you say?

Literate programming, if you haven't run across the concept before, is generally the idea of putting your programs and their documentation (and perhaps even the data) together, interwoven, so as to surround the code of (and/or to generate) our program with descriptions of how they work, why we made them to work that way, and so on.

Program Description

As mentioned, this blog post (or, the source for it, if you are looking at an HTML version), contains everything need to create package-emacs-29.sh.

The actual programs' code starts here, with some boiler-plate leads with a short description of the program

#!/usr/bin/bash
# package-emacs-29 - Create an Emacs 29 Snapshot Pre-Release for Windows
#
# Copyright 2023 Corwin Brust <corwin@bru.st>
#
# This program is distributed under the terms of the GNU Public
# License version 3 or (at your option) any later version.
#

Build Environment

To keep this post from getting out of hand, in terms of length, I'm not discussing installing MSYS/MINGW, nor installing Emacs dependences within MSYS. Nevertheless, you will need to have done these things for this script to work: it must run under either the MINGW32 or MINGW64 command shell, depending on whether or not you are building a 64 bit Emacs.

Additionally, this script looks for certain folders and files in specific places on the system where it is run. Since the program, after the initial "commentary", starts with setting up variables for (among other things) where to find things on the local system we can discuss the detail in terms of the script's source; for now, let's look at a picture:

  c:\
    emacs-build\
	    upload\
	    install\
	    dist\
	      emacs-29-deps.zip
	      emacs-29-deps-src.zip*
	    git\
	      emacs-29\
		admin\nt\dist-build\*
		  build-deps-zips.py*
		  build-zips.sh*

  * not a dependancy, probably mentioned in the blog post

This represents a directory structure on the hypothetical Windows build machine of a once a future Emacs dev, imagined. My environment looks a little different, as we will shortly see.

Other Even More Actually Official Windows Packaging Scripts

With folder structure setup something like the picture, the .py and .sh scripts in admin/nt/dist-build/, at the bottom of the list, represent the tooling provided with the Emacs' source for building Emacs and packaging releases binary releases for Window.

The python script works, with some crude patching, for Emacs 29, I think. It's job is to create zip files containing any DLLs that we need to ship with Emaccs but which aren't built when we build Emacs ("deps.zip"), and the sources (generally including sources of sources) for those ("deps-src.zip").

The program this post produces (package-emacs-29.sh), does much the same job of fthe second script pictured above: build-zips.sh. It does not do all of the jobs that program could/did do, although it does do several things build-zips.sh never has.

The build-zips.sh program did (but this program won't):

  • make releases (rather than a "snapshot" pre-release)

  • make a pre-release (rather than a "snapshot")

  • allow selecting a branch (vs being "hard-coded" to emacs-29)

  • take command line options (-c is clean-up: nice. none of that, here)

  • put "snapshot-LIVVYY-MM-DD" into the names of snapshots it makes

This program does (but build-zips.sh hasn't ever):

  • put the git short version ID into the names of the snapshots it makes

  • create only snapshot (not release or pre-release) builds

  • zip the sources used to build

  • create the executable self-installer

  • create sha256 hash summaries

  • GPG sign

  • create GNU FTP upload descriptor files

  • safe for repeated use

  • up-to one release/install per revision to the emacs-29 branch

Both programs make (full) emacs-$SLUG.zip, emacs-$SLUG-no-deps.zip, and leave intact an "installed" Emacs, as a by product. This program can install a set of Emacs builds side-by-side, up to one per commit to the emacs-29 branch.

Neither program has (or likely will) setup your MSYS/MINGW environment or its constituent requisite dependencies, nor setup the folder structure pictured above apart from creating folders for something the program is actually building, etc.

Aside: ask me sometime about the important the difference between a snapshot and a release/pre-release build is, sometime. No doubt if I could remember/understand the exact detail here my/upstream existing scripting would be that much closer to "works perfectly enough for me".

VARIABLES

After the "sh-bang" line and top-of-program commentary, the program code starts by creating a number of variables. Our goal is to evaluate as much of the (starting state of) the build environment and our "project" or "task" (e.g. "emacs 29 snapshot") up front, before doing anything else. If nothing else, this let's us make reports. Let's discuss this line-by-line:

Make your changes here :)

FROM

A fully qualified path to a folder, which must contain a folder called "git". That "git" folder must, in turn, contain a folder called "emacs-29", which must be the root of a checkout of the emacs-29 branch.

FROM=/c/users/corwi/emacs-build

DEPS

Fully qualified path to emacs-29-deps.zip: a "flat" archive of "extra" DLLs, meant to be unpacked atop the bin/ folder contained in emacs-$SLUG-no-deps.zip, it works equally well unpacked atop the installed emacs created by make install.

DEPS=$FROM/deps/emacs-29-deps.zip

MV

Emacs major version. This is to make other vars more readable, I guess. I dodn't expecting twiddling it to make the script build Emacs 30 or create release builds.

MV=29

SRC

The folder where we'll find emacs sources.

SRC=$FROM/git/emacs-$MV

EV

The full version string, from Emacs' top-level Makefile

EV=$(grep '^version=' $SRC/Makefile | perl -pne 'chomp;$_=[split/=/]->[-1]')

TO

The folder under which we collect installed and packaged versions of Emacs. A folders will be created under "upload" and under "install", so those sub-directories must exist.

TO=/d/emacs-build

SLUG

This is used to formulate release filenames. Is the major version (29), plus a hyphen, plus the short revision ID from the current branch (expected to be emacs-29) in the SRC folder.

SLUG=29-$(cd $SRC; git rev-parse --short HEAD);

EB

Emacs basename used to formulate release filenames. It is "emacs-" followed by SLUG.

  EB="emacs-${SLUG}"

  IB="${TO}/install"
  IN="${IB}/${EB}"
  UP="${TO}/upload/${EB}"
  NO="${UP}/${EB}-no-deps.zip"
  FU="${UP}/${EB}.zip"
  OE="${IB}/emacs-${EV}-installer.exe"
  SE="${UP}/${EB}-installer.exe"
  SZ="${UP}/${EB}-src.zip"

  DIR=/d/projects/emacs-build/new-hotness/emacs-29-pre.directive
  PF=/c/users/corwi/emacs-build/foo.txt

  echo "FROM: ${FROM}
    MV: ${MV}
   SRC: ${SRC}
    EV: ${EV}
    TO: ${TO}
  DEPS: ${DEPS}
  SLUG: ${SLUG}
    EB: ${EB}
    IB: ${IB}
    IN: ${IN}
    UP: ${UP}
    NO: ${NO}
    FU: ${FU}
    OE: ${OE}
    SE: ${SE}
   DIR: ${DIR}"

  if [[ -d $IN ]] ; then
      echo "NOTICE: build dir exits: $IN"
  else
      (cd $SRC; git clean -fxd;
       ((./autogen.sh \
	     && ./configure --with-modules \
			    --without-dbus \
			    --with-native-compilation \
			    --without-compress-install \
			    --with-tree-sitter \
			    CFLAGS=-O2 \
	     && make install -j 20 \
		     NATIVE_FULL_AOT=1 \
		     prefix=$IN
	) | tee $TO/${EB}-make.log
       )  && echo "1..OK make"
      ) || (echo "ERROR: prep upload ($?)"; exit 1);
  fi

  # pause for testing
  echo "Pausing for 10 seconds for tests."
  echo "Presss C-z then: make prove";
  echo "When finished, type: fg<RET>"
  user_input=''
  sleep 10;

  if [[ ! -d $IN ]]; then
      echo "ERROR install folder is missing: $IN"
      exit 1
  fi

  if [[ -d $UP ]]; then
      echo "NOTICE: install folder exists: $UP"
  else
      ((mkdir $UP \
	    && cd $IN \
	    && zip -vr9 $NO . 2>&1 \
	    && unzip -vd bin $DEPS \
	    && zip -vr9 $FU . 2>&1) \
	   | tee $TO/${EB}-zip.log \
	   && echo "2..OK zip files") \
	  || ( echo "FAILED ($?)" ; exit 2 )
  fi

  if [[ -r $SE ]] ; then
      echo "NOTICE: self-installer exists: $SE";
  else
  ((cd $IB \
	&& cp $SRC/admin/nt/dist-build/emacs.nsi . \
	&& makensis -v4 \
		    -DEMACS_VERSION=$EV \
		    -DVERSION_BRANCH=$EV \
		    -DOUT_VERSION=$EV \
		    emacs.nsi \
	&& mv $OE $SE) \
       | tee $TO/${EB}-esi.log \
	   && echo "3..OK executable self installer") \
	   || (echo "ERROR: creating self installer ($?)"; exit 3)
  fi


  # clean-up the build folder
  cd $SRC
  git clean -fxd | tee $TO/${EB}-clea.log

  # archive sources
  ((zip -9r $SZ . -x .git/ .git/\* \
	| tee $TO/${EB}-src.log
   ) && echo "4..OK archive sources"
  ) || (echo "ERROR: archive sources ($?)"; exit 4);

  cd $UP

  # create SHA256 sums
  ((for f in *.{zip,exe} ;
   do
       sha256sum.exe $f ;
   done) | tee $UP/${EB}-sha256sums.txt
   ) | tee $TO/${EB}-sums.log

  # sign release files
  (for f in $UP/*.{txt,exe,zip} ;
   do
       gpg --pinentry-mode=loopback \
	   --passphrase-file=$HOME/emacs-build/foo.txt \
	   --batch --yes -b $f
   done) | tee $TO/emacs-${SLUG}-sign.log

  # create upload directives
  (for f in *.{zip,exe,txt} ;
   do
       cat $DIR \
	   | perl -p -e "s/__FILE__/$f/" \
		  > $f.directive ;
   done) | tee $TO/emacs-${SLUG}-dirs.log

  # sign directives
  (((for f in *.directive ;
   do
       gpg --pinentry-mode=loopback \
	   --passphrase-file=$HOME/emacs-build/foo.txt \
	   --batch --yes --clearsign $f ;
   done)  | tee $TO/emacs-${SLUG}-sidr.log
   ) && echo "5..OK prep upload"
   ) || (echo "ERROR: prep upload ($?)"; exit 4);

  # (export SLUG=29-$(git rev-parse --short HEAD); (./autogen.sh && ./configure --with-modules --without-dbus --with-native-compilation --without-compress-install --with-tree-sitter CFLAGS=-O2 && make install -j 20 NATIVE_FULL_AOT=1 prefix=/d/emacs-build/install/emacs-${SLUG}) |tee /d/emacs-build/install/emacs-${SLUG}.log)
  # makensis -v4 -DEMACS_VERSION=29.0.60 -DVERSION_BRANCH=29.0.60 -DOUT_VERSION=29.0.60 emacs.nsi
  # (for f in *.{zip,exe} ; do sha256sum.exe $f ; done) | tee emacs-29.0.60_1-sha256sums.txt
  #  for f in *.directive ; do gpg --pinentry-mode=loopback --passphrase-file=$HOME/emacs-build/foo.txt --batch --yes --clearsign $f ; done
  # for f in *.{txt,exe,zip} ; do gpg --pinentry-mode=loopback  --passphrase-file=$HOME/emacs-build/foo.txt --batch --yes -b $f ; done
  #for f in *.{zip,exe,txt} ; do cp /d/projects/emacs-build/new-hotness/emacs-29-pre.directive $f.directive; perl -pi -e "s/__FILE__/$f/" $f.directive ; done