My build command for Emacs 29 Snapshots

2023-03-21 23:59:59

Emacs 29 Build Command

TBH, I've been building emacs "manually" for months.

I have scripts for this, of course. Naturally, they are broken. It turns out not to be all that hard to get me to bail out from fixing (even) my own scripts, and revert to building Emacs by hand. Here's what I ran this evening, to make a new snapshot from the emacs-29 branch:

  (export BIF=/d/emacs-build/install \
	  SLUG=29-$(git rev-parse --short HEAD); \
   (./autogen.sh \
	&& ./configure \
	       --with-modules \
	       --without-dbus \
	       --with-native-compilation \
	       --without-compress-install \
	       --with-tree-sitter \
	       CFLAGS=-O2 \
	&& make install -j 20 \
		NATIVE_FULL_AOT=1 \
		prefix=${BIF}/emacs-${SLUG}) \
   | tee ${BIF}/emacs-${SLUG}.log)

Complexity is a dangerous attractor

I let my various "helper" programs get way too complicated. Pretty soon I didn't feel like playing with them at a time when they did not do what I wanted.

I think it may have been somewhat of an intended consequence: I have been curious how much complexity/flexibility I, myself, as someone with a high degree of enthusiasm for building GPL compliant distributable binary releases of Emacs from source for use on Windows, would "enjoy" in my code to automate this thing.

Other stuff was broken too

There are currently two key scripts within the Emacs (both in admin/nt/dist-build).

The first is "build-deps.py". This creates the "deps" and "deps-src" zips, of binaries and sources for Emacs dependencies on Windows. Several hard-coded lists tell Emacs which MSYS/MINGW packages to include and how to recursively find the dependencies and source distributions of each, as presently installed in the build environment. I recently got this working again and am very glad to be able to keep using it. Here are my present diffs vs Emacs 29 as of 8b6a0de964d:

diff --git a/admin/nt/dist-build/build-dep-zips.py b/admin/nt/dist-build/build-dep-zips.py
index 71105a071ec..8ce2be1a681 100755
--- a/admin/nt/dist-build/build-dep-zips.py
+++ b/admin/nt/dist-build/build-dep-zips.py
@@ -1,6 +1,6 @@
 #!/usr/bin/python3

-## Copyright (C) 2017-2023 Free Software Foundation, Inc.
+## Copyright (C) 2017-2022 Free Software Foundation, Inc.

 ## This file is part of GNU Emacs.

@@ -20,13 +20,11 @@
 import os
 import shutil
 import re
-import functools
-import operator

 from subprocess import check_output

 ## Constants
-EMACS_MAJOR_VERSION="28"
+EMACS_MAJOR_VERSION="29"

 # This list derives from the features we want Emacs to compile with.
 PKG_REQ='''mingw-w64-x86_64-giflib
@@ -39,7 +37,10 @@
 mingw-w64-x86_64-librsvg
 mingw-w64-x86_64-libtiff
 mingw-w64-x86_64-libxml2
-mingw-w64-x86_64-xpm-nox'''.split()
+mingw-w64-x86_64-gmp
+mingw-w64-x86_64-xpm-nox
+mingw-w64-x86_64-tree-sitter
+mingw-w64-x86_64-sqlite3'''.split()

 DLL_REQ='''libgif
 libgnutls
@@ -51,7 +52,11 @@
 librsvg
 libtiff
 libxml
-libXpm'''.split()
+libgmp
+libXpm
+libXpm-noX4
+libtree-sitter
+libsqlite3-0'''.split()


 ## Options
@@ -103,7 +108,7 @@ def ntldd_munge(out):

         ## if it's the former, we want it, if its the later we don't
         splt = dep.split()
-        if len(splt) > 2 and "msys64" in splt[2]:
+        if len(splt) > 2 and "mingw64" in splt[2]:
             print("Adding dep", splt[0])
             rtn.append(splt[0].split(".")[0])

@@ -114,26 +119,44 @@ def ntldd_munge(out):
 ## Packages to fiddle with
 ## Source for gcc-libs is part of gcc
 SKIP_SRC_PKGS=["mingw-w64-gcc-libs"]
-SKIP_DEP_PKGS=frozenset(["mingw-w64-x86_64-glib2"])
+SKIP_DEP_PKGS=["mingw-w64-glib2" "mingw-w64-ca-certificates-20211016-3"]
 MUNGE_SRC_PKGS={"mingw-w64-libwinpthread-git":"mingw-w64-winpthreads-git"}
 MUNGE_DEP_PKGS={
     "mingw-w64-x86_64-libwinpthread":"mingw-w64-x86_64-libwinpthread-git",
     "mingw-w64-x86_64-libtre": "mingw-w64-x86_64-libtre-git",
 }
+SRC_EXT={
+    "mingw-w64-freetype": ".src.tar.zst",
+    "mingw-w64-fribidi": ".src.tar.zst",
+    "mingw-w64-glib2": ".src.tar.zst",
+    "mingw-w64-harfbuzz": ".src.tar.zst",
+    "mingw-w64-libunistring": ".src.tar.zst",
+    "mingw-w64-winpthreads-git": ".src.tar.zst",
+    "mingw-w64-libxml2": ".src.tar.zst",
+    "mingw-w64-ncurses": ".src.tar.zst",
+    "mingw-w64-openssl": ".src.tar.zst",
+    "mingw-w64-pango": ".src.tar.zst",
+    "mingw-w64-python": ".src.tar.zst",
+    "mingw-w64-sqlite": ".src.tar.zst",
+    "mingw-w64-xpm-nox": ".src.tar.zst",
+    "mingw-w64-xz": ".src.tar.zst",
+}

 ## Currently no packages seem to require this!
 ARCH_PKGS=[]
 SRC_REPO="https://repo.msys2.org/mingw/sources"


-def immediate_deps(pkgs):
-    package_info = check_output(["pacman", "-Si"] + pkgs).decode("utf-8").splitlines()
+def immediate_deps(pkg):
+    package_info = check_output(["pacman", "-Si", pkg]).decode("utf-8").split("\n")
+
+    ## Extract the "Depends On" line
+    depends_on = [x for x in package_info if x.startswith("Depends On")][0]
+    ## Remove "Depends On" prefix
+    dependencies = depends_on.split(":")[1]

-    ## Extract the packages listed for "Depends On:" lines.
-    dependencies = [line.split(":")[1].split() for line in package_info
-                    if line.startswith("Depends On")]
-    ## Flatten dependency lists from multiple packages into one list.
-    dependencies = functools.reduce(operator.iconcat, dependencies, [])
+    ## Split into dependencies
+    dependencies = dependencies.strip().split(" ")

     ## Remove > signs TODO can we get any other punctuation here?
     dependencies = [d.split(">")[0] for d in dependencies if d]
@@ -149,18 +172,16 @@ def extract_deps():
     print( "Extracting deps" )

     # Get a list of all dependencies needed for packages mentioned above.
-    pkgs = set(PKG_REQ)
-    newdeps = pkgs
-    print("adding...")
-    while True:
-        subdeps = frozenset(immediate_deps(list(newdeps)))
-        newdeps = subdeps - SKIP_DEP_PKGS - pkgs
-        if not newdeps:
-            break
-        print('\n'.join(newdeps))
-        pkgs |= newdeps
+    pkgs = PKG_REQ[:]
+    n = 0
+    while n < len(pkgs):
+        subdeps = immediate_deps(pkgs[n])
+        for p in subdeps:
+            if not (p in pkgs or p in SKIP_DEP_PKGS):
+                pkgs.append(p)
+        n = n + 1

-    return list(pkgs)
+    return sorted(pkgs)


 def download_source(tarball):
@@ -208,7 +229,13 @@ def gather_source(deps):
         ## Switch names if necessary
         pkg_name = MUNGE_SRC_PKGS.get(pkg_name,pkg_name)

-        tarball = "{}-{}.src.tar.gz".format(pkg_name,pkg_version)
+        ## src archive is usually a .tar.gz
+        if pkg_name in SRC_EXT.keys():
+            src_ext = SRC_EXT[pkg_name]
+        else:
+            src_ext = ".src.tar.gz"
+
+        tarball = "{}-{}{}".format(pkg_name,pkg_version,src_ext)

         download_source(tarball)

@@ -257,7 +284,7 @@ def clean():

 if( args.l ):
     print("List of dependencies")
-    print( deps )
+    print( extract_deps() )
     exit(0)

 if args.s:

The other one, build-zips.sh, has the rather simpler problem of sequencing the Emacs build. I had some issues with that. I wrote something to help automate my sequence for validating file sets and preparing them for life on a GNU FTP server. When a problem or two came up during Emacs 28 builds I got the help I needed from the mailing lists, emails, etc. Look, I really don't know when I decided to rewrite it, okay? Anyway, long story short, since my stuff was working pretty okay, quite a lot of the time, I don't know exactly when I stopped being able to make build-zips.sh do its thing, but it (like my own scripts) fail me when building the emacs-29 branch.

In conclusion, it's going fine

As I hope I've said at the top of the post, it all really does boil down to this one command that's not very hard to type/remember/find.

For my next amazing trick

For now, in terms of scripting my Emacs 29 builds, I'm still noodling. I consider the complex problem domain of logging. Wither, I ponder, printing only the very most interesting messages, as created by configure/make/etc? Whence summarizing the rest, perchance in a tidy way. Especially, I muse: I have done too much, already?

C.f. scripts and blah in the emacs-28 folder