autoconf-2.64: the phantom macro menace

While I should write it down on autotools mythbuster in a most generic fashion, since I found this today, I wanted to write down these notes for other developers. With autoconf-2.64 there can be a problem with “phantom macros” as in macros that are called, but seem not to produce any code.

In particular, I noticed a configure failure in recode today. The reported error out is the following:

checking for flex... flex
checking lex output file root... lex.yy
checking lex library... -lfl
checking whether yytext is a pointer... yes
checking for flex... (cached) flex
./configure: line 10866: syntax error near unexpected token `fi'
./configure: line 10866: `fi'

Looking at the actual configure code, you can easily see what the problem is around line 10866:

if test "$LEX" = missing; then
  LEX="$(top_srcdir)/$ac_aux_dir/missing flex"
  LEX_OUTPUT_ROOT=lex.yy
  else


fi

In sh, you probably know already, “else fi” is invalid syntax; but what is the code that produces this? Well, looking at configure.in is not enough, you need to also check an m4 file for the package:

# in configure.in:
ad_AC_PROG_FLEX

# in m4/flex.m4
## Replacement for AC_PROG_LEX and AC_DECL_YYTEXT
## by Alexandre Oliva 
## Modified by Akim Demaille so that only flex is legal

# serial 2

dnl ad_AC_PROG_FLEX
dnl Look for flex or missing, then run AC_PROG_LEX and AC_DECL_YYTEXT
AC_DEFUN(ad_AC_PROG_FLEX,
[AC_CHECK_PROGS(LEX, flex, missing)
if test "$LEX" = missing; then
  LEX="$(top_srcdir)/$ac_aux_dir/missing flex"
  LEX_OUTPUT_ROOT=lex.yy
  AC_SUBST(LEX_OUTPUT_ROOT)dnl
else
  AC_PROG_LEX
  AC_DECL_YYTEXT
fi])

So there are calls to AC_PROG_LEX and AC_DECL_YYTEXT macros, so there should be code in there. What’s happening? Well, maybe you remember a previous post where I listed some user-advantages in autoconf-2.64 :

Another interesting change in the 2.64 release which makes it particularly sweet to autotools fanatics like me is the change in AC_DEFUN_ONCE semantics that makes possible for macros to be defined that are executed exactly once. The usefulness of this is that often times you get people to write bad autoconf code, that instead of using AC_REQUIRE to make sure a particular macro has been expanded (which is usually the case for macros using $host and thus needing AC_CANONICAL_HOST), simply call it, which would mean the same check is repeated over and over (with obvious waste of time and increase in size of the generated configure file).

Thanks to the AC_DEFUN_ONCE macro, not only it’s possible to finally define macros that never gets executed more than once, but also most of the default macros that are supposed to work that way, like AC_CANONICAL_HOST and its siblings, are now defined with that, which means that hopefully even untouched configure files will be slimmed down.

Of course, this also means there are more catches with it, so I’ll have to write about them in the future. Sigh I wish I could find more time to write on the blog since there are so many important things I have to write about, but I have not enough time to expand them to a proper size since I’m currently working all day long.

Indeed the two macros above are both once-expanded macros, which means that autoconf expand them before the rest of the now-defined macro. Now, the solution for this is using M4sh properly (because autoconf scripts are not pure sh, they are M4sh, which is a language in the middle between sh and m4). Instead of using if/then/else, you should use AS_IF; indeed changing the above macro to this:

AC_DEFUN(ad_AC_PROG_FLEX,
[AC_CHECK_PROGS(LEX, flex, missing)
AS_IF([test "$LEX" = missing], [
  LEX="$(top_srcdir)/$ac_aux_dir/missing flex"
  LEX_OUTPUT_ROOT=lex.yy
  AC_SUBST(LEX_OUTPUT_ROOT)dnl
], [
  AC_PROG_LEX
  AC_DECL_YYTEXT
]])

allows autoconf to understand the flow of the code and produces the proper sh code (this is true sh code) in the final configure file:

checking for flex... flex
checking for flex... (cached) flex
checking lex output file root... lex.yy
checking lex library... -lfl
checking whether yytext is a pointer... yes

(see how the two checks for flex are both up the list of checks?).

Unfortunately there are more problems with recode, but at least this documents the first problem, which I’m afraid is going to be a common one.