Macro Processor Overview

for fun and profit ? Bob Eager 28 October 2023 Macro Processing 1 28-Oct-23

What is a macro processor? A very useful software tool, invented over 60 years ago but still relevant In its simplest form, a macro processor is a program that copies a stream of text from one place to another, making some kind of systematic set of replacements as it does so It processes one or more inputs (the source text), generating one or more outputs (the replacement text) This is done using rules (macros) defined by the user; these can be simple or extremely complex Macro processors may be embedded in other programs; they may also be standalone programs, which means that it is easy to use them for processing any kind of text at all More powerful macro processors add a degree of intelligence into their operation, instead of blindly making replacements of one piece of text by another For example, they can deal with text where the format of what is to be replaced may have a complex, perhaps repetitive, structure. Macro Processing 2 28-Oct-23

What can it be used for? The possibilities are endless Saving a lot of typing Changing the format of a file so that it is maintained in a user- friendly form, then transformed into what is actually needed For example, firewall rules Defining undefined opcodes for an assembler, but defining new mnemonics that expand to a data definition Extending an existing programming language Extracting selected data from a file Program parameterisation (e.g., a parameter might determine whether debugging statements are to be included in a program) Data format conversion, for example: To format and convert files marked up with HTML As part of a microprogam translator for a bit slice computer implementation Macro Processing 3 28-Oct-23

Examples of macro processors (1) Many assemblers have, historically, included a macro processor, to ease the writing of large programs or to encode system calls DEC (originators of the now defunct PDP series of machines) had particularly good macro assemblers IBM have used them from very early on, although they are somewhat clunky The C language, and others, have incorporated a preprocessor with simple macro facilities In the case of C, this is relatively basic, but it is used extensively for system header files, as well as at the user level Macro Processing 4 28-Oct-23

Examples of macro processors (2) General purpose macro processors first generation These are standalone programs that process input and output connected directly to files (or pipes, or ) One of the earliest was GPM (General Purpose Macrogenerator), in 1965 Relatively limited, but did its job There have been more recent implementations in C Macro Processing 5 28-Oct-23

Examples of macro processors (3) General purpose macro processors second generation STAGE2 was designed as the second stage of the Mobile Programming System Written in a language called FLUB (First Language Under Bootstrap) Could implement itself on a new machine (or using a bootstrap written in FORTRAN) Line oriented m4 - extension of an earlier macro processor, m3! Written by Dennis Ritchie for an AP-3 minicomputer Widely available, particularly on UNIX{-like} systems Used for many configuration purposes Relatively primitive Macro Processing 6 28-Oct-23

Examples of macro processors (4) General purpose macro processors second generation (continued) ML/I (Macro Language I) Developed as part of Ph.D. work by Peter Brown Stream oriented Unlike many others, there is no special character introducing a macro call There is no distinction between the definition phase and the processing phase, and new definitions can be made on the fly We will use ML/I in our examples Macro Processing 7 28-Oct-23

Example of macro processing By default, ML/I copies from standard input to standard output, so let s see how it works in a command window We will label input (with numbers) and output, to make things clearer; output to standard error will also be labelled 1) Output) 2) Output) Hello, world! Hello, world! Second line Second line Here we have a rather over-engineered copying program! Macro Processing 8 28-Oct-23

Now lets use one of the 20 or so built in macros in ML/I: 3) MCNOTE Hello Error) Hello Error) detected in line 3 of source text Notice that nothing goes to the output, but the side effect is a message to standard error MCNOTE is a call on a built-in macro (or operation macro) called, well, MCNOTE, whose replacement text is null, but which has the side effect of generating a message Macro Processing 9 28-Oct-23

Another operation macro is MCINS, which defines an insert 4) MCINS %. There is no output as MCINS only has a side effect of defining % as an insert marker, with . as its closing delimiter Inserts allow calculation of expressions as well as insertion of all kinds of things 5) %S2. Output) 5 This inserts the value of system macro variable 2 (which is the source line number) Macro Processing 10 28-Oct-23

Another important operation macro is MCDEF, which allows us to define an actual macro 6) MCDEF Robert AS Bob 7) Robert Output) Bob No surprises here The difference between the above, and a global edit, is that ML/I works in units of atoms whole words or punctuation characters, so you get 8) Roberta did not help Output) Roberta did not help 9) ROBERT is a different word Output) ROBERT is a different word It can be seen that ML/I is also case dependent Macro Processing 11 28-Oct-23

Notice that the newline after Robert appeared after Bob, because that wasn t part of the macro; it was just some more input But the newline after MCDEF (for example) was part of the MCDEF macro, so it was absorbed; we can choose to do that too We will now define something called a skip; it s a bit like a macro, but it just stops anything inside it being processed: 10) MCSKIP MT,<> (no output again); we ll explain this later Now define another macro: 11) MCDEF Promote to NL 12) <The %A2. is now %A1.! 13) > And call it: 14) Promote Robert to Managing Director Output) The Managing Director is now Bob! Notice the replacement of Robert by Bob, too Macro Processing 12 28-Oct-23

What happened: We defined a macro called Promote, with secondary delimitersto and then newline (keyword NL) The first argument (between macro name and delimiter 1 (to) is Robert The second argument (between delimiters 1 and 2) is Managing Director Replace everything between macro name (Promote) and final delimiter (newline) with the text between < and > Evaluate what is there for further calls: The calls on inserts (%A1. and %A2.) get replaced with arguments 1 and 2 respectively Evaluation is repeated, so Robert becomes Bob until there is no more replacement to be done The macro name is also known as delimiter 0 (zero) Macro Processing 13 28-Oct-23

ML/I is very enthusiastic, and will replace stuff wherever it can; to control this, we use skips Define a skip (with at least a name, but usually one or more secondary delimiters) with MCSKIP: 15) MCSKIP Farage This will delete all occurrences of Farage: 16) Who is Farage Output) Who is More usually, it would be something like: 17) MCSKIP Delete ; 18) Delete this and this; but not this Output) but not this To just protect text from macro calls, use the Text option, which copies arguments unchanged: 19) MCSKIP T,[ ] 20) [This was really done by Robert] Output) This was really done by Robert Macro Processing 14 28-Oct-23

To copy across the delimiters too, use the Delimiter option: 21) MCSKIP DT, COMMENT ; 22) COMMENT Robert wants comments unchanged; Output) COMMENT Robert wants comments unchanged; This is how we protect the arguments to MCDEF from being evaluated straight away (usually for the replacement text; we use a skip such as the one we used earlier, the brackets < and >; they get stripped away by omitting the D option, but we allow them to be nested by using the Matched option: MCDEF Promote to NL <The %A2. is now %A1.! > Macro Processing 15 28-Oct-23

ML/I has about 20 built in macros, called operation macros: Some change the name environment (i.e. create and delete macros) MCDEF, MCSKIP, MCINS Some are functions operating on text (e.g. to get its length) MCSUB, MCLENG Some create new macro variables, make decisions, etc. MCGO, MCNOTE, MCPVAR, MCALTER These provide enough to do practically anything you want ML/I works in units of text called atoms: A single punctuation character A word surrounded by white space or punctuation Hence Robert and Roberta being distinct macro names Macro Processing 16 28-Oct-23

Macro variables hold numeric or textual information, and can be created as desired; they have simple numeric names: System variables (S1, S2, ) control ML/I Permanent variables (P1, P2, ) allow communication between macros Temporary variables (T1, T2, ) provide workspace inside complex macros Character variables (C1, C2, ) pass text between macros Startlines are imaginary characters inserted (optionally) at the start of each line; they allow macros to be line rather than stream based Macro Processing 17 28-Oct-23

Conclusion General purpose macro processors are surprisingly useful, and can save a lot of work For a copy of these slides, see: http://macros.bobeager.uk That page includes other useful links Macro Processing 18 28-Oct-23