Forth and Productivity

I've been in a spell of maximum effortless productivity lately programming in FORTH and wanted to jot down how it feels before the enchantment goes away.

After writing this, I realize this post requires at least some familiarity with what FORTH is, and in the end I present some demonstrative FORTH code which definitely requires FORTH knowledge. I have left some comments describing what is happening, but it's better if you know at least the absolute basics of FORTH.

What a surprisingly wonderful language FORTH is. To me at least. I normally enjoy programming in these "strange and exciting but not really useful" languages like the odd APLs and LISPs, and FORTH originally seemed to fit this category to me.

These languages usually have something radically different about them that makes them interesting to try out. It feels fresh to program in them because of the jarring paradigm shift they introduce. And in each excursion into one of these languages, something of an enlightenment is experienced by the programmer. For the lay-programmer who cuts his teeth in imperative or object oriented languages, functional programming is what would typically introduce this feeling. When I was discovering functional programming myself, I remember talking to a couple friends about some of the most, at the time, mind blowing concepts and chuckling in enthusiasm at their shocked befuddlement. "What do you mean you can't just change your variables? How can you achieve anything?" towards immutability. "Infinite recursion? Come on. That's impossible." towards tail recursion.

I love those moments. I'm a programmer by heart and sometimes by trade¹ and these interesting, wonderful aspects about programming affirm the mysticism I find in programming. Specifically the kind of programming I escape to after facing the banality abundant in the field these days. I find this quote by Charles Moore² best expresses it:

You have to get inside the world of software, the world of imagination; and you create your own world. You create your own problems and so it is almost frightening, the amount of power you’ve got. I’ve always had the sort of dream—I’ve always favored software over hardware. The hardware is the nuisance you have to put up with in order to have software. If you want to talk about a religious aspect of FORTH, it is to say, “Do you need hardware?” Can you conceive of a way of representing these ideas, of making these castles in the air without any particular underpinning? I’m sure that chips will get down to molecular size, but there is still going to be that matter at the core of things. Can you take energy fields, somehow, and weave them to make software? Maybe that’s what the layers and layers of operating systems and languages are doing, taking you so far away from the hardware that you forget it is even there.

From the section "Interview with Charles Moore" in FORTH Dimensions - Volume 5, Number 2

I'll cut short the rant about how modern software sucks, you've heard it a million times, but its impossible not to do so when a programming language like FORTH exists. By no means is it the industry revolutionizing, loud and emoji-spangled-readme-d VC-backed programming language of the 21st century, but the lack of abundant praise on the internet³ is definitely not a sign of FORTH being a boring old whatever language. It is anything but. And it is so in the exact manner that the common "Keep It Simple, Stupid" crowd would lose their heads towards. Oh the false dichotomy of KISS…

FORTH is both simple, and, in the most earnest sense of the word, wonderful. It is FUN. Remember when you got into programming because you thought you would MAKE things? and that the process would be FUN? No? then I don't think you'll find the rest of this interesting. Go away. Otherwise, uh yeah. FORTH is a good time.

Why is FORTH fun? For starters, it lets you do anything you want. No guard-rails. You can define 2 to be 2.5. You can make your own if statement or case or for or what have you. Its got this dictionary where you can put things in. What things? Whatever you want. Functions, Values, Arrays, Garbage; It's where your definitions go. You can accidentally write over a function if you overflow an array. Is this level of freedom dangerous? Not inherently, but you would probably write over functions more in FORTH than in other languages. And what if you do? The system probably crashes. Skill issue.

I adore this about programming in FORTH because of the second, and probably most important aspect of FORTH: Transparency.

FORTH unapologetically exposes its guts to the programmer—or should I say, Operator. The distance between a key-stroke and the avalanching effects of intricate inner machinations resulting in a computation is tiny. The abstraction layer is microscopical. Dammit it doesn't even have a parser. Imagine that! A programming language venerated for its expressive power doesn't have a big ole parser with 32 levels of AST transformation. FORTH has no syntax. It has one dimension to it: the word. Normally whitespace delimited, but, characteristically, you can change the delimiter. Hell, the functions for parsing and evaluating are exposed to you, and, get this, you are encouraged to abuse these.

As an aside, I find it extremely refreshing that the general community around FORTH encourages bending the language for your own purposes. Charles Moore himself says this in his "Programming a Problem-Oriented Language"

Make variables as GLOBAL as possible. Why not? You can save some space and clarify your requirements. For instance, how many Is, Js and Ks do you need? In most cases a single copy in COMMON would suffice (you have to declare them, remember, and may as well put them in COMMON); you can redefine it locally if you must; and it is of interest that you must.

You think that's wild? Get a load of this:

Use words with mnemonic value. Unfortunately what is mnemonic to you may not be mnemonic to me; and I'm the one who judges. Also unfortunately, mnemonic words tend to be long, which conflicts with:

Use short words. You don't want to type long words, and I don't want to read them. In COBOL this means avoid dashes and avoid qualification, though both can be useful upon occassion[sic].

So let me suggest a compromise: abbreviate in some consistant[sic] fashion and stick to your own rules. I can probably figure out the rules you're using. You might even mention them in a comment.

I'm not even quoting out of context. In fact, you should read the whole thing; it's worth it.

There's many instances of similar sentiments by the old masters in FORTH communities. My point is, not a lot of programming communities uphold such values. Which other programming community wants you to go crazy with your tricks? Imagine the C community going "Oh yeah! Abuse those macros!" No way. Community attitude is very important. It is what you'll interact with when you don't want to spend your morning writing a PNG encoding library. That day, it's the community that decides whether or not you'll find an ultra-modularized, ultra-configurable, monadic, type-safe library, or one with an AbstractImageEncoderBuilder which you'll have to parameterize with PNGEncoderBuilder<ArrayList<U64>>, or maybe a blazingly fast, zero-copy, async compatible one, or maybe one that makes 8 million malloc calls upon a call to encoderInit(). The community colors your experience of the language whether you like it or not, and I like that in FORTH it's very much a wild west.

Here's the thing though. You don't really find yourself pulling in third party code in FORTH all the time⁴. It's kinda easy to just write a bunch of functions. And it's fun. The situation is that either the problem is straight forward and well specified enough that your fingers autonomously emit the library you want, or it's interesting enough that you get to have fun exercising FORTH's expressive power. The principle in FORTH is that you keep things simple, and you program a language that expresses your problem and then use it to solve the problem. I am excited by these opportunities to write code. I can't say I'm always excited–sometimes its just tedious and I reach for some kinda C-FFI– but more so than not compared to when I'm using other languages.

This is because FORTH code is terse and easy to crank out. I've attempted expressing that while under another one of these inspired spells in this post, but I'll try getting at the gist of it briefly here. FORTH is easy to write because all the data passing is implicit due to the stack. You don't annotate and say "Here's the function XLERB I'm calling ( with the arguments Foo, Bar, and Baz. Thank you. );" You just say "XLERB" with the arguments on the stack. XLERB's definition doesn't have to have a signature either. It just assumes what it wants is on the stack. This whole thing may not seem like much but your code ends up very dense and terse with no notation for passing the data around to noise it all up.

The stack thing takes getting used to though, especially to those of us who'd been brought up on ALGOL-y syntax. I can't really remember the exact moment it clicked for me, and I wouldn't even proclaim it clicked. I just slowly got used to keeping track of things on the stack until I developed habits and principles about how to organize stack items. It also takes some practice to know when some problem is better solved with the aid of variables or the stack, but it's not the end of the world if you haven't got there yet. Lord knows it took me a while. You usually know after one too many >R's and ROTS and stack diagrams/comments.

Once you start getting it though, oh boy. It feels nice. Sometimes you don't even notice it. I find myself going "Oh wait this would've taken so much longer in [INSERT LANGUAGE], lol." often. It's all the features coming together. The stack for the implicit data-passing. The transparent minimal-abstraction-ness of it. The lack of syntax. The way these all elegantly come together at the cost of a couple wrinkles here and there. It's wonderful. It's lovely to program in.

I've gone through this whole post without showing any example code because honestly you've just gotta write FORTH to feel it. I didn't want to come off like I'm showing some cherry-picked examples throughout and making the claim that those examples prove that everyone should use FORTH everywhere, and all the time. In the end, there's a time and place. Right tool for the job and all that. However, I will leave off with maybe a handful snippets of fun I've had with FORTH. By fun, I mean times where I wrote something that wasn't just a straight-forward chain of function calls (though that concatenative aspect is also incredibly valuable to me), but had something clever to it which also highlighted FORTH's strengths.

This one is from my ongoing attempt to write a text editor. I was programming the Gap buffer, and after putting in the cursor movement primitives, I wanted to make one I can parameterize. There's some things I need to clear up beforehand. I have these global variables to store the start(S) and ends(E) of the gap(G) and the buffer(B). They are normal FORTH VARIABLE's and are prefixed with > to denote that they're indices. For example >BS is the variable containing the address of the Start of the Buffer.

I begin by defining a utility for early returns:

\ For the uninitiated, : begins a definition by the name of the next
\ word (BEQ in this case) and ; ends the definition. POSTPONE means
\ when this definition is called, compile the next word. For example,
\ POSTPONE = here means when BEQ is called, it puts = into wherever it
\ is called. IMMEDIATE modifies BEQ so that instead of getting
\ compiled into whatever definition it is invoked in, it is
\ immediately executed during compilation instead, like a macro.
\ In effect, BEQ is a macro that expands to "= IF EXIT THEN".

: BEQ POSTPONE = POSTPONE IF POSTPONE EXIT POSTPONE THEN ; IMMEDIATE

This is a macro that encodes the idea of "If these two are equal, there's no work to do".

I then use it to bound the cursor movement, i.e, If the cursor (represented as the gap) is at the left-most position, and you press the left arrow key, you don't go anywhere:

\ For the uninitiated, @ fetches the value at a memory location, C!
\ stores a character at a memory location, and +! increments a value
\ at a memory location.  Variables are memory locations that leave
\ their address on the stack when invoked.

: P>  >GE @ >BE @ BEQ >GE @ C@ >GS @ C! 1 >GS +! 1 >GE +! ;
: P<  >GS @ >BS @ BEQ -1 >GS +! -1 >GE +! >GS @ C@ >GE @ C! ;

P> moves the cursor (point) this way -->, and P< this way <--.

Now, I wanted to have a general purpose function for moving the point N times each direction; Right if it is a positive number, and left if it is a negative number.

\ ['] puts the execution token of the next word on the stack. EXECUTE
\ executes the execution token on top of the stack.

: NP DUP 0< IF -1 * ['] P< ELSE ['] P> THEN
     SWAP 0 ?DO DUP EXECUTE LOOP DROP ;

I don't know about you but I find this definition really satisfying. It was satisfying to write too. I was testing and playing with other parts of the text editor when i said to myself "I'm tired of writing P> P> P> P>", and this definition kinda flowed out of my fingers.

Here's another similar example. It's from the same project, but it has to do with a function called INS which inserts a character at the point:

: INS >GS @ C! 1 >GS +! FULL? IF BUFGROW THEN ;

This definition inserts a character supplied on the stack into the buffer and adjusts the gap accordingly. It conditionally grows the buffer if it fills up.

Then I ran into the same problem of not wanting to type "CHAR A INS CHAR B INS CHAR C INS". I'd rather "S" This is sample text" PASTE". And again, the following definition came about naturally.

: PASTE  OVER + SWAP ?DO I C@ INS LOOP ;

None of these examples are mind blowing in their algorithmic brilliance. I also admit, to somebody that doesn't speak FORTH, all the DUP's and @'s and +!'s are probably a turn off. It definitely takes getting used to. What I love is how elegant the results tend to be. How what would be a superfluous overstated solution in a C-like language is naturally expressed in a handful of words. How the language allows and encourages you to mess with its mechanisms to get what you want in the case of ['] for instance. I have been having a blast with it.

I'm sorry if this post turned out gushy. It's more of a love letter than anything, and that from an amateur FORTH user⁵. FORTH rewards your curiosity and tinkering spirit like no other language. I've found much joy in programming in it daily as I have in learning about it. Jonesforth was the most rewarding and mind blowing read I've done in the last year. The surrounding literature around forth is always interesting. The community, especially the people down at r/Forth are kind and inspiring. And there's a myriad interesting FORTH related projects about. Amidst the currernt uninspiring and sloppy vibe surrounding the computing community, I'm happy to have started the new year discovering FORTH.