Using the -shared option to GCC, you can compile your C source file into a self-contained shared library (with the suffix *.so). Once this is done, OCTAL will be able to load and use your machine. Here's an example of a GNU make command that will properly compile your machine:
squaregen.so: squaregen.c machine.h gcc -O3 -g -pedantic -shared squaregen.c -o squaregen.so
OCTAL's build system will be able to automatically compile and install machines for the end user, so a distribution pack consisting of the machine source and documentation should suffice.
Note that you no longer need to link to util.o, since all host utility functions are made accessible through the Package. See machine.h for more information on the Package.