LaTeX (pronounced "laytech") is the most widely used system for typesetting mathematical and complex (cross referencing, bibliographical citations and indexes) documents. Like HTML, it is a markup language: you describe what you want in a plain text file, and then use a program to convert this description to PostScript or PDF.
You must edit LaTeX source files using a text editor. A particularly good choice is Emacs, which includes support (AUCTeX) for editing LaTeX files.
To learn how to use LaTeX, begin by consulting LaTeX: from Quick and Dirty to Style and Finesse, by Tony Roberts. More information can be found at LaTeX Project.
The standard way to use LaTeX is via the shell (command interpreter), which is possible on Linux, Mac OS X, and Windows/Cygwin. But it is also possible to use a special-purpose application, such as TeXShop on Mac OS X, or MiKTeX on Windows (without Cygwin).
On Linux, Mac OS X or Windows/Cygwin, if LaTeX is not already installed, you can install the teTeX distribution, or the newer TeX Live distribution. On Mac OS X, teTeX may be installed via Fink, and both teTeX and TeX Live may be installed using Mac Ports.
The Perl script Latexmk ("mk" stands for "make") automates the process of running LaTeX enough times to resolve cross references, bibliographic citations and indexing. It can also use supporting programs (see below) to turn LaTeX's output (a DVI file) into PostScript or PDF.
On Mac OS X, I recommend viewing PDF files using the open source application Skim, which is capable of redisplaying a PDF document as the file changes. To turn this feature on, check the "Check for file changes" box in the Sync tab of the Preferences window. Sadly, Adobe Reader is incapable of tracking file changes.
A moderately involved example of what can be achieved using LaTeX can be found in the file example.tex, which uses the files macros.tex (some general purpose settings and macros that I've found useful) and commands.tex (some macros for typesetting formal language theory).
To turn example.tex into PDF on Linux, Mac OS X or Windows/Cygwin using latex, run the following commands (after copying all three files into a new directory):
latex example.tex latex example.tex dvips -Ppdf example.dvi ps2pdf example.ps
(latex had to be run again, because there were undefined cross-references after the first run.)
To turn example.tex into PDF on Linux, Mac OS X or Windows/Cygwin using pdflatex, run the following commands (after copying all three files into a new directory):
pdflatex example.tex pdflatex example.tex
(pdflatex had to be run again, because there were undefined cross-references after the first run.)
On Linux and Mac OS X, you can use the program smlnjtrans to generate LaTeX transcripts of SML/NJ sessions.