Jitter is a digital timing issue that affects digital devices, such as DVD players and CD transports. High levels of jitter can result in a decrease in sound quality, and in worst cases, audible artifacts. High-end digital audio products aim to minimize jitter in any way they can, sometimes by rebuilding the audio signal completely.
If there isn't a way to sync the clock inside the source to the clock inside the receiver / preamp, jitter is likely. No such mechanism exists with HDMI, making connections with it much more likely to have jitter as compared to optical or coax.
Certain implementations of USB DACs can nearly eliminate jitter.
For more info on jitter, check out the Wikipedia page.
Read HomeTheaterReview.com's reviews of the Ayre QB-9 USB DAC and the Benchmark DAC 1 PRE.