Bores Signal Processing - Introduction to DSP - basics: quantisation

When the signal is converted to digital form, the precision is limited by the number of bits available.

The diagram shows an analogue signal which is then converted to a digital representation - in this case, with 8 bit precision.

The smoothly varying analogue signal can only be represented as a 'stepped' waveform due to the limited precision.

Sadly, the errors introduced by digitisation are both non linear and signal dependent.

Non linear means we cannot calculate their effects using normal maths.

Signal dependent means the errors are coherent and so cannot be reduced by simple means.

This is a common problem in DSP. The errors due to limited precision (ie word length) are non linear (hence incalculable) and signal dependent (hence coherent). Both are bad news, and mean that we cannot really calculate how a DSP algorithm will perform in limited precision - the only reliable way is to implement it, and test it against signals of the type expected. The non linearity can also lead to instability - particularly with IIR filters.

The word length of hardware used for DSP processing determines the available precision and dynamic range.

Uncertainty in the clock timing leads to errors in the sampled signal.

The diagram shows an analogue signal which is held on the rising edge of a clock signal. If the clock edge occurs at a different time than expected, the signal will be held at the wrong value. Sadly, the errors introduced by timing error are both non linear and signal dependent.

A real DSP system suffers from three sources of error due to limited word length in the measurement and processing of the signal:

• limited precision due to word length when the analogue signal is converted to digital form
• errors in arithmetic due to limited precision within the processor itself
• limited precision due to word length when the digital samples are converted back to analogue form

These errors are often called 'quantisation error'. The effects of quantisation error are in fact both non linear and signal dependent. Non linear means we cannot calculate their effects using normal maths. Signal dependent means that even if we could calculate their effect, we would have to do so separately for every type of signal we expect. A simple way to get an idea of the effects of limited word length is to model each of the sources of quantisation error as if it were a source of random noise.

The model of quantisation as injections of random noise is helpful in gaining an idea of the effects. But it is not actually accurate, especially for systems with feedback like IIR filters.

The effect of quantisation error is often similar to an injection of random noise.

The diagram shows the spectrum calculated from a pure tone

• the top plot shows the spectrum with high precision (double precision floating point)
• the bottom plot shows the spectrum when the sine wave is quantised to 16 bits

The effect looks very like low level random noise. The signal to noise ratio is affected by the number of bits in the data format, and by whether the data is fixed point or floating point.