More Decimal Places Do Not Automatically Mean a Better Measurement

A digital scale displays 100.00 g. Another displays 100 g. The first shows more decimal places, but the display alone does not establish that its answer is closer to the mass of the object. It tells you something about how the instrument presents a reading. To judge the measurement, you need to know how the instrument behaves against a suitable reference and under the conditions of use.

Four ideas often become tangled in the word “precise”: resolution, repeatability, bias, and accuracy. Separating them helps explain why a number can look impressively detailed and still be misleading.

A small display step is one property

Resolution concerns the smallest distinguishable change in a measurement system or its indication. A display that advances in 0.01 g increments has a finer displayed step than one that advances in 1 g increments. This may help detect small changes, but it does not guarantee that each final digit represents reliable information about the object.

An instrument can display digits produced by an imperfect sensor, a drifting calibration, or software rounding. Extra digits cannot repair those limitations. The advertised resolution should therefore be read alongside the stated measurement range and performance conditions.

Repeat the measurement

Suppose a reference object has an accepted mass of 100.00 g for the purpose of this example. One scale repeatedly gives readings of 102.01, 102.00, and 102.02 g. The readings cluster closely, but they are displaced from the reference.

A second scale gives 99.6, 100.4, and 100.0 g. Its readings are more spread out, even though their average in this small invented sample is close to the reference. Three readings are not a complete instrument evaluation; they simply make two different properties visible.

The first pattern suggests good short-term repeatability alongside a systematic offset. The second suggests more variation under the stated conditions. Calling one instrument “better” requires knowing how it will be used and what level of uncertainty is acceptable.

Conditions define the comparison

Repeatability concerns repeated measurements under specified conditions held sufficiently alike. Reproducibility concerns agreement when specified conditions change, such as operators, locations, or instruments. A result that repeats well in one quiet room may behave differently after transport or when used by several people.

That does not make repeatability unimportant. It means the claim has a scope. “The reading repeated within this range on this instrument over ten minutes” is more informative than “the measurement is precise” without further detail.

NIST's terminology guidance also cautions against using qualitative concepts loosely as numerical labels. A standard deviation can quantify the spread of repeated results. It should be identified as that statistic, including the conditions and data behind it, rather than simply called “the accuracy.”

Calibration supplies a reference relationship

Calibration establishes how indications relate to reference values under specified conditions. An adjustment may be a separate operation. A label saying a device was calibrated is not a promise that it remains suitable indefinitely or in every environment.

NIST does not prescribe one universal recalibration interval for every instrument. Stability, required performance, contractual or regulatory requirements, and environmental conditions can all matter. An arbitrary annual sticker is not a substitute for understanding the measurement process.

For a home example, a scale on a soft mat and the same scale on a rigid surface may receive loads differently. Repeating the reading ten times without correcting the setup can repeat the same setup problem. More repetition does not automatically eliminate a common source of error.

Reporting only what the evidence supports

An average can reduce some random variation, but averaging does not automatically remove bias. If every reading is shifted by approximately the same amount, their average can remain shifted. This is why both reference comparisons and repeated observations have a role.

Before trusting a highly detailed number, ask what was measured, over what range, with what instrument, and against what reference. Look for uncertainty or a performance statement that applies to the actual conditions. If those details are absent, the extra decimal places are presentation detail. They are not independent proof that the underlying measurement deserves more confidence.