You are here:
Geoprocessing > Geoprocessing environments > Environment categories

# About coverage precision

Release 9.2

Print all topics in : "Environment categories"

Related Topics

### What is coordinate precision?

Coordinate precision refers to the mathematical exactness of a coordinate and is based on the possible number of significant digits that can be stored for each coordinate. ESRI ArcInfo coverages are stored in either single- or double-precision coordinates. Single precision stores up to seven significant digits for each coordinate. This means that values of 4,999,999.6 units or 5,000,000.4 units are rounded off to 5,000,000 units when stored in single precision. Double precision stores up to 15 significant digits—typically 13 to 14—and retains a mathematical precision of much less than 1 meter at a global extent. Note that mathematical precision does not, by itself, define accuracy. It is, however, a major factor in coverage resolution.

### When to use double precision

It is probably best to use double precision if the coordinate resolution of a coverage must be maintained past six significant digits. Here are some of the conditions that meet this requirement:
• Your coverage requires a high-level of accuracy—for example, a parcels layer.
• You need to use a map projection whose coordinate values exceed the available coordinate precision.
• You are changing datums—for example, from NAD27 to NAD83.

How does precision work?

The computer can only discern a limited number of decimal places, depending on the precision being used. For single-precision coordinates, the computer assumes that values such as 1.2345678 and 1.23456789 are equal, because numbers beyond the seventh digit are ignored. You might imagine a small halo surrounding each coordinate that is equal to the resolution of coordinate storage on the computer. Coordinates with overlapping halos are seen by the computer to represent the same location.
Coverages stored with either single, or double precision coordinates can be used interchangeably. For example, you can display single and double precision coverages over each other, overlay them, merge them, and so on. This capability, however, does not replace the need to be clear and deliberate about how you encode, store, and manage each coverage. Base your decision of which type of precision to use on the desired level of coordinate accuracy to be maintained for each coverage.
The coordinate systems for many map projections use large coordinate values. For example, State Plane and UTM coordinates contain values in the 2 million to 6 million range. In these cases, you can use double-precision coordinates to maintain accuracy of less than one unit, extending beyond the decimal point.
As mentioned, double-precision coverages can store up to 15 significant digits. This is sufficient to map any point on the earth to better than a millimeter of accuracy. However, double-precision coverages require additional disk space for storing coordinates. Thus, it may be worthwhile to store coverages that require high levels of accuracy, such as parcels, in a double-precision coordinate system and coverages needing less accuracy, such as soils, in a single-precision coordinate system.

Precision rules

You can define the coordinate precision of new coverages and derived coverages by two rules: creation and processing.

### Creation rule

The creation rule specifies the precision with which to create all new coverages. Anytime a new coverage is created, the coordinate precision of the coverage is defined by the current creation rule. Single precision is the default. Thus, if you want to create double-precision coverages, you must set the precision for coverages to DOUBLE.

### Processing rule

The processing rule specifies the precision with which to create all derived coverages. When a coverage is derived from one or more existing coverages, such as the result of the Buffer or Update tools, the coordinate precision of the derived coverage reflects the current processing rule. For example, the output coverage precision can be the highest of a set of input coverages. Thus, if all input coverages are in a single-precision coordinate system, the output coverage will be single precision; if at least one of the coverages is double precision, the output coverage is double precision. By setting LOWEST for the precision of derived coverages, the precision of derived coverages will be taken from the input coverage with the lowest precision. Thus, if all input coverages are in double precision except one that is in single precision, the derived coverage's precision will be single.
To create double-precision coverages regardless of the precision of the input coverages, set the processing rule to DOUBLE before you begin.

Please visit the Feedback page to comment or give suggestions on ArcGIS Desktop Help.
Copyright © Environmental Systems Research Institute, Inc.