Geoprocessing > Automating your work with models > Model concepts and terms
Intermediate data and the scratch workspace |
|
|
Release 9.3
Last modified January 26, 2009 |
Print all topics in : "Model concepts and terms" |
When you run a model, output data is created for each process in the model. Some of the data created is of no use after the model is run since it was only created to connect to another process that creates new output. Such data is called intermediate data. To determine if any data variable contains intermediate data, right-click the data variable—if there is a check mark next to Intermediate, then the data is considered intermediate. If there is no check mark, the data is not intermediate and will not be deleted.
Intermediate data will only be deleted when
- You execute the model using its dialog.
- You execute the model from the command line or a script.
Intermediate data will not be deleted if you run the model from the ModelBuilder window. This allows the has-been-run state of the model to be saved between sessions (which means that each time you open a previously run model in ModelBuilder, you will not need to rerun the entire model).
In ModelBuilder, all derived data elements are automatically flagged as intermediate.
If you want to delete intermediate data after running a model in the ModelBuilder window, click the Model menu and click Delete Intermediate Data.
Delete intermediate data
Not all derived data is automatically flagged as intermediate. A data variable that is in/out cannot be flagged as intermediate, since the input data exists.
Learn more about in/out derived data.
NOTE: By default, ModelBuilder assumes that all derived data (with the exception of in/out derived data) is considered intermediate, even derived data that is the final result of a model, as shown in the figure below.
It's rare that you would want your final output to be deleted because it means the model has no permanent result. In most cases, these final variables are marked as model parameters, and model parameters cannot be intermediate data.
Learn more about model parameters.
If you want to preserve intermediate data, you have three choices:
- Always run the model from ModelBuilder, since intermediate data is never deleted when the model is run from ModelBuilder. Conversely, never run the model using its dialog, from the command line, or in scripting. This is an impractical solution, since there is nothing to prevent you from running a model using any of these methods.
- Make the data variable a model parameter. Since model parameters are exposed to the user in the tool dialog, the expectation is that they want to preserve the data, so model parameters cannot be flagged as intermediate.
- Remove the intermediate flag by right-clicking the data variable and clicking Intermediate. The Intermediate option is unavailable if the data variable is a model parameter, an in/out data variable, or project data.
Auto-generated locations
When you open a tool from ArcToolbox or inside ModelBuilder and provide the necessary input datasets, the location (the workspace pathname followed by the dataset name) of the derived output data is automatically generated. This autogenerated name is constructed using the following logic:
- The location in the Scratch Workspace environment variable. If this variable is blank, then
- The location of the Current Workspace environment variable. If this variable is blank, then
- The location of the first project data you add to the tool. If this location is read-only, then
- Your system temp directory.
The output data variable will contain the autogenerated name regardless of whether the variable eventually becomes intermediate data, managed data, or a tool parameter.
Learn more about auto-generated names.
Setting the scratch workspace
Here are some things you should be aware of regarding writing intermediate scratch data:
- If the scratch workspace is an ArcSDE workspace, there is additional computational cost communicating with the database. You should always avoid having an ArcSDE connection as your scratch workspace.
- As with any computer application that writes data, you need to have enough space on disk to write the data. Avoid disks that are nearly full.
- If the scratch workspace is somewhere other than a disk on your computer, there is computational overhead involved with writing data across the network.
NOTE: It is recommended that you use a file geodatabase for your scratch workspace.
The in_memory workspace
You can write intermediate data to your computer's memory rather than to a workspace on disk. Writing intermediate data to memory is extremely fast compared to writing to disk.
NOTE: Only tables and feature classes (points, lines, polygons) can be written to the in_memory workspace. The in_memory workspace does not support extended geodatabase elements such as subtypes, domains, representations, topologies, geometric networks and network datasets. Only simple features and tables can be written.
To write to your computer's memory, use the pathname in_memory, as follows:
in_memory\results
You can think of in_memory as a special geodatabase workspace. You can use in_memory to write intermediate data.
NOTE: Do not set your scratch workspace environment to the in_memory workspace. Use in_memory only for outputs that you know to be simple features and tables.
The example below shows using in_memory with the Frequency tool.
Rules about using in_memory workspace are
-
NOTE: Only tables and feature classes (points, lines, polygons) can be written to the in_memory workspace. The in_memory workspace does not support extended geodatabase elements such as subtypes, domains, representations, topologies, geometric networks and network datasets. Only simple features and tables can be written.
- You cannot create feature datasets or folders in the in_memory workspace.
- Data written to memory will be deleted when you exit the application (such as ArcMap or ArcCatalog), so only use in_memory for intermediate (temporary) data.
- You cannot edit or modify features contained in an in_memory feature class.
-
NOTE: Do not set your scratch workspace environment to the in_memory workspace. Use in_memory only for outputs that you know to be simple features and tables.
Managing the in_memory workspace
When you write data to the in_memory workspace, you are consuming your computer's internal memory, and if you write too much data, you will consume the memory an application such as ArcMap needs to execute efficiently. Eventually, the application will slow down, causing all tasks (such as refreshing the map) to become very slow.
At some point, the advantages of using the in_memory workspace are offset by the slowdown of the application. When this point is reached depends on many factors such as the amount of available internal memory, the amount of virtual memory (swap space), the amount of data to be written to the in_memory workspace, and the number of applications open on your computer. In general, you should avoid using in_memory for large datasets and use a workspace on disk instead.
When using the in_memory workspace, you should delete your intermediate data in ModelBuilder when no longer needed so you free up memory.
Learn how to delete intermediate data.
You can also use the Delete tool to delete data in the in_memory workspace. You can delete individual tables and feature classes, or you can delete the entire in_memory workspace, which will delete all of its tables and feature classes. An empty in_memory workspace will be recreated after you've deleted it.
In_memory source location
A table or feature class based on data written to the in_memory workspace will have the source location of GPInMemoryWorkspace, as shown in the following illustration from ArcMap:
The long string of characters enclosed in brackets {} following GPInMemoryWorkspace is a unique identifier created and used by ArcGIS.