Configure DataDownload Tab
Release 9.3.1 E-mail This Topic Printable Version Give Us Feedback

Introduction

The Data Download page is an optional component providing clip-zip-ship data download functionality for vector data. The data download interfaces with an ArcGIS Server Map Service and Geoprocessing service to provide the download clip/zip functionality. Note: The Data Download page can be configured for raster data, but requires customization best carried out through ESRI Professional Services. If you are interested in contracting support for raster Data Download, please contact portal@esri.com to discuss your project.



When downloading data, a user will draw the area they wish to download, select layers for download, select an output projection and format, and supply their E-Mail address. The Geoportal extension data download page will then take the data from the input fields and use them as inputs to the data download geoprocessing service. When the request has been completed, an E-Mail message will be sent to the user with an attached ZIP file of their data (or alternatively, a link to a virtual directory to pick up the file can be sent instead. This customization is discussed in 'Option: Send Link to Virtual Directory'). For more information on how to use the feature, see How to Use the Data Download Feature.



The figure below shows the primary components of the Data Download functionality and the communication interfaces among them.







IMPORTANT: The Data Download customization does not come standard as part of the default Geoportal. The sample discussed in this section should be used as a starting point to create a customized task that suits your organization's needs. To implement the task, you can use the clip-zip-ship functionality provided with the Geoportal extension installation files, or you can configure Data Download with your own geoprocessing task.



For Data Download to be functional in the Geoportal, the following must be present and configured:

Configuration and customization of python files will be necessary. For information on how to customize geoprocessing tasks using python, see GP Service example: Clip and ship,Python scripts to zip and unzip data, and the Geoprocessing Resource Center.

Configure DataDownload Tab

This section describes how to configure the Geoportal to enable data download from a custom map document using the Clip and Ship geoprocessing task. This is a sample, and may require customization and testing by your organization to suit your organization's needs. Configuring Data Download for the Geoportal entails four major steps:
  1. Create the necessary map service and geoprocessing service maps
  2. Configure the ClipAndShip task itself for use in the geoprocessing service
  3. Publish the maps as ArcGIS Server map and geoprocessing services
  4. Configure the Geoportal to use the services in the Data Download tab
For detailed descriptions of topics discussed in this document, please read the following ArcGIS Desktop Help topics. To access the first three, open ArcGIS Desktop Help and activate the "Contents" tab. Open the section on "Geoprocessing", and then the section titled "Geoprocessing with ArcGIS Server". The final section addresses how to get additional information on ArcGIS publishing and configuration topics mentioned in this document.



Create the necessary map service and geoprocessing service maps

Two map documents must be prepared for the ClipAndShip task. The documents can be named whatever suits your organization. In this guide, the first document is called "Data_for_Download.mxd", and will be used to create a map service (within ArcGIS Server) that will appear on the Download Data page of the Geoportal. The second map document will be called "ExtractData_GeoProcessTask.mxd" in this guide, and will be the basis for a Geoprocessing (GP) service created within ArcGIS Server. This GP service will be used to clip and deliver the data to the end user. The ExtractData_GeoProcessTask.mxd will have the same layers as the Data_for_Download.mxd except for the additional geoprocessing tool layer.







Prepare Map Documents
  1. Prepare Data_for_Download.mxd

    To prepare the "Data_for_Download.mxd" document, do the following:
    • Open a map document (a new one or existing one that you want to use for "Data_for_Download.mxd") in ArcMap, and add the feature layers the map will use.
    • Save the document as an mxd file and name it "Data_for_Download.mxd".
  2. Prepare ExtractData_GeoProcessTask.mxd

    • Copy the ClipAndShip.zip file from your <Geoportal Distribution CD>\Desktop Tools\Geoprocessing Tools folder and onto your local machine.
    • Unzip ClipAndShip.zip into the file system of the machine running ArcGIS Server. For example, you can unzip the zip file under C:\ drive of the Windows server running ArcGIS Server.

      Ex: C:\ClipAndShip


    • Make a copy of the map that you used for "Data_for_Download.mxd", and rename it to "ExtractData_GeoProcessTask.mxd".
    • In ArcMap, verify that the same layers you used in your "Data_for_Download.mxd" are present in the "ExtractData_GeoProcessTask.mxd" file.
    • In ArcMap, import the tool "ClipShip2" into an ArcToolBox panel. You can import the tool by opening the ArcToolbox window. Then, right click on the ArcToolbox at the top level of tools in the ArcToolbox window, and select "Add Toolbox". Browse to the folder where you unzipped the ClipAndShip.zip file, select the ClipShip2 toolbox, and click the 'Open' button.







    • In the Toolbox window, expand the "Clip and Ship version 2" toolbox name, and then expand the "Clip, Zip, and Email utilities" tool.
    • Drag and drop the "ExtractDataTool" script into the ArcMap table of contents, placing it just below the Data Frame heading (in this case, the Data Frame is named "Layers"):








    • Right-click the ExtractDataTool layer and click Open. You should see a dialog box similar to the one shown below.








    • Close the dialog box and save the file as "ExtractData_GeoProcessTask.mxd".
Configure the ClipAndShip task itself for use in the geoprocessing service

Before the tool is published, some customization needs to take place. Follow the steps below to prepare the ClipAndShip Task to be used in the geoprocessing service.



Set Polygon Parameters

  1. Open ArcCatalog and connect to the unzipped ClipAndShip file, using the "Connect to Folder" button.
  2. Navigate to the ExtractDataTool script under the "ClipShip2" toolbox, as shown below.







  3. Right click "ExtractDataTool", and click "Properties" in the pop-up menu.
  4. Highlight the "Area To Download" attribute in the "Parameters" tab.







  5. In the "Parameter Properties" for Area to Download, find the "Schema" property. Click the "..." next to it in the "Value" column. A browsing window will open.
  6. Navigate to your unzipped ClipAndShip folder, then to the "ToolData", and then to "Templates.gdb". Then, click the "ClipPolygon" feature class, and click the "Add" button.







  7. Click "OK" to close the ExtractDataTool Properties dialog box.
  8. Expand "Clip, Zip, and Email Utilities" in ArcCatalog, right click "Clip By Layer String" script, and select "Properties" from the menu.







  9. Highlight the "Clip Features" attribute in the "Parameters" tab.







  10. In the "Parameter Properties" for ClipFeatures, find the "Schema" property. Click the "..." next to it in the "Value" column. A browsing window will open.
  11. Navigate to your unzipped ClipAndShip folder, then to the "ToolData", and then to "Templates.gdb". Then, click the "ClipPolygon" feature class, and click the "Add" button.







  12. Click "OK" to close the ExtractDataTool Properties dialog box.


Configure the Email Server

The geoprocessing task should be configured with your specific email server information. You can also customize the email title and message. The tool scripts below configure the messages that are sent to users who will download data from your Portal.

Test the Script Configurations by Running the ExtractDataModel



IMPORTANT: This next step will test that you have correctly configured the data download scripts. If this test does not result in a zip file successfully mailed to your email inbox, then do not proceed to the next step. This test will fail if there are problems with how the scripts are configured, or communication problems with your smtp server.
  1. Open the mxd that has the Geoprocessing Task as a layer.
  2. In the Table of Contents in ArcMap, right click on that ExtractDataTool layer, and select "Open" from the menu. An interface with input fields should appear.
  3. Enter the information below:
    • Output Coordinate system: WGS 1984
    • Output Format: GMLSF - (.gmlsf)
    • Area To Download: Select the "Add features interactively" radio and define an extent to clip from the map
    • Use Features From: do not select this radial
    • Layers to Clip: the name of the layer you want to clip, exactly as it is shown in the ArcMap Table of contents
    • Email address: your email address
  4. After inputting the parameters, click OK.
  5. If it runs successfully, then do not save any changes. Close the map document. If it does not run successfully, then you must resolve the geoprocessing errors before proceeding. To resolve the errors, check the scripts that you edited earlier and communication with the smtp server. If you make any edits to the scripts, make sure that you save the edits and that the updates are present in the ExtractDataTool in your ExtractData_GeoProcessTask.mxd.
  6. When the script runs successfully from ArcMap, then close the map document and proceed to the next step.
Publish the maps as ArcGIS Server map and geoprocessing services

You can use ArcGIS Server manager or ArcCatalog to publish the "Data_for_Download.mxd" and the "ExtractData_GeoProcessTask.mxd" files. Please read instructions from the ArcGIS Desktop Help topic "Publishing geoprocessing services" for the detailed process to publish into ArcGIS Server.



You will publish "Data_for_Download.mxd" as a map service, and "ExtractData_GeoProcessTask.mxd" as a geoprocessing service, following the instructions below. You must be a member of the agsadmin group on the ArcGIS Server machine in order to publish the services.



Publish the Data_for_Download.mxd as a Map Service
  1. Open ArcCatalog.
  2. In the table of contents window at the left, connect to the machine hosting ArcGIS Server by expanding the "GIS Server" node and double clicking "Add ArcGIS Server".







  3. The following wizard should appear. Select "Manage GIS Services" and Click Next.







  4. At the next screen, input the URL to the ArcGIS Server, and its host name (in the example below, the ArcGIS Server machine is called "xyz"). Click Finish.







  5. Highlight the ArcGIS Server machine that you just connected to, and then move the mouse to the window to the right. With the "content" tab active in this window, right click on a blank space in the window and select "Add New Service", as shown below:







  6. A wizard will open. Populate the first window of the wizard with the name of your extract data service ("Data_for_Download" in this example). Select "Map Service" from the "Type" dropdown list. See the example below:







  7. Click Next.
  8. In the next window, browse to the "Data_for_Download.mxd" file in your ClipAndShip folder. Leave the rest of the input fields as default, except the final dropdown. Select the default ArcGIS Server cache folder location, as shown below:







  9. Click Next.
  10. In the next window, accept the defaults and click Next:







  11. In the next window, also accept the defaults and click Next:







  12. In the next window, also accept the defaults on the next panel, and click Next:







  13. Accept defaults on final screen and click Finish:







  14. Your Data_for_Download.mxd is now published as a map service.
Publish the ExtractData_GeoProcessTask.mxd as a Geoprocessing Service
  1. Open ArcCatalog if it's not already open.
  2. In the table of contents window at the left, connect to the machine hosting ArcGIS Server using a "GIS Server" connection, like you did previously when you had to publish the Data_for_Download map service.
  3. In the "content" window to the right, with the "Content" tab active, right click on a blank space in the window and select "Add New Service". A wizard will open.
  4. Populate the first window of the wizard with the name of your extract data geoprocessing service ("ExtractData_GeoProcessTask" in this example). Select "Geoprocessing Service" from the "Type" dropdown list. See the example below:







  5. Click Next.
  6. In the next screen, select the "A Map" radial button, and browse to the ExtractData_GeoProcessTask.mxd. Set the other input fields as shown below (substituting "xyz" for your ArcGIS Server machine name):







  7. Click Next.
  8. Accept defaults as shown below, and click Next.







  9. Accept defaults as shown below and click Next.







  10. Accept defaults as shown below and click Next.







  11. Accept settings on final screen and click Finish.









Refresh ArcGIS REST Cache

After publishing the two services, it is necessary to refresh the ArcGIS REST Cache so they can be accessed by your geoportal.
  1. Open a web browser and type the following (substituting your ArcGIS Server machine name for "xyz"): http://xyz:8399/arcgis/rest/admin
  2. The ArcGIS REST API Admin page should appear. Log in to the page with your ArcGIS Server admin account.







  3. After logging in, you will see the REST API Admin page. Select the "Clear Cache Options" link:







  4. On the next page, click the "Clear Cache Now" link:







  5. Logout and close the browser.


Configure the Geoportal to use the services in the Data Download tab

You will likely want to configure the Geoportal to set up a list of output file formats and a list of output projections on the Data Download tab.



The ClipAndShip configuration is located in the "<downloadData>" element in your gpt.xml file, located in the \\geoportal\WEB-INF\classes\gpt\config folder. To configure this element, do the following:
  1. Open gpt.xml file in a text editor and scroll to the "<downloadData>" element.
  2. Find the <taskUrl> element, and update it with the ArcGIS Server REST URL to your published ExtractData_GeoProcessTask geoprocessing service. To get the REST URL, to the ExtractDataTool, open a web browser and enter the following address (entering your server instead of "xyz"): http://xyz:8399/arcgis/rest/services/. The ExtractData_GeoProcessTask service should be listed in this list. Click on the "ExtractDataTool" link in the middle of the page. When the page redirects, copy the url that is in the web browser and paste it in the <taskUrl> element, as shown below.

    In this example:

    taskUrl="http://xyz:8399/arcgis/rest/services/ ExtractData_GeoProcessTask/GPServer/ExtractDataTool"

  3. Find the <mapServiceUrl> element and update it with the ArcGIS Server REST URL to the published Data_for_Download. You can get the REST URL to the map service by going to the same ArcGIS REST services link as you did for the geoprocessing service (http://xyz:8399/arcgis/rest/services/) and clicking on the link to your map service.

    In this example:

    mapServiceUrl="http://xyz:8399/arcgis/rest/services/Data_for_Download/MapServer"

  4. Find the <mapServiceType> element and update it with the value "dynamic". This represents your data download map service's type, usually tiled or dynamic. Even if your map is a tiled map, enter "dynamic" as the value, as the display map will not display correctly if this value reads "tiled".
  5. Find the <projections> element, and enter the projections that you want to support. For more information on defining a list of projections, see the "Additional projections support" information later in this Section.

    In this example:

    <projections>

    <projection key="Mercator (world)" alias="54004" resKey="catalog.download.projection.54004"/>

    <projection key="WGS 1984 UTM Zone 10N" alias="32610" resKey="catalog.download.projection.32610"/>

    <projection key="WGS 1984 Web Mercator" alias="102113" resKey="catalog.download.projection.102113"/> <projection key="WGS 1984" alias="4326" res

    Key="catalog.download.projection.4326"/>

    </projections>

  6. Find the <formats> element, and enter the supported output formats. For more information on defining a list of output formats, see the "Additional Output Format Support" later in this document.

    In this example:

    <formats>

    <format key="GML - (.gml)" alias="gml" resKey="catalog.download.outputFormat.gml"/>

    <format key="GMLSF - (.gml)" alias="gmlsf" resKey="catalog.download.outputFormat.gmlsf"/>

    <format key="MIF - (.mif)" alias="mif" resKey="catalog.download.outputFormat.mif"/>

    <format key="ACAD - (.dwg)" alias="dwg" resKey="catalog.download.outputFormat.acad"/>

    <format key="IGDS - (.dgn)" alias="dgn" resKey="catalog.download.outputFormat.igds"/>

    </formats>

  7. Save the file and close it.
  8. Restart the geoportal web application.


Additional Projections Support

To add more projections, you must copy the corresponding projection files you wish to support from ArcGIS installation folder into the ClipAndShip folder. Do this by following the steps below:

  1. Go to the Coordinate Systems folder in your ArcGIS installation folder (ex: "C:\Program Files\ArcGIS\Coordinate Systems").
  2. From the sub folders of Coordinate Systems folder, copy the projection (.prj) files and paste them into CooSys folder under ClipAndShip\ToolData folder (ex: C:\ClipAndShip\ToolData\CooSys).
  3. Open the gpt.xml file (located at \\geoportal\WEB-INF\classes\gpt\config folder) in a text editor.
  4. Add a new <projection> element into the <projections> element; the new <projection> element should include the following attributes:
    • "key": the exact projection file name. For example, if you add the projection file called NAD 1983 UTM Zone 10N.prj, key should be NAD 1983 UTM Zone 10N.
    • "resKey": used to reference a display name for this projection. This display name will show up in a dropdown list in Download Data tab. It has a naming convention that puts "catalog.download.projection." and appends the alias string. You do not have to follow the naming convention, but it is recommended in order to organize the names of your projections in the interface. Whatever new name you use in this file must be referenced as a new key in the gpt.properties file.
    • alias: an optional field and it is additional identifier for each projection. The number you can use for alias is in http://resources.esri.com/help/9.3/arcgisserver/apis/javascript/arcgis/help/jshelp/pcs.htm
    For example, in the code below, we have added the last <projection> element, for NAD 1983 UTM Zone 10:

    <projections>

    <projection key="Mercator (world)" alias="54004" resKey="catalog.download.projection.54004"/>

    <projection key="WGS 1984 UTM Zone 10N" alias="32610" resKey="catalog.download.projection.32610"/>

    <projection key="WGS 1984 Web Mercator" alias="102113" resKey="catalog.download.projection.102113"/>

    <projection key="WGS 1984" alias="4326" resKey="catalog.download.projection.4326"/>

    <projection key="NAD 1983 UTM Zone 10N" alias="26910" resKey="catalog.download.projection.26910"/>

    </projections>

  5. Open the gpt.properties file in your \\geoportal\WEB-INF\classes\gpt\resources folder
  6. Add the display names for the added projections in the "# download data" section, using your "resKey" described above as a guide. For example, we could add the key below to match the resKey in the gpt.xml file:

    catalog.download.projection.26910 = NAD 1983 UTM Zone 10 North

  7. Save the gpt.properties file and the gpt.xml file. You will have to restart the geoportal web application for you changes to take effect.



Additional Output Format Support



Shapefile and Geodatabase

It is possible to add other formats to the Format drop down list in the Data Download interface. Three common formats that many users may want are shapefile, personal geodatabase, and file geodatabase. To add these to the list of formats, paste the three new <format> entries shown below into the gpt.xml file within the <formats> element, and update gpt.properties to reference the three new resKey values. For details on what the attributes for the <format> elements represent, and specifics on updating the gpt.properties, see instructions in the "Other Formats" subsection further below:



<format key="GEODATABASE_MDB - (.mdb)" alias="mdb" resKey="catalog.download.outputFormat.mdb"/>

<format key="GEODATABASE_FILE - (.gdb)" alias="gdb" resKey="catalog.download.outputFormat.gdb"/>

<format key="SHAPE - (.shp)" alias="shp" resKey="catalog.download.outputFormat.shape"/>



Other Formats

For other output formats, the ESRI Data Interoperability extension must be installed and enabled in ArcGIS Desktop for the map service basemap and geoprocessing task basemap. Also, the Data Interoperability extension for ArcGIS Server must also be installed on the machine hosting ArcGIS Server. Finally, you will have to alter the Data Download Task python scripts to handle the additional data format. This requires some python programming knowledge. Because this customization is not included in the Geoportal extension general documentation, the Geoportal extension User Forum is a good place to find out how other users may have added additional data types for download.



If you have licensed the Data Interoperability extension and configured the python scripts to handle an additional data type, adding the data type to your Geoportal interface is comparatively simple. The steps below describe how to add additional output formats to the formats list in your gpt.xml file so they display in the list on your Geoportal's Data Download page.

  1. The Data Interoperability extension uses Feature Manipulation Engine (FME) technology from SafeSoftware. To get the information that you will need to update the list of output formats, you will need to reference the FME website. Our example below shows adding the ESRI ArcInfo format to the list.

    • Obtain the format identifier and file extension information by accessing the FEM website and clicking the link next to the format in which you are interested. This will redirect to a .pdf file giving additional details.
    • Find "Format Identifier" and "Typical File Extensions". For ESRI ArcInfo Export, format identifier is "E00" and typical file extension is ".e00".
  2. In the gpt.xml file, insert an additional <format> element with the following attributes into the <formats> element to include the supported output format.
    • "key": {Format Identifier} - ({typical file extension}). In the ArcInfo example, it should be E00 - (.e00)
    • "alias": optional - repeat the file extension.
    • "resKey": assigns the key for referencing the display name as defined in gpt.properties for this format. Should be unique. In this example, resKey is catalog.download.outputFormat.e00.
    • The new E00 entry has been added to the default list in the example below:



      <formats>

      <format key="GML - (.gml)" alias="gml" resKey="catalog.download.outputFormat.gml"/>

      <format key="GMLSF - (.gml)" alias="gmlsf" resKey="catalog.download.outputFormat.gmlsf"/>

      <format key="MIF - (.mif)" alias="mif" resKey="catalog.download.outputFormat.mif"/>

      <format key="ACAD - (.dwg)" alias="dwg" resKey="catalog.download.outputFormat.acad"/>

      <format key="IGDS - (.dgn)" alias="dgn" resKey="catalog.download.outputFormat.igds"/>

      <format key="E00 - (.e00) " alias = "e00" resKey="catalog.download.outputFormat.e00"/>

      </formats>

  3. Open the gpt.properties file in your \\geoportal\WEB-INF\classes\gpt\resources folder
  4. Add the display names for the added format in the "# download data" section, using your "resKey" described above as a guide. For example, we could add the key below to match the resKey in the gpt.xml file:

    catalog.download.outputFormat.e00= E00 - (.e00)
  5. Save the gpt.properties file and the gpt.xml file. You will have to restart the geoportal web application for you changes to take effect.

Option: Send Link to Virtual Directory

IMPORTANT: Before continuing with this customization, it is recommended that you first complete the standard Data Download customization previously discussed, as the following customization assumes critical steps have already been carried out.



The default Data Download sample sends the user an email with a zip file output from the data download clip-zip-ship process as an attachment. This solution works for small data sets, but will fail with large data sets (e.g. greater than 5 Mb) due to mail server limitations on file sizes of attachments. This document outlines a procedure with code to customize the Geoportal Data Download sample that sends the user an email with a web link to the output zip file. This custom process allows the user to request large data download packages (greater than 5 Mb). Users can pick up the zip file at their convenience without clogging up their Mail Server inbox with large file attachments. For the Geoportal hosting organization, this process allows for greater service provision and greater security control.

System Requirements

In addition to the requirements for implementing a Geoportal with the Clip-Zip-Ship (CZS) Data Download sample, the hosting organization will need to provide a web server or FTP server with a directory that will be used for posting and user-pickup of output CZS zip files.

Customization Details

Customizations were limited to changes to Clip-Zip-Ship toolbox, specifically, the sendmail.py Python script. At issue with the sample CSZ toolbox is that the zip file created is removed from the file system at the end of the geoprocessing execution. A second issue is that the name of the resulting zip file is the same for all users.



The sendmail.py Python script was edited to make a copy of the CSZ zip file with a unique file name and store it on a file system directory that will be used for hosting the Web/FTP directory for user pickup. The script appends the web-accessible URL link to the file to the email and forwards the email to the mail server for delivery to the user.

Steps to Configure

Security Recommendations

Users should not be allowed to browse the Web or FTP folder used for hosting the Clip-Zip-Ship zip file outputs. If allowed, they could pick up other user's data request packages. This can be mitigated by setting access permissions on the folder to Read-only. If some data sets need to be restricted further, consider setting up a SSL-enabled web folder that requires a user to log in with user authentication credentials when they try to access the CSZ output file from the email link.



Operations and Maintenance Recommendations

Create a scheduled process to periodically remove and delete zip files on the downloadDir folder that are older than a certain date.

Hide Data Download from Anonymous Users

If you only want registered users to be able to use the Data Download tab in your geoportal, there are three things that should be considered:
  1. The Data Download tab can be hidden from anonymous users.
  2. The download page itself can be protected from access by specifying a security constraint within \\geoportal\WEB-INF\web.xml (a commented out example is provided by default). This requires container based authentication (i.e. it's a container like Tomcat that actually protects the resource, setting up the Geoportal in single sign-on mode requires container based authentication).
  3. The ArcGIS Server service's associated with data download might also need protection.
Below, instructions for hiding the download tab from anonymous users is described. See your servlet container documentation for how to protect the download page itself, and ArcGIS Server documentation for how to protect the data download service.