How to download data from public feature services with either ArcGIS Pro, or Python only

When I do demos for clients, I normally download my client's public data first and then add their data to my demo. They often ask how I got access to their data, even though they knew it was public, and they ask how I was able to download it because they also want to be able to download third-party data. So, this article shows you how I find the data, and then a few different ways the data can be downloaded.

How I find the data:

First to find the data, I use Google's site search operator, which is nice functionality provided by Google that most people do not know about - see here for more info: https://developers.google.com/search/docs/monitor-debug/search-operators/all-search-site. For example, if I want to find all data in Queensland in Australia, that is related to "fire" but I only want to search Queensland government websites, and I only want to find ArcGIS web services, then my search term in Google would be: site://*.qld.gov.au/*/rest/services fire. This will only return the results I want. I can also filter by service type, e.g. site://*qld.gov.au/*/rest/services/*/MapServer fire will filter the results to include map services only, and changing MapServer to FeatureServer, will only return feature services.

If I remove "fire" from the search term, then the results returned will show me all of the ArcGIS Server REST Services Directories for any Queensland government websites (that are indexed by Google) that end in .qld.gov.au. This is only one way I find data quickly. Another way I get data is by finding the web mapping application I am interested in, then I turn all of the map layers off, then I open the browser Developer Tools and go to the Network tab, clear the results, then turn on the map layer I want, and then I can quickly see where the data is coming from. This is how many developers quickly find web services. There are other methods I use too, however, this article is mostly about how I download the data.

How I download the data:

Now that I have found the feature service I want, I just need to go to ArcGIS Pro, then click on the Map tab, then Add Data, and Data from Path, and enter the URL for the feature service (this also works with secure feature services if a valid access token is in the URL). Then I can use the Copy Features geoprocessing tool in ArcGIS Pro to copy the data to a file geodatabase (or enterprise geodatabase). I also have the correct layer symbology from the feature service by doing it this way. This works well for small one-off tasks, however, if you want to automate this, or you do not have ArcGIS Pro, then Python is easy to use.

With Python, I download the data as GeoJSON using this Python script I wrote below. Note, this script does not use ArcPy, so anyone with Python can run this, without having access to ArcGIS software.

import json
import requests
import os

output_folder = 'output/data'

if not os.path.exists(output_folder):
    os.makedirs(output_folder)

feature_service_urls = ["https://sampleserver6.arcgisonline.com/arcgis/rest/services/Water_Network/FeatureServer"]

for feature_service_url in feature_service_urls:
    params = {
        "f": 'json'
    }

    response = requests.post(feature_service_url, params)
    result = json.loads(response.text)

    service_name = feature_service_url.split('FeatureServer')[0].split('/')[-2]

    for layer in result['layers']:
        file_name = layer['name'].lower().replace(' ','_').replace('-','').replace('__','_')
        id = layer['id']

        feature_layer_url = f'{feature_service_url}/{id}'
        query_url = f'{feature_layer_url}/query'

        params = {
            "f": 'json'
        }

        response = requests.post(feature_layer_url, params)
        result = json.loads(response.text)
        object_id_field = result['objectIdField']

        params = {
            "f": 'json', 
            "returnCountOnly": 'true',
            "where": '1=1'
        }

        response = requests.post(query_url, params)
        result = json.loads(response.text)
        no_of_records = result['count']

        print(f'{layer["name"]} ({layer["id"]}) - ({(no_of_records)} records)')
        
        records_downloaded = 0
        object_id = -1

        geojson = {
            "type": "FeatureCollection",
            "features": [] 
        }

        while records_downloaded < no_of_records:
            params = {
                "f": 'geojson', 
                "outFields": '*',
                "outSR": 4326, # change the spatial reference if needed (normally GeoJSON uses 4326 for the spatial reference)
                "returnGeometry": 'true',
                "where": f'{object_id_field} > {object_id}'
            }

            response = requests.post(query_url, params)
            result = json.loads(response.text)

            if(len(result['features'])):
                geojson['features'] += result['features']
                records_downloaded += len(result['features'])
                object_id = result['features'][len(result['features'])-1]['properties'][object_id_field]
    
            else:
                ''' 
                    this should not be needed but is here as an extra step to avoid being 
                    stuck in a loop if there is something wrong with the service, i.e. the
                    record count stored with the service is incorrect and does not match the 
                    actual record count (which can happen).
                '''
                break

        if(records_downloaded != no_of_records):
            print(f'--- ### Note, the record count for the feature layer ({layer["name"]}) is incorrect - this is a bug in the service itself ### ---')
            print('-'*50)

        output_file = os.path.join(output_folder, f'{file_name}.geojson')

        with open(output_file, 'w') as f:
            f.write(json.dumps(geojson, indent=2))
        

This script saves the results to a GeoJSON file for each feature layer within each feature service that is being downloaded. These GeoJSON files can then be added to ArcGIS Pro or QGIS, or previewed via an online viewer like https://geojson.io/ (assuming you have used 4326 as the output spatial reference).

The scripts I have written for my workflows are more complicated than this script above. For my scripts, I normally use an ArcGIS Online web map as the script input, then I use Python to download all of the data in the web map by looping through each of the feature layers, and I also create layer files (.lyrx) from the URLs using arcpy.management.SaveToLayerFile, however, SaveToLayerFile does not work with secure URLs, so in my code, I use a proxy via FastAPI to keep the URL secure but without the token, so that my SaveToLayerFile function still works. Then, I use Python to modify the data sources in the .lyrx files to map to my new file geodatabase, and then I use Python to create an ArcGIS Pro project (.aprx file) with the downloaded data and correct symbology. So, I can copy a secure web map with secure services to an ArcGIS Pro project, without needing to open ArcGIS Pro.

If you need more details on how I do any of the above, or if you have feedback for improving this article, then please message me on LinkedIn.

If you work for Esri, and you see anything wrong with what I am doing above in my script (in terms of being compliant with Esri's license terms and conditions), then please send me a message, and I will update this article.