Python > FME: Getting the most out of GIS Automation

Introduction

Geographic Information Systems (GIS) are vital in today's world. Automation is an essential yet relatively new component of GIS, as it helps to streamline repetitive tasks and enhance data processing efficiency.

Python programming and Feature Manipulation Engine (FME) are two powerful tools used for GIS automation. Both tools have their strengths and weaknesses, and choosing between them can be a daunting task for GIS Programmers. In this article, we will explore the downsides of FME and why Python is the preferred GIS Automation tool. I would like to note that as a GIS Programmer, this article is to relay my personal opinion.

FME vs Python: The Sandbox Theory

The Sandbox Theory is a theory I've created to convey the principles of Python and FME to an inexperienced or elementary GIS Programmer.

"You have two sandboxes, FME and Python. You are not sure which one you want to play in, so you decide to give both a try!

You move to play in the FME sandbox, but you must pay $5 to get your shovels, buckets, shells, and shape molds. You see that all of the tools are conveniently laid out for you, woo! You pay the $5, take your shoes off, and jump right in. You want to build a very complex sand castle, and you realize that while you have a plethora of buckets available, there isn't one that feels right to you. You realize that you are now restricted to the tools that have been provided to you, and while you can still make a pretty neat sand castle, you can't create the exact sand castle you want. You notice that while the tools are organized for you, you don't really have an idea where to start. You wonder if it would be wise to ask someone, but you feel hesitant as your sand castle is unique and the guidance from someone else might not be exactly what you're looking for. Alas, you utilize your buckets, shapes, shells, and tools to create a sand castle. Satisfied, you move over to the Python sandbox.

Upon entering the Python sandbox, you find nothing but sand. You look around for tools to use, but you can not find any. You ask yourself "Should I have paid someone?". Staring at the vast, flat surface, you wish for your special bucket, you begin to think about what the bucket looks like, what curves and ridges it has, and its size, and suddenly, it appears in the sandbox. You come to the realization that although you have no tools to use, you can create them. This means that you can eventually create your ideal sand castle, so long as you are willing to create the items you require to make it in the first place.

As you create your buckets, tools, and shapes, you are unable to create a shell. You don't know how, or why, but it is the last piece you need to create. An individual spots you and walks over to you, you explain your problem and they explain that many others have wanted similar shells, and walks you through the exact process of creating one.?

What you realize is, that although the options presented were convenient, the power to create your own from scratch (and most importantly, at no cost) has allowed you to gain a better understanding of your sand castle, it is your own, unique to you, how powerful.

Beyond GIS Scripts

FME is restricted to a workflow style of achieving your data manipulation and management needs. Whereas Python has different avenues that will allow you to achieve anything data related. For myself, I've really gotten into using Python's 'Tkinter' module for graphical user interface (GUI) creation. Discovering that by creating simple yet effective GUIs, you create an environment for anyone to execute GIS-related tools and functions. Although it reminds me of the 32-bit applications I use to install as a kid, its deliverance does not need to be designed with a contemporary mindset to maintain its effectiveness. See below for a GUI I developed in a few hours that allows for spatial / non-spatial data editing.?

I mean, despite some design flaws it has the same fundamental functionality as ArcCatalog or ArcGIS Pro Attribute's panel. It does whatever you program it to do, I could have used the empty space to add a filter, switch selection, and even calculate field. The possibilities are endless. If you're a project manager who wants to edit an FME workspace, good luck. Whereas here, anyone can figure it out. This is why my position always leads towards Python, if made correctly, anyone can utilize it.

Extract, Transform, and Load (ETL)

Put simply, ETL is the process of moving data from one system to another or from one format to another. In my opinion, this is the extent of FME's capabilities. Although one might think FME is an infinite source of possibilities, however, its?capabilities?do not go beyond a specialized focus, and this might be why companies who focus on GIS automation gravitate towards Python. While FME does have some programming capabilities through its native interfaces, it's not nearly as flexible as Python. Throughout my research, I've discovered over eight Python libraries that can conduct ETL on spatial and non-spatial data. Whereas with FME, you only have access to the tools given to you, it's very constrictive.

Conducting ETL on big data is not easy by any means, so perhaps FME is preferred as its graphical user interface is user-friendly, but if you can't step into the mindset of being able to interpret the text as a developer, then you pigeonhole yourself from continuous improvement and growth across other programming languages.

GIS Automation in the workplace, and why people shy away from it!

GIS automation was never restricted to running a Python script manually. Your scripts do not need to be self-contained within a Python file, where you must open it in your preferred IDE and press run; and in my experience, this narrative is the exact reason why people I've worked with reject GIS automation using Python.

"What do I need to install?", "Will it work on everyone's computer?" "Why can't I see what's inside your program?", "Let's just do it the?old-fashioned?way for now", "How long will it take you to make?".

Whether you're a GIS Coordinator or the CEO of a conglomerate, by rejecting GIS Automation you are rejecting one of the most effective ROI that you can achieve in a GIS Department. GIS automation allows for your products to have minimal to non-existent variation, better quality, efficiency, and eliminated from human-error. Yet, it is met with so much dismissal because you have to be removed from another project to complete an internal task. You eliminate waste while providing the maximum value to customers with the lowest possible amount of investment. So, if you're working in a mapping department and your team is loaded with work, you're losing profits on chargeability, and you haven't implemented GIS automation in your day-to-day tasks, then you can't say that you couldn't have had it any other way. Because GIS programmers with experience in Python exist!

After presenting my capstone on Map Automation using Python, a first-year student commented that by implementing an automation tool like this, I am not attempting to remove jobs from the industry, and replacing them with scripts. I couldn't come up with a justification for it until a few hours later. My ideas for GIS automation were never to remove GIS Technician / Analyst jobs but to allow these techs and analysts to focus on the more interesting aspects of working in a GIS environment, such as remote sensing, photogrammetry, and the actual analytics of data. Making static maps gets old, very quickly...

Jidoka

Jidoka is automation with a human touch, and it's a very effective principle implemented in lean manufacturing (and in my scripts). Ideas are still automated, but they have human oversight to ensure that the results are accurate and meet industry/company standards.

FME for GIS Automation does not incorporate the concept of Jidoka in the same way that Python can. Since FME is focused on the automation of data transformations and does not have the same flexibility and customizability as Python. While FME does have some built-in QC measures, they are not as extensive as Python.

Exception handling and error reporting have become a large part of my development projects. This involves setting up my scripts to detect errors or anomalies during execution and to automatically notify me to take corrective action. This way, my scripts can continue to run without interruption, while ensuring that errors are identified and resolved in a timely manner.

FME, on the other hand, does not have the same level of support for exception handling and error reporting as Python. Again, while FME does have some limited capabilities in this area, it is not as robust as what can be achieved with Python. This means that if an error occurs during processing, FME may simply stop or produce incorrect results without notifying the developer. Say incorrect results were believed to be correct, you are now placing incorrect deliverables in the hands of your client. What's even worse is that they're paying you for this and you're turning your standard 10% profit margin, all for a useless workflow that you believe is right because it runs without any errors.

Conclusion

I do believe that FME is a powerful tool for ETL workflows, but there are several reasons why Python may be the obvious choice for GIS users. With FME, you have the cost, you have the steep learning curve, and restricted to spatial data. Python's versatility, flexibility, and intuitive learning process makes it the preferred GIS automation tool.

If you are someone who is rejecting everyday GIS automation ideas, don't. GIS Programmers are coming into the industry with the ideas to better everyone's lives, and more importantly,?the knowledge to implement them.?Your team members will thank you for it, your client will thank you for it, and you'll be able to keep up with changes in the industry. GIS Automation with a Python backbone is the future of the entire industry.

Until next time,

Ross