When your abandoned ideas come together
I "open-sourced" a side project of mine last week. The readme of the repository is more of a mini-blog post than a true "about" page because I feel the story of how it came to be is probably the most interesting part.
Dubbed "Flexli Engine" (because I had the domain lying around) it's my own take on a low/no-code workflow service. There are a lot of those out there, but I wanted to try my hand at something that was designed for event-driven, high scale automation across any API, and serverless, of course.
If you want to learn how it all works the user guide goes into detail on the concepts, features, and some tutorial content.
This started out simply as me wanting to do something fun with AWS dev in my spare time as I wasn't getting to do much of that during 2023. I went digging through my GitHub and found a private repo from my Jamf days: create the ability to chain together MDM commands when a device enters a smart group. This was an idea from a literal hallway conversation.
Jamf Pro, a mobile device management (MDM) solution, can create dynamic groups of managed devices based on attributes in their records. If a device enters or leaves a group the server can send a webhook to a destination. My original idea was to have a server that could accept those events and then run defined commands on devices in response - a workflow. Most of the basic design for the workflow schema was already here and it became my starting point.
While the original design was very focused on workflows for managed Apple devices, I quickly expanded it to cover general API operations (it was natural as the Jamf API has to be used to issue those MDM commands). I wanted to take any events and run any number of different workflows off those events if they matched. As I iterated on the design and the code I found myself going back into my archives to pull out other past projects that I never did anything with.
In the engine's workflows, API requests are actions and you need to decide what you want to do with the data that gets returned. I had decided that the workflow's data, the state, would never be modified unless the user explicitly did so. I needed an easy to use, but powerful system for selecting and modifying data without exposing any kind of code or custom scripting.
I had another project from a year prior, that I actually finished, called data-mapper which provides an API to transform JSON data using templates. It heavily uses JMESPath, a JSON querying language, to allow a lot of flexibility in how data is selected or modified as it passes through. JMESPath also supports a set of built-in functions as well as exposing custom functions which made it perfect for the transform feature of my workflow engine.
At this point I'm starting to write workflows to test out my ideas and figure out what I'm missing. One major gap is I needed to add conditional logic onto the actions. If a test against the state data didn't pass I wanted to be able to skip the action or stop the workflow.
领英推荐
The work was already done! I had already written a pretty robust condition system for another prototype years back. The goal then had been to approach smart groups as a shared service, but with more options than what could be done in Jamf Pro's existing system. The criteria could even reference other attributes in the same object for evaluations (something that's not possible with Jamf's implementation). I had written all the code and tested it, but had never gone the next step of creating the proof of concept service. All that work plugged right in the workflow engine with barely any modification.
The end result is something that grew to a point I had to stop and decide what I was going to do with. The engine had a full control plane API. I could define third-party APIs as connectors and use them in workflows. The workflow schema supported a huge range of control flows and actions like emitting internal events or running other nested workflows. I added JMESPath expressions to fill in gaps for things like datetime operations. I had to write a full user guide! Not only for the benefit of some I was sharing this with, but also for myself as I was writing workflows and had to have a reference to how all the features worked. The whole thing worked, worked well, and I was getting deep into feature work that probably was never going to see the light of day.
Which brings us to now, and releasing the code into the wild. It's there mainly for my own satisfaction. It's also very messy with unfinished features, lacking resilience, and a code structure not meant for production (just meant for rapid iteration).
It is also the culmination of multiple ideas from over multiple years that somehow all meshed together. I think the business world calls that synergy. There's probably a conference talk in here somewhere. We're all always coming up with ideas, and we can't always finish everything we start, but sometimes these things all click at a point and something really great comes from it.
Links
As an aside, I highly recommend writing your own user docs if you really want to know if what you've done makes sense. I not only caught bugs and unintended behavior, but I actively went back and changed things because I disliked how I had to explain it.
Helping organizations succeed with Apple devices.
6 个月There is something very satisfying about pulling snippets of code from “long lost” projects and stitching them together to create something new and useful. Thanks for sharing your work and your process with us.