Rethinking node-based AI platform with Phygital+

This spring, I was invited to conduct a series of sessions on reimagining the Phygital+ product. It's a multipurpose creative platform with a node-based interface for design and art tasks, allowing you to build procedural chains from various neural networks. This approach to user interaction with complex systems, where goals are multifaceted and require prototyping and experimentation, resonates with me. Also, I had already started independently testing this platform shortly before receiving the invitation, making it an exceptionally intriguing direction.

Here, I had the opportunity to apply my diverse experience as working with node-based editors such as Rhino Grasshopper, VVVV (beta and gamma), Blender Geometry and Shader Nodes, Spark AR Studio, as well as more specific ones like Cables, QGIS Processing Modeler, ComfyUI, and my experience with Habidatum in interface design involved developing node-based solutions for processing spatial-temporal data and creating metrics as well.

I cannot divulge all the details of my work. However, I'd be delighted to shed light on some essential elements of my vision for developing such platforms, which I endeavored to unfold during the analysis and proposal development for Phygital+ and touch on some outcomes.

Vision

I believe a creative platform should meet specific requirements that arise from the very nature of the creative process:

1. Principle of total experimentation: the platform should facilitate a nonlinear creative process, including contemplation, generating multiple variants, switching between tasks, revisiting previous experiments, blending procedures and results, and other inherent aspects of unrestricted creativity.
2. A closed-loop cycle: users should be able to input a source for processing and go through all intended stages without jumping out of the platform, ensuring they remain within a singular task cycle.
3. Easy jump out (without contradicting the previous point): during a free creative process, it's impossible to satisfy all needs within the platform alone. Therefore, the ease of input and output should enable users to seamlessly integrate their familiar external tools into the creative pipeline. The platform should assist but not replace creative freedom.
4. Not only onboarding but ideally offering suggestions or even building the required solutions on the fly. Every platform adds complexity to user life. But this complexity creates space for generating intricate and unique content. Users may not be experts on how to manage it in a platform. However, over-simplification stifles the creative process, so we must help users navigate the complexity.
5. It's a space for community communication, mutual assistance, co-creative, and learning together with the community. Don't miss to integrate tools that support this interaction.
6. Allow users to play multiple roles: content creator, assistant, mentor, client, and add-on developer. Embracing these roles expands the platform's outreach and significance.

This is not a final list but the most critical points.

User interface paradigmes

Currently, I can highlight three types of AI-based creative platforms, differing in how users interact with AI algorithms:

1. Collection-based: This type consists of a set of unrelated AI algorithms. Users store input and output data and can reuse them without the ability to memorize and reuse pipelines. It's like a self-service store where you take a product and decide whether to use it separately or combine multiple ingredients to create your recipe, but you need your kitchen and jotting down the steps on paper. A notable example is Runway.
2. Canvas-based: This interface offers an infinite canvas where users can place images, texts, and other assets. Then you can freely apply various algorithms to them or use templates with pre-defined procedures. Users can freely arrange elements on a two-dimensional canvas, organizing space for experimentation and filling it as a creative sandbox. The main drawback is the lack of automated pipelines and dependencies between input assets and output results. A notable example is Fermat.
3. Node-based: In this case, users also have a two-dimensional canvas, but their elements include not only assets (images, texts, or 3D models) but also processes that they connect to form diverse chains, creating pipelines that can be reused by changing input assets and algorithm parameters within the chain. This is not only a creative sandbox. It's also like a music synthesizer, where you interconnect many inputs and outputs with wires to create a new composition. Phygital+ play on this field.

Finally

Ultimately, my proposal included the following key points:

1. The new paradigm for organizing and structuring nodes, new taxonomy for input-output and processing nodes, defining how they interact, setting up preferences, storing and editing information, grouping to complex nonlinear pipelines, and supporting autonomy and multi-threading.
2. Interface organization that allows convenient interaction with pipelines while maintaining the benefits of a creative sandbox, where users can organize their workflow freely and build spaces for thinking inherent to the "canvas-based" type. Users would have the flexibility to balance pipeline-aware and canvas-aware two strategies using just explicit properties of the interface's elements.
3. An alternative system of interaction through AI-driven chats while retaining the capabilities of a node-based editor. Interfaces based on Large Language Models (LLM) were beginning to emerge. I was also presented with the intriguing challenge of reimagining the platform based on this concept. However, I made one more step—a hybrid approach combining nodes and chat, which I haven't seen in the market yet.
4. General recommendations. Sometimes they are obvious, but they can have crucial warth for the healthy becoming of such platforms, like creating user extensions, collaborative editing, libraries of user-generated content, and other essential features.

In conclusion, this has been an incredible experience that unites three of my passions: creativity and creative processes, artificial intelligence, and user experience and interfaces. I thank Oleg Yusupov , CEO of Phygital+ Generative AI Workspace , for inviting me to participate in this endeavor. No matter the platform's direction for its development, I can only wish them the best of luck; they are a team with a remarkable vision.