Visualizing Proteins in 3D on Mobile Devices using Flutter (Faking 3D Graphics)

As a programmer, a considerable amount of your time is dedicated to work projects. However, sometimes, due to reasons like curiosity or a desire for change in your direction, you delve into researching and attempting to write code that may pose a new challenge for you. Some time ago, I came across a few video clips by Kooshiar Azimian , the CEO of 310.ai , discussing the generation of new protein molecules using AI. Out of curiosity, I decided to test this product on their website. It was fascinating. At the same time, I was thinking whether this application could be developed for mobile as well. In fact, two curiosities came to my mind simultaneously. The first was displaying a 3D molecule using a mobile app, and the second was exploring Flutter's capabilities in 3D graphics. At the Flutter Forward conference held in January 2023, discussions about Impeller were presented, which besides the features and performance it would bring, was intended to enable 3D graphics for Flutter. After some research I conducted in the past few days, its stable version has not yet been released for Android. Due to using Windows for development, I couldn't assess Impeller's capabilities on iOS and macOS and had to settle for watching a few clips. However, the fact that it is not yet usable for Android was important to me. Hopefully, we'll have access to Impeller with its special features as soon as possible. Ultimately, I decided to explore and implement various ways to visualize 3D protein molecules using Flutter.

?Firstly, I needed to learn more about the structure of protein molecules. In my research journey, I became familiar with rcsb.org and later with the Protein Data Bank (PDB). Information that I knew nothing about before. It seemed like an interesting challenge.

The first solution was to display 3D through GLB files using the model_viewer_plus package or similar packages. It seemed like a suitable solution with very good quality. In the following, I encountered several issues. Firstly, for each molecule, a several-megabyte file needed to be downloaded from the reference site. Secondly, if operational activities like those of the 310.ai site were to be performed and a new molecule displayed, there would be no reference site available, and the GLB file would need to be created by the backend and then received by the mobile app. Furthermore, aside from the several-megabyte size of each file, if there are many users, it is likely to impose a heavy workload on the server.

?The second solution I found after investigation was using WebView and displaying a web page that could directly display the PDB file. For this, I used Mol*, which has other alternatives as well. But then I thought, if a web page is to be displayed, what's the need for a mobile app? After testing this solution, I also realized it runs very slowly and doesn't provide a suitable display on a small mobile screen. It even encountered problems on an older phone and only displayed a chessboard. Furthermore, some of the issues with the first solution persisted in this solution as well. So, I had to look for another solution.

For the third solution, I went back to exploring 3D visualization in Flutter. Initially, I tried to use Native Code, which would require me to work with Swift and Kotlin, leaving no reason to use Flutter. In my own investigations, I came across a video by Craig Labenz and Filip Hrá?ek , which had an interesting topic, "Observable Flutter: Faking 3D graphics"! They didn't reach the conclusion they wanted after 2 hours, but it gave me a clue for a multi-day curiosity. I had to turn to CustomPainter class and paint method and vector_math_64 library. More importantly, solving the required mathematical issues was crucial. I had to recall what I learned in high school and university. Over 20 years had passed and recalling them was challenging for me. After hours of studying and browsing the internet, I started completing my code through the third solution. After completing parts of the code, the project was ready to run. Running it revealed interesting issues for me. It clarified why Craig used the term "Faking 3D graphics"; it became fully justified to me. Even he might not have been entirely sure, but he chose the best possible name. Solving all perspective and object stacking issues had to be done by me. Moreover, all drawing methods in the Canvas class are created for 2D drawings. For example, to draw a circle, you can use drawCircle, but there's no method for drawing a sphere. Also, the important Offset method only takes x and y coordinates and doesn't involve z. For solving the sphere drawing, I used coloring and scaled them for perspective. Using a gradient for coloring 3D objects may not seem very logical, but at present, it's the simplest shortcut available. The challenges and difficulties, as usual, reveal themselves gradually. For drawing lines to show connections between atoms, I had to use drawLine method, which further challenged perspective as it couldn't draw a line with varying thickness from start to end. Lines further away could be thinner, but for lines with different z at the start and end, it was unsolvable. Also, for creating a 3D image, lines shouldn't be drawn from the center of two spheres but from the minimum distance between them, which meant solving several trigonometric equations. I decided to speed up the work by turning to ChatGPT and Gemini. First, I implemented Gemini's solution, which seemed simpler. But, it had many issues, and I couldn't achieve the necessary result. Then I used the solution provided by ChatGPT. Even that wasn't flawless, and I had to spend several hours to solve this mathematical problem. A significant experience I gained was that after getting help from AI, its solution must be tested for correctness. It hasn't matured enough to provide the final solution.

After completing the initial phase of the project and resolving some errors, I reached a somewhat reliable preliminary result. This is just the beginning, reminiscent of programming days with C, Pascal, and C++. A time when we had to fulfill most of our needs on our own. Completing such a project would require much more time, especially in mathematical computations and gaining sufficient knowledge about protein molecule structures. But for me, as an initial effort in both areas, it was sufficient. Perhaps in the future, I'll decide to further develop this project. But what became clear to me is that Flutter has not been developed to the extent required in the realm of 3D, and we must wait for the stable version of the Impeller rendering engine.

The project source code is also available for those interested to review and use. Additionally, if you search for "protein-visualization" on GitHub, you'll find source code in various programming languages. If I decide to complete this project, I'll likely seek help from them. There are always interesting topics for curiosity and creativity.

Those who want can access the project: https://github.com/KavehKavandi/protein3d_flutter_custom3d