Striking the Right Balance: Rapid Prototyping and Clean Code in Applied Research and Machine Learning

As an applied scientist conducting research in industry, I frequently face the challenge of writing clean code and adhering to software engineering best practices when the end solution to a technical problem is unknown. Research often requires exploring multiple approaches and iterating through various solutions before achieving the desired outcome. This struggle is not unique to me, as I frequently witness fellow researchers and engineers facing similar hurdles during code reviews.

The approach that has proven most effective for me is rapidly prototyping research code in a Jupyter Notebook environment. This rapid prototyping approach allows me to achieve a working solution as quickly as possible, even if it results in what is often called "grad school code." Tackling challenging research problems typically requires multiple attempts and solutions, and the initial focus is finding a viable solution, even if there are more efficient and elegant implementations. Inevitably, the code often becomes less structured as the solution diverges from our initial attempt.

Once we achieve the initial technical solution, it becomes not just important but crucial to go back and refactor large portions of the code with the end solution in mind. This step involves restructuring the code to follow software engineering best practices, promoting maintainability, reliability, and reuse. It is also a responsibility to document critical technical design decisions while fresh in your mind, ensuring that future team members can understand the system and technical design choices without confusion.

While this approach may seem inefficient, as it involves writing the code twice, I have yet to discover a better way to balance the need for rapid prototyping and exploration with the necessity of producing clean, maintainable code for production environments.

A common pitfall, however, occurs when developers neglect the follow-up step to refactor and write a clean version of the code once the initial code is working, often due to tight schedule deadlines or management direction. This results in difficult-to-maintain code sneaking into production software, creating brittle systems, a snowball of technical debt, and software with insufficient documentation.

I'm eager to learn from your experiences and insights. How do you balance the need for rapid prototyping and exploration with the necessity of producing clean, maintainable code in your work? Have you discovered any best practices that have helped you in this regard? What challenges have you faced in this balancing act, and how have you overcome them? Or have you found effective ways to ensure that the refactoring and documentation steps are not overlooked, even under tight deadlines? Your input is invaluable in this ongoing discussion.