The Art of Statistical Programming: When Creativity blooms from Restrictions and Repetitions

"Restrictions breed creativity". Often, original and unconventional ideas come up only when we are constrained. This may be with respect to any scenario. Repetition on the other hand is often perceived to end up in monotony, lack of originality and hence lack of creativity. But, on the other side of the story: "Repetitions breed familiarity". Familiarity in turn will gradually result in new ideas, revamp the usual thought process and induce creativity in a phased manner.

    The following piece of writing, is based on my learnings from the work scenario with respect to statistical programming, the initial phases of struggles, repetitions and restrictions, the point of realization that programming is indeed an art which is often misconstrued as a mere craft, and ultimately the emergence of creativity from the whole exercise.

    Programming, in simple words, is the process of setting up a bunch of instructions via a computer, to facilitate automatic performance of tasks. Statistical programming refers to the use of programming tools to perform statistical analysis and related computations.

    The overall idea behind programming is to make life simple by facilitating automation. Sometimes, work requirements pose the need for development of customized programs for certain statistically oriented computational tasks, or a macro-like function for performing a string of repetitive computations. For example, a program which performs a list of statistical analyses and computations on several variables at a single go. In such cases, the program is gradually built from scratch using a set of pre-written codes over and over again in a structured way using a set of "for loops" and "if loops". "For loops" are used for repeating the computations/functionality and "if loops" are used for constraining or for setting conditions for performing a computation. These are just basic functions described for the sake of illustration, and there are many more looping and other utility functions in any programming language.

    The initial stages of building such customized functions/programs would involve a lot of time in terms of structuring, testing and debugging. This stage involves a lot of repetitions, in terms of writing similar pieces of programs several times, using similar arrangement and working towards a prespecified end result/output template. The programs in this stage could be very lengthy and highly inefficient.

    However, with more and more time spent with the same set of functions, computational goals and fixing bugs, the program evolves into a better shape, exhibiting the attractive features of brevity and efficiency. More unique ways to structure the program efficiently come in. Gradually, ideas creep in and more enhancements are made to the functionalities and standard output templates. The enhancements may be adding more nuanced analysis features or data visualization tools within the macro-program, and also restyling the output template, in terms of layout, design, form, etc. Programming is certainly a skill, which is the reason that it's considered to be a craft. But applying this skill requires a lot of innovative thinking and creativity, because there are multiple ways to carry out a programming task. Hence programming is more of an art than a craft.

    More and more familiarity with the programming environment, basic chunk of syntax, analysis/computational tasks, etc., result in better understanding and the thought process is gradually modified to spark new ideas with respect to programming. More new add-ons like enabling different types of transformations on variables, addition of robust analysis methods apart from conventional ones, creative ways of summarization, for example, using heat maps to summarize key results from a model, inclusion of clear input arguments for users, appropriate analysis decisions automatically taken based on nature of dependent variables (binary/multinomial/time-to-event variables), etc. The attractiveness of the program can be augmented by giving it an interactive interface look. For example, customized R functions translated into shiny apps or via packages like "plotly". End user requirements also generate a lot of new ideas/improvements incorporated into the programs.

    Thus, building programs using repetitive codes for performing repetitive tasks (macros) constrained by end user's stated requirements, run time limits and programming environment rules, bring about a gradual change to the conventional thought process and programming ways, ultimately resulting in a lot of creative augmentations to the entire activity.

    Hence the new saying is, "Repetitions breed familiarity which when blended with restrictions breeds efficiency which in turn results in the blooming of creativity."