Single Point of Failure – SPOF The power of 2

Many people have written about Hardware, Software, Business and other systems. My write-up is more about how businesses need to organize teams – especially Product Development team.

For last 30+ I had been developing many products starting with Financial Accounting, Jewelry ERP, POS, Trade Finance, HR, Task Management and others – and now Process Safety Risk Management.

I came to Hong Kong after getting trained on Tandem computers – in those time it was the first twin-cpu mainframe with fault-tolerant feature. Being an Industrial engineer and having studied Reliability, this stayed on with me from then – redundancy. I always ensured I had two resources for some of the critical tasks – like programming, customer support, sales and marketing, HR, Accounting. Even in the toilet I ensure two rolls are always replenished. Same applies to groceries (bread, sugar, eggs, butter etc) – except for my wife! (I can hear you saying even one is difficult to manage!)

In my first business Cosmic, from 1991, I used Magic RAD Tool (from Israel) and it was easier to maintain the code as it was very structured; off course with many other limitations like physical dongle, support was pretty bad, pricing was not transparent, some runtime behaviors could not understand. But it served me well in my initial years until 2000. But one thing I ensured there will be at-least 2 programmers trained at any point of time.

Off-course some worst cases even two redundancy may fail at the same time. This is what happened during 2008 financial tsunami. I had depended on two of my programmers working with me for 10+ years to protect the source code and IP which I had been developing for 18+ years – but with my own negligence to learn the new Technology ASP.NET/Java, lack of code review, standardization and documentation and too much trust/reliance on individuals, I lost the entire code (though I had some backups – the spaghetti codes could not be maintained by any new programmer(s) when I tried to hire and hence had to shut down that business and look for another career).

It was a tough lesson I learnt and I was almost bankrupt unable to pay mortgage, with young children admitted to expensive international school and at the verge of mental breakdown - even suicidal thoughts crossed  my mind once. The banks I was having account for 18+ years were not willing to give any loan nor give any holiday for mortgage payments. But I did not give up. I was positive that things will improve and there will be some light at the end of the tunnel (I guess a trait from my mother).  

I wrote to then Chief Executive of Hong Kong, Mr. Donald Tsang with my plight and requesting to help release my Provident Fund I had accumulated which I could not take a loan nor use (explaining to him that in India, one could take a low interest loan based on Provident Fund).

Next week I received a call from Citi bank offering me 1/2million HK dollars loan without any security with a 5 year repayment time. I grabbed it immediately. I have to thank Citi Bank for their timely support. I still think it was because the letter I wrote to HKSAR CEO. May be it was a coincidence. I never will know. By 2014, I cleared all the debts and I was on my financial recovery but still not out of the woods with increasing education expenses, growing children and family needs.

I also have to thank Ronald Cheung CEO of Lifewood who hired me in 2010 as Vice President of Operations to expand the BPO business (we dreamt together for CHINDIA BPO, cooperation between Chinese Dragon and Indian Elephant which never materialized – I had designed a logo at that time which we presented to many leading Indian BPOs - though everyone showed interest no one was willing to invest). I am glad I was able to bring many new projects from new markets like Hong Kong Trade Development Council Exhibition Data Digitization, Cathay Pacific Aircraft Maintenance Digitization, China Construction Bank Credit Card processing, Dental Corp Australia (now BUPA) account payable processing, Harrington Brooks UK Debt recovery and many more.

When I started the product development of Haz360 at Iresc, I also ensured that a minimum of two developers hired at the same time and code is maintained in bit bucket for version controlling. For small companies / startups with bootstrapped funding it is an ongoing battle between product development and sales/marketing. However it is important that knowledge is spread between multiple resources instead of depending on a single person for product support. Founders and sponsors need to keep this in mind when developing products, otherwise product development investments may not recoupable. Organization structure should be maintained that everyone should be replaceable. Some of the large companies can afford Pair-programming (an Agile technique in which two sit at the same workstation with one coding and another reviewing – switching roles frequently) which may not be possible for SMEs.

I remember in 1998, when I was hired to develop Trade Finance software for a bank, the IT person in that bank was afraid that he and his assistant would loose their jobs because he was the only person in the bank who knew how to operate the mainframe system and by replacing the mainframe software with PC based new software, they may be made redundant. Initially he was very reluctant to share any information about the operation, reports etc. I had to work with end users and learn about the current operations of Import, Export, Placement, Borrowing, Deposits, Loans and how those got updated to the Financial Accounting (chart of accounts). When we completed the implementation and got the approval from Hong Kong Monetary authority to retire the old mainframe system, the IT persons were retrained to support the new system and both were relieved about their loss of job. 

The point I am making is, yes for organizations it is always better to retain existing employees for their contribution, too much dependency on one individual could be avoided.

This is the reason why most corporates HR policy ensure every employee must take a 14 day mandatory paid holiday to ensure the systems and processes can work even without that specific employee. In fact, most corporates also use this opportunity to

• Review the employee and manager, also if someone is doing too much or too less
• Cross train other employee for the role
• Improve safety (identify any bullying or other work hazards if employees are burnt out)
• Plan for succession
• Motivate employee by rotating jobs or promoting to other tasks
• Improve delegation (hand-offs)
•  Identify any frauds

In Chinese this symbol (Xi) is used for marriage - meaning double happiness for the bride and groom - like families, product development need pairs - the power of 2.

Hope you enjoyed my article – please do comment on your views.