SuperForecasting: Ten Investment Practices to Improve Performance

Investing requires good judgment, which is blurred by bounded rationality in an uncertain world. Early-stage venture investing is further complicated by scant information in highly competitive, rapidly changing markets. Even with rigorous due diligence, most early-stage investments come down to an informed hunch.

In Seeking Alpha: How Investors Derive Signal from Noise, I drew from Munger, Kahneman and Tetlock and identified 25 tendencies that introduce bias and noise in investment decisions. Biased, noisy investment decisions impair investment performance. As Charlie Munger acknowledged, “I came to the psychology of human misjudgment almost against my will; I rejected it until I realized that my attitude was costing me a lot of money.”[1]

This article shifts the discussion to ten investment practices that any investment firm can use to reinforce a rigorous investment process and improve investment judgment. These ten practices build on each other so I have ordered them chronologically within the investment process. I include examples from my firm NGP Capital describing how we have refined these practices over the past two decades and drawn selectively from best practices at other venture firms.

Establishing a shared strategy, criteria and process for investments is vital. Rapidly changing, competitive markets will test even experienced investment teams who have worked together for decades. Investing by exception or changing investment criteria or processes midstream introduces noise and confuses the team.

The ten practices discussed here can be incorporated into any investment process. This article uses venture capital as a reference point, but I believe these practices are applicable to investing generally. They may be elements of an investment process but do not constitute a comprehensive process. While this article is long, it is highly condensed and much more could be said on each of these points and on the process overall.

I hope these practices are a catalyst for conversation. I would welcome your thoughts and best practices that work for you.

?

Ten Strategies to Improve Investment Judgment:

1.????? Investment Filters: A to do list is clutter, a not to do list is discipline. Investment filters are a not to do list that reinforces discipline. All investors use filters to efficiently screen potential investments and align effort with fund strategies and objectives. Early, growth and late-stage funds screen for company size and maturity. Funds that focus on specific sectors and geographies filter for companies within their target areas. Once past the initial filter, funds may screen companies by funding requirements, valuation expectations, target ownership stake, and their role as active or passive investors. High level filters should be easy to apply freeing time for investments that meet initial criteria and require further information to determine whether to proceed with an investment.

Filters benefit both entrepreneurs and investors reinforcing efficient use of time and effort. Entrepreneurs should be even more diligent in filtering prospective investors to limit disclosure of confidential information, which should be divulged only once investors and entrepreneurs have mutually confirmed interest. Investors are generally scrupulous with confidential information, but they see many companies and their allegiance and fiduciary duty are with their portfolio companies.

Thoughtful investors conserve energy and focus on high potential, prequalified investments where good judgment is required to determine whether to proceed further and invest. Buffett and Munger identified four investment filters in their letter to shareholders in 1977, at least three of which are judgment calls. They focus on businesses (1) they understand, (2) with favorable long-term prospects, (3) operated by honest and competent people, and (4) available at a very attractive price. All venture firms would agree that these are worthy targets, yet many diverge from these criteria in practice. Few investors have requisite technical expertise in deep tech startups. Many are willing to bid aggressively for high potential investments that promise outsized returns if successful.

?

2.????? Circle of Competence: Elaborating on their first investment filter regarding businesses they understand, Buffett advised: "Know your circle of competence, and stick within it. The size of that circle is not very important; knowing its boundaries, however, is vital.” Berkshire Hathaway has outperformed the market while avoiding technology, a sector that has outperformed the market for decades. Much of the venture market is driven by the FOMO – the Fear of Missing Out. Berkshire Hathaway, which Buffett acquired in 1964 and turned into his investment vehicle, is a sixty-year case study that belies FOMO. Residing in Nebraska far away from the financial markets has enabled Buffett to avoid the madness of crowds and focus on what he does best.?

Early stage investors focus on Founder Market Fit: founders with the requisite insight and expertise to capitalize on an opportunity above all other competitors. Startups with Founder Market Fit have the “Right to Win.” Investors would do well to apply a similar logic to their investments: Why are we so lucky? What expertise or insight gives us the Right to Win?

The Circle of Competence applies well to venture capital much as it applies to Berkshire Hathaway and private equity. Through our Fund of Funds program, NGP Capital has partnered with several boutique venture firms that focus on their area of expertise and outperform the venture market by a wide margin.

At NGP Capital, we are thesis driven investors focusing on investment themes that are sufficiently narrow to develop expertise and insight yet broad enough to develop pattern recognition across a dozen or more related investments. Shared investment themes help cohere and gather insights across our global investment practice. Having managed investment themes for the past two decades, NGP Capital has found that it takes at least two years to develop insight and apply pattern recognition. Our best investments typically come three to five years after we have launched an investment theme.

3.????? Advisors: Roman statesman and philosopher Seneca advised, “Cherish some man of high character, and keep him ever before your eyes, living as if he were watching you, and ordering all your actions as if he beheld them.” Two centuries later Roman emperor Marcus Aurelius offered similar guidance: “Search men’s governing principles, and consider the wise, what they shun and what they cleave to.” I am fortunate to have had several wonderful mentors across my career, including Don Lucas, one of the earliest Silicon Valley investors and initial lead investor in nine unicorns, including Oracle and Cadence Design Systems, back when billion-dollar companies were truly unicorns. My best investments were those in which he figuratively sat beside me, and I could see the opportunity through his eyes.

?

Every firm, whether a startup or investment fund, would benefit by assembling an Advisory Board. Advisory Boards serve a different function than Boards of Directors, which have a fiduciary responsibility to investors. Advisory Boards, when wisely assembled, have a broader purview. When wisely constructed and managed, Advisory Boards are like a sports franchise having five consecutive lottery picks: they enable firms to tap expertise that would not be available in any other capacity.

Advisors can serve different purposes so each relationship is unique to the person and situation. Advisory Boards work best when skills and perspectives are complementary and relevant to the situation. Otherwise, it may be best to engage advisors individually. Advisors should be fully informed on the business so they have prepared minds and appropriate context as matters arise that require their input. ?

Advisors generally offer three types of expertise: (1) domain expertise specific to a market sector; (2) functional expertise such as financial, technical, sales or marketing; and (3) investment or company building expertise earned through decades of experience. Advisors help fill gaps, offer perspective, build bench strength, and can assist on specific projects.

At NGP Capital, we have benefited from advisors for two decades. Advisors have helped guide firm development and refine our investment strategy. Industry experts have helped accelerate entry into new investment areas and avoid pitfalls on early investments. In the absence of dedicated advisors, several advisory services have emerged to provide due diligence support on an ad hoc basis with expertise across a wide range of areas.? ?

?? ?

4.????? Checklists: No pilot takes off without going through a checklist. Airline accidents per million flights declined by 95% in the 1960s after checklists and other safety measures were adopted. Checklists have since been adopted in healthcare, construction and manufacturing to simplify operations and improve safety.[2]

Both Kahneman and Munger assembled checklists of topics and questions to consider when making a decision. Munger spoke often about the benefits of a checklist and believed investors should develop their own checklists according to their objectives. Thus, it is not surprising that the checklists used by Kahneman and Munger differed reflecting their varied pursuits.[3]

At NGP Capital, we use several checklists to guide our investment process. We have checklists for our due diligence process, to assess product market fit and founder market fit, and to rate prospective investments. Checklists are a compendium of institutional wisdom. As checklists evolve, they are a good way to monitor personal and organizational development. As an evolving document, the contents of the checklist are less important than the forethought they require and discipline they instill.??

????

5.????? Project Teams – Inverters: Charlie Munger advised “invert, always invert.” He also said, “Tell me where I may die so I don’t go there,” which seemed to work well as he lived almost a century. Munger described himself as a foil for Buffett investigating what could go wrong with any potential investment. Much like Munger’s inverter, Kahneman advised designating a ‘decision observer’ for important decisions.[4] SG Warburg, an early leader in investment banking, assigned two different project teams to prepare cases for and against a potential investment and decided only once they fully understood both the investment rationale and risks.[5]

Inverters and project teams reinforce psychological distancing to gain perspective, boost creativity and mitigate biases. Investment committees serve a similar purpose, but they engage at the end with little ability to counteract biases already embedded in the process. Involving neutral observers at the outset encourages a healthier investment process. In Superforecasting[6], Tetlock observed that project teams, especially complementary teams of experts with diverse perspectives, outperformed individual experts in their global forecasting contests.

NGP Capital, like most venture firms, does our best in project teams. In due diligence for prospective investments, we include an “inverter” in each project team. The inverter keeps the checklist ensuring the project team addresses key investment risks identified by the firm. An able inverter offers a different perspective, balances that natural optimism of the project team, and serves as a wingman covering blind spots of the sponsoring partner. The inverter must be respected by the sponsoring partner and project team to be effective. Partners with domain expertise from another region offer a distanced, respected perspective to the process. ?

One final, important note on inverters. The inverter is a difficult role and is effective only if the firm has inculcated an open-minded culture that welcomes differing viewpoints. Munger acknowledged that he was effective because Buffett was open minded and welcomed contrarian views. Amazon nurtures an open culture and has a bedrock principle that leaders be willing to change their minds as new facts are uncovered.[7]

6.????? Perpetual Beta Mode: Precommitment bias is a pernicious tendency. Those who determine candidate fit within the first five minutes of an interview do both the candidate and their company a disservice.[8] Similarly, potential investors who decide to invest in the first meeting based on affinity with the CEO risk biasing all future diligence in favor of an investment.? As the legal phrase ‘confirmatory due diligence’ suggests, it is natural for project sponsors to pre-commit and overlook non-confirmatory evidence.

To counter the precommitment tendency, savvy investors delay judgment until all data are gathered and assessed. Tetlock describes this as ‘active open-mindedness’, a thinking style in which an analyst operates in ‘perpetual beta mode’, a term drawn from software development in which programs are constantly improved prior to release of a final version. Tetlock observed that “the strongest predictor of rising into the ranks of superforecasters is perpetual beta, the degree to which one is committed to belief updating and self-improvement. … What makes them so good is less what they are than what they do – the hard work of research, the careful thought and self-criticism, the gathering and synthesizing of other perspectives, the granular judgments and relentless updating.”[9]

Yet the corollary to ‘perpetual beta’ is analysis paralysis. Venture capital operates in highly competitive, rapidly changing and volatile markets with limited information. In such markets, the balance between foresight and hastiness is razor thin. Accustomed to operating in the ‘fog of war’, U.S. Secretary of State and Chairman of the Joint Chiefs of Staff Colin Powell applied the 40/70 rule suggesting that focusing on principle issues enabled prudent decisions to be made with 40-70% of total information. He believed that requiring full information can lull decision makers into a false sense of security and increase chances of making a big mistake.[10] Similarly, Tetlock struck a balance advising forecasters to delay intuition but not to ban it.

?

7.????? Scenario Analysis: Traditional discounted cash flow models do not readily apply to early-stage venture investing. Nobel laureate William Sharpe, who developed the capital asset pricing model (CAPM), observed that early-stage investments produce stochastic outcomes that do not follow a normal distribution. Investment outcomes have fat tails[11] and are better assessed by evaluating a set of expected discrete scenarios. His firm Financial Engines, which applied scenario analyses to value companies operating in rapidly changing markets, had a successful initial public offering and was ultimately acquired for $3 billion.[12]

At NGP Capital, we evaluate investments applying a weighted average expected return across five scenarios based on winner, upside, base case, downside and loser potential outcomes. Using five scenarios instead of three enables us to develop a more granular view and reflect fat tails in scenarios one and five that can have a material impact on weighted average expected returns. This granular scenario analysis also encourages investors to think more broadly and consider the winner and loss scenarios more carefully. We encourage investors to write detailed descriptions for each scenario outlining what we need to believe for the company to deliver favorable outcomes and how to mitigate risks that could yield disappointing results. These scenario analyses inform our action plan after the investment to help realize upside potential. ????

8.????? Premortems and Pre-parades: One way to be smarter at the start is to imagine ways a project may fail or succeed before deciding to invest or launch the project. Research shows that ‘prospective hindsight’ increases the ability to correctly anticipate future outcomes by up to 30%.[13] Prospective hindsight may be used either to determine whether to proceed or preventive measures that can reduce the risk of failure. Many firms use premortems to anticipate potential risks, but a one-sided process may reinforce risk aversion, which may lead investors to forego promising but risky ventures.

Sequoia Capital uses both premortems and pre-parades to identify risks and anticipate opportunities both for their own firm and portfolio companies. Managing Partner Roelof Botha credits this process with Sequoia’s decision to diversify into both early stage and public market investing.[14] Amazon development teams work backwards and draft a press release describing a product as it is ready to ship before product development begins. At NGP Capital, premortems and pre-parades are an extension of our scenario analyses encouraging deeper assessment of potential risks and opportunities.

9.????? Decision Structure: Early-stage venture investing is largely a bet on the founders. As Tom Alberg, founder of Madrona Ventures observed, “Even with rigorous due diligence, most early-stage investments come down to an informed hunch.”[15] That many of today’s iconic tech firms – Amazon, AirBnB, Peloton, Pinterest and Uber to name just a few – had difficulty raising initial funding reinforces that early-stage investing is more art than science.

In an uncertain world, good outcomes can come from bad decisions and bad outcomes from good decisions. As former head of Goldman Sachs Robert Rubin reminds us, “Decisions shouldn’t be evaluated only based on results. Even the best decisions are probabilistic and run a real risk of failure, but the failure wouldn’t necessarily make the decision wrong.”[16] Instead, both startups and investment firms should hone their decision-making process to increase the likelihood and magnitude of success.

Decision structure varies significantly across firms. Some firms vest decision authority in one person, others in an investment committee, and still others across all partners or the entire investment team. Even in distributed voting systems, group think can produce polarized outcomes when discussions disproportionately favor one viewpoint.

Kahneman recommended the Delphi method to tap the wisdom of the group and avoid group think. In the Delphi method, each person submits an anonymous vote at the outset. If voting results are clear, then discussion focuses on how best to implement the decision. Otherwise, if the vote is split, discussion elicits different points of view in a collective effort to develop a shared perspective. As Amy Herman has shown in how art is perceived, people see things differently even when we are all looking at the same thing.[17] How much more will perceptions differ when we consider amorphous, unproven technologies in dynamic markets. In such circumstances, investors will make better decisions when they coalesce around a shared perspective that accounts for different views on company prospects.?

At NGP Capital, rigorous diligence and robust discussions help identify differing assumptions that encourage shared perspectives even when opinions differ. We encourage broad input yet do not require consensus to proceed as we have found that the most disruptive opportunities often elicit differing views.?

????

10.?? After Action Reviews: Success has many fathers and failure is an orphan.[18] Failure offers many lessons, but these learnings may be lost if failure is abandoned and forgotten. Many firms use post-mortems to excavate lessons from failure that might otherwise be swept under the rug. I prefer After Action Reviews, which is a more neutral term and encourages teams to develop a balanced scorecard on all investments, both successes and failures.

Learning from success may be even more elusive. Firms celebrate successes with victory laps. As firms perpetuate an aura of success, victory laps can become victory miles or marathons. Celebrations may blur the distinction between good practices and outcomes and, with many clamoring for credit, success may prevent firms from accurately identifying and codifying best practice. After Action Reviews reinforces replicable practices that can be institutionalized to promote future success and encourages an honest assessment of favorable circumstances that might not be readily repeated in the future.

After Action Reviews are a standard military practice that seek to derive learnings and codify best practice from any major initiative. Memories are faulty and unreliable guides for After Action Reviews. Instead, the military has found that keeping detailed records of key decisions is essential for balanced After Action Reviews. At NGP Capital, we write investment and pre-closing memos that explain our investment rationale, risks, and key assumptions that informed our initial investments. These are the source documents that inform our After Action Reviews.

At NGP Capital, we use After Action Reviews at two stages in the investment process: (1) a post investment review three to six months after an initial investment and (2) a post exit review after completing the sale of an investment. In the Post Investment Review, we compare our pre-investment assumptions with our current understanding of the business to reassess our posture and action plan with the company. In a Post Exit Review, we assess what we did well and could have done better regardless of outcome on any investment. Since investments are collective decisions, we focus on what rather than who emphasizing key lessons and recommended adjustments to NGP best practices.

?

Concluding Thoughts:

Investing requires good judgment, which is blurred by bounded rationality in an uncertain world. ?Early-stage venture investing is further complicated as we operate in rapidly changing markets with limited information. Even with rigorous due diligence, most early-stage investments come down to an informed hunch.

Munger, Kahneman and Tetlock came from different disciplines but shared a commitment to good judgment. Comparing their assessments and proposed remedies enables thoughtful investors to triangulate and develop sound investment practices appropriate for their firms. ?

As a continuation of my observations in Seeking Alpha, this article identifies ten practices to improve investment performance for venture and private equity firms. I have offered examples from NGP Capital where we have implemented and refined these practices over the past two decades as well as other venture firms to illustrate implementation and benefits of these practices.

These ten practices are mutually reinforcing, and I have ordered them chronologically as they may occur in the investment process. There is a lot here as more process is needed to manage a global investment practice. Some of this may be downsized or approached differently when partners work together closely in one office. Yet this list is not intended to be comprehensive, and much more could be said on each of these practices. Our investment business is becoming more data driven and we rely increasingly on our proprietary, intelligent data platform to filter companies, identify high potential investments and evaluate prospective investments. The timing and sequencing of decisions are also important in managing escalating commitments.

Venture investors operate in highly competitive, fast moving, uncertain environments. Good judgment is more important in such circumstances, yet timely decisions with limited information exert pressure on prudent practices as is evident in the highly cyclical nature of our business.

Establishing a shared strategy, criteria and process for investments is vital. Rapidly changing, competitive markets will test even experienced investment teams who have worked together for decades. After Action Reviews facilitate institutional learning and continuous improvement.

The venture industry is an apprenticeship business. It is a wonderful profession as we are constantly learning as technology advances and the industry evolves. I would welcome your thoughts and best practices that work for you.

May the best be yet to come!

?

[1] Bevelin(2018). Seeking Wisdom.

[2] See Guwande, A. (2009). The Checklist Manifesto.

[3] See Kahneman (2021), Appendix B and Bevelin (2018), Appendix 4. Bevelin’s list is considerably longer than Kahneman’s as it includes both mental models and biases.

[4] Kahneman, D. (2021).?Noise: A Flaw In Human Judgment. See Appendix B for a checklist for Decision Observers.

[5] Chernow, R. (2012). The Warburgs.

[6] In 2011, IARPA recruited over 25,000 forecasters who participated in a four-year project and made over one million predictions about a wide range of world events. The Good Judgment Project (GJP) sponsored by Wharton won the contest besting subject matter experts and intelligent community specialists. The GJP used superforecasting practices developed by Philip Tetlock based on over 30 years of research.

[7] Comment from Tom Alberg, who was a Board director at Amazon for 25 years. Source: Alberg, T. (2021). Flywheels: How Cities are Creating Their Own Futures.

[8] Geoff Smart, author of Who: The A Method for Hiring applies structured, final round interviews for executives that can extend a half day or more. Kahneman also devotes a chapter to interviewing in Noise: A Flaw In Human Judgment. ?

[9] Tetlock and Gardner (2016).?Superforecasting.

[10] Powell, C. & J.E. Persico (2010). My American Journey.

[11] Fat tails, or khartosis in statistical parlance.

[12] Financial Engines was acquired in 2018 for $3.02 billion by Hellman & Friedman, a private equity firm.

[13] Mitchell, D. J., Edward Russo, J., & Pennington, N. (1989). Back to the future: Temporal perspective in the explanation of events.?Journal of Behavioral Decision Making,?2(1), 25-38.

[14] Roelof Botha described Sequoia’s use of pre-mortems and pre-parades on The Tim Ferris Show.?

[15] Alberg (2021). Flywheels.

[16] Robert Rubin, In an Uncertain World.

[17] Herman, A. (2017). Visual Intelligence: Sharpen Your Perception, Change Your Life. The book includes striking examples using art to show that we see things differently even when we carefully observe the same things.

[18] Attributed to John F. Kennedy.