Evaluating Quantum Computing Research

Quantum Computing has the potential to transform the world in front of us.? It’s no wonder then that we in industry as well as those in academia want to share the work we are doing.? At the same time the concepts can be quite complicated, and it can get hard to separate the signal from the noise.? I wanted to share the criteria I use when working with my own Quantum product team to make sure that we set an appropriate bar. I’d like to help the industry mature and unnecessary hype is antithetical to that outcome.? There are four key questions to ask whenever you see quantum news / research:

Question 1:?Is the work simulating the physics or demonstrating the physics in hardware?

Simulation is an important tool for research.? We use it extensively in our work to bring a quantum supercomputer based on topological qubits to reality.? Typically, one uses classical computers with a software workload to simulate the inputs and system under design to predict what the outcome would be.? In some cases, simulation can also include physical devices.? Either way the goal is the same: try to test your hypothesis up front so that when you do the actual hardware (e.g., fabricate a real chip or device then test it) you can measure against it.? We find the technique to be a super important part of chip fabrication because once dialed in, we can leverage the results to find a better path as well as characterize the outcomes leading to a faster time to discovery.

This question is important when analyzing publications because I often see articles published that don’t describe whether the work is a simulation vs actual physical experiment.? It doesn’t mean the experiments or publications aren’t worthy.? But, for example, if I simulated a new hypersonic airplane design via a computer program it is not the same thing as building and flying an actual plane.? What’s important is to clearly label the described work, and for simulations, to explain the practical benefit derived.

Question 2:?Does a quantum experiment have a demonstrable advantage over a classical computer?

The point of a quantum computer is to answer the kinds of hard problems that would literally be impossible on even the largest classical supercomputers.? Practically speaking this means that while you could run any algorithm on a classic supercomputer, the run times required may be so long as to make it useless to try (e.g., decades to even billions of years of compute time). ?This concept has been coined “quantum advantage” but has been liberally applied to many domains that when analyzed don’t really add up.

Bottom line: doing research today to help further the current generation of machines we are building makes sense as test cases and experimental systems.? The research can even spur new classical algorithms that wind up performing equally or better than the researched quantum algorithm. ?But we shouldn’t claim “quantum advantage” unless and until it is proven to have an advantage over an optimized workload run on a classical computer in a reasonable run time duration.??And to truly justify the cost and complexity of a quantum computing solution what we really want to find are exponential speedups, where in most cases the classical solution simply won’t be efficient or even practical.

Question 3:?Does the quantum solution advance the state of the art for the science it is exploring?

We are building quantum computers because we want to advance real science in a way that is not possible today.? For example, can we help find solutions for food scarcity?? Climate change?? Better battery technology? ?Designing and running basic quantum circuits as part of the R&D effort towards that end is legitimate; it is helpful as we design the new computers to pick a real-world problem for the test workload. ?We just need to be careful not to imply that a test workload like that has advanced the state of the art for the science explored.? The litmus test for this question would be validation by domain experts who do primary research and / or product development in the area being explored (e.g., chemistry or materials researchers).? If not, then the experiment should appropriately be classified as part of the quantum computing R&D effort with potential in the future to advance the state of the art.? You can learn more about achieving true quantum advantage here:?Disentangling Hype from Practicality: On Realistically Achieving Quantum Advantage.?

Question 4:?How many rQOPS would be required to demonstrate quantum advantage for the problem being explored?

Truly demonstrating practical quantum advantage requires far more sophisticated machines than anyone in the industry has today.? It can get confusing to understand whether progress is truly getting us to the goal or not.? As an example, we do not believe today’s race for noisy qubit count is the point.? It is like trying to get to the moon by building taller buildings here on earth. ?Sure, you can be happy each new building is taller than the last one. ?But it will never get you to the moon!? And similarly, there are systems which claim the greatest number of raw noisy qubits, but when you factor in the poor fidelity rates it means one still cannot execute real quantum circuits. ?Given the complexity, it is understandable why it can be hard to track what is real vs hype.

We have offered the Quantum Computing Implementation Levels to the industry for feedback to help frame the roadmap:?

• Level 1 – Foundational tracks these Noisy Intermediate Scale Quantum (NISQ) computers by qubit count.?
• Level 2 – Resilient gives us the first logical qubit that can be reliably used, as measured by logical error rate.?
• Level 3 – Scale will arrive when we have a true quantum supercomputer and for the first time demonstrate useful quantum advantage by advancing science in a way that is not possible today.? A Level 3 machine will be measured by reliable Quantum Operations Per Second or rQOPS.?

To solve the hardest problems, we believe you must have at least one million rQOPS, with only one in a trillion logical error rate (the only way you could execute a complex quantum circuit for the duration required to get a genuine answer).? Given a single Level 2 logical qubit will be made from hundreds or even thousands of physical qubits, today’s machines with just a few hundred noisy qubits and low fidelity are a long way off!

An Example

Recently IBM published a research paper that makes the claim that quantum computers soon will be able to beat classical computers through error mitigation vs. achieving error correction leveraging a ferromagnetism test case (Evidence for the utility of quantum computing before fault tolerance). ?The paper picked up quite a bit of coverage in the popular press.? Speaking on behalf of an industry which is already investing significant R&D in the assertion that Quantum Computing can and will exceed classical for some scenarios, investments that further that goal are a good thing.? This research leverages existing Level 1 computers and informs us of the computer science required for us to one day solve these problems.? I think it is useful to run it through the framework to understand what it does and does not do:

• Question 1 – Simulation or demonstrated in hardware? ?In this case the work leverages a real machine which can help advance hardware R&D.
• Question 2 – Can the problem be solved on a classical computer?? Yes, it can.? And even more recently, this paper demonstrates achieving a higher rate of accuracy using a simulation on a classical system: Efficient tensor network simulation of IBM’s kicked Ising experiment. ?
• Question 3 – Does the experiment advance the state of the art for the science being explored?? Not for the Ising model, which was the underlying test case motivating the experiment.? Rather, the experiment attempts to show that a real quantum machine could achieve results like a classical calculation. Credit to IBM for calling out this distinction.
• Question 4 – How many rQOPS does the scenario take to demonstrate quantum advantage? ?To truly advance the state of the art for the Ising model we will need at least 1 million rQOPS due to the complexity of the circuit required.? The goal of the paper though was to assert that error mitigation could lead to quantum advantage sooner without the requirement of error correction. ?As the community has pointed out, while the experiment is a good fit for quantum computing R&D purposes, don’t confuse it with a solution that achieves quantum advantage.

So, while we should applaud the experimentation for the purposes of quantum computing R&D, as an industry we still have a lot of research and development required to reach the scale required for a true practical quantum advantage.? Truly advancing the state of the art for real science problems will require getting a machine out of Level 1 and at least to Level 2.? Solving the hardest problems will require us to get to Level 3.? It is unclear how certain types of noisy qubits can even accomplish that task as a truly scalable machine that fully demonstrates quantum advantage must have size, speed, and controllability (Assessing requirements to scale to practical quantum advantage).? I will note that the realization of the true scale requirements is what led to Microsoft’s decision to pursue the tougher path of a hardware corrected topological qubit.

Engaging In Industry

As an industry we are all collectively committed to bringing quantum computing to fruition.? Our goal with the Quantum Computing Implementation Levels and rQOPS definitions is to help create maturity in the space and cultivate a shared understanding.? We look forward to feedback on the frameworks.

More Reading

If you are interested in this space, you may find the following links useful:

What does it mean to achieve scale?

• Disentangling Hype from Practicality: On Realistically Achieving Quantum Advantage
• Assessing requirements to scale to practical quantum advantage
• Microsoft achieves first milestone towards a quantum supercomputer - Microsoft Azure Quantum Blog

Estimating scale using practical tools

• Azure Quantum Resource Estimator empowers you to create algorithms for quantum at scale
• An introduction to resource estimation
• Learn how the Resource Estimator works

Real examples of solving hard science with quantum circuits

• Elucidating Reaction Mechanisms on Quantum Computers (better fertilizer production to assist with food scarcity)
• Quantum computing enhanced computational catalysis (advance carbon capture technology to help fight climate change)