Debunking algorithmic qubits
Executive Summary: Quantinuum’s H-Series computers have the highest performance in the industry, verified by multiple widely adopted benchmarks including quantum volume We demonstrate that an alternative benchmark called algorithmic qubits is deeply flawed, hiding computer performance behind a plurality voting trick and gate compilations that are not widely useful.
Recently a new benchmark called algorithmic qubits (AQ) has started to be confused with quantum volume measurements. Quantum volume (QV) was specifically designed to be hard to “game,” however the algorithmic qubits test turns out to be very susceptible to tricks that can make a quantum computer look much better than it actually is. While it is not clear what can be done to fix the algorithmic qubits test, it is already clear that it is much easier to pass than QV and is a poor substitute for measuring performance. It is also important to note that algorithmic qubits are not the same as logical qubits, which are necessary for full fault-tolerant quantum computing.
To make this point clear, we simulated what algorithmic qubits data would look like for two machines, one clearly much higher performing than the other. We applied two tricks that are typically used when sharing algorithmic qubits results: gate compilation and error mitigation with plurality voting. From the data above, you can see how these tricks are misleading without further information. For example, if you compare data from the higher fidelity machine without any compilation or plurality voting (bottom left) to data from the inferior machine with both tricks (top right) you may incorrectly believe the inferior machine is performing better. Unfortunately, this inaccurate and misleading comparison has been made in the past. It is important to note that algorithmic qubits uses a subset of algorithms from a QED-C paper that introduced a suite of application oriented tests and created a repository to test available quantum computers. Importantly, that work explicitly forbids the compilation and error mitigation techniques that are causing the issue here.
As a demonstration of the perils of AQ as a benchmark, we look at data obtained on both Quantinuum’s H2-1 system as well as publicly available data from IonQ’s Forte system.
We reproduce data without any error mitigation from IonQ’s publicly released data in association with a preprint posted to the arXiv, and compare it to data taken on our H2-1 device. Without error mitigation, IonQ Forte achieves an AQ score of 9, whereas Quantinuum H2-1 achieves AQ of 26. Here you can clearly see improved circuit fidelities on the H2-1 device, as one would expect from the higher reported 2Q gate fidelities (average 99.816(5)% for Quantinuum’s H2-1 vs 99.35% for IonQ’s Forte). However, after you apply error mitigation, in this case plurality voting, to both sets of data the picture changes substantially, hiding each underlying computer’s true capabilities.
Here the H2-1 algorithmic performance still exceeds Forte (from the publicly released data), but the perceived gap has been reduced by error mitigation.
“Error mitigation, including plurality voting, may be a useful tool for some near-term quantum computing but it doesn’t work for every problem and it’s unlikely to be scalable to larger systems. In order to achieve the lofty goals of quantum computing we’ll need serious device performance upgrades. If we allow error mitigation in benchmarking it will conflate the error mitigation with the underlying device performance. This will make it hard for users to appreciate actual device improvements that translate to all applications and larger problems,” explained Dr. Charlie Baldwin, a leader in Quantinuum’s benchmarking efforts.
There are other issues with the algorithmic qubits test. The circuits used in the test can be reduced to very easy-to-run circuits with basic quantum circuit compilation that are freely available in packages like pytket. For example, the largest phase estimation and amplitude estimation tests required to pass AQ=32 are specified with 992 and 868 entangling gates respectively but applying pytket optimization reduces the circuits to 141 and 72 entangling gates. This is only possible due to choices in constructing the benchmarks and will not be universally available when using the algorithms in applications. Since AQ reports the precompiled gate counts this also may lead users to expect a machine to be able to run many more entangling gates than what is actually possible on the benchmarked hardware.
What makes a good quantum benchmark? Quantum benchmarking is extremely useful in charting the hardware progress and providing roadmaps for future development. However, quantum benchmarking is an evolving field that is still an open area of research. At Quantinuum we believe in testing the limits of our machine with a variety of different benchmarks to learn as much as possible about the errors present in our system and how they affect different circuits. We are open to working with the larger community on refining benchmarks and creating new ones as the field evolves.
To learn more about the Algorithmic Qubits benchmark and the issues with it, please watch this video where Dr. Charlie Baldwin walks us through the details, starting at 32:40.
About Quantinuum
Quantinuum, the world’s largest integrated quantum company, pioneers powerful quantum computers and advanced software solutions. Quantinuum’s technology drives breakthroughs in materials discovery, cybersecurity, and next-gen quantum AI. With over 500 employees, including 370+ scientists and engineers, Quantinuum leads the quantum computing revolution across continents.
Today’s LLMs are often impressive by past standards – but they are far from perfect
Quietly, and determinedly since 2019, we’ve been working on Generative Quantum AI. Our early focus on building natively quantum systems for machine learning has benefitted from and been accelerated by access to the world’s most powerful quantum computers, and quantum computers that cannot be classically simulated.
Our work additionally benefits from being very close to our Helios generation quantum computer, built in Colorado, USA. Helios is 1 trillion times more powerful than our H2 System, which is already significantly more advanced than all other quantum computers available.
While tools like ChatGPT have already made a profound impact on society, a critical limitation to their broader industrial and enterprise use has become clear. Classical large language models (LLMs) are computational behemoths, prohibitively huge and expensive to train, and prone to errors that damage their credibility.
Training models like ChatGPT requires processing vast datasets with billions, even trillions, of parameters. This demands immense computational power, often spread across thousands of GPUs or specialized hardware accelerators. The environmental cost is staggering—simply training GPT-3, for instance, consumed nearly 1,300 megawatt-hours of electricity, equivalent to the annual energy use of 130 average U.S. homes.
This doesn’t account for the ongoing operational costs of running these models, which remain high with every query.
Despite these challenges, the push to develop ever-larger models shows no signs of slowing down.
Enter quantum computing. Quantum technology offers a more sustainable, efficient, and high-performance solution—one that will fundamentally reshape AI, dramatically lowering costs and increasing scalability, while overcoming the limitations of today's classical systems.
Quantum Natural Language Processing: A New Frontier
At Quantinuum we have been maniacally focused on “rebuilding” machine learning (ML) techniques for Natural Language Processing (NLP) using quantum computers.
Our research team has worked on translating key innovations in natural language processing — such as word embeddings, recurrent neural networks, and transformers — into the quantum realm. The ultimate goal is not merely to port existing classical techniques onto quantum computers but to reimagine these methods in ways that take full advantage of the unique features of quantum computers.
We have a deep bench working on this. Our Head of AI, Dr. Steve Clark, previously spent 14 years as a faculty member at Oxford and Cambridge, and over 4 years as a Senior Staff Research Scientist at DeepMind in London. He works closely with Dr. Konstantinos Meichanetzidis, who is our Head of Scientific Product Development and who has been working for years at the intersection of quantum many-body physics, quantum computing, theoretical computer science, and artificial intelligence.
A critical element of the team’s approach to this project is avoiding the temptation to simply “copy-paste”, i.e. taking the math from a classical version and directly implementing that on a quantum computer.
This is motivated by the fact that quantum systems are fundamentally different from classical systems: their ability to leverage quantum phenomena like entanglement and interference ultimately changes the rules of computation. By ensuring these new models are properly mapped onto the quantum architecture, we are best poised to benefit from quantum computing’s unique advantages.
These advantages are not so far in the future as we once imagined – partially driven by our accelerating pace of development in hardware and quantum error correction.
Making computers “talk”- a short history
The ultimate problem of making a computer understand a human language isn’t unlike trying to learn a new language yourself – you must hear/read/speak lots of examples, memorize lots of rules and their exceptions, memorize words and their meanings, and so on. However, it’s more complicated than that when the “brain” is a computer. Computers naturally speak their native languages very well, where everything from machine code to Python has a meaningful structure and set of rules.
In contrast, “natural” (human) language is very different from the strict compliance of computer languages: things like idioms confound any sense of structure, humor and poetry play with semantics in creative ways, and the language itself is always evolving. Still, people have been considering this problem since the 1950’s (Turing’s original “test” of intelligence involves the automated interpretation and generation of natural language).
Up until the 1980s, most natural language processing systems were based on complex sets of hand-written rules. Starting in the late 1980s, however, there was a revolution in natural language processing with the introduction of machine learning algorithms for language processing.
Initial ML approaches were largely “statistical”: by analyzing large amounts of text data, one can identify patterns and probabilities. There were notable successes in translation (like translating French into English), and the birth of the web led to more innovations in learning from and handling big data.
What many consider “modern” NLP was born in the late 2000’s, when expanded compute power and larger datasets enabled practical use of neural networks. Being mathematical models, neural networks are “built” out of the tools of mathematics; specifically linear algebra and calculus.
Building a neural network, then, means finding ways to manipulate language using the tools of linear algebra and calculus. This means representing words and sentences as vectors and matrices, developing tools to manipulate them, and so on. This is precisely the path that researchers in classical NLP have been following for the past 15 years, and the path that our team is now speedrunning in the quantum case.
Quantum Word Embeddings: A Complex Twist
The first major breakthrough in neural NLP came roughly a decade ago, when vector representations of words were developed, using the frameworks known as Word2Vec and GloVe (Global Vectors for Word Representation). In a recent paper, our team, including Carys Harvey and Douglas Brown, demonstrated how to do this in quantum NLP models – with a crucial twist. Instead of embedding words as real-valued vectors (as in the classical case), the team built it to work with complex-valued vectors.
In quantum mechanics, the state of a physical system is represented by a vector residing in a complex vector space, called a Hilbert space. By embedding words as complex vectors, we are able to map language into parameterized quantum circuits, and ultimately the qubits in our processor. This is a major advance that was largely under appreciated by the AI community but which is now rapidly gaining interest.
Using complex-valued word embeddings for QNLP means that from the bottom-up we are working with something fundamentally different. This different “geometry” may provide advantage in any number of areas: natural language has a rich probabilistic and hierarchical structure that may very well benefit from the richer representation of complex numbers.
The Quantum Recurrent Neural Network (RNN)
Another breakthrough comes from the development of quantum recurrent neural networks (RNNs). RNNs are commonly used in classical NLP to handle tasks such as text classification and language modeling.
Our team, including Dr. Wenduan Xu, Douglas Brown, and Dr. Gabriel Matos, implemented a quantum version of the RNN using parameterized quantum circuits (PQCs). PQCs allow for hybrid quantum-classical computation, where quantum circuits process information and classical computers optimize the parameters controlling the quantum system.
In a recent experiment, the team used their quantum RNN to perform a standard NLP task: classifying movie reviews from Rotten Tomatoes as positive or negative. Remarkably, the quantum RNN performed as well as classical RNNs, GRUs, and LSTMs, using only four qubits. This result is notable for two reasons: it shows that quantum models can achieve competitive performance using a much smaller vector space, and it demonstrates the potential for significant energy savings in the future of AI.
In a similar experiment, our team partnered with Amgen to use PQCs for peptide classification, which is a standard task in computational biology. Working on the Quantinuum System Model H1, the joint team performed sequence classification (used in the design of therapeutic proteins), and they found competitive performance with classical baselines of a similar scale. This work was our first proof-of-concept application of near-term quantum computing to a task critical to the design of therapeutic proteins, and helped us to elucidate the route toward larger-scale applications in this and related fields, in line with our hardware development roadmap.
Quantum Transformers - The Next Big Leap
Transformers, the architecture behind models like GPT-3, have revolutionized NLP by enabling massive parallelism and state-of-the-art performance in tasks such as language modeling and translation. However, transformers are designed to take advantage of the parallelism provided by GPUs, something quantum computers do not yet do in the same way.
In response, our team, including Nikhil Khatri and Dr. Gabriel Matos, introduced “Quixer”, a quantum transformer model tailored specifically for quantum architectures.
By using quantum algorithmic primitives, Quixer is optimized for quantum hardware, making it highly qubit efficient. In a recent study, the team applied Quixer to a realistic language modeling task and achieved results competitive with classical transformer models trained on the same data.
This is an incredible milestone achievement in and of itself.
This paper also marks the first quantum machine learning model applied to language on a realistic rather than toy dataset.
This is a truly exciting advance for anyone interested in the union of quantum computing and artificial intelligence, and is in danger of being lost in the increased ‘noise’ from the quantum computing sector where organizations who are trying to raise capital will try to highlight somewhat trivial advances that are often duplicative.
Quantum Tensor Networks. A Scalable Approach
Carys Harvey and Richie Yeung from Quantinuum in the UK worked with a broader team that explored the use of quantum tensor networks for NLP. Tensor networks are mathematical structures that efficiently represent high-dimensional data, and they have found applications in everything from quantum physics to image recognition. In the context of NLP, tensor networks can be used to perform tasks like sequence classification, where the goal is to classify sequences of words or symbols based on their meaning.
The team performed experiments on our System Model H1, finding comparable performance to classical baselines. This marked the first time a scalable NLP model was run on quantum hardware – a remarkable advance.
The tree-like structure of quantum tensor models lends itself incredibly well to specific features inherent to our architecture such as mid-circuit measurement and qubit re-use, allowing us to squeeze big problems onto few qubits.
Since quantum theory is inherently described by tensor networks, this is another example of how fundamentally different quantum machine learning approaches can look – again, there is a sort of “intuitive” mapping of the tensor networks used to describe the NLP problem onto the tensor networks used to describe the operation of our quantum processors.
What we’ve learned so far
While it is still very early days, we have good indications that running AI on quantum hardware will be more energy efficient.
We recently published a result in “random circuit sampling”, a task used to compare quantum to classical computers. We beat the classical supercomputer in time to solution as well as energy use – our quantum computer cost 30,000x less energy to complete the task than Frontier, the classical supercomputer we compared against.
We may see, as our quantum AI models grow in power and size, that there is a similar scaling in energy use: it’s generally more efficient to use ~100 qubits than it is to use ~10^18 classical bits.
Another major insight so far is that quantum models tend to require significantly fewer parameters to train than their classical counterparts. In classical machine learning, particularly in large neural networks, the number of parameters can grow into the billions, leading to massive computational demands.
Quantum models, by contrast, leverage the unique properties of quantum mechanics to achieve comparable performance with a much smaller number of parameters. This could drastically reduce the energy and computational resources required to run these models.
The Path Ahead
As quantum computing hardware continues to improve, quantum AI models may increasingly complement or even replace classical systems. By leveraging quantum superposition, entanglement, and interference, these models offer the potential for significant reductions in both computational cost and energy consumption. With fewer parameters required, quantum models could make AI more sustainable, tackling one of the biggest challenges facing the industry today.
The work being done by Quantinuum reflects the start of the next chapter in AI, and one that is transformative. As quantum computing matures, its integration with AI has the potential to unlock entirely new approaches that are not only more efficient and performant but can also handle the full complexities of natural language. The fact that Quantinuum’s quantum computers are the most advanced in the world, and cannot be simulated classically, gives us a unique glimpse into a future.
The future of AI now looks very much to be quantum and Quantinuum’s Gen QAI system will usher in the era in which our work will have meaningful societal impact.
At this year’s Q2B Silicon Valley conference from December 10th – 12th in Santa Clara, California, the Quantinuum team will be participating in plenary and case study sessions to showcase our quantum computing technologies.
Schedule a meeting with us at Q2B
Meet our team at Booth #G9 to discover how Quantinuum is charting the path to universal, fully fault-tolerant quantum computing.
Join our sessions:
Tuesday, Dec 10, 10:00 - 10:20am PT
Plenary: Advancements in Fault-Tolerant Quantum Computation: Demonstrations and Results
There is industry-wide consensus on the need for fault-tolerant QPU’s, but demonstrations of these abilities are less common. In this talk, Dr. Hayes will review Quantinuum’s long list of meaningful demonstrations in fault-tolerance, including real-time error correction, a variety of codes from the surface code to exotic qLDPC codes, logical benchmarking, beyond break-even behavior on multiple codes and circuit families.
Wednesday, Dec 11, 4:30 – 4:50pm PT
Keynote: Quantum Tokens: Securing Digital Assets with Quantum Physics
Mitsui’s Deputy General Manager, Quantum Innovation Dept., Corporate Development Div., Koji Naniwada, and Quantinuum’s Head of Cybersecurity, Duncan Jones will deliver a keynote presentation on a case study for quantum in cybersecurity. Together, our organizations demonstrated the first implementation of quantum tokens over a commercial QKD network. Quantum tokens enable three previously incompatible properties: unforgeability guaranteed by physics, fast settlement without centralized validation, and user privacy until redemption. We present results from our successful Tokyo trial using NEC's QKD commercial hardware and discuss potential applications in financial services.
Wednesday, Dec 11, 5:10 – 6:10pm PT
Quantinuum and Mitsui Sponsored Happy Hour
Join the Quantinuum and Mitsui teams in the expo hall for a networking happy hour.