BACQ: Delivering an application-oriented benchmark suite for objective multi-criteria evaluation of quantum computing performance, a key to industrial uses

Quantum computing is expected to revolutionize multiple technical fields and activity sectors, from optimization in logistics or in finance, to simulation for research, engineering or industry, passing through cryptography.

Measuring the progress towards the quantum advantage and the realization of such promises, with objectivity and reliability, is of high interest for potential end-users and crucial for the future development of the domain, now subject of hype and high competition.

The challenges, especially to achieve comparable measurements, comes from the diversity of the hardware platforms, their specificities in terms of physical characteristics and applications, their maturity that can still be low, and the potential rapid evolution of the technologies.

The BACQ project precisely aims at developing, exploiting and promoting a reliable, objective and long-lasting measurement instrument of the quantum computing practical performance, by considering a set of benchmarks at a level close to real applications. The project targets high-level evaluation, meaningful for industrial users. The proposed methodology consists in the aggregation of low-level technical metrics and a multi-criteria analysis providing operational performance indicators of the different quantum computing solutions and pointing out the service qualities of interest to the end-users. Dedicated to the whole value chain setting up from quantum hardware development to industrial use cases, BACQ is complementary to the benchmarking initiatives only focusing on low-level hardware physical criteria. Sharing of the benchmark suite, as broadly as possible, is an important objective to establish common reference measurement methods and ensure no bias with inclusion of all the technologies.

The envisioned benchmark suite will be based on the resolution of several classes of problems covering important application domains of quantum computing:

• Optimization
• Linear systems solving
• Quantum physics simulation
• Prime factorization

Relevant criteria will be defined for the resolution of each problem, some being hardware-agnostic and others dependent (low level): computation time, latency, problem size, approximation rate, resolution probability, accuracy, fidelity… Importantly, the project also considers energetic criteria for the evaluation of the energetic performance of the machines. The aggregation of the criteria and multi-criteria analysis will allow fully explainable and transparent notations, allowing comparisons between different quantum machines and with classical computers, as well as determination of the practical advantages of each quantum machine with respect to specific applications. The project will address both analog machines (quantum simulators and annealers) and gate-based machines, Noisy Intermediate Scale Quantum (NISQ) and Fault Tolerant Quantum Computing (FTQC). The followed practical approach is to have a suite of benchmarks, adaptative to some extent, appropriate to the capabilities of the available machines and able to demonstrate their respective advantages, including, in the longer term, exponential speedup of specific algorithms on FTQC machines.
Consultations of the technology providers, as well as end-users is key to develop an instrument that meet the needs. Once specified and developed, the benchmark suite will be available to the community for its use. To achieve a universal tool, the project will seek to establish cooperation with similar initiatives worldwide. Standardization will be another way to get consensus and adoption at large scale, considering the European committees CEN-CENELEC and the international ones, ISO, IEC and IEEE.

As part of the project, a first action has already been launched regarding Q-Score, which Eviden developed on the MAXCUT optimization problem, to test and validate the benchmark on different types of quantum machines. Within this first action to establish the universality of the benchmark, discussions and collaborations have already been engaged with European and international technology providers regarding various hardware platforms, superconducting, photonic, spins, neutral atoms, trapped ions, and NV centres in diamond.

The BACQ project, with 3-year duration and 4 M€ budget, including 2.5 M€ funding by France 2030 via ANR within MetriQs-France, gathers a consortium composed of Thales as coordinator, Eviden, an Atos business, CEA, CNRS, Teratec and LNE.

A consortium of quantum experts

Thales is the coordinator of the BACQ project and will participate in the development of benchmark criteria related to linear systems solving and optimization, with particular focus on noise and errors, on both NISQ and FTQC machines. Furthermore, Thales will be involved in the aggregation of the criteria developed in the project based on the multi-criteria decision tool MYRIAD developed for capturing consensus preferences and providing an explicit global rating.

Eviden will participate in the development of technical benchmark criteria related to optimization through the adaptation of the existing QScore/MAXCUT for a large variety of quantum machines. They will also contribute on the development of a new QScore/Many-body based for simulation of quantum physics.

CEA will contribute with three teams. Two of them will work on benchmark criteria related to simulations of many-body quantum physics problems with gate-based and analog machines (e.g., accuracy of the preparation, evolution and measurement of entangled states tested against analytical solutions). The third team will be involved in the evaluation of analog and gate-based machines on several classes of optimization problems (from easy-sparse, to difficult-dense problems), linear systems (at different sizes), and integer factorizations (with an increasing complexity).

CNRS will define, test and promote the energetic efficiency of simple gate-based quantum machines, focusing on the hardware-agnostic algorithmic level. The contribution includes developing a simulation module and use it to optimize energetic efficiencies. The promotion and acceptance of the benchmark will directly benefit from the strong implication of the CNRS team, which is leading the IEEE newly launched Quantum Energy Initiative Working Group, to propose a standard on energy efficiency of quantum computing.

Teratec will participate in the establishment of the relations to the ecosystem to promote the project: consultations with industrial stake-holders, end-users and technology providers, dialogue and cooperation in Europe and at international, with e.g., similar benchmarking initiatives.

LNE is the coordinator of MetriQs-France. The public institution will participate in BACQ for developing the relations with the stakeholders, for promotion and exploitation, notably in metrology and standardization, taking advantage of the connection with different organizations AFNOR, CEN-CENELEC, ISO/IEC. LNE will make particular use of its expertise in the development and implementation of reference measurement methods. In addition to this, by its trusted third party status, LNE will guarantee the impartiality and the independence of the benchmarks.

MetriQs-France

As part of the National Quantum Strategy of France, MetriQs-France is the national program on measurements, standards, test & evaluation of quantum technologies. With the ambition to support innovation and establishment of a sound and thriving quantum industry, the program aims at developing, exploiting and promoting reference measurement capabilities, validated and harmonized, for reliable, comparable, and impartial characterizations and performance evaluations of these emerging technologies. Such measurement are required to demonstrate indisputably the possible advantage of the quantum technologies and to enable their adoption by industry and market. Coordinated by LNE the program addresses the measurement needs on the national priorities, including quantum computing, quantum sensing, quantum communication and the enabling technologies. It comprises both deployment of an infrastructure of reference for measurement & test, and R&D collaborative projects.

General presentation of the partners

Thales (Euronext Paris: HO) is a global leader in advanced technologies within three domains: Defence & Security, Aeronautics & Space, and Digital Identity & Security. It develops products and solutions that help make the world safer, greener and more inclusive.
The Group invests close to €4 billion a year in Research & Development, particularly in key areas such as quantum technologies, Edge computing, 6G and cybersecurity.
Thales has 77,000 employees in 68 countries. In 2022, the Group generated sales of €17.6 billion.

Eviden designs the scope composed of Atos’ digital, cloud, big data and security business lines. It will be a global leader in data-driven, trusted and sustainable digital transformation. As a next generation digital business with worldwide leading positions in digital, cloud, data, advanced computing and security, it brings deep expertise for all industries in more than 53 countries. By uniting unique high-end technologies across the full digital continuum with 57,000 world-class talents, Eviden expands the possibilities of technologies for enterprises and public authorities, helping them to build their digital future. Eviden is an Atos Group business with an annual revenue of c. €5 billion.

CEA has role to inform public decision-making and to provide the scientific and technological resources needed by the country's driving forces (companies and local authorities) to better master the major societal changes: energy transition, digital technology, health of the future, defence and global security. Its 20,000 employees work at the heart of the regions in 9 centres equipped with very large research infrastructures, within the framework of academic and industrial partnerships in France, Europe and internationally. Thanks to the excellence of its fundamental research, the CEA also stands out from other European RTOs (Research and Technology Organisations).

CNRS, the French National Center for Scientific Research is one of the most recognised and renowned public research institutions in the world. For more than 80 years, it has continued to attract talent at the highest level and to nurture multi-disciplinary and interdisciplinary research projects at the national, European and international levels. Geared towards the public interest, it contributes to the scientific, economic, social and cultural progress of France. The CNRS is above all 33,000 women and men, more than 1,000 laboratories in partnership with universities and other higher education institutions bringing together more than 120,000 employees and 200 professions that advance knowledge by exploring the living world, matter, the Universe, and the functioning of human societies. The CNRS ensures that this mission is carried out in compliance with ethical rules and with a commitment to professional equality. The close relationship it establishes between its research missions and the transfer of acquired knowledge to the public makes it today a key player in innovation in France and around the world. Partnerships with companies are at the heart of its technology transfer policy, and the start-ups that have emerged from CNRS laboratories bear witness to the economic potential of its research. The CNRS provides also access to research findings and data, and this sharing of knowledge targets many audiences: scientific communities, the media, decision-makers, economic players and the general public. For more information: www.cnrs.fr.

Teratec is an association created in 2005 to bring together industrial users, suppliers and academic structures around HPC, AI and simulation. Since 2018, the TQCI (Teratec Quantum Computing Initiative) has been bringing together major industrial users around the development of quantum computing in order to better understand its uses, develop training and increase skills in this field, in particular through the organisation of seminars and support for setting up collaborative projects.

LNE, Laboratoire National de Métrologie et d’Essais, is the national metrology institute of France and an internationally recognized testing laboratory. Providing reliable measurements is at the core of LNE's mission entrusted by the French Ministry of the Economy and Finance in charge of Industry, to serve economy and society. Relying on excellent scientific & technical skills, LNE group, with around 1.000 people, has multiple activities: research in measurement science, testing and calibration services, technical assistance for e.g., regulation and standardization, certification, training. As a strategic orientation, LNE aims at supporting innovation and industrial competitiveness, notably in the field of emerging technologies, especially nanotechnologies, AI and quantum technologies. Regarding the latter, LNE has a long-standing experience for the development of quantum standards and sensors, to provide traceability to the International System of units (SI) at the best accuracy level.

 

Contacts presse

LNE, Coordinator of MetriQs-France:
Valérie Mulot • 01 40 43 40 93 • valerie.mulot@lne.fr

Thales, Coordinator of BACQ:
Philomène Emptaz • +33 6 59 06 98 76 • philomene.emptaz@thalesgroup.com