Demonstrate the reliability of your artificial intelligence systems

An AI system is a piece of hardware or a computer program that can make decisions, make recommendations, adapt to new situations, and learn from data in order to perform specific tasks autonomously or assisted.

Image

Why and when to qualify the AI ​​features of an intelligent system?

In order to optimize its operation and energy consumption (frugality), to secure its use and to explain its decision-making process (explainability with a view to acceptability), it is essential to qualify the various AI functionalities of the intelligent system, as well as the evaluation methods used, before it is put into operation.

Analyzing the performance and robustness of AI features helps assess its reliability in performing the requested task and meeting the regulatory requirements related to bringing your product to market.

Validating an AI system will provide an opportunity to ensure that the model and the data used are reliable and interpretable to obtain a quality and robust analysis.

Thanks to the structured analysis methods developed by our technical experts and adapted to your product, which put the technical functioning of AI to the test, it then becomes possible to challenge the AI decision-making process and identify all possible improvements.

Integrating this approach as early as possible in the integration or development phase of your product to identify the improvements to be made will limit development costs but also to be able to reassure about the performance of the AI model used in your system, to make the right decision.

The analysis of this performance is essential in the most critical areas of activity, such as the high-risk systems defined in the AI Act, such as health, surveillance or defense.

Thanks to the test reports sent by LNE, you will be able to document, for example, the robustness and explainability of the algorithms developed or used.

An AI system qualification is part of your product's risk management analysis in order to:

• Attest to its reliability
• Identify to mitigate biases by determining their cause
• Define the security of the algorithm so that it can be used in critical sectors
• Meet the need for trust of integrators and users, and thus reduce complaints for inappropriate response

Validating the operation of an AI system through a methodical qualification thus makes it possible to provide a set of proofs (documentation) and to bring you into compliance with the standards as could be required in the MDR, the Machine Directive or the AI Act.

Going through a third-party organization helps to strengthen the confidence of end-users and regulators in the AI system developed, and to facilitate its deployment to the market.

A complete proposal:

To counter cyberattacks, LNE also offers to carry out a security audit of your AI system in order to identify its various vulnerabilities and strengthen its security.

Setting up a process certification of your AI and/or ISO 42001 is also a way to meet regulatory requirements, thus attesting to the implementation of a quality process and controlled organizational management.

Data: a qualitative and environmental impact

In order to reduce the energy impact of the development of AI systems, it is necessary to use the right amount of evaluation or training data and to optimize the size of the models used. However, the reduction in the amount of data must not impact the quality of the result.

The quality and representativeness of the data used during the validation of AI are therefore essential, and must be controlled.

Thanks to their expertise in the field, our technical experts will be able to assist you in guiding you on the use, at the right level, of controlled and relevant data that will allow you to obtain a quality, reliable and robust analysis.

By doing so, you will also contribute to reducing your energy consumption in this area.

What solutions can be used to qualify an AI system?

Your algorithm or AI system can be autonomous (software, application, etc.) or integrated into a piece of hardware or a device (medical device, personal assistance robot, smart camera, etc.).

As a third-party organization, LNE has been working since 2008 through its AI Assessment Laboratory (LE. AI) to the deployment of technical solutions and the development of methods for qualifying AI systems in order to secure their use and build confidence.

Image

Within the LE. AI, you have access to a range of services that allow you to carry out simulations or immerse your system in a simulated dynamic reality, with different levels of realism. Depending on your need for validation or support, our experts will guide you on the most appropriate solution.

How can you ensure the reliability of an evaluation of an AI system?

LNE supports end users, integrators and developers of AI systems during the different phases of product development.

Thanks to our expert's expertise in data completeness and representativeness, we can support you in the development, or adjustment, of your analysis methods and the evaluation plan of your AI system.

This analysis can be completed by carrying out:

• An assessment of the performance and robustness of your system's AI features to validate it
• An analysis of the qualification of the learning or assessment database (real or synthetic)
• Technical assistance on defining the metrics to be used

Attentive to regulatory developments and able to provide an answer to your questions about the various regulations that apply to your AI system, we offer:

• Support in the interpretation of regulatory and normative requirements via a conformity assessment (e.g. literature review according to the requirements of the AI Act) and by answering your specific questions
• An assessment of the learning process or assessment of AI functionality in relation to AI certification needs
• A literature and methodological analysis of an AI feature according to its application domain (influencing factors, learning protocol and limits, evaluation protocol and defined metrics, etc.)
• An analysis of the cybersecurity of your AI system

A catalogue or tailor-made training can be offered to you depending on the practical questions you need to answer.

Examples of products qualified by LNE: video protection systems, air traffic management, voice assistants, medical devices integrating AI, etc.

Namely:

1. Metrics serve as basic metrics and help trace the origin of identified underperformance, so it's essential that they are well-defined based on the AI system and its usage.
2. The data from the product will also need to be qualified and annotated to evaluate the product as accurately as possible.

Technical expertise in general-purpose generative AI

With in-depth expertise in audio, text, images and videos, our technical team can meet your various needs for the qualification and validation of your AI system:

• Civil applications in automatic speech understanding or transcription, machine translation, diarization and detection of named entities,
• Dual voice comparison applications for forensics, translation,
• Speech recognition, character recognition (OCR, text content analysis)
• Recognition of images, people or objects, shapes in TV/video documents,
• Checking for anomalies.

LNE has tools* to evaluate automatic speech transcription systems, and can support you in the development and integration of generative AI for general use.

Our skills are being expanded across all sectors of activity that require the secure use of AI, such as health, energy, defence, surveillance, among others.

*tools:

• Datomatic for data organization and building a baseline
• Evalomatic to test the relevance of software by comparing its results to the references provided

How do you qualify a robot with AI on board?

LNE has unique infrastructures that allow you to validate your AI model scenarios embedded in a hardware device.

Through a simulation or immersion in a virtual environment, you collect data that can be used to identify any weaknesses and explore the functional limits of your solution.

Simulation will save you time on data collection and optimize your development costs.

Typical products evaluated: intelligent sensors and surveillance cameras, drones, robots with 2D or 3D cameras, Lidar, Sonar, mobile devices (autonomous vehicles, personal assistance, etc.), etc.