# Why AI Trustworthiness is defined before development begins > A practical systems approach to trustworthiness analysis, AI Assurance and Human-Centred Design for AI-based systems in high-risk environments. Published: 2026-07-22 | Author: Kseniia Goncharenko Canonical: https://welldigit.com/en/journal/ai-trustworthiness-before-development Artificial intelligence becomes a part of operational workflows where decisions carry real consequences. The question is no longer whether an AI technology is capable of producing useful outputs, but whether the system as a whole can be trusted to support human decision-making. Before any AI technology is selected and building is started, we work through three engineering questions: what the system is permitted to know and do, how we can demonstrate that it behaves as intended, and what information a human must see before acting on its output. How these questions are answered determines whether trustworthiness is built into the system or added around it later. Our approach is structured around three connected engineering blocks:

Trustworthiness analysis

which focuses on understanding the system before it is built.

AI assurance

which demonstrates that the system continues to perform as intended within its defined conditions of use and when requests fall outside its authorised operational scope.

Human-centred design

which ensures that AI-generated outputs are understandable, relevant, and support informed operational decision-making.

Together these blocks provide the engineering foundation for responsible AI development in high-risk environments. Under the EU AI Act's classification of high-risk AI systems, this approach is becoming relevant to a growing range of AI-enabled products and decisions that organisations must design, deploy, and govern responsibly. ## AI risk assessment requires more than failure analysis The EU AI Act formalises what many organisations already recognise: that AI used in high-risk environments requires more than performance testing. It also requires AI risk management. What is less often discussed is why standard risk analysis tends to fall short, even when it is done carefully. Most methods concentrate on failure modes: the system provides an incorrect answer, hallucinates, or generates an unexpected output. These are genuine risks, and identifying them is important, but they account for only part of the overall issue. > AI is not deterministic. A system that produces a correct output today may produce an incorrect one tomorrow under conditions that seem identical. This is why AI risk assessment is different from standard software testing. AI risk assessment cannot be limited to a list of system failures and generally accepted practices. To ensure the reliability of an AI-based system, a detailed trustworthiness analysis is required. It provides a level of specificity that general risk frameworks cannot deliver. ## What does AI trustworthiness analysis require for high-risk AI systems?

A complete AI trustworthiness analysis requires defining end users and their responsibilities, high-level tasks with AI-role, operational scenarios including degraded conditions (when request is outside its scope), system authority boundaries, and a functional hazard assessment that maps each system function to its potential failure consequences. General risk frameworks and model evaluation alone are insufficient.

The first question we ask is not, "What can this AI do?" It is, "What is this AI permitted to do, for whom, and within which operational domain?" This question is particularly important for logic- and knowledge-based AI systems (LKB AI), or in simple words AI-driven knowledge base management systems, where system behavior is determined not only by model capabilities but also by the quality, scope, and governance of the knowledge they are authorised to access. Two further concerns belong in this same stage.

The first is security threats at the point of knowledge ingestion: what happens if someone tries to corrupt the knowledge the system relies on or feeds it content designed to manipulate its output. Knowledge base poisoning and prompt injection are the most relevant threats for logic- and knowledge-based AI systems. It is easier to control what enters the system than to correct what it has already generated.

The second is bias and ethical considerations. Before anything is built, we define whether the system could reinforce unfair treatment of anyone it affects, and we design against automated scoring or ranking of individuals wherever that risk exists.

## From risk identification to continuous monitoring Once the system boundary is defined, AI risk assessment begins to determine the level of risk associated with the system and whether that risk is acceptable for its intended operation. For each function the system performs, in each operational scenario, we identify what could go wrong and what the consequences would be for the end user and for the operation. The assessment covers the system as a whole: the AI output, AI role in appropriate function, and the contribution of the authorised person who reviews each output before it is used. A failure that reaches a qualified reviewer is not the same as one that reaches a system with no oversight layer. Each function hazard is assessed on two dimensions: severity and likelihood. Together they determine the risk level and what is required in mitigation. > A listed risk does not mean that it has already been mitigated. Mitigation requires a specific design solution, architectural control, or procedural requirement that reduces likelihood or severity. Every mitigation should be documented. Every residual risk should be evaluated for acceptability in the intended operational context. And the work continues after launch. Assumptions change. Sources are updated. Environments evolve. Continuous monitoring keeps the assessment valid throughout the system's operational life. ## AI Assurance: from verification and configuration management to operational monitoring Defining trustworthy behaviour is only the first step. AI Assurance provides evidence that the system continues to behave in accordance with that definition throughout its operational life. An AI-based system can produce a confident, well-structured answer from a knowledge base that is outdated, incomplete, or wrong. That failure mode does not appear in a demo. It appears months later, when someone acts on information the system was certain about, and should not have been. This is why we treat the knowledge base itself as something that has to be assured, separately from the model that draws on it. Assurance is not limited to verifying correct outputs. It also demonstrates that the system consistently behaves as specified, including when information is unavailable, uncertain, or falls outside its authorised operational scope. > If a request falls outside the defined boundary, the AI-based system says so. It does not improvise, extrapolate, or produce an answer that appears plausible but is not grounded in anything it was authorised to know. Every piece of information the system uses is versioned and dated. We verify, rather than assume, that the knowledge base covers what it is intended to cover, and we re-verify that coverage continuously, not only at launch. Whenever a source document, knowledge base, or model version changes, we re-run the same verification scenarios that originally validated the system, rather than assuming yesterday's correctness still holds. None of this is visible to the end user during normal operation. That is precisely the point. Assurance is the engineering discipline that makes the visible parts of the system trustworthy. When done properly, it prevents failures before users ever encounter them. ## Human-Centred Design: building for independent judgement This is also the part that requires the most deliberate design efforts. An AI system working in a high-risk environment should not ask a person to trust its output. It should give that person what they need to judge the AI-generated output for themselves. Every AI recommendation the system produces should carry a traceable source, a confidence indication, and a path back to the underlying source. The person using it can check the reasoning before they rely on it, not after. There is a more difficult problem beneath this, and it deserves to be named directly. Adding a human review step is the standard answer to "how do we keep AI in check." But a reviewer does not begin from a neutral position. Once someone sees what the AI recommended, that recommendation shapes what they notice and how they interpret the evidence before them, even when they believe their judgment is independent. Human-in-the-loop does not automatically restore that independence simply by existing. It must be intentionally designed for. This is why traceability is not merely a form of documentation; it is the foundation of explainability. It is the mechanism that enables a person to reconstruct the reasoning, not just approve or reject a conclusion they did not form themselves. That difference changes what the interface must display and how the system must record what it retrieved, why, and with what level of confidence. > A system designed this way respects the professional judgement of the person using it, rather than replacing it. ## Why we build this way As AI adoption accelerates, discussions about trustworthiness have become increasingly common. Hallucinations, explainability, bias, transparency, robustness, governance, and regulatory compliance dominate AI conferences, white papers, and product roadmaps. However, our approach was shaped by more than these discussions. Years of work in aviation, a safety-critical environment, taught us that incorrect information in the hands of a qualified person can have serious consequences, and that an AI-based system producing outputs without giving that person enough basis to assess them is not a neutral tool. It becomes a source of risk within the workflow. But experience also showed us that well-designed systems act as safety barriers. They can prevent errors before they reach the people who would otherwise need to detect them on their own.

These observations remain valid regardless of industry or project size. The principles of robustness and trustworthiness must be established during AI-based system development. Adding them later is harder, more expensive, and often incomplete.

If you are working through these questions for your own AI-based system, we'd be happy to share how we approach them in practice.