The problem of bias in AI systems is often discussed as if it were a problem of the algorithm. In reality, however, AI learns from us – from our decisions, our data, and our biases. If we truly want to talk about non-discrimination in AI, we must begin by changing our own ways of thinking and acting.
Law and social reality
The debate on discrimination in AI systems is increasingly focusing on the question of whether existing and proposed legal frameworks are capable of genuinely preventing inequalities before they turn into concrete violations of individual rights. This question is particularly significant in the context of protecting human rights in a world destabilised by conflicts, nationalism, and a return to a discreditable past, especially when it comes to gender equality.
The Artificial Intelligence Act (AI Act), although criticized by many, represents an important shift in thinking about the relationship between humans and technology. The very move from reacting to problems that have already occurred toward earlier risk management is a significant step. However, it does not seem to be sufficient. Why? There are at least two reasons.
Two sources of the problem: social and technological
The first is social and cultural in nature. We still too rarely ask ourselves why so many people – including women themselves – do not perceive the problem of discrimination or do not see the need to use feminine forms of job and professional titles (so-called feminatives). Why do young children still understand that a doctor, lawyer, scientist, or pilot can be either a woman or a man, yet a few years later, when asked to draw a scientist, they overwhelmingly draw a man? In languages such as Polish, where both masculine and feminine forms exist, this is not merely a linguistic issue. Language shapes awareness. Language reinforces social patterns of “visibility” and “invisibility.”
The second reason is structural and technological. We too often speak about AI as if it were a neutral tool, as if it simply processed information and produced objective results. This is not how it works. Every AI system operates not on reality itself, but on a model of it and a model is always the result of a series of prior decisions. Someone selects the data, someone decides which features are relevant, someone determines what will be considered a desirable outcome. In other words, before the system begins to “decide,” an entire chain of decisions has already been made about what is to be visible, measurable, and relevant.
The myth of technological neutrality
In this sense, the neutrality of technology is largely a myth. Every technical architecture reduces something. It reduces a human being to data, context to variables, and a decision to an outcome that fits within a defined threshold. From a technical perspective, this is necessary, because a system cannot operate on the full complexity of human life. The problem, however, is that we often mistake this necessary simplification for objectivity.
A model does not understand a person in a social, moral, or legal sense. It sees rather a statistical profile, correlations, and probabilities. The law, however, protects the person, not a statistical pattern. This tension is fundamental. Technology simplifies, whereas the law must take into account dignity, context, and the individual situation of a human being. In practice, this means that a system may be highly effective according to its own logic and at the same time deeply unjust from the perspective of the individual. It may perfectly optimise accuracy, speed, or cost reduction, but it cannot, by itself, optimise for dignity, equality, or justice. These values do not appear in the model automatically. They must be introduced through design, organisational, and legal decisions.
The AI Act – important but insufficient
Regulations, therefore, although very necessary, do not solve the broader problem of discrimination. The AI Act focuses on risk management, human oversight, data quality, and control mechanisms. All of these elements are essential. Although, they assume that every risk can be identified and described early enough. In reality, some risks only become apparent in practice – when the system begins to operate within a specific institutional context, affects real people, and becomes embedded in concrete organisational practices. Therefore, the AI Act should be treated rather as a good starting point than as a final solution to the problem.
It is crucial to understand – also by supervisory authorities – that AI systems do not merely reflect social reality but also co-shape it. They simplify it, eliminate edge cases to generate more consistent and scalable results, and thus may reinforce existing mechanisms of exclusion.
Where does bias arise?
This leads to another problem: even if discrimination has already occurred, detecting and proving it can be extremely difficult. When we talk about bias, we too often focus exclusively on the algorithm itself. In reality, bias can arise at many stages of an AI system.
First, at the input stage. Data may be incomplete, historically biased, or simply unrepresentative. If past decisions reflected inequalities, the system will easily learn to reproduce them. Second, the problem may lie in the model itself. Someone defines the objective function, someone decides what exactly is to be optimised: efficiency, cost reduction, speed. These are not neutral technical settings, but normative decisions that influence who will be rejected, classified as risky, or deemed unreliable. Third, the problem appears at the output stage. Even if the system is formally intended only to support human decision-making, in practice its recommendation may function as a strong suggestion that no one actually questions (the problem of over-reliance on AI).
A good example is credit scoring systems. Two individuals may have very similar financial profiles, but if the system uses location as a proxy variable, it may generate different results solely due to place of residence. This leads to indirect discrimination. Similar mechanisms have been observed in recruitment, where systems trained on historical data learned to favour male candidates because they reproduced previous hiring patterns. In healthcare, systems have been used that treated cost as a proxy for medical need. Because some groups historically had poorer access to healthcare, they generated lower costs, and the system inferred that they needed less care. This clearly shows that so-called bias is not merely a technical error. It is most often a reflection of historically shaped social reality.
The evidentiary problem and access to legal protection
From a legal perspective, proving such discrimination is particularly difficult. Traditional anti-discrimination law instruments rely on comparing situations, identifying a clear criterion of differentiation, and establishing a causal link. In AI systems, however, the logic of decisions is often dispersed, opaque, and difficult to reconstruct. Without access to data, documentation, system logic, and the manner of its deployment, the right to effective legal protection may become illusory.
We therefore need not only tools such as algorithmic audits or outcome testing for discriminatory effects, but also rights that make these tools genuinely usable. This primarily means access to relevant datasets, obligations of transparency and explainability, and in certain situations also a shift in the burden of proof. The European Union attempted to address some of these issues through the proposed AI Liability Directive. Although not perfect and often a far-reaching compromise, it was an attempt to create procedural instruments better adapted to AI-related disputes. Unfortunately in the context of a deregulatory shift, the proposal was withdrawn. As a result, we must still rely mainly on the national legal systems of EU Member States, which are diverse and often not adapted to AI-related cases.
Against this background, the question of responsibility becomes particularly important. Who should actually be responsible for preventing discrimination in AI? The traditional approach to liability does not fit the architecture of these systems. AI does not operate linearly. It is an ecosystem. We have developers designing the model, data providers, AI system providers integrating different models and adapting them to a specific context, organisations deploying the system, and individuals using it in practice. Responsibility is therefore distributed. The AI Act partially addresses this issue by indicating what the provider of a high-risk AI system is responsible for and emphasising that its role in ensuring compliance is crucial. After all, many of the most important decisions are made long before the system is deployed in an operational environment. If an AI system is designed from the outset to maximise efficiency or reduce costs, it may systematically worsen the situation of certain groups. Controversies often concern not only the results themselves, but already the way “risk” is defined and measured. In terms of liability, we should therefore rather ask who had control, who could foresee the risk, and who had a real possibility to prevent the harm.
“Ethics by design” – marketing slogan or real obligation?
A similar operationalisation must be applied to concepts such as “ethics by design” or “fundamental rights by design.” These are very popular today, but if they are not translated into measurable obligations, they will remain marketing slogans. Ethics cannot be an add-on to a finished product. Ethical issues arise from the very beginning: when defining the objective, selecting data, setting evaluation metrics, and determining decision thresholds. These are not purely technical decisions, but decisions reflecting specific values and priorities.
Examples of facial recognition systems that achieved significantly worse results for women and people with darker skin tones do not simply demonstrate a “technical flaw.” They show the consequences of specific decisions regarding data and system design. If we take ethics seriously, we must translate it into obligations such as mandatory impact assessments – for fundamental rights and data protection – systematic testing across different groups, proper documentation of system limitations, and ensuring real, not merely apparent, human oversight.
Equality ≠ sameness
At this point, one more important reservation must be added: equality does not mean sameness. Not every difference in outcomes is automatically unlawful discrimination. The law does not prohibit all differentiation between people or all differences in decision outcomes. It primarily prohibits differentiation that is arbitrary, disproportionate, or based on prohibited grounds.
This is important because in the debate on AI it is very easy to fall into the simplification that every statistical inequality must mean unlawful discrimination. This is not the case. What matters is the basis on which differentiation occurs, the purpose it serves, and whether it can be objectively justified. Sometimes differentiation is not only permissible but even necessary. Real equality does not consist in mechanically treating everyone identically, regardless of their situation. Such automatism may be convenient for the system, but it is not always fair for the individual.
In many cases, meaningful differentiation better protects individual rights and better responds to real needs. This applies both to people in a more difficult situation, requiring additional support or protection, and to those whose competencies, abilities, or predispositions justify different treatment. A person with a disability, a digitally excluded person, or someone in a particularly vulnerable social or economic situation may require a different approach in order for the outcome to be genuinely fair.
AI technology is very good at classification, segmentation, and prediction. It is much worse at understanding which differences between people have legal and moral significance and which should not influence a decision at all. This is precisely why the apparent neutrality of a system may make it harder to notice injustice. Not every difference in outcome is a problem, but the problem arises when a system differentiates people according to illegitimate, hidden, or disproportionate criteria, and its technical form masks this fact.
The problem of asymmetry
The legal remedies at our disposal are insufficient for a person affected by an AI system’s decision. The main problem is asymmetry. The person concerned often does not even know that an AI system was used, does not understand how the decision was made, and has no real possibility to challenge it. Transparency in the use of AI by institutional and business users is therefore crucial. It consists not only in knowledge that an AI system is being used, but also in an appropriate level of explainability and effective enforcement mechanisms.
We therefore need better expertise among providers and users of AI systems, as well as more critical reflection on technology design on the one hand, and well-prepared and competent supervisory authorities on the other, so that our legal systems, organisations, and society are capable of responding to the enormous scale of the reproduction of inequalities.
