The fundamentals of Artificial Intelligence

Generative Artificial Intelligence & Transformer architectures

Generative AI is a subset of deep learning. It uses deep neural networks and can process both labeled and unlabeled data using supervised, unsupervised, and semi-supervised methods.

Generative AI is a class of artificial intelligence that creates new content learned from existing data distributions.

The process of learning from existing content is called "training" and results in the creation of a statistical model.

When given a prompt, generative AI uses this statistical modelMathematical equations encoding patterns extracted from training data. to predict the most probable output and generate novel content that resembles the training distribution without copying it verbatim.

To understand its uniqueness, we must compare it with the traditional (discriminative) approach.

Distinction between discriminative and Generative AI

The difference lies in the nature of the probability distribution the model seeks to learn.

Discriminative AI (traditional approach)

Discriminative models (classification, regression) focus on the decision boundary. They do not attempt to understand how data is generated, only how to separate classes.

• Mathematically, they learn the conditional probability P(Y|X): what is the probability that the label is Y given input X?
• Use cases: spam detection, facial recognition, sales forecasting.

Generative AI (creative approach)

Generative models aim to model the underlying structure of the data itself.

• Mathematically: they learn the joint probability P(X, Y) or the distribution P(X).
• Mechanism: this enables them to model how features are distributed and, through statistical sampling, generate entirely new instances.

The Transformer architecture & self-attention

Introduced in 2017 by Google Brain researchers in the seminal paper "Attention Is All You Need" (Vaswani et al.), the Transformer architecture sparked the technological breakthrough behind modern LLMs (Large Language Models) such as BERT, GPT, Gemini, and Claude.

The Transformer is a deep neural network architecture designed to handle sequential data (text, code, DNA), breaking free from the bottlenecks of earlier architectures through parallel processing rather than sequential processing.

Breaking away from recurrent networks (RNNs and LSTMs)

Before 2017, the standard for natural language processing was Recurrent Neural Networks (RNNs), which read data sequentially, word by word, from left to right.

• This sequential bottleneck meant that to process word n, the RNN had to finish computing word n-1. This prevented parallel computation across GPUs, making training slow and difficult to scale.
• The memory bottleneck: over long sequences (such as a full paragraph), the network tended to "forget" the beginning of the sentence by the time it reached the end (the vanishing gradient problem).

The Transformer solves both issues by processing the entire input sequence simultaneously rather than sequentially. It can analyze a sentence, paragraph, or entire document in a single pass, enabling massive parallel training on GPU clusters.

The self-attention mechanism

At the core of the Transformer lies the self-attention mechanism—an algorithm that allows the model to weigh the importance of every word in a sequence relative to all other words in order to build an accurate contextual representation.

Unlike older static embedding methods like Word2Vec, where a word with multiple meanings had a single fixed vector representation, the Transformer dynamically recalculates a word's meaning based on its surrounding context.

Educational illustration:

Consider the sentence: "The animal did not cross the street because it was too tired." To a computer, the pronoun "it" is ambiguous. To resolve this ambiguity, the attention mechanism computes a relevance score (attention weight) between "it" and every other word in the sentence. Mathematically, the model assigns a high attention weight between "it" and "animal", and a low weight between "it" and "street". The vector representing "it" thus absorbs the contextual features of "animal". Conversely, if the sentence were "The animal did not cross the street because it was too wide," attention would focus on "street".

From unsupervised to self-supervised learning

Transformers require colossal volumes of training data. Manually labeling petabytes of text—marking every subject, verb, or entity—is humanly impossible. Instead, they rely on Self-Supervised Learning.

The model generates its own supervisory signals directly from raw text:

• Masked Language Modeling: words are randomly masked in a sentence (e.g., "The cat chases the [MASK]") and the model must predict the missing token using bidirectional context.
• Next-Token Prediction (Causal Language Modeling): the model learns to predict the next token in a sequence, as used by GPT (Generative Pre-trained Transformer) models.

It is this ability to learn autonomously across the entire web, combined with the parallel architecture of Transformers, that gave rise to the apparent "intelligence" of today's foundation models.

Core differences between training data and weights

To understand the internal mechanics of Transformers, it is essential to distinguish external training resources ("data") from the internal processing and storage mechanism ("parameters").

"Data" refers to the training corpus—raw information external to the model that the algorithm analyzes during training:

• It consists of billions of sentences, books, software repositories, and web articles.
• It serves as instructional material: during self-supervised pre-training, the model "reads" this data to extract statistical regularities.
• Once training is complete, the raw data is not stored inside the model. It was used to calibrate the network and is then discarded. The model does not "contain" texts like a relational database; it retains their statistical representation.

"Parameters" form the brain of the model—the internal variables of the artificial neural network that define how the model processes information:

• They consist of numerical values (weights W and biases b) organized into massive multidimensional matrices. These billions of parameters are adjustable variables within the network.
• Each parameter is comparable to a microscopic tuning dial or an artificial synapse.
• Initially, parameters are set to random values (the model knows nothing). During training, the optimization algorithm progressively adjusts each parameter to minimize prediction error across the training corpus.
• The frozen set of trained parameters constitutes the final "model" (such as GPT-4 or Claude). They are the mathematical crystallization of what the AI has learned.

Why do Large Language Models (LLMs) hallucinate?

In the context of language models, a hallucination refers to the generation of content that appears coherent, syntactically fluent, and confidently stated, yet is factually inaccurate, ungrounded in the training corpus, or logically contradictory to the provided prompt.

Unlike classic software bugs, hallucination is not a technical malfunction, but an intrinsic consequence of the probabilistic nature of LLMs.

Distinction between syntax and semantics

It is essential to correct a common misconception: modern hallucinations are almost never grammatical errors.

• Thanks to the Transformer architecture, models master grammatical and stylistic rules effortlessly. The generated sentence is structurally flawless.
• Semantic drift: the error occurs at the level of factual meaning. The model fills "gaps" in its probability distribution with the most statistically plausible sequence of words, even if the resulting statement is factually false.

Simply put, an LLM is optimized for plausibility (what sounds like a convincing human answer) rather than veracity (what is objectively true).

Technical causes of hallucinations

• Lossy compression: an LLM does not store facts like a database; it stores statistical associations across its weights. During training, it compresses a vast amount of web text into a finite number of parameters, causing a loss of precision on rare or specific facts.
• Next-token prediction objective: the model has no internal concept of "truth" or the "real world". Its sole mathematical objective is to predict the most probable next token. If a factually incorrect token has a high probability in context, the model will select it.
• Training data noise: if the training corpus contains contradictions, outdated claims, or falsehoods (as the open internet inevitably does), the model will reproduce those inconsistencies.

Typology of hallucinations

In technical analysis, two primary categories of hallucinations are distinguished:

1. Intrinsic hallucinations, which directly contradict the source context provided in the prompt (for example, given meeting notes stating a meeting took place on Monday, the summary claims it occurred on Tuesday).
2. Extrinsic hallucinations, where the model fabricates external facts unverifiable from the context (for example, inventing non-existent legal case law or attributing a quote to the wrong author).

Mitigating hallucinations

While hallucinations cannot be completely eliminated from a probabilistic engine, proven engineering techniques keep them under tight control:

• RAG (Retrieval-Augmented Generation): the generative model is connected to a trusted external knowledge base and instructed to formulate its answer strictly from retrieved documents, drastically reducing fabrication (grounding).
• RLHF (Reinforcement Learning from Human Feedback): during post-training alignment, human evaluators penalize fabricated answers, training the model to state "I don't know" rather than invent facts when uncertain.

Steering the system: Prompt Engineering

Prompt engineering is the art and science of designing and optimizing text inputs (prompts) to effectively guide a Large Language Model (LLM) toward a specific output without modifying its internal weights.

Unlike traditional deterministic programming, prompting constrains the model's probability distribution to steer generation. This is known as "in-context learning": the model learns temporarily from the information supplied inside its attention window, and that temporary state vanishes as soon as the session ends. The model's underlying weights remain frozen.

Here are the foundational techniques every business leader should know:

• Zero-shot prompting: asking the model to perform a task without providing any prior examples (e.g., "Translate this text into English").
• Few-shot prompting: providing the model with a few "input-output" examples inside the prompt to demonstrate the expected format or logic before asking the actual question. This dramatically improves output reliability.
• Chain-of-Thought (CoT): instructing the model to break down its reasoning step by step ("Think step by step") before delivering the final answer. This enables the model to solve complex mathematical or logical problems while reducing hallucinations.

Fine-tuning: domain adaptation and alignment

Fine-Tuning is a supervised learning process performed after a model's initial pre-training. It is a transfer learning technique.

Unlike Prompt Engineering, Fine-Tuning permanently updates the internal parameters (weights) of the neural network.

The process involves taking a general-purpose pre-trained foundation model (such as Gemini or Llama) that has absorbed broad web knowledge, and training it further on a smaller, highly curated, domain-specific dataset.

Why use fine-tuning?

There are three primary reasons to fine-tune a model:

1. Domain specialization: adapting a model to medical, legal, or proprietary enterprise terminology and codebases.
2. Style and format alignment: training the model to consistently respond in a strict tone, schema, or structure (e.g., structured JSON, brand voice, or standardized compliance reports).
3. Cost and latency reduction: a smaller, well-fine-tuned model can often match or exceed the performance of a giant general-purpose model on a specific task at a fraction of the inference cost and latency.

Note on PEFT (Parameter-Efficient Fine-Tuning): today, retraining all billions of parameters in a model is rarely necessary. Techniques like LoRA (Low-Rank Adaptation) modify only a tiny fraction of parameters (often less than 1%) to achieve comparable domain adaptation at minimal compute cost.

Multimodal Artificial Intelligence

Modern AI is converging toward multimodality—the ability to simultaneously process and relate different signal types (text, images, audio, video) within a unified mathematical representation (latent space).

Unlike unimodal models built for a single task (such as a text-only translation model), a multimodal system can process, understand, and generate information across multiple signal types or modalities simultaneously: text, images, audio, video, code, and sensor data.

Core modalities

• Text-to-Text: reasoning, translation, summarization.
• Text-to-Image / Image-to-Text: visual synthesis or image understanding via diffusion networks and visual encoders.
• Natively multimodal: models (such as GPT-4o or Gemini) trained from the ground up on interleaved media, capable of "seeing" and "hearing" directly without separate transcription pipelines.

AI agents (text-to-action)

The next frontier is the AI agent: rather than merely generating a text response, it can invoke external tools (web browsing, code execution, API calls, CRM updates) to autonomously execute multi-step workflows.