NLP & LANGUAGE

Tokenization

NLP & Language

Definition

Tokenization is the process of splitting a text into smaller units (tokens) that the AI model can process. Different models use different strategies: some split by words, others by subwords. Understanding tokenization helps estimate API costs (calculated per token) and optimize prompts to stay within the context window limits.

Related terms

AI Fundamentals

Token

A token is the basic unit of text processed by language models. It can correspond to a word, part of a word, or a punctuation character. For example, 'automation' might be a single token, while 'automatization' could be split into multiple tokens. The cost of AI services is often calculated based on the number of tokens processed.

NLP & Language

Context Window

The context window is the maximum amount of text (measured in tokens) that an AI model can process in a single prompt. Models with larger context windows can analyze longer documents at once. Claude offers windows up to 200K tokens, allowing you to analyze entire technical manuals or complex contracts without splitting them.

NLP & Language

NLP (Natural Language Processing)

NLP is the field of AI concerned with the interaction between computers and human language. It includes tasks such as sentiment analysis, entity extraction, machine translation, and text generation. For SMEs, NLP enables automating the analysis of emails, customer reviews, legal documents, and internal communications.

EXPLORE

More terms in NLP & Language

NLP (Natural Language Processing)

Sentiment Analysis

Sentiment analysis is the NLP technique that automatically determines the emotional tone of a text: positive, negative, or neutral. Companies use it to monitor customer reviews, analyze brand sentiment on social media, and quickly identify satisfaction issues. It can be applied to thousands of texts in seconds.

Named Entity Recognition (NER)

NER is an NLP technique that automatically identifies and classifies entities such as names of people, companies, locations, dates, and monetary amounts within a text. In business, it is used to extract key information from contracts, invoices, emails, and legal documents, saving hours of manual data entry work.

BERT

BERT (Bidirectional Encoder Representations from Transformers) is a language model developed by Google that understands the context of a word by looking both left and right in the text. It is particularly effective for comprehension tasks such as classification, question answering, and semantic search. Many enterprise search systems use BERT behind the scenes.

Text Classification

Text classification is the task of automatically assigning one or more categories to a document. Practical applications include: routing support tickets by priority, cataloging emails by department, classifying accounting documents by type. Modern models achieve accuracy above 95% on many business classification tasks.

Information Extraction

Information extraction is the automatic process of identifying and structuring relevant data from unstructured texts. For example, extracting order numbers, delivery dates, and amounts from supplier emails, or personal data from scanned documents. It drastically reduces data entry time and human errors in SMEs.

See all glossary terms →

Want to apply AI in your business?

Talk to us. The first call is free and no commitment.

Book a call