A model parameter is a learned numerical value inside a machine learning model. In neural networks, parameters are primarily weights and biases that are adjusted during training to reduce prediction error.
In a Large Language Model (LLM), parameters encode statistical structure learned from training data. They influence token probabilities, factual associations, style, reasoning patterns, and many other behaviors, but they are not a database of explicit records.
Parameter count is often used as a rough measure of model scale. Larger models can have more capacity, but parameter count alone does not determine quality. Architecture, training data, tokenization, post-training, inference compute, and evaluation coverage all matter.
Some training methods update all parameters. Parameter-efficient methods such as LoRA (Low-Rank Adaptation) update a much smaller set of adapter parameters while leaving the base model frozen.
Sparse architectures complicate parameter-count comparisons. A sparse expert model may contain many total parameters but activate only a subset for each token.
Google's Machine Learning Glossary defines a model as a learned set of parameters and structure in its model entry.
The LLM Knowledge Base is a collection of bite-sized explanations for commonly used terms and abbreviations related to Large Language Models and Generative AI.
It's an educational resource that helps you stay up-to-date with the latest developments in AI research and its applications.