Glossary / AI
Feature
In interpretability research, a unit of an AI's internal representation that corresponds to a human-understandable concept, such as the Golden Gate Bridge.
Individual neurons in a model often respond to several unrelated concepts at once, so they cannot be read directly. A 2022 study by Nelson Elhage and colleagues explained this as superposition: the model packs more concepts than it has neurons by overlapping them[1].
Features are what you get by re-decomposing neuron activity into combinations of concepts. Artificially strengthening or suppressing a feature has been shown to shift the model's answers along that concept.
Related terms
- Mechanistic interpretabilityThe research field that tries to read an AI's internal computation part by part and explain, as a mechanism, why it produced an answer.
- Sparse autoencoderAn auxiliary neural network used to decompose an AI's internal activity into combinations of a very large number of features, only a few of which are active at once.
- NeuronThe smallest unit of computation in a neural network. It weights and sums its inputs and produces an output depending on the result.