awesome-llm-interpretability

Series of posts discussing open research problems in the field of Mechanistic Interpretability (MI), which focuses on reverse-engineering neural networks.

Proposes the Quantization Model for explaining neural scaling laws in neural networks.

Explores the phenomenon of 'grokking' in deep learning, where models suddenly shift from memorization to generalization during training.

Discusses hyperdimensional computing, a novel method involving hyperdimensional vectors (hypervectors) for more efficient, transparent, and robust artificial intelligence.

Method that uses gradients for a linear approximation of activation patching in neural networks.

Introduces causal scrubbing, a method for evaluating the quality of mechanistic interpretations in neural networks.

Group of researchers focusing on AI alignment.

Non-profit AI research lab that focuses on interpretability and alignment of large models.

Research group discussing various topics on interpretability.

at Google work on

Identifies backward chaining circuits in a transformer trained to perform pathfinding in a tree.

A comprehensive exploration of the initial five layers of the InceptionV1 neural network, focusing on early vision.

Examines small neural networks to understand how they learn group compositions, using representation theory.

Use LLMs to build interpretable classifiers of text data

Causal mediation analysis as a method for interpreting neural models in natural language processing.

ChainPoll, a novel hallucination detection methodology that substantially outperforms existing alternatives, and RealHall, a carefully curated suite of benchmark datasets for evaluating hallucination detection metrics proposed in recent literature.

. This survey paper provides an up-to-date review of the literature on LLMs, which can be a useful resource for both researchers and engineers..

Analyzing attention maps from BERT transformer.

Code for automatically generating, simulating, and scoring explanations of neuron behavior.

A carefully compiled list that summarizes the diverse functions of the attention heads.

A local web app for transformer activation tracing, causal interventions, and sparse autoencoder experiments.

Comgra helps you analyze and debug neural networks in pytorch.