Can Sparse Autoencoders Explain Why Agents Choose Tools?

Dataiku’s open-source Kiji Inspector applies sparse autoencoders to agent decision tokens to investigate why agents choose specific tools, according to an InfoQ presentation published November 12, 2026. ¹ The question for teams running agents is whether these internal signals diagnose a decision or simply support a plausible account of it.

The presentation also discusses causal ablation, layer-activation extraction via patched vLLM engines, and production monitoring workflows. ¹ These are ingredients for an investigation, but their inclusion alone does not establish that a particular feature caused a tool choice.

What problem is Kiji Inspector trying to solve?

In the presentation, “Opening the Black Box: Critical Explainability for AI Agent Tool Selection with Kiji Inspector,” Hannes Hapke describes Dataiku’s project applying sparse autoencoders directly to decision tokens to investigate agent tool selection. ¹

A tool choice is an operational decision: it determines what an agent does next. A post-hoc explanation can sound reasonable without showing which internal signals drove the choice. The presentation frames its approach as moving beyond superficial chain-of-thought narratives. ¹

Analysis: An explanation useful for oversight should do more than tell a coherent story. It should help investigators distinguish cases, test a hypothesis, or predict what might change when a suspected signal changes. That is a standard for evaluating the method, not a result the presentation description claims to have established.

The description does not specify the models used, layers examined, meaning of “decision token,” or measured results. ¹ Without those details, readers cannot assess reproducibility, coverage across agent configurations, or the strength of the evidence from the description alone.

What sparse autoencoders on decision tokens can and can’t show

The presentation says Kiji Inspector applies sparse autoencoders directly to decision tokens to expose why agents pick tools. It also mentions extracting layer activations. ¹ The description does not explain how decision tokens are defined or how features are identified.

Analysis: A feature associated with a tool choice is not necessarily the reason for it. It could track the prompt topic, an instruction, or another property that co-occurs with the tool call. A correlation may still help investigators identify patterns worth testing. Tool choice can also depend on whether the agent had enough information in the first place, a separate question explored in When Should an Agent Ask for More Information Before Deciding?

Treat a candidate feature as a lead, not a verdict. Ask:

• Does it recur across comparable decisions to use the same tool?
• Does changing the feature change the tool choice?
• Does that change leave unrelated behavior intact?

These are proposed evaluation questions, not findings reported in the source. The description does not say which features were validated or how well they generalized. ¹

A visualization can help a team form a hypothesis. By itself, it is not evidence that the hypothesis is correct.

What causal ablation for AI agents adds — and what it still doesn’t settle

The presentation description says the talk discusses causal ablation. ¹ Analysis: One way to evaluate a candidate signal is to alter or remove it and check whether tool-selection behavior changes. The source does not specify an ablation procedure or report results. ¹

An activation correlated with a tool choice is an observation. A controlled intervention that changes the targeted choice may provide stronger evidence, but it still needs controls: if the intervention broadly degrades behavior, the changed choice does not establish that the candidate feature was specifically responsible.

Analysis: Test whether an intervention affects behavior beyond the decision under study, causes broader degradation, or has effects limited to particular prompt contexts. Even a change in tool choice would not show that one feature was the sole cause; agent behavior may depend on multiple signals.

The practical bar should remain modest: ablation strengthens an explanation only when investigators can distinguish the targeted effect from collateral changes.

What instrumentation needs to exist before any of this runs in production

The presentation covers extracting layer activations through patched vLLM engines and describes production monitoring workflows for software architects and senior engineering leaders. ¹ It does not specify supported models, layer configurations, runtime overhead, or deployment topology. ¹

Analysis: Before building a monitoring pipeline, map each activation capture to the decision it is intended to explain. A team might preserve prompt and tool context, identify the decision token, capture the activations under study, record the selected tool, and retain metadata needed to reproduce the analysis. This is an implementation checklist, not a design reported by the presentation.

There is a practical trade-off: too little context may make attribution difficult, while more capture can increase storage and processing demands. Measure those costs rather than assuming an offline interpretability workflow transfers directly to live serving. Evaluate recorded cases separately from live monitoring, and confirm which models and configurations the instrumentation supports before relying on it.

What breaks when interpretability meets production oversight

The description covers production monitoring workflows but reports no production performance, failure rates, or deployment outcomes. ¹ Describing a monitoring approach does not establish that it has run reliably at scale.

Analysis: Before using this approach for oversight, a team should ask:

• How often does the system surface a candidate feature?
• How often does an ablation change the targeted decision?
• How often are results ambiguous or difficult to reproduce?
• Can investigators act on a finding and observe the expected effect?

Those answers must come from the team’s own evaluation setting; the source provides no benchmark numbers or thresholds. ¹

Prompts, models, tools, and routing policies may change over time, so a feature that once tracked a decision may stop being informative. A feature may correlate with a choice without causing it, and an ablation that changes a choice may also affect other behavior.

Analysis: Interpretability output belongs alongside ordinary traces, task evaluations, and policy checks—not in place of them. A feature plot or coherent narrative should not be the sole approval signal for an agent decision. For a related view of agents as an operations problem rather than an autonomy demo, see a physics experiment co-pilot as a test of operational AI.

How to evaluate agent tool selection explainability before relying on it

Analysis: Start with a small set of known tool-selection cases. Identify candidate sparse features, test them with controlled ablations, and compare the results with observed choices. Add controls designed to challenge the explanation: cases where the same tool is chosen in different contexts, cases where a candidate feature appears without that tool being selected, and interventions that should not affect the decision. These tests can help distinguish a feature specific to the choice from one that merely accompanies similar inputs.

Track whether investigators agree on the evidence, results reproduce across runs, ablation changes the intended decision, collateral behavior shifts, and monitoring costs. The presentation reports no thresholds to adopt. ¹

A reasonable rule: use the method for diagnosis once it produces repeatable, decision-specific evidence. Do not treat a feature plot or plausible account as a proven explanation until it survives intervention and controls.

What to watch next

The presentation brings together sparse autoencoders applied to decision tokens, causal ablation, patched vLLM activation extraction, and production monitoring as elements of an investigation into agent tool selection. ¹ The description does not provide the experimental setup, model and layer choices, ablation results, production behavior, or stated limitations. ¹

Analysis: For platform leaders, the question is whether the organization will fund the instrumentation and validation needed to turn internal signals into repeatable evidence—and whether that evidence is strong enough to affect an oversight decision. Teams that cannot inspect model internals at all face a different problem; one angle is covered in Can Pitfall-Anticipation Models Help Steer Black-Box Agents?

Kiji Inspector may help teams investigate why an agent chose a tool. ¹ Watch for published ablation results, named model and layer configurations, and reproducibility evidence from teams running it. Until those are available, treat its output as evidence to test, not an explanation to trust.

FAQ

What is Kiji Inspector used for?

The InfoQ presentation describes Dataiku’s open-source Kiji Inspector as a tool for investigating why AI agents choose particular tools, using sparse autoencoders on decision tokens. ¹

Do sparse autoencoders prove why an agent selected a tool?

No conclusion of that strength is established by the available source description. A feature associated with a choice is evidence to investigate; causal ablation and suitable controls are needed before treating it as causal evidence. ¹

What is causal ablation in agent interpretability?

In this context, causal ablation refers to testing a candidate internal signal by altering or removing it and checking whether the agent’s tool-selection behavior changes. The presentation is described as discussing causal ablation, but the supplied source does not specify its exact procedure or results. ¹

What engineering infrastructure does this approach require?

The presentation discusses extracting layer activations with patched vLLM engines and production monitoring workflows. ¹ The supplied source does not specify the required models, layers, runtime cost, or deployment configuration, so teams would need to validate those before adoption.