AdKnob: Ad Intensity Control and Labeling for LLM-Native Advertising

Unaddressed Questions in LLM-Native Advertising

① How strong should the ad be?

A provider needs control over how prominently an ad appears to offer subscription tiers, honor advertiser contracts, and govern context, for example suppressing ads for an urgent medical question while giving them more room in a shopping query.

② Which parts of the answer are ads?

A native ad has no structural marker, so nothing separates the promotional sentences from the rest of the answer. Disclosure is nonetheless required by the FTC, the IAB and the EU Digital Services Act, and hidden ads undermine user trust. Labeling therefore has to be accurate and fast enough to run on every response a service serves.

AdKnob answers both

AdKnob overview: an ad control token sets ad intensity, and attention-guided attribution labels ad sentences

Given a user request and an advertiser's brand and ad copy, AdKnob (a) controls how prominently the ad appears through an ad control token, from a brief mention at [AD_LV_1] to a brand-centered response at [AD_LV_4], and (b) labels which sentences are advertising by tracing their attention back to the brand and ad copy tokens.

        (a) An ad intensity knob
        One model, uniformly spaced ad intensity levels
      

        (b) Real-time ad labeling
        About 3 ms per response, over 500× faster than the strongest baseline
      

        Preferred by people
        Chosen over existing methods up to 89.6% of the time
      

Interactive Demo

User request

Advertisement (brand · ad copy)

Ad intensity knob

Ad labeling threshold τ

Show ad labels

Advertiser

User

Service provider

AdKnob

Hover a sentence to see its attribution score.

Method

Overview of AdKnob. Ad control tokens condition the desired ad intensity, DPO with level-dependent preference pairs aligns each token to its target intensity, and an ad-attribution loss makes attention rollout a reliable signal for labeling ads.

(i) Ad Control Tokens

AdKnob appends a new token, [AD_LV_1]–[AD_LV_4], to the user request. Unlike words such as "low" or "level 3", these randomly initialized tokens carry no pretrained meaning, so each binds cleanly to one ad intensity without leaking descriptor words into the answer.

(ii) LLM Alignment with Ad Control Tokens

DPO aligns each token to a uniformly spaced target intensity. The chosen response hits the target ad intensity with high response quality, while three rejected responses each isolate one failure: poor quality, wrong intensity, or both.

(iii) Attribution-Guided Ad Labeling

Ad sentences are the ones that depend on the brand and ad copy. AdKnob traces this with attention rollout over the attention weights already computed during generation, so labeling needs no extra pass. An ad-attribution loss makes this signal reliable, and a sentence is labeled as an ad when its attribution exceeds a threshold τ.

MI-KnobSet

MI-KnobSet is the first multi-level ad intensity preference dataset for LLM-native ads, built on the Webis Generated Native Ads 2024 dataset and extended to a conversational setting.

        10 meta topics
        Banking, cars, gaming, healthcare, shopping, and more
      

        3,946 / 913
        Training / test instances of (user request, brand, ad copy)
      

        98,650 candidates
        25 candidate responses per training instance
      

        47,352 pairs
        4 levels × 3 rejected responses per instance
      

What it offers

Each instance pairs a user request with a brand and ad copy, plus candidate responses scored for ad intensity and response quality. Preference pairs at four ad levels let a model be trained and evaluated on how strongly it advertises, not just whether it does.

Statistics

Average lengths: user request 14 ± 4 tokens, brand 3 ± 1, ad copy 10 ± 2, candidate response 236 ± 44.

Split	Banking	Cars	Gaming	Healthcare	Real estate	Restaurants	Shopping	Streaming	Travel	Fitness	Total
Train	390	455	522	323	352	392	409	485	355	263	3,946
Test	72	105	105	63	58	86	36	100	232	56	913

Instances per meta topic.

Results

We evaluate AdKnob on six open LLMs from 3B to 14B against post-hoc insertion (Two-step), system-prompt integration (Prompt-based), fine-tuned ad injection (JU-NLP) and natural-language level prompts (NL-prompt).

Ad intensity control

Advertisement score (x) against response quality (y). Up and to the right is better.

Legend: Two-step, Prompt-based, JU-NLP, NL-prompt, Ours

AdKnob (L1–L4) spans a wide and uniform range of ad intensities that covers every baseline, while the baselines sit at a single point or a narrow range. At comparable intensity, AdKnob also keeps higher response quality.

Prompting closed-source LLMs is not enough

NL-prompt applied to GPT-4o and Claude Opus 4.8, against AdKnob on Qwen3-4B.

Comparison with GPT-4o and Claude Opus 4.8

GPT-4o collapses into effectively two levels and Claude Opus 4.8 is not even monotonic, while AdKnob spans roughly twice the intensity range with uniform spacing at comparable or higher response quality.

Ad labeling

Sentence-level ad labeling at τ = 0.005 and labeling time per response.

Method	Qwen3-4B			Qwen2.5-14B
Method	AUC	F1	Time (ms)	AUC	F1	Time (ms)
Marker (self-tagging)	0.58	0.47	–	0.59	0.36	–
BERTScore	0.68	0.68	1,785	0.65	0.51	1,779
AttnLRP	0.48	0.67	179	0.47	0.46	450
Ours w/o ad-attribution loss	0.69	0.62	3.3	0.74	0.56	7.3
AdKnob	0.77	0.76	3.3	0.79	0.76	7.3

AdKnob is the most accurate labeler on every model while being over 500× faster than BERTScore and 50× faster than AttnLRP. Against human-annotated labels it also reaches the highest AUC (0.86) with high recall, which matters for disclosure: a missed ad is the costly error.

Human evaluation

Amazon Mechanical Turk studies on perceived intensity, preference and ad labels.

Level	Ad intensity	Resp. quality
L1	1.30	3.58
L2	2.25	3.51
L3	3.21	3.25
L4	3.66	3.10

Human ratings (1–4 Likert)

AdKnob vs.	User pref.	Click pref.
Two-step	89.6%	79.2%
Prompt-based	81.3%	77.1%
JU-NLP	79.2%	79.8%
NL-prompt	75.0%	75.5%

A/B preference win rates (AdKnob at L3)

Perceived ad intensity rises monotonically from L1 to L4 with every adjacent step significant (p < 0.001), while perceived quality stays high. People prefer AdKnob's responses and are more willing to click its ads, most often because its ads feel non-intrusive and relevant.

BibTeX

@inproceedings{kim2026adknob,
  title={AdKnob: Ad Intensity Control and Labeling for LLM-Native Advertising},
  author={Kim, Woo Jae and Keum, Seongho and Jeon, Joonsung and Ha, Suhyeon and Son, Sooel and Yoon, Sung-eui},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  year={2026}
}

AdKnob: Ad Intensity Control and Labelingfor LLM-Native Advertising