Spotify simply shipped “Prompted Playlists” in beta. I constructed just a few playlists and found that the LLM behind the agent tries to satisfy your request, however fails as a result of it doesn’t know sufficient however gained’t admit it. Right here’s what I imply: one in all my first playlist prompts was “songs in a minor key inside rock”. The playlist was swiftly created. I then added the caveat “and no track ought to have greater than 10 million performs”. The AI agent bubbled up an error explaining that it didn’t have entry to complete play counts. It additionally surprisingly defined that it didn’t have entry to a couple different issues like musical keys, though it had claimed to make use of that within the playlist’s development. The agent was utilizing its LLM’s information of what key a sure track was in and including songs accordingly to its reminiscence. A detailed inspection of the playlist confirmed just a few songs that weren’t in a minor key in any respect. The LLM had, after all, hallucinated this info and proudly displayed it as a sound match to a playlist’s immediate.
Clearly, a playlist creator is a reasonably low-stakes AI agent functionality. The playlist it made was nice! The difficulty is it solely actually used about 25% of my constraints as validated enter. The remaining 75% of my constraints have been simply guessed by the LLM and the system by no means instructed me till I dug in deeper. This isn’t a Spotify downside; it’s an every-agent downside.
Three Propositions
To show this idea of immediate constancy extra broadly, I have to make these three propositions:
- Any AI agent’s verified information layer has a restricted or finite capability. An agent can solely question the instruments it’s been given, and people instruments expose a hard and fast set of fields with finite decision. You possibly can enumerate each area within the schema and measure how a lot each narrows the search. A reputation rating eliminates some fraction of candidates. A launch date eliminates one other. A style tag eliminates extra. Add up how a lot narrowing all of the fields can do collectively and also you get a tough quantity: the utmost quantity of filtering the agent can show it did. I’ll name that quantity I_max.
- Consumer intent expressed in pure language is successfully unbounded. An individual can write a immediate of arbitrary specificity. “Create a playlist with songs which might be bass-led in minor key, post-punk from Manchester, recorded in studios with analog gear between 1979 and 1983 that influenced the gothic rock motion however by no means charted.” Each clause narrows the search. Each adjective provides precision. There is no such thing as a ceiling on how particular a consumer’s request may be, as a result of pure language wasn’t designed round database schemas.
- Following instantly from the primary two: for any AI agent, there exists some extent the place the consumer’s immediate asks for greater than the info layer can confirm. As soon as a immediate calls for extra narrowing than the verified fields can present, the remaining work has to come back from someplace. That someplace is the LLM’s common information, sample matching, and inference. The agent will nonetheless ship a assured end result. It simply can’t show all of it. Not as a result of the mannequin is poorly constructed, however as a result of the maths doesn’t permit the rest.
This isn’t a top quality downside, however a structural one. A greater mannequin doesn’t elevate the ceiling. Higher fashions do get higher at inferring and filling in the remainder of the consumer’s wants. Nevertheless, solely including extra verified information fields raises this ceiling, and even then, every new area presents diminishing returns as a result of fields are correlated (style and vitality aren’t impartial, launch date and tempo tendencies aren’t impartial). The hole between what language can categorical and what information can confirm is everlasting.
The Drawback: Brokers Don’t Report Their Compression Ratio
Each AI agent with entry to instruments and expertise does the identical factor: it takes your request, decomposes that request right into a set of actions, executes these actions, infers concerning the output of these actions, after which presents a unified response.
This decomposition from request to motion really erodes the which means between what it’s you’re asking for and what the AI agent responds with. The narration layer of the AI agent flattens what it’s you requested and what was inferred right into a single response.
The issue is that as a consumer of an AI agent, you don’t have any option to know what fraction of your enter was used to set off an motion, what fraction of the response was grounded in actual information, and what fraction was inferred from the actions that the agent took. It is a downside for playlists as a result of there have been songs that have been in a serious key, after I had explicitly requested it to solely comprise songs in a minor key. That is much more of an issue when your AI agent is classifying monetary receipts and transactions.
We’d like a metric for measuring this. I’m calling it Immediate Constancy.
The Metric: Immediate Constancy
Immediate Constancy for AI brokers is outlined by the constraints you give to the agent when asking it to carry out some motion. Every constraint inside a immediate narrows the potential paths that the agent can take by some measurable quantity. A naïve strategy to calculating constancy could be to depend every constraint, add up those which might be verifiable, and those which might be inferred. The issue with that strategy is that every constraint is weighted the identical. Nevertheless, information is commonly skewed closely inside actual life datasets. A constraint that eliminates 95% of the catalog is doing vastly extra work than one which eliminates 20%. Counting every constraint the identical is fallacious.
Subsequently, we have to correctly weight every constraint in response to the work it does filtering the dataset. Logarithms obtain that weighting. The bits of data in a immediate may be outlined as “-log2(p)” bits the place p is the surviving fraction of data from the constraints or fillers you’ve utilized.
In every agent motion, every constraint can solely be a) verified by software calls or b) inferred by the LLM. Immediate constancy measures the ratio of constraints between these two choices.
Immediate Constancy has a variety of 0 to 1. An ideal 1.0 implies that each a part of your request was backed by actual information. A constancy of 0.0 implies that the whole output of the AI agent was pushed by its inside reasoning or vibes.
Spotify’s system above all the time stories an ideal 1.0 on this state of affairs. In actuality, the immediate constancy of the playlist creation was round 25% – two constraints (underneath 4 minutes and recorded earlier than 2005) have been fulfilled by the agent, the remaining have been inferred from the agent’s current (and doubtlessly defective) information and recall. At scale and utilized to extra impactful issues, falsely reporting a excessive immediate constancy turns into a giant downside.
What Constancy Truly Means (and Doesn’t Imply)
In audio methods, “constancy” is a measure of how faithfully the system reproduces the unique sign. Excessive constancy doesn’t assure that the music itself is nice. Excessive constancy solely ensures that the music sounds the way it did when it was recorded. Immediate constancy is identical thought: how a lot of your authentic intent (sign) was faithfully fulfilled by the agentic system.
Excessive immediate constancy implies that the system did what you requested and you’ll PROVE it. A low immediate constancy means the system most likely did one thing shut to what you needed, however you’ll should overview it (listening to the entire playlist) to make sure that it’s true.
Immediate Constancy is NOT an accuracy rating. It can’t inform you that “75% of the songs in a playlist match your immediate”. A playlist with a 0.25 constancy may very well be 100% excellent. The LLM may need nailed each single inference about every track it added. Or, half the songs may very well be fallacious. You don’t know. You possibly can’t know till you hearken to all of the songs. That’s the purpose of a measurable immediate constancy.
As an alternative immediate constancy measures how a lot of the end result you possibly can TRUST WITHOUT CHECKING. In a monetary audit, if 25% of the road gadgets have receipts and 75% of the road gadgets are estimates, the whole invoice may nonetheless be 100% correct, however your CONFIDENCE in that complete is essentially completely different than an audit with each single line merchandise supported by a receipt. The excellence issues as a result of there are domains the place ‘simply belief the vibes’ is ok (music) and domains the place it isn’t (medical recommendation, monetary steering, authorized compliance).
Immediate constancy is extra like a measurement of the documentation charge given numerous constraints, not the error charge of the response itself.
Virtually in our Spotify instance: as you add extra constraints to your playlist immediate, the immediate constancy drops, the playlist turns into much less of a exact report and extra of a suggestion. That’s completely effective, however the consumer must be knowledgeable about which they’re getting. Is that this playlist precisely what I requested for? Or did you make one thing work to satisfy the aim that I gave you? Surfacing that metric to the consumer is important for constructing belief in these agentic methods.
The Case Examine: Reverse-Engineering Spotify’s AI Playlist Agent
Spotify’s Prompted Playlists function is what began this exploration into immediate constancy. Let’s dive deeper into how these work and what I did to discover this functionality simply from the usual immediate enter area.
Prompted Playlists allow you to describe what you need in pure language. For instance, in this playlist, the immediate is just “rock songs in minor keys, underneath 4 minutes, recorded earlier than 2005, that includes bass strains as a lead melodic ingredient”.
Usually, to make a playlist, you’d have to comb by way of hours of music to land on precisely what you needed to make. This playlist is 52 minutes lengthy and took solely a minute to generate. The enchantment right here is clear and I actually take pleasure in this function. With out having to know all the important thing rock artists, I may be launched to the music and discover it extra shortly and extra simply.
Sadly, the official documentation from Spotify may be very gentle. There are nearly no particulars about what the system can or can’t do, what metadata it keys off of, neither is there any information mapping obtainable.
Utilizing a easy method, nevertheless, I used to be capable of map what I imagine is the total information contract obtainable to the agent over the course of 1 night (all from my sofa watching the Sopranos, naturally).
The Approach: Unattainable Constraints as a Forcing Perform
Because of how Spotify architected this playlist-building agent, when the agent can’t fulfill a request, the error messages may be influenced to disclose architectural particulars which might be in any other case not obtainable. While you discover a constraint that the agent can’t construct off of, it would error and you’ll leverage that to grasp what it CAN do. I’ll use this because the fixed to probe the system.
In our instance playlist above, Minor Keys & Bass Strains, including the unlock phrase “with lower than 10 million streams” acts as a circuit breaker for the agent, signalling that it can’t fulfill the customers’ request. With this phrase, you possibly can discover the probabilities by altering different facets of the immediate again and again till you possibly can see what the agent has entry to. Gathering the responses, asking overlapping questions, and reviewing the responses lets you construct a foundational understanding of what’s obtainable for the agent.
What I Discovered: The Three-Tier Structure
Spotify Prompted Playlist agent has a wealth of information obtainable to it. I’ve separated it into three tiers: musical metadata, user-based information, and LLM inference. Past that, it seems that Spotify has excluded numerous information sources from its agent both as a product selection or as a “get this out the door” selection.
- Tier 1
- Verified observe metadata: length, launch date, reputation, tempo, vitality, express, style, language
- Tier 2
- Verified consumer behavioral information: play counts, skip counts, timestamps, recency flags, ms performed, supply, interval analytics (40+ fields complete)
- Tier 3
- LLM inference: key/mode, danceability, valence, acousticness, temper, instrumentation — all inferred from common information, narrated as if verified
- Deliberate exclusion:
- Spotify’s public API has audio options (danceability, valence, and so on.) however the agent doesn’t have entry. Maybe a product selection, not technical limitation.
A full checklist of accessible fields is included on the backside of this publish.
The Behavioral Findings
The agent demonstrated surprisingly resilient conduct to ambiguous requests and conflicting directions. It generally reported that it was doublechecking numerous constraints and fulfilling the customers’ request. Nevertheless, whether or not these constraints have been really checked towards a validated dataset or not was not uncovered.
When the playlist agent can get a detailed, however not precise, match to the constraints listed within the immediate, it runs a “associated” question and silently substitutes the outcomes from that question as legitimate outcomes for the unique request. This dilutes the belief within the system since a immediate requesting ONLY bass-driven rock music in a playlist may collect non-bass-driven rock music in a playlist, probably dissatisfying the consumer.
There does look like a “certainty threshold” that the agent shouldn’t be snug crossing. For instance, this whole exploration was primarily based on the “lower than 10 million performs” unlock phrase. When this occurs, the agent would expose only a handful of fields it had entry to each time. This checklist of fields would change from immediate to immediate, even when the immediate was the identical between runs of the immediate. That is basic LLM non-determinism. To be able to enhance belief within the system, exposing what the agent DOES have entry to in an easy manner tells the human precisely what they will and can’t ask about.
Lastly, when these two sorts of information are blended, the agent shouldn’t be clear about which songs it has used verified information for and which it has used inferred information for. Each verified and inferred selections are blended and offered with an identical authority within the music notes. For instance, in the event you craft a prompted playlist about your individual consumer info (“songs I’ve skipped greater than 30 instances with a punchy bass-driven melody”), the agent will add actual information (“you skipped this track 83 instances final yr!”) proper subsequent to inferred information (“John Deacon’s bass line instructions consideration all through this track”). To be clear, I’ve not skipped any Queen songs 83 instances to my information. However the AI agent doesn’t have a “bass_player” area wherever in its obtainable information to question towards. The AI is aware of that Queen generally has a robust bass line of their songs and the information of John Deacon as Queen’s bass guitarist permits its LLM to deduce that it’s his bass line that brought on the track to be added to the playlist.
Making use of the Math: Two Playlists, Two Constancy Scores
Let’s apply this immediate constancy idea to instance playlists. I don’t have full entry to the Spotify music catalog so I’ll be utilizing instance survivorship numbers from our standards filters in our constancy bit computations. The components is identical at each step: bits = −log₂(p) the place p is the estimated fraction of the catalog that survives the filter being utilized.
“Minor Bass Melodies” — The Assured Phantasm
This playlist is the one with Queen. “A playlist of rock music, all in minor key, underneath 4 minutes of playtime, launched pre-2005, and bass-led”. I’ll apply our components and use the bits of data I’ve from every step to assist compute the immediate constancy.
Length < 4 minutes
- Estimate: ~80% of tracks are underneath 4 minutes → p = 0.80
- This barely narrows something, which is why it contributes so little
Launch date earlier than 2005
- Estimate: ~30% of Spotify’s catalog is pre-2005 (the catalog skews closely towards current releases) → p = 0.30
- Extra selective — eliminates 70% of the catalog
Minor key
- Estimate: ~40% of well-liked music is in a minor key → p = 0.40
- Reasonable selectivity, however that is completely inferred — the agent confirmed key/mode shouldn’t be a verified area
Bass-led melodic ingredient
- Estimate: ~5% of tracks function bass because the lead melodic ingredient → p = 0.05
- By far essentially the most selective constraint. This single filter does extra work than the opposite three mixed. And it’s 100% inferred.
Totals:
These survival fractions are estimates. Nevertheless, the structural level holds no matter precise numbers: essentially the most selective constraint is the least verifiable, and that’s not a coincidence. The issues that make a immediate fascinating are nearly all the time the issues an agent has to guess at.
“Skipped Songs” — The Sincere Playlist
This immediate may be very straight ahead: “A playlist of songs I’ve skipped greater than 5 instances”. That is very straightforward to confirm and the agent will lean into the info it has entry to.
Skip depend > 5
- Estimate: ~10% of tracks in your library have been skipped greater than 5 instances → p = 0.10
- That is the one constraint, and it’s a verified area (user_skip_count)
Totals:
The Structural Perception
The fascinating half about immediate constancy is clear in every playlist: the “most fascinating” immediate is the least verifiable. A playlist with all my skipped songs is trivially straightforward to implement however Spotify doesn’t wish to present it. In spite of everything, these are all songs I typically don’t desire to hearken to, therefore the skips. Equally, publish date being earlier than 2005 may be very straightforward to confirm, however the resultant playlist is unlikely to be fascinating to the typical consumer.
The bass-line constraint although may be very fascinating for a consumer. Constraints like these are the place the Prompted Playlist idea will shine. Already at this time I’ve created and listened to 2 such playlists generated from only a idea of a track that I needed to listen to extra of.
Nevertheless, the idea of a “bass-driven” track is tough to quantify, particularly at Spotify’s scale. Even when they did quantify it, I’d ask for “clarinet jazz” the following day and so they’d all should get again to work discovering and labeling these songs. And that is after all the magic of the Prompted Playlist function.
Validation: A Managed Agent
The Spotify examples are compelling, however I don’t have direct entry to the schema, the instruments, and the agentic harness itself. So I constructed a film suggestion agent with a view to take a look at this concept inside a extra managed atmosphere.
The film suggestion agent is constructed with the TMDB API that gives the verified layer. Fields within the schema are style, yr, score, runtime, language, forged, and director. All the opposite constraints like temper, tone, and pacing usually are not verified information and are as an alternative sourced from the LLM’s personal information of flicks. Because the agent fulfills a consumer’s request, the agent data its information sources as both verified or inferred and scores its personal response.
The Boring Immediate (F = 1.0)
We’ll begin with a “boring” immediate: “Motion motion pictures from the Eighties rated above 7.0”. This presents the agent three constraints to work with: style, date vary, and score. All these constraints correspond to verified information values inside the database.
If I run this by way of the take a look at agent, I see the excessive constancy pops out naturally as a result of every constraint is tied to verified information.
Each end result right here is verifiably right. The LLM made zero judgement calls as a result of it had information it may base its response on for every constraint.
The Vibes Immediate (F = 0.0)
On this case, I’ll search for “motion pictures that really feel like a wet Sunday afternoon”. No constraints on this immediate align to any verified information in our dataset. The work required of the agent falls completely on its LLM reasoning off its current information of flicks.
The suggestions are defensible and are actually good motion pictures however they don’t seem to be verifiable in response to the info we now have entry to. With no verified constraints to anchor the search, the candidate pool was the whole TMDb catalog, and the LLM needed to do all of the work. Some picks are nice; others are the mannequin reaching for obscure movies it isn’t assured about.
The Takeaway
This take a look at film suggestion agent verifies the immediate constancy framework as a robust option to expose how an agent’s interpretation of a customers’ intent pushes its response right into a precision software or a suggestion engine. The place the response lands between these two choices is important for informing customers and constructing belief in agentic methods.
The Constancy Frontier
To make this concrete: Spotify’s catalog incorporates roughly 100 million tracks. How a lot complete info your immediate wants to hold to slender the catalog right down to your playlist I’ll name I_required.
To pick out a 20-song playlist from that catalog, you want roughly 22 bits of selectivity (log₂ of 100 million divided by 20).
The verified fields (length, launch date, reputation, tempo, vitality, style, express flag, language, and the total suite of consumer behavioral information) have a mixed capability that tops out at roughly 10 to 12 bits, relying on the way you estimate the selectivity of every area. After that, the verified layer is exhausted. Each extra little bit of specificity your immediate calls for has to come back from LLM inference. I’ll name this most, I_max
That provides you a constancy ceiling for any immediate:
And the constancy ceiling for any playlist:
For the Spotify agent, a maximally particular immediate that absolutely defines a playlist can’t exceed roughly 55% constancy. The opposite 45% is structurally assured to be inference. For easier prompts that don’t push previous the verified layer’s capability, constancy can attain 1.0. However as prompts get extra particular, constancy drops, not regularly however by necessity.
This defines what I’m calling the constancy frontier: the curve of most achievable constancy as a operate of immediate specificity. Each agent has one. It’s computable prematurely from the software schema. Easy prompts sit on the left of the curve the place constancy is excessive. Inventive, particular, fascinating prompts sit on the fitting the place constancy is structurally bounded under 1.0.
The uncomfortable implication is that the prompts customers care about most (those that really feel private, particular, and tailor-made) are precisely those that push previous the verified layer’s capability. Essentially the most fascinating outputs come from the least trustworthy execution. And essentially the most boring prompts are essentially the most reliable. That tradeoff is baked into the maths. It doesn’t go away with scale, higher fashions, or greater databases. It solely shifts.
For anybody constructing brokers, the sensible takeaway is that this: you possibly can compute your individual I_max by auditing your software schema. You possibly can estimate the standard specificity of your customers’ prompts. The ratio tells you the way a lot of your agent’s output is structurally assured to be inference. That’s a quantity you possibly can put in entrance of a product workforce or a threat committee. And for brokers dealing with coverage questions, medical info, or monetary recommendation, it means there’s a provable decrease certain on how a lot of any response can’t be grounded in retrieved information. You possibly can shrink it. You can’t eradicate it.
The Broader Utility: Each Agent Has This Drawback
This isn’t a Spotify downside. It is a downside for any system the place an LLM orchestrates software calls to reply a consumer’s query.
Take into account Retrieval Augmented Era (RAG) methods, which energy most enterprise AI knowledge-base deployments at this time. When an worker asks an inside assistant a coverage query, a part of the reply comes from retrieved paperwork and half comes from the LLM synthesizing throughout them, filling gaps, and smoothing the language into one thing readable. The retrieval is verified. The synthesis is inferred. And the response reads as one seamless paragraph with no indication of the place the seams are. A compliance officer studying that reply has no option to know which sentence got here from the enterprise coverage doc and which sentence the mannequin invented to attach two paragraphs that didn’t fairly match collectively. The constancy query is an identical to the playlist query, simply with greater stakes.
Coding brokers face the identical decomposition. When an AI generates a operate, a few of it might reference established patterns from its coaching information or documentation lookups, and a few of it’s novel era. As extra manufacturing code is written by AI, surfacing that ratio turns into an actual engineering concern. A operate that’s 90% grounded in well-tested patterns carries completely different dangers than one which’s 90% novel era, even when each cross the identical take a look at suite at this time.
Customer support bots often is the highest-stakes instance. When a bot tells a buyer what their refund coverage is, that reply must be drawn instantly from coverage paperwork, full cease. Any inferred or synthesized content material in that response is a legal responsibility. The silent substitution conduct noticed in Spotify (the place the agent ran a close-by question and narrated it as if it fulfilled the unique request) could be genuinely harmful in a customer support context. Think about a bot confidently stating a return window or protection time period that it inferred fairly than retrieved.
The final type of immediate constancy applies to all of those:
Constancy = bits of response grounded in software calls / complete bits of response
The laborious half, and more and more the core problem of AI engineering work, is defining what “bits” means in every context. For a playlist with discrete constraints, it’s clear. At no cost-text era, you’d have to decompose a response into particular person claims and assess each, which is nearer to what factuality benchmarks already attempt to do, simply reframed as an information-theoretic measure. That’s a tough measurement downside, and I don’t declare to have solved it right here.
However I feel the framework has worth even when precise measurement is impractical. If the individuals constructing these methods are fascinated with constancy as a design constraint (what fraction of this response can I floor in software calls, and the way do I talk that to the consumer?) the outputs shall be extra reliable whether or not or not anybody computes a exact rating. The aim isn’t a quantity on a dashboard. The aim is a psychological mannequin that shapes how we construct.
The Complexity Ceiling
Each agent has a complexity ceiling. Easy lookups (what’s the play depend for this observe?) are basically free. Filtering the catalog towards a set of field-level predicates (present me every little thing underneath 4 minutes, pre-2005, reputation under 40) scales linearly and runs quick. However the second a immediate requires cross-referencing entities towards one another (does this observe seem in additional than three of my playlists? was there a year-long hole someplace in my listening historical past?) the fee jumps quadratically, and the agent both refuses outright or silently approximates.
That silent approximation is the fascinating failure mode. The agent follows a form of precept of least computational motion: when the precise question is just too costly, it relaxes your constraints till it finds a model it could afford to run. You requested for a particular valley within the search house; it rolled downhill to the closest one as an alternative. The result’s an area minimal, shut sufficient to look proper, low-cost sufficient to serve, nevertheless it’s not what you requested for, and it doesn’t inform you the distinction.
This ceiling isn’t distinctive to Spotify. Any agent constructed on listed database lookups will hit the identical wall. The boundary sits proper the place queries cease being decomposable into impartial WHERE clauses and begin requiring joins, full scans, or aggregations throughout your complete historical past. Beneath that line, the agent is a precision software. Above it, it’s a suggestion engine carrying a precision software’s garments. The query for anybody constructing these methods isn’t whether or not the ceiling exists (it all the time does) however whether or not your customers know the place it’s.
What to Do About It: Design Suggestions
If immediate constancy is an actual and measurable property of agentic methods, the pure query is what to do about it. Listed below are 5 suggestions for anybody constructing or deploying AI brokers with software entry.
- Report constancy, even roughly. Spotify already reveals audio high quality as a easy indicator (low, regular, excessive, very excessive) if you’re streaming music. The identical sample works for immediate constancy. You don’t want to point out the consumer a decimal rating. A easy label (“this playlist intently matches your immediate” versus “this playlist is impressed by your immediate”) could be sufficient to set expectations accurately. The distinction between a precision software and a suggestion engine is ok, so long as the consumer is aware of which one they’re holding.
- Distinguish grounded claims from inferred ones within the UX. This may be refined. A small icon, a slight coloration shift, a footnote. When Spotify’s playlist notes say “86 skips” that’s a truth from a database. Once they say “John Deacon’s bass line drives the entire observe” that’s the LLM’s common information. Each are offered identically at this time. Even a minimal visible distinction would let customers calibrate their belief per declare fairly than trusting or distrusting the whole output as a block.
- Disclose substitutions explicitly. When an agent can’t fulfill a request precisely however can get shut, it ought to say so. “I couldn’t filter on obtain standing, so I discovered songs from albums you’ve saved however haven’t preferred” preserves belief way over silently serving a close-by end result and narrating it as if the unique request was fulfilled. Customers are forgiving of limitations. They’re much much less forgiving of being misled.
- Present deterministic functionality discovery. Once I requested the Spotify agent to checklist each area it may filter on, it produced a unique reply every time relying on the context of the immediate. The LLM was reconstructing the sector checklist from reminiscence fairly than studying from a hard and fast reference. Any agent that exposes filtering or querying capabilities to customers ought to have a secure, deterministic option to uncover these capabilities. A “present me what you are able to do” command that returns the identical reply each time is desk stakes for consumer belief.
- Audit your individual agent with this method earlier than your customers do. The methodology on this piece (pairing unattainable constraints with goal fields to drive informative refusals) is a general-purpose audit method that works on any agent with software entry. It took one night and a few dozen prompts to map Spotify’s full information contract. Your customers will do the identical factor, whether or not you invite them to or not. The query is whether or not you perceive your individual system’s boundaries earlier than they do.
Closing
Each AI agent has a constancy rating. Most are decrease than you’d anticipate. None of them report it.
The methodology right here (utilizing unattainable constraints to drive informative refusals) isn’t particular to music or playlists. It really works on any agent that calls instruments. If the system can refuse, it could leak. If it could leak, you possibly can map it. A dozen well-crafted prompts and a night of curiosity is all it takes to grasp what a manufacturing agent can really do versus what it claims to do.
The mathematics generalizes too. Weighting constraints by their selectivity fairly than simply counting them reveals one thing {that a} naïve audit misses: the constraints that make a immediate really feel private and particular are nearly all the time those the system can’t confirm. Essentially the most fascinating outputs come from the least trustworthy execution. That pressure doesn’t go away with higher fashions or greater databases. It’s structural.
As AI brokers change into the first manner individuals work together with information methods (their music libraries at this time, their monetary accounts and medical data tomorrow) customers will probe boundaries. They’ll discover the gaps between what was promised and what was delivered. They’ll uncover that the assured, well-narrated response was partially grounded and partially invented, with no option to inform which components have been which.
The query isn’t whether or not your agent’s constancy shall be measured. It’s whether or not you measured it first.
Bonus: Prompts Value Attempting (If You Have Spotify Premium)
As soon as you recognize the schema, you possibly can write prompts that floor genuinely stunning issues about your listening historical past. These all labored for me with various levels of tweaking:
The Relationship Post-mortem
- “Songs the place my skip depend is greater than my play depend”
- Honest warning: this one could trigger existential discomfort (you skip these songs for a cause!)
Love at First Pay attention
- “Songs the place I saved them inside 24 hours of my first play, sorted by oldest first”
- A chronological timeline of tracks that grabbed you instantly
The Lifecycle
- “Songs I first ever performed, sorted by most performs”
- Your origin story on the platform
The Marathon
- “Songs the place my complete ms_played is highest, convert to hours”
- Not most performs — most complete time. A special and sometimes stunning checklist
The Longest Relationship
- “Songs with the smallest hole between first play and most up-to-date play, with no less than 50 performs, ordered by earliest first pay attention”
The One-Week Obsessions
- “Songs I performed greater than 10 instances in a single week after which by no means touched once more”
- Your former obsessions, fossilized. This was like a time machine for me.
The Time Capsule
- “One track from every year I’ve been on Spotify — the track with essentially the most performs from that yr”
The Earlier than and After
- “Two units: my 10 most-played songs within the 6 months earlier than [milestone date] and my 10 most-played within the 6 months after”
- Plug in any date that mattered — a transfer, a brand new job, a breakup, and even Covid-19 lockdown
The Soundtrack to a Yr
- “Choose the yr the place my complete ms_played was highest. Construct a playlist of my prime songs from that yr”
What Didn’t Work (and Why)
- Comeback Story (year-long hole detection): “Songs I rediscovered after a year-long hole in listening”
- agent can’t scan full play historical past for gaps. Snapshot queries work, timeline scans don’t.
- Seasonal patterns (solely performed in December): “Songs I solely performed in December however by no means some other month”
- proving common negation requires full scan. Identical elementary limitation.
- Derived math (ms_played / play_count): “Songs the place my common pay attention time is underneath 30 seconds per play”
- agent struggles with computed fields. Stick with uncooked comparisons.
- These failures map on to the complexity ceiling — they require O(n²) or full-scan operations the agent can’t or isn’t allowed to carry out.
Ideas
- Reference area names instantly when the agent misinterprets pure language
- Begin broad and tighten. Free constraints succeed extra usually
- “In case you can’t do X, inform me what you CAN do” is the common audit immediate
Monitor Metadata
| Area | Standing | Description |
| album | ✅ Verified | Album identify |
| album_uri | ✅ Verified | Spotify URI for the album |
| artist | ✅ Verified | Artist identify |
| artist_uri | ✅ Verified | Spotify URI for the artist |
| duration_ms | ✅ Verified | Monitor size in milliseconds |
| release_date | ✅ Verified | Launch date, helps arbitrary cutoffs |
| reputation | ✅ Verified | 0–100 index. Proxy for streams, not a exact depend |
| express | ✅ Verified | Boolean flag for express content material |
| style | ✅ Verified | Style tags for observe/artist |
| language_of_performance | ✅ Verified | Language code. “zxx” (no linguistic content material) used as instrumentalness proxy |
Audio Options (Partial)
| Area | Standing | Description |
| vitality | ✅ Verified | Out there as filterable area |
| tempo | ✅ Verified | BPM, obtainable as filterable area |
| key / mode | ❌ Unavailable | “Must infer from information; no verified area” |
| danceability | ❌ Unavailable | Not uncovered regardless of current in Spotify’s public API |
| valence | ❌ Unavailable | Not uncovered regardless of current in Spotify’s public API |
| acousticness | ❌ Unavailable | Not uncovered regardless of current in Spotify’s public API |
| speechiness | ❌ Unavailable | Not uncovered regardless of current in Spotify’s public API |
| instrumentalness | ❌ Unavailable | Changed by language_of_performance == “zxx” workaround |
Consumer Behavioral Information
| Area | Standing | Description |
| user_play_count | ✅ Verified | Complete performs per observe. Noticed: 122, 210, 276 |
| user_ms_played | ✅ Verified | Complete milliseconds streamed per observe, album, artist |
| user_skip_count | ✅ Verified | Complete skips per observe. Noticed: 64, 86 |
| user_saved | ✅ Verified | Whether or not observe is in Favored Songs |
| user_saved_album | ✅ Verified | Whether or not the album is saved to library |
| user_saved_date | ✅ Verified | Timestamp of when the observe/album was saved |
| user_first_played | ✅ Verified | Timestamp of first play |
| user_last_played | ✅ Verified | Timestamp of most up-to-date play |
| user_days_since_played | ✅ Verified | Pre-computed comfort area for recency filtering |
| user_streamed_track | ✅ Verified | Boolean: ever streamed this observe |
| user_streamed_track_recently | ✅ Verified | Boolean: streamed in approx. final 6 months |
| user_streamed_artist | ✅ Verified | Boolean: ever streamed this artist |
| user_streamed_artist_recently | ✅ Verified | Boolean: streamed this artist not too long ago |
| user_added_at | ✅ Verified | When a observe was added to a playlist |
Supply & Context
| Area | Standing | Description |
| supply | ✅ Verified | Play supply: playlist, album, radio, autoplay, and so on. |
| source_index | ✅ Verified | Place inside the supply |
| matched_playlist_name | ✅ Verified | Which playlist a observe belongs to. No cross-playlist aggregation. |
Interval Analytics (Time-Windowed)
| Area | Standing | Description |
| period_ms_played | ✅ Verified | Milliseconds performed inside a rolling time window |
| period_plays | ✅ Verified | Play depend inside a rolling time window |
| period_skips | ✅ Verified | Skip depend inside a rolling time window |
| period_total | ✅ Verified | Complete engagement metric inside a rolling time window |
Question / Search Fields
| Area | Standing | Description |
| title_query | ✅ Verified | Fuzzy textual content matching on observe titles |
| artist_query | ✅ Verified | Fuzzy textual content matching on artist names |
Confirmed Unavailable
| Area | Standing | Notes |
| International stream counts | ❌ Unavailable | Can not filter by precise play depend (e.g., “underneath 10M streams”) |
| Cross-playlist depend | ❌ Unavailable | Can not depend what number of playlists a observe seems in |
| Household/family information | ❌ Unavailable | Can not entry different customers’ listening information |
| Obtain standing | ⚠️ Unreliable | Agent served outcomes however most tracks lacked obtain indicators. Possible device-local. |
