After reviewing the context that PromptQL sends to the LLM, I noticed it includes a large number of
CREATE FUNCTION definitions. What’s the best way to remove or exclude these from the context?
After reviewing the context that PromptQL sends to the LLM, I noticed it includes a large number of
Thanks for this, @Hoang 
I have a feeling these are necessary in order to pass the available set of “tools” PromptQL has access to when working with the LLM (as they’re exposed as SQL functions in your schema).
I’m curious: did you encounter an explicit issue with your Prompt overflowing the context window? If so, we can definitely talk about ways of minimizing the context that’s sent to the LLM. Or, are you just curious about methods of tweaking costs/performance?
Either way, I’ll share this internally and see if we can get someone from the engine team to share context (pardon the pun 
)!
No problem. The issue we have is of course related to cost. With such a large amount of input tokens, we are forced to use a large model like gpt-4.1, which leads to a huge cost. Each request is currently costing more than $0.5, at such a price it is impossible to put it into prod.
Functions are SQL representations of Commands (and Model with arguments) in the metadata. If you don’t intend PromptQL to use these Commands for upserting/modifying data, I would recommend to remove these Commands from the metadata (use cli: ddn command remove "*"). This will drastically reduce the tokens. In general keep only the relevant Models and Commands definitions.
We are also working on a strategy that reduces the token usage by first filtering out which Models/Commands should be kept in context before sending the main prompt to LLM. This can be helpful in scenarios, where the supergraph is very large, and cannot be trimmed down due to the nature of intended PromptQL agent.
1 Like