Since a couple days ago, I have been getting the following error about 40% of the time when making queries through the PromptQL console:
Error calling DDN: Error calling DDN: status code: 503 response text: upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
I’ve noticed this on multiple projects (namely ready-aardvark-7500 and tender-crappie-9017), and I am unable to view traces in the Insights tab due to another internal error.
We would like to have this fixed so we can return to using PromptQL in our workflows; we’ve checked on our end and don’t see any issue with our connector setup, credentials, etc. We’d appreciate any guidance on resolving this issue.
The 503 likely means the underlying query engine is panicking due to the excessive memory your query is using.
From those 40% of requests, can you identify any patterns in the generated SQL that looks memory-intensive? Additionally, which connector(s) are you using?
Answering these questions will make it easier to point you in the right direction.
The generated SQL we were running involved some large unions across several tables, and we’re using the hasura/postgres connector.
Actually, it looks like the 503 errors have been resolved; we tried some memory-intensive queries this morning and found that they ran just fine. Is there any chance this could have been related to the GCP outage last week?
Is there any chance this could have been related to the GCP outage last week?
It’s certainly possible, @mic0 — however, I’ve seen this behavior before. Since you’re using PromptQL, I’d recommend switching to the hasura/postgres-promptql connector. This has SQL pushdown built into the connector, which results in more performant queries.