Openai allows for strict schemas in chat completions. I need this because a model is returning more than just text. But, I still need the streaming output and it seems that I cannot have both. Is this a correct assumption or is there any way to specify an output schema for the model instead of just text? I couldn't find a way to modify the request to openAi to pass any schema to the chat completion api.
1 Like
Hello @neilb,
OpenAI's Chat Completion API supports structured outputs, but it doesn't enforce strict schemas, nor does it natively validate the output against a schema during streaming.
Here are the key considerations:
- Streaming and Structured Outputs:
- When using the Chat Completion API in streaming mode, you receive tokens incrementally. This makes it challenging to validate against a strict schema until the full response is received.
- The API doesn't provide native schema validation during generation. You must implement post-processing or validation in your application after receiving the full response.
- Non-Streaming Mode:
- In non-streaming mode, you could request the model to adhere to a schema by explicitly prompting it to output data in a specific structure (e.g., JSON, XML).
- You'd then validate the response in your application after receiving it in full.
- Workaround for Schema in Streaming:
- You can design your prompt to guide the model toward generating a predictable output structure. For example, you could ask the model to output strict JSON and monitor the stream for well-formed data.
- However, you must handle validation and error recovery manually. If the model generates invalid JSON during streaming, your application needs to catch and correct it.
- Using the API with Retool:
- Retool does not have native support for enforcing schemas directly within OpenAI API integrations.
- You can preprocess responses in Retool by using JavaScript or a backend service to enforce and validate schema compliance after receiving the complete output from the API.