Output Evaluation
Offline evaluation framework to accelerate development
Last updated
Offline evaluation framework to accelerate development
Last updated
Evaluations allow users to check if the outputs of a LLM step prompt meets a specified criteria. It also allows users to rapidly validate new prompts.
An evaluation determines if your sample (inputs) and prompt generate an output that meets the given criteria and returns true or false.
Sample: a set of inputs that you will use to evaluate against a set of criteria.
Criteria: a rule or specification written in natural language to evaluate the output of your prompt.
E.g. The output must be a JSON object that begins with { and ends with }.
E.g. The output must not include a conclusion.
Evaluation: returns true or false based on the criteria you specified, along with the rationale.
LLM Step History: a history of executions, including the inputs, prompt, and output. Rerun the same inputs on new prompts by selecting the executions with the inputs you want to rerun.
Lets use an example workflow to guide you on how to perform an evaluation. We have a workflow that has only a LLM step and a topic input:
On the LLM step, we write a prompt to tell the LLM to write a simple 50-word story about the topic:
We test it with the topic "unicorns" and we get a nice 50-word story about unicorns:
So, everything seems to be working as expected. We can now create an Evaluation! So we hit the "Create Evaluation" button. This will give us some suggested Criteria to evaluate against.
We now have our first Sample and Criteria for evaluation:
We can run an evaluation for the created sample by hitting the "evaluate" button:
So evaluation succeeded. The inputs of the sample were used to execute the workflow to generate a new output, and that output was evaluated against the defined evaluation criteria. Lets try now to modify the original prompt, we will just ask it to say some nonsense:
So we head back to the Evaluation tab and run the evaluation again. This time the evaluation criteria that controlled that the story is about the required topic failed. So, in this way, we can check that the changes we make to the prompt don't break previous behavior.
Under the history tab, we can see all the executions of the LLM step. We can use any of those executions as samples for evaluation:
You can use the evaluation criteria that you previously defined to evaluate a prompt you are currently working on. Just use the "Test + Evaluate" option and it will evaluate the output of the LLM right there.
