Large Language Models

Large Language Models (LLMs) are a category of machine learning models that generate human-like text. They are trained on extensive amounts of data and are capable of understanding and generating responses based on given inputs as well as powering conversational experiences.

You have likely interfaced them in ChatGPT, the OpenAI playground or tools like NotionAI. They are an extremely versatile tool that can perform tasks ranging from text generation, code generation, natural-language processing to name just a few. We started AirOps to make it easier to build powerful solutions with these models as they play a growing role in business operations.

Supported Models

*Test and experiment to find the best model for your use case. It will always be a tradeoff between speed, capability, cost, context window.

OpenAI Chat Model Specifications

An OpenAI chat model (GPT 3.5, GPT 4) request has 3 components.

• System Prompt: define the AI's persona, the objective and behavior, and specific tasks or rules (Note: this definition varies from user-to-user and prompt-to-prompt. Too many System instructions, and the output may degrade - so feel free to experiment by adding the instructions in the User prompt! )

  System: 
  You are a renowned poet. 
  You write poems in the tone and style of Dr. Seuss.
  Only include topics appropriate for children.
  

• User Prompt: provide the model with inputs or conversation to get the desired output

  User: 
  Please write an exciting poem targeted toward kindergarten students with the following:
  Poem Title: The First Day of School
  Topics: Recess, Art Class, Snack Time
  

• Assistant Prompt: use the assistant prompt to help the model "learn" the desired output format. This is especially helpful to generate arrays, JSON, or HTML without extra "chat" text ("Sure! This is an array..."),

  System: You output an array of 5 strings with nouns that are kitchen objects.
  
  User: Give me an output of 5 utensils
  
  Assistant: ["spoon", "fork", "knife", "spatula", "chopsticks"]
  
  User: Now, give me an array of 5 {{my_input}}
  

In AirOps you can include multiple User / Assistant pairs in order to give the LLM examples of how to respond.

Claude Specifications

We are big fans of Claude at AirOps. Key Differences to consider with Claude:

• Claude does not have a "System" prompt - you will need prompt with the "Human" and "Assistant" only!
• Claude is generally faster than OpenAI models
• Claude has the largest context window available (100K tokens)
• Claude is not as good as GPT-4 at generating content in most cases but please test
• Claude is chatty - it tends to include a "Chat" response such as "Sure! Here is a ..."

Because Claude can be a little...verbose, prompting may take additional testing and specification. The prompt we used for GPT required slight modification so that it would return an array only:

Human: 
You output an array of 5 strings with nouns that are kitchen objects. 
Give me an output of 5 utensils. Return the array only.

Assistant: ["spoon", "fork", "knife", "spatula", "chopsticks"]

Human: Now, give me an output of 5 {{my_input}}. Return the array only.

Reference Inputs or Outputs in the LLM Step

To reference an input (defined in the "Start" step) or output variable (generated by a step), you will need to use Liquid syntax:

For a detailed guide to referencing inputs, check out How to Reference Variables.

Streaming

In any LLM step you will see a checkbox called "Stream?". Checking this box will create the typedown effect we have become used to with products like ChatGPT.

• What is streaming?: Streaming allows you to see output generated word-by-word (or token-by-token), just like ChatGPT.
• How do I enable streaming?: To stream the text, click the checkbox "Stream?" in the LLM step
• Where can I see streaming?:": You can see the text stream in the "Test" panel of the App Studio or the "Run App" tab of your published app
• Why would I want to stream?:
  • Rapidly test prompts: If you want to test prompts quickly, you can watch your output generate in real-time and validate your prompt faster instead of waiting for the entire response to finish.
  • Deliver an API with streaming: If streaming text in real-time is a better user-experience for your end-user, you should consider streaming as an option. On the other hand, if you're generating large amounts of text, perhaps streaming is not the ideal option for your end-user.
• The AirOps Frontend SDK makes it possible for you to stream Workflow outputs directly into your user experience.