To support the incoming Language Model Computer architecture, we extend OpenAI’s messages format to include additional information, and a new role called computer:

# The user sends a message.
{"role": "user", "type": "message", "content": "What's 2380*3875?"}

# The assistant runs some code.
{"role": "assistant", "type": "code", "format": "python", "content": "2380*3875"}

# The computer responds with the result of the code.
{"role": "computer", "type": "console", "format": "output", "content": "9222500"}

# The assistant sends a message.
{"role": "assistant", "type": "message", "content": "The result of multiplying 2380 by 3875 is 9222500."}

Anatomy

Each message in the LMC architecture has the following parameters (format is only present for some types):

{
  "role": "<role>",       # Who is sending the message.
  "type": "<type>",       # What kind of message is being sent.
  "format": "<format>"    # Some types need to be further specified, so they optionally use this parameter.
  "content": "<content>", # What the message says.
}
ParameterDescription
roleThe sender of the message.
typeThe kind of message being sent.
contentThe actual content of the message.
formatThe format of the content (optional).

Roles

RoleDescription
userThe individual interacting with the system.
assistantThe language model.
computerThe system that executes the language model’s commands.

Possible Message Types / Formats

Any role can produce any of the following formats, but we’ve included a Common Roles column to give you a sense of the message type’s usage.

TypeFormatContent DescriptionCommon Roles
messageNoneA text-only message.user, assistant
consoleactive_lineThe active line of code (from the most recent code block) that’s executing.computer
consoleoutputText output resulting from print() statements in Python, console.log() statements in Javascript, etc. This includes errors.computer
imagebase64A base64 image in PNG format (default)user, computer
imagebase64.pngA base64 image in PNG formatuser, computer
imagebase64.jpegA base64 image in JPEG formatuser, computer
imagepathA path to an image.user, computer
codehtmlHTML code that should be executed.assistant, computer
codejavascriptJavaScript code that should be executed.assistant, computer
codepythonPython code that should be executed.assistant
coderR code that should be executed.assistant
codeapplescriptAppleScript code that should be executed.assistant
codeshellShell code that should be executed.assistant
audiowavaudio in wav format for websocket.user