The following functions are designed for language models to use in Open Interpreter, currently only supported in OS Mode.

Display - View

Takes a screenshot of the primary display.

interpreter.computer.display.view()

Display - Center

Gets the x, y value of the center of the screen.

x, y = interpreter.computer.display.center()

Keyboard - Hotkey

Performs a hotkey on the computer

interpreter.computer.keboard.hotkey(" ", "command")

Keyboard - Write

Writes the text into the currently focused window.

interpreter.computer.keyboard.write("hello")

Mouse - Click

Clicks on the specified coordinates, or an icon, or text. If text is specified, OCR will be run on the screenshot to find the text coordinates and click on it.

# Click on coordinates
interpreter.computer.mouse.click(x=100, y=100)

# Click on text on the screen
interpreter.computer.mouse.click("Onscreen Text")

# Click on a gear icon
interpreter.computer.mouse.click(icon="gear icon")

Mouse - Move

Moves to the specified coordinates, or an icon, or text. If text is specified, OCR will be run on the screenshot to find the text coordinates and move to it.

# Click on coordinates
interpreter.computer.mouse.move(x=100, y=100)

# Click on text on the screen
interpreter.computer.mouse.move("Onscreen Text")

# Click on a gear icon
interpreter.computer.mouse.move(icon="gear icon")

Mouse - Scroll

Scrolls the mouse a specified number of pixels.

# Scroll Down
interpreter.computer.mouse.scroll(-10)

# Scroll Up
interpreter.computer.mouse.scroll(10)

Clipboard - View

Returns the contents of the clipboard.

interpreter.computer.clipboard.view()

OS - Get Selected Text

Get the selected text on the screen.

interpreter.computer.os.get_selected_text()