What I learned creating a Node.js library for AI computer use

I am incredibly excited about computer use, I cannot hide it. I also think every PM in tech should understand it well enough to explore the possibilities it opens up. So I built a Node.js library called Automify to learn more about what computer use looks like when it is not a standalone tool, but a capability inside an application.

I was not only interested in making a model click a button or run a command. I wanted to understand the product shape around it: how an app gives the model a task, passes in data, limits what it can touch, observes what happened, and gets back a result the rest of the system can use.

Automify is my way of learning what building blocks are needed there. It supports AI computer use and command use across browsers, terminals, desktops, Docker containers, and QEMU virtual machines, but the larger idea is simpler: make this capability callable from whatever Node.js app, script, test runner, internal tool, or backend workflow needs it.

import { initAutomify, jsonOutput } from "automify";

const automify = initAutomify({
  provider: {
    type: "openai",
    apiKey: process.env.OPENAI_API_KEY,
    model: "gpt-5.5"
  }
});

const browser = await automify.browser({
  startUrl: "https://aldovincenti.github.io/automify/demo.html"
});

const run = await browser.do("Add the person from data, then read the saved record.", {
  data: {
    firstName: "Ada",
    lastName: "Lovelace"
  },
  output: jsonOutput("person_record", {
    id: "string",
    firstName: "string",
    lastName: "string"
  })
});

console.log(run.parsed.id);

The learning process

This is where building the library became useful as a learning exercise. Every feature forced a product decision.

Structured output made me think about downstream consumers: a workflow is much more useful when it returns parsed JSON instead of a vague paragraph. Domain allowlists made me think about scope: a browser agent should not be free to wander anywhere. Command policies made me think about permissions: "run the tests" is different from "do anything in this shell". Screenshots, logs, hooks, and recordings made me think about auditability: if something goes wrong, people need a trail they can inspect.

The more I worked on it, the less I saw computer use as a magical agent capability, and the more I saw it as a product surface. A powerful one, but still a product surface. It needs boundaries, states, outputs, failure modes, and clear contracts.

Technically, Automify wraps a model loop around concrete execution environments. The application gives a task, optional data, policies, and an output shape. Automify observes the current state, asks the model for the next action, translates that action into browser, terminal, desktop, Docker, or QEMU operations, and keeps the evidence needed to inspect the run afterwards. For desktop control, it uses nut.js to turn model instructions into real mouse, keyboard, and screen actions.

Why PMs should care

Computer use becomes interesting for product teams when it can be added to software they already own. It does not have to be a new assistant, a new tab, or a separate automation product. It can be a capability inside an existing support tool, QA workflow, admin panel, onboarding flow, or operations dashboard.

That changes the shape of possible features. A data-entry tool could extract information from documents or existing systems and enter it into a custom website. A QA tool could reproduce a bug through the browser, save screenshots, and attach the exact observed state to a ticket. An operations dashboard could combine a web check with a terminal command and return a report in the format the rest of the application already expects.

The important part is not that the AI "uses a computer" in a flashy way. The important part is that an existing product can delegate a bounded task, pass in the right context, keep the result inspectable, and continue its normal workflow afterwards.

The project

Automify is open source, published as the automify npm package, and licensed under MIT. The documentation lives at aldovincenti.github.io/automify and the code is on GitHub.