How helpful is operator, OpenAI’s new AI agent?

The tool, which was billed as a “research preview,” is only available to people who pay $200 a month for the company’s highest subscription tier, ChatGPT Pro. It gives users the ability to direct an AI agent that can use a web browser, fill out forms and take other actions on a user’s behalf.

AI agents are all the rage in Silicon Valley right now. Some industry insiders think they’re the next big step in AI capabilities, because an AI agent that can use a computer can actually accomplish valuable real-world tasks rather than just provide assistance. Many of the leading AI companies, including Google and Anthropic, are testing autonomous agents that they claim companies will eventually be able to “hire” as full-fledged workers.

I upgraded my ChatGPT subscription to put Operator through its paces and see what an AI agent could do for me.

On the surface, Operator looks a bit like regular ChatGPT, except that when you give it a job — “Buy me a 30-pound bag of dog food on Amazon,” for example — Operator opens a miniature browser window, types “Amazon.com” into the address bar and starts clicking around, trying to follow your instructions.

It might ask a few clarifying questions. (Do you want chicken-flavored or beef-flavored food? Overnight shipping or two-day?) Then, once it’s feeling confident it has made the right choice, Operator prompts you for a final confirmation, puts the dog food in your cart and places the order. (Operator won’t enter passwords or credit card numbers — you have to take over the minibrowser and type those things in yourself — but it does the rest on its own.)

The whole point of Operator is that you don’t have to supervise it; it can carry out tasks in the background while you’re doing other things. But I found myself glued to the window, mesmerized by the sight of a self-driving web browser clicking on buttons, typing words into boxes and selecting from drop-down menus all on its own. Look, Ma, a computer using a computer!

Operator also did impressively well on a few relatively simple tasks I gave it:

— It successfully ordered lunch on DoorDash for my colleague Mike and sent it to his house. (I didn’t tell it what to order him, but Operator chose a Mexican restaurant, picked out a handful of dishes for him and even tipped the delivery person $7.)

— It responded to hundreds of unread LinkedIn messages for me after I gave it control of my LinkedIn profile (although, to my horror, it also registered me for a webinar).

— It made $1.20 for me by setting up accounts on websites that offer small cash rewards for filling out surveys. (It might have made more, but I started to feel guilty for spamming the surveys with fake, robot-written answers.)

But Operator also failed at a bunch of other tasks and revealed its limitations:

— It couldn’t scan my recent columns and add them to my personal website, because Operator’s browser was blocked from entering The New York Times’ website. (It’s also blocked from a number of other sites, including Reddit and YouTube. The Times is suing OpenAI and Microsoft for copyright infringement related to the training of AI models.)

— It wouldn’t play online poker for me. (Operator responded, “I’m unable to assist with gambling or related activities,” which seemed like a reasonable rejection, given the chaos a gambling bot could create.)

— And it was prevented from logging into a number of sites by CAPTCHA tests (which I found reassuring, given that the whole point of CAPTCHAs is to deter robots).

In all, I found that using Operator was usually more trouble than it was worth. Most of what it did for me I could have done faster myself, with fewer headaches. Even when it worked, it asked for so many confirmations and reassurances before acting that I felt less like I had a virtual assistant and more like I was supervising the world’s most insecure intern.

This is, of course, early days for AI agents. AI products tend to improve from version to version, and it’s a good bet that the next iterations of Operator will be better. But in its current form, Operator is more an intriguing demo than a product I’d recommend using — and definitely not something most people need to spend $200 a month on.

That said, I think it’s a mistake to write off AI agents. When they become more capable, they could start to substitute for human workers in some occupations. (OpenAI and Meta have already said they’re building AI engineer agents.) And some experts worry that more powerful, unrestrained AI agents could pose safety risks if they learn to carry out commands like “drain a bank account” or “execute a cyberattack.”

Setting a bunch of AI agents loose on the internet could also provoke a backlash from web publishers, e-commerce sites and other businesses that rely on human-generated traffic to pay their bills. (If you’re a business buying ads on Amazon, you want those ads to be seen by humans, not bots pretending to be humans.) In the future, I can imagine more websites taking steps to block AI agents or steer them toward certain pages or products.

Right now, AI agents are too incompetent to be much of a threat. But it doesn’t take much imagination to envision a near future where most of the web will consist of robots talking to robots, buying things from robots and writing emails that only other robots will read.

The self-driving internet is almost here, in other words; get your clicks in while you can.

This article originally appeared in The New York Times.