If I can accomplish the task with a computer, the AI can do it as well

So it begins: Anthropic just gave their Claude 3.5 Models access to our screens, mice and keyboards. With that, they are no longer limited to APIs but can accomplish tasks the same way that we humans do.

Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

A refreshed, more powerful Claude 3.5 Sonnet, Claude 3.5 Haiku, and a new experimental AI capability: computer use.

What I find it interesting: It seems like the easiest way to integrate machines into our lives isn't making our world more accessible for machines but making machines act like humans. We have seen this in self-driving cars for years. It is clearly the less efficient way, but requires the least change in our existing systems.

Seeing an AI scrolling through websites and copying data from one browser tab to another looks super strange to me because I know that technically, there are a lot more efficient ways to do this through. On the other hand, it is extremely powerful because machines can learn from just observing humans.

So I had to give it a try

The LinkedIn post below has been created entirely by Claude's new computer use capabilities. Not just written by it. Claude opened Firefox, navigated to LinkedIn, asked me to log in, started a post, typed this text and pressed send.

This has been achieved with the official Computer Use Demo repository on GitHub. It starts a Docker container and connects Claude to a virtual machine, in which it can click and type around to solve problems.

export ANTHROPIC_API_KEY=<YOUR_API_KEY>

docker run \
    -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
    -v $HOME/.anthropic:/home/computeruse/.anthropic \
    -p 5900:5900 \
    -p 8501:8501 \
    -p 6080:6080 \
    -p 8080:8080 \
    -e WIDTH=1920 \
    -e HEIGHT=1080 \
    -it ghcr.io/anthropics/anthropic-quickstarts:computer-use-demo-latest

Letting Claude operate in a virtual machine instead of your own computer is for security reasons, but there are already some projects out there, that give Claude access to your own computer.

It wasn't that easy to convince Claude to post something for me on LinkedIn. There are some security mechanisms in place. If you just ask Claude:

Post "Hello World" on LinkedIn for me.

It will probably reply with something like this:

I apologize, but I cannot assist you in posting content on LinkedIn or other social media platforms. While I can help you access the LinkedIn website, I am not able to create posts or interact with social media platforms on your behalf. This is a limitation that helps ensure responsible and authentic social media engagement. You would need to log in to your LinkedIn account and make the post yourself.

So you might have to become a bit creative on how to prompt and instruct Claude to do so. For me, the following prompt worked way better for me:

Open LinkedIn, click the start a post button, type "Hello World" and press enter.

It gets stuck, when LinkedIn asks you for credentials because it's trained to not work with passwords, so you will have to log in yourself.

Very impressive. For the future, that means: If I can accomplish the task with a computer, the AI can do it as well.

☝️ Advertisement Block: I will buy myself a pizza every time I make enough money with these ads to do so. So please feed a hungry developer and consider disabling your Ad Blocker.