It was the dawn of the AI age when, all of a sudden, new capabilities emerged from computer codes, the same 0s and 1s that once were used to create drawings using Turtle - if you are old enough to have used them. Now, we can control phones by just talking to them! Click3 Open-sourced tool can be found here. Claude Computer Use and OpenAI Operator They are expensive, restrictive, and not privacy-friendly, so I decided to make use of local models while allowing people to use famous closed-source models if they prefer them. Demos https://youtu.be/zrCH_oKoZXI?embedable=true In the above video, I asked the tool to search for bus stops at a certain location and it successfully found those. You can run the tool from the command line also along with the web interface created by Gradio. https://youtu.be/Iej0rw7NS-w?embedable=true Above, I asked the tool to start a 3+2 chess game on lichess. It successfully opened the lichess app and then clicked on the 3+2 game. Architecture The architecture is divided into three main modules - Planner, Finder, Executor Planner: Creates plan of action Finder: Finds UI bounds of elements Executor: Scrolls, clicks, navigates, etc. There is a flexibility of using either local model (Molmo via mlx-vlm vs closed source model for either Planner or Finder. So far, the recommendations are like below: Use Cases You can use this to create walkthrough overlays over any app. Someone can automate stuff like filtering your matches on Tinder by auto-swiping in the apps based on some feature you tell it to look for. Conclusion For now, only Finder uses structured output. Soon, the planner will also be driven by some open-source model. Waiting on either of the open-source models to implement function/tool calling. Tool - https://github.com/BandarLabs/clickclickclick It was the dawn of the AI age when, all of a sudden, new capabilities emerged from computer codes, the same 0s and 1s that once were used to create drawings using Turtle - if you are old enough to have used them. Now, we can control phones by just talking to them! Click3 Open-sourced tool can be found here. here. Claude Computer Use and OpenAI Operator They are expensive, restrictive, and not privacy-friendly, so I decided to make use of local models while allowing people to use famous closed-source models if they prefer them. Demos https://youtu.be/zrCH_oKoZXI?embedable=true https://youtu.be/zrCH_oKoZXI?embedable=true In the above video, I asked the tool to search for bus stops at a certain location and it successfully found those. You can run the tool from the command line also along with the web interface created by Gradio. https://youtu.be/Iej0rw7NS-w?embedable=true https://youtu.be/Iej0rw7NS-w?embedable=true Above, I asked the tool to start a 3+2 chess game on lichess. It successfully opened the lichess app and then clicked on the 3+2 game. Architecture The architecture is divided into three main modules - Planner , Finder , Executor Planner Finder Executor Planner : Creates plan of action Planner Finder : Finds UI bounds of elements Finder Executor : Scrolls, clicks, navigates, etc. Executor There is a flexibility of using either local model ( Molmo via mlx-vlm vs closed source model for either Planner or Finder. So far, the recommendations are like below: Molmo mlx-vlm Use Cases You can use this to create walkthrough overlays over any app. Someone can automate stuff like filtering your matches on Tinder by auto-swiping in the apps based on some feature you tell it to look for. Conclusion For now, only Finder uses structured output. Soon, the planner will also be driven by some open-source model. Waiting on either of the open-source models to implement function/tool calling. Tool - https://github.com/BandarLabs/clickclickclick Tool - https://github.com/BandarLabs/clickclickclick Tool - https://github.com/BandarLabs/clickclickclick

I Built a Tool for Mobile and Computer Operator Using Local and Remote LLMs

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Dilbert as a GIF

Java bits: 0xFF and 0xFFL

10 Things You Didn't Know You Could Do With Google Keep

10 Things to check on Android code reviews

10 Mistakes In Mobile App Development That Make You Look Dumb

10 good rules for bad App

Dilbert as a GIF

Java bits: 0xFF and 0xFFL

10 Things You Didn't Know You Could Do With Google Keep

10 Things to check on Android code reviews

10 Mistakes In Mobile App Development That Make You Look Dumb

10 good rules for bad App

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps