All Visual Agent Builder Tools Compared

To test a bunch of visual agent builders/agent orchestration tools, I came up with a "simple enough but in-depth enough" test: Integrate a Chatbot with a Pinecone vector DB.

The things I realised

• I'm looking for something that lets me separate orchestration from business logic.
  • I want help building the business logic, without being required to run that logic in someone else's proprietary infrastructure.
  • I also want help running the infrastructure so I don't have to.
• One click deploy to production is pretty well supported in the ecosystem

Winner?

In terms of winner it's really hard to say. Microsoft's autogen has 40k stars on GitHub, but I really didn't find much in the way of an ecosystem (I think maybe because they released a big update recently, and only now has an opportunity for more ecosystem contribution), whereas n8n has a thriving ecosystem, is open source & self hostable, but doesn't make it easy to edit/run from an IDE, so more complicated stuff is limited to editing in a browser

Make.com is super robust and well adopted, but closed ecosystem and targets less technical users (and so is less capable).

I liked Windmill. Maybe slightly too far on the flexible end of flexible vs easy to use, but it gives you most of the quality of life features as others, without limiting you much at all. Not a huge ecosystem around it either though.

I've sorted these in descending order of how which order I'd pick them. The use case I was judging against is building a vector DB Chatbot, that I expect to maintain in production.

Windmill

• ✅ Open source
• It doesn't give you as much out of the box as n8n or Langflow (in terms of how many prebuilt blocks are available and drop-in ready)
• Windmill lets you directly edit the code for a node
• Unfortunately trying to use the Pinecone scripts available in the Windmill Hub didn't work - they were >2 years out of date, so you'd have to rewrite the scripts with the up to date packages.
• There are some features that allow you to use your IDE and keep your scripts, workflows etc in sync with your servers, including live preview in your IDE.

Autogen

• ✅ Open source
• Autogen Studio is the GUI builder.
• Seems like a great tool: All config is just JSON, so you can move between python/autogen studio reasonably easily (although not with live/hot reload).
• Despite looking robust, there's not much traction in the ecosystem around it, so you don't get much for free and will have to write a lot from scratch.

n8n

• ✅ Open source
• Easy to get a basic tool up and running
• I couldn't work out how to build something more complex
  • For example allowing an agent to use the filters in the Pinecone db, alongside a query. There was no way to create a tool that presented a query and filter_username param to the agent, and then passed those down to Pinecone.
• No IDE support

Langflow

• ✅ Open source
• It's heading in a good direction but it's a little immature.
• Blocks were generally drop-in compatible, making it super quick and easy to build something. But only for a very low level of complexity.
  • For example a lot of the blocks will have the definition for their output (as_dataframe), and that causes a name collision and prevents the flow from running. So you have to go and rename a bunch of stuff if you have more than one tool for an agent as an example.
• No integration with an IDE. The flow and code is stored as JSON in a postgres db so the only editor is the browser.

Make

• ❌ Open source
• The canvas is very mature, but very constraining
• They have a lot of integration, but only at an abstracted level so more technical features are unavailable
• I couldn't get the embedding from OpenAI into a Pinecone, even though it's possible to just make the query to the OpenAI API
• No IDE support
• No code export

Stack AI

• ❌ Open source
• Need to sign up to "Pro" to test anything aside from the basic features (which means doing a sales call).
• More custom implementations are limited - for example you can only choose from a selection of vector embeddings No IDE integration

Dify

• ✅ Open source
• It has a very limited set of blocks.
• Primarily for building workflows where LLMs do 90% of the work?
• Only lets you use their "knowledge" blocks, no 3rd party providers. So you have to manually import knowledge by uploading docs, or through API access.

Retool

• ❌ Open source
• Does Apps (drag/drop UI components and connect them to logic) as well as Workflows (using a canvas to build logic and adding custom code)
• Medium number of integrations, so for a lot of things you'd need to write the integration manually

Relay

• ❌ Open source
• Very restricted capabilities - you can only query LLMs or use agents - no option for getting an embedding or using Pinecone (aside from making http requests directly)

I want to use a tool like n8n or Make for this kind of thing. I just struggle because I think you'd need to duct-tape together a variety of other tools to actually achieve what you need. Maybe if I had more knowledge of all the other no code tools in the ecosystem I'd find it easier to get running with them.