Common mistakes

Building an effective AI agent usually takes some experimentation.

Many of the most common problems aren't caused by the technology itself, but by unclear instructions, overly broad use cases or the knowledge available to the agent. This guide covers common mistakes to look out for and practical ways to build better agents in Askollo.

Estimated time: 5-10 minutes

What you'll learn

By the end of this guide you'll understand:

• The most common mistakes when building agents
• Why these mistakes affect agent performance
• How to avoid them
• How to diagnose problems with an existing agent
• Simple ways to improve your agents over time

1. Trying to make one agent do everything

One of the easiest mistakes to make is giving an agent too many responsibilities.

For example, you might create a People Assistant and ask it to:

• Answer HR policy questions
• Help with expenses
• Support recruitment
• Create onboarding plans
• Answer IT questions
• Draft company communications

This can make it harder to define how the agent should behave and what knowledge it needs.

Instead: give your agent a clear purpose

Start with a specific problem. For example, an HR Policy Assistant helps employees find accurate answers to questions about company policies and procedures.

You can always expand its capabilities later or create additional agents for different tasks.

2. Giving your agent vague instructions

Your agent relies on its instructions to understand what you expect from it. An instruction like "Help employees with HR questions" provides very little guidance about how the agent should respond.

Instead: be specific

Answer employee questions using our company HR policies. Keep responses clear and concise. Use the available company knowledge when answering. If the information isn't available, say that you don't have enough information rather than guessing.

Think about the behaviour you want and communicate it clearly.

3. Connecting too much knowledge

It can be tempting to give your agent access to everything. More knowledge doesn't automatically mean a better agent.

Connecting large amounts of irrelevant information can make it harder to keep the agent focused on its intended purpose.

Instead: choose relevant knowledge

Ask yourself what information this agent actually needs to do its job.

An HR Policy Assistant might need access to your employee handbook and HR policies, but probably doesn't need access to sales documentation.

4. Using outdated or poor-quality knowledge

Your agent can only work with the information available to it. If its underlying knowledge is incomplete, outdated or contradictory, its responses may reflect those problems.

Instead: review your sources

Before blaming the agent, check:

• Is the correct information available?
• Is it up to date?
• Are there multiple documents containing conflicting information?
• Is important context missing?

Good agents depend on good underlying knowledge.

5. Not telling your agent what to do when it doesn't know

There will be situations where your agent doesn't have enough information to answer a question. Without clear guidance, it may still attempt to provide an answer.

Instead: define its boundaries

Tell your agent how to respond when information isn't available. For example:

If you cannot find enough information in the available company knowledge to answer the question, explain that you don't have enough information rather than making an assumption.

This helps users understand the agent's limitations and builds trust in its responses.

6. Only testing the happy path

It's easy to test an agent with questions you already know it can answer. But real users won't always ask perfectly structured questions.

Instead: test different scenarios

Try:

• Common questions
• Complicated questions
• Vague questions
• Unexpected questions
• Questions outside the agent's purpose
• Questions where the answer doesn't exist

For example, an HR Policy Assistant should know how to respond appropriately if someone asks it to write a sales proposal.

Testing outside the expected path helps you understand your agent's boundaries.

7. Changing too much at once

If an agent isn't performing well, it can be tempting to rewrite its instructions, change its knowledge and adjust multiple settings at the same time. The problem is that you won't know which change actually improved the response.

Instead: make targeted changes

Use a simple testing loop:

Test → Identify the issue → Make one change → Test again

Compare the response before and after each meaningful change. This makes improving your agent much easier.

8. Testing it yourself and nobody else

When you've built an agent, you already know how it's supposed to work. Real users don't. They may ask questions differently, expect different behaviour or use the agent in ways you didn't anticipate.

Instead: get other people involved

Ask colleagues to test your agent before sharing it more widely. Don't tell them exactly what to ask; see how they naturally interact with it. Their behaviour can reveal problems you wouldn't find yourself.

9. Publishing before you're confident in the basics

Your agent doesn't need to be perfect before other people use it, but it should reliably perform its core job.

Before publishing, check:

• Does the agent have a clear purpose?
• Are its instructions clear?
• Is the right knowledge connected?
• Have you tested common questions?
• Have you tested unusual questions?
• Does it behave appropriately when it doesn't know something?
• Would you be comfortable with colleagues using its responses?

If the answer is yes, you're ready to start sharing it.

10. Treating your agent as finished

Publishing an agent isn't the end of the process. The questions people ask will change. Your organisation's knowledge will change. You'll also learn more about how people actually use the agent.

Instead: keep improving

Regularly review your agent and update:

• Instructions
• Knowledge sources
• Common use cases
• Known issues

The best agents improve over time.

A simple formula for better agents

When building an agent, focus on four things:

• Clear purpose. What specific problem does this agent solve?
• Clear instructions. How should it behave?
• Relevant knowledge. What information does it need?
• Thorough testing. Does it work when real people use it?

If something isn't working, start by checking these four areas.

Frequently asked questions

Should I create one powerful agent or several focused agents?

Start with focused agents built around clear use cases. This makes their purpose, knowledge and expected behaviour easier to define and test.

How detailed should my instructions be?

Detailed enough that the agent understands its purpose, expected behaviour and boundaries. Avoid adding instructions simply for the sake of making them longer.

Why is my agent giving poor answers?

Start by checking its instructions and connected knowledge. Then test the specific question again after making a targeted change.

How do I know when my agent is ready?

Your agent should perform reliably on its main use cases and respond appropriately when it can't answer something. It doesn't need to be perfect.

Next steps

Avoiding these common mistakes will give you a stronger starting point, but good agents still require testing and refinement. Continue to Test and improve your agent.

You may also want to explore:

• Build your first agent
• Sharing and permissions
• Search and Agent Builder together