Evals pt. 1: Choosing a Project

This is the first piece in a series of me documenting my journey with learning AI evals. I use LLMs daily and want to learn how to build agents properly, but also how to build and maintain AI systems. Its a calculated bet on my part, up-skilling into an area with AI that I think will become very in-demand in the future.

What are AI evals?

The name is pretty self-explanatory; we are evaluating AI systems. From my research, there are a multitude of methods we can employ to carry out these evaluations.

Choosing a project

I figured if I’m going to be self-educating, it makes sense to base the project on a subject I’m interested in. Football (soccer) is my thing, so I decided to build an agent that sits atop a dataset. I want to be able to chat with this agent and have it answer any questions I may have. Eventually, I want it to be able to do things like produce scouting reports.

Why this project? Apart from being interested in the topic, there are many opportunities for me to learn how to build evals across many different types and difficulties, so I’ll hopefully get to experience building evals of varying difficulty.

Next steps

I’ll need to source the data I want. I don’t fancy building scraping infrastructure, so I’ll to get the majority of the data from api-football.com.

In the next article, I will go through the steps I took to get the data cleaned and ready for the agent.