How to Tell Whether Your AI Project Actually Works

A notebook that prints 94% accuracy proves very little. Plenty of learners finish a model or a chatbot, see it run, and move on without being able to say whether it is any good. Evaluation is the skill that turns a demo into evidence, and it is usually the first thing a mentor, interviewer or client probes. This guide shows how to test a classic Machine Learning model and a Retrieval-Augmented Generation (RAG) application honestly, using methods you can apply with free tools.

Why "It Runs" Is Not the Same as "It Works"

A model can score well for the wrong reasons. Three problems account for most inflated results: information leaking from the test data into training, a test set that does not resemble real use, and a metric that hides the mistakes that matter. Each is easy to introduce and easy to miss, which is why careful evaluation is a skill in its own right and not an afterthought.

Start With a Baseline

Before training anything sophisticated, build the simplest possible predictor. For classification, that might mean always predicting the most common class. For regression, it might mean always predicting the average. A plain logistic or linear regression is a sensible second baseline

A baseline gives every later number a meaning. If an XGBoost model beats logistic regression by a fraction of a percentage point, the extra complexity may not be justified. If it beats the naive predictor by a wide margin, you have a result worth explaining.

Keep Test Data Genuinely Separate

Watch for data leakage

Leakage happens when the model gains access to information it would not have in real use. The classic mistake is fitting preprocessing, such as scaling, imputing missing values or selecting features, on the whole dataset before splitting it. The test rows then influence the training process. The scikit-learn documentation on common pitfalls and recommended practices describes this problem and recommends wrapping preprocessing and model together in a Pipeline, so that every step is fitted only on training data.

Leakage also hides inside the data itself. A churn dataset might contain a column recording that an account was closed, which is the answer in disguise. Duplicate or near-duplicate rows that land on both sides of the split cause the same trouble.

Split the data the way the model will be used

A random split is not always appropriate. For a stock or sales forecast, shuffling rows lets the model learn from the future, so a chronological split or walk-forward validation is more honest. If your data has several rows per customer, keep each customer entirely in training or entirely in testing. Ask one question: in real use, what will the model know, and what will it have to predict?

Choose Metrics That Match the Cost of Mistakes

Accuracy can mislead badly on imbalanced data. If 5% of customers leave, a model that predicts that nobody leaves is 95% accurate and completely useless. Google's Machine Learning Crash Course explains accuracy, precision and recall and why class imbalance changes how they should be read.

In plain terms, precision asks how many of the cases the model flagged were correct, and recall asks how many of the real cases it found. Which matters more depends on the problem. A fraud alert that misses cases and a spam filter that hides genuine mail fail in different ways, and the right balance depends on what each error costs.

For regression, report the average error alongside the baseline, and explain what it means. Mean absolute error treats all misses equally, while root mean squared error punishes large misses more heavily. An average error of four lakh rupees means something different for a flat priced at twenty lakh than for a house priced at two crore, so state the error in relation to the values being predicted.

Look at the Errors, Not Just the Score

A single number cannot tell you where a model fails. Pull out thirty to fifty wrong predictions and read them. Group them by cause. You may find that a price model struggles with the highest-priced homes, or that an image classifier confuses two visually similar classes.

A sentiment model offers a good local example. Reviews and messages from Punjab are often written in a mix of English, Hindi and Punjabi, sometimes in Roman script. A model trained mainly on clean English text may handle such input poorly, and the score on a standard English test set would never reveal it. Testing on a small sample of the text you actually expect to receive does.

Evaluating a RAG Application

RAG is a design in which a system first retrieves relevant passages from a document collection and then gives them to a large language model, so that the answer is based on those documents instead of only on the model's general training. Because there are two stages, there are two places to fail, and they should be tested separately.

Test retrieval on its own

Write twenty to fifty realistic questions and note, for each, the passage in your documents that contains the answer. Then check whether that passage appears among the top results the retriever returns. The share of questions where it does is a simple and useful retrieval score. If it is low, no choice of language model will rescue the answers, and you should adjust the chunk size, the number of retrieved passages or the embedding approach and measure again.

Test whether answers are grounded

Once retrieval is reliable, read the generated answers against the retrieved text. Mark any claim the passages do not support. Include questions your documents cannot answer, and check that the system says it does not know instead of inventing something plausible. A chatbot that confidently answers an unanswerable question has a real defect, even if the demo looks polished.

Keep your question set fixed and re-run it after each change. That turns tuning from guesswork into a comparison.

Document What You Found

A strong project write-up is short and specific. State the problem and the data source, including its licence. Describe how you split the data, what the baseline scored, which metrics you chose and why, what the error analysis revealed, and what the project cannot do. Add the library versions and random seeds so someone else can reproduce your numbers.

The limitations section matters most. A reviewer can tell the difference between a project you understand and a tutorial you followed by asking why you made a choice and what went wrong. Having honest answers to both is worth more than a higher score you cannot explain.

Where Feedback Fits In

Evaluation is hard to teach yourself because you do not know which mistakes you have missed. Review from someone experienced, who asks about your split and your baseline, speeds things up considerably. When looking at AI training in Punjab, it is worth asking whether projects are reviewed against these criteria, and not only whether they are completed.

Learners still deciding where to begin can look at the AI courses available across Punjab. Grow2Grab's course pages, for example, list projects such as a customer churn predictor and a RAG support chatbot. Whichever provider you consider, the checks above apply equally to every project it describes.

A Short Checklist Before You Call a Project Finished

• Does it beat a simple baseline, and by how much? • Is the test data untouched by preprocessing, tuning or duplicates? • Does the split reflect how the model would be used? • Does the chosen metric reflect the cost of different mistakes? • Have you read at least thirty wrong predictions and grouped them? • For RAG, have you tested retrieval and answer grounding separately? • Does your write-up state what the project cannot do?