Blog

MAIA has been running on WhatsApp for Peruvian, Panamanian, Colombian and other Latin American micro-entrepreneurs and smallholder farmers for two and a half years. Over that period, the hallucination problem has been a central element to the work, and has changed in very interesting ways. We've observed three general shifts in this period, each requiring a different response. It really shows how important constant monitoring and improvement is when using a rapidly evolving tool like AI.<h3>Phase one: fixing data distribution (2023-early 2024)</h3>When MAIA launched, the frontier models of the day were good at some things, but not as good what our users needed. A small store owner in Cusco asking about how to register with SUNAT, or a coffee farmer in Huila asking about a specific fertilizer subsidy, would routinely get answers from generic AI that were for silicon valley entrepreneurs, with made up details to fill the gaps. This was simply a data distribution problem. The open web that trained these models is overwhelmingly North American and European. The specific procedural and regulatory knowledge our users care about lives in PDFs on ministry websites, in trainer manuals, and in the heads of local extension officers, and not well represented in pre-training.The fix was country-specific retrieval-augmented generation (RAG). We built knowledge bases of the actual documents our users were asking about: curriculum content, ministry guidance, local regulatory texts, and partner-supplied training material. We forced the model to answer from retrieved chunks rather than from its priors. Where the retrieval came up empty, we had the system say so rather than improvise.That worked well. It was a lot of work to build and maintain, but it dropped the hallucination rate on regionally specific questions to something acceptable for a coaching context.<h3>Phase two: the chatbot sweetspot (mid 2024 to mid 2025)</h3>But over time the frontier moved. GPT-4 class models, then the next generation, then the one after that, all got dramatically better at factuality. This can be seen in our data and in third party benchmarks. Eg the Vectara hallucination leaderboard started showing top models below 5% on grounded summarisation tasks. Anthropic's CEO went on record at a developer event suggesting frontier models hallucinate less than humans on some factual tasks. For us, this period was the easiest stretch in terms of hallucinations. The same RAG pipeline that took heroic effort to build in 2023 was now backstopped by a model that, even without retrieval, was much less likely to invent a Peruvian ministry that does not exist or a tax regime that was repealed three years ago. All the neat little prompting tricks that used to be important (eg 'you are an expert in X') were simply built into the models by the providers themselves and didn't matter much anymore. This allowed us to focus on other things like proactive engagement and nailing the coaching loop. Life was good.<h3>Phase three: the overconfidence era (late 2025 to now)</h3>In the last several months something has shifted. Model improvement is accelerating and aggregate hallucination benchmarks are improving. But these headline numbers are not what production deployments are showing. What we have observed, and what independent researchers are now documenting, is a change in the character of the hallucinations. Models follow complex multi-step instructions far better than they did 18 months ago. They produce structured outputs more reliably. They are excellent at coding and at chaining tool calls. But when they are wrong, they are wrong with more conviction and less hedging than the previous generation. They refuse less. They volunteer fewer "I am not sure about this" caveats. The same training pressures that make a model good at agentic coding, where the right move is to commit to an action and execute, appear to be reducing the model's willingness to express uncertainty in factual responses, making them in some ways worse hallucinators than before.This is not just our impression. OpenAI's own evaluations show that o3 and o4-mini, their reasoning-focused models, hallucinate more than their predecessors o1 and o3-mini, despite being smarter on benchmarks. Independent analysis attributes this to reduced refusal rates: the newer models answer questions they should have declined, and when they answer, they sound just as confident as when they actually know. Comparative benchmarks released in early 2026 show one major frontier model hallucinating at over 80% on the hardest factual recall tests, while still leading the field on raw capability. Another lab's flagship hallucinates less often but produces confident fabrications when it does fail, making the errors harder to spot than the more obviously confused output of older models.Model providers are post-training their flagships for the use cases that pay the bills, which are coding assistants and agentic workflows. Both reward decisiveness and instruction-following. Neither rewards epistemic humility about an emerging market's agricultural regulations. The optimisation pressure is pointing away from the behaviour we need.<h3>What this changed in our engineering</h3>The implication for us was concrete: internal tests of new frontier models started showing worse rather than better coaching performance without proper adapations in everything else around the model (prompt, RAG, programming). The marginal hallucination that gets through to a user today is more likely to sound right than the ones we caught in 2023, which means downstream checks matter more.Over the past several months we have added a layer of programmatic checks to MAIA that sits between the model and the user. These are not glamorous and they are not novel research. They are the boring engineering equivalent of a second (third, fourth) pair of eyes, and aren't just LLM but also old school programmatic protections. For categories of claim where we know there are risky failure modes, we force validations and check them. Where validation fails, the system either rewrites with stronger grounding or falls back to a safe answer that acknowledges the limit of its knowledge. Those acknowledgements used to be done by the models themselves, but that is less so the case with new models. This is in line with where serious production AI is heading. The 2026 consensus, visible across enterprise guardrail vendors and academic work on hallucination detection, is that grounding plus runtime validation is now the standard architecture, not a paranoid extra. The era when you could ship a wrapper around a frontier model and trust it on facts is over.<h3>What this means for you&nbsp;</h3>Building an AI business coach, or any other AI tool, is a rapidly moving target. Old models are deprecated, and new models provide significant potential improvements for your tool, but could really hurt performance if not incorporated carefully. Particularly for an AI coach as frontier labs all chase coders. Constant monitoring and adaptation are critical.

Hallucination risk is changing. Quickly.

There's now twenty years of randomized-evaluation evidence on whether business training helps micro and small enterprises grow. The authoritative review is <a data-mce-href="https://voxdev.org/voxdevlit/training-entrepreneurs" href="https://voxdev.org/voxdevlit/training-entrepreneurs" target="_blank" rel="noopener">McKenzie and Woodruff's VoxDevLit on Training Entrepreneurs</a>. Their bottom line is that classroom-based training has modest but real effects on business practices and survival, while more intensive individualized consulting (Bloom et al. in India, Bruhn et al. in Mexico, Iacovone et al. in Colombia) produces substantially larger gains in productivity and profits. The catch is cost: a single consultant working with a small firm runs anywhere from $4,000 (Nigeria) to $30,000+ (Colombia) to $75,000 (Bloom's India experiment) per firm. Consulting works. Consulting at scale, on a microenterprise budget, is the open problem.Innovations for Poverty Action just published <a data-mce-href="https://poverty-action.org/leveraging-digital-tools-womens-skills-development-and-economic-empowerment-evidence-low-and-middle" href="https://poverty-action.org/leveraging-digital-tools-womens-skills-development-and-economic-empowerment-evidence-low-and-middle" target="_blank" rel="noopener">an evidence review that adds 22 recent studies focused on digital tools</a> (apps, video, SMS, WhatsApp, online platforms) for women's economic empowerment. The first section, on digital delivery of business training, sits squarely on this same question: can digital tools close the gap between cheap-but-modest classroom training and effective-but-expensive consulting?<h3>What the studies found</h3>Of the six studies, only one produced sustained gains in business outcomes:<ul><li>Estefan et al., Guatemala (2023): App-based course with 28 short videos, plus three personalized 30-minute video consultations with a business advisor. Result: +6% sales, +23% profits, +5% knowledge, with a 2.3:1 benefit-cost ratio. The researchers explicitly identified the personalized consulting as what drove engagement with the app.</li></ul>The other five all underperformed on business outcomes: <ul><li>Davies et al., Mexico/Guatemala (2024): Live Zoom course, nine two-hour sessions, 61% completion. Real instructors, real engagement, real short-term gains in planning practices (+28%) and sales (+24%). Six months later: gone. Women adopted practices during training and abandoned them afterward.</li><li>Cassidy et al., Ethiopia (2024): Smartphone app with reminders, cash-prize lotteries, and chat rooms designed to replicate networking. 22% completion. No impact on practices, sales, or profits.</li><li>Mehmood, Kenya (2024): 150 SMS messages in narrative format with an interactive chatbot. Engagement collapsed within months. No long-term effect on knowledge, sales, profits, or business survival.</li><li>de Oliveira, Brazil (2024): 15-hour WhatsApp course in video and audio, with reminders and small monetary incentives ($4–8) for adopting specific practices. No impact on practices, survival, profits, or revenue.</li><li>Cole et al., India and Philippines (2022): Weekly recorded phone calls with financial heuristics in a soap-opera narrative. Modest improvements in practice adoption (0.06–0.12 SD); the study was underpowered to detect profit effects.</li></ul>A quick reading might conclude "digital business training mostly doesn't work." But there are clear patterns here quite consistent with the bigger picture from McKenzie &amp; Woodruff. Estefan worked. Davies worked, briefly. Cole moved practices for the lowest-skilled participants. Why? Look at what Estefan had that the others didn't: a real human responding to a specific entrepreneur's specific situation, three times over the course of the program. The course content itself was identical for every participant. What varied, what the researchers identified as the active ingredient, was the personalized layer. Not the videos, not the WhatsApp reminders, not the digital vouchers for chicken inventory. The advisor.Now look at Davies, which is the most instructive failure case. Live human instructors, two-hour sessions, 61% completion. By digital training standards, that's terrific engagement. Women genuinely learned. They adopted practices during the training. And then they stopped. The researchers themselves note that women "abandoned them in the long-term" and flagged that the marketing content "was ineffective and could be improved by providing more specific actionable steps." Generic content delivered to a group, on a fixed nine-session schedule, didn't carry over once the schedule ended. Live humans weren't enough. The delivery was not personalized (same content, group setting) and not sustained (course ends, you're on your own).The other failures fit the same shape. Cassidy's app tried to substitute for relationship with engagement features like reminders, lotteries, chat rooms, and got 22% completion. Mehmood's SMS course tried to substitute with narrative quality and chatbot interactivity, and engagement decayed. De Oliveira tried to substitute with cash incentives, paying entrepreneurs to adopt specific practices, and it didn't work because the prescribed practice wasn't necessarily the right next step for each business.Generic reminders, lotteries, narrative formats, peer chat rooms, and cash-for-behavior all share something: they're attempts to engineer engagement with content that wasn't built for the individual receiving it.<h3>Coaching is Personalized and Sustained</h3>My read of these results is that the active ingredient in successful business training, digital or otherwise, isn't really the digital part, the curriculum part, or even the human part. It's:Personalization — the support adjusts to this entrepreneur's specific situation, priorities, and bottlenecks. Sustained engagement — something pulls the entrepreneur back into action over time, rather than ending when the curriculum does.This is exactly what makes consulting more effective than classroom training in the McKenzie-Woodruff synthesis: consultants tailor advice to the specific firm and stay involved over months. Estefan's Guatemala study delivered a light-touch digital version of that mechanism (three personalized advisor sessions woven through an otherwise generic video course) and it produced consulting-style results at training-style costs. The other five digital interventions in the IPA review delivered neither personalization nor persistence, and tried some shallow substitutes like reminders, lotteries, peer chat rooms, narrative formats, cash incentives. None of those substitutes closed the gap.The reframe matters because it separates the mechanism (personalization plus persistence) from the delivery channel (human consultant, AI system, peer group, hybrid model). The McKenzie-Woodruff evidence tells us the mechanism works; the IPA review tells us the substitutes for it don't. Neither tells us whether the mechanism can be delivered cost-effectively at scale through software.<h3>What this means for MAIA</h3>We've built our service around this focus: personalized, sustained engagement delivered through WhatsApp, with an AI coach that holds context across weeks and months, adjusts to the individual's actual situation, and pulls people back into action through a sprint cadence rather than a course completion. None of the studies in the IPA review tested an AI-delivered version of what Estefan's human advisors did, and the McKenzie-Woodruff literature doesn't yet have AI-mediated coaching RCTs to consider. The question of whether AI can deliver the consulting mechanism at a fraction of the cost is genuinely open. But the research is clear about what the high potential mechanism is, and the design choices that have failed are the ones we've also been careful to avoid.

Coaching vs Courses

<h3>Reading Impact From the Conversation: Predicting Whether MAIA Worked, Without Asking</h3>We face a typical problem with our own internal impact surveys (which are different from <a data-mce-href="https://mypeasesoria.org/blog/impact-numbers" href="https://mypeasesoria.org/blog/impact-numbers" target="_blank" rel="noopener">external impact evaluations</a>): the people who answer your internal impact survey might be special, and not represent the non-respondents, the silent majority. If their outcomes look systematically different from the respondents', our internal impact metrics would be biased. This bias could be in two directions. It could be that only successful power users bother to respond to the survey, biasing results upwards. Or it could be the case that, because many of our users are given the tool through a government agency like Tu Empresa or AMPYME, which also provide financial support to struggling businesses, users see the survey as an opportunity to get financial assistance and bias the results downards.For MAIA, this is really important. We coach small business owners over months. Our outcome surveys at 60 and 90 days ask whether their costs went down and their sales went up. The active users who respond are giving us valuable signal, but they're a minority of our users, and only give this information at a few points in time. Are they representative of non-respondents? How does this impact change over time beyond the survey questions?Over the past few months we built and tested an answer. We built a model that predicts how a user would have answedr the impact surveys, based on their MAIA conversations and resulting profile and diagnostic. At the cohort level, its predictions match actual survey outcomes within a few percentage points. Using a metric from credit scoring, the AUC is 0.83. Importantly, all of the signals were derived blind to the outcome variable. We then examine those same conversation features for the entire user base and compare respondents to non-respondents. The results show they look the same: the respondents are not a biased subset.<h3>What MAIA already knows about a user</h3>Every active MAIA user has a structured profile that the system rebuilds periodically from their conversation history. The profile captures things like the user's business sector and stage, their stated goals, their current operational practices, which of the eight capability pillars (customer and market, offer quality, sales channels, operations and supply, finance and records, costing and pricing, people and productivity, planning and risk) MAIA assesses them at, and how those assessments have changed over time.We hypothesized that some of these structured fields might predict survey outcomes. To test it, we trained a predictor against the users who had answered the surveys, then evaluated which signals carried the most weight. The model itself is deliberately simple. The specific model family matters less than the features, and we wanted something we could inspect and reason about rather than a black box.The signals that turned out to predict outcomes most strongly were exactly the ones with the most intuitive relationship to actual business performance:Whether the user has adopted a cost-tracking template. This is a structured tag in the profile. It captures whether MAIA has confirmed, from conversation evidence, that the user is using a recurring method to track their unit costs. The intuition is direct: you can't manage what you don't measure. Users who adopt a cost-tracking habit are dramatically more likely to report substantial cost reductions.The trajectory of business sophistication over time. MAIA's diagnostic rates each user across the eight capability pillars. The static assessment matters somewhat, but the *direction of movement* matters more. Users whose pillars are improving over months tend to be the users showing real outcomes. The improving slope is itself a signal of an engaged, learning operator.Pillar regression, the warning sign. When MAIA's assessment of a user's capability *decreases* on a pillar between rebuilds (for example, from "Proficient" back to "Developing"), it's usually because something concrete went wrong: a habit lapsed, a process broke, something didn't get re-implemented. Regressions are rare but informative. Users with multiple regressions over their lifecycle are systematically less likely to report impact.Whether the conversation contains evidence of completed actions. This one we extract from the conversation itself, not just structured fields. When a user reports, in their own past-tense words, that they did something MAIA proposed ("subí el precio a S/12," "ya no compro de ese proveedor," "ahorré como S/200 esta semana"), it's the strongest single predictor. Adoption is the whole game, and reported execution is the visible signal of adoption.There are other signals in the model, and the exact weighting matters less than the picture they paint together: a user who tracks costs, whose capabilities are growing, whose progress isn't regressing, and who reports concrete actions taken. That's a user who's likely to report impact when surveyed. None of this is mysterious. The signals work because they describe the underlying mechanisms by which a small business actually improves.<h3>The validation step</h3>First, calibration on respondents. For users who *did* answer the surveys, we compared their predicted outcome distribution (developed blind to the outcome variable) to their actual responses. The model was accurate enough that the cohort-level predicted impact rate matched the cohort-level actual impact rate within a few percentage points, and an AUC of 0.83.Second, and more importantly, a feature-distribution comparison on non-respondents. The crucial question was whether non-respondents look similar on those key input features. If non-respondents systematically had lower cost-tracking adoption, fewer action reports, more pillar regressions, then our surveys would be capturing only the happy customers. If they didn't, the silence isn't correlated with poor outcomes.The distributions overlapped closely. The non-responding active users look, by every signal we can measure, much like the responding active users. This is reassuring, but it comes with a limit we name below: the check only covers the features we have. If non-respondents differ on something we don't observe, we can't see it from this analysis alone.<h3>What this enables</h3>A few useful things follow from having a working predictor.Population-scale impact monitoring. We no longer need to wait for survey responses to know whether a recent product change moved the needle. As a preliminary check, we can score every active user weekly and track the predicted impact distribution over time. Surveys remain the ground truth at 60 and 90 days, but the predictor gives us a leading indicator on a much larger N.Earlier and broader experimentation. When we test a prompt change or a new coaching mode, we can score the user population both before and after to see if the predicted distribution shifts. We don't need a fully powered RCT to see whether a change is plausibly working. The predictor offers a fast, cheap signal that we can validate later with surveys.Honest segment reporting. We can break down predicted impact by country, sector, acquisition source, or any other axis without losing statistical power to small response counts. Partners in Panama, Peru, and Colombia each get a meaningful read of how MAIA is performing for users in their region without us having to collect hundreds of survey responses per segment.Catching regression early. A user whose predicted impact score drops over time (capability pillars regressing, no execution evidence, no template adoption) gets flagged in our internal review. We can look at what changed in their coaching and either intervene or learn.<h3>The thing we're not claiming</h3>We are not claiming the predictor *is* impact. It's a model of how observable conversational and profile signals correlate with self-reported business outcomes. If the underlying surveys are themselves biased in some way the predictor doesn't capture, because both the predictor and the survey draw from the same observed signals, the model would inherit that bias. That's why our true north will always be external impact evaluations with a clear identification strategy.We're also not claiming this is the right approach for everyone. Profile-based prediction works for MAIA because we have a structured profile that captures meaningful business state. A simpler chatbot without that profile wouldn't have the inputs to do this. The general point, *triangulate self-report against signals from the conversation itself*, is portable. The specific implementation isn't.<h3>Where this goes next</h3>A few open questions we're working on:- Are the predictive signals stable across cohorts and time? Our validation was on a recent cohort. We need to re-validate quarterly as the user base evolves and as we change the product.- How much does the predictor improve when we add longer-horizon outcome data? Right now we validate against 60 and 90-day surveys. As we accumulate impact data at 6 months and 12 months, we expect the picture to deepen, particularly on the sales side, where we hypothesize wins compound over time.- Can we predict the *type* of impact (cost vs sales) and not just whether impact occurred? The two paths <a data-mce-href="https://mypeasesoria.org/blog/defense-and-offense--two-different-wins" href="https://mypeasesoria.org/blog/defense-and-offense--two-different-wins" target="_blank" rel="noopener">look mechanistically different</a>, and the predictive signals may differ between them.The measurement problem we opened with isn't unique to MAIA. Any coaching service that interacts with users over time, collects partial survey responses, and wants to know its true impact distribution is facing the same question. We've put a first answer on the table for our case. If you're working on something similar, we'd love to compare notes.

Predicting Whether MAIA Worked

When we designed MAIA, we assumed a good coach is a good coach. The same patient questioning, the same nudges toward tracking numbers, the same encouragement to take action. These would help any small business owner improve their bottom line by both spending less and selling more.A recent deep dive into thousands of MAIA's users in Peru, Panama, and Colombia shows that actually, users tend to focus exclusively on one or the other. Cost-cutting wins and sales-growing wins are not just two outcomes of the same coaching. Users who reduced their costs and the users who grew their revenue are pretty different, and the conversations where MAIA helped them look completely different too.Some users are playing defense, some are playing offense. And what surprised us is that in the first few months of the coaching relationship, most users play one game or the other, not both.<h3>What the data shows</h3>MAIA asks two outcome questions to active users: one about cost reduction at around 60 days, and one about sales increases at around 90 days. When we matched the users who answered both, we expected to see overlap. That is, to see the best performing users winning on both fronts. Instead, cost-positive users and sales-positive users were almost entirely different entpreneueurs. Wins clustered on one side or the other.This isn't because MAIA is coaching them differently. MAIA doesn't run a "cost reduction mode" versus a "sales growth mode." The coaching takes an early focus based on the user's priorities, their current pain, the decision in front of them this week. Users self-select into defense or offense based on where they are in their business and what's worrying them. MAIA meets them there.<h3>What defensive coaching looks like</h3>Cost wins come from a small repertoire of defensive moves. Reading through the conversations of users who reported meaningful monthly cost reductions, the same patterns come up again and again:Avoiding a bad investment. A poultry farmer in Panama was about to invest several hundred dollars in a slow-growing chicken variety because a neighbor had recommended it. MAIA walked through the unit economics with him (feed cost per day times growth cycle) and the user ran the math himself. His conclusion was blunt: "yo no voy a cometer esa locura." He switched to a faster-cycling variety and pocketed the difference. The MAIA win wasn't that MAIA added anything. It was that MAIA prevented something.Free or cheaper substitutes. Several agricultural users in Peru and Panama replaced purchased inputs with on-farm or low-cost alternatives surfaced by MAIA: chicken manure as fertilizer, agua de tilapia for irrigation, mulch from on-farm material. None of these are MAIA's invention. They're well-known practices the user hadn't connected to their own situation. They tend not to hear about them because suppliers are just prompting the costly solutions they sell, not the cheap or free alternatives. The savings are real and recurring.Formalization that unlocks tax benefits. A small services business in Panama spent two months working through MAIA on AMPYME registration and the new tax structure for micro-enterprises. The result wasn't new sales, it was that the business stopped paying the wrong amount of tax every month. Maybe not a sexy marketing campaign, but real money.Saying no to a tempting purchase. A user in Panama was considering a used commercial refrigerator. MAIA pulled together the maintenance plus electricity calculation alongside the price, and it became clear the unit would cost more to run than it would save. He decided not to buy.What unites these stories is the shape of the moment. The user was about to spend money, MAIA inserted itself before the spend, and the math made the decision. The coaching is negative: what not to do. The job is to be a second opinion at high-stakes purchase moments with the user's best interests front and center, rather than those of a supplier.<h3>What offensive coaching looks like</h3>Sales wins look pretty different. They come from doing something new, often after MAIA has helped think through whether it would actually be profitable.Price increases after margin math. A small artisan in Peru produced handwoven items and was selling key pieces at prices that, on close inspection, didn't cover materials and time. MAIA worked through the unit cost (materials plus hours times an hourly rate) and suggested a higher price, from S/50 to S/60 on one product line, with similar increases on others. She tried it and most customers accepted. The same volume produced meaningfully more income.Adding a new product line. A bodega owner in Peru added baked goods (kekes) to her shop after MAIA helped run the ingredient cost and pricing math. The marginal revenue was modest but consistent: new revenue from existing customers and existing foot traffic.Customer reactivation. A small services provider in Peru drafted a reactivation message to dormant past clients with MAIA's help. Within the first week, several reactivated. The conversation was about how to generate new value from existing assets like her contact list, with no new investment needed.Channel additions and promotional campaigns. WhatsApp Business catalog setup, a sorteo escolar that drove foot traffic, an aniversario promo with a free small item to attract new customers. Concrete, time-bounded experiments that produced traceable revenue.What unites these stories is that the user was about to do something (or could be persuaded to do something), and MAIA worked through the details with them to pressure-test the economics, sharpen the execution, and prepare the assets. The coaching is positive: what to do, with confidence the math works. The mechanic is execution support. MAIA prepares the user to act.<h3>What a naive chatbot would miss</h3>Here is where it gets interesting, and where we think the design of the coaching engine actually matters. A naive Q&amp;A chatbot just answers whatever the user brings to the conversation. It happily stays in whichever mode the user started in. A cost-focused user asks a cost question, gets a cost answer, comes back next week with another cost question, gets another cost answer. Same for sales. The user stays in their lane, and the chatbot never notices the other half of the business is untouched.That's a missed opportunity. The farmer who saved money on chicken variety still has pricing decisions he hasn't examined. The artisan who raised her prices still has input costs she's never audited. The defensive user will plateau on savings and never discover the offensive moves available to her. The offensive user will grow revenue while quietly bleeding margin on costs she never looked at.A good team needs good offense and good defense, and MAIA is built with that in mind. The coaching engine runs a structured diagnostic across eight pillars of the business: customer and market understanding, offer quality, sales channels, operations and supply, finance and records, costing and pricing, people and productivity, and planning and risk. Each pillar is rated on a four-level scale from Emerging to Extending, and MAIA tracks which areas the user has and hasn't addressed. As the first-priority areas stabilize, which is usually around the time this first impact result is fielded, MAIA proactively surfaces the next one. The defensive user who cut costs in month two gets nudged toward a pricing conversation in month four. The offensive user who added a product line gets pulled into a margin audit. The goal is not to keep the user in whatever game they showed up playing, but to move them, over time, toward improvement across the whole business.This is the difference between a chatbot and a coach. A chatbot waits to be asked. A coach has a view of the whole business and decides what to focus on next. We think this will be observable in the longer-term impact numbers. The early exchanges are about meeting the user where they are, defense or offense, whichever they brought. The real sustained coaching relationship helps users achieve both, proactively addresses all key dimensions of their business, and builds personal initiative along the way. Since we've only begun fielding those surveys, longer term business impact results are still to come. What we can say is that users stick with MAIA over long periods with remarkably little churn, which suggests the relationship is delivering broad utility that continues to feel worth their time.

Defense and Offense: Two Different Wins

Part 3 in our series on MAIA's architecture---With AI tools becoming more accessible, more organizations are exploring AI-powered services like coaching bots, advisory tools, training apps. The technology holds a lot of promise. But what does it actually take to go from "we'll use AI" to a service that works reliably for real users?Based on our experience building MAIA across five countries for the past 2.5 years, here are the core capabilities that a well-functioning AI coaching service needs, and a rough sense of how much of the work each one represents (based on lines of code, which is an interesting but imperfect measure).1. Knowing What to Say — and How to Say It (~26%)The largest piece of the puzzle is instructing the model. To be useful across a variety of use cases and contexts, you actually need a library of prompts, not just one, because a coaching service needs to behave differently depending on who it's talking to, what stage they're at, and what they need. A first-time user in Panama exploring formal registration needs different guidance than a returning user in Kenya refining their pricing strategy. This includes decision logic that routes a user's question to the right capability and prompt, if your chatbot will do more than just generic 'talk do your documents' advice. Should this trigger a web search? Generate a marketing image generation? Offer a diagnostic? Simply answer? This library is never "done." It evolves constantly as you learn what works.2. Reliability and Delivery (~22%)The gap between a demo and a product is reliability. Messages need to arrive once and only onc, even when services go down. When an upstream provider has an outage, the system needs to gracefully switch to another provider rather than go silent. When a user sends a voice note, it needs to be transcribed and processed. When two messages arrive at the same instant, they can't corrupt each other's state, and when users multi-message (common in WhatsApp) you need your tool to behave like a human and combine them, rather than behave like a chatbot and respond one at a atime. This is the plumbing. No one sees it, but you definitely notice when its not right, and it significantly impacts usage.3. Proactive Coaching (~19%)A service that only responds when spoken to isn't really coaching. Nearly a fifth of MAIA's codebase is dedicated to reaching out: personalized weekly outreach, smart follow-ups that review the recent conversation to decide whether a check-in is useful, engagement logic that adjusts frequency based on how active each user is, structured multi-week sprints, and onboarding sequences tailored to each country program. This entire subsystem exists independently of the regular response flow, and has to be built around the unique economics of WhatsApp.4. Memory and Understanding (~11%)AI models don't inherently remember users between conversations. Building persistent memory means extracting structured information from each interaction, particularly when chats span months and years as some of ours do. This includes things like what the user's business does, their financial situation, their marketing channels, their goals, and maintaining an evolving profile that gets better over time. Unlike generic chatbots which can struggle with coherent focused relevant memory, MAIA's business focus allows for a much smarter and more targeted memory. Moreover, MAIA goes beyond recall to a structured diagnostic that assesses users across seven business pillars (like sales, finance, operations, and digital tools), so that advice is calibrated to where someone actually is and what is their key next priority.5. Context Assembly (~7%)Even with a good model and good instructions, the quality of an AI response depends on what information it can see. Before every interaction, the system assembles a context window: the user's business profile, their diagnostic results, recent conversation history, any active coaching program, and relevant knowledge base references. Think of it like briefing a human coach before each session. The better the briefing, the better the session.6. Conversation Handling (~10%)Real users don't behave like demos. They send three messages in a row before the bot can respond. They reply to a messages out of order, from last week. They forward something from another chat. They go quiet for two months and come back. Handling all of this gracefully, like batching rapid messages, transcribing audio, threading quoted replies, managing session state, preventing the bot from talking to itself in edge cases, is another significant layer of work.7. The AI Model Call (~7%)Here is the part that most people picture when they think "AI service": sending a request to a language model and getting a response. It includes connecting to the model provider, parsing the response, and having fallback options when one provider is unavailable.8. Localization and Cultural Fit (~5%)Operating across countries isn't just translation. Coaching advice about formal business registration is completely different in Peru versus Kenya. The relevant government agencies, legal frameworks, financial products, and market conditions are all different. The system needs to resolve the right language, the right country context, and the right institutional references for each user, and allow per-country overrides without requiring a new deployment.ConclusionEach of these capabilities is essential, and they do overlap. Your proactive coaching engine needs your memory system to know who to reach out to. Your context assembly depends on your reliability layer to ensure profiles are current. Your instruction library depends on your localization system to serve the right version. The LLM is the engine, but a well-functioning service is the whole car: navigation, fuel system, chassis, dashboard. Each one is a product design decision that impacts user satisfaction, tool usage and ultimate impact.

Building an AI Coaching Service: The Key Ingredients

Two data sources are starting to paint a picture of Maia's early impact on entrepreneurs.<h3>ILO Colombia Study: Behavior Change in Coffee Country</h3>The International Labour Organization conducted an independent pre-post survey of coffee farmers in Colombia's Huila region—a baseline and follow-up two months later. The results were:Business practices: &nbsp; - Farmers keeping financial records: 56% → 78% &nbsp; - Formalized businesses: 2% → 17% &nbsp; - 55% of users reported directly implementing changes based on MAIA interactionsAgricultural practices: &nbsp; - Harvesting only mature beans: 80% → 85% &nbsp; - Minimizing waste/residues: 72% → 87% &nbsp; - Proper fertilizer application: 88% → 91% &nbsp; - Using climate-appropriate seeds: 83% → 87%The formalization jump—from 2% to 17%—is particularly notable. Formalization is notoriously hard to move. One farmer shared: "MAIA helped me formalize my coffee brand and create the company with all the legal requirements."What users valued most was having expert guidance available on demand. As one producer put it: "It's like having the agronomist right there on your phone." Another noted: "I used to mix fertilizers and composts together. Now I learned there are products that shouldn't be combined—MAIA explained that to me."<h3>Peru User Surveys: Self-Reported ROI</h3>Separately, we recently began surveying MAIA users in Peru and Panama at months 2 and 3 about changes in their business finances. The questions are simple: have your costs gone down due to Maia? Have your sales gone up due to Maia? Users choose between no change, up to S/150/month (~10% of average MSME profits), or more than S/150/month.From the first 165 responses: Cost Reduction (Month 2): 44% reported lower costs, Sales Increase (Month 3): 58% reported higher sales.&nbsp;These numbers are a floor. At months 2-3, MAIA's coaching relationship is just beginning. User profiles are still taking shape, and most haven't completed full coaching cycles. The progression from 44% (month 2) to 58% (month 3) illustrates momentum building over time. But still just using these early results, the average user reports S/47/month in cost savings and S/60/month in sales gains (at the bin midpoints). At a ballpark program cost of $1/user/month, that's a 12-17x ROI.According to <a data-mce-href="https://voxdev.org/voxdevlit/training-entrepreneurs" href="https://voxdev.org/voxdevlit/training-entrepreneurs" target="_blank" rel="noopener">https://voxdev.org/voxdevlit/training-entrepreneurs</a>, standard entrepreneurship training programs (typically 5-7 days in a classroom at ~$177/participant) produce around 5% sales gains and 10% profit improvement. The most effective "personal initiative" programs cost ~$750/participant and achieve 30% profit gains over two years. MAIA is showing comparable behavior change at a fraction of the cost: $12/year versus $177-750 for traditional training—and delivered continuously rather than as a one-time event, meaning its impact should rise over time while the impact of one-time training fades.<h3>What We're Learning</h3>The ILO study surfaced an important insight: "MAIA facilitates progressive adoption of digital technologies in rural areas, especially when combined with technical support and contextualized content." Users valued personalized, locally-relevant guidance. This is what enabled adoption of an advanced AI tool by users with very low tech comfort.These are early numbers. The surveys are self-reported, sample sizes are modest, and respondents still in the first months of the coaching relationship. But the direction is clear: users are changing behavior, and they're attributing it to MAIA.We'll keep measuring and sharing what we learn.

Blog

Hallucination risk is changing. Quickly.

Coaching vs Courses

Predicting Whether MAIA Worked

Defense and Offense: Two Different Wins

Building an AI Coaching Service: The Key Ingredients

Impact Numbers