Copilot & Foundry: The Microsoft AI Stack for Small Business · Lesson 10 of 10
Shipping, Measuring, Iterating
Time: ~30 min reading + ~30 min worksheet (capstone) Goal: Close out the Northside engagement professionally. Learn the rollout, metrics, and improvement-loop patterns that separate good deliveries from great ones. Do the capstone.
---
There's a gap between "the agent works" and "the agent is delivering value" that kills a lot of AI projects. Module 10 is about closing that gap.
For Northside, we've built v1 in Copilot Studio (Module 8) and added Foundry for the service history agent (Module 9). Total cumulative investment: ~$100K over Year 1, $146K in Year 2. Mike and Lisa are invested — both financially and personally. They need to see this through.
Your job for the next several weeks is rollout, measurement, and iteration. And then the handoff.
Two rules that will save you every time:
Rule 1: Never big bang
A 45-person company. You are not enabling this for all 45 on the same day. You're doing it in waves. Even if the pilot went perfectly.
Rule 2: Have an off switch
Always have a way to disable the agent or roll back cleanly if something surprising happens. Configuration as code, version control on Copilot Studio exports, Foundry agent versioning. You want to be able to say "we rolled back to Tuesday's version" without hesitation.
The Northside rollout plan
Week 1 (pilot → extended pilot): Pilot group of 8 expands to 20. New users include field techs (who bring the service history use case), a broader mix of office roles.
- Send out a short kickoff message (Teams announcement).
- Run one live 15-minute demo (or record it).
- Enable Teams app for this group only.
- Rachel available for support.
Week 2 (full rollout): Enable for remaining 25 users.
- Repeat the kickoff pattern.
- Drop-in office hours Tuesdays at 10am and Thursdays at 2pm for two weeks.
- Collect questions and issues in a shared Teams channel.
Weeks 3–6 (stabilization):
- Weekly check-ins with Mike, Lisa, Rachel.
- Biweekly content gap analysis (see below).
- Address recurring issues with topic additions, content updates, or agent prompt tweaks.
Week 7 (pilot close-out):
- Present metrics.
- Document what we learned.
- Plan Phase 2.
The three tiers of metrics.
Usage metrics (the raw data)
These answer "is the agent being used?"
- Active users per week — unique users who asked at least one question
- Conversations per week — total conversation count
- Questions per conversation — depth of engagement (higher = users trust it more)
- Deflection rate — % of conversations that didn't escalate to Rachel
- Escalation rate — % of conversations that did escalate
- Content gap rate — % of conversations where the agent failed to answer at all
Quality metrics (the user experience)
These answer "is it delivering value?"
- Thumbs-up/down ratio — user satisfaction at the message level
- Conversation completion rate — % of conversations that ended naturally vs. were abandoned
- Follow-up question frequency — Rachel receiving clarifying questions from users who got bad answers
- Content-citations-present rate — especially for the Foundry service history agent, are citations always appearing?
Business impact metrics (the ROI)
These answer "is the business better off?"
- Rachel's self-reported time saved — weekly, qualitative + hours
- Volume of escalations to Rachel — should be dropping
- Technician on-site efficiency — for the service history agent, average time-on-call vs. pre-deployment baseline (if measurable)
- Customer satisfaction indicators — informal signals from customers about whether the tech got things right the first time
The metrics dashboard
Don't send a spreadsheet. Build a one-page PDF or a Power BI page with:
- Three headline numbers (active users, deflection rate, Rachel's hours saved)
- A trend chart (last 8 weeks)
- Top 5 content gaps (questions that failed)
- Three recent success stories (specific examples)
Mike and Lisa need this monthly. Anything less is a signal you're not paying attention.
AI agents aren't "shipped and done." They're shipped and tuned. Here's the loop that keeps them getting better.
Weekly: content gap review (15 min)
Look at what the agent failed on or escalated last week. Triage:
- Fixable with content: add a doc or update the handbook
- Fixable with a new topic: author one in Copilot Studio
- Fixable with prompt tuning: adjust the Foundry system prompt or the Copilot Studio agent's guidance
- Escalation-appropriate: these are supposed to go to Rachel. Leave alone.
Biweekly: user feedback synthesis (30 min)
Review thumbs-down responses. Are users consistently unhappy about a specific topic? Pattern-match; fix upstream.
Monthly: regression testing
Before making big changes, run your test set (50–100 known questions with known good answers). Did you break anything? Do this especially when upgrading Copilot Studio versions, tweaking prompts, or re-indexing RAG content.
Quarterly: architecture review
Is the current architecture still serving the client? Is the knowledge base approaching limits? Are users asking for things the current stack can't do? This is where you plan the next graduation.
What exists in writing after the project closes.
Architecture doc
The authoritative description of what was built. Includes:
- System diagram (the Pattern 2 sketch from Module 9)
- Component list with versions
- Azure resource inventory with resource group names
- Integration points with external systems
- Credentials inventory (where they live, rotation schedule)
Runbook
What to do when something goes wrong:
- Agent not responding: where to check, what to restart
- Cost spike: how to investigate
- Content update: step-by-step process
- Rollback: how to revert to previous version
- Contact list for escalation
Knowledge base schema
For the Foundry agent specifically:
- What's in the Azure AI Search index
- How data flows from ServiceTitan into the index
- Refresh cadence
- Schema of the documents (fields, metadata)
Governance doc
- Purview configuration
- DLP policies in effect
- Sensitivity labels applied
- Audit log access
- Data retention policies
User guide
For Rachel and other non-technical stakeholders:
- How to add a document to the agent's knowledge
- How to respond to an escalation
- How to export metrics
- What's "normal" vs. "call the IT partner"
Most consultants skip the documentation step. Don't. It's the thing clients remember when their next vendor asks "who built this?"
Three ways this engagement can settle into steady state.
Option 1: Full managed service (recommended)
You stay as the retainer partner. Monthly fee ($1K-$3K/month for Northside-scale). You monitor, maintain, respond to issues, run the improvement loop, and plan the next phase. Client has a single neck to grab. This is the best outcome for both sides.
Option 2: Client takes it over
Client's internal IT (or their existing IT partner relationship) handles day-to-day. You're on call for significant changes at project rates. Works if they have capable internal IT that's been involved in the build.
Option 3: Hybrid
Client handles day-to-day and content updates. You handle architecture-level work, Foundry changes, anything requiring Azure expertise. Retainer is smaller ($500-$1K/month).
For most SMBs, Option 1 or Option 3. Pure client takeover (Option 2) often regresses within 6 months because the improvement loop fades.
With the helpdesk agent shipped and the service history agent stabilizing, what's next?
For Northside, the natural progression:
Phase 3 candidates
- Customer-facing quote/estimate bot — Lisa's idea from Module 9. Pattern 3 territory. $100K+ build.
- Dispatch assist agent — helps dispatchers optimize schedules, triage incoming calls. Pattern 1 or 2.
- Proposal/bid assistant — helps the sales team write and price bids. Pattern 1 or 2, depending on complexity.
- New hire onboarding agent — dedicated agent for onboarding. Pattern 1.
Don't propose all of these at once. Sequence them:
- Phase 3 (6–9 months out): Pick one. Ideally the one that shows up as the biggest bottleneck. Ship it.
- Phase 4 (12–18 months): Second agent. By now the client is sophisticated and the pattern is repeatable.
- Phase 5 (18–24 months): Start thinking about agent-to-agent orchestration. Multiple specialists under one roof, coordinating.
When you have 3–5 specialist agents, the orchestrator pattern (Module 5) starts paying real dividends. Users get one entry point. Specialists handle their domains. Agents even talk to each other for cross-domain questions.
This is where Microsoft's A2A (agent-to-agent) features become important. As of 2026, both Copilot Studio and Foundry support various flavors of this:
- Handoff patterns (one agent delegates to another)
- Parallel coordination (two agents work on parts of the same request)
- Hierarchical orchestration (a top-level agent supervises specialists)
You don't need to sell this on day one. But you should be aware it exists. When a client with 3 agents asks "can you make them talk to each other?" — you say yes, and you have a framework for how.
The final week of the original engagement. What actually happens:
Final meeting with Mike, Lisa, Rachel
- Present the metrics dashboard
- Walk through three success stories
- Show the documentation
- Discuss transition to managed service
- Surface Phase 3 options (without pushing)
- Get a testimonial (in writing or video)
Internal post-mortem
- What worked?
- What was harder than expected?
- What would you change?
- What case studies can you use (with permission) for future business development?
Contract transition
- Statement of work for managed service phase
- Signed
- Effective the day after current engagement ends
Now you have a reference client. A paying retainer. A template for the next engagement. And a real case study.
That's the whole arc.
---
- Rollout in waves, never big bang. Always have an off switch.
- Measure three tiers: usage, quality, business impact. Dashboard monthly.
- Improvement loop is forever — weekly content review, biweekly feedback, monthly regression, quarterly architecture.
- Documentation is the deliverable clients remember — architecture doc, runbook, knowledge base schema, governance doc, user guide.
- Managed service retainer is the best handoff model — for client continuity and for your recurring revenue.
- Sequence future phases deliberately. One agent at a time. Earn the next one.
- A2A orchestration is the long-term horizon — know it exists, plan for it when the client has 3+ specialist agents.
---
For the final exercise of this course, you'll design your own Microsoft-stack AI engagement from scratch. This is the worksheet — not just a theoretical exercise. Do it seriously. It's the artifact you'd use to pitch the engagement to a real owner.
Pick a real or hypothetical client you know well. SMB, 25–200 employees, M365-based.
Produce:
- A client summary (who they are, what they do, where they hurt)
- An architecture doc (which pattern, component list, reasons for choices)
- A 90-day rollout plan (week-by-week)
- A 3-year TCO (seven cost categories, realistic numbers)
- A success metrics plan (what you'll measure, targets, dashboard structure)
- A handoff/managed-service proposal (options, pricing, your recommendation)
Think of it as a pitch document. If you were walking into a real owner meeting, this is what you'd bring.
The worksheet gives you the scaffolding. The thinking has to be yours.
---
Next up: You're done with the lessons. Do the capstone. Then flip through the Resources folder — it's the working material you'll reach for on real engagements.
---
Time: ~60 min (this one is bigger than the others — it's the final deliverable) Goal: Design a complete Microsoft-stack AI engagement from scratch. This is the artifact you'd walk into an owner meeting with.
---
Client name: ______________________________
Size (headcount): ______________
Industry: ______________________________
M365 licensing in place: ______________________________
Other key systems: ______________________________
Decision-maker(s): ______________________________
---
What specific problem is this engagement solving? One paragraph. Be concrete — not "they need to be more efficient" but "their HR manager spends 10 hours a week answering repeated questions that are documented but scattered."
```
```
Who feels the pain most acutely? ```
```
Why hasn't it been solved yet? ```
```
---
Apply the decision tree (Module 5). What pattern are you recommending?
Pattern: (1 / 2 / 3)
Justification (2–3 sentences): ```
```
Component list:
- User interface: ______________
- Agent platform(s): ______________
- Knowledge sources: ______________
- Model(s): ______________
- Integrations/connectors: ______________
- Governance layer: ______________
- (Optional) Foundry components: ______________
Architecture sketch (text diagram): ```
```
---
Weeks 1–2: Discovery & Prep
```
```
Weeks 3–4: Build
```
```
Weeks 5–6: Pilot
```
```
Weeks 7–9: Rollout
```
```
Weeks 10–13: Measure & Iterate
```
```
Key milestones (3–5):
- ______________________________________________________
- ______________________________________________________
- ______________________________________________________
- ______________________________________________________
- ______________________________________________________
---
Use the seven-category framework from Module 6.
Annual recurring (Year 1+)
| Line | Year 1 | Year 2 | Year 3 | |---|---|---|---| | User licenses (M365 Copilot, etc.) | $____ | $____ | $____ | | Platform capacity (Copilot Studio) | $____ | $____ | $____ | | Azure consumption (if applicable) | $____ | $____ | $____ | | Governance / Purview | $____ | $____ | $____ | | Training & change management | $____ | — | — | | Annual recurring subtotal | $____ | $____ | $____ |
One-time
| Line | Amount | |---|---| | Consulting build | $____ | | Any one-time licensing | $____ | | One-time subtotal | $____ |
Ongoing managed service
- Year 1 partial: $____
- Year 2: $____
- Year 3: $____
3-Year Cumulative TCO: $____________
Hidden costs you've flagged (at least three):
- ______________________________________________________
- ______________________________________________________
- ______________________________________________________
---
Primary metrics (what you'll report monthly):
- ______________ | Target by day 90: ______________
- ______________ | Target by day 90: ______________
- ______________ | Target by day 90: ______________
Secondary metrics (diagnostic):
- ______________________________________________________
- ______________________________________________________
- ______________________________________________________
Business impact metrics:
- ______________________________________________________
- ______________________________________________________
Dashboard structure (sketch the one-page layout):
```
```
---
In 60 seconds to the owner, how do you explain what they get for this investment?
```
```
If challenged on TCO, what's your one-page defense? ```
```
---
Your recommended handoff model:
- ☐ Full managed service
- ☐ Client takeover
- ☐ Hybrid
Proposed monthly retainer: $____________
What the retainer includes:
- ______________________________________________________
- ______________________________________________________
- ______________________________________________________
What's out of scope (project-priced):
- ______________________________________________________
- ______________________________________________________
---
Name three real risks to this engagement. For each, write one mitigation.
Risk 1: ______________________________________________________ Mitigation: ______________________________________________________
Risk 2: ______________________________________________________ Mitigation: ______________________________________________________
Risk 3: ______________________________________________________ Mitigation: ______________________________________________________
---
Looking ahead past the initial engagement: what's Phase 2? Phase 3? Don't pitch them now — just note them so you have a sequenced roadmap in your head.
Phase 2 (6–12 months out): ______________________________________________________
Phase 3 (12–24 months out): ______________________________________________________
---
Once you've filled this in, read the whole thing as if you were the owner. Three checks:
- Is it specific? If everything in it could apply to any company, you've described a template, not an engagement. Go back and add client-specific detail.
- Is the math honest? Cross-check the TCO against Module 6. Any line item that looks too low is probably wrong.
- Is the value claim defensible? If the owner asks "how do you know?" for any number or projection, can you answer without flinching?
If yes on all three — congratulations. That's a deliverable. You can pitch that tomorrow.
If no on any — revise. Capstones aren't graded, but the next owner meeting will be.
---
Check yourself: If you produce one of these for every discovery conversation you have this year, your close rate will be noticeably different by Q4. The scaffolding is the value — don't skip it.
That's the end of the course. The Resources folder has the working material (glossary, decision tree, prompt library, governance checklist, AI-102 crossover). Go build something.