AI Impact Assessment Specialist
AI-focused specialism of Impact Assessment Specialist
AI Impact Assessment Specialist
Assesses the social, economic and developmental effects of AI systems and AI-enabled interventions, such as advisory chatbots for farmers, AI triage in clinics or automated targeting of cash assistance, on the people and institutions they reach. Unlike an impact assessment specialist who uses AI to measure programme impact, this role treats the AI system itself as the subject: who benefits, who is harmed or excluded, and whether it contributes to development results.
- Level
- Senior / Advisor 6–10 years
- Works with AI
- Governs AI
- Career family
- AI Ethics, Governance and Quality Assurance
- Key tools
- Python (pandas), Stata, KoboToolbox +7
Without AI
The role today: Impact Assessment Specialist
The core responsibilities, as most job descriptions still list them.
- Design impact assessments of AI systems before deployment (ex ante) and after deployment (ex post), with clear assessment questions.
- Map affected stakeholders, including groups excluded by phone access, language or literacy.
- Develop theories of change for AI-enabled interventions, including assumptions about uptake, trust and accuracy.
- Assess distributional effects across gender, age, disability, location and income groups.
- Measure intended and unintended outcomes, such as changed service-seeking behaviour, shifts in frontline roles or over-reliance on AI advice.
- Combine system performance data, such as accuracy and error types, with field evidence on how people actually use the system.
- Assess economic effects, including costs, savings and who bears the cost of AI errors.
- Engage affected communities through participatory methods to capture their experience of AI-enabled services.
- Produce assessment reports with recommendations to scale, adapt, restrict or stop an AI system.
What AI changes
Traditional impact assessment examined the effects of policies, programmes and projects on people and the environment. When the intervention itself includes AI, the specialist must assess the system's accuracy and failure patterns alongside its social and economic effects, and look for harms that averages hide, such as exclusion of people without phones or who speak minority languages. The AI system becomes the object of assessment rather than a tool for doing it.
With AI
The role with AI: AI Impact Assessment Specialist
What the job looks like once AI is part of the daily work.
- Analyse AI system logs to assess reach and use patterns across user groups.
- Compare AI-driven decisions with human or earlier decisions to estimate inclusion and exclusion errors.
- Assess the accuracy of AI advice or classifications against expert review in the contexts where they are used.
- Identify unintended effects on staff roles, trust in services and reliance on automated advice.
- Use AI-assisted coding of user feedback, with human review, to identify reported harms and benefits.
- Review developer documentation, such as model cards, for intended use and known limitations.
- Feed assessment findings into AI risk registers and decisions on scaling or redesign.
A typical week
3 of these 5 tasks already involve AI.
- Designing a mixed-methods assessment of an SMS advisory chatbot used by 40,000 smallholder farmers in two languages.
- Analysing usage logs showing that women are 18% of chatbot users but 45% of registered farmers.
- AI-assisted task: Running focus groups with community health workers on how an AI triage tool has changed their referral decisions.
- AI-assisted task: Comparing exclusion errors of an AI-based targeting model with the previous community-based method in 30 villages.
- AI-assisted task: Drafting the stakeholder map for an ex ante assessment of AI-assisted beneficiary selection in a school feeding programme.
What stays human
- Deciding with affected communities which impacts matter most and for whom.
- Judging whether benefits for the majority justify harms or exclusion for a minority.
- Interpreting system data in light of local context, power relations and access barriers.
- Recommending whether an AI system should scale, change or stop.
Skills gap check · 2 minutes
Impact Assessment Specialist jobs are being redesigned around AI. Are your skills ready?
Employers increasingly expect M&E professionals to work confidently with AI. Answer 8 quick questions to see where you stand against the “AI Impact Assessment Specialist” role, and what to learn before the gap costs you your next opportunity.
- Your AI readiness level
- Your skills gap for this role
- What to learn first
Free · no sign-up · your answers stay private
AI in practice
Real examples of how people in this role use AI.
For an AI poverty-scoring model used to select cash transfer recipients, the specialist compares model decisions with a verification survey of 1,200 households and finds exclusion errors of 31% for female-headed households against 19% overall, and recommends a community appeals channel.
Using 18 months of RapidPro logs from a farmer advisory chatbot, the specialist analyses use by gender, district and language and finds that Tigrinya users abandon conversations twice as often, which prompts a translation quality review.
Agronomists rate 300 randomly sampled chatbot answers and judge 12% incorrect or unsafe for local conditions; the specialist then follows up with affected farmers to see whether the advice was acted on.
An AI assistant does first-pass coding of 2,500 complaints about an AI health triage line; the specialist reviews a 10% sample, refines the codes and reports delayed referrals as the main theme to the health ministry.
Career path
Where people usually come from, and where this role can lead.
Before
This role
Skills
Most important first within each group.
How well do you match? Tick the skills you already have.
Your ticks stay on this page. Nothing is saved or sent.
M&E methodology
Data and technical
Ethics, data protection and governance
Professional
Sector knowledge
What to learn first
In priority order for someone moving into this role.
- Machine learning fundamentals — Judging accuracy claims and failure patterns of the systems being assessed requires knowing how models are trained and tested.
- Evaluating AI systems — Expert-rated samples and error analysis show whether AI advice is good enough for the people relying on it.
- Gender and inclusion analysis — Many AI harms show up only in disaggregated data on who uses, benefits from or is excluded by a system.
- AI risk assessment — Assessment findings are most useful when they feed directly into risk ratings and safeguards.
- NIST AI Risk Management Framework — Its map and measure functions line up closely with impact questions about context and affected groups.
Software and tools
Tool names open the official website in a new tab. “Roles” shows other careers that use the tool.
AI assistants
Data analysis
Dashboards and BI
Data collection
Qualitative analysis
Feedback and community channels
Summary in the EvalCommunity framework: Assess the social, economic and developmental impacts of AI systems and interventions.
The AI-enhanced title is a proposed career title, not a standardized occupational classification.
Cart
LAND A JOB REFERRAL IN 2 WEEKS (NO ONLINE APPS!)
Sign Up & To Get My Free Referral Toolkit Now: