Monitoring and Evaluation
of Social Programs
Kathy Lindert, World Bank
Qualidade do Gasto Publico no Brasil
June 26-27, 2003
1
Outline
Objectives of M&E
Conceptual Framework, example
Monitoring (role, steps, etc.)
Evaluation (role, steps, etc.)
Dissemination and Uses
Institutional Issues and Sustainability
2
Objectives of M&E
Monitoring and evaluation can help policy makers
in many ways:

As planning and budgeting tool: to ensure effective allocations of
government resources – planning, decision-making and
prioritization

As a management tool by revealing the performance of ongoing
activities at the sector, program or project level, to inform decisions
about expansion, modification, or elimination of programs
depending on their effectiveness

As an accountability mechanism, so that managers can be held
accountable for the performance of their projects/programs and so
that government can be held accountable for its performance

To obtain and maintain political support, institutional resilience
(even across administrations), and financing for successful
activities
3
Implementação
Objetivos
Marco Conceitual
•
Longo prazo, ampla
melhoria na sociedade
Resultados
•
Efeitos intermediários
dos produtos sobre os
clientes
Produtos
•
Produtos e serviços
produzidos
Metas
(Impactos)
• Tarefas empreendidos
Atividades
Insumos
para transformar insumos
em produtos
• Financeiros, humanos, e
recursos materiais
4
Example: Secondary Education
Goal
(Impacts)
• Higher income levels;
increase access to higher
skill jobs
Outcomes
•
Increased skills; more
employment opportunities
Outputs
•
Number of youths completing
secondary school
Activities
Inputs
• Schooling
•
$, facilities, teachers,
materials
5
Monitoring
6
Monitoring of Implementation and Results
Monitoring is a continuous process of collecting and
analyzing information to compare how well a program
is being implemented against expected results
Monitoring tracks both implementation and results
Enfoque tradicional: Monitoring of activities


Definir atividades e responsabilidades
Monitoring activities is important
No entanto:

É possível executar as atividades, cumprir as
responsibilildades, mas não fazer progresso até as metas
Question: how will you know when you have been
successful?
“Se não se reconhecer o sucesso, vai se valorizar o
fracasso”
7
Results
Main Types of Monitoring
Impact
Results Monitoring
Outcome
Implementation
Output
Activity
Implementation Monitoring
(enfoque tradicional)
Input
8
Key Steps for Monitoring
Acordo sobre os resultados a monitorar
Seleção de indicadores de estos
resultados
Medição de “baseline”
Eleição de metas quantitativas
Gerenciamento com relação às metas
9
1. Acordo sobre os resultados a
monitorar
Outcomes make explicit the intended objectives of
governmental action – the vision
Consensus about these outcomes, and this vision, is
important


Participation in agreeing on outcomes is key
Key stakeholders should be involved (e.g., government, civil society,
donors, etc.)
Desenhar para sucesso

Evitar resultados impossíveis
Motivar pessoal com uma visão inspiradora

Criar linha de vista (“line of sight”) para todos que trabalham no
sistema entre as suas ações e o resultado desejado
“Se você não sabe para onde vai, todos os caminhos estão
certos”
10
2. Seleção de indicadores
Outcome indicators are not the same as outcomes
Indicadores devem medir resultados mas tambem
processos e productos
Um resultado corresponderá típicamente a mais
de um indicador
Respondem à pergunta:
“Cómo conheceremos o sucesso, se o vermos?”
11
2. Seleção de indicadores,
cont.
A good performance indicator must be:
Clear
(Precise and unambiguous)
Relevant
(Appropriate to subject at hand)
Economic
(Available at reasonable cost)
Adequate
(Must provide a sufficient basis to assess
performance)
Monitorable (Must be amenable to independent validation)
12
3. Establishing Baseline Data on
Indicators
A performance baseline is…
Information (quantitative or
qualitative) that provides data at
the beginning of, or just prior to,
the monitoring period. The
baseline is used to:


Learn about recent levels and
patterns of performance on the
indicator; and to
Gauge subsequent policy, program,
or project performance
13
Data Sources
May Be Primary or Secondary
PRIMARY data are collected directly by the
organization, for example, through surveys,
direct observation, and interviews.
SECONDARY data have been collected by
someone else, initially for a purpose other
than that of the program. Examples include
survey data collected by another agency, a
Demographic Health Survey, or data from a
financial market.

Secondary data often can save money in
acquiring data, but be careful!
14
Dados de “Baseline”
Pesquisas
de painel
Entrevistas com
interlocutores
chaves
Conversas
com
pessoas
afetadas
“Grupos
de foco”
Análises
dos
registros
officiais
Observação de
participantes
Visitas
de
campo
Pesquisas
únicas
Observação
direta
Census
Entrevistas
das
comunidades
Métodos informais/menos estruturados
Experiências
controladas
Questionários
Métodos formais/mais estruturados
15
Practicality
Are the data associated with the
indicator practical?
Ask whether…
Quality data are currently available
 The data can be procured on a regular and
timely basis
 Primary data collection, when necessary, is
feasible and cost-effective

16
4. Metas
Targets are the quantifiable levels of the indicators
that a country wants to achieve at a given point in
time
Precondições


Clara compreensão do passado recente
Conhecimento dos recursos disponíveis
Considerações políticas


Talvez o mais importante sendo o custo de não cumprir
a meta…
Gerenciamento para sucesso



Seja realista!
Faixas?
Nota: gestão por resultados não depende críticamente das
metas
17
5. Monitoria e gestão por
Metas
“Abertura” faz parte do enfoque

Desenvolver estratégias de divulgação para o
público, internet, etc.
Monitoramento deve ser contínuo
Informações devem correr “verticalmente” e
“horizontalmente”
A nenhum ponto do sistema devem ser
coletados dados sem ser analisados

Cria incentivos em favor da qualidade dos dados
18
5. Monitoria
Fidelidade
Validade
Medida clara e direta
do desempenho relevante
Metodologia estável
e consistente
Pontualidade
Dados atualizados e
suficientemente frenqüente
para influenciar decisões gerenciais
19
5. Monitoria: Freqüência
Quanto mais observações tem, tanto
mais os diagnósticos serão
Seguros
 Sofisticados
?
Tempo
Melhorar acesso
aos mercados
Accesso
Accesso

Tempo
Melhorar acesso
aos mercados
20
Evaluation
21
O papel das avaliações
Results-Based Evaluation
• An assessment of a planned, ongoing, or completed
intervention to determine its relevance, efficiency,
effectiveness, impact and sustainability. The intent is to
incorporate lessons learned into the decision-making
process.
Evaluation Addresses
“Why” Questions: What caused the changes we are monitoring
“How” Questions: What was the sequence or processes that led
to successful (or not) outcomes
“Compliance/ Accountability Questions”: Did the promised
activities actually take place and as they were planned?
22
Uses of Evaluation
To make resource decisions
To re-think about the causes of a problem
To identify issues around an emerging
problem, i.e. children dropping out of
school
To identify alternatives
To build support of public sector reform /
innovation
To help build consensus among
stakeholders on how to respond to a
problem
23
Evaluation Means Information on:
Strategy Operation
Whether we are doing the right things
Rationale/justification
 Client satisfaction

Whether we are doing things right


Effectiveness in achieving expected
outcomes
Efficiency in optimizing resources
Learning
Whether there are better ways of
doing it
Alternatives
 Best practices
 Lessons learned

24
General Types of Evaluations
& PERs
Evaluations can occur at many levels:




Evaluations of individual staff performance
(including project managers)
Evaluations of organizational units
Evaluations of programs (focus of this
presentation)
Evaluations of sectors/policy areas (e.g., crosssectoral issues, fiscal policy, overall education
policy)
Public expenditure reviews generally evaluate
the broadest level (evaluations of
sectors/policy areas), but they also draw on
evaluations of programs
25
Types of Program Evaluations
Process Implementation Evaluations:




Provides detailed information on whether program
is operating as intended (are we doing things
right?)
Provides continuous feedback to assist managers
Provides detailed information to those interested
in scaling up or replicating
Identifies bottlenecks
Cost-Benefit Evaluations:


Calculates the cost of the intervention relative to
results
Are we doing things efficiently?
26
Types of Program Evaluations, cont.
Impact Evaluations:




Measure net effect of program to determine whether or
not objectives are being achieved
Tries to establish causality between activities and
impacts
Impacts include both those that were intended and
those that were not
Impact evaluations require a clearly established
counter-factual:


What would have happened without the project?
Generally need to compare those “with” the project
(beneficiaries, “treatment group”) and those “without” the
project (“control group”) both before and after project
27
Stages of Evaluation
Clarify objectives of evaluation
Explore available information
Design evaluation
Form evaluation team (capacity?)
Collect information (sample, questionnaire, pilot,
data collection, entry, quality control, etc.)
Analysis
Reporting, disseminating, discussing results
Using results: feedback to policies, budgets,
planning
28
Dissemination and Uses of
Information
29
Divulgação
Informar sobre o estado dos projetos
Prover índices sobre problemas
Explicar problemas
Criar oportunidades para considerar
melhoras o alternativas
Prover informação sobre tendências
30
Usando as Conclusões
As a planning and budgeting tool:


Formular/justificar orçamentos
Ajudar alocação operacional de recursos
As a management tool:

Melhorar os serviços
As a performance/accountability tool:




Responder às demandas para “accountability”
Motivar pessoal
Monitorar fornecedores
Provocar debate sobre problemas
Reforçar confiança pública
31
Institutional Issues and
Sustainability
Demand for M&E is crucial


Structural requirements (ex. national law that mandates M&E of
all federal programs)
Budgetary and planning links
Papéis claros

Papel formal para organismos de planejamento e finanças
Credibilidade


Sistema deve entregar boas e más notícias!
Proteção dos gerentes contra conseqüências gravas
Responsabilidade (“Accountability”)

Papéis da sociedade civil, mídia…
Capacidade técnica
32