Banco Interamericano
de Desenvolvimento
Management for Development Results
at Sub-national Governments
Monitoring and Evaluation of
Priority Programs
MAPP
Marcos C. Holanda
[email protected]
RBM - MAPP
RBM – Governance Strategy
MAPP – Operational Instrument
Banco Interamericano
de Desenvolvimento
It is important to
differentiate
Banco Interamericano
de Desenvolvimento
Planning Projects selection
Administration Projects execution
Management Systemic coordination of projects
based on results
What RBM is About
1.
2.
3.
4.
5.
6.
Banco Interamericano
de Desenvolvimento
Do less but do Better.
Outcomes prevail over outputs.
Outcomes put citizens first. Outputs put government first.
Outcomes as driving force to programs integration.
Outcomes to differentiate good from bad programs.
A permanent cycle of learning: Planning-Budgeting-MonitoringEvaluation.
7. An instrument for recognition and compensation of good
performance of government employees.
Banco Interamericano
de Desenvolvimento
GOVERNMENT PLAN
IMPLEMENTATION
DEZIGN
GOVERNMENT STRATEGIC OUTCOMES
DEPARTMENT STRATEGIC OUTCOMES
DEPARTMENT PROGRAMS
DEPARTMENT OUTPUTS
DEPARTMENT PROJECTS AND ACTIVITIES
Requirements for a
successful RBM
Banco Interamericano
de Desenvolvimento
 To have information – outcome and output indicators
 To know how to use the information – information
affect budget decisions
 To have a budget based on results – RBM defines
the budget and not the contrary
 To know what it is and how to do RBM –
Institutional and operational capacity to implement RBM
Guidelines for RBM
Banco Interamericano
de Desenvolvimento
1. Operational flexibility in searching for results.
2. Medium term budget.
3. Increments in the budget are based on increments in the results.
4. Look for “good or better practices” and not for “best practices”.
MAPP
Banco Interamericano
de Desenvolvimento
Monitoring
MAPP
Planning
Evaluation
MAPP
Banco Interamericano
de Desenvolvimento
1. PLANNING  Selection of Projects  Application Card
2. MONITORING  Outputs  MAPP Monitoring
Macro Evaluation
a) Government Outcomes
b) Integration Matrix
3. EVALUATION  Outcomes 
Micro Evaluation
a) Design Evaluation
b) Impact Evaluation
Banco Interamericano
de Desenvolvimento
PLANNING  Project Application Card Banco Interamericano
de Desenvolvimento
1. The problem to be faced.
2. The government outcomes to be targeted.
3. The department outcomes to feed the government ones.
4. The outputs to be delivered.
5. The outcomes and outputs indicators.
6. The targeted population or region.
7. The time frame of expected outcomes.
8. The sources of financing.
9. The expected efficiency (benefits-costs analysis).
10. The expected integration with others projects.
PLANNING  Project Application Card
Banco Interamericano
de Desenvolvimento
PLANNING  Project Application Card
Banco Interamericano
de Desenvolvimento
PLANNING  Project Application Card
Banco Interamericano
de Desenvolvimento
PLANNING  Project Application Card
Banco Interamericano
de Desenvolvimento
MONITORING  MAPP Monitoring
Banco Interamericano
de Desenvolvimento
MONITORING  MAPP Monitoramento
Banco Interamericano
de Desenvolvimento
MONITORING  MAPP Monitoramento
Banco Interamericano
de Desenvolvimento
MONITORING  MAPP Monitoramento
Banco Interamericano
de Desenvolvimento
EVALUATION  Macro Evaluation
Government Outcomes
Social Outcomes
Indicadores
Nota SAEB e ESPAECE
Distorção Idade Série
Taxa Escolar Líquida:
Ensino Fundamental
Taxa Escolar Líquida:
Ensino Médio
Taxa de Analfabetismo
(>15 anos)
Economic Outcomes
Sinalização
▬
▲
▼
▲
Taxa Mortalidade Infantil
▲
▬
▲
Razão Mortalidade Materna
▲
Taxa Internação AVC (> 40
anos)
Taxa de Roubos
▼
(de 7 a 14 anos)
Taxa de Furtos
Taxa de Homicídio
▼
▬
▼
Renda Domiciliar Per
Capita (R$)
Porcentagem Extrema
Pobreza
▲
Índice de GINI
▲
▲
Indicadores
Saldo de Empregos Formais
(Novos Postos)
Sinalização
▲
Banco Interamericano
de Desenvolvimento
Governance Outcomes
Indicadores
Sinalização
Div / PIB (%)
▲
Div / RCL (%)
▲
Taxa de Crescimento do PIB
(%)
▬
Resultado Primário
▲
Fluxo de Comércio Exterior
▲
Custeio / RCL (%)
▲
PIB Interior / PIB RMF (%)
▼
Pessoal / RCL (%)
▲
Gastos Finalísticos / RCL
(%)
▬
Investimento / RCL (%)
▼
Número de Acessos às
Páginas WEB do Governo
▲
Consumo de Energia
Residencial: Interior / RMF
(%)
▲
Empregos Interior /
Empregos RMF (%)
▬
EVALUATION  Macro Evaluation
Integration Matrix
Banco Interamericano
de Desenvolvimento
Programs Integration Matrix
Government Outcome # 1
Government
Outcome # 2
Government
Outcome # 3
Based
Budget
Economic Growth
Program
1
Program
2
Program
3
Program
4
Program
5
Program
6
Result
EVALUATION  Micro Evaluation
Design Evaluation
Banco Interamericano
de Desenvolvimento
Logical Framework
Inputs
Activities
Monitoring
Outputs
Outcomes
Outputs delivered
X
Planned Outcomes
Impacts
EVALUATION  Micro Evaluation
Design Evaluation
OUTCOMES
Banco Interamericano
de Desenvolvimento
OUTPUTS
a) Education:
. Primary education schooling rate
a) Education:
. Proportion of teachers in primary education with bachelor degrees
. Secondary education schooling rate
. 4th grade promotion rate.
. Proportion of teachers in secondary education with bachelor degrees
. Libraries, reading rooms and computer labs per public school; and
. Computer equipments per public school.
b) Health:
. Infant mortality rate
b) Health:
. Prop. of women assisted during the first 3 months of pregnancy
. Cerebral Vascular Accident (CVA) admittance rate
. Proportion of children (2 years old or younger) assisted
. Proportion of the population assisted by the Family Health Program
c) Living Conditions:
c) Living conditions:
. Urban water supply coverage rate
. Proportion of urban domiciles with regular water supply
. Proportion of urban domiciles with sewage
d) Employment and income:
. Average residential energy consumption
. Formal employment quality index
. Average establishments’ size
e) Rural Development:
.Gross value of the agricultural production per rural establishment
. Proportion of energy consumption in rural areas
. Urban sewage coverage rate
d) Employment and income:
. Secondary enrollment – total enrollment ratio
. Health professionals per 1,000 inhabitants
. Paved roads – municipality area ratio.
e) Rural development:
. Mean value of rural loans
. Assisted producers per rural establishment
EVALUATION  Micro Evaluation
Impact Evaluation
Banco Interamericano
de Desenvolvimento
Program Evaluation: Criança Fora da Rua Dentro da Escola
Probability (Recidivism) = F(age, gender, approach period, activity,
per capita family income, guardian
education, benefit)
Variables
Coeficient
dY/dX
P Value
Age
0.0136
0.0035
0.002
Gender
0.2493
0.0619
0.000
Approach period
0.1606
0.0441
0.001
Activity
0.2590
0.0664
0.000
Per Capita Family Income
-0.0050
-0.0013
0.000
Guardian Education
-0.1698
-0.0013
0.000
Benefit
-0.5001
-0.1344
0.000
Constant
-0.8609
-
0.000
8397
Pseudo R2
0.0639
Number of observations
Source: IPECE
LESSONS
Banco Interamericano
de Desenvolvimento
1. Strategic leadership(Governor, Mayor) and operational
leadership(line secretary) are crucial.
2. It is very Important a clear understanding of RBM concepts .
3. It is important to show that RBM is about opportunity and not
a threat.
4. The selection of output and outcome indicators can be very tricky.
LESSONS
Banco Interamericano
de Desenvolvimento
5. It is important to explain the difference between Monitoring and
Evaluation:
Monitoring  Good Accounting
Evaluation  Good Economics
6. Before impact evaluation, ask for design evaluation (quick
evaluation) for the main Program/Project.
7. Keep the RBM policy as simple as possible.
8. RBM requires 3 P’s ( a different kind of PPP):
Patience + Persistence + Passion
Banco Interamericano
de Desenvolvimento
Thanks!
[email protected]