Sensor Fusion
Applied to Soccer Robots
LEIC
27 de Fevereiro
de 2003
Prepared by: Pedro Marcelino
Oriented by: Prof. Pedro Lima
TFC “Sensor Fusion”– Pedro Marcelino
“Sensor Fusion”
Topics
LEIC
27 de Fevereiro
de 2003
•
•
•
•
•
•
•
•
Motivation
Sensors Caracteristics
Sensors as Members of a Team
Sensor Models
Observation Integration
Implemented Algoritms
Experimental Results
Conclusions
TFC “Sensor Fusion”– Pedro Marcelino
“Sensor Fusion”
Topics
LEIC
27 de Fevereiro
de 2003
•
•
•
•
•
•
•
•
Motivation
Sensors Caracteristics
Sensors as Members of a Team
Sensor Models
Observation Integration
Implemented Algoritms
Experimental Results
Conclusions
TFC “Sensor Fusion”– Pedro Marcelino
“Sensor Fusion”
Motivation
• Increased interest in the developing of multi-sensor robots
• Limitations in the reconstruction of environments
• Observation errors, bad calibrations or partial and incomplete
information of the world
• Cooperation to resolve ambiguities
• Robust and consistent description of the world
• Team with a common goal and shared knowledge, so it can take the
right decisions.
LEIC
27 de Fevereiro
de 2003
TFC “Sensor Fusion”– Pedro Marcelino
“Sensor Fusion”
Topics
LEIC
27 de Fevereiro
de 2003
•
•
•
•
•
•
•
•
Motivation
Sensors Caracteristics
Sensors as Members of a Team
Sensor Models
Observation Integration
Implemented Algoritms
Experimental Results
Conclusions
TFC “Sensor Fusion”– Pedro Marcelino
“Sensor Fusion”
Sensors Caracteristics
•
•
•
•
Sensor Complexity
Observation Error
Observation Disparity
Multiples Points of View
LEIC
27 de Fevereiro
de 2003
TFC “Sensor Fusion”– Pedro Marcelino
“Sensor Fusion”
Topics
LEIC
27 de Fevereiro
de 2003
•
•
•
•
•
•
•
•
Motivation
Sensors Caracteristics
Sensors as Members of a Team
Sensor Models
Observation Integration
Implemented Algoritms
Experimental Results
Conclusions
TFC “Sensor Fusion”– Pedro Marcelino
“Sensor Fusion”
Sensor as a Team Member
•
•
•
•
•
•
LEIC
27 de Fevereiro
de 2003
Multi-Sensorial System = Team of Sensors
Each sensor is considerer an individual
Each sensor make local decisions
Each sensor implements its actions
The Team coordinate the activity of its members
Information exchange to resolve conflits and validation
of observations
• Makes the Team Decision Problema a simple Estimation
Problem
TFC “Sensor Fusion”– Pedro Marcelino
“Sensor Fusion”
Topics
LEIC
27 de Fevereiro
de 2003
•
•
•
•
•
•
•
•
Motivation
Sensors Caracteristics
Sensors as Members of a Team
Sensor Models
Observation Integration
Implemented Algoritms
Experimental Results
Conclusions
TFC “Sensor Fusion”– Pedro Marcelino
“Sensor Fusion”
Sensor Models
• Observation Model
 It is a static description of the sensor performance,
realting the observation with the state of teh
environment
 Front Camera Model
C
 Up Camera Model
CL
LEIC
27 de Fevereiro
de 2003
TFC “Sensor Fusion”– Pedro Marcelino
“Sensor Fusion”
Front Camera Observation Model
Variância (m)
Modelo da Câmara da Frente
1.4
1.2
1
0.8
0.6
0.4
0.2
0
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
Distância (m)
Modelo da Câmara da Frente XX
Modelo da Câmara da Frente YY
LEIC
27 de Fevereiro
de 2003
TFC “Sensor Fusion”– Pedro Marcelino
“Sensor Fusion”
Up Camera Observation Model
Modelo da Câmara da Cima
Variância (m)
0.25
0.2
0.15
0.1
0.05
0
0
0.5
1
1.5
2
2.5
3
3.5
Distância (m)
Modelo da Câmara da Cima XX
Modelo da Câmara da Cima YY
LEIC
27 de Fevereiro
de 2003
TFC “Sensor Fusion”– Pedro Marcelino
“Sensor Fusion”
Sensor Models
• State Model
 Relates the observation of a sensor with a given
location and its internal state
 Perspective change to a common frame so that the
observation can be compared
LEIC
27 de Fevereiro
de 2003
TFC “Sensor Fusion”– Pedro Marcelino
“Sensor Fusion”
State Model
• Each feature is represented as
with a gauss distribuition
• Mean
• Variance
• Angle with central axis
• Distance to feature
• New variance results from the
perspective transformation to
a global frame
LEIC
27 de Fevereiro
de 2003
TFC “Sensor Fusion”– Pedro Marcelino
“Sensor Fusion”
Sensor Models
• Dependency Model
 Describe sthe relation between the observations and
the actions of each sensor
 Team Utility Function
 Team Decision Fucntion
 Groups Rational Aximos
 Each member makes a decision that maximizes its
Team Utility Function
LEIC
27 de Fevereiro
de 2003
TFC “Sensor Fusion”– Pedro Marcelino
“Sensor Fusion”
Topics
LEIC
27 de Fevereiro
de 2003
•
•
•
•
•
•
•
•
Motivation
Sensors Caracteristics
Sensors as Members of a Team
Sensor Models
Observation Integration
Implemented Algoritms
Experimental Results
Conclusions
TFC “Sensor Fusion”– Pedro Marcelino
“Sensor Fusion”
Observation Integration
• Each feature is modeled by a gauss distribution, using
Bayes Law
• If the Mahalanobis distance is less than 1, then there is
agreement and the team member will cooperate, to
estimate the feature position, otherwise, there is
desagreement and the team member observation will not
be used
LEIC
27 de Fevereiro
de 2003
TFC “Sensor Fusion”– Pedro Marcelino
“Sensor Fusion”
Observation Integration
• Two bayes observers showing agreement
LEIC
27 de Fevereiro
de 2003
TFC “Sensor Fusion”– Pedro Marcelino
“Sensor Fusion”
Observation Integration
• Two bayes observers showing desagreemnet
LEIC
27 de Fevereiro
de 2003
TFC “Sensor Fusion”– Pedro Marcelino
“Sensor Fusion”
Topics
LEIC
27 de Fevereiro
de 2003
•
•
•
•
•
•
•
•
Motivation
Sensors Caracteristics
Sensors as Members of a Team
Sensor Models
Observation Integration
Implemented Algoritms
Experimental Results
Conclusions
TFC “Sensor Fusion”– Pedro Marcelino
“Sensor Fusion”
Implemented Algoritms – Ball Detection
• Ball detection in front
Camera
LEIC
27 de Fevereiro
de 2003
TFC “Sensor Fusion”– Pedro Marcelino
“Sensor Fusion”
Implemented Algoritms – Ball Detection
• Ball detection un Up
Camera
LEIC
27 de Fevereiro
de 2003
TFC “Sensor Fusion”– Pedro Marcelino
“Sensor Fusion”
Camera Models
Comparação das Posições Observadas pelas
Câmaras e a Posição Real da Bola
ao longo de uma Linha Recta
Eixo YY (m)
0.15
0.05
-0.05 0
0.5
1
1.5
2
2.5
3
3.5
4
-0.15
Eixo XX (m)
Posição Câmara Cima
Posição Câmara Frente
Comparação dos Erros de Leitura segundo Eixo
XX ao longo de uma Linha Recta
Comparação dos Erros de Leitura segundo Eixo
YY ao longo de uma Linha Recta
LEIC
0.6
Erro (m)
Erro (m)
0.8
0.4
0.2
0
0.25 0.5 0.75
1
1.25 1.5 1.75
2.25 2.5 2.75
3
3.25
0.12
0.1
0.08
0.06
0.04
0.02
0
0.25 0.5 0.75
Distância ao Robot (m)
Erro Câmara Cima XX
27 de Fevereiro
de 2003
2
Posição Real
Erro Câmara Frente XX
1
1.25 1.5 1.75
2
2.25 2.5 2.75
Distância ao Robot (m)
Erro Câmara Cima YY
TFC “Sensor Fusion”– Pedro Marcelino
Erro Câmara Frente YY
3
3.25
“Sensor Fusion”
Sensor Models Diagram
Observation of Sensor 1 Observation of Sensor 2
Observation
Model
Observation
Model
Sensor Model
State
Model
State
Model
Change of perspective to
Global Frame
Dependency
Model
Decision and Integration
of Observation
Team Utility
Function
Structure that keeps all
decisions made by the
team members
New Fusion Validation
Variance Increase with
Time
LEIC
27 de Fevereiro
de 2003
TFC “Sensor Fusion”– Pedro Marcelino
“Sensor Fusion”
Topics
LEIC
27 de Fevereiro
de 2003
•
•
•
•
•
•
•
•
Motivation
Sensors Caracteristics
Sensors as Members of a Team
Sensor Models
Observation Integration
Implemented Algoritms
Experimental Results
Conclusions
TFC “Sensor Fusion”– Pedro Marcelino
“Sensor Fusion”
Experimental Results
Comparação da Trajectória da Bola com a Fusão e
dados Observados ao longo de uma Linha Recta
Eixo YY (m)
0.15
0.05
-0.05 0
0.5
1
1.5
2
2.5
3
3.5
4
-0.15
Eixo XX (m)
Posição Câmara Cima
Posição Câmara Frente
Posição Real
Posição Fusão
Comparação dos Erros segundo Eixo XX ao longo
de uma Linha Recta
Comparação dos Erros segundo Eixo YY ao longo
de uma Linha Recta
LEIC
0.6
Erro (m)
Erro (m)
0.8
0.4
0.2
0
0.25 0.5 0.75
1
1.25 1.5 1.75
2
2.25 2.5 2.75
3
3.25
0.12
0.1
0.08
0.06
0.04
0.02
0
0.25 0.5 0.75
Distância ao Robot (m)
Erro Câmara Cima XX
27 de Fevereiro
de 2003
Erro Câmara Frente XX
1
1.25 1.5 1.75
2
2.25 2.5 2.75
3
3.25
Distância ao Robot (m)
Erro Fusão XX
Erro Câmara Cima YY
TFC “Sensor Fusion”– Pedro Marcelino
Erro Câmara Frente YY
Erro Fusão YY
“Sensor Fusion”
Topics
LEIC
27 de Fevereiro
de 2003
•
•
•
•
•
•
•
•
Motivation
Sensors Caracteristics
Sensors as Members of a Team
Sensor Models
Observation Integration
Implemented Algoritms
Experimental Results
Conclusions
TFC “Sensor Fusion”– Pedro Marcelino
“Sensor Fusion”
Conclusions
•
•
•
•
•
•
Real time fusion of the world information
Good estimative of features localization
Makes system more robust, eliminating sporadic errors
Coerent World decription
Use of Bayes Teorem to solve the decision problem
It is a really good method to be used in modern robotics, which
should be used whenever possible to determine the position and
orientation of the environment features that surrond the robot
LEIC
27 de Fevereiro
de 2003
TFC “Sensor Fusion”– Pedro Marcelino
“Sensor Fusion”
Future Work
To be developed during the Master
• Sensor Fusion of several robots
• Other players detection
• Team players detection
• Sensor Fusion to determine robot position
LEIC
27 de Fevereiro
de 2003
TFC “Sensor Fusion”– Pedro Marcelino
“Sensor Fusion”
Sensor Fusion Diagram
BlackBoard
Dependency Model
global.worldmodel.*
World Model
Local Sensor Fusion
Algoritm of Other Robots
Global Sensor Fusion Algoritm
Dependency Model
Local Sensor Fusion Algoritm
BlackBoard
local.up.*
local.front.*
local.sonars.*
local.odometry.*
Observation and
State Model
Up Camera
Front Camera
Sonars
Odometry
Sensors
Up Camera
Front Camera
Sonars
Odometry
LEIC
27 de Fevereiro
de 2003
TFC “Sensor Fusion”– Pedro Marcelino
Team Members
LEIC
27 de Fevereiro
de 2003
• Docentes do IST:
• Pedro Lima (coordenação) - DEEC
• Luis Custódio (coordenação) - DEEC
• Carlos Pinto Ferreira (professor associado) - DEM
• Alunos de Doutoramento (EEC):
• Miguel Garção
• Alunos Finalistas (TFC):
• Bruno Damas - LEEC
• Pedro Pinheiro - LEIC
• Hugo Costelha - LEEC
• Gonçalo Neto - LEEC
• Cláudio Gil – LEIC
• Miguel Arroz – LEIC
• Bruno – LEIC
TFC “Sensor Fusion”– Pedro Marcelino