PTTMetro - PTT.br
The Brazilian Metropolitan
IXP Project
http://ptt.br/
LINX 78
21st August 2012
Eduardo Ascenço Reis <[email protected]>
Antonio Galvao de Rezende Filho <[email protected]>
PTT.br Engineering Team <[email protected]>
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
1
CGI.br – NIC.br - CEPTRO.br
CGI.br
The Brazilian Internet Steering Committee (CGI.br) purpose is for coordinating and integrating
all Internet service initiatives in Brazil, as well as promoting technical quality, innovation
and the dissemination of the services available. The CGI.br is comprised of members from the
government, the corporate sector, the third sector and the academic community, and as such
constitutes a unique Internet governance model for the effective participation of society in decisions
involving network implementation, management and use.
NIC.br
The Brazilian Network Information Center (NIC.br) is a non-profit civil entity that implements
the decisions and projects designed by CGI.br.
CEPTRO.br
The Center of Study and Research in Network Technology and Operations (CEPTRO.br) is
responsible for services and projects that aim to improve the quality of Internet in Brazil,
working more specifically with Internet infrastructure.
PTTMetro/PTT.br is part of CEPTRO.br activities.
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
2
Reference Name
IXP - Internet eXchange Point
PTT – Ponto de Troca de Tráfego
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
3
Brazilian PTT History and Internet Key Points
Year
Location (City/State)
Event
1989
São Paulo / SP
.br TLD Activation
1991
São Paulo / SP
Brazil First Connection to Internet via FAPESP (Dawn of Brazil Internet)
1996
São Paulo / SP
PTT ANSP/FAPESP Operation Start
1997
São Paulo / SP
Registro.br Operation Start (Brazil NIR)
2000
Porto Alegre / RS
PTT RSIX Operation Start (RNP UFRGS)
2000
São Paulo / SP
PTT OPTiX-LA Operation Start (Optiglobe Inc, now Tivit)
2002
Curitiba / PR
PTT PriX Operation Start (RNP UFPR)
2002
Brasilia / DF
PTT FIX Operation Start (RNP)
2004
São Paulo / SP
PTTMetro (CGI.br)
2004
Rio de Janeiro / RJ
PTTMetro (CGI.br)
2004
Brasilia / DF
PTT FIX move to PTTMetro (CGI.br)
2004
Barueri / SP
PTT ANSP/FAPESP moved to NAP do Brasil (Terremark Latin America)
2005
Curitiba / PR
PTT PRIX moved to PTTMetro (CGI.br)
From Peering in Brazil presentation @ LACNIC XI / Salvador, BA – Brazil - Interconnection Schemes and Strategies for ISPs Tutorial
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
4
Brazilian PTT History and Internet Key Points
Year
Location (City/State)
Event
1989
São Paulo / SP
.br TLD Activation
1991
São Paulo / SP
Brazil First Connection to Internet via FAPESP (Dawn of Brazil Internet)
1996
São Paulo / SP
PTT ANSP/FAPESP Operation Start
1997
São Paulo / SP
Registro.br Operation Start (Brazil NIR)
2000
Porto Alegre / RS
PTT RSIX Operation Start (RNP UFRGS)
2000
São Paulo / SP
PTT OPTiX-LA Operation Start (Optiglobe Inc, now Tivit)
2002
Curitiba / PR
PTT PriX Operation Start (RNP UFPR)
2002
Brasilia / DF
PTT FIX Operation Start (RNP)
2004
São Paulo / SP
PTTMetro (CGI.br)
2004
Rio de Janeiro / RJ
PTTMetro (CGI.br)
2004
Brasilia / DF
PTT FIX move to PTTMetro (CGI.br)
2004
Barueri / SP
PTT ANSP/FAPESP moved to NAP do Brasil (Terremark Latin America)
2005
Curitiba / PR
PTT PRIX moved to PTTMetro (CGI.br)
From Peering in Brazil presentation @ LACNIC XI / Salvador, BA – Brazil - Interconnection Schemes and Strategies for ISPs Tutorial
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
5
PTT Close Proliferation Problem (e.g. São Paulo city in the past) - 1/3
City Area
PTT
Optix-LA
PTT
ANSP/FAPEP
PTT
Diveo
PTT
Telcomp
Customer
AS
Too many Connections
High Cost
From Peering in Brazil presentation @ LACNIC XI / Salvador, BA – Brazil - Interconnection Schemes and Strategies for ISPs Tutorial
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
6
PTT Close Proliferation Problem (e.g. São Paulo city in the past) - 2/3
City Area
PTT
Optix-LA
PTT
ANSP/FAPEP
PTT
Diveo
PTT
Telcomp
?
Customer
AS
From Peering in Brazil presentation @ LACNIC XI / Salvador, BA – Brazil - Interconnection Schemes and Strategies for ISPs Tutorial
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
7
PTT Close Proliferation Problem (e.g. São Paulo city in the past) - 3/3
City Area
PTT
Optix-LA
PTT
ANSP/FAPEP
PTT
Diveo
PTT
Telcomp
Customer
AS
IP Transit
ISP
NSP
From Peering in Brazil presentation @ LACNIC XI / Salvador, BA – Brazil - Interconnection Schemes and Strategies for ISPs Tutorial
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
8
PTTMetro – Brazilian Metropolitan IXP Project
PTTMetro (PTT- Internet Exchange Point) is the project of the
Brazilian Internet Steering Committee (CGI.br) that provides the
necessary infrastructure for the direct interconnection between the
diverse networks that operate in a metropolitan region.
http://cgi.br/
http://www.cgi.br/internacional/
http://ptt.br/
From Peering in Brazil presentation @ LACNIC XI / Salvador, BA – Brazil - Interconnection Schemes and Strategies for ISPs Tutorial
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
9
PTTMetro – PTT.br – Project # Operation
PTTMetro
CGI.br project started in 2004 and is still active.
PTT.br
Operation division responsible to deploy and maintain PTTMetro locations.
PTT.br has a partnership to support its operation with entities that host PIX
on all locations. Not complete list of PIX's hosts types:
● National Education and Research Network (RNP)
● Academic (e.g. Universities)
● Information Technology Government Companies (e.g. Procempa, Prodest, Prodepa)
● Internet Data Centers (IDC)
● Internet Service Providers (ISP)
● Internet Network Providers (NSP)
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
10
PTTMetro/PTT.br – 2004 Start in 3 Locations
Initial Locations: Brasília, Rio de Janeiro and São Paulo
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
11
PTTMetro/PTT.br – August 2012 – 20 Locations in Operation
1. Americana
2. Belém
3. Belo Horizonte
4. Brasília
5. Caxias do Sul
6. Campina Grande
7. Campinas
8. Curitiba
9. Florianópolis
10. Fortaleza
11. Goiânia
12. Londrina
13. Natal
14. Porto Alegre
15. Recife
16. Rio de Janeiro
17. Salvador
18. São José dos Campos
19. São Paulo
20. Vitória
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
12
PTTMetro/PTT.br – IXP Locations and Brazilian AS Distribution
1. Americana
2. Belém
3. Belo Horizonte
4. Brasília
5. Caxias do Sul
6. Campina Grande
7. Campinas
8. Curitiba
9. Florianópolis
10. Fortaleza
11. Goiânia
12. Londrina
13. Natal
14. Porto Alegre
15. Recife
16. Rio de Janeiro
17. Salvador
18. São José dos Campos
19. São Paulo
20. Vitória
Brazilian Autonomous Systems (AS) ~1700
States: 26 + Federal District
Population: ~194 M
PTT.br IXP Locations
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
13
PTTMetro/PTT.br – IXP Locations
All PTTMetro Locations are Isolated
The main purpose for PTTMetro locations is to allow the local traffic to
stay in the location.
The Brazilian Internet Steering Committee do not intend to interconnect PTTMetro
Locations and compete with telecommunications companies.
PTTMetro project stimulates and support inter locations transport services done
by participants companies.
PTTMetro have defined some transport modes in order to optimize common
resources and reduce costs that are already in production
(e.g. Campinas <=> São Paulo by Fasternet company)
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
14
PTTMetro/PTT.br – Proliferation of IXPs Satellite Model
Small
IX
Small
IX
Small
IX
Local
IX
(e.g. Americana)
Medium
IX
Concentrator
IX
(e.g. Campinas)
Large
IX
National
IX
(e.g. São Paulo)
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
15
PTTMetro/PTT.br – Participants AS per Location
Sao Paulo
Campinas
Florianopolis
Sao Jose dos Campos
Belem
Porto Alegre
Rio de Janeiro
Goiania
Brasília
Vitoria
Curitiba
Belo Horizonte
Campina Grande
Americana
Natal
# Participants AS
Total:
Unique:
Salvador
Londrina
Fortaleza
Recife
Caxias do Sul
615
412
Values from Aug 2012
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
16
CGI.br – NIC.br – PTT.br - Financial Model
Brazilians
(persons or companies)
R$
Domains / IP / ASN
Registration
.br
Return Benefits
Help Improve Internet
Authoritative Name Systems
DNS
.br
NIC.br divisions / activities
Registro.br
CERT.br
Cetic.br
Ceptro.br
PTT.br / IPv6.br/ SIMET.nic.br / etc
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
17
PTTMetro/PTT.br - Financial Model HighLights
PTTMetro / PTT.br costs are sustained by CGI.br / NIC.br, despite
Autonomous Systems (AS) participants needs for ports (number and capacity).
Each participant AS has only costs to reach one or more Interconnection Points (PIX)
of a specific PTT.br location in order to access its peering fabric.
There is no startup and recurrent fees for AS participants at all PTT.br locations.
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
18
PTTMetro/PTT.br – Operational Responsibilities
NIC.br
PTT.br is responsible for:
Installation, Management, Administration, Monitoring and Supporting all IX locations.
Always keeping Neutrality and High Quality as main targets.
Highlight: PTT.br is responsible to provide enough capacity to sustain bandwidth demand
on connection between PIX: (e.g. 1GE, 10GE, nx 10GE with DWDM).
PIX - Interconnection Point
Each entity that hosts a PTT.br PIX is responsible to provide continuous and high quality:
● IDC Infrastructure
Rack Space, Electrical Energy, Refrigeration, Physical Security,
Operational Hand, etc
● Dark Fiber Pair (preferably redundant) to Central PIX
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
19
PTTMetro/PTT.br - Regional Metro Ethernet Non-Blocking Switch/Peering Fabric
AS A
Each PTT.br Location Infrastructure
Router
IXP
PIX A
AS D
Router
PIX D
PIX
Central
PIX B
Router
AS B
PIX C
Router
PIX: Interconnection Point
AS C
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
20
PTTMetro/PTT.br – High Availability Network Topology Model
PIX Central
C1
PIX Central
C2
Core
Access
PIX
Remote
PIX: Interconnection Point
Dark Fiber Pair
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
21
PTTMetro/PTT.br – São Paulo Location – PIX Topology
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
22
PTTMetro/PTT.br – São Paulo Location – PIX Distribution
Central / Core PIX
C1 in operation
Remote / Access PIX
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
23
PTTMetro/PTT.br – Peering Agreements
Multi-Lateral Peering Agreements (MLPA)
IP (v4/v6) traffic exchange between all participants.
BGP sessions established between participants and IXP route servers.
Different MLPA VLANs for IPv4 and IPv6.
Bilateral Peering Agreements (BPA)
Shared VLAN Mode
IP traffic exchange only between participants.
Use the same MLPA VLANs.
Dedicated VLAN Mode
IP traffic exchange only between participants (e.g. Internet Transit).
Use a dedicate VLAN for each peering agreement.
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
24
PTTMetro/PTT.br – Reference Model
Fundamental Definition
IP (v4|v6)
AS A
AS B
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
25
PTTMetro/PTT.br – Basic Operational Model – Allowed Traffic
AS A
MAC A
MAC B
AS B
IP (v4|v6)
PTT.br
Location
Peering Fabric
Allowed Frames with Ethertypes:
● 0x0800 - IPv4
● 0x0806 - ARP
● 0x86dd - IPv6
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
26
PTTMetro/PTT.br – Extended Operational Model – Inter AS Traffic
Allowed Multiple Connections to a Location for
Redundancy and/or Load Balancing
AS A
MAC A
MAC B
AS B
MAC C
AS C
PTT.br
Location
Peering Fabric
AS A
MAC A'
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
27
PTTMetro/PTT.br – Extended Operational Model – Intra AS Traffic
Denied Multiple Connections to a Location for Internal Traffic (local loop)
AS A
MAC A
MAC B
AS B
MAC C
AS C
PTT.br
Location
Peering Fabric
AS A
MAC A'
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
28
PTTMetro/PTT.br – Extended Operational Model – Multiple AS per Single Port
Shared L2 structure outside PTT.br operation domain
AS A
MAC A
NSP
Transport
Provider
AS D
PTT.br
Location
Peering Fabric
MLPA
MAC D
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
29
PTTMetro/PTT.br – Extended Operational Model – Multiple AS per Single Port
Use Different VLAN Tags for Logical Isolation (IEEE 802.1Q)
AS A
MAC A
VLAN
A
VLAN
A
NSP
Transport
Provider
PTT.br
Location
Peering Fabric
MLPA
VLAN C
VLAN
B
AS D
MAC D
VLAN
B
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
30
PTTMetro/PTT.br – Redundant Authoritative DNS Servers Service
Anycast Copies at 14 PTT.br IX Locations
Root:
.BR cctld:
L.root-servers.net
*.DNS.BR
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
31
PTTMetro/PTT.br – Economical Summary
PTTMetro - Economically Interests for Telecommunication Companies
●
●
●
Reduction of Transit and Interconnection Costs
Some Brazilian Access ISP reported to exchange around 30-40% of their
Internet traffic at Sao Paulo PTT.br.
Last/First Mile Provisioning
IP Services Interconnection
Using dedicated VLANs to provide L2 logical isolation for:
Internet Transit (IPv4 and IPv6), Backup, Storage, VoIP, etc
●
Inter Locations Transport
●
Interconnection Points (PIX) Hosting
●
Neutral Facilities for the Brazilian Telecommunications Agency Regulated Interconnections
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
32
PTTMetro/PTT.br – All Locations Aggregated Exchanged Traffic Growth
25000
20000
Terabytes
15000
10000
5000
0
Au
g/
20
07
Oc
t /2
00
7
De
c/2
00
7
Fe
b/
20
08
Ap
r/ 2
00
8
Ju
n/
20
08
Au
g/
20
08
Oc
t/2
00
8
De
c /2
00
Fe
b/
8
20
09
Ap
r/ 2
00
9
Ju
n/
20
09
Au
g/
20
09
Oc
t/2
00
9
De
c/2
00
9
Fe
b/
20
10
Ap
r/ 2
01
0
Ju
n
/2
01
0
Au
g/
20
10
Oc
t/2
01
0
De
c/2
01
0
Fe
b/
20
11
Ap
r
/2
01
1
Ju
n/
20
11
Au
g/
20
11
Oc
t/2
01
1
De
c/2
Month
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
33
01
1
Fe
b/
20
12
Ap
r/ 2
01
2
Ju
n/
20
12
PTTMetro/PTT.br – All Locations Aggregated Exchanged Traffic
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
34
PTTMetro/PTT.br – London 2012 Olympic Games Broadcast Traffic
London 2012 Internet Broadcast Transmission
Sample at Porto Alegre - RS PTT.br Location
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
35
PTTMetro/PTT.br – PTT Forum 6 – São Paulo, Brazil
http://ptt.br/pttforum/
The 6th PTT Fórum event will take place in São Paulo, Brazil on December 03 and 04,
2012.
PTT Fórum normal audience is composed by Brazilian Autonomous Systems (AS) and
PTTMetro/PTT.br IX participants (some of them are international companies).
Last year the 4th PTT Fórum had around 400 participants.
The event will be translated between Portuguese and English.
The 6th PTT Forum event will take place in São Paulo, Brazil on December 3 and 4,
2012 and it will be part of the Brazilian Internet Infrastructure week together with an
IPv6 Forum, ISOC ION Conference and GTER/GTS (Network Engineering, Operation
and Security Working Groups).
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
36
PTTMetro/PTT.br – Thanks & Contact
Thanks | Obrigado
http://ptt.br/
[email protected]
LINX 78 – 21st August 2012 – PTTMetro/PTT.br Engineering Team <[email protected]> - NIC.br - http://ptt.br/
37