###############################
External Network Considerations
###############################
.. _networking-external_allocations:
Allocations
===========
We've got a lot of IP addresses and ranges allocated to us by various parties.
Here's an attempt to keep track of them all.
+----------------------+-----------------+--------------------------------+--------------+
| Address/block | Where | What | From Whom |
+======================+=================+================================+==============+
| 128.220.70.0/24 | Malone VLAN 13 | Cluster JHU Internal ("oldcs") | JHU IT |
+----------------------+-----------------+--------------------------------+--------------+
| 128.220.251.32/29 | Malone DMZ | Cluster JHU DMZ ("ff") | JHU IT |
+----------------------+-----------------+--------------------------------+--------------+
| 128.220.35.176/28 | Malone VLAN 35 | CS public subnet | CS |
+----------------------+-----------------+--------------------------------+--------------+
| 10.161.159.216/29 | Malone VLAN 159 | CS private subnet | CS |
+----------------------+-----------------+--------------------------------+--------------+
| 2606:2B00:0:410::/64 | ??? | JHU IPv6 network | JHU IT |
+----------------------+-----------------+--------------------------------+--------------+
Internally, the subnets from JHU IT are allocated to:
+-------------------+----------------------------+---------------------------------+
| Address/block | Controller | What |
+===================+============================+=================================+
| 128.220.70.0/25 | Various | ACM servies and physical hosts |
+-------------------+----------------------------+---------------------------------+
| 128.220.70.128/26 | Gomes via Openstack | ACM virtual machines |
+-------------------+----------------------------+---------------------------------+
| 128.220.70.192/26 | Gomes via OpenStack | User virtual machines |
+-------------------+----------------------------+---------------------------------+
| 128.220.251.32/29 | Magellan, Gomes | See tables below |
+-------------------+----------------------------+---------------------------------+
.. note :: As of this writing, 251.38 is unallocated.
Security Policies
-----------------
Network security manages the JHU border gateway policy for ``128.220.70.0/24``
and requires us to have a clean-slate report to their scanning tools for
external access to be granted. Contact network.security@jhu.edu to get the
policy adjusted, but please try to keep the tables below up-to-date, too!
This means, among other things, that we are obligated to not attempt
IP-address-based restrictions that would keep the following IP addresses from
probing our systems: ``10.181.169.162``, ``10.181.169.163``,
``10.181.169.164``, ``10.15.69.217``, ``10.15.69.218``, ``10.15.69.219``,
``128.220.242.60``, ``10.131.228.26``, and ``10.132.160.55``.
Thankfully, for the most part, public services offered by our cluster are not
restricted by IP address anyway.
Naming
------
Details of how subnet DNS entries are managed can be found in :ref:`dns_external`.
DHCP or other Dynamic Configuration
-----------------------------------
Our allocations from CS can be managed by CS's DHCP server; for somewhat
obvious reasons they don't want us running our own on their network. To adjust
the MAC/IP map, send mail to ``support@cs``.
Our direct allocations are manually managed and do not use dynamic
configuration.
Cluster Common Considerations
=============================
The cluster is a big mess internally that gets services exposed on a handful
of IP addresses, both inside the JHU firewall and outside and has somewhat
interesting egress rules. This page attempts to document the thinking
behind some of our port maps, but is *non-authoritative* (the authority, of
course, is what is configured on the cluster gateway).
For the moment, we use ``shorewall`` to manage our network configuration.
.. _enet-egress:
Multi-Provider Egress and Tracking
----------------------------------
We have two providers configured, in ``/etc/shorewall/providers``::
csprov 1 0x1000 main $NET_IF_CS 128.220.70.1 track $NET_IFS_INTERNAL
ffprov 2 0x2000 main $NET_IF_FF 128.220.251.33 track $NET_IFS_INTERNAL
The ``track`` directive ensures that we route responses back out the interface
on which things arrived. ``/etc/shorewall/rtrules`` describes the egress
rules. This file differs between Magellan and Gomes, but in rough schematic::
#SOURCE DEST PROVIDER PRIORITY
- 10.0.0.0/8 csprov 26000
$NET_CIDR_OS_A_CS - csprov 26000
$NET_CIDR_OS_A_FF - ffprov 26000
$NET_CIDR_OS_U_CS - csprov 26000
These rules ensure that, unless otherwise indicated by the ingress-attached
tracking labels, that outbound traffic to JHU-internal RFC1918 addresses
egress via the behind-firewall interface. The ``$NET_CIDR_OS`` lines
dictate how egress from our OpenStack VMs is routed -- the ``_A_`` regions
are for VMs under administrative control while ``_U_`` are for VMs running
user code.
The contents of ``/etc/shorewall/masq`` follow along. Again, this file differs
between Magellan and Gomes, but in rough sketch::
$NET_CS_IF $NET_OS_A_CS_CIDR
$NET_FF_IF $NET_OS_A_FF_CIDR
$NET_CS_IF $NET_OS_U_CS_CIDR $AEOLUS_CS_EXT
.. _enet-ingress:
Ingress
-------
It will probably be clearer to present the contents of
``/etc/shorewall/rules`` in a tabular form:
Magellan
````````
+-------------+-----------+------------+-----------------------------------------------+
| IP Address | Port | JHU Public | Description |
+=============+===========+============+===============================================+
| .70.63 | | | |
| (magellan) | | | |
| +-----------+------------+-----------------------------------------------+
| | TCP | No | |
| | 22 | | Magellan itself listening on SSH |
| +-----------+------------+-----------------------------------------------+
| | UDP | Yes | Alt. address for 128.220.251.36 file server |
| | 7000,7005 | | |
+-------------+-----------+------------+-----------------------------------------------+
| .70.64 | | | |
| (magellan2) | | | |
| +-----------+------------+-----------------------------------------------+
| | UDP | Yes | Alt. address for 128.220.251.35 file server |
| | 7000,7005 | | |
+-------------+-----------+------------+-----------------------------------------------+
| .251.34 | | DMZ | |
| (seattle) | | | |
| +-----------+------------+-----------------------------------------------+
| | | | (User firewall free egress address: |
| | | | :ref:`enet-egress` |
| +-----------+------------+-----------------------------------------------+
| | UDP | | |
| | 7000,7005 | | AFS scratch and mirror server |
+-------------+-----------+------------+-----------------------------------------------+
| .251.36 | | DMZ | |
| (magellan) | | | |
| +-----------+------------+-----------------------------------------------+
| | TCP | | |
| | 22 | | Magellan itself listening on SSH |
| +-----------+------------+-----------------------------------------------+
| | UDP | | |
| | 7000,7005 | | AFS homedirs and services server |
+-------------+-----------+------------+-----------------------------------------------+
Gomes
`````
+-------------+-----------+------------+-----------------------------------------------+
| IP Address | Port | JHU Public | Description |
+=============+===========+============+===============================================+
| .70.55 | | | |
| (astrolabe) | | | |
| +-----------+------------+-----------------------------------------------+
| | TCP | Yes | |
| | 80, 443 | | Mirrors web server |
+-------------+-----------+------------+-----------------------------------------------+
| .70.65 | | | |
| (centaur) | | | |
| [enet-acm]_ | | | |
| +-----------+------------+-----------------------------------------------+
| | TCP | Yes | |
| | 22 | | All-users SSH server (conch) |
| +-----------+------------+-----------------------------------------------+
| | TCP | Yes | |
| | 80, 443 | | User web server (web.vm) |
| +-----------+------------+-----------------------------------------------+
| | TCP 25, | Yes | |
| | 465, 587 | | ACM mail service (centaur.vm) |
+-------------+-----------+------------+-----------------------------------------------+
| .70.74 | | | |
| (nagios) | | | |
| +-----------+------------+-----------------------------------------------+
| | TCP 80 | Yes | Nagios worker machine (bigbrother.trinidad) |
| +-----------+------------+-----------------------------------------------+
| | ICMP | | Nagios worker machine (bigbrother.trinidad) |
+-------------+-----------+------------+-----------------------------------------------+
| .70.79 | | | |
| | | | |
+-------------+-----------+------------+-----------------------------------------------+
| .70.82 | | | |
| (belthazar) | | | |
| +-----------+------------+-----------------------------------------------+
| | TCP | Yes | |
| | 22 | | :doc:`../../services/egg` |
+ +-----------+------------+-----------------------------------------------+
| | TCP | Yes | |
| | 80,443 | | Mailman web interface (lists.acm.jhu.edu) |
+ +-----------+------------+-----------------------------------------------+
| | | | (User firewalled egress address: |
| | | | :ref:`enet-egress` |
+-------------+-----------+------------+-----------------------------------------------+
| .70.84 | | | |
| | | | |
+-------------+-----------+------------+-----------------------------------------------+
| .70.90 | | | |
| | | | |
+-------------+-----------+------------+-----------------------------------------------+
| .70.91 | | | |
| | | | |
+-------------+-----------+------------+-----------------------------------------------+
| .251.35 | | DMZ | |
| (batman) | | | |
| +-----------+------------+-----------------------------------------------+
| | TCP 22 | | All-users SSH server (conch.ff.uvm) |
| +-----------+------------+-----------------------------------------------+
| | TCP | | |
| | 4242 | | Quassel IRC agent (quassel.vm) |
| +-----------+------------+-----------------------------------------------+
| | TCP | | |
| | 6080 | | Sandstorm alias |
+-------------+-----------+------------+-----------------------------------------------+
| .251.37 | | DMZ | |
| (london) | | | |
+-------------+-----------+------------+-----------------------------------------------+
.. [enet-acm] For historical reasons, we have an A record in DNS for our
domain. This IP address should probably have the "canonically ACM"
things listening on it. At present, this address is *inside the JHU
firewall*.
.. todo:: It might be nice to have this table generated automatically
from the contents of the various rules files, actually. No?
Services Without the Cluster
============================
For the sake of eliminating SPOFs on critical services, the following
services are run on hosts entirely outside the cluster gateway. So even if
everything falls over, authentication and name resolution should continue to
function.
+-------------+-----------+------------+----------------------------------+
| IP Address | Port | JHU Public | Description |
+=============+===========+============+==================================+
| .70.76 | | | |
| (typhon) | | | |
| | | | |
| +-----------+------------+----------------------------------+
| | TCP | Yes | SSH |
| | 22 | | |
| +-----------+------------+----------------------------------+
| | TCP | Yes | :doc:`dns` |
| | 53 | | |
| +-----------+------------+----------------------------------+
| | TCP | Yes | :doc:`../auth/ldap` |
| | 389 | | |
| +-----------+------------+----------------------------------+
| | UDP | Yes | :doc:`dns` |
| | 53 | | |
| +-----------+------------+----------------------------------+
| | UDP | Yes | :doc:`../auth/kdc` |
| | 88 | | |
| +-----------+------------+----------------------------------+
| | UDP | Yes | AFS Fileserver (esp. replicas) |
| | 7000 | | |
| +-----------+------------+----------------------------------+
| | UDP | Yes | AFS DBs |
| | 7002,7003 | | |
| +-----------+------------+----------------------------------+
| | UDP | No | AFS VolSer |
| | 7005 | | |
| +-----------+------------+----------------------------------+
| | UDP | No | AFS BosServer |
| | 7007 | | |
+-------------+-----------+------------+----------------------------------+
| .70.53 | | | Mail server |
| (crimea) | | | |
| | | | |
| +-----------+------------+----------------------------------+
| | TCP | Yes | SSH |
| | 22 | | |
| +-----------+------------+----------------------------------+
| | TCP | Yes | Mail ingress |
| | 25 | | |
| +-----------+------------+----------------------------------+
| | TCP | Yes | Mailing list web interface |
| | 80/443 | | |
| +-----------+------------+----------------------------------+
| | UDP | Yes | AFS Fileserver reservation |
| | 7000 | | Crimea is not an AFS server now! |
+-------------+-----------+------------+----------------------------------+
| .35.178 | | | (Services here are replicas from |
| (echidna) | | | Typhon) |
| | | | |
| +-----------+------------+----------------------------------+
| | TCP | Yes | SSH |
| | 22 | | |
| +-----------+------------+----------------------------------+
| | TCP | Yes | :doc:`dns` |
| | 53 | | |
| +-----------+------------+----------------------------------+
| | TCP | Yes | :doc:`../auth/ldap` |
| | 389 | | |
| +-----------+------------+----------------------------------+
| | UDP | Yes | :doc:`dns` |
| | 53 | | |
| +-----------+------------+----------------------------------+
| | UDP | Yes | :doc:`../auth/kdc` |
| | 88 | | |
| +-----------+------------+----------------------------------+
| | UDP | Yes | AFS Fileserver (esp. replicas) |
| | 7000 | | |
| +-----------+------------+----------------------------------+
| | UDP | Yes | AFS DBs |
| | 7002,7003 | | |
| +-----------+------------+----------------------------------+
| | UDP | No | AFS VolSer |
| | 7005 | | |
| +-----------+------------+----------------------------------+
| | UDP | No | AFS BosServer |
| | 7007 | | |
+-------------+-----------+------------+----------------------------------+
| .35.191 | | | (Most services here are replicas |
| (chicago) | | | from Typhon) |
| | | | |
| +-----------+------------+----------------------------------+
| | TCP | Yes | SSH |
| | 22 | | |
| +-----------+------------+----------------------------------+
| | TCP | Yes | :doc:`dns` |
| | 53 | | |
| +-----------+------------+----------------------------------+
| | TCP | Yes | :doc:`../auth/ldap` |
| | 389 | | |
| +-----------+------------+----------------------------------+
| | UDP | Yes | :doc:`dns` |
| | 53 | | |
| +-----------+------------+----------------------------------+
| | UDP | Yes | :doc:`../auth/kdc` |
| | 88 | | |
| +-----------+------------+----------------------------------+
| | UDP | Yes | AFS Fileserver (esp. replicas) |
| | 7000 | | |
| +-----------+------------+----------------------------------+
| | UDP | Yes | AFS DBs |
| | 7002,7003 | | |
| +-----------+------------+----------------------------------+
| | UDP | No | AFS VolSer |
| | 7005 | | |
| +-----------+------------+----------------------------------+
| | UDP | No | AFS BosServer |
| | 7007 | | |
+-------------+-----------+------------+----------------------------------+
.. _networking-external_cluster-uplink-cabling:
Cluster Uplink Cabling
======================
You may also wish to refer to
:ref:`networking-internal_cluster-switch-cabling` for the inside job.
+---------------------------------------+--------------------+
| Host and port | Neighbor |
+=======================================+====================+
| Gomes leftmost (eth0) | oldcs (70) |
+---------------------------------------+--------------------+
| Magellan leftmost (eth0) | oldcs (70) |
+---------------------------------------+--------------------+
| Magellan next leftmost (eth1) | DMZ |
+---------------------------------------+--------------------+
| Crimea leftmost | oldcs (70) |
+---------------------------------------+--------------------+
| Chicago eth1 | oldcs (70) |
+---------------------------------------+--------------------+
| Typhon eth0 | oldcs (70) |
+---------------------------------------+--------------------+
| Echidna | cs public (35) |
+---------------------------------------+--------------------+