add support for hostfiles

README.md

@@ -16,6 +16,8 @@ The URLs used for Shibboleth authentication and the DNS-Portal are specified at
 
 An alternative to this program would be gaining AXFR access to the authorative DNS server, which has not been granted to us, yet.
 
+*Update:* Due to the problems inherent with this approach as outlined above, I resorted to downloading RWTE^3H's `/etc/hosts` file, which is accesible without any login. It contains the necessary information in a less structured manner, so I added some support in `generate.py` to detect and work around that. Using the new `required` config option, one can eliminate all the irrelevant hosts in that file. This method is more stable and thus `zonedl.py` probably won't get fixed.
+
 ## `generate.py`
 
 `generate.py` takes one or more zone files as input, reads some configuration file and generates and returns a SSH configuration file. This allows management of CNAME aliases and multiple A records and at the same time use consistent host keys and configuration options.
@@ -23,4 +25,3 @@ An alternative to this program would be gaining AXFR access to the authorative D
 Currently, the possible options which one can configure is quite limited to the most pressing use cases of my config. This could be improved easily.
 
 The configurations file lives either at `./sshgen.cfg` or the location given by `--cfg`. Select a preset with `--preset`. A sample configuration file is available. It configures the location of the zone file(s), the domain stripping and proxy presets, and the various rewriting/exclusion/aliasing/agent settings.
-

generate.py

@@ -68,6 +68,7 @@ def retrieve_hosts():
     i = {}
 
     for k in d:
+        try:
             z = dns.zone.from_text(get_zone_file(k), relativize=False)
 
             # TODO AAAA records (and others)
@@ -87,6 +88,36 @@ def retrieve_hosts():
                     target = []
                     h[rdata.target] = target
                 target.append(name)
+        except dns.zone.UnknownOrigin:
+            for line in get_zone_file(k).splitlines():
+                if line.startswith('#') or len(line.strip()) < 3:
+                    continue
+                parts = line.split()
+                addr = i.get(parts[0])
+                if addr is None:
+                    addr = []
+                    i[parts[0]] = addr
+                addr.append(parts[1])
+                host = h.get(parts[1])
+                if host is None:
+                    host = []
+                    h[parts[1]] = host
+                if len(parts) > 2:
+                    for alt in parts[2:]:
+                        if alt == '#':
+                            break
+                        host.append(alt)
+
+    req_set = set(s.strip() for s in config['excludes']['required'].split(',') if s.strip())
+    def intersects(s):
+        cmp_set = s.copy()
+        for x in s:
+            parts = x.split('.')
+            for n in range(len(parts)):
+                start = -1 - n
+                cmp_set.add('.'.join(parts[start:]))
+        return len(req_set.intersection(cmp_set)) > 0
+    h = {k: v for k, v in h.items() if intersects(set([k]+v))}
 
     fin = False
     while not fin:

sshgen.example.cfg

@@ -35,6 +35,8 @@ hosts = ap-[a-z0-9]+.fsmpi.rwth-aachen.de,
 	sw-[a-z-]+.(fsmpi|asta).rwth-aachen.de,
 aliases = fsmpi.rwth-aachen.de,
 	asta.rwth-aachen.de
+required = fsmpi.rwth-aachen.de,
+	asta.rwth-aachen.de
 
 [aliases]
 learninglinux.fsmpi.rwth-aachen.de = lls