Please fix the fucking network

Please fix the fucking network. There are so many problems with it. Impossible…

Unknown author, Hohenheim 13.05.23 –

We have been receiving emails like this in the past few days. It’s understandable, considering the outages we’ve had in the past week, turning Hohenheim into an internet wasteland. Believe us, we want nothing more than to have the finest internet flowing through the fiber optics. However, let’s take a look at why it’s sometimes not that simple.

First, let’s provide an overview of the locations in Hohenheim:

23 buildings
50 Layer-2 switches
1660 connections in the user network
2x 1 Gbps uplink towards the student union

As you can see, currently, all the connections converge at Schwerzstrasse 1 and then split into two lines towards the student union.

Line 1: Egilolfstrasse 49, 51, 53, 57, 59; Fruhwirthstrasse 3-13; Welfenstrasse 80; and Chausseefeld 1-9.
Line 2: Egilolfstrasse 41, 43, 45; and Schwerzstrasse 1-3.

Both lines are equally loaded.

That’s the infrastructure setup.

Timeline:

In early April, we were requested to provide internet to Schwerzstrasse 3, which had been renovated and is now available as a dormitory.
During the initial inspection, we were informed that Schwerzstrasse 2 would undergo major renovation and the power would be shut off there.
At that time, Schwerzstrasse 2 still housed the uplink for the entire Hohenheim, and that required power. So, it was planned to move that hardware to Schwerzstrasse 3 on the same day. Fiber optics between the buildings are available and can be used.

April 19, 2023
We drive towards Schwerzstrasse 2 to check if there are enough lines to move the uplink towards Schwerzstrasse 3. We find a chaos of fiber optics and start taking notes.
According to the notes, everything should be fine, except for the shortage of cables. Ancient plug connections were used for the fiber optics.

 

April 24, 2023
We took time off to tackle the cabling in the new dormitory.
We arrived at 2 p.m. and immediately started equipping the new cabinet.
However, we encountered the first problem during the reconstruction of the uplink. The individual ports towards Stuwe remained offline. A measurement revealed that the attenuation was too high. You can imagine it as stuffing cotton in your ears. You hear much less and don’t understand everything.
The same applies to light; the further away or worse the material, the less you can see. That’s why many front doors have frosted glass. Light enters, but you can only see a little from the other side.
We suspect that the fiber optic between Schwerz.3 and Schwerz.1 might be defective or contaminated, acting like frosted glass.
As a result, we installed the uplink in Schwerz.1 to mitigate this attenuation. However, only one of the two lines came online there as well. We started trying different lines between Schwerz.1 and Schwerz.2 and eventually found at least 2 out of 24 fibers where the switch received enough information to provide a bit of internet.
That should be sufficient for one night, and we returned to Tübingen. Departure at 7 p.m.

 

Schwerzstrasse 3

April 25, 2023
Since we couldn’t find any documentation regarding Hohenheim in our records (at least the previous responsible person didn’t leave any), we went to Hohenheim with pen and paper in hand.
We diligently conducted measurements and discovered that the damaged line must be between Schwerz.2 and the university’s data center.
The staff at the data center were very helpful and showed us the relevant fibers and a switch, which belonged to Stuwe. We included it in our documentation. However, we don’t have access to it, so we couldn’t perform much diagnosis.
As a precaution, we replaced the fiber optic cable in the data center. We also noticed on a map in the machine room that there must be another building between the university and Schwerzstrasse. We headed there but unfortunately didn’t find anyone available.
Disappointed, we replaced a few more fiber cables in the patch panel of Schwerzstrasse 2 with higher-quality ones, hoping to improve the poor connection. Departure at 6 p.m.

Uplink Hohenheim

April 26, 2023
We drove back, this time to the mysterious Building 1.21. In the basement, we found a cabinet with our fibers. We replaced the connecting cable between the panels in the cabinet and headed towards the university. However, the attenuation was still the same. Cleaning the fiber optics didn’t yield any results, so we switched to another fiber to rule out any faults. Unfortunately, it didn’t work at all.
We dismantled everything and drove back to Tübingen since the shared car reservation was only for 5 hours, and we had already spent 4.5 hours there.
We returned in the evening to continue the troubleshooting. Since the lines worked fine in Schwerzstrasse 2, we installed a homemade amplifier solution until we could reach the data center again. Departure at 9 p.m.

If it works, it aint stupid

April 27, 2023
Early in the morning, we headed straight to the data center. By now, we had become adept at pleading for entry, and we replaced the fiber optics with another pair. Then we proceeded towards Emil-Wolf-Strasse (Building 1.21) and made the necessary connections. Lo and behold, the attenuation value improved from -23 dBm to a fantastic -7 dBm (significantly better).
The patch cable for Egilolfstrasse 45 had a bend and, lacking a better replacement, we substituted it with a multimode fiber (not ideal). Will it work? We quickly documented the changes and headed home. Departure at 4 p.m.

April 28, 2023
Disappointment struck. The uplink was now running smoothly, but Egilolfstrasse 45 seemed to be causing problems. We returned and replaced the multimode patch cable with a single-mode cable. From that point on, things started working smoothly again, and we happily returned home.

May 5, 2023
All the residents from Schwerzstrasse 2 have now moved to Schwerzstrasse 3. We noticed that four rooms were not properly patched (apologies for the mix-up!). We went there and corrected the connections. From Schwerzstrasse 2, we removed the equipment from the floors before the major renovation began. It was quite eerie there, but it seemed like there had been a good party.

May 7, 2023
Initial reports of disruptions in Egilolfstrasse 53-59. Evening outages were observed. We discovered numerous errors in the log. We went there in the evening and replaced the module at the switch (those things keep breaking).

Fiber Module SFP

 

May 8, 2023
Under load, especially in the evenings, we noticed an increasing number of errors. As a result, the traffic briefly decreased and then recovered. You may have experienced internet outages, but we noticed them through the error counter:

Receive:
Error Frames: 227,636,249
Transmit:
Error Frames: 0

Both of these values should be 0. It seems that either the fiber or the module is faulty.

Once again, we visited the location and swapped the fiber to eliminate it as a potential issue. However, we still encountered the same problem.

Two tests are pending:

  • Checking the port on the Stuwe switch.
  • Testing or replacing the fiber module at the switch.

We sent an email to the responsible person since we cannot access external hardware.

May 12, 2023
We received a response from the responsible person at Stuwe.

The examination and, if necessary, replacement will be carried out by the HEDV as part of the uplink expansion. However, the uplink expansion can only take place after the work on Schwerzstrasse 2 is completed.

According to his opinion, the uplink switch needs to be moved to another building, but which building is still uncertain. Until then, everything will remain as is.
Unfortunately, the responsible person will not be available on Friday.

That’s the summary of the events in the past few weeks.
Tomorrow, we will make another round of calls and try to find a solution with the responsible person, if they are available. We are just as frustrated as you are, and we are doing everything we can to provide you with a good internet connection. However, as you can see, sometimes it’s not just technical issues that are at play.

As soon as we have any new updates, we will notify you here.

Until then, have a pleasant evening!

Hohenheim 25th of April 2023

All Hohenheimer dormitories are connected to our network through a central hub.

Until now, this hub was located at Schwerzstrasse 2.

The building is undergoing renovation and will be without power for a while.

The plan was to relocate the hub to the newly renovated Schwerzstrasse 3.

Unfortunately, the connection between the buildings was so poor that we had to move the hub to Schwerzstrasse 1, which delayed the internet interruption accordingly. We planned for 10 minutes, but it took 3 hours.

During operation, we found that the connection there works, but is also too poor for continuous use.

Therefore, we will replace the connections again at around 12 p.m. today.

To keep the interruption short, we will send two teams. We expect the interruption to last less than an hour, ideally just a few minutes.

Update 4:50pm: Thanks to qualified feedback, an unexpected new problem that occurred during the repair of the fiber optic problem was identified and fixed around 4:30 pm. I’ll quote the translated message and thank you on behalf of the Netz-AK.

Hi, we know you’re working hard to switch over to Schwerzstraße. Since we haven’t heard anything new about the switch attempt at 12:00 since this morning’s blog post, we wanted to let you know (in case it hasn’t already been done) that if everything is back to normal, our flat still has problems. Greetings

Nevertheless, we also understand messages that explain to us that the internet is important for studying.

The fiber optic problem was traced to the edge of our network and optimized soon.

Hohenheim March 24th 2023

Good evening,
the maintenance work that was announced lasted from 2 PM to 6:30 PM.
The internet outage lasted longer than expected due to unexpectedly high attenuation of the fiber optic cable.
However, those moving from Schwerzstr 2 to 3 can now expect gigabit in every room.
Everything should be working everywhere again now, but if it’s not, please try restarting your router before contacting us.
Gruss
Lucas and Heinrich

Hohenheim: Fruwithstraße and Egilofstraße from 53 onwards

As part of the cleanup work that the Network Working Group in Hohenheim is finally able to tackle, I plan to publish some events on our blog.

Here’s a short text explaining why we’re worried about the cabling in the mentioned dormitories, since only a few have looked at the cables in their rooms.

I’ll compare the cables from the dormitories in the Hibuka building in the French Quarter in Tuebingen, where I lived extensively. There, we had cables for the doorbell at the front door (2 wires), cable for the telephone connection TAE (6 wires), cable connection (2 wires TV, 2 wires radio), and a double network socket (two cables with 8 wires each).

Why two network cables for one room? If one cable is damaged, the other can be used as a replacement, and you don’t have to lay a new cable from the room to the basement.

The TERA cabling is much more economical. Over a single cable with 8 wires, two wires are used for TV, two for the combination of doorbell and telephone. This leaves 4 wires to establish a 100 Mbit network connection.

If one wire is damaged, the internet network access is gone.

If one wire of the network cable is damaged in the French Quarter, there will only be 100 Mbit instead of 1000 Mbit or we switch to the spare cable.

Given the fact that TERA cables are only manufactured on demand and this type of cabling seems to be very rare, I assume that the savings in installation costs compared to the risks in operation have not prevailed.

From a purely network perspective, we have only half a network cable, without any reserves in the mentioned dormitories.

Nothing will change without construction activity. But building is not within the scope of our volunteer working group.

If there is a cable, we will make the best internet access possible for you.

Even though we can’t do much at the moment, please inform us about any possibly defective network connections and other problems. If only your apartment knows about a problem, nothing will improve.

Thanks for your interest,
Heinrich from the Netz-AK (short for Netzwerk Arbeitskreis = Network Work-group)

https://en.wikipedia.org/wiki/TERA

Upgrade Schwerzstrasse 1

We need to make some unplanned changes to our hardware in your basement.

We can start working from 6 PM in Hohenheim.

Your internet will be down while we work, but we’ll finish by 10 PM. After that, all rooms will be back online, and the speed will be increased to 1 GBit from the previous 100 MBit. You can check the speed by running our speedtest.

 

Speed test

We have set up an internal test for you some time ago. At netz-ak.de/speedtest, you can see how much your connection in your dorm room can handle.
We hope that all devices connected to our network via cable have a ping of less than 5 and an insignificant jitter.

A high jitter of over 5 indicates that the device from which you are measuring is connected via Wi-Fi and may be affected by what we call the 2.4 GHz problem.

Both download and upload should be just below the respective maximum of your building, i.e. 100 MBit/s or 1 GBit/s (1000 MBit/s). You can find out how fast the connections in your dormitory are under the expandable map on the  facts and figures page.

Currently, none of the dormitories are reaching the limit of their connection.

Pestalozzistraße 67 – Netz-AK

Good morning!

Yesterday, 07. of March 2023, we were able to complete the wiring at about 20 o’clock.

We had started around 18:30. So if during that time “the internet was gone”, it was because there was no cable connected at our end.

Starting from the upper floors, you should have come back online when plugged in. We finished at 8pm.

In your rooms the left, oddly labelled network socket should work now.

Until yesterday, some rooms were unsystematically missing its wiring and we didn’t know which room was connected where.

Normally we announce maintenance work longer in advance, but this time it was urgent.

Please contact us if you have any problems!

Your volunteers of the Netz-AK (short for Netzwerk Arbeitskreis = Network Work-group)

A warm welcome to the Pestalozzistraße 67!

The dormitory will be opened today and as far as we know it will go online over our network.

Unfortunately this morning we still don’t know when that will be and are waiting for the go of the StuWe technicians.

However, you are right next to the university with plenty of internet and the other dorms are all online, too.

Get to know your new neighbours, the internet will reach you soon.

As soon as the network is at your house, it works like everywhere else: You plug in your own router and create your own little network with WiFi.

Please use our tutorials and don’t buy too expensive devices (never over 40€!), but also nothing outdated to save 5 €.

Just read the tutorials!  😉

See you around
your volunteers of the Netz-AK (short for Netzwerk Arbeitskreis = Network Work-group)

Partial outage Egilolfstraße 49 & 51 2022-12-28

Today we exchanged a damaged network switch in Hohenheim. The outage affected around 24 connections. During replacement, the connected houses in Chausseefeld and Welfenstraße were also briefly offline.

Unfortunately another device failed during restart. We can only exchange this device tomorrow on December 29th, 2022. Until then around 24 connections will be offline in Egilolfstraße.