Please fix the fucking network. There are so many problems with it. Impossible…
Unknown author, Hohenheim 13.05.23 –
We have been receiving emails like this in the past few days. It’s understandable, considering the outages we’ve had in the past week, turning Hohenheim into an internet wasteland. Believe us, we want nothing more than to have the finest internet flowing through the fiber optics. However, let’s take a look at why it’s sometimes not that simple.
First, let’s provide an overview of the locations in Hohenheim:
23 buildings
50 Layer-2 switches
1660 connections in the user network
2x 1 Gbps uplink towards the student union
As you can see, currently, all the connections converge at Schwerzstrasse 1 and then split into two lines towards the student union.
Line 1: Egilolfstrasse 49, 51, 53, 57, 59; Fruhwirthstrasse 3-13; Welfenstrasse 80; and Chausseefeld 1-9.
Line 2: Egilolfstrasse 41, 43, 45; and Schwerzstrasse 1-3.
Both lines are equally loaded.
That’s the infrastructure setup.
Timeline:
In early April, we were requested to provide internet to Schwerzstrasse 3, which had been renovated and is now available as a dormitory.
During the initial inspection, we were informed that Schwerzstrasse 2 would undergo major renovation and the power would be shut off there.
At that time, Schwerzstrasse 2 still housed the uplink for the entire Hohenheim, and that required power. So, it was planned to move that hardware to Schwerzstrasse 3 on the same day. Fiber optics between the buildings are available and can be used.
April 19, 2023
We drive towards Schwerzstrasse 2 to check if there are enough lines to move the uplink towards Schwerzstrasse 3. We find a chaos of fiber optics and start taking notes.
According to the notes, everything should be fine, except for the shortage of cables. Ancient plug connections were used for the fiber optics.
April 24, 2023
We took time off to tackle the cabling in the new dormitory.
We arrived at 2 p.m. and immediately started equipping the new cabinet.
However, we encountered the first problem during the reconstruction of the uplink. The individual ports towards Stuwe remained offline. A measurement revealed that the attenuation was too high. You can imagine it as stuffing cotton in your ears. You hear much less and don’t understand everything.
The same applies to light; the further away or worse the material, the less you can see. That’s why many front doors have frosted glass. Light enters, but you can only see a little from the other side.
We suspect that the fiber optic between Schwerz.3 and Schwerz.1 might be defective or contaminated, acting like frosted glass.
As a result, we installed the uplink in Schwerz.1 to mitigate this attenuation. However, only one of the two lines came online there as well. We started trying different lines between Schwerz.1 and Schwerz.2 and eventually found at least 2 out of 24 fibers where the switch received enough information to provide a bit of internet.
That should be sufficient for one night, and we returned to Tübingen. Departure at 7 p.m.
April 25, 2023
Since we couldn’t find any documentation regarding Hohenheim in our records (at least the previous responsible person didn’t leave any), we went to Hohenheim with pen and paper in hand.
We diligently conducted measurements and discovered that the damaged line must be between Schwerz.2 and the university’s data center.
The staff at the data center were very helpful and showed us the relevant fibers and a switch, which belonged to Stuwe. We included it in our documentation. However, we don’t have access to it, so we couldn’t perform much diagnosis.
As a precaution, we replaced the fiber optic cable in the data center. We also noticed on a map in the machine room that there must be another building between the university and Schwerzstrasse. We headed there but unfortunately didn’t find anyone available.
Disappointed, we replaced a few more fiber cables in the patch panel of Schwerzstrasse 2 with higher-quality ones, hoping to improve the poor connection. Departure at 6 p.m.
April 26, 2023
We drove back, this time to the mysterious Building 1.21. In the basement, we found a cabinet with our fibers. We replaced the connecting cable between the panels in the cabinet and headed towards the university. However, the attenuation was still the same. Cleaning the fiber optics didn’t yield any results, so we switched to another fiber to rule out any faults. Unfortunately, it didn’t work at all.
We dismantled everything and drove back to Tübingen since the shared car reservation was only for 5 hours, and we had already spent 4.5 hours there.
We returned in the evening to continue the troubleshooting. Since the lines worked fine in Schwerzstrasse 2, we installed a homemade amplifier solution until we could reach the data center again. Departure at 9 p.m.
April 27, 2023
Early in the morning, we headed straight to the data center. By now, we had become adept at pleading for entry, and we replaced the fiber optics with another pair. Then we proceeded towards Emil-Wolf-Strasse (Building 1.21) and made the necessary connections. Lo and behold, the attenuation value improved from -23 dBm to a fantastic -7 dBm (significantly better).
The patch cable for Egilolfstrasse 45 had a bend and, lacking a better replacement, we substituted it with a multimode fiber (not ideal). Will it work? We quickly documented the changes and headed home. Departure at 4 p.m.
April 28, 2023
Disappointment struck. The uplink was now running smoothly, but Egilolfstrasse 45 seemed to be causing problems. We returned and replaced the multimode patch cable with a single-mode cable. From that point on, things started working smoothly again, and we happily returned home.
May 5, 2023
All the residents from Schwerzstrasse 2 have now moved to Schwerzstrasse 3. We noticed that four rooms were not properly patched (apologies for the mix-up!). We went there and corrected the connections. From Schwerzstrasse 2, we removed the equipment from the floors before the major renovation began. It was quite eerie there, but it seemed like there had been a good party.
May 7, 2023
Initial reports of disruptions in Egilolfstrasse 53-59. Evening outages were observed. We discovered numerous errors in the log. We went there in the evening and replaced the module at the switch (those things keep breaking).
May 8, 2023
Under load, especially in the evenings, we noticed an increasing number of errors. As a result, the traffic briefly decreased and then recovered. You may have experienced internet outages, but we noticed them through the error counter:
Receive:
Error Frames: 227,636,249
Transmit:
Error Frames: 0
Both of these values should be 0. It seems that either the fiber or the module is faulty.
Once again, we visited the location and swapped the fiber to eliminate it as a potential issue. However, we still encountered the same problem.
Two tests are pending:
- Checking the port on the Stuwe switch.
- Testing or replacing the fiber module at the switch.
We sent an email to the responsible person since we cannot access external hardware.
May 12, 2023
We received a response from the responsible person at Stuwe.
The examination and, if necessary, replacement will be carried out by the HEDV as part of the uplink expansion. However, the uplink expansion can only take place after the work on Schwerzstrasse 2 is completed.
According to his opinion, the uplink switch needs to be moved to another building, but which building is still uncertain. Until then, everything will remain as is.
Unfortunately, the responsible person will not be available on Friday.
That’s the summary of the events in the past few weeks.
Tomorrow, we will make another round of calls and try to find a solution with the responsible person, if they are available. We are just as frustrated as you are, and we are doing everything we can to provide you with a good internet connection. However, as you can see, sometimes it’s not just technical issues that are at play.
As soon as we have any new updates, we will notify you here.
Until then, have a pleasant evening!