Yesterday around 200PM I received an urgent LinkedIn message. The person said I had helped them a few years ago with an IT issue and they had an emergency today. Two hours later I was at their Bluffton, IN location ready to assist them. My contact person was in another state and we spoke on the phone for communication. The Bluffton trucking location had two buildings: Dispatch and the Shop. Each one had a 24-port gigabyte ethernet switch. The two switches were connected via an SFP fiber optic cable running underground.
My contact person had spent the morning trying to get the switch in Dispatch to respond and it never was completely down but it wasn’t perfect either. Sometimes the internet worked for the computers and sometimes not. The phone system also was up and down. The night before, there was an alarm for one of the switches that said a problem occurred but I don’t remember the details. They tried re-booting it and sending it instructions via the network and nothing seemed to work.
My contact wanted an IT person(me) on site because he tried instructing the local employees but they lacked the knowledge to help my contact. They did their best, but it wasn’t enough. My contact wanted to see if disconnecting the switches from each other would help. The problem the local employees were having was disconnecting the fiber optic cable from the Dispatch office switch. Four or five people attempted to pull the connector out and all failed. It took me several minutes but I finally got it disconnected. A thumbtack was used to pry the plug out of the connector. . It was in pretty tight and I had to pry it out with damaging the small fiber optic cable.
Once the two switches were disconnected from each other, the situation improved and we got the network back to normal and phones and internet, email all worked as usual. I checked back today and it’s still working normally.