The Internet Update

2007-04-20 23:49:45.000 – Jon Cotton,  Observer

NULL

Last night I really didn’t have time to write an in-depth account of recent activities on the summit. Our Internet was still down and work was needed to restore it. There was a balance of writing a comment/something/anything and hoping to post it in a small window of uptime, diagnosing the problem, finishing the day’s record check and trying to gather all the materials for the morning forecast. As far as our Internet connectivity goes, the aftermath shows that the issue was water in the valley connection and probable misalignment of the summit antenna due to ice and wind. After hours and hours of combined shift manpower we are back up. Knowing the symptoms will help with a hopefully faster resolution in the future.

By the way, thanks for the support. Many of you have been keen for the myriad details of Spring Storm ’07 (capitalization used for effect). I caught some serious flack for posting a light-hearted comment last night when many people wanted status updates. The tone was supposed to be a “quick update we’re doing great and I’m excited to write more when things ease up” feel. Sometimes I share life-at-the-top stories and in other comments I enjoy the more scientifically amazing aspects of this job. Let me know what you think via email. For those interested in the technical details of our Internet issues read onward!

All of the Observatory’s data runs over a 17 mile point-to-point wireless radio link to the roof of our Weather Discovery Center in North Conway. The link also provides all of our internal phones and one of our outside lines. We have three additional phones lines over a separate wireless system to Littleton, NH. What crashed Monday night was the 17 mile link to North Conway and with it all the data, which is why the website wasn’t updating. The other shift did a great job of diagnosing but sometimes the only thing we can do is wait until the weather improves. The most likely cause they determined was that severe icing and high wind loads on the dish must have knocked off the alignment. Our shift got on it and spent Wednesday night clearing feet of ice off the dish and slightly tweaking the aiming direction. Thursday morning Kyle spent aligning the dish and aligning a secondary backup grid antenna with no luck. Thursday afternoon, it was found without doubt that water in the antenna cable down on the roof of the Weather Discovery Center was the cause. It is an interesting troubleshooting situation because of course the hammering of the summit seemed the obvious cause (maybe the antenna was misaligned, maybe it wasn’t) but we intitially overlooked the buckets of rain the valley experienced. The connection was refit as best it could and the link was stable for several hours. Overnight, the connection flaked out again.

Here comes the geek speak. For large portions of time the radio software was reporting a solid link with few errors (measured in signal power, quality and errored seconds). However the radio error lights would flick on and off and closer examination showed the RF link was oscillating several times a second. The radio we use is a split quad T1/Ethernet bridge with configurable bandwidth and channel provisioning. The Ethernet bridge acts like the 802.11 devices hooked up in coffee shops around the nation. We use this channel as dedicated bandwidth for our video conferencing system and “Live From the Rockpile” presentations. Dual T1s carry all the rest of our data and a single T handles our internal phones. The major development of T-carriers was done by Bell Labs, I believe, in the 1950s and 60s and used primarily for voice telecommunications until the 1990s when the Internet boom added data transport into the T-carrier’s popular repertoire. T1s have very rigid specifications on exactly how the data is modulated on the line with set framing, sampling, error reporting and alarms. As it relates to this narrative, all the devices on a T1 line need a single clock source in order to synchronize. Our router in the valley sets the clock signal on one line, the second T1 reads from the first, the radio matches on both lines and the summit router is configured to read the clock source from the valley router.

As long as the link is stable, the T1s remain synchronized allowing data and voice a smooth flow. Last night showed an oscillating wireless link and the T1s were desperately trying to sync. The router kept trying to grab a clock signal and the radio software reported red alarms. The Ethernet bridge on the radio was subject to the same RF link of course but Ethernet as a communications standard (802.3u) is more tolerable of flakiness than T-carriers. Where the T1 data channel would give me a viable connection once every 15 minutes last night, the Ethernet bridge channel yielded connection for a few seconds every minute.

By hooking directly into the Ethernet switch on the radio, I was patiently able to grab enough model data for a complete summits forecast as well as the state forecast we record for New Hampshire Public Radio. Having such nice weather helps considerably. I’d rather forecast a sunny day on bare bones data than last week’s storm complications.

On Friday, the summit staff finished up the antenna dish alignment while Mike in the valley fixed up the cable connection on the roof. It looks like all systems are go. The T1s are purring, bridges are spanning 17 miles and the summit has the world at its finger tips.

Jon Cotton,  Observer