On July 8th, 2008 I wrote a post raving about Verizon FiOS, a high-speed fiber-to-home internet solution that has clearly kicked the butt of all the competition on a performance/cost basis. I’d been using the service for around two years at that juncture. On October 20th, 2008 I wrote another post, this time chronicling the extreme distress I was experiencing with my Verizon FiOS connection. It is now November 5th, 2008 and my issue is still not resolved. The problem began on 10/16 and continues to the present. I have spent 10+ hours on the phone with Verizon over a period of days and have opened multiple tickets including PADQ01JC660 and PADQ01KD8X (which was closed for an unknown reason) and now PAFS010562.

The Problem:

Verizon’s first tier technical support is decent, they can fix 99% of mom and pop problems. This means if you have a standard problem (e.g. router died or needs to be rebooted, you need to enter a password, ip needs to be renewed, etc.) you’ll most likely have no problem getting rapid support. The issue is with escalation. After the first level of support their are “Network Technicians”, these are the people who are supposed to analyze and resolve complex issues. Unfortunately, multitudinous experiences indicates:

• Network Technicians do not communicate concerning tickets.
• Network Technicians do not perform necessary troubleshooting on tickets.

I should note, as a Network Engineer, I understand some of the dilemma faced by network technicians. First, one is constantly bombarded by a large number of false positives. People will insist they have a problem that is your fault when it is their own. Second, network technicians generally tend to enjoy working on problems more than communicating about problems. Okay, this is natural…but this has been ridiculous. Ignore it once, okay – not the best idea but understandable. Ignore it twice – okay, bad idea. Ignore it three (four, five) times and now we are getting to the point of inciting righteous anger on the part of the consumer.

The Resolution:

I can’t remark on the specifics of resolving this issue, since I am not within the Verizon NT group, but I will comment generally on ways to resolve this sort of consumer abuse:

• Ensure network techs. are not overtasked. A network tech. will let “questionable” problems fall through the cracks when he is over-engaged by “real” problems.
• Enable a linking method for tickets and an analysis system that will detect repeat callers and allow for appropriate escalation to resolve the issue.
• Offer a web-based ticketing system with tickets automatically visible via phone call. Allow consumers to view and respond to ticket modifications.[1. This way if a ticket is closed, the consumer knows it...rather than waiting a day or two to call back in about the issue to find out that the NT never did anything with the issue.]
• Its all about communication. If a network tech. doesn’t believe its a real issue he needs to communicate this back to the first tier tech., and the first tier tech. needs to talk to the consumer more…But in no case should a ticket simply be dropped.

UPDATE 11/07/08: Well, life is back to normal…after around two weeks. I called in and told them I would remain on the line until the NT was available. They told me he would call back within 48 hours. I insisted on knowing what the tech. thought was the problem. The NT said he would call back in four hours. I still insisted on knowing what the tech. thought was the problem, this didn’t get very far…I concluded by asking the first tier helpdesk to inform the NT that I was placing all Verizon related tickets online and that if this news made it into mainstream press the NT could be assured Verizon higher-ups would be looking for someone to sacrifice. I received a call-back within an hour or two. The problem had been resolved. It had been an issue with the configuration of their Juniper switches…I am happy now but think that my suggestions above still carry significant weight. It shouldn’t have taken two weeks to make a configuration change.

Image thanks to striatic’s generous creative commons.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

On Thursday, 10/16/08 I tried to log onto the internet. No connection. Between 10/16-10/17 I would make seven major phone calls to Verizon in an attempt to resolve this issuing totaling approximately 250 minutes (over 4 hours). Verizon for an unknown reason[3. Not only unknown to me but also unknown to them. They did not know why I was disconnected, just that they had done it.] disconnected my FiOS service[4. Second time this has happened since establishing service with them.]. Okay, no big deal – reestablish the service[5. Unfortunately it takes hours for this to occur. As a Network Engineer I am still unable to comprehend what could possibly take so long in the assignment of an IP address when all the physical wiring actually exists and has for some time.]. They create an entirely new account and inform me at the same time that I can no longer have my less-expensive 5 Mbps/2 Mbps subscription but must upgrade to a 10 Mbps/2 Mbps. Again, not a huge deal – $10/mo. – but doesn’t make an unhappy customer happy when you screw up and then demand he pay a higher price for the service.

The installation was supposed to be completed by the end of that day. At the end of the day I called back, ohh – somebody screwed up the order. I would talk to approximately three people between that call and calls the following day attempting to get the order straightened out. They placed at least two new orders and finally one of them kicked through. In all, it was around 48 hours without internet access.

Then the internet access finally returned. Great! I’m a happy man…Till its slower than DSL. I run some speedtests. Yup, certainly slower than DSL – and ping tests against Google/Yahoo’s servers revealed that occasional timeouts where occurring and response times were slow (over 100 ms typical). More phone calls to Verizon[6. Unfortunately my cell records have not updated to show me the length of those calls, so I cannot report their exact duration - but it was several and the time was significant.]. Yes, my service is now working. No, the service is very slow. How do I know? Well, I used a bandwidth test. Run through the normal diagnostics – resetting wireless router, plugging directly into FiOS connection, resetting the FiOS box outside my house – sending the issue over to a tech. to look at. The pings drop down – Google/Yahoo now in the 10 ms range, but the speed is still horrific. Maybe its the bandwidth tester – well, I trust Speakeasy’s test but, sure, let’s try Verizon’s. Verizon shows even more dismal speeds. Ahh, forget Verizon’s speed test let’s try speedtest.net. Now if I go to the server in Newark I get 10 Mbps, but almost everywhere else its slow as anything. Will I run the FiOS settings optimizer[7. Glorified IE/TCP optimizer.]? Sure. Still nothing. I’m tired, I’m going in circles. I talk to a supervisor. Same thing. They want to send me a new router. Sure, you can do that, but since I’m plugging directly into the connection and bypassing the router it’s not the router. Okay, we’ll have a technican look at it. Great – never hear from them again.

The connection has slowly improved. Dropped packets have decreased, but overall the network is slow. Streaming video doesn’t work well – unless you happen to hit 1% of the servers that seem to be operating on Verizon’s network at high speeds.

So, that is my situation as of today. Two days without internet – now extremely slow internet. Calling into technical support is an exercise in frustration. I can’t speak to someone who understands the problem is a technical issue on Verizon’s side. My guess? Either my circuit settings where entered wrong or they have a bad router somewhere along the line. 99% of my traffic is going through the bad router – if I happen to hit a server that takes another route (doesn’t use that router) then I get good speeds.

Anyone else having this problem? The phone support from Verizon has improved in that I don’t spend 15 minutes dialing through automated voice support – but the help desk support doesn’t have a clear escalation path beyond themselves and so I am at an ends. Vague promises of a “network tech.” looking into it and no followup leaves me with little hope of a long-term solution.

UPDATE:

Spoke to a technician at Verizon who knew what I was talking about and again submitted a ticket to the network tech. guys. His name was Dan…If any Verizon people do read this, he needs a promotion – his networking knowledge is far beyond most of his peers. In any case, he agrees the issue is one with the route being taken to reach the final website. To give everyone an idea here is a tracert to google.com (note the length of the route, its extremely, ridiculously long):

<!– @page { margin: 0.79in } P { margin-bottom: 0.08in } –>

1 1 ms <1 ms <1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 4 ms 4 ms 4 ms L100.FTTP-02.PHLAPA.verizon-gni.net [68.238.240.1]
3 7 ms 6 ms 7 ms G1-0-881.LCR-08.PHLAPA.verizon-gni.net [130.81.110.108]
4 7 ms 6 ms 7 ms P14-2.LCR-02.STTLWA.verizon-gni.net [130.81.29.6]
5 7 ms * 7 ms 0.so-6-0-0.XL2.PHL6.ALTER.NET [152.63.3.81]
6 13 ms 12 ms 11 ms 0.so-6-0-0.XL4.IAD8.ALTER.NET [152.63.0.130]
7 13 ms 11 ms 12 ms 0.ge-3-3-0.BR2.IAD8.ALTER.NET [152.63.40.229]
8 22 ms 49 ms 99 ms 204.255.169.2
9 88 ms 86 ms 87 ms tbr1.wswdc.ip.att.net [12.123.8.106]
10 580 ms 576 ms 724 ms cr1.wswdc.ip.att.net [12.122.16.89]
11 580 ms 484 ms 96 ms cr2.phlpa.ip.att.net [12.122.4.53]
12 115 ms 97 ms 96 ms cr2.cl2oh.ip.att.net [12.122.2.209]
13 87 ms 87 ms 86 ms cr1.cl2oh.ip.att.net [12.122.2.125]
14 88 ms 87 ms 86 ms cr1.cgcil.ip.att.net [12.122.2.205]
15 99 ms 99 ms 99 ms cr1.st6wa.ip.att.net [12.122.31.162]
16 88 ms 86 ms 87 ms tbr1.st6wa.ip.att.net [12.122.23.130]
17 87 ms 86 ms 87 ms gar8.st6wa.ip.att.net [12.122.111.17]
18 98 ms 96 ms 117 ms 12.89.209.14
19 111 ms 96 ms 97 ms 209.85.249.32
20 103 ms 98 ms 89 ms 66.249.95.208
21 91 ms 92 ms 91 ms 72.14.233.37
22 92 ms 92 ms 104 ms 216.239.48.143
23 93 ms 106 ms 117 ms 209.85.251.125
24 102 ms 94 ms 104 ms 74.125.31.2
25 105 ms 104 ms 104 ms cg-in-f99.google.com [209.85.171.99]

Unfortunately 99% of the sites I visit appear to be routing in this same, obtuse method. 1% take a different router and operate at blazing fast speeds.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Growing up I lived in Westerlo New York. No, its not a big town. Yes, that’s the reason you’ve never heard of it. In any case, its way out in the middle of nowhere and to this day large portions of the town can only access the internet via 56k…err, make that 28.8k. As a teenager I spent so many hours waiting for a page to load – I learned to read a book at the same time I was surfing the web (a habit that stays with me to this day).

When I moved to Pennsylvania I experienced what high speed internet was on Philadelphia Biblical University’s campus. Granted, the speeds weren’t that amazing – but since I worked oftentimes over winter and summer breaks when most of the students were away things cracked along at amazingly fast speeds. The bonded T1′s providing 3 Mbps of internet connectivity.

For a while I moved back to dial-up, until I was hired by Collages.net Inc. Then I moved up to DSL – and through several different providers including Verizon and Speakeasy. This continued until Charity and I bought our current home – and they had just begun FiOS layouts. So we had the FiOS run.

What exactly is FiOS? I’m glad you asked. Lets take a minute to talk about the different types of connectivity one can have:

• Dial-Up – Uses traditional copper lines and communicates via analog signals. At the sender’s end the modem translates everything into analog and then it must be converted back into digital on the receiving. Okay, that’s not the big problem – the real problem is that dial-up operates at 56kbps at its fastest – and that’s slow.
• Cable Modem – Operates over the same lines that the cable video network runs over. Much bigger pipes but everyone in a local area shares the same pipe. If you are the only person or one of a small number using cable, things go fast – but start to build up a lot of users at a local node and internet speeds will begin to deteriorate. Common speeds are around 1-8 Mbps (~1000-8000 Kbps).
• DSL – Runs over traditional phone lines but at much higher speeds than dial-up yet usually at lower speeds than cable’s theoretical maximums. Oftentimes speeds where in the range of 1-2 Mbps.
• FiOS – Uses dedicated fiber run to the individual home, providing exceptional speeds as high as 20-40 Mbps currently (~20,000-40,000 Kbps) and starting at 5 Mbps!

As you can begin to see from the above breakdown of service types (yes, I ignored satellite, ISDN, etc.) – FiOS kicks butt for two reasons. First, it has a dedicated line and second its speed is excellent. Not to say that cable can’t catch up, but it will require significant infrastructure upgrades by the cable providers.

So, anyways, I’ve really enjoyed FiOS and have very few complaints. But, to be fair, I’ll list the issues I’ve encountered with FiOS:

• The technicians who installed my FiOS where telecommunications guys and didn’t really understand how FiOS or the internet in general worked. Verizon has been cross-training these guys, but they are still relative newbies. Perhaps they are better now with two years of experience under their belts.
• Verizon at one point canceled my line due to a billing mistake on their end. Their network did not recognize I had been successfully connected and yet I had internet access, so out of the blue I lost internet connectivity when they decided one day to terminate my connection. They rectified this within 3-5 days.
• Verizon is horrific if you have to call them. Granted, I barely ever have to call for technical support – but if you do, be ready for your patience to be tried. The worst part is the automated phone system. By the time you reach a technician you are ready to strangle someone. The technician’s themselves aren’t bad, though sometimes lack the ability to effectively troubleshoot the issue (I’m a network guy). The worst though is after dialing through their automated system and waiting on hold for 10-15 minutes you receive this message, “We are sorry, all lines are busy. Please call back later.” What?!?!
• They only warranty their routers for a year. If your router dies after that period expect to either pay them a hefty fee (and it will die) or go to Best Buy/Circuit City and pick up a new unit (unfortunately, you’ll need to be a geek to figure out how to configure the router).

Okay, I know those sound like a lot of negatives…But its the best experience I’ve had thus far from an ISP. I’m pleased with the speeds and with the technology. If Verizon could work on the customer service aspect they’d have one killer service.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.