Held hostage by Worldlink, Inc.

•April 28, 2009 • Leave a Comment

I, personally, have been a long time customer of Worldlink, Inc as a co-location provider. They’ve always had issues, like not being available for reboots on the weekends, not having adequate power to support the equipment in the event of a power outage, not honoring their uptime guarantee. But they were cheap, so I expected a few issues.

Then they raised their prices and I started shopping. I didn’t put much effort into it though, as I had a million other things that were more important. Then Lori, the “billing” person, goes on vacation and the Senior Systems Administrator is left to take my payment. He goofs and I see a double-bill on my account. I send this email to “billing” on April 9th.

It appears that you’ve double charged me for the 207.90. I will be removing my server from your facility on April 25th at 10:00pm. Please make the necessary arrangements.

Joe Julian

I had also spoken with Rick on the phone to that effect, and when Lori returned, I also told her that I was canceling and she acknowledged receipt of that email.

I call on Friday the 24th to confirm my appointment the next day to pull the servers. I talk to Ryan who says he has to check with his boss about it and will call me back. He gets my number and reads it back to me. I point out that 5:00 is quickly approaching and suggest he should get back to me before then. He assures me that he’ll call me back either way.

Either way? I have a contract that states that it allows 24x7x365 access.

I don’t get a call back.

Although I’ve got my phone with me all day Saturday to ensure no missed calls with this changeover rapidly approaching, I receive none. We run some errands, watch a movie while eating dinner. I check my email at about 8:30 and notice this email had arrived about 5:31:

Joe,

Rick here. You called Friday as I was on my way out and you talked to Ryan. He said you were going to call back a little before 5PM. I shot off a quick email to the powers that be to check and make sure that pickup of the server was OK, and they responded in the negative. Apparently, they have yet to receive a proper cancellation in 30 days in advance.

It puts me in an uncomfortable position to say so, but I guess I get to be the voice here… No one can meet you at 10:30 tonight to remove the gear. What you REALLY need to do is do the cancellation thing. I’d say your best shot is to call Monday through Friday between noon and 3, hit zero, and pursue with them. I know those are some pretty slim hours, but they’re actually there a lot more… I’m just trying to give you the best shot.

In any case, I know what they’re going to do is to have you FAX or postal mail a written cancellation in. It should contain the date, your name, desire to cancel and remove equipment, and the effective date (ASAP).

Like I said, I’m not thrilled to pass along billing news like this to ANY customer, but I felt obligated to fire off some kind of update.

Thanks,

Rick Kunkel
Worldlink DNS Administrator
(206)361-8785

First of all, what does cancellation have to do with access? As a matter of fact, it should mean just the opposite. I head down there as I begin making phone calls. I call the “emergency” number. The “emergency” extension. Rick’s home. I’m sending emails. Find Rick’s Facebook account and message him there. Nobody’s responding. I continue to try to get a response, calling every 15 minutes to no avail. Finally, at 2:30am, I am defeated. I go home.

I awake the next morning and begin again at 8:00. I call my colleague Brad who rents a half-rack and hosts the Ed Wyse email server. He schedules to meet me there at 6:00pm and at least get the Ed Wyse box out.

About mid day I get a reply on Facebook from Rick:

Joe,

I talked with my bosses, and my hands are tied. I do not like being in this position. They say that you will have to talk to billing on Monday, and there’s nothing I could or should do. I’m sorry.

If your machine is down, is there something that we could do to reset it? I am away this weekend, but it’s possible that Ryan could reset. I have been instructed to not allow access at this point.

–Rick

At 6:00, I meet with Brad. As we’re headed for the door I think, “Hey! I’ve got an idea!” I have Brad wait a moment while I shut down the server from my laptop (already booted in the car) so that I don’t have to plug in the monitor and keyboard to shut it down. We then go over to the door. Brad swipes his RFID key and the light blinks… Red. It’s supposed to blink green for access. He tries again, still red. He tries calling the emergency line, and Rick’s cell phone. Nobody returns his calls. He too has asked to cancel his contract at the end of the month, but again is supposed to have unlimited access. With no way to get in to the building, Brad leaves. I have another down server that I can’t access.

I start to leave then decide I’m going to take this further. I pull into a parking lot and start doing searches. I remember that they’re a corporation so I look up the corporate record at the Secretary of State’s web site and get the names of the principals, Lori and Darwin Hill. I start searching for their names and discovered this juicy tidbit:

FOR IMMEDIATE RELEASE
January 14, 2004
Gregoire Settles with Worldlink

SEATTLE — Attorney General Christine Gregoire announced a settlement today with Worldlink, Inc., a Shoreline-based Internet provider accused of engaging in unfair business practices.

Worldlink, run by the husband and wife team of Darwin and Lori Lee Hill, offers Internet services including dial-up accounts and DSL service. Under the settlement, entered in King County Superior Court, the company has agreed to pay $17,876 in restitution, civil penalties and fees and stop the business practices alleged to be unfair by the Attorney General in the complaint.

Worldlink is accused of providing poor service to customers, including charging for services not provided, failing to cancel consumers’ accounts after representing they would do so, placing charges on consumers’ credit cards or debit cards without authorization, placing numerous unfair restrictions on cancellation, and financially penalizing consumers who canceled. Worldlink is also accused of charging customers for service they could not deliver because of technical problems.

As part of the agreement, Worldlink must adhere to specific rules of fair business operation. Those rules include providing adequate notice of contract renewals and charges, canceling customer accounts when requested within a specific period of time, resolving customer complaints in a timely, professional manner and maintaining a record of consumer complaints and resolutions.

Worldlink is also required to cease billing consumers who were subjected to the alleged unfair practices and provide adequate information to any collection agencies or credit reporting agencies whose records may incorrectly reflect the consumer’s obligation to pay.

The case was handled by Gregoire’s High Tech Unit.

and

Settlement announced with Internet provider

OLYMPIA — The Attorney General’s Office yesterday announced a settlement with a Shoreline-based Internet provider accused of engaging in unfair business practices such as placing charges on consumers’ credit cards without authorization.

The company, Worldlink, has been operating since 1995 and has about 10,000 dial-up and high-speed subscribers, according to Darwin Hill, co-owner with his wife, Lori Hill. Darwin Hill said the decision to pay the state nearly $18,000 in restitution, civil penalties and fees and to adhere to certain business practices was not an admission of wrongdoing.

Rather, he said, it was a business decision to avoid continuing struggles with the Attorney General’s Office. The office said it had collected about 30 consumer complaints.

It accused Worldlink of providing poor service, including charging for services not provided, failing to cancel consumers’ accounts after saying it would, and financially penalizing customers who canceled.

A consent decree commits Worldlink to providing adequate notice of contract renewals and charges, among other rules of operation.

Oh, crap!

I knew I had to make this happen quickly.

I had a public records search run on them and got their phone number and address.

I tried the phone numbers, and one of them had a recording that was Lori’s voice. I left a message explaining that if this wasn’t taken care of right away that I was going to find out who the investigator was and let them know that they’re up to the same old tricks.

No reply.

I then decided to drive over to their house.

They live on 145th, not far from the freeway. As there’s no street parking there, I had to park about a block away. I walked over, went up the driveway and knocked on the door.

Darwin answered the door, and I could see that they had sat down to dinner. I asked if I should come back after dinner, and he told me that it was okay and asked me what I wanted. I told him my name (which I know for a fact he knows who I am) and he gave me a blank look. In fact, he was so committed to not knowing what I was talking about, I think if I had asked, he would have denied knowing what Worldlink was. But I didn’t let him. I told him that Rick had talked to either himself or Lori and had been told I wasn’t to be allowed entry. I told him that it “feels” like the same thing’s happening again that they got in trouble for back in 2004, but that I was sure that couldn’t be the case (again, he denies of knowledge of what I’m talking about). He asked what “Rick” I had talked to, I told him. He said he would call Rick and have him call me. I thanked him and walked away.

I knew he was lying. He was lying about what he knew. He was lying about calling Rick. I went back to my car and waited.

After 25 minutes, I walked back to the front door. All the shades were pulled. The lights were all off.

I knocked.

I waited.

2 minutes pass and just as I touch the door to knock again LOUDER, my phone rings. It’s Rick. Apparently, Darwin didn’t like me calling his bluff.

Less than a half hour later, I’m back at the colo. Ryan arrives with his dad and helps me remove the rails from the rack as I remove the servers.

The rest was easy.

We’re now in Netriver‘s Colocation facility. It has top-of-the line power facilities and environmental design. The rest has yet to be tested, but they look like there’s a lot more behind them than Worldlink seemed to have.

Bottom line, avoid Worldlink, Inc of Shoreline, Wa at all costs.

Commercial Quality Helpdesk and Resource Management System

•April 16, 2009 • Leave a Comment

I’ve switched us over to GLPI for our helpdesk ticket management system, and I am a new fan.

I found GLPI while looking for a piece of software to manage our resources. We had a series of failures of our Zyxel Prestige 662HW-61’s where the bridge chip would start dropping packets. Once they started dropping them, they would just keep getting worse until they were completely unusable. As we have over 25 of them that were all approaching the end of their warranty, we were a bit concerned. So far, Zyxel has taken good care of us. As a result of needing to keep track of which modems had been changed out, which were in transit, and which were good – I looked for a better tool than a spreadsheet.

I found GLPI. Not only does it keep track of all the hardware, their locations, purchase dates, warranty periods, applicable contracts, etc., it also has this nice helpdesk system that integrates with the resources. Now trouble tickets are attached to the resource that caused the problem, greatly improving the ability to identify cronic equipment problems.

Like any software packages, there are a few deficiencies. This package is written by a french group (one of which is Remi Collet, of the remi repository fame). Some of their english translations are a bit lacking. They are, however, quite willing to update the dictionaries with corrections when submitted. There’s a feature to authenticate users against imap accounts that is supposed to be able to be configured to use a preset domain name. The setup for that is there, but the execution was missing. Lastly, there’s a “network links” system that, when complete, will save a lot of time tracing down wires. Unfortunately, it’s only half implemented.

Overall, this is a very powerful piece of software that will greatly simplify management of my helpdesk resources, and our equipment.

SWITCHed to freeSWITCH

•April 6, 2009 • Leave a Comment

The evaluation and changeover is complete from Callweaver to freeSWITCH. Callweaver didn’t do what we needed. freeSWITCH did. Not that difficult of a decision to make.

The freeSWITCH learning curve isn’t all that steep. Yes, it’s a complete departure from what you were used to with Asterisk/Callweaver, but it’s far more logical and can be as detailed as you like. Unlike CW/* dialplans are processed in the order that you load them. Put your specific dialplans first, then drop through to your more and more generic ones. It’s simple.

The support I’ve received from the respective IRC channels is beyond comparison. In CW, my responses were rare and frequently pointed me to things I should try programming myself, as the things I needed didn’t exist. In the freeSWITCH channel, not only were people personable and helpful, they were there all the time! Newbies, though referred to the wiki, are told which pages of the wiki hold the information that’s useful to them. They’ll even help you brainstorm methods for getting the resultant dialplan you’re trying to accomplish, should you be trying to do something “interesting”.

And the best part, we have working DTMF! We can now voice authorize credit card transactions if necessary. A feature that was going to kill this project and cause the principals of this company to pull the VoIP system and put the old Panasonic switches back in. Now we can move forward and save tens of thousands of dollars per year by switching to VoIP comany wide.

Thank you to everyone in #freeswitch, and to the developers that made this possible.

Callweaver vs. Freeswitch

•April 1, 2009 • 1 Comment

I’ve gotten a bit sidetracked from my main project to handle an issue that’s been plaguing a couple of our stores. DTMF (rfc2833) over VoIP.

Somehow, Level 3 Communications picked Sonus to provide them with HDMGs. Sonus has botched the rfc2833 implementation in RTP causing everybody to write special cases for their software to work around Sonus’ inflated ego.

Callweaver, an asterisk fork, has not kept up with the industry’s needs. In callweaver’s irc channel, I was pointed to freeswitch’s implmentation of a Sonus workaround to see if it could be integrated into callweaver. I decided to look at freeswitch as an alternative instead.

Not only is freeswitch’s development more active, it was designed by an asterisk developer who recognized the design shortcomings in asterisk and decided to start from scratch with a threaded framework. Additionally, of course, it has the workaround for the Sonus issue that we need.

The down side is that freeswitch uses a more flexible configuration design, where the config files are in xml. The problem with that is that now I have to learn a whole new configuration system and re-implement the features we need. This will just take time that I was hoping to spend on the virtualization project.

Within the next day or two, I should have all the bugs worked out and will report on my assessment of freeswitch.

SAN, the good, the bad, the failure

•March 26, 2009 • Leave a Comment

Storage Area Networks (SAN) uncouple the storage system from the physical server. The arguments for and against SANs being:

Should a server fail, simply roll in a new server, move the HBAs over from the failed server and boot. With hot-swap drive trays, it may actually be quicker to swap over DAS drives
A Fiberchannel SAN interface is faster Newer SAS drives on smart array controllers are arguably just as fast, or faster due to reduced communication overhead
SANs can be expanded without downtime DAS can be expanded, in many circumstances, without downtime
SANs shared storage ability results in greater utilization of the storage media NAS can also provide greater utilization of storage media
SAN hosts do occasionally fail. Their failures can affect the entire network architecture and are not quickly and easily recovered. Their failures, can also cause data corruption that., if using SAN replication, will be replicated to the failover SAN device.
SAN storage is much more costly than DAS

Since there is no way that I can justify the cost of installing multiple SAN hosts, or ensuring the data’s integrity, I’ve eliminated this option from my list of possibilities. It’s reliance on a single piece of hardware per storage cluster gives it the same reliability disadvantages as a DAS RAID1 mirror. The speed advantages it offers is better suited to high volume web sites and would be underutilized in our network.

About our network

•March 26, 2009 • Leave a Comment

Ed Wyse is a beauty supply wholesaler in the Pacific Northwest, servicing licensed professionals in Washington, Oregon and Idaho. We have 26 locations with a real-time point-of-sale, inventory, and payment processing system. Our satellite locations use thin clients running Fedora 8, while our main office is a mix of various operating systems.

Our network is built using IPSEC VPNs over DSL, with the as yet unrealized potential for dial-up backup.

With the exception of our mail server, which is in a colocation facility, our servers are racked in our corporate headquarters in a secure environment with redundant power with a generator backup and UPS, and redundant environmental control systems.

My goal is to work toward eliminating SPFs and provide a fully redundant cluster of systems. Physically this goal will require a storage system that is both distributed and redundant. The possibilities I considered were SAN, NAS, and DAS with raid1, raid5, raid6, drbd, dfs, iscsi, or glusterfs (or various combinations thereof).

Other physical impediments to SPF are the actual servers themselves, network architecture, and internet connection.

Logical impediments include services that don’t allow for failover or load sharing, poorly designed hardware OS’s that don’t follow specifications, and security requirements.

A history of my 0DT 0SPF project

•March 26, 2009 • Leave a Comment

Several months ago when a service core dumped and the heartbeat failover failed to fail over, I decided that I wanted to reconfigure our systems with the goal being 0 downtime. I spent several weeks researching the best way to handle such a goal and came to the conclusion that this idea is still pretty young. There are no good enterprise quality pieces that eliminate single points of failure (SPF).

In this blog, I will attempt to document the pieces I’ve reviewed and the steps required to make those pieces work.