Ed Wyse's Guru's Blog

Using rrDNS to allow mount-failover with GlusterFS

edwyseguru — Mon, 09 Jan 2012 17:40:39 +0000

Using round-robin DNS (rrDNS) makes for an easy way to ensure your clients can mount their volumes regardless of whether a server is down. To do this, create a DNS entry that points to all of your servers. (if you’re not using nsupdate to manage your dns entries, you’re doing it wrong. )

# nsupdate -k${UPDATE_KEY}
Creating key...
> zone domain.dom
> update add glusterfs.domain.dom. 86400 A 10.0.0.1
> update add glusterfs.domain.dom. 86400 A 10.0.0.2
> update add glusterfs.domain.dom. 86400 A 10.0.0.3
> send

If this worked, you should now be able to see (note that the order may be different every time):

# host glusterfs.domain.dom
glusterfs.domain.dom has address 10.0.0.2
glusterfs.domain.dom has address 10.0.0.1
glusterfs.domain.dom has address 10.0.0.3

Now the rest is easy, use this new dns entry to mount your volumes:

mount -t glusterfs glusterfs:testvol1 /mnt/testvol1

Bad customer service

edwyseguru — Tue, 15 Nov 2011 01:17:10 +0000

I hate being treated like a newb about as much as I hate support people reading half of a support ticket and dismissing it as invalid.

ME:

I’ve tried sending dtmf via SIP INFO or via RFC2833, and neither works consistently. If I dial my POTS line, and dial digits on the voip phone keypad, I can see sip info packets if I set it to info, or rtp rfc2833 packets if I set it that way, but on the pots phone, I hear maybe 1 in 10 tones.

THEM:

INFO DTMF is not supported on our service for calls to or from the PSTN. You must use RFC2833. RFC2833 is fully supported by all of our underlying carriers for both inbound and outbound service, so if you are having a problem, the issue is likely between your SIP phone and the PBX.

When I called 12066230560 and entered 256 it immediately interrupted the IVR. I then received an ‘Incorrect Entry’ response. When I enter #1, I am able to reach the directory.

This call was placed from my Polycom phone and an MX system that is configured to utilized RFC2833 only.

Thanks,

ME:

As I said, I get the same results either way.

THEM:

The problem, as was noted in the previous response, is likely related to the DTMF relay between your phone and your PBX, as this issue cannot be reproduced from our PBX when calling your POTS line.

In addition, what you hear when pressing tones should not make any difference as RFC2833 is out-of-band and has nothing to do with the audible tones.

Thanks,

ME:

When you say, “between your phone and your pbx”, are you referring to my Polycom phone to my Freeswitch PBX, or my Panasonic phone to my KSU? If the former, then that would prevent the RFC2833 packets from showing up in a packet capture that I’ve reviewed with Wireshark between my Freeswitch PBX and your Acme Packet SBC. If the latter, then I would have the same results utilizing the IVR as you.

When referring to what I hear, I was referring to the Panasonic phone on the POTS KSU. If I’m not hearing DTMF at the POTS end and the RFC2833 is leaving my Freeswitch PBX correctly then the only possibility that I can imagine is that the problem is within your realm of responsibility.

THEM:

I am referring to the Polycom to Freeswitch connection. The inbound leg of the call to the POTS line is the same for my call tests and your call tests. I do not have any problems originating traffic via nexVortex and terminating to your POTS line and then relaying DTMF.

Are there other numbers that you have problems passing DTMF to?

Thanks,

ME:

Well then, as I said in my very first message, “I can see … rtp rfc2833 packets … but on the pots phone, I hear maybe 1 in 10 tones.”

Yes, I cannot use DTMF consistently anywhere that we’ve tried that uses an IVR system. The bank, the alarm company… even just calling a POTS line with an old-fashioned simple handset.

I mean really…. If you’re going to answer a question, at least make an attempt at understanding it first. I give way better support than this over IRC to people that I have no financial interest in helping. You’d think that the vendor that gets money from us every month could do a bit better.

Gluster Aquired by RedHat for $136M

edwyseguru — Tue, 04 Oct 2011 20:05:45 +0000

In the latter part of 2008, we had a server that started to have problems. That’s when we discovered that DRBD would fail miserably if you tried switching the active state back and forth rapidly. This caused a split-brain situation and we had our entire company down for half a day while I got critical systems back up, and I spent more than a month tracking down everything that was botched.

I had to find a new solution.

By March of 2009, I had found and learned how to use GlusterFS. This was no easy task, as it turned out. The documentation was hit and miss. Settings may or may not be for the current version. Terms were used incongruously. Though they had a mailing list, questions to that list went unanswered. The IRC channel was silent.

Eventually, I figured it out on my own and decided to take my newfound knowledge and edit the wiki, and help people in the IRC channel. Now, if you’ve ever tried to get support from an IRC channel, most of the time you get surly narcissists who feel that demeaning people is somehow helpful. I vowed that our channel would be different. My narcissism would not be surly.

I have no idea over the past two and a half years how many people I helped, but I do know that my example and influence have created one of the nicest and most helpful IRC support channels I’ve ever encountered. This in large part is thanks to the wonderful people that have joined me on #gluster: Samuli ‘samppah’ Heinonen, Louis ‘semiosis’ Zuckerman, Jeff ‘jdarcy’ Darcy, Adam ‘m0zes’ Tygart, Todd ‘Heebie’ Hebert and others.

Today, RedHat and Gluster announced the acquisition. Anand Babu (AB) Periasamy, Gluster CTO stated in the webcast, “The community was central to our success.” Brian Stevens, CTO and VP, Worldwide Engineering, Red Hat also acknowledged the community was a part of RedHat’s decision in his announcement.

I’ve been assured that the check is in the mail for my part in this acquisition, but that the envelope and postage have more value than the check.

All kidding aside, my sincere congratulations to everybody, employee, community and end-user that’s made Gluster such a success. And congratulations to RedHat on filling this gap in their enterprise offerings.

GlusterFS 3.1.6 performance tests

edwyseguru — Fri, 16 Sep 2011 23:53:47 +0000

Testing on our Drupal 6 intranet server. Simple ab test.

Volume Name: intranet Type: Distributed-Replicate Status: Started Number of Bricks: 4 x 3 = 12 Transport-type: tcp Bricks: Brick1: ewcs2:/var/spool/glusterfs/a_intranet Brick2: ewcs4:/var/spool/glusterfs/a_intranet Brick3: ewcs7:/var/spool/glusterfs/a_intranet Brick4: ewcs2:/var/spool/glusterfs/b_intranet Brick5: ewcs4:/var/spool/glusterfs/b_intranet Brick6: ewcs7:/var/spool/glusterfs/b_intranet Brick7: ewcs2:/var/spool/glusterfs/c_intranet Brick8: ewcs4:/var/spool/glusterfs/c_intranet Brick9: ewcs7:/var/spool/glusterfs/c_intranet Brick10: ewcs2:/var/spool/glusterfs/d_intranet Brick11: ewcs4:/var/spool/glusterfs/d_intranet Brick12: ewcs7:/var/spool/glusterfs/d_intranet

Defaults

$ ab -n 100 -c 5 http://intranet/ This is ApacheBench, Version 2.3 Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking intranet (be patient)…..done

Server Software: Apache/2.2.3
Server Hostname: intranet
Server Port: 80

Document Path: /
Document Length: 16883 bytes

Concurrency Level: 5
Time taken for tests: 113.311 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 1733000 bytes
HTML transferred: 1688300 bytes
Requests per second: 0.88 [#/sec] (mean)
Time per request: 5665.557 [ms] (mean)
Time per request: 1133.111 [ms] (mean, across all concurrent requests)
Transfer rate: 14.94 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 27 58.8 2 281
Processing: 2141 5593 2214.2 4936 11377
Waiting: 1985 5436 2175.5 4762 11001
Total: 2141 5620 2213.0 4941 11378

Percentage of the requests served within a certain time (ms)
50% 4941
66% 5546
75% 6706
80% 7483
90% 9540
95% 10590
98% 11245
99% 11378
100% 11378 (longest request)

With performance.cache-refresh-timeout set to 30

$ ab -n 100 -c 5 http://intranet/ This is ApacheBench, Version 2.3 Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking intranet (be patient).....done Server Software: Apache/2.2.3 Server Hostname: intranet Server Port: 80 Document Path: / Document Length: 16908 bytes Concurrency Level: 5 Time taken for tests: 124.092 seconds Complete requests: 100 Failed requests: 45 (Connect: 0, Receive: 0, Length: 45, Exceptions: 0) Write errors: 0 Total transferred: 1734285 bytes HTML transferred: 1689585 bytes Requests per second: 0.81 [#/sec] (mean) Time per request: 6204.579 [ms] (mean) Time per request: 1240.916 [ms] (mean, across all concurrent requests) Transfer rate: 13.65 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 15 33.8 1 184 Processing: 3521 6189 2428.1 5970 13399 Waiting: 3214 6031 2442.0 5861 13288 Total: 3522 6204 2427.7 5971 13399
Percentage of the requests served within a certain time (ms) 50% 5971 66% 6533 75% 7080 80% 7501 90% 9621 95% 12271 98% 12967 99% 13399 100% 13399 (longest request)

Using the local nfs layer method: http://community.gluster.org/p/nfs-performance-with-fuse-client-redundancy/

$ ab -n 100 -c 5 http://intranet/ This is ApacheBench, Version 2.3 Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking intranet (be patient).....done Server Software: Apache/2.2.3 Server Hostname: intranet Server Port: 80 Document Path: / Document Length: 16926 bytes Concurrency Level: 5 Time taken for tests: 87.674 seconds Complete requests: 100 Failed requests: 0 Write errors: 0 Total transferred: 1737300 bytes HTML transferred: 1692600 bytes Requests per second: 1.14 [#/sec] (mean) Time per request: 4383.720 [ms] (mean) Time per request: 876.744 [ms] (mean, across all concurrent requests) Transfer rate: 19.35 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 18 56.3 1 490 Processing: 917 4344 3324.3 3495 16214 Waiting: 719 4227 3312.7 3445 16031 Total: 955 4363 3325.0 3498 16215
Percentage of the requests served within a certain time (ms) 50% 3498 66% 4357 75% 4706 80% 4909 90% 10750 95% 13920 98% 15947 99% 16215 100% 16215 (longest request)

Unified File and Object Storage on GlusterFS 3.1 Howto

edwyseguru — Fri, 26 Aug 2011 00:20:48 +0000

This tutorial expects that you know how to install a working GlusterFS filesystem and add the swift components.[1]

Create a volume for storing the authentication data. This must be named “auth”.

gluster volume create replica 2 auth server1:/data/auth server2:/data/auth
Create a volume for your storage. This will be referred to as an “account”.

gluster volume create replica 2 account1 server1:/data/account1 server2:/data/account1
Run gluster-object-config

# /usr/bin/gluster-object-config
Enter external IP for Storage-Server: 192.168.0.10
Enter Super Admin key: P-assword-1
Enter FileSystem (Press Enter for default Glusterfs):
Enter FileSystem (Press Enter for default Glusterfs):
Enter Object server port (Press Enter for default 6010):
Enter Container server port (Press Enter for default 6011):
Enter Account server port (Press Enter for default 6012):
Enter ’1′ for https ’2′ for ‘http’ (Recommended is https):2
Start the object service

# gluster-object-start
Prepare the authentication store

# gluster-object-prep -A https://192.168.0.10/auth/ -K P-assword-1
Add an administrative user to the account

# gluster-object-add-user –admin -A https://192.168.0.10/auth/ -K P-assword-1 account1 adminuser1 adminpass1

To test, use “st” from the openstack-swift package:

# st -A https://192.168.0.10/auth/v1.0 -U account1:adminuser1 -K adminpass1 stat
Account: AUTH_bc4de365fd9e8a6a262d99056f5bd47f
Containers: 0
Objects: 0
Bytes: 0

That’s all there was to it.

[1] If you don’t, come see us on IRC – #gluster on freenode.

Very minimal CentOS image howto

edwyseguru — Thu, 07 Oct 2010 15:33:44 +0000

I’ve seen it said by someone at Redhat, “The favorite whipping boy around here is anaconda. Every slow or failed install is anaconda’s fault. Never mind that someone here wrote it,” and I agree. I don’t like the anaconda installs, I think even the minimum install is bloated for a minimalist VM approach, so here’s how I like to make my minimal CentOS install.

First, download the current centos-release and centos-release-notes for your architecture from your favorite mirror. I’m doing this with Fedora 13, adjust for your favorite flavor.

su -
TARCH=x86_64
wget ftp://mirrors.kernel.org/centos/5.5/os/${TARCH}/CentOS/centos-release-5-5.el5.centos.${TARCH}.rpm
wget ftp://mirrors.kernel.org/centos/5.5/os/${TARCH}/CentOS/centos-release-notes-5.5-0.${TARCH}.rpm

Next, let’s create a work directory to do this under. I have lots of space in my home directory, so I’m going to use that:

WKDIR=/home/jjulian/centos_image
mkdir -p ${WKDIR}/image
cd ${WKDIR}

We’re going to use yum to do our install, so we’re going to create a custom yum config:

cat >yum.conf <

We initialize the rpm database and install the centos-release rpms:

rpm -r ${WKDIR}/image --initdb
rpm -ivh --nodeps -r ${WKDIR}/image centos-release*rpm

Now we just install the yum and rpm using yum:

yum -c yum.conf --nogpgcheck install rpm yum

I’m sure there’s some way to leave gpgcheck enabled, but I haven’t looked for it. If someone wants to figure that out and let me know, I’ll update my article.

Since we created the rpm database using our parent OS, it may not be using the correct db version, so let’s recreate it using the tools we just installed. We’re going to reinstall using these tools, so we’ll need the rpms we downloaded earlier as well.

cp centos-release*rpm ${WKDIR}/image/root
cp /etc/resolv.conf ${WKDIR}/image/etc/
mount -t proc none ${WKDIR}/image/proc
chroot ${WKDIR}/image
rm -rf /var/lib/rpm/*
rpm --initdb
rpm -ivh --nodeps ~/centos-release*rpm
yum -y groupinstall Base
yum clean all

So now we have a fairly minimalist centos tree (397 packages) weighing in at about 820 Mb. You can install other packages now, set up some initial configurations, etc.

You could have an even slimmer install if you wanted to be more selective. I find the Base group to be sufficiently slim for most cases.

If you’re using pxe for nfs rooted VMs, then your image is done. Move the image where you want it, export it and see the next article about nfs rooted vm’s.

If you need a bootable grub image for a stand-alone vm, ask me and if I get enough requests I’ll write about how to do that too.

GlusterFS in an Enterprise setting

edwyseguru — Fri, 11 Jun 2010 18:07:03 +0000

About a year ago I began my experiment with GlusterFS as a 0DT SAN. There were a few hiccups at the beginning, and a couple of version changes to get to the point where I consider it stable, but now I don’t know how anyone lives without it.

My test configuration started with 3 servers and 4 drives each. I created single LVM volumes on each drive, consisting of half the drive so I have room for snapshots. I mount them each under /cluster as 0,1,2, and 3 and share them using glusterfsd.

volume posix0
type storage/posix # POSIX FS translator
option directory /cluster/0 # Export this directory
end-volume

volume locks0
type features/locks # Implement posix locks, not working 100%
subvolumes posix0
end-volume

volume brick0
type performance/io-threads # Performance enhancement
option thread-count 8
subvolumes locks0
end-volume

volume posix1
type storage/posix
option directory /cluster/1
end-volume

volume locks1
type features/locks
subvolumes posix1
end-volume

volume brick1
type performance/io-threads
option thread-count 8
subvolumes locks1
end-volume

volume posix2
type storage/posix
option directory /cluster/2
end-volume

volume locks2
type features/locks
subvolumes posix2
end-volume

volume brick2
type performance/io-threads
option thread-count 8
subvolumes locks2
end-volume

volume posix3
type storage/posix
option directory /cluster/3
end-volume

volume locks3
type features/locks
subvolumes posix3
end-volume

volume brick3
type performance/io-threads
option thread-count 8
subvolumes locks3
end-volume

volume server
type protocol/server
option transport-type tcp
subvolumes brick0 brick1 brick2 brick3
option auth.addr.brick0.allow *
option auth.addr.brick1.allow *
option auth.addr.brick2.allow *
option auth.addr.brick3.allow *
end-volume

Each of the clients then connects to the 3 servers and handles AFR and distribution, effectively doing a raid 1(3)+0(4).

volume fs1_cluster0
type protocol/client
option transport-type tcp
option remote-host fs1.ewcs.local
option remote-subvolume brick0
end-volume

volume fs1_cluster1
type protocol/client
option transport-type tcp
option remote-host fs1.ewcs.local
option remote-subvolume brick1
end-volume

volume fs1_cluster2
type protocol/client
option transport-type tcp
option remote-host fs1.ewcs.local
option remote-subvolume brick2
end-volume

volume fs1_cluster3
type protocol/client
option transport-type tcp
option remote-host fs1.ewcs.local
option remote-subvolume brick3
end-volume

volume fs2_cluster0
type protocol/client
option transport-type tcp
option remote-host fs2.ewcs.local
option remote-subvolume brick0
end-volume

volume fs2_cluster1
type protocol/client
option transport-type tcp
option remote-host fs2.ewcs.local
option remote-subvolume brick1
end-volume

volume fs2_cluster2
type protocol/client
option transport-type tcp
option remote-host fs2.ewcs.local
option remote-subvolume brick2
end-volume

volume fs2_cluster3
type protocol/client
option transport-type tcp
option remote-host fs2.ewcs.local
option remote-subvolume brick3
end-volume

volume fs3_cluster0
type protocol/client
option transport-type tcp
option remote-host fs3.ewcs.local
option remote-subvolume brick0
end-volume

volume fs3_cluster1
type protocol/client
option transport-type tcp
option remote-host fs3.ewcs.local
option remote-subvolume brick1
end-volume

volume fs3_cluster2
type protocol/client
option transport-type tcp
option remote-host fs3.ewcs.local
option remote-subvolume brick2
end-volume

volume fs3_cluster3
type protocol/client
option transport-type tcp
option remote-host fs3.ewcs.local
option remote-subvolume brick3
end-volume

volume repl0
type cluster/replicate
subvolumes fs1_cluster0 fs2_cluster0 fs3_cluster0
end-volume

volume repl1
type cluster/replicate
subvolumes fs1_cluster1 fs2_cluster1 fs3_cluster1
end-volume

volume repl2
type cluster/replicate
subvolumes fs1_cluster2 fs2_cluster2 fs3_cluster2
end-volume

volume repl3
type cluster/replicate
subvolumes fs1_cluster3 fs2_cluster3 fs3_cluster3
end-volume

volume distribute
type cluster/distribute
subvolumes repl0 repl1 repl2 repl3
end-volume

volume writebehind
type performance/write-behind
option aggregate-size 32MB
option cache-size 64MB
subvolumes distribute
end-volume

volume ioc
type performance/io-cache
option cache-size 64MB
subvolumes writebehind
end-volume

This configuration allows for any 2 drives in a replicate set to fail, or even any 2 servers, without loss of operation. Only 1/6th of the entire available storage capacity is shared, but with the dramatically reduced cost of storage over other redundant systems, this is a negligible problem.

Here’s a graphic representation of the configuration:

Held hostage by Worldlink, Inc.

edwyseguru — Tue, 28 Apr 2009 23:52:34 +0000

I, personally, have been a long time customer of Worldlink, Inc as a co-location provider. They’ve always had issues, like not being available for reboots on the weekends, not having adequate power to support the equipment in the event of a power outage, not honoring their uptime guarantee. But they were cheap, so I expected a few issues.

Then they raised their prices and I started shopping. I didn’t put much effort into it though, as I had a million other things that were more important. Then Lori, the “billing” person, goes on vacation and the Senior Systems Administrator is left to take my payment. He goofs and I see a double-bill on my account. I send this email to “billing” on April 9th.

It appears that you’ve double charged me for the 207.90. I will be removing my server from your facility on April 25th at 10:00pm. Please make the necessary arrangements.

Joe Julian

I had also spoken with Rick on the phone to that effect, and when Lori returned, I also told her that I was canceling and she acknowledged receipt of that email.

I call on Friday the 24th to confirm my appointment the next day to pull the servers. I talk to Ryan who says he has to check with his boss about it and will call me back. He gets my number and reads it back to me. I point out that 5:00 is quickly approaching and suggest he should get back to me before then. He assures me that he’ll call me back either way.

Either way? I have a contract that states that it allows 24x7x365 access.

I don’t get a call back.

Although I’ve got my phone with me all day Saturday to ensure no missed calls with this changeover rapidly approaching, I receive none. We run some errands, watch a movie while eating dinner. I check my email at about 8:30 and notice this email had arrived about 5:31:

Joe,

Rick here. You called Friday as I was on my way out and you talked to Ryan. He said you were going to call back a little before 5PM. I shot off a quick email to the powers that be to check and make sure that pickup of the server was OK, and they responded in the negative. Apparently, they have yet to receive a proper cancellation in 30 days in advance.

It puts me in an uncomfortable position to say so, but I guess I get to be the voice here… No one can meet you at 10:30 tonight to remove the gear. What you REALLY need to do is do the cancellation thing. I’d say your best shot is to call Monday through Friday between noon and 3, hit zero, and pursue with them. I know those are some pretty slim hours, but they’re actually there a lot more… I’m just trying to give you the best shot.

In any case, I know what they’re going to do is to have you FAX or postal mail a written cancellation in. It should contain the date, your name, desire to cancel and remove equipment, and the effective date (ASAP).

Like I said, I’m not thrilled to pass along billing news like this to ANY customer, but I felt obligated to fire off some kind of update.

Thanks,

Rick Kunkel
Worldlink DNS Administrator
(206)361-8785

First of all, what does cancellation have to do with access? As a matter of fact, it should mean just the opposite. I head down there as I begin making phone calls. I call the “emergency” number. The “emergency” extension. Rick’s home. I’m sending emails. Find Rick’s Facebook account and message him there. Nobody’s responding. I continue to try to get a response, calling every 15 minutes to no avail. Finally, at 2:30am, I am defeated. I go home.

I awake the next morning and begin again at 8:00. I call my colleague Brad who rents a half-rack and hosts the Ed Wyse email server. He schedules to meet me there at 6:00pm and at least get the Ed Wyse box out.

About mid day I get a reply on Facebook from Rick:

Joe,

I talked with my bosses, and my hands are tied. I do not like being in this position. They say that you will have to talk to billing on Monday, and there’s nothing I could or should do. I’m sorry.

If your machine is down, is there something that we could do to reset it? I am away this weekend, but it’s possible that Ryan could reset. I have been instructed to not allow access at this point.

–Rick

At 6:00, I meet with Brad. As we’re headed for the door I think, “Hey! I’ve got an idea!” I have Brad wait a moment while I shut down the server from my laptop (already booted in the car) so that I don’t have to plug in the monitor and keyboard to shut it down. We then go over to the door. Brad swipes his RFID key and the light blinks… Red. It’s supposed to blink green for access. He tries again, still red. He tries calling the emergency line, and Rick’s cell phone. Nobody returns his calls. He too has asked to cancel his contract at the end of the month, but again is supposed to have unlimited access. With no way to get in to the building, Brad leaves. I have another down server that I can’t access.

I start to leave then decide I’m going to take this further. I pull into a parking lot and start doing searches. I remember that they’re a corporation so I look up the corporate record at the Secretary of State’s web site and get the names of the principals, Lori and Darwin Hill. I start searching for their names and discovered this juicy tidbit:

FOR IMMEDIATE RELEASE
January 14, 2004
Gregoire Settles with Worldlink

SEATTLE — Attorney General Christine Gregoire announced a settlement today with Worldlink, Inc., a Shoreline-based Internet provider accused of engaging in unfair business practices.

Worldlink, run by the husband and wife team of Darwin and Lori Lee Hill, offers Internet services including dial-up accounts and DSL service. Under the settlement, entered in King County Superior Court, the company has agreed to pay $17,876 in restitution, civil penalties and fees and stop the business practices alleged to be unfair by the Attorney General in the complaint.

Worldlink is accused of providing poor service to customers, including charging for services not provided, failing to cancel consumers’ accounts after representing they would do so, placing charges on consumers’ credit cards or debit cards without authorization, placing numerous unfair restrictions on cancellation, and financially penalizing consumers who canceled. Worldlink is also accused of charging customers for service they could not deliver because of technical problems.

As part of the agreement, Worldlink must adhere to specific rules of fair business operation. Those rules include providing adequate notice of contract renewals and charges, canceling customer accounts when requested within a specific period of time, resolving customer complaints in a timely, professional manner and maintaining a record of consumer complaints and resolutions.

Worldlink is also required to cease billing consumers who were subjected to the alleged unfair practices and provide adequate information to any collection agencies or credit reporting agencies whose records may incorrectly reflect the consumer’s obligation to pay.

The case was handled by Gregoire’s High Tech Unit.

and

Settlement announced with Internet provider

OLYMPIA — The Attorney General’s Office yesterday announced a settlement with a Shoreline-based Internet provider accused of engaging in unfair business practices such as placing charges on consumers’ credit cards without authorization.

The company, Worldlink, has been operating since 1995 and has about 10,000 dial-up and high-speed subscribers, according to Darwin Hill, co-owner with his wife, Lori Hill. Darwin Hill said the decision to pay the state nearly $18,000 in restitution, civil penalties and fees and to adhere to certain business practices was not an admission of wrongdoing.

Rather, he said, it was a business decision to avoid continuing struggles with the Attorney General’s Office. The office said it had collected about 30 consumer complaints.

It accused Worldlink of providing poor service, including charging for services not provided, failing to cancel consumers’ accounts after saying it would, and financially penalizing customers who canceled.

A consent decree commits Worldlink to providing adequate notice of contract renewals and charges, among other rules of operation.

Oh, crap!

I knew I had to make this happen quickly.

I had a public records search run on them and got their phone number and address.

I tried the phone numbers, and one of them had a recording that was Lori’s voice. I left a message explaining that if this wasn’t taken care of right away that I was going to find out who the investigator was and let them know that they’re up to the same old tricks.

No reply.

I then decided to drive over to their house.

They live on 145th, not far from the freeway. As there’s no street parking there, I had to park about a block away. I walked over, went up the driveway and knocked on the door.

Darwin answered the door, and I could see that they had sat down to dinner. I asked if I should come back after dinner, and he told me that it was okay and asked me what I wanted. I told him my name (which I know for a fact he knows who I am) and he gave me a blank look. In fact, he was so committed to not knowing what I was talking about, I think if I had asked, he would have denied knowing what Worldlink was. But I didn’t let him. I told him that Rick had talked to either himself or Lori and had been told I wasn’t to be allowed entry. I told him that it “feels” like the same thing’s happening again that they got in trouble for back in 2004, but that I was sure that couldn’t be the case (again, he denies of knowledge of what I’m talking about). He asked what “Rick” I had talked to, I told him. He said he would call Rick and have him call me. I thanked him and walked away.

I knew he was lying. He was lying about what he knew. He was lying about calling Rick. I went back to my car and waited.

After 25 minutes, I walked back to the front door. All the shades were pulled. The lights were all off.

I knocked.

I waited.

2 minutes pass and just as I touch the door to knock again LOUDER, my phone rings. It’s Rick. Apparently, Darwin didn’t like me calling his bluff.

Less than a half hour later, I’m back at the colo. Ryan arrives with his dad and helps me remove the rails from the rack as I remove the servers.

The rest was easy.

We’re now in Netriver‘s Colocation facility. It has top-of-the line power facilities and environmental design. The rest has yet to be tested, but they look like there’s a lot more behind them than Worldlink seemed to have.

Bottom line, avoid Worldlink, Inc of Shoreline, Wa at all costs.

Commercial Quality Helpdesk and Resource Management System

edwyseguru — Thu, 16 Apr 2009 22:16:27 +0000

I’ve switched us over to GLPI for our helpdesk ticket management system, and I am a new fan.

I found GLPI while looking for a piece of software to manage our resources. We had a series of failures of our Zyxel Prestige 662HW-61′s where the bridge chip would start dropping packets. Once they started dropping them, they would just keep getting worse until they were completely unusable. As we have over 25 of them that were all approaching the end of their warranty, we were a bit concerned. So far, Zyxel has taken good care of us. As a result of needing to keep track of which modems had been changed out, which were in transit, and which were good – I looked for a better tool than a spreadsheet.

I found GLPI. Not only does it keep track of all the hardware, their locations, purchase dates, warranty periods, applicable contracts, etc., it also has this nice helpdesk system that integrates with the resources. Now trouble tickets are attached to the resource that caused the problem, greatly improving the ability to identify cronic equipment problems.

Like any software packages, there are a few deficiencies. This package is written by a french group (one of which is Remi Collet, of the remi repository fame). Some of their english translations are a bit lacking. They are, however, quite willing to update the dictionaries with corrections when submitted. There’s a feature to authenticate users against imap accounts that is supposed to be able to be configured to use a preset domain name. The setup for that is there, but the execution was missing. Lastly, there’s a “network links” system that, when complete, will save a lot of time tracing down wires. Unfortunately, it’s only half implemented.

Overall, this is a very powerful piece of software that will greatly simplify management of my helpdesk resources, and our equipment.

SWITCHed to freeSWITCH

edwyseguru — Mon, 06 Apr 2009 15:14:11 +0000

The evaluation and changeover is complete from Callweaver to freeSWITCH. Callweaver didn’t do what we needed. freeSWITCH did. Not that difficult of a decision to make.

The freeSWITCH learning curve isn’t all that steep. Yes, it’s a complete departure from what you were used to with Asterisk/Callweaver, but it’s far more logical and can be as detailed as you like. Unlike CW/* dialplans are processed in the order that you load them. Put your specific dialplans first, then drop through to your more and more generic ones. It’s simple.

The support I’ve received from the respective IRC channels is beyond comparison. In CW, my responses were rare and frequently pointed me to things I should try programming myself, as the things I needed didn’t exist. In the freeSWITCH channel, not only were people personable and helpful, they were there all the time! Newbies, though referred to the wiki, are told which pages of the wiki hold the information that’s useful to them. They’ll even help you brainstorm methods for getting the resultant dialplan you’re trying to accomplish, should you be trying to do something “interesting”.

And the best part, we have working DTMF! We can now voice authorize credit card transactions if necessary. A feature that was going to kill this project and cause the principals of this company to pull the VoIP system and put the old Panasonic switches back in. Now we can move forward and save tens of thousands of dollars per year by switching to VoIP comany wide.

Thank you to everyone in #freeswitch, and to the developers that made this possible.