Building Chess.com - Part 7: Grass is Always Greener

In high school I was a parking attendant, then a lifeguard, and then a locksmith. Then briefly after college I had a job as a "marketing guy" (no idea what that means), a job teaching chess to kids, and then an internship last summer as a controller for a software startup (I built excel models...zzzzzz). But I mostly feel like I have been working for myself for the last 8 years. And let me tell you - the grass is ALWAYS greener. 

Think your job is BORING? You might be right. But ask yourself if this is the kind of excitement you want:

Late last night we brought the site down temporarily so we could add a bunch of new features that would #1 make the site faster, #2 allow us to start adding more web and database servers, and #3 have a perfect backup of our database (which stores EVERYTHING about the site). We finished the upgrade, put the site back online, and it was SCREAMING fast. I went to bed feeling SO happy that our hardware/database issues were finally behind us.

This morning at 9am (my time) I am on the site and suddenly a page doesn't load. Then my cellphone starts beeping with SMS alerts that Chess.com is down. Uh oh! Then I start calling Jay and Igor to see what is going on. They hadn't received any SMS alerts because my SMS alert account was empty. :( By now I am so stressed that I feel sick to my stomach. Why? Well, in a normal world and with a normal job when you make a mistake maybe a few people get upset. But when you run a website that 40,000+ people visit each day, the weight of knowing how many people might be upset, disappointed, frustrated, or otherwise unhappy is totally paralyzing. Maybe an appropriate analogy is you are in a stadium full of 40,000 people who have come to see a concert and you are the one who promised them that their favorite band would be there, but then they never show up. And the catering company doesn't have enough food. And you are standing in the middle of it totally helpless to do anything, but still feeling complete responsibility for it all. 

Anyway, Jay and Igor hop on and find that the database had crashed for some unknown reason. They restart the server and all is well. I'm starting to feel better, though somewhat sad about the people who probably came to the site, saw nothing, and left. 

Two hours later the site goes down AGAIN! And this time it goes down completely. Hard. For the count. We can't reboot the machine. It says "Disk mount failure" which is just a really bad message - not something you want to hear. But hey, no problems, right? We have perfect replication of data! So we get the other database all lined up and ready to launch so we can bring the site up. Then we notice that the data is looking weird. The database was 2 hours old! The replication failed when the first crash happened. That means that anyone who signed up in the last 2 hours was no longer a member. Games that had been started? GONE. Moves made in the last 2 hours? GONE.

By now I have two thoughts: #1 what kind of ulcer is this going to cause me, and #2 I wonder what kind of job I could get. Ok, I'm exaggerating kind of. I'm not going to get an ulcer, but I really did stop and think "Why am I doing this?? Why am I taking on all of this stress?? Why not just get a job??"" The grass is greener when you don't have the weight of the world on your head, right?

Then things went from bad to worse. Not only was that database old, but then we couldn't access the new machine. Finally after 90 nail-biting minutes we were able to recover the original database that was 100% accurate. We tested it, brought it back up, and finally, three agonizing hours later, got the site back up and running. 

So now this very moment while I am typing this up I am on hold with Dell so we can RMA (return) one of our servers (which I paid $3,000 out of pocket for) and get some new ones up and running. My neck hurts from holding the phone, my heart hurts from thinking of how many people I disappointed today, my head hurts from thinking about how to get all of this fixed, and my stomach hurts from the stress.

Jay and I try to keep our senses of humor while this is going on. We joke about calling up Google and getting job interviews. But when we really think about it, we're not  sure we're fit for jobs.  Sure they have better food, better benefits, more money, fancy company shirts, less stress, etc. And yes, the grass LOOKS greener. But then I hear so many people with jobs saying they want to start their own businesses and I think "you're crazy! don't do it! it is a living hell sometimes!". And that is the truth. The pay stinks. The hours stink. The lows are REALLY low. But then again, the highs can be really high. 

The grass is always greener...

Until next time,

- Erik 

p.s. RIGHT as I am about to submit this story the site goes down again!!! And I'm thinking "that's it! i quit!!" Turns out it was the hardware guy in the datacenter who is trying to fix our other machine - he accidentally unplugged the wrong server :D