Category Archives: IT

Hbase Slowdown and Region Inconsistencies

At work, we had an issue with Hbase where one of our regionservers was reporting its hostname and ip address to the Hbase master.  This was causing a lot of slowness to occur when regions were being accessed through the ip address and the hostname was returning the data.

First, I needed to isolate the problem to this node.  Since I knew the regionserver data is being shared on the underlying datanode, I wasn’t concerned with shutting the regionserver off.

Act I.  Shutting the regionserver off.

Well, that worked.  However, looking at the Hbase master, the ip is still checking in.  The next step is to see what kind of processes are running on the node.  ssh onto the regionserver host and run something like:

ps -ef | grep regionserver

When I did this, there was a orphaned regionserver process.  Take note of the child and parent PIDs in case you kill the process and still have an orphan.

kill <PID>

Now, I was free of any lingering regionservers.  Looking at the hbase master, the regionserver IP was finally listed on the dead nodes.

At this point, I brought the regionserver back.  However, as it came back, there were several regionservers still stuck in transition.  This is the dreaded RIT issue!

Act II.  Remove the Regions In Transition

The typical steps to fixing regions in transition are to stop hbase, remove the znodes, start hbase.

And that’s what I did.

So, I stopped hbase and watched the logs to see if there was going to be any errors.  Things shutdown rather quickly, so I logged into the Zookeeper Command Line Interface (zkcli).

hbase zkcli or zookeeper-client

in here you want to remove the /hbase znode.

rmr /hbase

Then I started hbase back up.

Act. III – Fixing Regions in Transition (Again)

So, we didn’t actually fix the regions.  Some of the regions were still tied to the orphaned regionserver process ip.  This is where you bring out the big guns.

hbase hbck -repair

When I ran this, it started to look at all the hbase tables and regionservers for inconsistencies.  This ended up repairing the regions that were tied to the orphaned process and found quite a few files that had no references at all.

In the end, these three steps fixed a majority of slowness for us.


4 Things You Must Do When Your Web Application Is Down [via Downhound]

Screen Shot 2015-09-01 at 9.10.33 AM

Downhound posted their first blog post today, and it’s an important one for Operations people to know.  Having been in operations for almost 8 years now, I’ve come to a realization that not everyone deals with outages in the same way.  There are some that will take any action possible (restart all the things) and some that investigate the problem but don’t take any action until they verify with someone.  I’ve been in both of those parties and more.

Downhound listed the following must dos:

  1. Above all, stay calm.
  2. Establish a situation room
  3. Look, but don’t touch
  4. Pause before fixing

Stressing on the need to stay calm is very important.  Junior and Senior Operations folks often get a surge of panic when a crisis hits but if you can take a deep breath and keep a cool head, figuring out the problem will be much easier.

They mention creating a new “room” for every outage.  I’ve never tried that and maybe I will some day.  However, I do like having a dedicated situation room that I can refer back to when assessing a new outage.  Slack rooms are technically free and unlimited, but I like being able to search one room for keywords and commands used previously.

The third “do” is pretty important.  This is the fact gathering stage where you figure out what is wrong with the system.  It’s important to assess why there is an issue.  Although, on one of these fact finding missions, I’ve accidentally cat’ed a large file instead of running a tail.

Pausing before fixing gives you the opportunity to think of the repercussions of your fix.  “Will there be a split brain scenario?” “What happens if you rerun that script?” “How long will this take?”  Often times that last question forces you to think about a long term vs a short term fix.  You might be able to get away with turning off puppet and manually editing a config file to get out of the hurdle and then turning around and editing the puppet files to make the config stick.

It’s also very important to remember that complaining, blaming and making fun of technical issues during the outage doesn’t help.  When SHTF you should fix the problem now and critique later.

Creating a WordPress site in Amazon Web Service’s Free Tier

Alright, so, we’ve set up our EC2 instance, and now we need to set up a database for our EC2 instance to access.

Head back to the console and select the RDS service.

Screen Shot 2013-12-12 at 8.13.33 AM

Select the “Launch a DB Instance” button.  Follow the Wizard as it makes it super easy to set up.  There are a few additional options you should make sure that you choose to avoid extra charges.

Screen Shot 2013-12-12 at 8.14.59 AM

I’m used to using WordPress with a mysql db, so I chose the mysql option.

Screen Shot 2013-12-12 at 8.15.13 AM

This isn’t really a commercial production site we’re creating and we want the free tier, so select No.

Screen Shot 2013-12-12 at 8.15.31 AM

The only changes you should make on the DB Instance Details are:

DB Instance Class: db.t1.micro

Multi-AZ Deployment: No

Allocated Storage: 5 (You can choose up to 5 GB for the free tier) It should be good enough for us.

DB Instance Identifier, Master Username, Master Password – Whatever you put in here, make sure you remember it.

Go next.




Screen Shot 2013-12-12 at 8.17.07 AM

Give your Database a name and choose the vpc and availability zone that you placed your instance in.  The reason you want them to be in the availability zone is so that you don’t incur cross AZ fees.  Imagine a datacenter, if all the traffic is within a single datacenter we don’t have to pay an outside provider to use their transfer lines, but if you need to cross to a different datacenter, you have to lease a line.  This is the same concept.


Screen Shot 2013-12-12 at 8.17.41 AM


After this, Review your setup and launch!

Next post will be about setting up an S3 repo.

Creating a WordPress site in Amazon Web Service’s Free Tier

I’ve been contemplating moving all of my hosted materials from my current hosting provider to my home server and saving on some of the costs. My sites are all relatively low traffic, so hosting on a home server would be easy. My problems are that I won’t have a dedicated IP, the server would need to be on all the time, and I’ll have to maintain the hardware. Well, since I’ve been using a lot of the AWS (Amazon Web Service) tools at work, I figured I can explore more of the options by utilizing it for personal use.

The nice thing is that AWS has a free tier for a year that I can try out. I don’t really have a plan (since this is just for fun and my own personal education) but if I did it would be something like: (1) setup instance, (2) setup wordpress, (3) write content, (4) track usage and performance, (5) evaluate. Warning, everything I write will be on-the-fly, so if you don’t understand how I got from one place to another, or want some clarity, feel free to write it in the comments and I’ll try to answer you.

Intro to AWS

First of all, make sure you sign up for an AWS account, and sign up for their free tier read more about it on .  For now, I’m going to utilize EC2 (Elastic Compute Cloud), RDS (Relational Database Service), S3 (Simple Storage Service), and EBS (Elastic Block Store).

Here are my description of those services.

EC2 – In physical computing terms, this is primarily the processing (cpu) and memory of a server.  In non-physical computing terms, the brains and the memory.

EBS – This is additional “persisitent” “hard drive” storage for the EC2 instance.  When you restart an EC2 instance, the “hard drive” will pretty much empty its contents and you start on a blank slate.  This is like having an external hard drive that you can remove from one machine and install into another.

RDS – This is the hosted MySQL part of our application.  Think of this as structured data that will help organize your content.

S3 – This is where we can store content in a really cheap way.  It also makes it a little faster to access the data since the storage is duplicated across several regions (Amazon’s version of datacenter locations).

Let’s start with these first.  There’s a whole bunch of utilities to use in a production setting, but for our simple setup this might be all we need at first.  I’ll describe additional tools as I use them.

Setting Up Your Instance

An instance in the AWS world is considered your server.  Sign into AWS, and head over to the Console (  Notice that region at the end of my link? You can choose whatever you want, but that’s what AWS defaulted me to.  The cheapest rates are usually in us-east-1.

First, let’s create the instance.  On the left of the console, click Instance.  Near the top you’ll see a button that says “Launch Instance”.  This will open up the wizard.  I’m using the generic Amazon Linux AMI, you can choose whatever Free Tier Eligible Operation System you are familiar with.  Make sure you choose t1.micro.  The defaults should pretty much be fine for now, so click “Review and Launch”, then “Launch”.

Congratulations!  You just setup a new Amazon instance.  My next post will be about setting up a database in RDS.


Setting up a Team Fortress 2 Server on Ubuntu

The install will take a while.  Near the end, we’ll end up downloading a 3.5 GB file.  I recommend screening the session, so you can close it when it starts the download and return to it later.


Download the install file:


Give permissions to execute, and execute it
This will download hldsupdatetool.bin to your current directory. Next we need to give this file execute permissions and the run it.

chmod +x hldsupdatetool.bin

Run the Steam server to pick up any new Steam updates:


Once updated, run the Team Fortress 2 server to download the TF2 server updates (this is where screen sessions become handy):

./steam -command update -game tf -dir .

Once the TF2 server downloads and updates, it’s time to setup the configuration. Here’s a link to get you started.

Now we start it:

./srcds_run -game tf +map ctf_2fort

And there you go! A simple way to start your own TF2 Server!

How-to: Automate your SSH Login with PuTTY


As many web developers can attest to, logging into your server through SSH (Secure Shell) is one of the more common day-to-day tasks (you can even use it as a secure tunnel for your traffic). It only makes sense to automate this process which in turn can save many many keystrokes.

This how-to is written with PuTTY and Windows in mind and requires several other tools that are available from PuTTY’s website. So from their download page, make sure you have these files:

  • PuTTY (putty.exe)
  • PuTTYgen (puttygen.exe)

Then to automate SSH login, do the following:

  1. Run PuTTYgen.
  2. Select SSH-2 DSA as the Type of Key to generate.
  3. Click generate and move your mouse around to generate randomness.
  4. Click “Save Private Key” and save it somewhere on your computer.
  5. Copy the entire content inside the box to your clipboard (this is your generated public key).
  6. Login to your SSH server.
  7. Create the file ~/.ssh/authorized_keys containing the generated public key(from step 3) on a single line.
  8. Make this file readable (chmod 755).
  9. Then open up PuTTY and navigate to Connection->Data and fill in the auto-login username.
  10. Navigate to Connection->SSH->Auth and under Private-key, browse to the file you had saved earlier on your computer.

That’s it! Now you can try logging in to your SSH server and it should login automatically. If it works, make sure you save your session so you don’t have to repeat these steps every time!

Hopefully these steps work for everyone! Let me know if there are any problems.

CentOS 5.5 Startup hangs at “Starting udev:”

As mentioned previously, I’m reusing a Dell Dimension E521 to run my media server in CentOS 5.5.  The funny thing is that I couldn’t start up the system because it would hang at “Starting udev”.  I looked at the boot sequence and noticed that it was trying to initialize peripheral drivers.  So, I went ahead and started disabling peripherals from the BIOS.  The sound card did the trick.

CentOS 5.5 Install Hangs At Enabling HT MSI Mapping

I’m installing CentOS 5.5 on an old Dell E521 in the hopes to use it for my new media server.  The first time I booted to the CD, I was presented with the normal Disc Boot Install options of running it graphical or text.  I chose graphical, and the system halts at a line that says: “Enabling HT MSI Mapping”.  I waited 5 minutes and there was no movement, so I restarted the system and was back at the Disc Boot Install screen.  From here, I decided to type in “linux noapic” and the install has gone through.  Just a little tip!

Ubuntu 9.10 to 10.04 update causes secondary video issues

I’ve been using Ubuntu 9.10 on my Lenovo Thinkpad T60p for a while and decided that I should update my OS to Ubuntu 10.04 to keep up with the kiddies.  My setup is having a Dell monitor attached to my Lenovo where I usually do most of my work.  After ~2-3 hours of updating packages, I was finally prompted to restart my system.  Once the 10.04 desktop came online, I noticed that there were a ton of squiggly rows on my secondary monitor which signalled a bad video output.

I decided to do a quick google search for “Ubuntu 10.04 upgrade squiggly monitor”.  This lead me to this forum post:

Several forum posters reiterated what the OP was mentioning and finally, someone had a solution:

“create file /etc/modprobe.d/radeon.conf with contents “options radeon new_pll=0 modeset=0″ and reboot.”

Boom, I did that and all is good now.  If you want to follow this bug, here is a link:


Converting file types in shell

I’ve been really wanting to convert the file type of my .cbr and create some php pages with them to view from anywhere. The first step I’m covering is how to convert the files into a compression format that I can work with.  First of all, .CBR and .CBZ files are .RAR and .ZIP, respectively.

So, starting with CBR:

for file in *.cbr ; do cp $file `echo $file | sed ‘s/\(.*\.\)cbr/\1rar/’` ; done


for file in *.cbz ; do cp $file `echo $file | sed ‘s/\(.*\.\)cbz/\1zip/’` ; done

Next time, I’ll write this in python.  Just for fun.