Hbase Slowdown and Region Inconsistencies

At work, we had an issue with Hbase where one of our regionservers was reporting its hostname and ip address to the Hbase master.  This was causing a lot of slowness to occur when regions were being accessed through the ip address and the hostname was returning the data.

First, I needed to isolate the problem to this node.  Since I knew the regionserver data is being shared on the underlying datanode, I wasn’t concerned with shutting the regionserver off.

Act I.  Shutting the regionserver off.

Well, that worked.  However, looking at the Hbase master, the ip is still checking in.  The next step is to see what kind of processes are running on the node.  ssh onto the regionserver host and run something like:

ps -ef | grep regionserver

When I did this, there was a orphaned regionserver process.  Take note of the child and parent PIDs in case you kill the process and still have an orphan.

kill <PID>

Now, I was free of any lingering regionservers.  Looking at the hbase master, the regionserver IP was finally listed on the dead nodes.

At this point, I brought the regionserver back.  However, as it came back, there were several regionservers still stuck in transition.  This is the dreaded RIT issue!

Act II.  Remove the Regions In Transition

The typical steps to fixing regions in transition are to stop hbase, remove the znodes, start hbase.

And that’s what I did.

So, I stopped hbase and watched the logs to see if there was going to be any errors.  Things shutdown rather quickly, so I logged into the Zookeeper Command Line Interface (zkcli).

hbase zkcli or zookeeper-client

in here you want to remove the /hbase znode.

rmr /hbase

Then I started hbase back up.

Act. III – Fixing Regions in Transition (Again)

So, we didn’t actually fix the regions.  Some of the regions were still tied to the orphaned regionserver process ip.  This is where you bring out the big guns.

hbase hbck -repair

When I ran this, it started to look at all the hbase tables and regionservers for inconsistencies.  This ended up repairing the regions that were tied to the orphaned process and found quite a few files that had no references at all.

In the end, these three steps fixed a majority of slowness for us.


King of Tokyo Board Game Review

A while back, I wrote a review on the board game King of Tokyo on Amazon.  I just checked it today and noticed that 200 people found my review helpful!  I still feel like this is an excellent game for those of you looking for a fun casual game to play with your group.

By Kingston Lee on November 26, 2012

King of Tokyo is a dice and card game where you are one of 6 Monsters trying to take over Tokyo. I found this to be a very fun and easy to learn (and teach) family game. There is a lot of luck (and some strategy) involved so anyone can win.

There are two ways to win:
1. Get the most Victory Points
2. Be the last monster standing

How a turn works:
1. Roll your dice
2. Re-roll the dice you do not want
3. Re-roll the dice you do not want (again!)
4. Count the results of your roll.

The roll can either give you energy points, victory points, heal your monster, or attack other monsters.
If you roll energy points, you can purchase upgrade cards to give your monster extra abilities to gain an advantage over your foes.
If you roll victory points, you get closer to winning the game by points.
When rolling attacks, there are specific rules that you should remember. When standing in Tokyo, you attack all the monsters outside of Tokyo. When standing outside of Tokyo, you attack the monster in Tokyo. Only one monster can be in Tokyo at a time (unless playing with 5-6 players, there’s a second area of Tokyo to rule).
When rolling heals, you can only heal when you are outside of Tokyo. This makes it more difficult for the guy trying to stay as the ruler of Tokyo.

How to gain victory points:
1. Roll 3 of a kinds with the numbered dice
2. Take over or stay in Tokyo
3. Buy cards that add victory points

Final words
If you decide to purchase this game, remember that the replay value can be very limited depending on the group you are playing with. I found that the first 20-30 times I played the game, the gameplay was fun because of the different cards. Game groups will love to play this before and/or in-between really intense sessions of longer games like Settlers of Catan, Battlestar Galactica, or Arkham Horror. Families will love playing this game with their kids.

Serial Podcast is back!

I’m super excited that Serial has returned for a second season. This time around, it’s a recent mystery about Bowe Bergdahl, a soldier who allegedly deserted his post and walked into enemy territory. Originally celebrated a hero, he became a villain as accounts of his desertion flooded the news.  A few questions come to mind. Will Sarah Koenig have a crush on Bowe? Is Mail Kimp going to support this one? It’s time to start the first episode.

Continue reading Serial Podcast is back!

Coffee: Ritual Seasonal Espresso – Dark Horse (11/2/2015, 11/14/2015, 11/23/2015)

I’ve been making a latte every morning since Sam surprised me with a La Pavoni for my birthday in 2012.  I’m finally getting around to taking notes on all the different bags of beans we’ve been getting.  I’ll be backfilling the notes I’ve been taking from a notebook, so some of the roast dates will be out of order.  Also, bear with me, as I learn to take better notes on coffee.

Ritual – Dark Horse


  • 50% YCFCU’s Organic Hama, Kochere, Ethiopia
  • 50% YCFCU’s Organic Aramo, Yirgacheffe, Ethiopia

Size: 12 oz.

Roaster’s Flavor Notes:

  • Chocolate-Vanilla Soft-Server Twist
  • Blackcurrant
  • Cantaloupe

Latte’s made: 16 (Whole Milk)

Kitchenaid Proline Grinder Setting 8

Samantha’s Notes: Deliciousness, I like it, Smooth, Foam is right (*I’ve upped my frothing technique by holding the pitcher from the bottom), Favorite Ratio of Coffee to Milk.

Kingston’s Notes: Mellow, not too acidic.  Not too dark.  We obviously like this one because we’ve been repeat customers.


Star Wars: The Force Awakens in Wired Magazine (Nov 2015)

Superfan JJ Abrams On Directing The Force Awakens

Fear Not, Star Wars Fans.  Your Movie is in Good Hands

You Won’t Live to See the Final Star Wars Movie

Star Wars: The Force Awakens’ Arsenal of Epic Battle Props

Star Wars Exclusive: Blueprints for the First Order’s Base

Continue reading Star Wars: The Force Awakens in Wired Magazine (Nov 2015)

Star Wars: The Force Awakens Trailers


[UPDATE 12/13/15: Star Wars: The Force Awakens “5 Days” TV Spot posted]  That’s right!  There’s only 5 days until the new Star Wars movie comes out!

[UPDATE 12/11/15: Star Wars: The Force Awakens “Secret” TV Spot (Official)]

[UPDATE 12/6/15: Star Wars: The Force Awakens “Legacy” Featurette (Comic Con Experience, Brazil)]

I’m collecting all the Official Star Wars The Force Awakens Youtube Trailers in here.  I just noticed that there are a ton of Comic Book Resource TV Spots and I’ll put those in another post to keep these separated.


Continue reading Star Wars: The Force Awakens Trailers

There's an exciting world out there, and I want to conquer it.