Chris Page – Software Engineer

Keeping an eye out for a better recording, but considering this happened just 30 mins ago — it’s the best I could do ( Drew Kerrigan actually recorded it on his iPhone here at work, good job Drew ). Without further ado, Lockerz CEO, Kathy Savitt, on CNBC Power Lunch! Congrats Kathy!

A CTO recently made a comment that some oddly high % of people above a particular salary range play N amount of video games a week.  I thought it was interesting, as I couldn’t disprove it, ha!  So, in effort to keep this page a little active, and not having the time to write out a well-thought-out technical tutorial, I decided to write about something fun, a video game I got into this week:  Storm8’s World War ( picture on the left ).  It’s a dice-based massively multi-player phone game for both Android and iPhone.

How it works?:

You pick your faction ( one of 5 global super powers ), each with their own benefits.  You then complete missions, earn money, buy bigger and badder units, and compete with other players.

What’s fun about it?:

One thing I particularly enjoy about World War is that it’s simply addictive – there are few controls and few options, so I don’t really need to deeply involve myself in the game to have fun with it. I play here and there.  I like this, because I’m so very busy these days – I haven’t really had time to join my buddies in the Starcraft 2 train, and I can only play my Wii in small dosages.  So phone games that I can carry around and play here and there is quite fulfilling.

Tips?:

I’ve only been playing for about a week, but like many other multi-player phone games, World War continues while you are offline.  So, it’s probably a good tip to build a lot of defense, so that other players don’t loot your cash flow while you’re gone!  When it comes to attacking, the best thing you can do is have a couple friends join your alliance.  The bigger your alliance, the more effective your attacks and defenses are.  Ie, if you are in an alliance of 4 people, you typically almost always win against an enemey from an alliance of 2 ( though not always, just usually ).

Join my alliance!:

If anybody wants to join the game, post your alliance code in the comments or shoot me an email — I’ll invite you to my alliance!

Enjoy,

[View with PicLens]

In addition to moving my servers to save costs, I ran into a two part issue that I lumped into: “I need to tune memory usage a bit”.

Part 1: Apache

Since I moved my Apache servers to lower memory instances, I was running into swap space usage that I could easily avoid, ie:

free -m
                      total       used       free     shared    buffers     cached
Mem:               268        245         22          0         71         53
-/+ buffers/cache:        120        147
Swap:               511         29        482

Some of the reasoning behind this was that, by default, Apache expects a bit more memory to be available than what I provided to it in the move. The fix was to introduce a few settings to lower child processes and limit concurrent connections to something more reasonable to the type of traffic my site really gets – which is near nothing most days.

The settings I dropped into apache were:

httpd.conf:

    #Low Memory Settings
    StartServers 1
    MinSpareServers 4
    MaxSpareServers 2
    ServerLimit 6
    MaxClients 6
    MaxRequestsPerChild 3000

I made the adjustments, cleared out the swap space with:

swapoff -a
swapon -a

Then restarted apache:

/etc/init.d/apache2 stop
/etc/init.d/apache2 start

And all was well in the world.

free -m
                       total       used       free     shared    buffers     cached
Mem:                268        207         60          0         31         79
-/+ buffers/cache:           97        170
Swap:                 511          0        511

Part 2-1: PHP

A bit simpler, my blog site was running into max memory allocation limits. I had left the default php.ini in place in the upgrade, so I needed to do a once over of configs and change memory_limit from 16M to something more reasonable for my site.

php.ini

memory_limit = 64M      ; Maximum amount of memory a script may consume (16MB)

Part 2-2: APC

Having Apache settings set for lower memory usage also allowed me more room to increase my APC cache limit a bit higher to keep more pages faster. From 30 MB to 50 MB.

apc.ini

extension=apc.so
apc.enabled=1
apc.shm_size=50

Other obvious solutions in consideration, switch to Rackspace to invert my memory/cpu requirement/cost ratios. Any other tips are welcome

So, recently I decided I was paying too much for my server because I was not maximizing performance across all the various daemons. So I decided to split my larger server into a handful of smaller servers to be able to fine tune each one to dedicated purposes. All went well, but I had some trouble for a few evenings figuring out how I could port forward localhost:3306 to the, now remote, database server. This should have been dirt simple with an iptables rule – but after digging in, I discovered MySQL treats localhost “special” by sending connections through the unix socket file, which is absolutely faster, but only works if the database daemon is on the same host as the connecting application.

After doing some research, I found it is possible to use a tool like socat and autossh to wrap an ssh tunnel to forward connections through the socket file to a remote IP over TCP. This however, was more complex and one off than I cared to explore for my simple problem. I finally resorted to using DNS and to stop using localhost as the host name. However, a few tid bits for the weary traveler:

• The mysql client library is responsible for selecting the protocol.
• PHP’s internal mysql libraries, unfortunately, as far as I could discover ( please correct me if I am wrong here ), do not allow you to select the protocol.
• So if you’re using “localhost” as your host name in a PHP mysql_connect, you’re forced to go through the socket file, however, you can use 127.0.0.1 instead of localhost to force TCP.
• The linux mysql-client package command line tool offers a –protocol=tcp flag if you want to force TCP. You can also set this as a default inside /etc/mysql/my.cnf under the [client] heading

my.cnf:

[client]
port            = 3306
socket          = /var/run/mysqld/mysqld.sock
protocol        = TCP

Again, this appears to work fine if you’re not using PHP as your client.

I hope this lesson learned ( use DNS ) comes as a helping hand to others out there. If anybody has some other suggestions, please do leave a comment!

I’ve been knee deep in performance and scalability for some time now, and have used and learned of many useful tools and techniques to help out. One of my favorite command line tools for seeing how well a single Apache server is churning out pages in development comes stock on Ubuntu, and Mac OS X: Apache Benchmark.

A simple performance test against the homepage of one of my client site’s using AB at the command line:

ab -t5 -n100 http://www.teamgzfs.com/

The results:

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking www.teamgzfs.com (be patient)
Finished 664 requests

Server Software:        Apache/2.2.8
Server Hostname:        www.teamgzfs.com
Server Port:            80

Document Path:          /
Document Length:        306 bytes

Concurrency Level:      100
Time taken for tests:   5.054 seconds
Complete requests:      664
Failed requests:        0
Write errors:           0
Total transferred:      465003 bytes
HTML transferred:       205326 bytes
Requests per second:    131.38 [#/sec] (mean)
Time per request:       761.143 [ms] (mean)
Time per request:       7.611 [ms] (mean, across all concurrent requests)
Transfer rate:          89.85 [Kbytes/sec] received

Connection Times (ms)
min  mean[+/-sd] median   max
Connect:       38   78  48.7     64     994
Processing:   114  540 226.2    493    1690
Waiting:      114  531 205.6    493    1485
Total:        191  618 236.0    558    1808

Percentage of the requests served within a certain time (ms)
50%    558
66%    587
75%    598
80%    617
90%    845
95%   1163
98%   1402
99%   1746
100%   1808 (longest request)

There is quite a bit of useful information here that can help you tune your code and server. It’s important to note however, that when working on a larger site, that expects quite a bit more traffic, you might want to investigate some more thorough solutions outside of just a single machine and ab. It is, however, a nice starting point into useful information.

One rather funny pitfall you can run into however, is if the host you are sending requests to is smartly secured – these types of tests become a bit useless, as they may have security settings to limit or delay requests – providing you with timeouts and/or inaccurate information. Best to run these types of things in a semi-developmental mode with those types of security settings turned down, and rely on bigger guns or fleets of boxes and scripts to hit a production secure site.

In addition to hitting just a landing page, you can use AB to send COOKIE or POST data too! This is very useful if you want to see how pages perform but need credentials to get in first. This is a little trickier to do using the -c, -T, -p, and -v flags. I noticed there are under-useful resources online to figuring it out with AB, so it would seem worthwhile to write it – as it took me some trickery to figuring it out as well:

Sending POST data to a login form:

First we create a file that contains our URL encoded post data. Note, AB expects the values to be URL encoded, but not the equal (=) or ampersands (&).

post_data.txt

username=foo%40bar.com&password=foobar

Capturing a cookie:

Here, we use the verbosity (-v) flag so we can see the response headers that come back — many sites will send back a cookie once authenticated, we’ll want to capture that cookie here. Though, some sites will not require it, I demonstrate it for the sake of example:

ab -v4 -n1 -T 'application/x-www-form-urlencoded' -p post_data.txt http://www.foobar.com/login

The returned response header will fly by quick, you’re looking for something like the following:

Set-cookie somesession=somerandomsessiondata...;

The session data may come back encrypted, unencrypted, a serialization, or just a number. That varies by site. The point here is you have a key/value pair for the cookie. All you need is the part up to the semi-colon ( not including the semi-colon ). Copy that “key=val” string, and use it when hitting other pages on the site you are testing, ie:

Using a cookie to test a page that requires a cookie:

ab -t10 -n100 -c 'somessession=somerandomsessiondata' http://www.foobar.com/login_required_page

This can become a lot of fun once you get the hang of it. Now you have the know how, go enjoy creating an arsenal of these scripts and start performance tuning your sites – or script hacking your favorite social network ( or obnoxious Blizzard clan website hahaha… drum roll for D3 – 2010? Please???! ).

As usual, it’s been awhile since the last post. As for pictures say a thousand words update:

Not pictured: “open bar, rock. the lockerz staff enjoying said open bar, rock. hundreds of lockerz fans enjoying the concert, rock. a great meal at the cafeteria in NYC, rock. oh, and Manhattan, ROCK!” Needless to say we had a blast! Thanks to all those supporting Lockerz, and all those who worked hard to throw an awesome launch party, as well as all those who continue to work hard every day to take the company to the next level!

In other news, we’ve been pouring our blood and sweat into work and home – doesn’t mean it hurts though My long time heterosexual life mate visited from Chicago over the weekend – we had a great time and already miss his company. In addition to his visit, my long time friend, college room mate, and colleague, Drew Kerrigan, stayed for 2 weeks as he joined Lockerz this month and is relocating from NoVA! A special congratulations to him!

So now we have at least 3 huge Diablo 3 fans ( Anthony, Myself, and Drew ) ready to pour vacation time down the drain

One last note! Happy birthday Peter!

Troopers in Lawrenceville

What a strange day! This may be a tad ignorant, but I really don’t know what the G20 is truly all about. I’ve only caught bits and pieces from other people – needless to say, it’s a big event. The office is just a few blocks from the convention center where the meetings are being held, and there are more cops, S.W.A.T. teams, and state troopers marching the streets than a small army.

Driving around, we’ve been behind vehicles tagged “diplomat” more than once, hit a few road blocks, and she had the unfortunate event of protesters banging on the car down Penn Avenue. I’ve caught rumors of tear gas being used, our office building was on lock down towards the end of the day, and on the way home the radio said President Obama was making an appearance in Oakland, just around the area we were staying for the last 2 months!

So, WTF is going on?! Ha… something good better come out of this ( aside of the awesome food Lockerz provided today to keep it’s employees safe during lunch ). We had to drive through Wilkensburg to get home, and that’s just no treat – cab driver once told us that Wilkensburg doesn’t get city funding, so it’s falling apart… crack whores calling out to their suga daddies – no lie. Though, given my ignorance to the world changing event downtown, perhaps I’m no better, ha.

Attached is a picture Drew Zhrodague’s mom took on her way for coffee.

Every once in awhile one has to write a script that is concerned with execution time. Consider the following snippet that utilizes epoch seconds to calculate execution time:

$start_time = microtime(true);
usleep(200000);  // sleep 2 seconds
$end_time = microtime(true);

echo 'Execution time:  ' . ($end_time - $start_time) . "\n";

Pretty straight forward and has many useful applications. By using built in php functions, `microtime` and `usleep`, it’s easy to benchmark operations, and even set time limits on operations. Consider the next snippet which will halt execution after 30 seconds:

$max_execution_time = 30;
$start_time = microtime(true);

while (1) {
    if ((microtime(true) - $start_time) > $max_execution_time)) {
        break;
    }

    //Do something
}

One more clever application, is to pace your scripts. Such may be useful when concerned with load:

while (1) {
     usleep(10000);  // sleep 1/10th of a second
    //Send a request... query a DB... do something
}

You can do quite a bit with these two functions. It never hurts to read up on microtime, and usleep. Enjoy!

Lil Wayne loves Pitt too!

Two weeks down, this blog is due for an update – time has been flying!

As far as home life in Pittsburgh goes – ROCK. I’m currently staying roughly between one of Carnegie Mellon and Pittsburgh University’s campus intersections – the architecture is gorgeous. I enjoyed a meal at the famous Primanti Brothers down the street, learned the public transit ( a work in progress… ), and discovered a few open mic hot spots I’ll be hitting up eventually – oh, and Lil Wayne at the Post Gazette Pavilion was also ROCK! I’ve had numerous other experiences and visited quite a few restaurants, but won’t detail all my eating habits here

As far as Lockerz goes… ROCK ON! I’ve made some new friends, thoroughly enjoyed many philosophical discussions, and am getting knee deep into working on the next best thing. Today was a moment for Lockerz history – as we saw an incredible traffic increase this morning – ranking 4 on Google Trends around 10 a.m.

The boost seemed grass roots in nature, as people are catching wind and telling their friends, who are telling their friends. The most personable for me was Amber’s friend made a comment about needing a Lockerz invite to join the site, not even realizing I work there. Small world? Needless to say, he got the invite.

I won’t detail much else about the job – you’ll have to keep tabs on the site! Need an invite? Let me know.