At some point in the not too distant future, the intarweb’s magazine consumers will be perusing a new magazines.com. I’ve got the responsibility of building it. One of the requirements is to allow our Merchandising and Marketing folk to get their jobs done without too much fuss. The system they use will consume sets of products. They might want to build a set of magazines in a certain category, under a certain price. What kind of tool am I talking about?

A Query Builder.

I can hear your eyes rolling already. We’ve all written them. But I’m going to share an extra-cool method for doing so.

Here’s the list of ingredients you’ll need.

• Your existing DBIC application.
• Experience with custom resultsets.
• An understanding of chained resultset usage.
• Moose
• MooseX::Method

Now that you’ve collected your ingredients, let’s prepare them.

First, note that we aren’t allowing our end users to build literal queries in this recipe. We are going to allow them to chain calls to our resultset. This is a much better option, in my opinion. A lot of the complexity can be removed by simplifying some heinous or dangerous SQL into a simple resultset method. With the help of Moose we will create a custom resultset that allows us to introspect the methods available for building and show them to the user automatically.

Now, we’ll be working on a Product example. Our end user needs to build a query that finds all the products in a price range, of a certain type and in a certain category.

Since you’re already versed in custom resultsets I’ll just show you the finished product:

package Our::ResultSet;
use Moose;
use MooseX::Method;
use MooseX::Util::TypeConstraints;
 
extends 'DBIx::Class::ResultSet';
 
subtype 'Currency'
	=> as 'Object'
	=> where { $_->isa('Math::Currency') };
 
coerce 'Currency'
	=> from 'Num'
		=> via { Math::Currency->new($_) };
 
method price => named (
	operator => { isa => 'Comparator', required => 1 },
	amount	=> { isa => 'Currency', required => 1, coerce => 1 }
) => sub {
	my ($self, $args) = @_;
 
	if($args->{'operator'} eq '=') {
		return $self->search({ price => $args->{'amount'}->as_float() });
	}
 
	return $self->search({
		price => {
			$args->{'operator'} => $args->{'amount'}->as_float()
		}
	});
};

That’s a lot of code, but the concept is very simple: Create your resultset’s methods with MooseX::Method and then you can do the following magic:

my $rs = $schema->resultset('Product');
my $map = $rs->meta->get_method_map();
foreach my $method (keys(%{ $map })) {
	# See MooseX::Method docs for what to do with this.
	my $signature = $method->signature();
	# Profit!
}

Having the method’s ’signature’ allows you to get a list of the parameters that the method expects, as well as the types and ‘required’ status. That sounds like the recipe for some go-getting developer to construct a query building tool!

One warning. MooseX::Method currently lacks a permanent API for acquiring this information. The current API is to call export on the signature. You can Dumper the resulting hashref and see pretty easily how to get what you need. Keep an eye on The CPAN or in irc.perl.org’s #moose for updates.

So to use this recipe, you’ll have to construct your own mechanism for taking the above information and generating a UI for your application. I got my Catalyst based query-builder working this afternoon.

If you have any questions feel free to drop into irc.perl.org and shoot a message to gphat. You can find me in #moose.

One of the most precious responsibilities afforded to me as a developer is the ability to name my projects. Moreover, as Development Manager I could — theoretically — dictate the names of all the projects done in my department. That’d be dictatorial and I don’t do it… but I do reserve the right to veto or break non-majority ties.

Here’s a listing of the projects names used on our ‘current’ projects:

• Greenspan (parent project of the others)
• Lockhart
• Kayhill
• Hopkins
• Osgood
• Downham
• Holbrooke

Greenspan was the ‘first’ in the new generation of projects we’ve done at $work. It established the system used to name others. I regret choosing a name that is so obvious to anyone who knows what the Federal Reserve Board, but it established the general rules we use:

• Words that break down into two words well (Os-Good, Down-Uhm, etc)
• Words that would seem at home when naming English cities
• Words that aren’t in people’s general lexicon

Since I also wear the hat of System Administrator, I get to pick names for the servers we run. The current naming scheme is based on the Greenspan project. We use the names of currencies for our servers: peseta, drachma, lira, pound, euro. We made a list of names on our wiki of prospective names so that when a new box arrives, we have a queue of names we can choose from. We usually vote unless the box is going to primarily serve a single person’s interest. In that case, they name it on their own.

It is said that to name a thing is the power to control it. I’m not sure if that is the underlying principal, but I can say that project naming and releasing procedures have always been extremely interesting to me. I love reading ChangeLogs and reading about the process that companies use to push out shiny new toys.

I think that codenames say something about the ideology of group that uses them. I’ve always been impressed by the huge list of codenames used by Intel for their products. We could use a scheme that generates informational names like fra0132 for “Franklin location, Rack 01, Position 32″. If we had a hundred boxes I’d consider it. But in a smaller installation all of these machines have a personality and to treat them like anonymous slaves would be a disservice.

These names are often difficult to get consensus on. I might love a name only to have another developer loathe it. It takes a lot of effort to cook them up. But it’s all worth it in the end.

Hasten Slowly, in the Latin.

A conversation with Jaxn tonight got me thinking: How ‘fast’ are the sites of the top 10 ’shopping’ sites (per Alexa)?

So I measured them. At first, it was a simple matter of emptying my cache and loading each site. But many sites, understanding how important it is to get things moving quickly, used some sort of onLoad function to do some more loading without screwing up the user’s experience. To facilitate those, and not penalize them, I took a screenshot of the measurements as soon as the progress bar filled and the site was useable.

UPDATE: The demo and trac links are down atm, as Ayalike is being moved to new hosting.

I’ve been hacking away for the last week or so on a content management system (henceforth CMS). We are going to need a CMS for some future $work projects and I’m generally unhappy with all that I’ve reviewed. So many of the existing ’solutions’ seem to be more in the business of providing you with a ‘platform’. Most of the folks I know build platforms for a living, we need only find a way to manage the content.

So with that in mind, Ayalike (pronounced like ‘a uh like’) aims to provide the following features:

• Multisite, multiuser, multirole
• Output agnostic
• Versioning
• Pluggable publishing
• Output processing

Three of those features are complete. Publishing works, but is currently not pluggable. Output processing is there as well, but I’ve yet to hook it into anything but unit tests.

That’s all kind of abstract, here’s what I’ve done in a more concrete form: This week I modified one of our internal applications to use a custom Template Toolkit provider that communicated with Ayalike and provided ‘preview’ functionality on the live site without changes to the application itself. The application functioned normally, rendering compiled templates from the filesystem, unless given a special parameter. In that case it requested the entry from Ayalike over HTTP using it’s wiki-esque page viewing feature. Obviously real apps would protect this feature with logins.

I’m not quite ready to tout it on the mailing lists or in IRC yet, but I’m very excited about the project and I hope that it might become the solution for my future work projects. Regardless, Catalyst needs a CMS.

Are you interested? Visit the Trac site. There’s even a working demo, cleared every half hour.

Oops. A bit late with this one.

• BPM 37093 is a white dwarf that, as it has cooled, has turned into a diamond. Our sun will do the same in about 5 billion years, as a hard crystalline core forms in it’s center.
• Planting 300,000 new trees and allowing them a full life would offset the carbon dioxide emissions of one 500-megawatt coal-fired power plant operating for a little more than a week.
• The brass ring on the end of a pencil that holds the eraser is called a ferrule.

It’s back. A little short this week, but I have to get back into the habit.

• Baby blue whales can gain up to up to 200lbs a day by drinking about 100 gallons of milk. Whale milk is 40-50% fat whereas cow’s milk is appromixately 4% fat.
• India is the world’s largest butter consumer. It’s people eat roughly 46% of the annual global supply — that’s over seven billion pounds.
• The darkest substance ever known was made this week.

My day to day responsibilities have changed pretty drastically in the last two years. I’ve gone from being the sole developer on a small, focused project to a manager over a department that is creating some some fairly large projects.

On second thought, it’s not so different. The challenges of the projects are the same. The wrinkle is the addition of more entities into the equation.

I do so much in a day that it takes me hours to decompress. I often cook up posts on my trip home only to succumb to the soothing caress of a game, T.V. show or conversation with the girlfriend. Today I:

• Spent time discussing project and development workflow.
• Finished setting up Nagios to monitor a bunch of machines.
• Added a requested feature to our order entry / customer service app.
• Tidied up some failing tests.
• Tested tomorrow’s shipping software.
• Played Weewar.

Tomorrow I intend to continue the process discussion. Having had some time to digest the information proposed I’ve got some ideas. With my current monitoring project wrapping up a few weeks early I can set my sights on the next one: automated testing of our codebase. I’m excited to spend some time with Verby made by my friend Yuval. It’s eerily similar to some code I had begun working on when I started thinking through this project. More on that in the future I hope.

I’m going to breath some life into my old What I Learned This Week posts, but marry them to my daily adventures in the development management world. I’m hoping that some upcoming projects will yield some post-worthy nuggets. If I’m lucky maybe I can come up with a talk to give at YAPC::NA 2008 in the next few months.

As a closing I’d like to thank Jon for suggesting Sierra Nevada Porter and my beautiful girlfriend for bringing pint glasses when she moved in.

Two months since my last post. It’s not really that I haven’t had anything to say, I assure you. I just don’t remember it long enough to write it out or I haven’t felt compelled to write about it in public. I guess. Who knows?

Imagine that I’m summarizing fanciful adventures and bracing, hard-fought lessons in this post. Go ahead, I’ll wait.

With that out of the way, we can quickly douse the fires of interest by describing the months of software development that culminated the creating of an order entry system for $work. That aspect of my life is harrowing and exciting. It’s a work in progress. More on that some other time, perhaps?

I’ve been… disconnected for a month or two. I rarely open NetNewsWire and I keep Twitterific closed. I’m not really sure why I’ve shunned the torrent of information I normally subject myself to. There was a spell of time — from September to mid November — where i struggled to produce a modicum of giveashit. This time was terribly busy and shutting down the founts of information was a requirement to maintain my sanity. My enthusiasm to reenter the ‘normal’ information overload has been tepid at best. I just trimmed 25 blogs out of my feed list and it still feels too big. Ok, another 20 more. Down to 80. That feels better.

I’d try and summarize the year, but I have some type of malady that muddies my memory past about 30 days. My girlfriend moved in, I shipped a big software project, nearly went insane and attended a Perl conference. All that’s happened in the last few months so it’s a poor summary.

So fuck that noise. Lets talk about what is going to happen in aught-eight:

• Be a good daddy
• Be a good boyfriend
• Enjoy my job more (working on managing up, delegating, componentizing and more)
• Go on vacation (Disney World for my brother’s wedding, March 30 - April 5)
• More

I’ve tried to write this post mutiple times. I finally succeeded. I’m not sure why my desire to talk to the world in this way has dried up. Surely I have things to say…

Things have been hectic as of late. Two weeks ago or so I began the rather stressful experience of converting the call center at work from an ancient terminal based order entry system to a fancy, modern Web 2.0-ified one. Many years of data was converted, and dozens of processes had to be rejiggered to work. We are still dealing with them today, actually. I’m watching the DEBUG messages scream by for a fulfillment run.

There are always things I want to do in my spare time, but there are a few that have been staring longingly at me from outside my cage lately. Being fettered to my desk working out the various problems that have arisen has made me long for these outside projects…

• A WeeWar clone that is more strategy and less Heavy Arty / Heavy Tank zerging
• Update my blog entries about DBIx::Class::QueryLog
• My Cat/DBIC based charting app
• Chart::Clicker
• Site Redesign

I’m attending the Pittsburgh Perl Workshop next week. My time there may give me some time to hack on these projects in my fave language. I also might take some time off to hack on them. I’ve earned it.

Friday night I released new version of my 2 Perl modules. I followed that up with more hacking over the last few days. Eventually I’ll release a new version of Clicker with some new features, but I got perturbed at it today and played the Xbox.

Speaking of XBox, Connect360 kicks ass. Reminds me of a conversation with someone last week about how the Mac platform has fewer but higher quality software. What I like even more is that so much of the software I’ve purchased is from indie developers.

Sunday was spent cleaning out my living room and garage, as a bunch of my furniture is being obsoleted by upcoming girlfriend move in. My brother brought his truck over and we made three runs to the recycling center and Goodwill truck located therein. There were also a bunch of boxes from various A/V purchases I’ve made over the years that were too big to break down and put in the trash. After we removed the furniture Jenn came over and shampooed the carpet downstairs and on the stairs themselves.

We roped the brother’s fiancée and saw Superbad after dinner and shopping in Murfreesboro. It won’t win any awards, but it was funny as hell.

Yesteday was a day of more cleaning up and whatnot. Jenn and I had dinner at family’s house in the evening.

Today I slept all day. Well, I hacked some too. That frustrated me more than it helped. That’s a bad thing considering how many bugs I need to fix and features I need to polish over the next few weeks. I’ll probably play Forza to compensate. Yeah, that’s a good excuse…

I just uploaded the 1.0.0 release of DBIx::Class::QueryLog to the CPAN.

The 1.0.0 label is mostly just a not to it’s maturity. I’ve been using QueryLog extensively at $job — most recently to tune a multi-million order conversion script — and a 1.0 is in order.

The only new feature is a refactoring of the ‘analysis’ code into a seperate object, as well as a new mode that totals up queries, combinining same queries and keeping a count and total time for all executions.

Have fun!

Learned markdown and used it for this post.

My gig at magazines.com began as a member of a two person team. Luck would have it that we were very compatible office mates as we shared similar interests, hours and working habits. Even our music taste was similar. We moved from a shared cube to a conference room table to an eventual custom 3 man office. When we moved from that office space to our current digs a special room (dubbed the ‘Cave’) was allocated to harbor developers. Cubes were inserted, but they were more elaborately connected desks than traditonal cubicles.

Time has passed. That early two person team has double to four and we are looking to hire another (send resumes to cwatson ta magazines dot com). A quick discussion between two programmers or a few questions from a coworker can easily knock an innocent bystander out of the zone.

I’ve been asked in recent planning what I think we should do about this. I’ve also just completed my first year as a manager, and I’ve got lots of stuff I did wrong. Here’s my laundry list:

• Private Offices

  Having a privacy and insulation from chit-chat and babble is a great way to stay in the zone and protect your flow.

• Common Area

  A response to the idea that having knowledge workers in a common area. This is a good idea, in theory, but every gain is countered with interruption. So rather than putting your programmers offices in an open room, give them an open room to meet in. Put some desks, power, comfy chairs, network jacks, caffeine and nerf guns into a room and let them meet there. Coordinate your meetings with…

• Department IRC / IM

  I idle in IRC and chat with other developers on projects we use. This lets me get help, provide help and socialize. Having your developers communicate this way gives you some of the benefits of the ‘community’ space but with the ability to minimize the window and get to work.

• Fast Laptops, Lots of RAM, Big Monitors

  Powerbooks or Thinkpads, chock full of RAM and a gigantic monitor to plug it in to. Mobility allows the programmers to bring their laptops together, work from home or take them to conferences. When a computer is the only tool used to do a time sensitive job, it makes no sense for it swap to disk.

• Solid Network

  Get a good firewall and spring for the most bandwidth that makes sense for your requirements. Waiting for a transfer from your colo to the office is murder when you are trying to get things done. So is fighting with arcane TCP/IP problems due to old firewalls.

• Lots of Storage and Processors

  Processors are fast and storage is abundant. Spring for the full terabyte and the quad-core. It’s worth it to avoid that full disk or that CPU bound process from ruining your day.

That’s my list for now. I’ll update after I get done with some more mind-numbing data conversion.

UPDATE: Jon Schutz recently posted an update to this idea. Please use his!

When handing a request Catalyst politely outputs each action executed (in order) along with it’s timings, like so:

2007/05/17 13:52:18 INFO [Catalyst : 1506] Request took 0.429198s (2.330/s)
.—————————————————————-+———–.
| Action                                                         | Time      |
+—————————————————————-+———–+
| /auto                                                          | 0.102865s |
| /default                                                       | 0.040768s |
|  -> /search/default                                            | 0.028275s |
| /end                                                           | 0.171297s |
|  -> Greenspan::Lockhart::View::TT->process                     | 0.165873s |
‘—————————————————————-+———–’

This information is helpful when trying to trace the flow of execution, find an error or when trying to isolate which action is hindering speedy throughput. But what does one do when multiple people are using a load-balanced app simulataneously? Tailing error_log on 3 webservers isn’t as much fun as the kids make it out to be! My work on DBIx::Class::QueryLog has me in the mode of providing performance information in the footer of the page. So I set out to do the same for Catalyst actions.

Turns out this is easier than it sounds. Catalyst keeps a Tree::Simple object in $c->stats() that details execution. Each node is a hashref containing (at minimum) ‘action’ and ‘elapsed’ keys. The former being the name of the action and the latter being how long it took to execute. Oddly, that latter value is not a plain number, but a float followed by an ’s’ to denote it is in seconds.

A quick snippet of code placed into the ‘end’ of my root controller can digest this tree and produce something useful for my templates:

my $tree = $c->stats();
 
my $dvisit = new Tree::Simple::Visitor();
$tree->accept($dvisit);
$c->stash->{'action_stats'} = $dvisit->getResults();

The value of the ‘action_stats’ key is now an array of hashrefs representing each action in the chain in execution order. A simple FOREACH and we’ve got ourselves a list of actions with their timings.

The actions could be arranged in order of time elapsed using Tree::Simple::Vistitor::Sort, but I personally prefer execution order. YMMV.

We are hiring at the office. It’s in Franklin, non-telecommute and LAMP.

The linked description basically sums it all up, but I’ll divulge a little more. We are looking for a mid-to-junior person: anywhere in that range. While being a Perl junkie would be cool, we are open to all types of experience. Once you’ve picked through a few languages you learn that it’s mostly syntactic sugar and tools anyway. We also wear extra hats (e.g. SysAd, DBA and Designer) so having additional areas of expertise is a plus.

We’re currently serving two masters: maintenance of an existing codebase and development of a wholly new one that already handles some tasks. The existing code has a reputation amongst people who’ve worked with it. I was actually warned away from this job due to it’s… personality. We’ve cleaned it up a lot, so that’s not an issue anymore.

Our newer codebase is based around Catalyst and DBIx::Class. Familiarity with these are not really required, as we can show you the ropes.

If this sounds like something you or someone you know might be interested in, shoot a résumé to cwatson at magazines dot com.

There’s a saying that’s floated around at my last two jobs. It was started by my one-time boss and continued by my current one. Difficult or irritating problems are often introducted by saying this:

I have an opportunity for you…

The idea, of course, is to frame a problem in it’s rosiest light. It’s become a joke because of the stigma we’ve attached to the word, but the idea is sound.

Lately I’ve been faced with some thorny issues. There are lots of little projects here and there that have been snowballing around and becoming somewhat cumbersome to general development. But I’ve been making the mistake of allowing these projects to sit in my mind as hinderances when I should be viewing them as opportunities.

It’s a silly mental game, but it’s a very important one.

I ran track every year in high school. I don’t run anymore but I like to think I learned a lot about how to run. I like to watch ‘professional’ runners because they make it look fluid and natural. One of the tips I learned is to leave your hands open and relaxed, not balled into a fist. If your hands are tight, it spreads to your forearms then your chest and eventually your whole body. In other words, if you start the run stressed, you doom yourself to having a harder time.

That’s a long way of saying that I’ve not been approaching my projects with the optimism they deserve. This negativity conducts itself to my ‘customers’. That’s a Bad Thing. This atitude is all the more important for people in driving positions of IT departments, as we are the guides into a magical¹ realm. As the keyholders, we have to enable, not hinder.

That puts the onus on IT/Development. So be it: That’s why they pay us the big bucks.²

There’s a yin to this yang. If your Development group is Perfect In All Ways, then your customers could simply depend on you to answer their questions and take them as The Truth. You probably aren’t. Neither are they. That’s why we should try to remember that this is supposed to be somewhat fun, and act accordingly. Solving hard problems is what we do.

1 - In the sense of the ‘Any sufficiently advanced technology is indistinguishable from magic’ saying, which is a very generous way to view my work. :)

2 - Yeah, right.

Months ago I implemented DBIx::Class::Storage::Statistics with the intent of making some sort of profiling tool. I finally got off my ass and did it. DBIx::Class::QueryLog should be on CPAN this weekend.

At $work we are replacing a legacy system with one based on Catalyst. One of the deliverables for this year is a new order entry application. Since our customer service folks are used to working in a terminal based system speed is of great concern to us.

So I installed DBIx::Class::QueryLog as the debugobj as described in it’s documentation, dropped it in the stash and added some code to our wrapper.tmpl:

      [% IF querylog %]
      <div class="featurebox">
        <h3>Query Log Report</h3>
        [% SET total = querylog.time_elapsed | format('%0.6f') %]
        <div>Total SQL Time: [% total | format('%0.6f') %] seconds</div>
        [% SET qcount = querylog.count %]
        <div>Total Queries: [% qcount %]</div>
        [% IF qcount %]
        <div>Avg Statement Time: [% (querylog.time_elapsed / qcount) | format('%0.6f') %] seconds.</div>
        <div>
         <table class="table1">
          <thead>
           <tr>
            <th colspan="3">5 Slowest Queries</th>
           </tr>
          </thead>
          <tbody>
           <tr>
            <th>Time</th>
            <th>%</th>
            <th>SQL</th>
           </tr>
           [% SET i = 0 %]
           [% FOREACH q = querylog.get_sorted_queries %]
           <tr class="[% IF loop.count % 2 %]odd[% END %]">
            <th class="sub">[% q.time_elapsed | format('%0.6f') %]
            <td>[% ((q.time_elapsed / total ) * 100 ) | format('%i') %]%</td>
            <td>[% q.sql %]</td>
           </th></tr>
           [% IF i == 5 %]
            [% LAST %]
           [% END %]
           [% SET i = i + 1 %]
           [% END %]
          </tbody>
         </table>
        </div>
        [% END %]
      </div>
      [% END %]

The first run yielded this:

Well. That select from users could use an index on username, eh?

Weee! That one disappeared. What are all these role selections? We use Catalyst::Plugin::Authorization::Roles but this looks fishy. We might could use an index on the role name but there are only 6 rows in that table. A bit more investigation finds a loop in our Root controller that iterates over a set of roles calling checkuserroles() to try and find out where we should redirect the user. So lets swap that out for creating an array of roles and using grep…

Kick ass! We are down to two measly queries. We reduced the number of queries by about 75% and decreased the time spent executing SQL by about 75% as well. These queries were being executed on every page load.

UPDATE:Yeah, the average statement time was broken during that. Oops.

Highlights: Avoided a root canal.

At what do we labor?

Seriously. Is it fleeting feelings? Some type of legacy?

Honestly, I’m not really sure what the hell I’m after. I don’t think about it very often.

But wait. Maybe I have inklings of an idea. I want to enrich the life of another. (And conversely, have mine enriched.) I want to raise a child that brings her own special to the world. I want to give back to the people who gave to me. I want to enjoy it all. I want to learn.

Let that be a lesson to you. Just when you think I’ve stopped, I deposit another mote of nonsense. Libations to a needy public.

Child number four for Jackson and Sabrina ‘entered’ the world at 9:15a weighing in one ounce under 9lbs.

Congratulations guys!

• emusic is all that iTunes isn’t. I can’t find good documentation but it looks like the label is getting a good share. Since they are indie I’m assuming the artist is getting taken care of. No DRM for me. Steve could learn something.
• Having many projects cooking at once is something I’m usually game for but work lately has been stressing me a bit. This feels odd to me as I’m never really stressed. It never bothers me after I leave but I feel bad for being snippy or pessimistic. I love what I do and I try and communicate that to everyone. I guess one of the prices of passion is that when it gets hard, I get ornery. I like the challenge.
• Why are people hiding behind ‘the troops’ when it comes to discussing Iraq? We know it’s a distraction to keep people away from the defense Gravy Train.
• Tax return is supposedly coming tomorrow. Time for a new center channel speaker!

Anyone who reads my blog for content knows that one of the few things I get fired up about is DRM. Since music is so important to me it makes sense that I would feel strongly about my rights in that arena. Today Steve Jobs, the guy in the driver’s seat of the 800lb gorilla of DRMed music, plucked at my heart strings.

In such a world, any player can play music purchased from any store, and any store can sell music which is playable on all players. This is clearly the best alternative for consumers, and Apple would embrace it in a heartbeat.

If Apple sold music unencumbered by DRM I would make purchases there. I still want physical, touchable compact discs for backup purposes but sometimes I just want something bad enough to download it.

I’m not sure I believe that The Steve really feels this way. Maybe he’s just trying to get some sway over the Big Four for licensing fees. But I’m sure as shit happy to hear someone so important in that market say the things I want to hear.

Talk does not a dollar earn. Show me the goods Steve. You are getting closer…

UPDATE: It has come to my attention (via Daring Fireball) that Apple does not allow indie labels to put their music on iTMS without DRM. For Shame. Are you lying to me Steve?