Jamie Thingelstad

Debating wether I should have 2 4G servers or 1 8G server at Linode for my personal server. 😕 One Ubuntu to manage sounds better.

Dunes.

Super fun day on the beach today.

Astounding beer selection at Saugatuck Brewing Company. Including 118 IBU IPA. Flight ordered.

Vacation starting now.

Found out this morning that Sona Mehring and I share common Perl skills! $_ FTW! 🙂

Good CaringBridge program meeting this morning → great progress!

Mom grilling for sisters birthday!

Updating to WordPress 3.6. Hope this is smooth. 😊

Did The Economist annual subscription price actually go up? Yikes.

Just discovered a ground hornets nest right in our backyard. Luckily nobody bitten.

Makes me chuckle how Canon camera firmware updates describe bug fixes as “Resolves a phenomenon where …”

Today’s xkcd highlights what might be one of the biggest security mistakes of our digital age: using passwords over passphrases. The comic does a great job of illustrating it. I’ve mentioned before that I use 1Password to store over 400 unique, random passwords. But how do I unlock 1Password? A passphrase that is long. Very long.

Sadly being an active 1Password user I can highlight that many websites fail if you use passwords that are greater than 20 characters, much less a 40 or 50 character passphrase.

I’ve been working with Yaron Koren’s new Miga project to provide an offline, mobile optimized way of accessing the nutrition data in Wikinosh. It works pretty well! However, we hit a couple of walls trying to get the 20,000+ food items in Wikinosh exported for use in Miga. If you are trying to page thru large data sets in Semantic MediaWiki read on.

Large Offsets

The best way to do this is to use queries of 500 items at a time and use incrementing offsets to retrieve each set of rows. This works really well and is easy to code. However, when you run this the first time if you have over 5,000 records you will notice that your while loop will never finish. Very odd. When you look at the data you will also see that after a few thousand rows, the first set of data just repeats over and over and you never reach the end.

A quick look in SMW_QueryProcessor.php on line 611 shows this:

$params['offset'] = array(
    'type' => 'integer',
    'default' => 0,
    'negatives' => false,
    'upperbound' => 5000 // TODO: make setting
 );

Now that makes sense. It turns out that if your offset value is greater than 5,000 it won’t be used. To make matters worse, this doesn’t generate an error it simply ignores the offset. Ugh! The “TODO” is still a to do, there is no setting for this. If your working with large data, you will want to increase the offset to something large enough to accommodate your largest set. Since Wikinosh has 21,000+ items, I set it to 30000.

Query Max

Now that you have your offset changed you can go ahead and run your export again. It will chug along and then hit 10,000 records and stop. What? Sure, you increased your offset however you probably still have the default $smwgQMaxLimit value of 10000. So, if your offset is greater than the query max, you get 0 rows and your done. This setting does have a way to override, set $smwgQMaxLimit = 30000; in your LocalSettings.php and you will be ready to go.

With these two changes in place, I can now easily page through 500 records at a time and get everything out exactly as I expect.

There are currently two bugs open on these behaviors. Check those for current status of these settings: SMW Ask query offset has a hardcoded limit and SMW Ask query offset should error if maximum offset is exceeded.

Spent a solid 4 hours cleaning the garage with Tammy. Looks great.

How long will a 3 year old cry and scream at the bottom of the stairs insisting someone “carry him!” up the stairs? You’d be amazed.