WikiApiary on Jamie Thingelstad

Envisioning a Cloud Native Wiki

Sun, 26 Nov 2017 17:37:37 -0500

A few years ago I dove deeply into the wiki ecosystem and learned all about MediaWiki, its plugins and various extensions. I started a wiki to index all of the other wiki’s called WikiApiary. The wiki movement was huge, and it still has a tremendous amount of energy and incredibly devoted users. Wikipedia, in my opinion, is one of the most amazing creations of the Internet.

However, the wiki movement has hit the skids lately. There was a significant rush of wiki hosting platforms early on, and those mostly didn’t work. Wikia is perhaps an exception however it’s driven itself into the hole of fandom in a big way. Sadly many wiki platforms continue to sit on top of ugly PHP code and MySQL databases, with old codebases and arcane syntax.

In recent years email newsletters have seen a resurgence. I’d like to see wikis make a similar revival but for that to happen, we need to have some new energy in the wiki ecosystem. We need a cloud-native wiki solution. What would that look like?

Cloud Native Wiki

First thing first wikis should enable communities of any size, so the cost of running a wiki needs to be as cheap as possible. A serverless approach seems to make the most sense. If nobody is using the wiki the cost of running it should be no more than the storage, and that can be very cost effective.

Store all wiki pages and objects in S3. Store the objects in an editable format, something like Markdown, but also store it as HTML as well so it can be served directly out of S3. Think of how static site generators work.
The wiki itself should live in Lambda functions exposed through API Gateway. All editing and modifying of objects in the S3 bucket should be done via this method.
The wiki should be self-managed by a central source to update it. Wiki’s have a history of being poorly maintained. You should be able to create an IAM credential and give the ability for the Lambda functions and infrastructure of the wiki to be updated automatically.

Let Millions of Wikis Flourish

Ideally, someone should be able to start a wiki by creating an AWS account and then creating the IAM account for provisioning and updating. That account should then do all the initial setup as well as updating over time.

The content would be held in Markdown files in S3, as well as rendered HTML. This content would be easily mirrored off of S3 so it could be taken somewhere else. If the Lambda functions were all removed, the wiki should still run in static mode with no editing.

If we had this kind of capability, perhaps we could see wikis reenter the landscape for all the good that they can provide.

Fri, 30 Oct 2015 20:20:00 -0500

Very cool Lambda tool. Needed this for my WikiApiary project.

Mon, 06 Oct 2014 19:48:16 -0500

The main thing I’ve learned from my WikiApiary project is how horribly bad the spam bot problem is in the MediaWiki world.

WikiApiary Stickers

Mon, 21 Jul 2014 00:00:00 -0500

StickerMule had a great deal a couple of weeks ago so I had these fun WikiApiary clear vinyl stickers made. They really came out great!

If you want one for your laptop or whatever send me a note and I’ll put one in the mail.

Why WikiApiary uses pages and not articles

Tue, 27 May 2014 00:00:00 -0500

A while back @Waldyrious posed this question about WikiApiary.

@WikiApiary in the lists of wikis by number of articles https://t.co/dKeb8eowPE, why not show the article count rather than the page count?
— Waldir Pimenta (@waldyrious) March 5, 2014

The full answer is actually found in my talk page on WikiApiary. Articles are a construct that each wiki can manage. MediaWiki tries to create these two classes of content: Pages and Articles. The intent being that pages are in some way less like “content” than articles. The metrics for determining this are configurable, but I’m guessing nearly everyone leaves it at the defaults. I felt that the page concept, just looking at content and not judging if it was better or worse was a simpler way to look at thousands of different wikis.

Sun, 18 May 2014 21:58:31 -0500

Tried to get the new WikiApiary Celery bots live tonight but “daemonizing” celeryd got the best of me for tonight.

Keybase and MediaWiki

Sun, 11 May 2014 00:00:00 -0500

I’m really intrigued by what Keybase.io is doing with identity. The ability to cryptographically prove your identity on the web without a centralized party like Twitter or Facebook owning the approval process is a needed function. I setup my profile and you can now prove on your own that I am who I say I am on Twitter, Github and five of my domains.

I’m trying to figure out how this could be extended to MediaWiki. I would love to be able to prove that my user account is me at:

This seems really hard. The method used on Github is to public a Gist, and you could certainly have a wiki user publish something on their user page, but that can also be edited by anybody.

But if we could do this, it would be a great way to allow Wikipedia editors to claim ownership with proof of their identities (if they wish) and would benefit thousands of self-hosted MediaWiki websites.

I think something more like the Twitter proof could work. How?

Have the user in question edit their User page. The contents of the edit don’t matter, the Summary field is what will be looked at. (Limited to 255 characters)
MediaWiki websites have a permanent revision history attached to each page. (This would be like a Tweet, see my change on May 11 2014)
This information is accessible via the MediaWiki API, and there is a revid attached to each revision.
This revision ID can be used to pull the proof forward for keybase.io.

Here are the last 5 revisions for my User page on WikiApiary (API call).

{
  "query": {
    "pages": {
      "43": {
        "ns": 2,
        "pageid": 43,
        "revisions": [
          {
            "comment": "This is my message for keybase.io to prove I am who I am!",
            "parentid": 884728,
            "revid": 904406,
            "timestamp": "2014-05-11T13:10:17Z",
            "user": "Thingles"
          },
          {
            "comment": "remove gittip button",
            "parentid": 569866,
            "revid": 884728,
            "timestamp": "2014-04-30T02:55:26Z",
            "user": "Thingles"
          },
          {
            "comment": "",
            "parentid": 569864,
            "revid": 569866,
            "timestamp": "2014-02-21T01:56:28Z",
            "user": "Thingles"
          },
          {
            "comment": "added badge",
            "parentid": 552628,
            "revid": 569864,
            "timestamp": "2014-02-21T01:54:59Z",
            "user": "Thingles"
          },
          {
            "comment": "add babel box (needs templates and styles)",
            "parentid": 528928,
            "revid": 552628,
            "timestamp": "2014-02-15T19:36:51Z",
            "user": "Thingles"
          }
        ],
        "title": "User:Thingles"
      }
    }
  },
  "query-continue": {
    "revisions": {
      "rvcontinue": 528928
    }
  }
}

Now that we have the revid we can find the information for this revision to continue to claim proof (API call). The key is that the title of the page “User:Thingles” matches with the user that made the change “Thingles”.

{
  "query": {
    "pages": {
      "43": {
        "ns": 2,
        "pageid": 43,
        "revisions": [
          {
            "comment": "This is my message for keybase.io to prove I am who I am!",
            "parentid": 884728,
            "revid": 904406,
            "timestamp": "2014-05-11T13:10:17Z",
            "user": "Thingles"
          }
        ],
        "title": "User:Thingles"
      }
    }
  },
  "query-continue": {
    "revisions": {
      "rvcontinue": 884728
    }
  }
}

And now I’ve proven my identity. The only trick is that the comment must contain all the data, and that should be easy since keybase already does something similar for Twitter.

And here is another example of me making a proven comment on my user page on Wikipedia (en) (API call).

{
  "query": {
    "pages": {
      "14604697": {
        "ns": 2,
        "pageid": 14604697,
        "revisions": [
          {
            "comment": "Here is my message for Keybase from Wikipedia.",
            "parentid": 551074105,
            "revid": 608054849,
            "timestamp": "2014-05-11T13:56:25Z",
            "user": "Thingles"
          }
        ],
        "title": "User:Thingles"
      }
    }
  },
  "query-continue": {
    "revisions": {
      "rvcontinue": 551074105
    }
  }
}

Sat, 10 May 2014 00:00:00 -0500

At least now the code doesn’t look like it was made by a monkey.

Paul commenting on the structure of my WikiApiary code after doing some refactoring.

Huge Impact of Linode Cloud Updates

Sun, 20 Apr 2014 00:00:00 -0500

I host all of my personal projects on two servers at Linode. Last week Linode announced new “cloud servers” with SSD’s, double RAM and a new chip architecture. I migrated both of my hosts over to the new servers that evening and the performance impact was immediately noticeable. WikiApiary is the most taxing project I run, and it was clearly much faster. This graph though of the WikiApiary API response time is the most telling. So much better!

Sat, 22 Mar 2014 12:09:00 -0500

Submitted pull request to add WikiApiary extension counts to Shields.io!

Mon, 20 Jan 2014 21:58:00 -0500

I was so disappointed this wiki wasn’t about actual fish hooks.

Wed, 08 Jan 2014 23:14:39 -0500

Got WikiApiary database moved to yoyo-migrations. Yoyo works well.

Thu, 02 Jan 2014 12:37:14 -0500

Fun day hacking on WikiApiary and hanging out at Urban Bean Coffee. Working on adding Celery to the bots. Thanks Paul for guidance!

Tue, 24 Dec 2013 09:52:00 -0500

Didn’t have my anti-spam settings right for Wiki on a Stick. It was blank, likely will just reset it.

One Year for WikiApiary

Sun, 22 Dec 2013 01:00:00 -0500

Yesterday WikiApiary had a very meta tweet when it wished itself Happy Birthday! A while back I realized that if you looked the edit history of the “Main Page” of a MediaWiki website you could infer the date the wiki was started, it’s birthday. WikiApiary then wishes wikis a happy birthday on that date. Yesterday was WikiApiary’s big day.

Happy Birthday WikiApiary http://t.co/wRk4Cs5iO0 #MediaWiki #wiki
— WikiApiary (@WikiApiary) December 22, 2013

The first year of WikiApiary has been great! The comments people make about it and the great contributions that many people have made to the wiki reflect the utility and interest in the data. WikiApiary was a holiday break project for me in 2012 and it’s continued to get additions and modifications from a number of people throughout the world. It is the first project I’ve started that I feel has gotten a true community around it and people that are moving it forward independent of what I’m doing. That is really great! This idea of a “Wiki to track other wikis” clearly caught on with some people.

In WikiApiary’s first year it has collected 1,855,979,520 statistics samples in its database, just 2.6GB of data. As of today, WikiApiary is collecting data from 9,555 active wikis. It shows 2,478,637 active users over 384,870,041 pages with 2,894,060,197 edits in the part of the wikiverse that it monitors.

Looking at visitor activity during this first year, WikiApiary had 32,416 visits with 105,895 page views. 9,346 of those visits were from MediaWiki.org. The top 10 countries visiting the site were:

Country	Visits	Pct
United States	10,243	31.6%
Germany	3,422	10.6%
United Kingdom	2,095	6.5%
Russian Federation	1,853	5.7%
France	1,084	3.3%
Canada	1,000	3.1%
Netherlands	814	2.5%
India	755	2.3%
Spain	714	2.2%
China	687	2.1%

WikiApiary visitors weigh heavier than average to Linux.

Operating System	Visits	Pct
Windows 7	14,989	46.2%
Linux	4,029	12.4%
Windows XP	3,876	12%
Mac OS	3,473	10.7%
Windows 8	1,707	5.3%

Chrome dominates the browser choice for WikiApiary visitors.

Browser	Visits	Pct
Chrome 26.0	2,377	7.3%
Safari 6.0	2,319	7.2%
Chrome 30.0	2,087	6.4%
Chrome 28.0	1,976	6.1%
Chrome 27.0	1,756	5.4%

2,455 of the 32,416 (7.6%) visits to WikiApiary were from logged in users. All websites statistics are from the amazing Piwik project. No data is shared with Google or other search engines.

WikiApiary is largely about graphs, so that seems like a logical way to explore the first year of WikiApiary. The number of active users on WikiApiary has roughly been around 30 for most of the year, peaking over 50. This doesn’t seem like a ton, but most wikis that are monitored actually have fewer than 5 active users. The total number of users is over 250 and grows steadily. Those are all real accounts too, no spam accounts. Registration is required to edit so this is a good reflection of engagement.

Edit activity on WikiApiary is mostly robotic. The bots are constantly tending to the data set and they do this with edits. You can see the edit rate jumped in October after I added tracking for MaxMind geo data as well as Whois records for wikis. Over 5 million edits in the first year.

Total pages of content largely reflect the number of wikis being tracked, plus the number of extensions and skins that exist. Notably you can see the initial load of sites in February and March. There were additional farmer bots that added in some reasonably sized farms in June. In October the pages spike again with the addition of more datasets.

WikiApiary is itself the 11th largest Semantic MediaWiki installation that it tracks. The largest is Gyvosios gamtos enciklopedija with over 16 million properties (think of a property as a data value). WikiApiary has over 3.3 million property values.

These 3.3 million properties are queried in MediaWiki templates so you can see the data. There are nearly 140,000 queries in WikiApiary.

Special Thanks

WikiApiary has had a lot of contributions with additions of wikis and help with templates and bots. Karsten Hoffmeyer has been a huge part of WikiApiary and is also an administrator on the site. Karsten helps with adding wikis and fending off the occasional bad edits. WikiApiary also has a very distinctive look from the Foreground skin which was built by my friend Garrick van Buren. Mark Hershberger has also been an active part of WikiApiary and is exploring ways that MediaWiki installs can automatically add themselves to WikiApiary. Huge thanks to Frederico Leva (Nemo) for linking extension pages on MediaWiki.org to their respective page on WikiApiary. This drives a lot of exposure for WikiApiary and provides great value to visitors of MediaWiki.org. A big thank you also to Paul DeCoursey who rewrote the Javascript code to embed the charts into the pages, and support multiple charts with usable controls.

Also, I think one of the things that makes WikiApiary unique is that it is built with MediaWiki and Semantic MediaWiki and the related suite of extensions. This is such a wonderful set of software and a special thanks to James HK, Jeroen De Dauw and Yaron Koren. All of them have helped out and provided input on WikiApiary at times in the first year.

Future Plans

I’ve got a ton of plans for WikiApiary, and I keep picking them off slowly. I’ve not had much time for the project the last couple of months but whatever time I have had has been going into rewriting the bots. The first versions were just hacked up and difficult to understand. I’m working on a rewrite that includes unit tests, a good object model and code that is easy enough to understand that I hope to get some more contributors involved. The other huge thing being added is parallel requests. Right now WikiApiary is limited in collecting from more sites due to how it collects, in serial. The new bots will do that in parallel and will dramatically change the cost of running a collection sequence. There should be no problem going from 10,000 to 100,000 or more wikis being monitored.

I would also like to see the Honey Bee MediaWiki extension get going which will be the first step of an extension that leverages WikiApiary inside of the wiki it’s running in.

Additionally I’d like to do a whole deeper level of analysis of MediaWiki websites and have been contacted by two groups who have written algorithms that do this and are interested in adding their code to WikiApiary. I hope to make that easier with the bot rewrite mentioned above.

I also want to provide a base Farmer class that can easily be extended so that bots that farm new wikis into WikiApiary are easier. My big objective is to finally pull in Wikia.

I’m proud of WikiApiary and plan on continuing to host it (no small feat actually, given its scale) and work on it. I see WikiApiary as one of my “decade project” so I don’t have to move too fast. I just keep things rolling the right way.

Sat, 30 Nov 2013 10:48:58 -0500

Stumped on this dumb Python import nosetests error. Help Jeroen De Dauw Paul
Finally figured out magic for relative module imports.
Got WikiApiary bots flowing through Travis CI and Coveralls (see badges)
Integrated WikiApiary bots into Landscape. Starting with brutal 19% rating.
Got my Landscape score from 18% to 43%. Still poor, but improving!

Thu, 28 Nov 2013 08:38:00 -0500

Made a profile for myself on Ohloh and also setup WikiApiary as a project on Ohloh.

Wed, 30 Oct 2013 04:47:00 -0500

1,000+ editors

@wikiworks: Most collaborative #SMW-based #wiki via @yolandagil research: @Mozilla Wiki #SMWCon

Sat, 19 Oct 2013 20:54:52 -0500

Running WikiApiary, a wiki that tracks as many MediaWiki sites as possible, sometimes you find some really strange stuff out there.

Stopping MediaWiki Spam with Dynamic Questy Captchas

Tue, 09 Jul 2013 00:00:00 -0500

This method of using Questy Captcha has been defeated by some spammers. Please check out my updated dynamic Questy Captcha method. MediaWiki websites are often plagued by spammers. It’s annoying in the extreme. If you setup a blank MediaWiki website and do nothing it is likely that within a couple of weeks your site will be found, and in a matter of days you will have thousands of spam user accounts and tens of thousands of pages of spam. There are a number of ways to stop wikispam. I tried using Recaptcha to little benefit. I still got a large number of spam registrations on my publicly available wikis. I’ve found the combination below to be incredible efficient.

I started to use QuestyCaptcha which is a plugin for the ConfirmEdit extension (WikiApiary) which uses a simple question/answer paradigm and that worked well. However, the hard part with Questy is figuring out what questions to use. Particularly if your wiki is global, you need to avoid using questions that are specific to one culture or location, or even language. What color is the sky? Well, in what language? For WikiApiary I want to make sure that people from anywhere are able to register. I started with simple questions, like “What is the name of this website?” but that was quickly defeated and spam registrations started showing up. What to do?

I decided to see if QuestyCaptcha could accommodate dynamic questions and it can! So, the first thing I did was decide to use a question that required someone to know something generic, but could also be easily found with a single click on a URL. My choice was to ask about the GMT time, because if you search for “gmt time” on Google, it tells you the answer. The PHP function gmdate will also give the answer. So I created two questions that asked for the day of the week and the hour (24 hour time) at GMT, and provide hyperlinks to the answer. This worked great!

Then I decided to go a little further and ask a very dynamic question. This time I generated an 8 character random string, and ask the user to identify one of the characters in the string. No language issue! No culture challenges! Simple. Here is the code for both of these solutions as it would appear in your LocalSettings.php file.

# Let's stop MediaWiki registration spam
require_once( "$IP/extensions/ConfirmEdit/ConfirmEdit.php" );
require_once("$IP/extensions/ConfirmEdit/QuestyCaptcha.php");
$wgCaptchaClass = 'QuestyCaptcha';

# Set questions for Questy
# First a couple that can be answered with a linked to Google search
$wgCaptchaQuestions[] = array (
    'question' => "What day of the week is it at <a href="http://google.com/search?q=gmt+time">Greenwich Mean Time</a> (GMT) right now?",
    'answer' => gmdate("l")
);
$wgCaptchaQuestions[] = array (
    'question' => "In 24-hour format, what hour is it in <a href="http://google.com/search?q=gmt+time">Greenwich Mean Time</a> (GMT) right now?",
    'answer' => gmdate("G")
);

# Now a more complicated one
# Generate a random string 8 characters long
$myChallengeString = substr(md5(uniqid(mt_rand(), true)), 0, 8);
# Pick a random location in those 8 strings
$myChallengeIndex = rand(0, 7) + 1;
# Let's use words to describe the position, just to make it a bit more complicated
$myChallengePositions = array ('first', 'second', 'third', 'fourth', 'fifth', 'sixth', 'seventh', 'eighth');
$myChallengePositionName = $myChallengePositions[$myChallengeIndex - 1];
# Build the question/anwer
$wgCaptchaQuestions[] = array (
    'question' => "Please provide the $myChallengePositionName character from the sequence <code>$myChallengeString</code>:",
    'answer' => $myChallengeString[$myChallengeIndex - 1]
);

# Skip CAPTCHA for people who have confirmed emails
$wgGroupPermissions['emailconfirmed']['skipcaptcha'] = true;
$ceAllowConfirmedEmail = true;

After putting these in place I’ve had nearly zero spam registrations (1 or 2 were clearly done by a human testing it). Now, can this be broken? Sure, easily. But not nearly as easy as static questions that could be harvested by a person and then put into a tool to automatically create accounts. In order to attack me, spammers would have to write a special handler that dealt with the randomness of the questions. This is very unlikely.

Feel free to use these examples, or, use other dynamic question/answer combinations. It’s not obvious that this type of configuration works with QuestyCaptcha, but it does and it allows for very powerful spam blocking.

Sat, 18 May 2013 22:10:00 -0500

Figured out the root cause of slowness on my MediaWiki farm.

Display MediaWiki job queue size inside your wiki

Sat, 20 Apr 2013 00:00:00 -0500

I wanted to find an easy way to show the current size of the MediaWiki job queue on WikiApiary. When you make changes to templates that are used on thousands of pages the queue can get backed up and it’s nice to have an easy way to keep an eye on this. The job queue is even one of the data points WikiApiary tracks and graphs. But I wanted something that was as close to realtime as possible. It wasn’t hard to do. This solution uses the External Data (see WikiApiary usage page) extension.

First we need to get the data. Let’s start by calling the siteinfo API method. The magic words are used to make this generic, and this should result in the right URL. If you are using a protocol-relative server setting you will have to modify this.

{{#get_web_data: url={{SERVER}}{{SCRIPTPATH}}/api.php?action=query&meta;=siteinfo&siprop;=statistics&format;=json
  | format=JSON
  | data=jobs=jobs}}

Now External Data has done the work and the value is stored for us. Now we get it by simply calling:

{{#external_value:jobs}}

I like to put that in a style so it’s big and obvious.

If you’re thinking ahead you’ll now be saying “Yeah, that’s neat, but it will be cached in MediaWiki for a hours!”. Yes, it will be unless you add a __NOCACHE__ directive to the page and use the MagicNoCache extension. This extension allows you to disable the MediaWiki cache on a page-by-page basis, very handy.

If you wanted to use this in multiple places you could even put an <onlyinclude> around it and transclude the job queue size in other pages, however I would be cautious about that if using the __NOCACHE__ directive as well.

Sun, 14 Apr 2013 22:58:52 -0500

Digging some ideas of putting Panic Status Board panels together with data from WikiApiary. Hmmm…

Tue, 12 Mar 2013 00:19:00 -0500

Took my Linode VPS to a loadavg of 13+ with a nasty bug in a WikiApiary bot. Fixed now. Happy LiSH got me on console!

Thu, 31 Jan 2013 21:10:48 -0500

I just created my first ever MediaWiki barnstar — You make WikiApiary awesome! I think this level += 1 me in wiki speak.

Sun, 06 Jan 2013 11:10:00 -0500

Attempting to write Python for WikiApiary that is not embarrassing. 🙂