MediaWiki to Wikee (or Ikiwiki or Git-Wiki)

By Albert on November 27, 2011 5:32 PM

I’ve been working on Wikee a bunch lately, and today I’m converting the Informed Banking wiki to be in git format (a format that will roughly work with GitWiki, Ikiwiki, and Wikee).

I should of course mention what Wikee is - I haven’t made an announcement about it yet - its a Rails 3 engine for adding a wiki to a rails 3 application. Its inspired by git-wiki and ikiwiki, and is generally awesome! I’ve got it setup on NeoCarz so far, and as I mentioned - I’m setting it up for Informed Banking today.

To do the conversion, I’m using an old ruby script I wrote that uses dbi to connect to mysql and grit to perform the git commands.

Its an old script, and I had to do some updating, but its working well again. If you are interested in converting a mediawiki installation, check out the script. It will likely take some work, but it might just save some time.

Here’s the general idea:

  1. git clone git://github.com/docunext/mediawiki2gitikiwiki.git
  2. or fork it
  3. cd mediawiki2gitikiwiki
  4. cp mydbsetup.rb.example mydbsetup.rb
  5. vim !$
  6. bundle install
  7. bundle exec ruby mw2iw.rb

As I mentioned, it will likely require some tweaking!

Additional information:

Requirements

  1 gem 'dbi'
  2 gem 'dbd-mysql'
  3 gem 'grit'

Example configuration file

  1 @mydb = Hash.new
  2 @mydb[:host] = 'DBI:Mysql:mediawiki:localhost'
  3 @mydb[:user] = 'username'
  4 @mydb[:pass] = 'password'
  5 @mydb[:prefix] = 'prefix_'
  6 @mydb[:gitpath] = '/path/to/repo'
  7 # if the imported wiki should go into a subdir
  8 @mydb[:subdir] = 'subdir/'
  9 @mydb[:extension] = '.mdwn'

Acceptance Testing in Rails3 with Rspec2, Capybara, and Devise

By Albert on November 21, 2011 12:33 AM

I finally figured out how to setup acceptance testing in Rails3 with Rspec2, Capybara, and Devise!

It actually wasn’t that hard, once I decided to manually login to create the user session instead of trying to use some magic sign-in method.

I researched the matter for awhile prior to manually creating the devise-based Rails3 session but could not find a simple example, so here’s one that is similar to what I’m doing:

require 'acceptance/acceptance_helper'

feature 'Vehicles', "Access vehicles" do

  background do
    Factory.create(:vehicle)
    User.create(
      :email => 'user@example.com',
      :password => 'password',
      :password_confirmation => 'password'
    )   
  end 

  scenario 'Viewing the index' do
    visit '/' 
    fill_in 'user[email]', :with => 'user@example.com'
    fill_in 'user[password]', :with => 'password'
    click_button 'Sign in'
    visit '/private-page'
    page.should have_content("My Special Content")
  end

end

Not too difficult, eh? I’m really starting to like Capybara - and I like how it integrates so well with Rspec2.

In researching this topic, I did find some useful posts, however. I read up about the “Database Cleaner” rubygem, and how folks are using it because selenium (an awesome testing framework I’ve used a bunch in the past) doesn’t work with transactional fixtures, and folks are using selenium because that’s how we can use capybara to test ajax integration. I’ve really been meaning to try out the “Database Cleaner” gem, just haven’t had the chance - but speaking of database-related gems, replicate by Ryan Tomayko is really terrific, in my humble opinion. It makes it really easy to import and export ActiveRecord objects from Rails applications from the command-line - even from one storage type to another, say from SQLite to MySQL, because it serializes the records as AR objects. The replicate gem can even handle associations, making it all that much more massively useful.

Pretty awesome, if you ask me.

Before closing, I should mention that while I’m using Devise to handle authentication, this method should work with others, like AuthLogic, or a custom built Rails session management system, though the key identifiers to enter in the credentials might be different. I’ve used AuthLogic in the past, and while it worked very well, I’ve since switched to Devise, and I have to say I like it better. There are a lot of third party extensions for it, and more importantly I found the documentation more accessible (and available).

Bundler / Rubygems and Git Source Problems?

By Albert on October 31, 2011 11:39 PM

I’m getting this odd error when trying a bundle install:

<NoMethodError: undefined method `yaml' for #<Psych::Nodes::Stream:0x9d78120>

Looks like the problem is related to yaml and psych. There is a related issue on Github, but this one appears new.

I’m trying to update bundler…

gem install bundler

Same problem with Bundler version 1.0.21 and Bundler version 1.1.rc, though in bundler 1.1.rc there is an improvement with the ever slow Rubygems metadata update:

Fetching gem metadata from http://rubygems.org/......

Yes! And the dots are even animated!!

Now trying to update psych. Still the same. Hmmm, I did add psych as a dependency in one of the rails3 engines I’m loading by a path. Let me try and take that out.

I’ve taken it out, and nothing fixed yet. Ack! I had to actually uninstall psych (v.1.2.2) to get it to work.

I reinstalled it, got the same error, but then tried psych v 1.2.1 and it worked.

Whoa! Then I tried running Rails and got a major error:

/home/albertlash/.rbenv/versions/1.9.3-rc1/lib/ruby/1.9.1/i686-linux/psych.so: warning: already initialized constant ANY
/home/albertlash/.rbenv/versions/1.9.3-rc1/lib/ruby/1.9.1/i686-linux/psych.so: warning: already initialized constant UTF8
/home/albertlash/.rbenv/versions/1.9.3-rc1/lib/ruby/1.9.1/i686-linux/psych.so: warning: already initialized constant UTF16LE
/home/albertlash/.rbenv/versions/1.9.3-rc1/lib/ruby/1.9.1/i686-linux/psych.so: warning: already initialized constant UTF16BE
/home/albertlash/.rbenv/versions/1.9.3-rc1/lib/ruby/1.9.1/psych/nodes/stream.rb:12: warning: already initialized constant ANY
/home/albertlash/.rbenv/versions/1.9.3-rc1/lib/ruby/1.9.1/psych/nodes/stream.rb:15: warning: already initialized constant UTF8
/home/albertlash/.rbenv/versions/1.9.3-rc1/lib/ruby/1.9.1/psych/nodes/stream.rb:18: warning: already initialized constant UTF16LE
/home/albertlash/.rbenv/versions/1.9.3-rc1/lib/ruby/1.9.1/psych/nodes/stream.rb:21: warning: already initialized constant UTF16BE
/home/albertlash/.rbenv/versions/1.9.3-rc1/lib/ruby/1.9.1/psych/nodes/sequence.rb:42: warning: already initialized constant ANY
/home/albertlash/.rbenv/versions/1.9.3-rc1/lib/ruby/1.9.1/psych/nodes/sequence.rb:45: warning: already initialized constant BLOCK
/home/albertlash/.rbenv/versions/1.9.3-rc1/lib/ruby/1.9.1/psych/nodes/sequence.rb:48: warning: already initialized constant FLOW
/home/albertlash/.rbenv/versions/1.9.3-rc1/lib/ruby/1.9.1/psych/nodes/scalar.rb:9: warning: already initialized constant ANY
/home/albertlash/.rbenv/versions/1.9.3-rc1/lib/ruby/1.9.1/psych/nodes/scalar.rb:12: warning: already initialized constant PLAIN
/home/albertlash/.rbenv/versions/1.9.3-rc1/lib/ruby/1.9.1/psych/nodes/scalar.rb:15: warning: already initialized constant SINGLE_QUOTED

It goes on, but that’s the idea. Installing psych 1.1.1 made it better.

First Attempt to Install Shapado

I’m trying to install shapado, a stackoverflow clone running on top of Ruby on Rails. It looks like a very solid piece of software. The shapado installation instructions are clear, but I’ve hit this snag:

Failed to connect to a master node at localhost:27017

Ah! Its because I don’t have mongodb running locally.

UPDATE on my efforts to (and failure to) install shapado:

I have given up on installing shapado, only because the version that uses Rails 3 requires a version on the mongo driver that as of yesterday was only available for Jruby, and as of today has been yanked from Rubygems. Hopefully they’ll bring a fresh version of the mongo driver back and it will be compatible with Ruby 1.9.

GitModel Rocks!

Tonight I’ve been working on GitModel, an ActiveModel powered ORM that uses Git as its storage mechanism.

Serializers

It originally used Yajl as its serialization layer, but I abstracted it out into a configurable module that can use json, yaml, or a custom interface I created to serialize to yaml front matter (similar to how Jekyll does it, and ikiwiki has a plugin for it, too).

The code is surprisingly simple and working on it was a breeze. The test coverage was pretty good, too, but I expanded it a bit.

Regulate

I plan to next use GitModel with regulate, which implements its own Git-powered ORM in a manner quite similar to GitModel.

Indexing

After that, if I have time, I’d like to look at improving the way the indexing is done in GitModel, exploring and evaluating some different options - including marshalling the index, as it doesn’t need to be human editable.

I’m also considering making a simple index with something like CDB, Tokyo Cabinet, or QDBM - maybe Memcache, too - its already included in GitModel for some components.

Relations

Beyond that I’d really like to have the syntax mimic that of Mongoid, and potentially even add relations like embeds_many and embedded_in. That would be a much more significant undertaking, obviously, and would require a bunch more indexing capabilities.

Questions

Folder Structure

I still can’t decide whether I want the storage files to be enclosed in a folder with a generic name like ‘attributes.json’, or have them stored in a file with the name of the object’s id. I used to want to have the plain file, but then I read the page on the ikiwiki site: switching to usedirs. That’s made me reconsider, however I think that argument refers to the rendering of the site, not necessarily the repository.

Integration Beyond (De-)Regulate

It would be really interesting if GitModel could integrate with something like jekyll or octopress. I’d also like to see an implementation of git-wiki using it, made with Rails of course. I’ve done a bunch of work on a git wiki fork, mainly trying to make it compatible with ikiwiki, which I was able to do.

I emailed the original author of GitModel earlier today:

Hi,

I have been working on a fork of regulate:

https://github.com/docunext/regulate

which is kind of like your balisong app. While the original regulate by quickleft also uses the attributes.json method, I switched to using front matter (sort of like yaml) after trying out jekyll.

Anyway, regulate implements a git interface that could probably be replaced by gitmodel, but it would need some flexibility as to the serialization process - i.e. making the database structure configurable so that the user could choose between json or yaml, and the database structure - either my-awesome-folder-name/attributes.serialized or my-awesome-filename.serialized.

If the latter sounds wacky, not to fret - Joey Hess of Debian awesomeness implemented the same in Ikiwiki.

Is this something you'd consider in the mainline? I've already done the conversion for regulate (and made it somewhat configurable). Abstracting the git interface out is something on the to-do list.

Albert

Hopefully we can collaborate, though I’d like to remove blob support, and I’m not sure the original author would be open to that.

The Source!

NOTE: Use of gitmodel from regulate is done through the “gitmodel” branch of regulate. Both are on GitHub, but note that regulate has a branch for gitmodel; master uses an internal interface.

Rails 3.1 Engines and Views Inheritance

By Albert on October 26, 2011 8:30 PM

I’m using Engines with Rails 3.1 extensively.

Ran into an interesting factor: when isolating the engine namespace, if the engine’s application controller is namespaced in a module without inheriting from ApplicationController (i.e. it inherits from ActionController::Base instead), the main app application layout will not be used.

A little background - with engines, most developers will want to either use the main app layout, or a layout included with the engine. For example, Rails Admin uses its own layout, where something like Forem will probably want to use the main app layout.

When merging the engine and the main app view layouts, there are a couple of pitfalls to be way of:

  • Inherited views causes colliding partial names - best to use specific partial paths like application/head instead of just head
  • View and path helpers - get used to using main_app.new_something_path

Want more specifics? I am trying to integrate the rails_blog_engine into NeoCarz v2 but it kept trying to use its own layouts - even after I removed them. To figure out what was going on, I compared and contrasted with Forem, which I am also using in NeoCarz v2 and which I know to use the main app layouts.

Categories