Moving from MediaWiki to SharePoint O365 - part 2

After getting our tools set up and acquainting ourselves with SharePoint O365, lets take a step back and think for a moment. The objective is to migrate all wiki content from MediaWiki to SharePoint O365. We can get to our content in MediaWiki, i.e. the raw content as the user types it in, or if need be the HTML-content produced by MediaWiki (using PHP with cURL). And I've got a way to upload HTML-content into pages in SharePoint. So this doesn't sound like such a challenge after all?

Wrong.

MediaWiki has a lot of user friendly formatting syntax. Like * for bulleted and # for numbered lists. Like ==heading==. Like [[some other article]]. Or how about [some external link]. And don't forget the [[Image:something|thumb|right]]. And then there's categories. And templates. And wikitables. And if you've had a bit of fun with the extensions, there's code highlighting. And math. And lots of other stuff.
SharePoint has HTML and an editor.


There's basically two ways to go about this. You could choose to grab the online (HTML) content, which doesn't present any MediaWiki syntax, and work with that. Just swap out the links pointing to a location on the wiki (like other articles, files, or images) to the location on SharePoint, and you're done. True and probably works fine for articles that use basic MediaWiki syntax and nothing more. If, however, you've had people using templates... Then you'll end up destroying the write-once-use-plenty they've set up. Or code highlighting - sure it will still look nice on your SharePoint. Trouble starts when somebody needs to make changes to the code - oops, that formatting and those colors are hard-coded? Or math... You'd end up copying pictures, and try and edit that later on.
The alternative is to use the original content, i.e. the one with the wiki-syntax. This will show template-calls, variables in templates, math source, code highlighted source, etc. But it will also show wiki-syntax for tables, headings, ordered and unordered lists, image positioning, etc, which would force us to translate all this 'common' wiki-syntax to HTML, while we get this for free in the online content.


Which is why I've chosen to use a combination of the two. It sounds complicated, but it actually makes things a lot easier.

  • All articles having no math, code highlighting or templates - which can be verified by scanning the wiki-syntax content - are copied over using their online content. Not a 1:1 copy of course, we still need to swap out image-, file- and intrawiki-links for links pointing to the new locations. 
  • On studying articles with math, it turns out the online content also has the original math source syntax (as alternate content to the image of the formula that it shows), so some processing can reformat the math-parts to work in SharePoint: take out the image-code, add a prefix and suffix to the math source syntax, and we're done.
  • For articles containing highlighted code, the online content of the highlighted code is not usable. But because every code block can still be recognized as such, we can combine the online content with the wiki-syntax content: we swap out all code blocks from the online content with the corresponding code blocks from the wiki-syntax content, and then update the code block prefix and suffix from wiki-syntax to SharePoint-syntax. (Actually: SyntaxHighlighter syntax. But we'll get to that later.)


What about categories?
I'm sorry my friend, but they've been lost to us. They are no more. You must let them go, and not let your mourning keep you from continuing to read (and work).
Seriously though, I haven't found a feasible way to get categories to work in SharePoint. There's a button for tags (but it's disabled in our installation) and in the settings there seems to be something like taxonomy, but I couldn't find anything which could emulate the Category tagging from MediaWiki.

But, so, templates and code highlighting and math, that works on SharePoint? We just need to throw it into a new format?
Sorry if I gave you that impression. SharePoint allows you to write HTML in their content editor. And if any of your HTML looks decidedly script-y, they'll automatically throw it into a Web part for you, just to be nice. Not that you're given a choice.
So, no, SharePoint doesn't have that stuff. The only thing still formatted like in MediaWiki is [[internal links]], everything else is done directly in HTML. And if you see the HTML the content editor produces... Oh boy... You'll think it's 1998 again.


Introducing... javascript

You being the Site owner and all, you'll be able to make changes to the master page, which includes adding scripts to it. So what I've done is made (a basic form of) templates, code highlighting, math and redirects by interpreting the page content through JavaScript. It is not super fast (as compared to code being executed on the server - although the server response times aren't great all by themselves either), and the user will see certain stuff loading/changing after the page has been loaded instead of it all being already parsed like in MediaWiki. But it does work, and it's more than you're getting from SharePoint out of the box. 

Step 1: jQuery

This one is easy. Download jQuery.
I suppose you could use a CDN, but then again do you really want (more) outsider scripts running on your confidential company information? I didn't think so.
Once you have it, you'll need to upload it. Go to your site in SharePoint, click the cog-wheel, go to Site contents, and open the Site assets library. Then click the Library button (top left somewhere), and in the toolbar that appears then, you'll see the Open with Explorer in the Connect & Export part.
If you find this button to be disabled, then you're probably not using Internet Explorer. Try again with the Microsoft browser, and the button should be available. Click it, and the library opens as a folder in Explorer. I've made a new folder 'JayVee' (call me vain) and copied the jquery.min.js to it.

Of course you've seen the upload button on the SharePoint library page itself, and seeing as it's only one file, wouldn't this have been faster? Maybe, but there's a whole lot more files that need to be sent there (thousands), and at that point you'll be very happy to know how to do this using Explorer, as the online upload only allows 99 files to be uploaded at once.

Also, consider donating to the people who bring you jQuery for free.


Step 2: templateCaller.js

This is my own script. I'm not a javascript master and never had the time to truly study the possibilities of jQuery, so I basically hacked this together from Google and StackOverflow. A lot of improvements to this script are probably possible. If you have some, please leave a comment on that page and enlighten me, so I can update it for myself and for other readers. 
The important thing for now is: it works.

You can find the code here. Copy it into a text file, rename it 'templateCaller.js' (it does more than that by now, but it started life as a script to work with templates, hence the name), and upload it to the same location as the jQuery script.


Step 3: MathJax

I work in an organisation where there's the occasional need to document some statistics and other math. We can use MathJax to have any existing math formulas be formatted nicely. Download an archive, because we don't want to use the CDN and risk letting scripts not under our control get a look at our confidential information.
Unpack the archive, and upload everything to SiteAssets/JayVee/MathJax. Consider donating to the people who bring you this for free.


Step 4: SyntaxHighlighter

There's a number of syntax-highlighting JavaScripts out there, and I had time only for a quick compare. So this may or may not be the script that suits your needs best. For me, it's a 90% match to my requirements: it can do JavaScript, SQL, PHP, batch, powershell, java and matlab (and many others, but these are the ones I care about most). It does not do SAS, which is a pity, but you can extend the script with brushes (there's one for every language it can highlight), so if need be I can eventually create a brush for SAS myself.
And very important (for me): it can automatically add line numbers. 

Download it from here, unpack, and put it in SiteAssets/JayVee/SyntaxHighlighter. Consider donating to Alex Gorbatchev, who made, maintains and improves this script and lets you use it for free.


Step 5: run the scripts on all pages

You really want to have the scripts available on all wiki pages, without having to add a block of code to each (Web parts, bleh). To do that, we're going to change the master page for wiki pages. What exactly happens to that master between the file as it is and a generated wiki page as you see it, I do not know, but the changes I've made to it work just fine. 

Using Internet Explorer, go to your Site settings (behind the cog-wheel). Under Web designer galleries you should find an entry Master pages. Click it, and you'll be brought to the library. Click the library link (top left somewhere) and then use the button to open the library in Explorer. From the explorer window, you can download the seattle.master file. I'm sure there's a story behind that filename, but I don't know what it is.
Using Eclipse (or your IDE of choice, or WordPad if need be), open the file. At a certain point you'll find this:

1
2
<SharePoint:CustomJSUrl runat="server" />
<SharePoint:SoapDiscoveryLink runat="server" />


Between these two lines, you need to add the references to the other scripts we're loading, making it:


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
<SharePoint:CustomJSUrl runat="server" />
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shCore.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shBrushAS3.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shBrushAppleScript.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shBrushBash.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shBrushBat.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shBrushCSharp.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shBrushColdFusion.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shBrushCpp.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shBrushCss.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shBrushDelphi.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shBrushDiff.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shBrushErlang.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shBrushGroovy.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shBrushJScript.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shBrushJava.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shBrushJavaFX.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shBrushMathematica.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shBrushMatlabSimple.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shBrushPerl.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shBrushPhp.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shBrushPlain.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shBrushPowerShell.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shBrushPython.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shBrushRuby.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shBrushSass.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shBrushScala.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shBrushSql.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shBrushVb.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shBrushXml.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/jquery.min.js"></script>
 <script type="text/javascript" src="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/templateCaller.js"></script>
<SharePoint:SoapDiscoveryLink runat="server" />

You'll notice syntaxHighlighter has a lot of scripts being loaded: that's actually just the core script (shCore.js) with a lot of brushes for specific languages. If you don't need all these languages, omit the corresponding lines. If you need more/other languages, make sure to upload the corresponding brush script from here (or here for a full list) and add a line in seattle.master for it.
At the end, we call jQuery and my own templateCaller.

A second change we need to do concerns css, we need to add a stylesheet for SyntaxHighlighter.
Find these lines:


1
2
 <SharePoint:CssRegistration Name="Themable/corev15.css" runat="server" />
</head>


And add a link to the stylesheet:


1
2
3
 <SharePoint:CssRegistration Name="Themable/corev15.css" runat="server" />
  <link type="text/css" rel="stylesheet" href="https://yourInstallation.sharepoint.com/sites/yourSiteNumber/SiteAssets/JayVee/syntaxHighlighter/shCoreDefault.css">
</head>


Step 6: what can you now do?

Now that everything is in place, how do you call the functionality offered by the scripts from within your page content? After all, migrating our MediaWiki content is one thing, but your users also have to know how to make new content on SharePoint.