Jon’s Project Blog

Several times I’ve had to save authenticated wiki content to my phone to view it on long flights (across the US). I found wget + Firefox 3.0 (Export Cookies extension) to be quite sufficient. The extension page explains a bit about how to use wget.

The basic strategy is:

1. Login to website, and export cookies and session information to a text file (“cookies.txt”).
2. Run wget and try to figure out what a reasonable download rate is. Some websites obviously don’t like bots, and so you may need to look through the wget options to slowly go throught the site. You may also need to periodically update your cookies or session info via the text file so don’t be too greedy.

As usual, you may want to look into the following wget options:

-I,-r,-l,-p,-k,-np

You may need to play around with the timing as websites generally don’t like being spidered. It also may be be illegal should check the Terms of Service.

The following is a sample commandline used to save webpages and attachments to a directory for offline use. I’ve noticed that firefox on the mac has issues with lack of extensions from wiki sites.

Example:

wget --random-wait --load-cookies=cookies.txt --save-cookies=cookies.txt --keep-session-cookies -r -l 2 -p -k -I /pages/ http://***:8080/pages/viewpage.action?pageId=xxxxxxx download/attachments

Just a quick post on the simple greasemonkey script that rips UCLA Bruincast video links (obviously you need access). I made it for my brother, so he can download lectures for classes he would like to save. Here’s the short greasy script. (Remember, rename the urls from “http:” to “rtsp:”). To grab the DSL links simply replace ‘LAN’ with the text of the link. Enjoy!

// ==UserScript==
// @name           BruinCastPageLinks
// @namespace      hawflakes.unoc.net
// @description    Grabs Links on a page
// @include        http://www.oid.ucla.edu/webcasts/courses/*
// ==/UserScript==
 
// Greasemonkey Script written by Jonathan Wong, Copyright 2009
// Script is freeware (Use at your own risk!).
 
Links = findXPathNodes("//a[text()='LAN']/@href");
PrintString="";
 
for(i=0;i<links .snapshotLength;i++)
{
	PrintString=PrintString+Links.snapshotItem(i).value+"\n";
}
 
GM_openInTab("data:text;charset=UTF-8," + encodeURI(PrintString).replace(/&nbsp;/g,"&amp;nbsp;"));
 
function findXPathNode(xpath, start,doc)
{
	var result = (doc == null ? document : doc).evaluate(xpath,(start == null ? document : start), null, XPathResult.ORDERED_NODE_SNAPSHOT_TYPE ,null);
	return (result.snapshotLength > 0 ? result.snapshotItem(0) : null);
}
 
function findXPathNodes(xpath, start,doc)
{
	return (doc == null ? document : doc).evaluate(xpath,(start == null ? document : start), null, XPathResult.ORDERED_NODE_SNAPSHOT_TYPE ,null);	
}
 
function elem(tagname,content)
{
	var ret = document.createElement(tagname);
	ret.innerHTML = content;
	return ret;
}
</links>

So apparently mencoder only seems to work for asf,wmv,avi,mpeg streams, and does not like real media streams (I think this is purely due to the fact I do not have real media codecs on this machine and therefore -ovc/-oac options would not work anyway). So I’ve defaulted back to using mplayer for ripping the real media streams. Fortunately, I had remembered seeing this, which pretty much explains what one needs to do.

For UCLA, the linked .rm file is actually just plain text that gives you the actual url. However, in the case of UCLA (like stanford), one can simply replace “http” with “rtsp”.


mplayer -noframedrop -dumpfile out.rm -dumpstream rtsp://url.rm


The most interesting piece of information is that on windows, it is possible for mplayer to rip streams even if it can’t actually play it. For example, my Windows machine (what I’m using atm) does not have real player or alternative/hacked rm codecs on it, and therefore I can’t actually play .rm. However, I was able to rip it and someone else was able to play it. This does make sense.

I haven’t posted in awhile, since I’m in the middle of several side projects and research. However, since many people that visit here seem to be looking at my attempts at coding with GWT, I figure I’ll blog about my MathML-in-gwt project I’ve been (ever so) slowly working on.

The goal of the MathML-in-GWT project was to make it easier for people to develop “math” applications for learning. One such application would be for generating basic math problem sets for arithmetic for 3rd – 8th graders. As a student who preformed below average on National Standardize testing in 1st and 2nd grade (how hard could those questions have been!), my parents enrolled me in the torture test of Kumon. If you don’t know what Kumon is, it’s essentially nothing other than the words “practice makes perfect” put into practice. For math I was given literally sets of 200 to 300 problems, many of which were repeated (I was lazy!). Students were typically timed, and if you didn’t finish a problem set within a given amount of time you didn’t pass. After two or three years of this test-taking mathematical torture, I was top my class in math (well almost at the top, the kid ahead of me did Kumon too). No matter what teachers in America say, American students could use more practice when it comes to solving math problems. However, why should anyone pay for Kumon? All they do is randomly generate numbers and print them out on cheap paper. Educational tools for generating such  simple tasks should be available (for free) and be open to encourage use.

I’ve prepared and example which I will reference further down. However, first I will explain a few issues about implementation and etc.

I know there is a gwt-mathml project also on Google Code, but it looks like it’s much more dead than mine. My implementation uses stock GWT (as usual). There are some issues such as IE requiring some voodoo MathPlayer plugin in order to display MathML. This requires adding several annoying header lines to trick IE into loading up MathPlayer. Other than that, I think my implementation is relatively clean and relatively straight forward, although it is still relatively simple (doesn’t support styles and such).

I decided to base all MathML elements on ComplexPanel, since it allows me to keep the relative hierarchy that MathML provides and I can easily mix and match it with normal GWT Widgets. (I haven’t tried putting it into forms or anything.) Briefly looking over MathML, it seems most tags take either one, two, or a list of parameters. Thus I created base “Generic classes (that use Generics) that handle one two or a list of parameters, just to make it easier to generate the whole MathML tag set quickly. I learned how to use Java before Generics were implemented and had dropped the language for a few years (while working at a .NET company). Thus, I decided it would be fun to play around with Generics which have been implemented in GWT 1.5+.

To handle the differences between MathPlayer (not requiring a namespaceURI, but other peculiarities)  and how Firefox renders MathML (requiring a namespaceURI),  I am using Deferred Binding using Replacement. It was relatively straight forward to implement and it makes the implementation “clean”. This was another first experience, since I hadn’t use deferred binding outside of the context of internationalization.

The header lines which must be used (so that Mathplayer functions) are as follows:

<head>
<object id="mathplayer" classid="clsid:32F66A20-7614-11D4-BD11-00104BD3F987">
</object>< ?import namespace="m" implementation="#mathplayer"?>
...</head>

Yup. That’s it. It’s a little annoying, but I think it’s the most straight-forward and preferably clean for developers.

Here is an example Fraction Problem Set Generator. Feel free to play around with it. It allows you to change the header of the problem set to create custom notes or instructions.

The example code can be downloaded here.