Jon’s Project Blog

Several times I’ve had to save authenticated wiki content to my phone to view it on long flights (across the US). I found wget + Firefox 3.0 (Export Cookies extension) to be quite sufficient. The extension page explains a bit about how to use wget.

The basic strategy is:

1. Login to website, and export cookies and session information to a text file (“cookies.txt”).
2. Run wget and try to figure out what a reasonable download rate is. Some websites obviously don’t like bots, and so you may need to look through the wget options to slowly go throught the site. You may also need to periodically update your cookies or session info via the text file so don’t be too greedy.

As usual, you may want to look into the following wget options:

-I,-r,-l,-p,-k,-np

You may need to play around with the timing as websites generally don’t like being spidered. It also may be be illegal should check the Terms of Service.

The following is a sample commandline used to save webpages and attachments to a directory for offline use. I’ve noticed that firefox on the mac has issues with lack of extensions from wiki sites.

Example:

wget --random-wait --load-cookies=cookies.txt --save-cookies=cookies.txt --keep-session-cookies -r -l 2 -p -k -I /pages/ http://***:8080/pages/viewpage.action?pageId=xxxxxxx download/attachments

Since I am currently attending Stanford, and I once searched the intraweb for a solution for downloading wmv streams, but couldn’t find anything specific, I’ll post my solution here.

Normally, SCPD requires you to login and then you watch the videos streamed using some wmv player. However, should you want to rewind or seek through a stream, this will prove to be impossible. Many commercial wmv recorders are not free, and simply play the video to download it. My solution is similar to this, but it is FREE (as in beer).

What you need:

1. Mencoder (An encoder that comes along with Mplayer) If you use windows, you should get the command-line non-gui download from the website since it will come with mencoder. Otherwise it will not.
2. Optional: Firefox + Greasemonkey for extracting the video URL if you’re lazy like me.
3. Optional: Python for executing mencoder and for converting http:// links to mms:// links.

To download from SCPD (or other unprotected wmv streams), simply execute the following at a command prompt:

mencoder mms://stream.wmv -ovc copy -oac copy -o output.avi

1. Replace mms://stream.wmv is simply the url with “http:” replace with “mms:”
2. Replace output.avi with the output file name. Extensions should govern the format that it spits out.

It will throw a bunch of text to the screen, and you simply leave the window open (or minimized) and wait for it to finish. However, since it’s free, I commonly download several streams at the same time, so when they are done I will have several streams downloaded instead of just one.

Download: Greasemonkey script for SCPD (gets links automatically when you view an SCPD video) (Updated 9/29/09)

If you are lazy like me, and don’t want to change the http://url to mms://url I’ve provided an mencoder script that, given an http url, will fill out the commandline and execute it for you. (Note: Many mac and linux users will have python already installed on their machines.)

Usage:

./stripSCPD http://url

Download: Python SCPD script

#!/usr/bin/python
 
import subprocess,sys
 
def strip(url):
    newurl = "mms"+url[url.find(":"):]
    print newurl
    vidname = url[url.rfind("/")+1:url.rfind("?")]
    print "mencoder %s -ovc copy -oac copy -o %s.avi" % (newurl,vidname)
    subprocess.Popen(["mencoder", newurl, "-ovc","copy","-oac","copy","-o", ("%s.avi" % (vidname,))])
 
if __name__=="__main__":
    strip(sys.argv[1])

Update (11/09/08):

I’ve just tried everything again in windows, and it seems to work. Note, for some reason mplayer doesn’t have the necessary codecs by default in the downloadable zip file. However, VLC, media player classic and Windows media Player seem to be able to seek. However only media player classic can increase the audio speed (in case you want to watch a stream 1.5 times fast and be able to comprehend what someone is saying even if they seem to be on helium!).