I'm James and I write code. Sometimes I blog.

Grabbing all of the links to a Youtube Playlist in Python

Thu, 31 Jan 2013 11:41:55 -0600

My friend Jesse wanted a way to grab all of the links to videos on a YouTube playlist. That’s no problem. I can easily cycle through links and spit them out into an HTML page.

There’s just one bug that I can’t seem to fix. Each and every link in the output goes to Rick Astley - Never Gonna Give You Up. I guess some bugs were never meant to be solved.

Mandelbrot Set in Matlab

Fri, 25 Jan 2013 12:06:00 -0600

I wanted to see if I could create a Mandelbrot Fractal image in Matlab. After quickly looking at the Mandelbrot Set entry on Wikipedia, I coded up the following and ran it:

I’m kinda surprised that it took so little code to create the image. I’m pleased with the results.

Finding a 3D Rotation Matrix for two vectors in Matlab

Tue, 25 Sep 2012 09:46:00 -0600

Update: The formulation of the input has been updated to allow two matrices of 3 by n column vectors and return a 3 by 3 by n matrix of rotation matrices.

Here’s the problem:

I have a 3 by 1 vector and I need to know the rotation matrix to manipulate that vector to another 3 by 1 vector.

Let’s call the first vector ‘u’ and the second vector ‘v’. I’m looking for a 3 by 3 rotation matrix ‘R’ that satisfies this simple property:

R*u=v

That’s it!

I’m sure that you’ve encourtered solving systems of linear equations in an algebra class. They are pretty painful, right? In those cases, you are given ‘R’ and ‘v’ and you are told to solve for vector ‘u’. By comparison, those are easy. Here we go with the solution:

Solving Many, Many 3 by 3 Eigenvalue Problems in Matlab

Tue, 21 Aug 2012 15:54:00 -0600

My research is requiring me to solve many, many small eigenvalue problems, all of which are 3 by 3 matrices. Rather than use the built-in “eig” function, I wrote my own.

Determining the Circumcircles of a list of Triangles in Matlab.

Sat, 30 Jun 2012 15:04:00 -0600

Imagine that we have a set of two dimensional points scattered across a space. For each point, the X coordinates are in variable X, and the Y coordinates are in variable Y. We can passed these coordiantes into the “delaunay” function and get “TRI”, which is a list of triangles. We can view the triangle mesh of the Delaunay triangulation by passing it to the MATLAB function “trimesh”, but there is no function for viewing the circumcirlces produced by this function. That’s what I’ve created today.

This second block isn’t my code. I found it on the Mathworks website. I’m posting it here to demonstrate that the above code works. It’s just used for plotting the circles.

Let’s make some triangle meshes!

This produces the elegant picture below.

Distance from point to the nearest point along a line segment in Matlab

Fri, 02 Mar 2012 16:44:00 -0600

Examining Exit Codes of 4 simple programs with bugs: Python, Ruby, Perl, PHP

Thu, 15 Dec 2011 11:09:00 -0600

Strict type checking is an aspect of the compiler to make sure that your programs have properly declared all of the necessary variable names prior to their use. I believe it is essential to all modern languages to implement proper errors in the event that an undefined variable is used. Let’s look at 4 popular scripting languages to see how they handle this problem.

For each program, we will examine three things: the source code, the execution, and the returned exit code. In modern operating systems, a program that concludes execution successfully returns a code of 0. If any problems arise in the course of the program, it should return something other than 0. I used the UNIX command “echo $?” to check the exit code.

All 4 programs will print the word “Hi” to the screen, then it will try to print an undefined variable “thisDoesNotExist”. After that, it will print “Hi again”. We would hope that the program will abort some time during the execution of the program and tell us that “thisDoesNotExist” is undefined. We would also hope that the program would never return an exit code of 0.

First, my favorite little language: Python

badName.py:

print "Hi"
print thisDoesNotExist
print "Hi again"

Execution and return code:

jcchurch@mccarthy:~/tmp/exitCodes$ python badName.py
Hi
Traceback (most recent call last):
  File "badName.py", line 2, in <module>
    print thisDoesNotExist
NameError: name 'thisDoesNotExist' is not defined
jcchurch@mccarthy:~/tmp/exitCodes$ echo $?
1

Python gives me a good error message “NameError: name ‘thisDoesNotExist’ is not defined”. It also returns a status code of 1. This is what we should expect from a modern language.

Second: Ruby

badName.rb:

puts "Hi"
puts thisDoesNotExist
puts "Hi again"

Execution and return code:

jcchurch@mccarthy:~/tmp/exitCodes$ ruby badName.rb
Hi
badName.rb:2: undefined local variable or method `thisDoesNotExist' for main:Object (NameError)
jcchurch@mccarthy:~/tmp/exitCodes$ echo $?
1

Ruby also gives a good error message “undefined local variable or method `thisDoesNotExist’” and an exit code of 1.

Third: Perl

Not every language designer agrees with me that strict type checking is essential. Perl requires us to use the statement “use strict;” to use strict type checking. I wish this were turned on by default.

badName.pl:

use strict;

print "Hi\n";
print $thisDoesNotExist;
print "Hi again\n";

Execution and return code:

jcchurch@mccarthy:~/tmp/exitCodes$ perl badName.pl
Global symbol "$thisDoesNotExist" requires explicit package name at badName.pl line 4.
Execution of badName.pl aborted due to compilation errors.
jcchurch@mccarthy:~/tmp/exitCodes$ echo $?
255

Perl doesn’t even print “Hi”. It gives the error message “Global symbol “$thisDoesNotExist” requires explicit package name”, which is a tad cryptic, but it does the job we want it to do. It also returns with a exit code of 255.

Fourth: PHP

I’m breaking my rule of never posting PHP code on this site. Like Perl, strict type checking is turned off by default. We can turn it on with the statement “error_reporting(E_ALL);”.

badName.php:

<?
error_reporting(E_ALL);
print "Hi\n";
print $thisDoesNotExist;
print "Hi again\n";
?>

Execution and return code:

jcchurch@mccarthy:~/tmp/exitCodes$ php badName.php 
Hi

Notice: Undefined variable: thisDoesNotExist in /home/jcchurch/tmp/exitCodes/badName.php on line 4
Hi again
jcchurch@mccarthy:~/tmp/exitCodes$ echo $?
0

PHP gives us a good error message: “Undefined variable: thisDoesNotExist”. But wait! It keeps going! We see “Hi again” AFTER the error message, which means this isn’t really an error message but a warning. When the program concludes, the exit code returned to the operating system is 0, which tells me that nothing went wrong in the program.

Conclusion

Python and Ruby demonstrate the same behavior: Execute as much of the program as possible until an error is reached. Once the error is reached, print an error and abort the program.

Perl compiles the entire program and notices the undefined variable at compile time. It prints the error without executing any of the code.

PHP executes as much of the code as possible. When it encounters the undefined variable, it happily prints a warning message and keeps on going. Finally, it tells us that there were no problems in the code with an exit code of 0.

Oh PHP. Maybe you will grow up as a language one of these days.

Top 10 List for Categorical Data in MySQL

Wed, 07 Dec 2011 03:39:00 -0600

Let’s say that you have a table with some categorical data. You want to make a Top 10 list of the most frequently visited categories. Here is an example table:

In this case, we have a table called “example” with one column called “label”. This is a bad example, in that it only has 10 items in the whole table.

mysql> SELECT * FROM example; +-------+ | label | +-------+ | A | | B | | C | | D | | A | | B | | C | | A | | B | | A | | E | | E | | E | | E | | E | +-------+ 15 rows in set (0.00 sec)

We wish to create a Top 10 List (err… Top 5 List) of the most frequently seen categories (or labels) in our table. Here is the code:

mysql> SELECT label, n FROM (SELECT label, count(*) AS n FROM example GROUP BY label) AS T1 ORDER BY T1.n DESC LIMIT 10; +-------+---+ | label | n | +-------+---+ | E | 5 | | A | 4 | | B | 3 | | C | 2 | | D | 1 | +-------+---+ 5 rows in set (0.00 sec)

There are two SELECT statements here! Let’s look at the inner SELECT statement. Here it is:

mysql> SELECT label, count(*) AS n FROM example GROUP BY label; +-------+---+ | label | n | +-------+---+ | A | 4 | | B | 3 | | C | 2 | | D | 1 | | E | 5 | +-------+---+ 5 rows in set (0.00 sec)

This bit of code groups labels together, pulls the label column and the count of each label (and names this count as ‘n’). It almost does the job, except the data isn’t sorted. You might wonder why we don’t use “ORDER BY n DESC” here. At this point, we’ve called a new column ‘n’, but we cannot use that new column in filtering or ordering.

Let’s move to the outer SELECT statement. I’ve replace the inner SELECT statement with “…”.

mysql> SELECT label, n FROM (...) AS T1 ORDER BY T1.n DESC LIMIT 10; +-------+---+ | label | n | +-------+---+ | E | 5 | | A | 4 | | B | 3 | | C | 2 | | D | 1 | +-------+---+ 5 rows in set (0.01 sec)

We are using the inner SELECT statement as a brand new temporary table for the outer SELECT statement. You can see the “AS T1”, which means that we are naming our inner SELECT statement “T1”. We order by T1’s column ‘n’ in descending order, limit the output to 10 records and again pull the columns ‘label’ and ‘n’.

In summary, creating a sorted ‘Top 10 List’ in MySQL requires a nested SELECT statement.

Linear Regression in Matlab

Thu, 13 Oct 2011 09:49:14 -0600

Determining if a graph is connected in Matlab

Thu, 16 Jun 2011 08:56:00 -0600

I needed a quick script to tell me if a graph was connected or not. My adviser usually tells me “MATLAB has everything!” No, it doesn’t, and that’s why we need to write simple code like this just to get more complicated jobs done.

Since I haven’t posted in a while, let’s do another script. This one will determine if a graph is symmetrical, which is another simple problem that MATLAB doesn’t have the code to do. As you can see, the code is very short.

Rotating Points around 3 axes in Matlab

Wed, 04 May 2011 16:25:02 -0600

I wrote this based on the content found on the Wikipedia page for Rotation Matrix and it worked on the first try. I fell like I need a victory lap.

Simplified Plotting (At least for me) in Matlab

Fri, 08 Apr 2011 15:59:00 -0600

Graph plotting in Matlab is rich with features that allow the user to create fantastic, beautiful plots. The downside to all of this is that when the user wants to create something simple, they must first return to the Matlab manuals.

I don’t care for any of this. I want to create simple 2D and 3D plots and I don’t really care about the current shape of the vector used in my plotting. I want a function which can accept multiple vectors filled with data and it figures out how the data should be plotted.

If I want a plot for a research paper, I’ll return to the manual to figure out something. For now, just throw the data on the screen as quick as Matlab can.

This function allows you to string together multiple vectors containing either 2D or 3D (or a combination of the two), with vectors in row-order or column-order (again, I don’t care), and throws it on the screen.

This supports up to 5 colors. More can be plotted, but once you’ve used your five colors, it loops back to the beginning.

Solving the Monty Hall Problem with a Monte Carlo approach in Java

Tue, 05 Apr 2011 14:48:00 -0600

mpipypi.py - Computing Pi using MPI Python bindings

Tue, 29 Mar 2011 17:00:00 -0600

Dijkstra's Algorithm in Matlab

Fri, 04 Mar 2011 17:19:00 -0600

Splitting Pages of a PDF in Python

Sun, 20 Feb 2011 23:44:00 -0600

For the past month, I’ve programmed in nothing but Matlab. Kinda sad, because I don’t even care for Matlab. I prefer Python.

I was going through a store called Tuesday Morning to look at the stuff for sale. It’s a bit of a junk store. They had a DVD for sale containing every Fantastic Four comic book on PDF from 1961 to 2004 for only $15. Being a comic fan, I had to get it.

Each issue is scanned into PDF format. Each 2 pages of the paper issue is combined into 1 page of the PDF. I didn’t like this, so I decided to write a Python script to cut each page of the PDF down the middle into two pages and stitch the document back together. This means that 1 page of a comic book equals 1 page of PDF.

I use is the handy pyPdf library. It’s doing all of the magic in this script. To execute this script:

$ python splitPages.py InputDocument.pdf OutputDocument.pdf

(Since the comics are intellectual property owned by Marvel, I’m going to not post screen shots.)

from pyPdf import PdfFileWriter, PdfFileReader
import sys

print "Reading", sys.argv[1]

output = PdfFileWriter()

left = PdfFileReader(file(sys.argv[1], "rb"))
right = PdfFileReader(file(sys.argv[1], "rb"))

left.decrypt('')
right.decrypt('')

pages = left.getNumPages()

for i in range(0, pages):
        # Grab the left page
        p = left.getPage(i)
        p.mediaBox.upperRight = (
            p.mediaBox.getUpperRight_x() / 2,
            p.mediaBox.getUpperRight_y()
        )
        output.addPage(p)

        # Grab the right page
        p = right.getPage(i)
        p.mediaBox.upperLeft = (
            p.mediaBox.getUpperRight_x() / 2,
            p.mediaBox.getUpperRight_y()
        )
        output.addPage(p)

print "Writing", sys.argv[2]

outputStream = file(sys.argv[2], "wb")
output.write(outputStream)
outputStream.close()

My O'Reilly Wish List

Wed, 26 Jan 2011 15:43:00 -0600

O’Reilly is currently holding a $500 giveaway. My entry form is a blog about the books, ebooks, and videos that I would purchase with $500. Beside every title is the dollar value of the product rounded up to the nearest dollar. After rounding, my wishlist totals $499. I think I hit the mark pretty close.

Paperback Books:

The Definitive ANTLR Reference - 37

Language Implementation Patterns - 35

Manga Guide to Calculus - 20

Hacking: The Art of Exploitation, Second Edition - 50

Ebooks:

Beautiful Data - 36

Beautiful Security - 32

Beautiful Architecture - 36

Beautiful Code - 36

jQuery Cookbook - 28

Data Analysis with Open Source Tools - 32

Statistics in a Nutshell - 35

Hadoop - 39

Bioinformatics Programming Using Python - 48

Videos:

Great R: Level 1 - 35

3D Visualization of an Adjacency Matrix in Matlab

Fri, 21 Jan 2011 17:30:00 -0600

An adjacency matrix is a data model to represent an undirected graph. Most adjacency matrices are two dimensional, which means you can use the built-in Matlab function “gplot” to visualize it. This function “gplot3” allows you to visualize adjacency matrices in 3D.

I created the above demonstration using Matlab’s built in Bucky Ball creator.

» [B XY] = bucky;
» gplot3(B, XY);

Here’s the code!

A Slinky in Matlab

Wed, 05 Jan 2011 17:30:07 -0600

I made this because I need to learn about 3D graphics, but I’m too lazy to learn Blender. This was made with 1 line of Matlab.

>> x = linspace(0, 200*pi, 10000); y = sin(x/100)+sin(x); z = cos(x); plot3(x, y, z)

Comparing Stock Volatility Using R

Sun, 02 Jan 2011 17:32:07 -0600

In my last post, I wrote a quick Python script to download 1 years worth of a stock’s history into a CSV file for given a list of stocks. Today, we are going to take those stocks an compare the volatility between each stock’s daily price change.

Measuring volatility is a simple calculation of the standard deviation of the percentage of the stock’s price change over a given year. The higher the standard deviation of percent change means higher volatility (and conversely, lower standard deviation means lower volatility).

This program takes a list of ticker symbols, opens up their corresponding CSV files (see yesterday’s post for more information), computes the standard deviation based on percent change, sorts that list, and prints.

In the results, you can see that I’m tracking 26 stocks. The three stocks with the lowest volatility are Proctor & Gamble (0.85%), Coca-Cola (0.99%) and Kraft (1.08%). The three stocks with the highest volatility are Micron Technology (3.24%), Netflix (3.44%), and Crosstex Energy Inc (3.62%).

Disclaimer: I’m not a broker and I haven’t been investing for very long. This information should not be used as investment advice, but it should be used as a tutorial on how to pull information out of a CSV file in the R language. How you interpret this data is up to you.

Here’s the data.

R version 2.7.1 (2008-06-23)
Copyright (C) 2008 The R Foundation for Statistical Computing
ISBN 3-900051-07-0

> source('stockSD.R')
          sdList
PG   0.008494452
KO   0.009928666
KFT  0.010838839
PFE  0.013281879
MSFT 0.013841909
YUM  0.014198831
SAP  0.014843072
FTR  0.015125052
ORCL 0.015317628
INTC 0.015906689
CPNO 0.016704188
AAPL 0.016787813
EMR  0.016905197
GE   0.017227925
GOOG 0.017538399
BIG  0.018467030
YHOO 0.018651226
GS   0.019989476
AMZN 0.020597942
BAC  0.023268540
C    0.023951356
F    0.023993128
RIMM 0.024404948
MU   0.032412592
NFLX 0.034474349
XTXI 0.036222946

Here’s the code.

symbols <- c("BAC", "YHOO", "RIMM", "MU", "MSFT", "GOOG", "PFE", "INTC", "C", "GE", "SAP", "FTR", "BIG", "ORCL", "AMZN", "XTXI", "GS", "KO", "F", "PG", "EMR", "AAPL", "KFT", "NFLX", "CPNO", "YUM")

sdList <- 1:length(symbols)

for (s in 1:length(symbols)) {
    infile <- paste(symbols[s], ".csv", sep="")
    df <- read.csv(infile)
    percent_change <- ((df$Close - c(0, df$Close[1:length(df$Close)-1]))/ c(0, df$Close[1:length(df$Close)-1]) )[-1]
    sdList[s] <- sd(percent_change)
}

# Create a frame for our standard devation list
sdDF <- data.frame(symbols, sdList)

# Order the standard devation list by standard devations in ascending order
sorted.sdDF <- sdDF[order(sdDF$sdList), ]

# Apply names to the standard devation list
rownames(sorted.sdDF) <- sorted.sdDF$symbols

# Take out the names 
sorted.sdDF <- sorted.sdDF[-1]

print(sorted.sdDF)