Its not a buffering issue, because no matter where I view it (home DLS, work T1) it still has the same performance. I have also fully buffered the videos before viewing them and even that didn't help things. Its also not a Mac problem as the same thing happens under Linux (no surprise) and Windows.
I realize that HTML 5 has a video tag built-in, but I don't think I can wait that long. :/