Back on May 15th, I wrote a 221-word entry on my first impression of The Matrix Reloaded. At last count (mid-afternoonish on June 17th), people have left 700 comments in the thread attached to that entry (7 of which are mine). Those 700 comments comprise a total of ~125,000 words (~180 per entry); that's about 3.3 150-page books. The HTML file is in excess of 1.3 MB in size, has been viewed about 5000 times in the past two weeks (during which about half of the comments were left), and is responsible, all by itself, for 5.3 gigabytes of data transferred from kottke.org this month.
In comparison to the rest of kottke.org (just a few more stats here), the most comments on previous thread was just over 200 (a Crouching Tiger, Hidden Dragon thread that is no longer online), there are around 3500 comments that have been left on the site in dozens of threads, and over the past 5 years, I have posted around 2500 entries to kottke.org, 27 of them since the Matrix post. My total bandwidth usage for June thus far is about 14.6 GB.
In short, the Matrix thread is unexpectedly large, and depending on how you look at it, is anywhere from 10% to 36% of my entire site, and significantly more than 50% of the total output since May 15th. If you were to look at all the content of the site in the aggregate, you might come to the conclusion that kottke.org is a Matrix Reloaded-related site even though it's been chugging along Reloaded-free for more than 5 years.
Given that the thread is still active (25 posts today alone), how do I deal with it going forward? I haven't read any of the conversation since around comment #200, and if it hits 1000 comments (which, 2 weeks ago seemed unlikely but now seems inevitable), the file approaches 2 MB in size, and sucks up 15 GB of bandwidth a month, it seems prudent for me to close the thread.
On the other hand, it continues to be valuable as a forum for a small number of participants and a larger number of readers...and from what I can tell, is a relatively high-quality conversation. Who am I to shut down a conversation that I'm not involved in? This may be my site, but the participants own the conversation. As much as it makes sense to shut it down, I'm inclined to let the participants go as long as they want. Who knows, maybe they'll even let me know when they're done.
Related: Sam Ruby's experiment of moderating discussions on weblogs.
Oh, and just for fun, I pasted the entire thread into Word and used the AutoSummary feature to come up with this 100 word summary of the thread:
The Matrix 1.1?
Wake up Neo...
Real vs Matrix.
Sorry, no "matrix within a matrix". "Wake up Neo."
Neo is human.
RTFB Neo!
Rewatch The Matrix.
Neo: Yeah.
Neo: Ahhh.
Neo: What?
Neo: Morpheus.
Neo: What?
Neo: Stop.
Neo: The Oracle.
Trinity: No, Neo.
Neo: Yeah.
The Matrix Overloaded
Matrix Within A Matrix?
3. DOUBLE MATRIX theory.
NEO: No.
Regarding a Matrix within a Matrix:
Aren't 2 matrices sufficient?
Neo: "No."
Neo: "No."
Morpheus: "Rest, Neo.
Morpheus: "Rest, Neo. (of the matrix)
Enter the Matrix
Matrix within a Matrix...
Matrix within a Matrix...
Matrix within a Matrix...
That's just begging to be set to music. Or to be included in a anthology of philosophical poetry.
Paged comments would be good too. Using some dynamic solution (like PHP), you could set a threshold for listing everything on one page. Once that is crossed, it would default to showing them in a paged format -- say 20 at a time or so. Navigation links would let folks jump to the end easily enough.
Wouldn't be too hard to put together, but I've never had the need myself...
(However, if a user decided to view all of the thread, more bandwidth would be used, but I suppose you will find that this happens so little of the time as to make it advantageous to spilt data up among pages.)
I don't know how to do this with MovableType, but it seems that to save yourself bandwith--because most users are not going to read all 700 comments when the page loads anyway--you could split up the comments by 50s or 100s, across multiple pages. This would save you bandwidth because for several reasons.
First, like Slashdot, most people aren't going to read the whole thread, so why load them the whole thread?
Secondly (and more tentatively) because a majority of external links--Google searches, trackbacks, etc--are not going to alert their users that the post is a forum, many people are just going to load the page looking for a weblog post on The Matrix: Reloaded, not knowing 700 comments are going to pop-up. These people will be loading MBs of data when they expected to get a little blog post, and most certainly will not read the whole thing.
Why load all this data for people who (a) are probably not expecting it and (b) probably won't read it alll.
I suggest you divide the posts across multiple pages, though providing a single-page or 'printer' view would be nice of you.
It feeds your huge ego...errrrrr... sense of self worth and makes you look cool.
Why give that up?
Anyway, your readers know they are given a privilege to pitch in their views in those instances you open up certain posts for commenting. I'm sure they wouldn't mind you closing the "Reloaded" thread.
"there is always a choice"
Before that PayPal icon shows up!
Because all the comments are organized sequentially, the entire 700-message thread has to be downloaded every time someone wants to read just the most recent posts. That's killing your bandwidth.
You need a threaded discussion board that provides title links to the posts. That way users only read what they want to read instead of having to download the entire discussion each time they want to read the 10 messages posted since yesterday.
It's YOUR site. It's YOUR money that pays the hosting costs. It's YOUR right to close a thread--any thread--without permission.
If it becomes a burden to you--especially a financial burden--then you have the responsibility to YOURSELF to close the thread.
Be selfish. Think of the impact it has on you before thinking of the impact it has on other people. Then, and only then, should you try to judge whether the impact it has on you is worth it to you.
* I do like the Creative Commons, though.
I'm with Greg H. here. It's your site; you have the right to do whatever you want to comments, including deleting, moderating, and closing discussions. Of course if you removed the thread from the site, you would have to deal with the backlash that may cause, so complete removal would be my last choice.
Still, it's quite a nice problem to have, isn't it? Far preferable to the opposite. ;)
also http://ter.air0day.com/index.php?script=matrixreloaded is funny; a handy abridged script, which should handle all questions; no more need for discusson!
Would be a pretty simple Perl script to write something to import the past comments though you might want to check with Steve to make sure thats allowed.
But the thing you should do right now, like Ry suggested, is archive all the old comments to a separate page, so when people check the thread they aren't see the old 2G. Your bandwidth will thank you, your readers will thank you.
also, there is the search engine angle to consider- having all the comments on one page probably enhances the pagerank in google, as more terms are likely to be found.
I think it's amazing that a 700+ comment thread can stay on topic for that long. Very impressive. That said, it seems a fair and good idea to do a summary of the comments, and move the rest of the file to a raw text/PDF file available for download should one really care to read it all.
As for what to do, I would write a script to export all the comments to a series of static HTML pages with appropriate navigation between them, link to the static version from the original post, and set the entry's "allow comments" to "None" (not "Closed"). This will keep all the comments in their original form in your MT database, but won't show them on the entry's page (but that's OK, because they'll be elsewhere in your static pages).
At that point, the link was almost an afterthought; I certainly had no idea it would be viewed so much. But since then, that comments page has been a significant referrer to our (admittedly small) site. I wonder if anyone else has had a similar experience.
I bet people'd buy it.
Weblogs don't need sophisticated commenting systems because posts like these are statistical outliers [how many sigma can dance on the head of a pin, anyhow?]. An "inefficient" commenting system it may be, but it works for the type of comments you want on a Weblog--reflective on the original entry, and not on each other so much.
Now, when someone wants to hack up a way to make [insert bloggish CMS here] and [insert *BB of choice here] speak to each other, well, then, cool. Of course, you're just doing an end-run of what Slashcode was designed to do, but for different reasons [which isn't a bad thing ... code should do what its users need, no more, no less].
In my view, Jason is providing a public service that is his to remove, alter, whatever - at his discretion. I appreciate his willingness to continue hosting it even as it has massively over-scaled expectations and the commenting system it's running on. And without so much as a single greedy banner ad! Thanks, Jason - I, for one, have gotten more good thinking out of this single page than all my other reading about Reloaded anywhere.
The interests of a theoretical "common good" would be to maintain the posts not only as an archive but the ongoing discussion as well; certainly the collective group has hit on a valuable topic and environment to discuss it in. I imagine it's still building steam and that traffic/posts will increase exponentially. The question seems to be: how much effort/expense should he go to in order to keep being our service provider? Is there anything the beneficiaries can do in a medium like the web to disperse the resources amongst them? Paypal for micropayments, anyone? [Salon now charges for a premium subscription for similar content that doesn't compete...]
I'm also concerned with persistence on the web, doing what we can to preserve an unexpected surge of quality content like this, without multiple "site has been moved" or "see the mirror site here" redirects. Quite a quandry...
The thing is, you've already made your choice on what to do. What you now have to understand is why you made that choice.
As for me (who has participated and thank you for the "box to fill with words"), I would give the people a deadline and provide them with links to other web sites whose purpose is Matrix discussions. Then when the deadline is reached, simply remove the submission form.
The comments can remain for prosperity (& google rankings), but Revolutions still has to come out and the thread does not seem like it is going to die by any other means.
Bandwidth issues notwithstanding, it would be great if the discussion could continue until the quality declines (which it may have by now). The PDF version seems like a great long-term solution. Thanks for hosting the thread, if no one has thanked you so far. It's invaluable!
You could publish your movie review in a publication and get paid a writer's fee and let the publication deal with all the responses and letters your commentary generates.
Not easy being provocative/tantalizing/evocative without escaping the weight of the response.
It could be the first signs of your celebrity and the loss of your anonymity. Dare I say you are becoming a BlogStar?
"It could be the first signs of your celebrity and the loss of your anonymity. Dare I say you are becoming a BlogStar?"
Becoming? I think Jason fairly well defines it, because he writes interesting stuff and links to other interesting stuff.
There doesn't seem to be one to paginate comments, though
I don't know if this one (a paginate plug in on the MT Plugin directory site) can do it. I'm sure Brad would know.
Breaking comments up into multiple pages would alleviate a lot of the problems, while letting people still add new posts.
Maybe you'd only want to show the first 20, 50, or whatever, comments on the first page, and then have pages of 50 or so from then on. Most users might never go past the first page, which will save you on bandwidth, and they'll have a better experience since they aren't downloading a 1.3 MB HTML file anymore.
For those ravenously following the discussion they could then bookmark the last page at the time as a way to easily find were they left off.
Can you tell I'm a UI Designer? Maybe we can develop Personas to help come up with the ideal solution for Jason's readership. hehe.
Just my thoughts.
If you can afford the bandwidth hogging, and you want to keep it open, go for it. But the separate page wouldn't be a bad idea, if you can find the hack.
if you're reading this (this who_owns... has already a lot of posts)
thanks for providing us your bandwidth for having the matrix discussions.
As you and a lot of other people have already stated, I believe the discussion forum is of a fairly high level with sometimes excellent and challenging philosophical issues. A lot of people learn a lot from it (they still do on June 25th)They gain a new insight into the film, and may even have some reflections into their own lives (which you have made possible, be proud of it).
I can assure you that the discussions will continue on your site. Hence I would recommend not to close it, if you agree with this, but to have the first 800 comments on a separate file or something that can be downloaded. This might ease you a little bit.
Thanks again J!
This thread is closed to new comments. Thanks to everyone who responded.

