Who owns the conversation on my web site?

Who owns the conversation on my web site? JUN 18 2003

Back on May 15th, I wrote a 221-word entry on my first impression of The Matrix Reloaded. At last count (mid-afternoonish on June 17th), people have left 700 comments in the thread attached to that entry (7 of which are mine). Those 700 comments comprise a total of ~125,000 words (~180 per entry); that's about 3.3 150-page books. The HTML file is in excess of 1.3 MB in size, has been viewed about 5000 times in the past two weeks (during which about half of the comments were left), and is responsible, all by itself, for 5.3 gigabytes of data transferred from kottke.org this month.

In comparison to the rest of kottke.org (just a few more stats here), the most comments on previous thread was just over 200 (a Crouching Tiger, Hidden Dragon thread that is no longer online), there are around 3500 comments that have been left on the site in dozens of threads, and over the past 5 years, I have posted around 2500 entries to kottke.org, 27 of them since the Matrix post. My total bandwidth usage for June thus far is about 14.6 GB.

In short, the Matrix thread is unexpectedly large, and depending on how you look at it, is anywhere from 10% to 36% of my entire site, and significantly more than 50% of the total output since May 15th. If you were to look at all the content of the site in the aggregate, you might come to the conclusion that kottke.org is a Matrix Reloaded-related site even though it's been chugging along Reloaded-free for more than 5 years.

Given that the thread is still active (25 posts today alone), how do I deal with it going forward? I haven't read any of the conversation since around comment #200, and if it hits 1000 comments (which, 2 weeks ago seemed unlikely but now seems inevitable), the file approaches 2 MB in size, and sucks up 15 GB of bandwidth a month, it seems prudent for me to close the thread.

On the other hand, it continues to be valuable as a forum for a small number of participants and a larger number of readers...and from what I can tell, is a relatively high-quality conversation. Who am I to shut down a conversation that I'm not involved in? This may be my site, but the participants own the conversation. As much as it makes sense to shut it down, I'm inclined to let the participants go as long as they want. Who knows, maybe they'll even let me know when they're done.

Related: Sam Ruby's experiment of moderating discussions on weblogs.

Oh, and just for fun, I pasted the entire thread into Word and used the AutoSummary feature to come up with this 100 word summary of the thread:

The Matrix 1.1?

Wake up Neo...
Real vs Matrix.
Sorry, no "matrix within a matrix". "Wake up Neo."

Neo is human.

RTFB Neo!

Rewatch The Matrix.
Neo: Yeah.
Neo: Ahhh.
Neo: What?
Neo: Morpheus.
Neo: What?
Neo: Stop.
Neo: The Oracle.
Trinity: No, Neo.
Neo: Yeah.

The Matrix Overloaded

Matrix Within A Matrix?

3. DOUBLE MATRIX theory.
NEO: No.

Regarding a Matrix within a Matrix:
Aren't 2 matrices sufficient?
Neo: "No."
Neo: "No."

Morpheus: "Rest, Neo.
Morpheus: "Rest, Neo. (of the matrix)

Enter the Matrix

Matrix within a Matrix...
Matrix within a Matrix...
Matrix within a Matrix...

That's just begging to be set to music. Or to be included in a anthology of philosophical poetry.

There are 50 reader comments

Matthew Aaron14 18 200312:14AM

Let's just hope this thread doesn't monstropolize as well. I can see the discussion ownership battle now...

MCRD15 18 200312:15AM

Oh no, I can see it now: There will soon be 2MB of comments on comments about the Matrix. Is this life imitating art? A sort of, Matrix within the Matrix? When will it all end?

Brad Choate28 18 200312:28AM

Installing mod_gzip would help.

Paged comments would be good too. Using some dynamic solution (like PHP), you could set a threshold for listing everything on one page. Once that is crossed, it would default to showing them in a paged format -- say 20 at a time or so. Navigation links would let folks jump to the end easily enough.

Wouldn't be too hard to put together, but I've never had the need myself...

Ry Rivard30 18 200312:30AM

I read a while back that one of Slashdot's tricks for high-tides of sever use is to decrease the number of comments on a default page (i e, people who aren't signed in with preferences set), because few people click onto the next page of comments anyway. This decreases the bandwidth load twofold ways: less text data is transmitted both on the loading of the main page and on loading of subsequent pages decreases inversely to increase it pages.

(However, if a user decided to view all of the thread, more bandwidth would be used, but I suppose you will find that this happens so little of the time as to make it advantageous to spilt data up among pages.)

I don't know how to do this with MovableType, but it seems that to save yourself bandwith--because most users are not going to read all 700 comments when the page loads anyway--you could split up the comments by 50s or 100s, across multiple pages. This would save you bandwidth because for several reasons.

First, like Slashdot, most people aren't going to read the whole thread, so why load them the whole thread?

Secondly (and more tentatively) because a majority of external links--Google searches, trackbacks, etc--are not going to alert their users that the post is a forum, many people are just going to load the page looking for a weblog post on The Matrix: Reloaded, not knowing 700 comments are going to pop-up. These people will be loading MBs of data when they expected to get a little blog post, and most certainly will not read the whole thing.

Why load all this data for people who (a) are probably not expecting it and (b) probably won't read it alll.

I suggest you divide the posts across multiple pages, though providing a single-page or 'printer' view would be nice of you.

dayment56 18 200312:56AM

this is just too much math.....

zerodoug14 18 2003 1:14AM

jason - the blogosphere has you...

zerodoug19 18 2003 1:19AM

;-)

anthony langford14 18 2003 2:14AM

the convesation pieces (Neo: Yeah. Neo: Ahhh.) are soo all you base.

spencer39 18 2003 2:39AM

that was a wonderful post. (the one about the post about the matrix reloaded, not the post about the matrix reloaded, althought that one might be cool, too.)

soitgoes05 18 2003 3:05AM

I think you should leave the matrix post up at all costs. That way you can always fall back on a "the post is still going with over 1,000,000 posts" post when things are slow.

It feeds your huge ego...errrrrr... sense of self worth and makes you look cool.

Why give that up?

Kingsley59 18 2003 3:59AM

All your comments are belong to us!

Willie Galang25 18 2003 4:25AM

Why not close the thread? It is already a valuable resource of "The Matrix: Reloaded" talk as it stands right now. I'm sure your site will go through a similar phase once "The Matrix: Revolutions" is released. Assuming you start a "Revolutions" thread, of course.

Anyway, your readers know they are given a privilege to pitch in their views in those instances you open up certain posts for commenting. I'm sure they wouldn't mind you closing the "Reloaded" thread.

angelday22 18 2003 6:22AM

I think Jason should install a PHPBB (http://www.phpbb.com/) or an Invision Power Board (http://www.invisionboard.com/) to handle vast amount of posts and conserve bandwidth. Seriously.

"there is always a choice"

Before that PayPal icon shows up!

jeremy_tai58 18 2003 6:58AM

Kottke reaches record setting comments on the Matrix Reloaded post just as The Matrix Reloaded has reached record setting numbers in the box office. Life imitating art.

Greg38 18 2003 7:38AM

Jason, you're experience the inefficiency of your commenting system. Movable Type's comment function is poorly designed to handle the level of discussion this thread generated.

Because all the comments are organized sequentially, the entire 700-message thread has to be downloaded every time someone wants to read just the most recent posts. That's killing your bandwidth.

You need a threaded discussion board that provides title links to the posts. That way users only read what they want to read instead of having to download the entire discussion each time they want to read the 10 messages posted since yesterday.

Paul27 18 2003 8:27AM

I think a threaded discussion board is overkill for a weblog entry. This one has gotten too big for its briches; take it out back and shoot it.

Greg H.45 18 2003 8:45AM

Has everyone in the blogosphere gone mad? People seem to be hesitant to claim ownership of anything, including their own content (Creative Commons*) and their own money that pays for bandwidth.

It's YOUR site. It's YOUR money that pays the hosting costs. It's YOUR right to close a thread--any thread--without permission.

If it becomes a burden to you--especially a financial burden--then you have the responsibility to YOURSELF to close the thread.

Be selfish. Think of the impact it has on you before thinking of the impact it has on other people. Then, and only then, should you try to judge whether the impact it has on you is worth it to you.

* I do like the Creative Commons, though.

Joshua Kaufman26 18 2003 9:26AM

This may be my site, but the participants own the conversation.

I'm with Greg H. here. It's your site; you have the right to do whatever you want to comments, including deleting, moderating, and closing discussions. Of course if you removed the thread from the site, you would have to deal with the backlash that may cause, so complete removal would be my last choice.

Mike26 18 2003 9:26AM

If you don't want to close the thread then modify the comment template to display the first 20 or 50 comments with a link to see the rest of them. Half the time people will read the first few and not bother with the rest, thus saving bandwidth.

Suw32 18 200310:32AM

I would agree with the suggestion of archiving off the older comments to a separate page so that they're still accessible, but not getting in the way. I'm sure that there'll still be a fair few people who want to read them in the future, if only because this whole Matrix is gonna run for a while yet.

Still, it's quite a nice problem to have, isn't it? Far preferable to the opposite. ;)

DiRK44 18 200310:44AM

"I suggest you divide the posts across multiple pages, though providing a single-page or 'printer' view would be nice of you." yeah that's what osnews does! +Ry Rivard :D

also http://ter.air0day.com/index.php?script=matrixreloaded is funny; a handy abridged script, which should handle all questions; no more need for discusson!

kellan02 18 200311:02AM

Point the discussion at a Quicktopic discussion board ala BoingBoing.

Would be a pretty simple Perl script to write something to import the past comments though you might want to check with Steve to make sure thats allowed.

But the thing you should do right now, like Ry suggested, is archive all the old comments to a separate page, so when people check the thread they aren't see the old 2G. Your bandwidth will thank you, your readers will thank you.

Brent Gustafson09 18 200311:09AM

So that's where all the bandwidth for our server is going. No wonder my site's been slow lately. :\

monkeyinabox40 18 200311:40AM

Being the last human on earth who hasn't seen the movie yet, I also have put off reading the comments in the original thread. I was interested in going back and reading it after I saw the movie, that was until I read this thread today. 3 - 150 page books of comments! That's madness!

jake10 18 200312:10PM

yeah- if anyone has a quick hack for modifying the MT entry page template to produce multipage comments, I'd definitely like to take a look at it. the thing is, the ideal solution would be smart enough to only invoke the multipage commenting if the comment number went above like 50- for less than that, i'd rather have all the comments on one page.

also, there is the search engine angle to consider- having all the comments on one page probably enhances the pagerank in google, as more terms are likely to be found.

Andrew16 18 200312:16PM

(Of course, by pointing out the post and thread, you draw more traffic to it, increasing your bandwidth issues...)

I think it's amazing that a 700+ comment thread can stay on topic for that long. Very impressive. That said, it seems a fair and good idea to do a summary of the comments, and move the rest of the file to a raw text/PDF file available for download should one really care to read it all.

Mark07 18 2003 1:07PM

I'd just like to be the first person to mention "power law distributions" here.

As for what to do, I would write a script to export all the comments to a series of static HTML pages with appropriate navigation between them, link to the static version from the original post, and set the entry's "allow comments" to "None" (not "Closed"). This will keep all the comments in their original form in your MT database, but won't show them on the entry's page (but that's OK, because they'll be elsewhere in your static pages).

Ryan27 18 2003 1:27PM

Beyond being responsible for a huge portion of your site content at this point, I wonder how powerful of a traffic driver that comments thread has been. Before anyone had any idea that the Reloaded thread would take over the world, I put in a comment (just went back and counted; it was #35). Someone had mentioned the Animatrix, which I had just watched, so I wanted to pitch in a little info about the way it fed into Reloaded. I also provided a link to a review I'd done.

At that point, the link was almost an afterthought; I certainly had no idea it would be viewed so much. But since then, that comments page has been a significant referrer to our (admittedly small) site. I wonder if anyone else has had a similar experience.

April39 18 2003 1:39PM

I feel in part responsible. I work with SpoonBoy and pointed the thread out to him because I knew he was really into The Matrix. I had no idea that he would contribute so much to it, but I'm glad he did.

boogah23 18 2003 3:23PM

it might seem a bit cheesy to do but i'd suggest closing the thread and pointing any people itching to discuss the movie further to either an offsite discussion forum geared towards matrix fans or a quicktopic. just an idea tho.

Brian26 18 2003 3:26PM

Maybe it'd be cool to (tediously) go through the comments and cut out the ones that aren't really in the flow of the conversation and actually publish a book of blog comments about the Matrix.

I bet people'd buy it.

Geof F. Morris44 18 2003 4:44PM

The thought occurs: close the thread, pull it off your site, name it something, and stick it on Kazaa or whatever file-sharing network you prefer. The commons still have access to the data, but they're not going on your server [and spending your money!] to get it.

Weblogs don't need sophisticated commenting systems because posts like these are statistical outliers [how many sigma can dance on the head of a pin, anyhow?]. An "inefficient" commenting system it may be, but it works for the type of comments you want on a Weblog--reflective on the original entry, and not on each other so much.

Now, when someone wants to hack up a way to make [insert bloggish CMS here] and [insert *BB of choice here] speak to each other, well, then, cool. Of course, you're just doing an end-run of what Slashcode was designed to do, but for different reasons [which isn't a bad thing ... code should do what its users need, no more, no less].

scott13 18 2003 5:13PM

The Architect speaks!

In my view, Jason is providing a public service that is his to remove, alter, whatever - at his discretion. I appreciate his willingness to continue hosting it even as it has massively over-scaled expectations and the commenting system it's running on. And without so much as a single greedy banner ad! Thanks, Jason - I, for one, have gotten more good thinking out of this single page than all my other reading about Reloaded anywhere.

The interests of a theoretical "common good" would be to maintain the posts not only as an archive but the ongoing discussion as well; certainly the collective group has hit on a valuable topic and environment to discuss it in. I imagine it's still building steam and that traffic/posts will increase exponentially. The question seems to be: how much effort/expense should he go to in order to keep being our service provider? Is there anything the beneficiaries can do in a medium like the web to disperse the resources amongst them? Paypal for micropayments, anyone? [Salon now charges for a premium subscription for similar content that doesn't compete...]

I'm also concerned with persistence on the web, doing what we can to preserve an unexpected surge of quality content like this, without multiple "site has been moved" or "see the mirror site here" redirects. Quite a quandry...

Joe Kaczmarek58 18 2003 6:58PM

So is this meta-commenting (commenting on a comments thread)?

The thing is, you've already made your choice on what to do. What you now have to understand is why you made that choice.

As for me (who has participated and thank you for the "box to fill with words"), I would give the people a deadline and provide them with links to other web sites whose purpose is Matrix discussions. Then when the deadline is reached, simply remove the submission form.

The comments can remain for prosperity (& google rankings), but Revolutions still has to come out and the thread does not seem like it is going to die by any other means.

Sean Tevis49 18 200310:49PM

I last read all the comments when it was around 400 posts - it took me the better part of a Saturday afternoon with the added time of clicking through the referenced URLs. Reading 700 comments would take me about 6-7 hours to wade through properly. The 'Reloaded' thread is one of the smartest, most thoughtful works on the movie as a work of literature that I have seen. I gather from this that kottke.org has an unusually intelligent readership.

Bandwidth issues notwithstanding, it would be great if the discussion could continue until the quality declines (which it may have by now). The PDF version seems like a great long-term solution. Thanks for hosting the thread, if no one has thanked you so far. It's invaluable!

SzaffireBlue28 19 200312:28AM

Hmm. Since I am not for cencorship in any way shape or form I would never discourage you from ceasing to write about huge popular cultural phenomenons ( ex: blockbuster films with alrge cult followings.) If you are squeamish about being proprietor then perhaps you could be a squatter of sorts and post the brilliant contents of your mind on another person's/corp. site and let them deal with bandwith/pulling the plug issues.

You could publish your movie review in a publication and get paid a writer's fee and let the publication deal with all the responses and letters your commentary generates.

Not easy being provocative/tantalizing/evocative without escaping the weight of the response.

It could be the first signs of your celebrity and the loss of your anonymity. Dare I say you are becoming a BlogStar?

Dean Allen10 19 2003 4:10AM

What's "The Matrix?"

Jeff Dowder36 19 2003 6:36AM

The personal computer is the anti-christ

Geof F. Morris16 19 200311:16AM

SzaffireBlue wrote:

"It could be the first signs of your celebrity and the loss of your anonymity. Dare I say you are becoming a BlogStar?"

Becoming? I think Jason fairly well defines it, because he writes interesting stuff and links to other interesting stuff.

Scarabic04 19 2003 2:04PM

I found the thread to be the most intelligent conversation on the subject that the internet had to offer. MeFites eschew the topic for the most part, and fanboy morons pollute just about every other forum around. The fanboy moron factor actually started to bug me here around comment 200. I think you'd be fine to close it. I don't think there's that much more to say. Perhaps re-packaging the existing comments would be beneficial for interested surfers. And with some pagination, you could cut way down on your bandwidth outgoing. As is, if you're going to incur a 1.3MB hit every time someone lands, you're going to go broke fast.

Jamison21 19 2003 6:21PM

Setting technical issues aside for the moment, pagination would seem to be the best solution.

There doesn't seem to be one to paginate comments, though
I don't know if this one (a paginate plug in on the MT Plugin directory site) can do it. I'm sure Brad would know.

Breaking comments up into multiple pages would alleviate a lot of the problems, while letting people still add new posts.

Maybe you'd only want to show the first 20, 50, or whatever, comments on the first page, and then have pages of 50 or so from then on. Most users might never go past the first page, which will save you on bandwidth, and they'll have a better experience since they aren't downloading a 1.3 MB HTML file anymore.

For those ravenously following the discussion they could then bookmark the last page at the time as a way to easily find were they left off.

Can you tell I'm a UI Designer? Maybe we can develop Personas to help come up with the ideal solution for Jason's readership. hehe.

nosy52 19 2003 8:52PM

Sell it to the owners of the official Matrix site and use the money to buy and distribute movie tickets for other movies to all the people who have been posting there. That might enable them to engage in conversations/chats/comments systems pertaing to other subjects, which could result in fairer distribution of bytes.

Adam Wood09 19 200311:09PM

This is why I don't blog.

Kutastha25 20 2003 3:25AM

I agree with those who beleive Jason can do whatever he wishes to do with this thread. If that means closing it down, fine. Perhaps, if he were to close it down, it would be kind of him to place that thread into a Word document or something for access to be downloaded by anyone who would like it. This way, only those who really know what it is will want to download it. There will be no accident of downloading a 2 MB page from the Internet.

Just my thoughts.

dowingba29 22 200310:29PM

Anyone who wants the thread going should donate money.

jocbrut07 23 2003 5:07AM

Installing mod_gzip is the best solution

Kip Ingram19 23 200312:19PM

It's your site and the decision has to be yours. It's nice that you ask for input, but it's your call in the end. The conversation could be spun off into a mailing list or an alt usenet group.

phoenix18 24 2003 4:18PM

that would make a kick-ass techno song...

cyberangel29 24 2003 4:29PM

Incredible, Jason. Wish I had time to read the whole thing!

If you can afford the bandwidth hogging, and you want to keep it open, go for it. But the separate page wouldn't be a bad idea, if you can find the hack.

Siddarta Gouthama37 25 2003 5:37AM

Jason,
if you're reading this (this who_owns... has already a lot of posts)
thanks for providing us your bandwidth for having the matrix discussions.
As you and a lot of other people have already stated, I believe the discussion forum is of a fairly high level with sometimes excellent and challenging philosophical issues. A lot of people learn a lot from it (they still do on June 25th)They gain a new insight into the film, and may even have some reflections into their own lives (which you have made possible, be proud of it).
I can assure you that the discussions will continue on your site. Hence I would recommend not to close it, if you agree with this, but to have the first 800 comments on a separate file or something that can be downloaded. This might ease you a little bit.
Thanks again J!

This thread is closed to new comments. Thanks to everyone who responded.