Ever since Twitter disabled access to their firehose last year, many users have been waiting with baited breath for its return.  The “firehose” refers to the stream of all public Twitter posts.  Currently, it’s only possible to get a small subset of all public posts, and many types of Twitter applications aren’t possible without access to the firehose, such as real-time track and trend analysis.

For a time, Twitter planned to allow firehose access through a service called Gnip, but in October stated that they would instead work on providing access themselves.  Since then, details have been sparse on the timeline or methods for which access will be given.

Last week however, some more information has been quietly released by means of an FAQ on the Twitter API website.  Here’s the question and answer in full (emphasis theirs):

When will the firehose be ready?

By late January, early February 2009. For at least Q1 2009, the “firehose” (the near-realtime stream of all public status updates on Twitter) will only be available to a small group of trusted partners. The firehose is a stream HTTP solution; a client connects to it and the stream begins, ceasing only when the client disconnects. Once we’re confident in the stability of the service, we’ll add partners on a case-by-case basis. We may allow a wider selection of clients to consume subsets of the public stream (that is, updates from a collection of user IDs or matching specific search terms). We do not intend to allow anonymous, unregulated public access to this stream for any number of legal, financial, and technical reasons.

There’s a few pieces of new information here.  First, that some kind of beta group will be given firehose access within a few weeks, using HTTP streams.  This sounds similar to the solution provided by Gnip to their users.

Perhaps more important though is the news that a full public stream of the type previously provided will not be returning.  While providing subsets of the public stream could be useful for things like groups, without the full firehose it’s very difficult (if not impossible) to provide a feature like real-time tracking, which has been eagerly awaited.

The three pieces of rationale for not making the firehose public (“legal, financial, and technical”) each bring up additional questions in turn.  Legally, is there a difference between providing public tweets in a full stream and providing tweets publicly by user (which requires knowing the username ahead of time)?  Do the financial motivations refer to saving money on servers and bandwidth, or by making money in providing access for a fee?  And technically, are the existing solutions (such as HTTP streams or XMPP) insufficient for the task, and if so, how?

Earlier this afternoon David Heinemeier Hansson and Jason Fried from 37signals did a live webcast in which they answered questions and talked about some of their upcoming projects.  They spoke about their upcoming book, plans for integrating across all their products, and how they feel about the iPhone and Android. Architectural Changes 37signals has been trying out Amazon’s EC2.  Rather than implementing it across all their products (a big decision to make, and time consuming to implement) they’ve started by trying it out on Tadalist, which is their smallest and simplest product.  This exemplifies one of the methodologies they use: avoiding huge decisions by first trying small implementations. They also did this with regards to translations.  Translating all 37signals products into multiple languages is a scary idea.  So instead, DHH tried translating only Basecamp to Danish, the other language he speaks. New Features and Designs One of the larger developments 37signals has been working on is a “37 ID”.  This is a global namespace across all of their products, in which users won’t have to login multiple times as they switch products.  It also will aid in product integration, which has been a common question among users.  In the past, they didn’t have a good way to determine which users were the same person across their products.  For example, they wouldn’t know if “dhh” on Basecamp was the same as “dhh” on Campfire.  It will also allow them to start selling their products in a suite, whereby customers could sign up for some or all of the products at once, perhaps with a bulk discount.  37 ID will be rolled out in phases, starting early next year. 37signals has also been working on redesigning the marketing of their sites, and has hired a new designer to work on the visual look-and-feel of their sites.  They’ve started with Highrise.  Below are a few screenshots I took of some of the designs they’ve been working with: The last picture is what they’re currently going with, and it should come out within the next week or two. Upcoming Book A few days ago 37signals announced an upcoming book, with the working title “Unconform”.  The announcement was sparse on details, but today they gave some more information on what to expect. Whereas their first book, the popular “Getting Real”, focused on software development and engineering, “Unconform” is more about business: team structure, hiring, competition, and getting the word out. One of the main themes of the book will be “small isn’t a stepping stone”. Companies should consider stopping at a small size, and not all businesses need to be massive to be successful.  DHH also described the book as pushing against the “lifestyle-business” idea, that small businesses like 37signals aren’t in the “real world”, or don’t have a “real business”. Mobile Jason and David got a few questions regarding their thoughts on the mobile environment, specifically the iPhone and Google’s G1 phone.  They stated that they’ve had an internal debate on whether or not to develop “official” iPhone apps, with the consensus being that they should work on making improvements to their API, and allow third parties to develop applications. They referred to the Twitter model of providing the best API they can, and letting developers work on creating clients on various platforms. When asked his opinion about the G1 phone, Jason Fried described it as more of a “me-too” device, not having many truly new features like the iPhone. For those who missed 37signals Live but would like to watch it, you can watch it on Justin.tv.