Twitter: Firehose will be not be made public, available only to "small group of trusted partners"

Ever since Twitter disabled access to their firehose last year, many users have been waiting with baited breath for its return.  The “firehose” refers to the stream of all public Twitter posts.  Currently, it’s only possible to get a small subset of all public posts, and many types of Twitter applications aren’t possible without access to the firehose, such as real-time track and trend analysis.

For a time, Twitter planned to allow firehose access through a service called Gnip, but in October stated that they would instead work on providing access themselves.  Since then, details have been sparse on the timeline or methods for which access will be given.

Last week however, some more information has been quietly released by means of an FAQ on the Twitter API website.  Here’s the question and answer in full (emphasis theirs):

When will the firehose be ready?

By late January, early February 2009. For at least Q1 2009, the “firehose” (the near-realtime stream of all public status updates on Twitter) will only be available to a small group of trusted partners. The firehose is a stream HTTP solution; a client connects to it and the stream begins, ceasing only when the client disconnects. Once we’re confident in the stability of the service, we’ll add partners on a case-by-case basis. We may allow a wider selection of clients to consume subsets of the public stream (that is, updates from a collection of user IDs or matching specific search terms). We do not intend to allow anonymous, unregulated public access to this stream for any number of legal, financial, and technical reasons.

There’s a few pieces of new information here.  First, that some kind of beta group will be given firehose access within a few weeks, using HTTP streams.  This sounds similar to the solution provided by Gnip to their users.

Perhaps more important though is the news that a full public stream of the type previously provided will not be returning.  While providing subsets of the public stream could be useful for things like groups, without the full firehose it’s very difficult (if not impossible) to provide a feature like real-time tracking, which has been eagerly awaited.

The three pieces of rationale for not making the firehose public (“legal, financial, and technical”) each bring up additional questions in turn.  Legally, is there a difference between providing public tweets in a full stream and providing tweets publicly by user (which requires knowing the username ahead of time)?  Do the financial motivations refer to saving money on servers and bandwidth, or by making money in providing access for a fee?  And technically, are the existing solutions (such as HTTP streams or XMPP) insufficient for the task, and if so, how?

Highlights from 37signals Live

Earlier this afternoon David Heinemeier Hansson and Jason Fried from 37signals did a live webcast in which they answered questions and talked about some of their upcoming projects.  They spoke about their upcoming book, plans for integrating across all their products, and how they feel about the iPhone and Android. Architectural Changes 37signals has been trying out Amazon’s EC2.  Rather than implementing it across all their products (a big decision to make, and time consuming to implement) they’ve started by trying it out on Tadalist, which is their smallest and simplest product.  This exemplifies one of the methodologies they use: avoiding huge decisions by first trying small implementations. They also did this with regards to translations.  Translating all 37signals products into multiple languages is a scary idea.  So instead, DHH tried translating only Basecamp to Danish, the other language he speaks. New Features and Designs One of the larger developments 37signals has been working on is a “37 ID”.  This is a global namespace across all of their products, in which users won’t have to login multiple times as they switch products.  It also will aid in product integration, which has been a common question among users.  In the past, they didn’t have a good way to determine which users were the same person across their products.  For example, they wouldn’t know if “dhh” on Basecamp was the same as “dhh” on Campfire.  It will also allow them to start selling their products in a suite, whereby customers could sign up for some or all of the products at once, perhaps with a bulk discount.  37 ID will be rolled out in phases, starting early next year. 37signals has also been working on redesigning the marketing of their sites, and has hired a new designer to work on the visual look-and-feel of their sites.  They’ve started with Highrise.  Below are a few screenshots I took of some of the designs they’ve been working with: The last picture is what they’re currently going with, and it should come out within the next week or two. Upcoming Book A few days ago 37signals announced an upcoming book, with the working title “Unconform”.  The announcement was sparse on details, but today they gave some more information on what to expect. Whereas their first book, the popular “Getting Real”, focused on software development and engineering, “Unconform” is more about business: team structure, hiring, competition, and getting the word out. One of the main themes of the book will be “small isn’t a stepping stone”. Companies should consider stopping at a small size, and not all businesses need to be massive to be successful.  DHH also described the book as pushing against the “lifestyle-business” idea, that small businesses like 37signals aren’t in the “real world”, or don’t have a “real business”. Mobile Jason and David got a few questions regarding their thoughts on the mobile environment, specifically the iPhone and Google’s G1 phone.  They stated that they’ve had an internal debate on whether or not to develop “official” iPhone apps, with the consensus being that they should work on making improvements to their API, and allow third parties to develop applications. They referred to the Twitter model of providing the best API they can, and letting developers work on creating clients on various platforms. When asked his opinion about the G1 phone, Jason Fried described it as more of a “me-too” device, not having many truly new features like the iPhone. For those who missed 37signals Live but would like to watch it, you can watch it on

From Gmail to MobileMe

When I purchased an iPhone 3G (upgrading from the first gen), I also purchased the companion MobileMe service, and I’ve been giving it a test run since.  At the time, I was hoping for a replacement to Google’s Gmail, but unfortunately, it’s not quite there yet.  I thought I’d take a few minutes to offer some tips to the MobileMe Email team, as well as start a discussion on email services in general.

[Read more...]

12 Notes on Setting up Gmail IMAP with Apple Mail

Just like Ryan, I’ve been in the process of moving to Gmail to handle all my email, now that they provide IMAP access. In no particular order, here are some notes and thoughts on the process of setting up Gmail IMAP with Apple’s

  1. Rather than restate all the steps I took, I’ll link to this post that I found useful. In particular, configuring Gmail’s Drafts, Junk, Trash and Sent folders to match those in Mail is nice.
  2. The way Gmail implements IMAP wasn’t very intuitive to me and took some getting used to. For example, deleting a message from a mailbox doesn’t really delete it, it just removes that label (or at least that’s how it’s supposed to work; more on that later). I found this page helpful in getting used to how IMAP functions match Gmail actions.
  3. I know this has probably been discussed ad nauseum, but I wish Gmail would loose the “labels” concept and move to “folders”. While I’ve eventually gotten used to it within Gmail, the problem is only made worse with IMAP. For example, if a message has two labels, you’ll see it in two folders, which means it’ll show as two unread messages.
  4. I figured as long as I was setting all this up, I’d use Gmail to backup all my old messages, as described by Ryan. This meant copying over my old mbox files, importing them to Mail, and moving them into the appropriate folders in Gmail. It’s definitely a slow process, but you can view the progress from the Activity window (under Window -> Activity). There were a few times when the moving failed: I believe it was due to cases where messages had very large bodies. Although it failed partway through, I was able to start over without having any duplicate messages, so it looks like Gmail is smart enough not to add an exact duplicate.
  5. To help speed the process a bit, I’d recommend turning off caching while doing any importing or moving of messages. This is done under “Keep messages for offline viewing” in the Advanced tab of the Gmail Account preferences. Once everything is set how you like it, you can turn cacheing back on. For a large number of messages the caching process can also take a while.
  6. It took me a while to find a setup for my outgoing messages that works for me. In the past, I’ve had my email client set to automatically BCC myself on any emails I send. This way my messages get categorized properly, and my emails will be in the same mailbox as the rest of a thread. With Gmail, my sent messages were getting put into the Sent Mail folder in Gmail, which means I never got a chance to filter them into any labels. BCCing myself didn’t help, since Gmail already had a copy of the message in Sent Items. What I’ve done now is to remove Gmail’s Sent folder as Mail’s Sent mailbox, which means the only copy that Gmail receives is the one that gets BCCed, which means I can have it filtered however I like. Update: Turns out I was wrong on this.  Any email sent through Gmail’s SMTP will be added to “Sent Mail”, so BCCing won’t help.
  7. An unfortunate annoyance with Gmail IMAP is that unless you have a message filtered to automatically archive (in Gmail context this means moving it to “All Mail”), it’ll show up twice in Mail: once in the Inbox, and once in “All Mail”. So far I haven’t found a way to avoid this.
  8. The filters in Gmail are rather basic and limited, especially compared to Mail. For example, I don’t think there’s any way to match messages whose subjects start with a string, instead of just containing the string.
  9. Rather than set up a complex set of filters for all the email I get, I got an idea from Ryan to only create filters for less important messages, like automated log messages. Anything important will to go my inbox, and then I can choose how to label it or to delete it. This also has the benefit that on my iPhone, I can just check my Inbox and have important new emails in one place.
  10. Although Google Help says that deleting a message will just remove its label, it won’t necessarily work this way in Mail. Although deleting a message will remove the label (or if it’s in the Inbox, archive it), moving it to the Trash will actually put it in the Trash, which puts it in line for deletion. I think you could prevent this by not having Mail’s Trash mailbox be Gmail’s Trash, but the whole “deleting a message to remove the label” feature isn’t something I really need to use.
  11. I love the new Todos in Mail, but getting them working in Gmail also required some changes. From what I can tell, you can have Mail store the Todos locally or on Gmail (although there doesn’t seem to be a simple Preferences change for this). If it’s on the server, each Todo is stored as an email, and if you want to specify a Calendar for an item, it means defining a new set of Calendars in iCal. If you store the items on the server, each corresponding message shows up in a nice human readable format, but unfortunately on the iPhone all you see is a Mime attachment. Hopefully in the future Apple will provide a simple way to access Todos on the iPhone.
  12. Reading back on these notes, I see how complicated Gmail IMAP can be! I don’t think it’s quite ready for average users yet. That said, it’s nice to know that 1) I now have all my email accounts coming to one place, 2) my old emails are backed up online, and 3) I can access all my email from my computer, my phone, or a browser.

Switching to Gmail

gmailOne of the things I had been meaning to do for some time was to switch all of my email over to my Gmail account. The idea is simple: I have a lot of different email accounts, and it’d be great to keep them all in one place, backed up, and always accessible. So, setting up Gmail to access each of my different email accounts (well, five of the most important, and the rest just forward to my gmail address) was trivial. The hard part, however, was getting all my previous messages into Gmail.

Here’s a quick overview of how to get all of your old emails into Gmail as painlessly as possible (and one way that preserves dates!).

[Read more...]

Updating Rubygems in Leopard (Mac OS 10.5)

After updating my Mac to Leopard (MacOS X 10.5), I noticed I had a couple issues when updating Rubygems. For instance, when running “gem update mongrel_cluster”, I would constantly get build errors. After doing some digging, I found that you need to set ARCHFLAGS to your system type. Here’s how you would update all your rubygems on Leopard:

sudo su
[Enter your password]
export ARCHFLAGS="-arch i386"
gem update

Note that you’ll want to change the i386 to the actual architecture you have (my MacBook, for instance, is an Intel processor, while you would want to use “ppc” for non-Intel Macs).

Once I had updated my gems, things started working fine again. For instance, mongrel_cluster was having issues configuring a new project, but after updating the gem using this method, it seemed to be working fine. Hope you find this useful!

Java through the ages

I wanted to point out a great article on ReadWriteWeb today, by Alex Iskold, about the history of Java and the missed opportunities the language has had over the years.  Even if you’re not a software developer, it’s still a great overview of a language that is both elegant and powerful, and the drama of free market competition.

Of course, don’t get the impression that Java is dead!  It’s anything but: kiosks and embedded devices run it, along with consumer-facing websites and plenty of enterprise software.  Many other technologies have been eating away at it over the years, and will continue to do so, unfortunately, for many of the reasons in the article.

Like Alex, I think the biggest issue Java faced (and got wrong) was with the web.  Before there was web 2.0 (and web 1.0 was still being explored), Java applets were the way to integrate interactivity inside web pages.  But applets were horribly slow, limited in their abilities, and complicated.  I think if Sun were able to do it all over again, they shouldn’t have focused on placing Java applets inside web pages, but focused more closely on making Java the core of web development.  Like Alex states, imagine how great it would be if we could manipulate the DOM using Java, say if it were an integrated part of the browser.

But, alas, that didn’t happen.  There’s still time to reinvent how Java and the web co-exist, but for the time being, it’s being edged out in favor of new technologies and languages.  So here’s to you, Java!

The future of AJAX web applications

There’s an interesting opinion up by Joel Spolsky, a software developer and founder of FogCreek Software, about where the direction AJAX-based web applications are headed. He makes an interesting, and I feel very apt, comparison with the olden days of mainframes and Lotus 1-2-3, and the current state of the interactive web. For instance, he likens the idea of sites like Google’s Gmail with Lotus 1-2-3, where the development team spent all of their time writing code and optimizing it for the current day’s limitations, rather than looking ahead and adding new wiz-bang features that would give them their “long-term competitive advantage.”

And I think Joel is completely right. Gmail, for one, has been stagnant for the last three years or so, and haven’t been preparing for the future. Check out this blog article from Lifehacker about a comparison between Gmail and Yahoo Mail. Their conclusion? Yahoo Mail has spent the last two years innovating and adding all sorts of new features, while Gmail has very little improvements (except, perhaps, incrementally increased storage levels).

[Read more...]

Apple iPhones reduced in price

iPhoneYesterday, Apple dropped the price on the 8gig iPhone (that killer, must-have gadget that’s apparently been selling like crazy since being introduced), along with new product announcements in their iPod lineup. Unfortunately, it looks like this price break puts a lot of us early adopters in a tight spot: those of us who shelled out the full $600 for the 8gig models are now realizing the price of purchasing early: about $200.

If you’re not already aware, if you bought your iPhone within 10 days of an announced price break, you’re entitled to receive the difference from Apple (provided you claim this within another 14 days of the announcement).

Should Apple reduce its price on any shipped product within 10 calendar days of shipment, you may contact Apple Sales Support at 1-800-676-2775 to request a refund or credit of the difference between the price you were charged and the current selling price. To receive the refund or credit you must contact Apple within 14 business days of shipment.

Sadly, we at Draconis bought out iPhones 16 days before the announcement: 2 days later and we would have qualified for that rebate. But I’m not bitter over it: I love my iPhone, was willing to part with the full price without expecting any kind of rebate, and anyway, these things are out of our control. Anyone else in the same boat as us?

The invotrak blog

The invotrak blogJust a quick note to let you know about the new blog, launching today. If you’re not already aware, we launched a new online invoice-tracking service recently, which helps freelancers and small-businesses keep track of the invoices they send to clients. The service is free, and we’ve seen quite a few people join up.

The blog is to keep much of the invotrak-related content in one place (without cluttering up this blog). It’ll have invotrak-related news (new features, notices, etc), as well as original content we think is of interest to users. If you’re interested in freelancing/entrepreneurship/etc, you may find the invotrak blog interesting.