For a time, Twitter planned to allow firehose access through a service called Gnip, but in October stated that they would instead work on providing access themselves.  Since then, details have been sparse on the timeline or methods for which access will be given.

Last week however, some more information has been quietly released by means of an FAQ on the Twitter API website.  Here’s the question and answer in full (emphasis theirs):

When will the firehose be ready?

By late January, early February 2009. For at least Q1 2009, the “firehose” (the near-realtime stream of all public status updates on Twitter) will only be available to a small group of trusted partners. The firehose is a stream HTTP solution; a client connects to it and the stream begins, ceasing only when the client disconnects. Once we’re confident in the stability of the service, we’ll add partners on a case-by-case basis. We may allow a wider selection of clients to consume subsets of the public stream (that is, updates from a collection of user IDs or matching specific search terms). We do not intend to allow anonymous, unregulated public access to this stream for any number of legal, financial, and technical reasons.

There’s a few pieces of new information here.  First, that some kind of beta group will be given firehose access within a few weeks, using HTTP streams.  This sounds similar to the solution provided by Gnip to their users.

Perhaps more important though is the news that a full public stream of the type previously provided will not be returning.  While providing subsets of the public stream could be useful for things like groups, without the full firehose it’s very difficult (if not impossible) to provide a feature like real-time tracking, which has been eagerly awaited.

The three pieces of rationale for not making the firehose public (“legal, financial, and technical”) each bring up additional questions in turn.  Legally, is there a difference between providing public tweets in a full stream and providing tweets publicly by user (which requires knowing the username ahead of time)?  Do the financial motivations refer to saving money on servers and bandwidth, or by making money in providing access for a fee?  And technically, are the existing solutions (such as HTTP streams or XMPP) insufficient for the task, and if so, how?

If you’re not already aware, if you bought your iPhone within 10 days of an announced price break, you’re entitled to receive the difference from Apple (provided you claim this within another 14 days of the announcement).

Should Apple reduce its price on any shipped product within 10 calendar days of shipment, you may contact Apple Sales Support at 1-800-676-2775 to request a refund or credit of the difference between the price you were charged and the current selling price. To receive the refund or credit you must contact Apple within 14 business days of shipment.

Sadly, we at Draconis bought out iPhones 16 days before the announcement: 2 days later and we would have qualified for that rebate. But I’m not bitter over it: I love my iPhone, was willing to part with the full price without expecting any kind of rebate, and anyway, these things are out of our control. Anyone else in the same boat as us?

In addition, there’s a new wireless stack, a new Firewire stack (very cool), and a slew of new drivers and other changes. Check out details about this release here, then download the kernel. Enjoy!

One of the big new features is the introduction of the LED back-lit screen:

LED back-lighting is touted to provide a more evenly lit screen with sharper images and colors without sacrificing battery life. All these I find to be true, the screen is without a doubt the best i’ve ever seen on a laptop, and better than a lot of desktop monitors I use. With the brightness up to full, even in the most well lit rooms, solid whites are almost blinding, which allows you to turn down the brightness and use less battery.

I was originally a Mac guy, switched to Linux as my desktop du jour, and then switched back with the MacBook, and I’m very glad I did.

DKIM is a simple means for verifying the origination of an email in an attempt to better track (and fight) spam and phishing messages. The method is simple: the sender encrypts the message body using it’s private key and stores this in the message header (non DKIM receivers, then, can safely ignore it and still deliver the message). A DKIM-enabled receiver looks up the originating domain’s record and extracts the public key. From Wikipedia: “The receiver can then decrypt the hash value in the header field and at the same time recalculate the hash value for the mail body that was received, from the point immediately following the “DomainKey-Signature:” header. If the two values match, this cryptographically proves that the mail did in fact originate at the purported domain, and has not been tampered with in transit.”

Using such a technique makes it nearly impossible to create forged emails like the kind hitting our inboxes purporting to be from PayPal, eBay, and others. From CNET: “In the long run, DomainKeys is more promising than existing antispam and antiphishing technologies, which rely on techniques like assembling a “blacklist” of known fraudsters or detecting such messages by trying to identify common characteristics.”

Microsoft has also put together a method for guaranteeing identity of email messages, called Sender ID, though it’s acceptance has been very slow (“because Microsoft initially did not agree to license patents in ways that are compatible with GNU General Public License”).

I think the DKIM technique towards spam/phishing message blocking is one of the most interesting and promising methods to come around. I’ve used Bayesian filters and blacklists, and spammers always find a way around it. But if we can get mail servers to all use this technique, I could easily see the eradication of nearly all phishing messages, and much-reduced spam messages (since it would be possible to track the sending habits of domains better).

What’s your take? In order for DKIM to be useful, both the sender and receiver must implement the DKIM method, which may mean upgrading your mail server software. One of the biggest downsides to this, unfortunately, is the extra overhead involved in email: each message requires cryptographic signing and a domain lookup, which can be quite taxing for already overworked email servers. Will you be using DKIM?

Additional Reading
Wikipedia: DomainKeys
CNET News: Promising antispam technique gets nod
DomainKeys Identified Mail Homepage
DKIM Signatures IETF Standards Draft

I see the whole certification process as having two main flaws (as least on the part of companies certifying their own products): (1) little pressure on the part of the certifying company to make the tests difficult or otherwise accurately prove a taker’s skills, and (2) lots of pressure on the part of the company to test the applicant’s knowledge of vendor-specific aspects. It seems to me that it’s in the interest of the certifying company to have lots of certified engineers out there who know the ins-and-outs of that company’s products and little about any competing products.

So, as someone who needs to hire a competent sysadmin, how does this help me? A potential sysadmin who’s certified as an MCSE or RHCE or whatever shows they can take a test and know the basics of that particular company’s product, but what about the millions of other things that sysadmin would be responsible for? Really, it seems these certifications are good for large companies interested in a sysadmin to manage many exact-same boxes and little else. For the majority of employers, especially startups and growing companies, I’d think someone who is more well-rounded in things they’d need to manage is much more useful (for instance, can a sysadmin fill in for a network admin should the need arise? do they understand infrastructure needs to make recommendations for expansion? etc). The best skill a sysadmin can have is the ability to learn as they go and adapt to changes. Having a certification in a particular OS doesn’t particularly help if its a heterogeneous network.

I think certification is a useful process but I’d like to see programs that are more comprehensive, easier to afford, and focus on general skills regardless of environment. What do you think? Have you gone through a certification process and, if so, how has it helped you? In the past, I’ve thought about getting certified myself, but never went through with it primarily for these reasons (as well as the cost).

Check it out. Though we don’t use RHEL on our production machines, a number of large IT departments do. Be sure to post your reactions here.

Every so often, I like to head out to a Starbucks (which costs a couple bucks for the TMobile service they provide, plus a Venti cup of whatever’s brewing) or a FreshCity (where it’s free), take my MacBook, and get some work done. I find it’s often good motivation when I pick up and move to some other place – different environs give me a nice motivational push. What’s your take? Do you think offering WiFi for free at Starbucks, Panera, FreshCity, McDonald’s, and other places would be harming or helpful?

Don’t Miss:
What Starbucks Can Learn From the Movie Palace
WiFi is the New Air Conditioning
Wi-Fi should be free
Wi-Fi: If Not Free, Then How?

In many ways, it still has the head of an graduate-school project grafted onto the body of an multinational corporation. The central tenet of its strategy is that its growing cadre of world-class computer scientists can design a network of machines that can store and process more information more efficiently than anyone else.

Mr. Reynolds estimated that Google’s computing costs are half those of other large Internet companies and a tenth those of traditional corporate technology users.

Interesting, but before you start thinking of building your own servers and file systems, remember that Google can benefit greatly from economies of scale: the extra time spent building those thousands of servers translates into greater savings; for most organizations, you might save a few hundred or thousand bucks going the DIY route, but you’ll be spending even more money in researching/building the five or so systems yourself to not make it worthwhile.

Anyway, a good read if nothing else for the WOW factor.