Archive for November, 2005

Spam

Wednesday, November 30th, 2005

Current laws designed to prevent spam haven't really helped - the international nature of the internet simply means that spammers route spam (and, on the other end of things, "generated leads") via countries without anti-spam laws so that they cannot be traced by law enforcement.

To eliminate spam, it is not necessary to be able to identify every single spam email. Catching 99% or even as low as 90% would probably do the trick. By reducing the income of spammers by a factor of 10-100, sending spam quickly ceases to be economically feasible.

Catching 90-99% of spam or more is quite possible with today's spam filters. The problem is that because the focus is on eliminating the damage done by spam (the expense of the bandwidth it uses) these spam filters are generally implemented on email servers rather than at the client end. This means that there is no easy way for users to give feedback when the spam filter makes an incorrect choice (no access to the emails marked as spam, and no easy way to report a missed spam as such). This means that people are inclined to turn the filters off (so that they don't miss any email) and also means that the filters are never "trained" to recognize the newest spam-detector-foiling techniques.

The first thing we need to do is have the most popular email clients contain Bayesian spam filters. Emails detected as spam are downloaded but put into a separate folder and users are not notified when new spam is downloaded as they are when new non-spam ("ham") is downloaded. This means that users never need to worry about false positives - they can always check back through their spam folders for a missed message. As the amount of spam decreases, it eventually becomes possible to look at every spam during slow periods, to make sure there were no false positives. These clients will have two extra buttons "delete as spam" and "false positive" that they can use to help train the spam filters.

Whenever the email client is connected to the internet, it uploads its latest changes to its filter data to a trusted central server. This server collates all the information from the clients and produces new filter data which is sent back to the clients. In this way, all spam filters can quickly be updated to recognize the latest spam keywords and filter-avoidance techniques.

How do we prevent the spammers from polluting the filters by sending a large amount of bad data to the servers? All clients are authenticated to the servers and a trust metric is set up. If the data sent by the client tends to agree with data sent from all the other clients, that client's trust rating goes up. If it tends to disagree, the trust rating goes down. That way, the damage that can be done by a particular client is very limited (the filter should be designed to be able to cope with a small amount of incorrect data).

The final change that can be made (and, I think, the one that would make the most difference) is educating end-users that responding to spam is a bad idea. If a large red flashing message saying something like "Warning! The message below is likely to be fraudulent in nature. Exercise extreme caution in giving any money or information to this person or organization" appeared above any email detected as spam, it would probably put off most of the potential clients of the spammers. This method also minimizes the negative consequences of false positives.

These methods work even if not everybody adopts them and for the most part they are most helpful for those who do adopt them (thus providing an incentive for adoption).

Please feel free to evaluate my spam solution against the spam solution evaluator:

Your post advocates a

( ) technical ( ) legislative ( ) market-based ( ) vigilante

approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)

( ) Spammers can easily use it to harvest email addresses
( ) Mailing lists and other legitimate email uses would be affected
( ) No one will be able to find the guy or collect the money
( ) It is defenseless against brute force attacks
( ) It will stop spam for two weeks and then we'll be stuck with it
( ) Users of email will not put up with it
( ) Microsoft will not put up with it
( ) The police will not put up with it
( ) Requires too much cooperation from spammers
( ) Requires immediate total cooperation from everybody at once
( ) Many email users cannot afford to lose business or alienate potential employers
( ) Spammers don't care about invalid addresses in their lists
( ) Anyone could anonymously destroy anyone else's career or business

Specifically, your plan fails to account for

( ) Laws expressly prohibiting it
( ) Lack of centrally controlling authority for email
( ) Open relays in foreign countries
( ) Ease of searching tiny alphanumeric address space of all email addresses
( ) Asshats
( ) Jurisdictional problems
( ) Unpopularity of weird new taxes
( ) Public reluctance to accept weird new forms of money
( ) Huge existing software investment in SMTP
( ) Susceptibility of protocols other than SMTP to attack
( ) Willingness of users to install OS patches received by email
( ) Armies of worm riddled broadband-connected Windows boxes
( ) Eternal arms race involved in all filtering approaches
( ) Extreme profitability of spam
( ) Joe jobs and/or identity theft
( ) Technically illiterate politicians
( ) Extreme stupidity on the part of people who do business with spammers
( ) Dishonesty on the part of spammers themselves
( ) Bandwidth costs that are unaffected by client filtering
( ) Outlook

and the following philosophical objections may also apply:

( ) Ideas similar to yours are easy to come up with, yet none have ever been shown practical
( ) Any scheme based on opt-out is unacceptable
( ) SMTP headers should not be the subject of legislation
( ) Blacklists suck
( ) Whitelists suck
( ) We should be able to talk about Viagra without being censored
( ) Countermeasures should not involve wire fraud or credit card fraud
( ) Countermeasures should not involve sabotage of public networks
( ) Countermeasures must work if phased in gradually
( ) Sending email should be free
( ) Why should we have to trust you and your servers?
( ) Incompatibility with open source or open source licenses
( ) Feel-good measures do nothing to solve the problem
( ) Temporary/one-time email addresses are cumbersome
( ) I don't want the government reading my email
( ) Killing them that way is not slow and painful enough

Furthermore, this is what I think about you:

( ) Sorry dude, but I don't think it would work.
( ) This is a stupid idea, and you're a stupid person for suggesting it.
( ) Nice try, assh0le! I'm going to find out where you live and burn your house down!

(I started doing so myself, but realised I could not be objective.)

Foggy night driving

Tuesday, November 29th, 2005

On our way back from Port Townsend the other weekend it was delightfully foggy out. Not so foggy that you can't see the front of the car (I've tried to drive in such conditions, and it's scary) but foggy enough that there were some really cool lighting effects in the stretches where there weren't any other cars. Streetlights, for example, start as a faint glow far in the distance. As you get closer, the blob of light resolves into a conical shape. As you pass through it it is like being on stage under a spotlight - bright underneath but blackness everywhere else. It reminded me of some the surrealism in the old Looney Tunes cartoons. Then a moment later all is black again apart from the stretch of road illuminated by the headlights and the receding/fading cone of light in the rear view mirror.

Overtaken by shadows

Monday, November 28th, 2005

My favorite bit of cycling back from work is freewheeling down the hill on 32nd Ave E. It's quite steep so you can go pretty fast but not so dangerous that you have to use the brakes. If its dark out then there is also an interesting lighting effect - as you pass under the streetlights your shadows seem to sneak up behind you and then zoom past before fading in the glow of the next light. It freaked me out a little when I first noticed it and didn't realize what it was.

Interview your future self

Sunday, November 27th, 2005

I thought of a cool idea while listening to This American Life on Friday. Imagine you could talk to a future version of yourself, 10 years older than you are now. What would you ask them? Record (audio or video) a set of questions for them, put them in a time capsule and in 10 years time record the answers. Splice the questions and answers together to make a interview across time instead of between two individuals. Of course, follow-up questions could be tricky.

Hot porn

Saturday, November 26th, 2005

I showed this to the wife this morning and she said I should post it.

Photographs - Our honeymoon (page 3 of 3)

Monday, November 21st, 2005

11th to 15th October - The Cinque Terra


The kitchen of the apartment we stayed at in Manarola


View from our apartment


We were staying at the very top of the orange house


The vineyards of Manarola. People have been farming these terraces for 1,000 years


Manarola has some incredible deep-water swimming


The Via Dell'Amore, between Manarola and Riomaggiore


The beach at Monterosso


Part of the path between Manarola and Corniglia


The footpath zig-zags up nearly 400 steps to the town of Corniglia


A cat that convinced Gennie to feed part of her lunch to him


View from the hike between Monterosso and Vernazza


Vernazza from the hiking path


Some of the many cats that decorate the Cinque Terra


16th October - Seattle

On our return to Seattle, we found (amongst other things) this message on the whiteboard...

...and this fun present from Beth and JT.

Photographs - Our honeymoon (page 2 of 3)

Monday, November 21st, 2005

7th to 11th October - Florence


The ceiling of the Baptistry


The Duomo


Giotto's tower


The top of the Duomo


Garden terrace outside our window


The Arno river

Pisa

Photographs - Our honeymoon (page 1 of 3)

Monday, November 21st, 2005

1st October - The Old Consulate Inn, Port Townsend


3rd to 4th October - Milan


Milan's Duomo is the fourth largest Cathedral in Europe


Vittorio Emanuele Arcade


4th to 7th October - Venice


San Marco square

Murano


The Bridge of Sighs


The Ducal palace


Gennie eating the best Gelato in Venice - at 10am!

Photographs - Summer/Autumn 2005

Monday, November 21st, 2005

A walk in the arboretum


New York, 2nd September


Theodore Roosevelt's birthplace


Gennie and her brother Tom

New Haven, Connecticut, 3rd September

The wedding of Beth Goldsmith and JT Russell


Halloween


5th November - the wedding of Johanna Schaeffer and Kody Dickerson


Kathleen


Kristi


Nurit and Sean

Andrew and Gennie's wedding - photos (page 4 of 4)

Friday, November 18th, 2005


Matt


Melissa and Matt


Aldo and Yarrow


Jeremy (who took most of these pictures) and his girlfriend Katie


Jo and her husband (then fiancee) Kody


Kathleen


Thus began the legendary dance-off between Tom and Bruno...


Christopher