Blind Five Year Old

In April, the Internet breathed a sigh of relief when Delicious was sold to AVOS instead of being shut down by Yahoo. In spite of Yahoo’s years of neglect, Delicious maintained a powerful place in the Internet ecosystem and remained a popular service.

Users were eager to see Delicious improve under new management. Unfortunately the direction and actions taken by Delicious over the last 8 months make me pine for the days when it was the toy thrown in the corner by Yahoo!

Where Did Delicious Go Wrong?

I know new management means well and have likely poured a lot of time and effort into this enterprise. But I see problems in strategy, tactics and execution that have completely undermined user trust and loyalty.

Bookmarklets

The one mission critical feature which fuels the entire enterprise falls into disrepair. Seriously? This is unacceptable. The bookmarklets that allow users to bookmark and tag links were broken for long stretches of time and continue to be rickety and unreliable. This lack of support is akin to disrespect of Delicious users.

Stacks

Here’s how they work. Select some related links, plug them into a stack and watch the magic happen. You can customize your stack by choosing images to feature, and by adding a title, description and comment for each link. Then publish the stack to share it with the world. If you come across another stack you like, follow it to easily find it again and catch any updates.

Instead of the nearly frictionless interaction we’ve grown accustomed to, we’re now asked to perform additional and duplicative work. I’ve already created ‘stacks’ by bookmarking links with appropriate tags. Want to see a stack of links about SEO, look at my bookmarks that are tagged SEO. It doesn’t get much more simple than that.

Not only have they introduced complexity into a simple process, they’ve perverted the reason for bookmarking links. The beauty of Delicious was that you were ‘curating’ without trying. You simply saved links by tags and then one day you figured out that you had a deep reservoir of knowledge on a number of topics.

Stacks does the opposite and invites you to think about curation. I’d argue this creates substantial bias, invites spam and is more aligned with the dreck produced by Squidoo.

Here’s another sign that you’ve introduced unneeded complexity into a product.

In just one sentence they reference stacks, links, playlists and topics. They haven’t even mentioned tags! Am I creating stacks or playlists? If I’m a complete novice do I understand what ‘stack links’ even means?

Even if I do understand this, why do I want to do extra work that Delicious should be doing for me?

Design

The visual makeover doesn’t add anything to the platform. Do pretty pictures and flashy interactions really help me discover content? Were Delicious users saying they would use the service more if only it looked prettier? I can’t believe that’s true. Delicious had the same UI for years and yet continued to be a popular service.

Delicious is a utilitarian product. It’s about saving, retrieving and finding information.

Sure, Flipboard is really cool but just because a current design pattern is in vogue doesn’t mean it should be applied to every site.

UX

There are a number of UX issues that bother me but I’ll highlight the three that have produced the most ire. The drop down is poorly aligned causing unnecessary frustration.

More than a few times I’ve gone across to to click on one of the drop down links only to have it disappear before I could finish the interaction.

The iconography is non-intuitive and doesn’t even have appropriate hover text to describe the action.

Does the + sign mean bookmark that link? What’s the arrow? Is that a pencil?

Now, I actually get the iconography. But that’s the problem! I’m an Internet savvy user, yet the new design seems targeted at a more mainstream user. Imagine if Pinterest didn’t have the word ‘repin’ next to their double thumbtack icon?

Finally, the current bookmarklet supports the tag complete function. You begin typing in a tag and you can simply select from a list of prior tags. This is a great timesaver. It even creates a handy space at the end so you can start your next tag. Or does it?

WTF!? Why is my tag all muddled together?

Delicious improved tagging by allowing spaces in tags. That means that all tags have to be separated by commas. I get that. It’s not the worst idea either. But the tag complete feature should support this new structure. Because it looks like it functions correctly by inserting a space after the tag. I mean, am I supposed to use the tag complete feature and then actually backspace and add a comma?

It’s not the best idea to make your users feel stupid.

Uptime

The service has been unstable, lately as poor as it was at the height of Twitter’s fail whale problem. I’ve seen that empty loft way too much.

What Should Delicious Do Instead?

It’s easy to bitch but what could Delicious have done instead? Here’s what I think they should have (and still could) do.

Filtering

An easy first step to improve Delicious would be to provide a better way to filter bookmarks. The only real way to do so right now is by adding additional tags. It would have been easy to introduce time (date) and popularity (number of times bookmarked) facets.

They could have gone an extra step and offered the ability to group bookmarks by source. This would let me see the number of bookmarks I have by site by tag. How many times have I bookmarked a Search Engine Land article about SEO? Not only would this be interesting, it maps to how we think and remember. You’ll hear people say something like: “It was that piece on management I read on Harvard Business Review.”

There are a tremendous number of ways that the new team could have simply enhanced the current functionality to deliver added value to users.

Recommendations

Delicious could create recommendations based on current bookmark behavior and tag interest. The data is there. It just needs to be unlocked.

It would be relatively straightforward to create a ‘people who bookmarked this also bookmarked’ feature. Even better if it only displayed those I haven’t already bookmarked. That’s content discovery.

This could be extended to natural browse by tag behavior. A list of popular bookmarks with that tag but not in my bookmarks would be pretty handy.

Delicious could also alert you when it saw a new bookmark from a popular tag within your bookmarks. This would give me a quick way to see what was ‘hot’ for topics I cared about.

Recommendations would put Delicious in competition with services like Summify, KnowAboutIt, XYDO and Percolate. It’s a crowded space but Delicious is sitting on a huge advantage with the massive amount of data at their disposal.

Automated Stacks

Instead of introducing unnecessary friction Delicious could create stacks algorthmically using tags. This could be personal (your own curated topics) or across the entire platform. Again, why Delicious is asking me to do something that they can and should do is a mystery to me.

Also, the argument that people could select from multiple tags to create more robust stacks doesn’t hold much water. Delicious knows which tags appear together most often and on what bookmarks. Automated stacks could pull from multiple tags.

The algorithm that creates these stacks would also constantly evolve. They would be dynamic and not prone to decay. New bookmarks would be added and bookmarks that weren’t useful (based on age, lack of clicks or additional bookmarks) would be dropped.

Delicious already solved the difficult human element of curation. It just never applied appropriate algorithms to harness that incredible asset.

Social Graph Data

Delicious could help order bookmarks and augment recommendations by adding social graph data. The easiest thing to do would be to determine the number of Likes, Tweets and +1s each bookmark received. This might simply mirror bookmark popularity though. So you would next look at who saved the bookmarks and map their social profiles to determine authority and influence. Now you could order bookmarks that were saved by thought leaders in any vertical.

A step further, Delicious could look at the comments on a bookmarked piece of content. This could be used as a signal in itself based on the number of comments, could be mined to determine sentiment or could provide another vector for social data.

Trunk.ly was closing in on this since they already aggregated links via social profiles. Give them your Twitter account and they collect and save what you Tweet. This frictionless mechanism had some drawbacks but it showed a lot of promise. Unfortunately Trunk.ly was recently purchased by Delicious. Maybe some of the promise will show up on Delicious but the philosophy behind stacks seems to be in direct conflict with how Trunk.ly functioned.

Analytics

Delicious could have provided analytics to individuals as to the number of times their bookmarks were viewed, clicked or re-bookmarked. The latter two metrics could also be used to construct an internal influence metric. If I bookmark something because I saw your bookmark, that’s essentially on par with a retweet.

For businesses, Delicious could aggregate all the bookmarks for that domain (or domains), providing statistics on the most bookmarked pieces as well as when they are viewed and clicked. A notification service when your content is bookmarked would also be low-hanging fruit.

Search

Delicious already has search and many use it extensively to find hidden gems from both the past and present. But search could be made far better. In the end Delicious could have made a play for being the largest and best curated search engine. I might be biased because of my interest in search but this just seems like a no-brainer.

Revenue

Building a PPC platform seems like a good fit if you decide to make search a primary feature of the site. It could even work (to a lesser extent) if you don’t feature search. Advertisers could pay per keyword search or tag search. I doubt this would disrupt user behavior since users are used to this design pattern thanks to Google.

Delicious could even implement something similar to StumbleUpon, allowing advertisers to buy ‘bookmark recommendations’. This type of targeted exposure would be highly valuable (to users and advertisers) and the number of bookmarks could provide long-term traffic and benefits. Success might be measured in a new bookmarks per impression metric.

TL;DR

The new Delicious is a step backward, abandoning simplicity and neglecting mechanisms that build replenishing value. Instead management has introduced complexity and friction while concentrating on cosmetics. The end result is far worse than the neglect Delicious suffered at the hands of Yahoo.

The knuckleball is a very effective pitch if you can throw it well. But not many do. Why am I talking about arcane baseball pitches? Because the Internet has a knuckleball problem.

Image from The Complete Pitcher

The Knuckleball Problem

I define the knuckleball problem as something that can be highly effective but is also extremely difficult. The problem arises when people forget about the latter (difficulty) and focus solely on the former (potential positive outcome).

Individuals, teams and organizations embark on a knuckleball project with naive enthusiasm. They’re then baffled when it isn’t a rousing success. In baseball terms that means instead of freezing the hitter, chalking up strikeouts and producing wins you’re tossing the ball in the dirt, issuing walks and running up your ERA.

If a pitcher can’t throw the knuckleball effectively, they don’t throw the knuckleball. But in business, the refrain I hear is ’X isn’t the problem, it’s how X was implemented‘.

This might be true, but the hidden meaning behind this turn of phrase is the idea that you should always attempt to throw a knuckleball. In reality you should probably figure out what two or three pitches you can throw to achieve success.

Difficulty and Success

The vast majority of pitchers do not throw the knuckleball because it’s tough to throw and produces a very low success rate. Most people ‘implement’ or ‘execute’ the pitch incorrectly. Instead pitchers find a mix of pitches that are less difficult and work to perfect them.

Yet online, a tremendous number of people try to throw knuckleballs. They’re trying something with a high level of difficulty instead of finding less difficult (perhaps less sexy or trendy) solutions. And there is a phalanx of consultants and bloggers who seem to encourage and cheer this self-destructive behavior.

Knuckleballs

In general I think mega menus suck. Of course there are exceptions but they are few and far between. The mega menu is a knuckleball. Sure you can attempt it, but the odds are you’re going to screw it up. And there are plenty of other ways you can implement navigation that will be as or even more successful.

When something has such a high level of difficulty you can’t just point to implementation and execution as the problem. When a UX pattern is widely misapplied is it really that good of a UX pattern?

Personas also seem to be all the rage right now. Done the right way personas can sometimes deliver insight and guidance to a marketing team. But all too often the personas are not rooted in real customer experiences and devolve into stereotypes that are then used as weapons in cross-functional arguments meetings. “I’m sorry, but I just don’t think this feature speaks to Concerned Carl.”

Of course implementation and execution matter. But when you consistently see people implementing and executing something incorrectly you have to wonder whether you should be recommending it in the first place.

Pitching coaches aren’t pushing the knuckleball on their pitching staffs.

Can You Throw a Knuckleball?

The problem is most people think they can throw the online equivalent of the knuckleball. And unlike the baseball diamond the feedback mechanism online is far from direct.

Personas are created and used to inform your marketing strategy and there is some initial enthusiasm and some minor changes but over time people get tired of hearing about these people and the whole thing peters out along with the high consulting fees which are also conveniently forgotten.

The hard truth is most people can’t throw the knuckleball. And that’s okay. You can still be a Cy Young Award winner. Tim Lincecum does not throw a knuckleball.

How (and When) To Throw The Knuckleball

This doesn’t mean you shouldn’t be taking risks or attempt to throw a knuckleball once in a while. Not at all.

However, you shouldn’t attempt the knuckler simply because it is difficult or ‘more elegant’ or the hottest new fad. You can take plenty of risks throwing the slider or curve or change up, all pitches which have a higher chance of success. In business terms the risk to reward ratio is far more attractive.

If you’re going to start a knuckleball project you need to be clear about whether you have a team that can pull it off. Do you really have a team of A players or do you have a few utility guys on the team?

Once you clear that bit of soul searching you need to be honest about measuring success. A certain amount of intellectual honesty is necessary so that you can turn to the team and say, you tossed that one in the dirt. Finally, you need a manager who’s willing to walk to the mound and tell the pitcher to stop futzing with the knuckleball and start throwing some heat.

TL;DR

The Internet has a knuckleball problem. Too many are attempting the difficult without understanding the high probability of failure while ignoring the less difficult that could lead to success.

Do I think Google’s policy around encrypting searches (except for paid clicks) for logged-in users is fair? No.

But whining about it seems unproductive, particularly since the impact of (not provided) isn’t catastrophic. That’s right, the sky is not falling. Here’s why.

(Not Provided) Keyword

By now I’m sure you’ve seen the Google Analytics line graph that shows the rise of (not provided) traffic.

Sure enough, 17% of all organic Google traffic on this blog is now (not provided). That’s high in comparison to what I see among my client base but makes sense given the audience of this blog.

Like many others (not provided) is also my top keyword by a wide margin. I think seeing this scares people but it makes perfect sense. What other keyword is going to show up under every URL?

Instead of staring at that big aggregate number you have to look at the impact (not provided) is having on a URL by URL basis.

Landing Page by Keywords

To look at the impact of (not provided) for a specific URL you need to view your Google organic traffic by Landing Page. Then drill down on a specific URL and use Keyword as your secondary dimension. Here’s a sample landing page by keywords report for my bounce rate vs exit rate post.

In this example, a full 39% of the traffic is (not provided). But a look at the remaining 61% makes it pretty clear what keywords bring traffic to this page. In fact, there are 68 total keywords in this time frame.

Clustering these long-tail keywords can provide you with the added insight necessary to be confident in your optimization strategy.

(Not Provided) Keyword Distribution

The distribution of keywords outside of (not provided) gives us insight into the keyword composition of (not provided). In other words, the keywords we do see tell us about the keywords we don’t.

Do we really think that the keywords that make up (not provided) are going to be that different from the ones we do see? It’s highly improbable that a query like ‘moonraker steel teeth’ is driving traffic under (not provided) in my example above.

If you want to take things a step further you can apply the distribution of the clustered keywords against the pool of (not provided) traffic. First you reduce the denominator by subtracting the (not provided) traffic from the total. In this instance that’s 208 – 88 which is 120.

Even without any clustering you can take the first keyword (bounce rate vs. exit rate) and determine that it comprises 20% of the remaining traffic (24/120). You can then apply that 20% to the (not provided) traffic (88) and conclude that approximately 18 visits to (not provided) are comprised of that specific keyword.

Is this perfectly accurate? No. Is it good enough? Yes. Keyword clustering will further reduce the variance you might see by specific keyword.

Performance of (Not Provided) Keywords

The assumption I’m making here is that the keyword behavior of those logged-in to Google doesn’t differ dramatically from those who are not logged-in. I’m not saying there might not be some difference but I don’t see the difference being large enough to be material.

If you have an established URL with a history of getting a steady stream of traffic you can go back and compare the performance before and after (not provided) was introduced. I’ve done this a number of times (across client installations) and continue to find little to no difference when using the distribution method above.

Even without this analysis it comes down to whether you believe that query intent changes based on whether a person is logged-in or not? Given that many users probably don’t even know they’re logged-in, I’ll take no for 800 Alex.

What’s even more interesting is that this is information we didn’t have previously. If by chance all of your conversions only happen from those logged-in, how would you have made that determination prior to (not provided) being introduced? Yeah … you couldn’t.

While Google has made the keyword private they’ve actually broadcast usage information.

(Not Provided) Solutions

Don’t get me wrong. I’m not happy about the missing data, nor the double standard between paid and organic clicks. Google has a decent privacy model through their Ads Preferences Manager. They could adopt the same process here and allow users to opt-out instead of the blanket opt-in currently in place.

Barring that, I’d like to know how many keywords are included in the (not provided) traffic in a given time period. Even better would be a drill-down feature with traffic against a set of anonymized keywords.

However, I’m not counting on these things coming to fruition so it’s my job to figure out how to do keyword research and optimization given the new normal. As I’ve shown, you can continue to use Google Analytics, particularly if you cluster keywords appropriately.

Of course you should be using other tools to determine user syntax, identify keyword modifiers and define query intent. When keyword performance is truly in doubt you can even resort to running a quick AdWords campaign. While this might irk you and elicit tin foil hat theories you should probably be doing a bit of this anyway.

TL;DR

Google’s (not provided) policy might not be fair but is far from the end of the world. Whining about (not provided) isn’t going to change anything. Figuring out how to overcome this obstacle is your job and how you’ll distance yourself from the competition.

Do as I say, not as I do. That seems to be Google’s philosophy when it comes to blog optimization.

Worst SEO Title Ever

What has finally pushed me over the edge into rant mode? It’s today’s Google+ announcement.

A bunch of numbers for your title. Really? Instead maybe you’d, you know, want to mention the introduction of search or that Google+ was now open to everyone. Those are actually really interesting and noteworthy items.

This isn’t a John Barth novel. The meta information around the number of improvements isn’t really relevant. Really, it’s not.

What query intent are you trying to match here? And yes, that matters.

Snippet Optimization

Google also continues to fail on snippet optimization. Yes, we know that the meta description isn’t a ranking factor. But the description is more important today since it’s used in the transmission of information to other platforms. So what does the snippet for this post look like?

At a glance can you tell what this is about? I certainly can’t. The default image here is useless, the title is nonsense and the description simply tells me that it’s available in other languages. Google can count to 100, seemingly in different languages. Congratulations.

Best Practices and Role Models

Does everyone have to follow best practices? No. All of this is optional. But Google is in a position where they should be setting an example. Google might want to take the Charles Barkley approach, but like it or not, you are a role model.

Or perhaps this is a deliberate thumb in the eye to the SEO community? We know that Google is willing to change titles when they think they’re not quite right. So maybe they just don’t think any of this is necessary? But I doubt that’s the case. Remember the adage about malice.

So please Google, take the time to perform the minimum of optimization on your vast collection of blogs (or give me and my team a call and we’ll get you square.) It’s good for you and it’s good for the search community.

[Update] Well, it looks like the Google Mobile Blog wants to fight for the Worst SEO Title Ever crown with their own numbers post.

 

Why are you still submitting your site and articles to directories? Sure, there was a time when directories were valuable. But that time has passed. So stop feeding their business and build your own instead.

Page Rank Ponzi

Directories are essentially a form of PageRank ponzi. They use your content to build their business – to build their trust and authority – and, in exchange, lease a small fraction of that trust and authority (e.g. PageRank) back to you.

You either give away or actually pay to provide them with content. They take your assets, gladly, and use it to do what you should be doing. Even if you get a small benefit from this exchange, you’re getting the short end of the stick.

Directory Heyday

There was a time when directories were useful and valuable. From the mid-to-late 90s to around 2003, directories were used by many to find sites and content. This was before tabbed browsing and broadband connections made it easy to get from one site to another. This was before search became the dominant way to navigate the web. This was before social platforms allowed you to tap your social graph and crowdsource information.

One only needs to look at the search volume for the term ‘web directory’ to see that this is an outdated method of online discovery.

Distribution

In the directory heyday it may have been difficult to get your site, article or blog post distributed. The web was not nearly as connected or fluid.

But today we have blogging platforms, a robust social graph and numerous social media outlets that give you an opportunity to capitalize on your own intellectual property instead of giving it away to others for peanuts.

We Are The Directory

Whether you call it curation or crowdsourcing there are other repositories that mimic and exceed the traditional directory. You might search Delicious. In fact, more people should. Or you might try out Trunk.ly.

We’re doing the work of directories every day.

Caffeine

In June of 2010, Google launched Caffeine and increased their ability to crawl and index the web. This was one of the last pieces of the puzzle in making directories obsolete.

Previously, directories might have been able to quickly surface new sites or content that hadn’t yet been found by Google. But that’s just not the case today. Google finds new content even in the dark and dusty corners of the Internet where Geocities pages lurk and survive.

Google Directory

So what does Google think about directories today?

Google shut down their directory. Read that again and think about what it means for the future and value of directories. And don’t get me started on the utter collapse of DMOZ. (No, I’m not even going to link there.)

As an aside, Google may want to consider a folder level URL removal so directory results (which return a 404) don’t clutter up SERPs.

Directory Spam

Most web directories are hastily thrown together arbitrage sites that serve as outposts for spam. Here’s a excerpt from an email sent to me by an ‘SEO Consultant’.

This is not SEO, at least not the SEO I practice. Some may reject this carpet bombing approach but subscribe to the idea that a handful of paid directories are worthwhile.

I say save your money.

Paid Link or Paid Listing?

Frankly, I’m still a bit irked that Google doesn’t view a paid listing as a paid link. The argument for paid directories is that they provide a certain level of curation that makes them valuable. You’re paying for someone to curate that directory – not for the link. This seems a very thin argument at best, and a bunch of claptrap at worst. Most, if not all, directories are pretty much a free-for-all as long as what you’re submitting isn’t complete spam or off topic. The level of curation is marginal, and I’m being nice.

Not only that, but it comes down to intent. For some reason I hear Jack McCoy yelling ‘intent follows the bullet’. It’s not a perfect analogy, but the general idea is that intent matters. Today, the intent for a directory listing is, quite simply, to secure a back link. So, what exactly is the difference between a paid link and a paid listing? There is none as far as I can tell.

Link Value

How valuable is that directory link anyway? I’m telling you that the value of these links declines every day. People aren’t using these sites. Newer technologies have replaced directories in the information ecosystem. The closure of the Google Directory should be a wake up call to anyone still clinging to this practice.

TL;DR

Traditional directories are an obsolete method of information discovery. Even if they provide some small benefit today, you’re paying a hefty price to support someone else’s dying business model. Stop PageRank ponzi and invest in the future and yourself instead.

In the past month I’ve left a number of comments on blogs only to find they were never published.

I’d like to believe that the blog owners simply didn’t see my comment. That it fell into their spam queue which they rarely, if ever, look at. Because the alternative is that they saw the post and decided to suppress it. Now, it’s their blog – their little corner of the Internet – but this type of censorship is troubling.

Comments Content

What about the content of my comments? To be fair, in some instances I was disagreeing with some or all of the content in that post. But I did so in a constructive manner, using links to my own thoughts on the topic or to other material to help round out my argument.

I regularly publish comments on this blog that are contrary to my own opinion. One only has to look at the comments on my Stop Writing For People post for examples. I’m free to respond and defend myself, but having the debate in the open is important. It builds trust, much like having bad reviews on a product is actually a good thing.

Comments are incredibly valuable because they provide additional information on the content. They make your content better through clarification, confirmation, addition and debate.

Comments = Content.

Comments are a rich source of meta information that deliver value to both readers and search engines. This extends to links as well! Relevant links in comments help create a web of information that users now and in the future will find useful.

Yet it is those links that may be at the root of the problem.

Comment Spam

It’s like the Internet version of a plague of locusts. One of the most popular ways to combat comment spam is to screen comments that have links. This is one of the default setting in Akismet.

It makes sense since many spammers will drop a link or links in comments. But links are not the problem. Spammers are the problem.

What’s wrong with contextual links to relevant content? This is not behavior that should be penalized. In fact, it should be encouraged. In many ways, the comment spam problem threatens the link graph.

Not only that but, anecdotally, it seems that comment spam sometimes pushes people to disable comments altogether. When the ratio of comment spam to real comments is too high, many simply give up. I understand the decision but it’s depressing that it gets to that point.

Outsourcing

Fed up with comment spam and general comment management, have we decided to outsource engagement to social networks? Twitter, Facebook, LinkedIn, and Google+ are all happy to provide venues in which comments can flourish. Make no mistake, these venues understand the value of comments.

Is our obsession with amplification and generating social proof robbing us of the real value of comments and conversation? Certainly there is some hope that it’s like a rubber band. The content goes out, but then snaps back, drawing more comments to your content. It works to a certain extent, but by how much and at what cost is an interesting debate.

The Filter Bubble

Of course these bloggers may have seen my comment and simply decided not to publish it. Eli Pariser argues that personalization and ‘invisible algorithmic editing’ as a real danger but I think comment censorship (whether intentional or accidental) is the true menace.

I believe much of the hype around the filter bubble is FUD. Personalization is rather minimal in most cases though I do agree with Gabriel Weinberg’s view of how to deal with personalization.

Personalization is not a black and white feature. It doesn’t have to be on or off. It isn’t even one-dimensional. At a minimum users should know which factors are being used and at best they should be able to choose which factors are being used, to what degree and in what contexts.

Personalization deals with the fact that some content isn’t being made readily visible. Comment censorship excises content from the Internet altogether.

Identity

So what could help get us out of this morass? How can we ensure comments are once again a vital part of the content ecosystem? Identity.

The reason why many embraced Facebook comments was because comments are attached to an identity. Not only that, but an identity that people cared about. This obviates the need for aggressive moderation. You might run into a troll, but it’ll be a troll you can clearly identify and block.

Identity essentially stops comment spam because you can’t post as Best Miami Attorneys. Comment moderation is suddenly manageable again.

Censorship

A commenting system that uses identity removes most of the uncertainty around comment censorship. If my comment isn’t published, it’s likely because that blogger made an active decision to toss it into the HTML version of The Bermuda Triangle.

If the filter bubble can be managed through making personalization transparent, so too can comment censorship. A third-party, identity-backed comment system could track the number of comments censored on each blog. A grade or score could then be shown to let users know how much of the conversation was being censored. In some ways it would be like Charity Navigator but for blogs.

So perhaps the blogger who touts the benefits of community actually censors 32% of blog comments. That might be an interesting thing to know.

Could this get messy? Sure. But you can build a system of checks and balances.

Reputation

Joan Jett might not care about her bad reputation but you should. Whether it’s a thumbs-up, thumbs-down, number of Likes, sentiment analysis, length of comments, spelling and grammar or other metrics, a savvy comment system could begin to assign reputation to each user.

So the censorship percentage wouldn’t be flat in nature. If you blocked a known troll, no worries. If you censored someone who had a history of abusive comments full of foul language, no problem.

On the other hand, it would be disturbing if you censor someone who consistently adds value to conversations. The reputation of those you censor would matter.

Confidence

I’d like to be confident that I’m not missing good comments that wind up going into spam.

I’d like to be confident that if I take the time and effort to comment on a blog that it will be published and, hopefully, spark further comment and conversation.

I’d like to be confident that the comments I read are not biased and simply a form of self-curated cheerleading.

“Confidence is contagious. So is lack of confidence.” – Vince Lombardi

The Internet desperately needs more confidence.

(IMPORTANT: Before I get to my story, if your Yahoo! email has been hacked I recommend that you immediately change your password, update your security questions and ensure your Yahoo! Mobile and Y! Messenger are both up-to-date. You should also visit Yahoo! Email Abuse Help and use this process if you are unable to login to your Yahoo! account. Also, make sure to read the comments on this post since there is a tremendous amount of good information there as well.)

(UPDATE 12/13/11: Yahoo has introduced second sign-in verification as an added security measure. It will require that you add a mobile phone number and verify it via a text message. Here’s the direct link to start using second sign-in verification.)

It happened just before we arrived at the San Francisco Zoo. We are at a red light on Sloat Boulevard when my phone started to vibrate.

Buzz. Buzz. Buzz. Buzz. Buzz. Buzz. Buzz. Buzz. Buzz. Buzz. Buzz. Buzz. Buzz.

Had the rapture come a day late? No. I was getting undeliverable messages. Lots of them. My Yahoo email had been hacked!

Here are the two important lessons I learned as a result.

I Have Good Friends

I didn’t want our day at the Zoo ruined, me staring into my phone resetting passwords and figuring out what happened. So I put the problem on the back burner and proceeded to have a fun family day.

But I did take time to quickly tap out a response to people who replied to the spam coming from my hijacked account. Why? Because they took the time and effort to give me a heads up that I had a problem. These were good people. Good friends.

The thing is, I’d gotten a number of these same emails lately from other hacked Yahoo accounts. I figured these people knew they’d been compromised and I didn’t need to respond. With the shoe on the other foot, I realized those emails were comforting even though I was well aware of the problem.

I’ll shoot off an email the next time I get a hacked email from someone.

Yahoo Email Security Failed

The odds are that I will get another one of those emails because I learned just how easy Yahoo makes it for hackers.

Upon getting home I went about securing my account. On a lark, I checked Yahoo’s ‘View your recent login activity’ link.

Sure enough at 10:03 AM my account was accessed from Romania. This obvious login anomaly didn’t set off any alarms? Shouldn’t my security questions have been presented in this scenario? I have never logged in from Romania before.

I’ve never logged in from outside the US. Yahoo knows this. In fact, Yahoo knows quite a bit about my location.

My locations puts me in three states: California, New York and Pennsylvania. I also have location history turned on, so it’s not just my own manually saved locations (some of which are ancient), but Yahoo’s automated location technology keeping track of me.

Do you see Romania in this list? I don’t.

Why is Yahoo making it this easy for spammers to hijack accounts? Make them work a little bit! At a minimum, make them spoof their location.

Yahoo should have noted this anomaly and used my security questions to validate identity. I still would have had to change my password (which wasn’t that bad) but I would have avoided those embarrassing emails.

A simple rule set could have been applied here where users are asked to validate identity if the login (even a successful one) is outside of a 500 mile radius of any prior location.

I’ve had a Yahoo account for over 10 years without a problem, even as I moved my business accounts over to Gmail.

Yesterday I thanked those friends who had my back. Unfortunately, Yahoo wasn’t one of them.

In February Aaron Bradley sent me an email to let me know that I had a duplicate content problem on this blog. He had just uncovered and rectified this issue on his own blog and was kind enough to give me a heads up.

Comment Pagination

The problem comes in the way that WordPress handles comment pagination. The default setting essentially creates a duplicate comment page.

Here’s what it looks like in the wild. Two pages with the same exact content.

http://blog.wolframalpha.com/2011/04/18/new-age-pyramids-enhance-population-data/comment-page-1/

http://blog.wolframalpha.com/2011/04/18/new-age-pyramids-enhance-population-data

That’s not good. Not good at all.

Comment-Page-1 Problem

The comment-page-1 issue offends my own SEO sensibilities, but how big of a problem is it really?

There are 28 million inurl results for comment-page-1. 28 million!

Do the same inurl search for comment-page-2 and you get about 5 million results. This means that only 5 million of these posts attracted enough comments to create a second paginated comment page. Subtract one from the other and you wind up with 23 million duplicate pages.

The Internet is a huge place so this is probably not a large percentage of total pages but … it’s material in my opinion.

Change Your Discussion Settings

If you’re running a WordPress blog I implore you to do the following.

Go to your WordPress Dashboard and select Settings –> Discussions.

If you regularly get a lot of comments (more than 50 in this default scenario) you might want to investigate SEO friendly commenting systems like Disqus, IntenseDebate or LiveFyre.

Unchecking the ‘break comments into pages’ setting will ensure you’re not creating duplicate comment pages moving forward. Prior comment-page-1 URLs did redirect, but seemed to be doing so using a 302 (yuck). Not satisfied I sought out a more permanent solution.

Implement an .htaccess RewriteRule

It turns out that this has been a known issue for some time and there’s a nice solution to the comment-page-1 problem in the WordPress Forum courtesy of Douglas Karr. Simply add the following rewrite rule to your .htaccess file.

RewriteRule ^(.*)/comment-page-1/ $1/ [R=301,L]

This puts 301s in place for any comment-page-1 URL. You could probably use this and keep the ‘break comments into pages’ setting on, which would remove duplicate comment-page-1 URLs but preserve comment-page-2 and above.

Personally, I’d rather have the comments all on one page or move to a commenting platform. So I turned the ‘break comments into pages’ setting off and went a step further in my rewrite rule.

RewriteRule ^.*/comment-page-.* $1/ [R=301,L]

This puts 301s in place for any comment-page-#. Better safe than sorry.

Don’t Rely on rel=canonical

Many of the comment-page-1 URLs have a rel=canonical in place. However, sometimes it is set up improperly.

Here the rel=canonical actually reinforces the duplicate comment-page-1 URL. I’m not sure if this is a problem with the Meta SEO Pack or simple user error in using that plugin.

Many times the rel=canonical is set up just fine.

The All in One SEO Pack does have a Canonical URL option. I don’t use that option but I’m guessing it probably addresses this issue. The problem is that rel=canonical doesn’t stick nearly as well as a 301.

So even though this post from over three months ago has a rel=canonical, the comment-page-1 URL is still being returned. In fact, there are approximately 110 instances of this on this domain alone.

Stop Comment-Page-1 Spam

23 million pages and counting. Sure, it would be nice if WordPress would fix this issue, but short of that it’s up to us to stop this. Fix your own blog and tell a friend.

Friends don’t let friends publish duplicate content.

Yesterday Google rolled out an algorithm change “designed to reduce rankings for low-quality sites—sites which are low-value add for users, copy content from other websites or sites that are just not very useful.”

Google’s share of the search market must have been suffering, right? Wrong. comScore puts Google at 63.0% in January 2009, 65.4% in January 2010 and 65.6% in January 2011. People were not defecting.

Not only that but Google is the leader in search engine customer satisfaction according to the American Customer Satisfaction Index. So why the change?

Google Means Well

I believe Google thinks they’re making things better. I don’t see a Machiavellian scheme behind every Google action. I like these guys. Meet any of the people at Google and you realize they’ve drunk deep from the search quality kool-aid. They are true believers! On top of that, they’re usually amiable and generous with their time.

But … the road to hell is paved with good intentions.

Google Got Bullied

What’s shocking is how Google got pressured into making this change. Vivek Wadhwa, Paul Kedrosky, Jeff Atwood, Michael Arrington, Rich Skrenta and others played Google. A bunch of upper-class, highly-educated technophiles convinced Google that search quality was in jeopardy. Was search quality really an issue or was this a matter of taste?

A reminder, you can please some of the people all of the time and all of the people some of the time but you can’t please all of the people all of the time.

The Echo Chamber

A good marketer knows that they are not the target market. If you’re reading this, you are not the primary search user. You might be a power user, but you are in the minority my friend.

Perhaps there is more under the hood, but from where I sit Google chose qualitative feedback over quantitative feedback. The problem? That qualitative feedback was biased. The Silicon Valley echo chamber flexed its muscle and Google acquiesced.

Red Herring

What is disappointing is that Google decided to tackle the subjective (content quality) instead of the objective (link fraud). Do we truly think that JC Penney, Forbes and Overstock are outliers? The answer is an unquestionable no.

What’s a bigger threat to search quality? The blatant and rampant manipulation of trust and authority via link fraud or the creation of content (of varying quality) to meet query intent?

What Changed

A staggering 11.8% of queries were impacted by this algorithm change. I’m curious about how Google effected this change.

Did they re-weight current signals or create new signals? Google acknowledges that data from the Personal Blocklist Chrome extension was not used. That doesn’t mean other new signals or data weren’t used. But even if Google did introduce other new signals, to impact 11.8% of the queries it seems reasonable to believe that current signals were also re-weighted.

That assumption and hours of SERP review lead me to the following conjecture.

• Trusted TLDs (org, gov, edu) were given more weight
• Exact Match Keyword Domains were given more weight
• Forums were given more weight
• On-Site text was given more weight

The last presents itself in an odd way. Sites that look like they were last touched in 2003 are ranking well. It’s as if Google sought a ‘no style’ version of the web. This also includes a number of long form blogs. Sadly, many of these same sites are bloated with AdSense. Now, AdSense is everywhere so … that’s to be expected. But the position of the ad units on many of these sites is completely against any UX standard.

This is a very simplistic and blunt analysis. I’m sure others will tease out other differences and we’ll never know for sure what changed. But what it tells me is that Google changed quantitative measures to meet a pre-determined qualitative goal.

The Real Story

Google passed judgment on the quality and value of sites in what seems like a very subjective manner. How exactly did these sites and specific pages rank so well in the past? What suddenly changed? Did the pogosticking rate creep up? Did internal satisfaction metrics of the ‘reasonable surfer’ change? I’m not hearing any of that. I’m hearing subjective terms like ‘quality’, ‘value’ and ‘useful’ being thrown around.

Google is setting their own perceived metric of value in conflict with other signals, metrics and feedback. The message? Google doesn’t trust us to know any better. It’s not about what we want. It’s about what Google thinks we should want.

The idea that Google altered current signals to effect a perceived content quality metric should terrify you.

It’s all very well and good when those changes don’t impact you. You guffaw at Mahalo’s demise. But what happens when they come for you? What happens when you’re suddenly the target? How will you feel when your content is called into question?

The needs of the many outweigh the needs of the few.

Blekko Spam

Just prior to Farsight 2011, Blekko removed twenty sites from its search results.

“These sites are the worst spam publishers on the Web according to our users,” said Rich Skrenta, CEO of Blekko. “They are literally responsible for millions of pages on the Web that our users say are just not helpful and they’d prefer they were banned permanently. So we’re going to do that for them.”

Blekko has some interesting functionality around spam so I can see why they’d want to highlight it based on the recent spam/content farm meme surrounding search. That’s understandable. But censorship is not the answer.

Blekko Users

There is precious little data as part of this announcement. How big is Blekko? Quantcast and Compete show that the monthly unique visitor count is anywhere between 16,000 and 143,000. However, to mark anything as spam you have to be a Blekko user.

The November 2010 public launch of Blekko provided some insight into numbers and usage.

Blekko has been testing its solution to search with roughly 8,000 beta testers who have created more than 3,000 different slashtags. Blekko tells us that 11% of its existing user base come back to the site on a weekly basis.

I was a beta tester. So were a number of my colleagues – innovators, technologists and SEOs. As a search marketer we were eager to try out a competing search engine. I’m not a Google apologist.

Without hard data the math gets fuzzy, but the total number of registered users seems relatively small and is likely still composed of innovators. Do these people represent everyone?

Blekko Searches

The other missing piece of data is the searches related to these spam complaints. We don’t know the types of searches that were performed, nor the result set that was presented to users. Are the spam complaints a measure of the sites or a measure of the quality of results returned by Blekko?

Are spam complaints produced on general search queries or long tail queries? Is the incidence of spam complaints for specific sites different based on query type? (Information vs Transaction vs Navigation.)

The spam interface also leads to another question. How many of the spam complaints were made without visiting the site in question?

Blekko Censorship

Aaron Bradley took the words out of my mouth in his Blekko, Can I Please Have My Spam Back? post.

At the end of the day, I have no respect for a search engine that censors my results based on notions of quality, rather than relevancy.  It ceases to be comprehensive, it smacks of elitist righteousness and – most of all – decisions about the validity of content are being made on my behalf by people I don’t know.

Quality and taste are subjective. The fact that Blekko has chosen to use the feedback from a biased minority to censor results for the majority is unfortunate. Is the message that mainstream users don’t know enough to make their own decisions, the right decisions? If I search for ‘food’, should unhealthy foods be removed from search results?

In all seriousness, would Blekko remove specific books that users had marked as spam? According to the American Library Association, this would mean Catcher in the Rye, To Kill a Mockingbird and The Color Purple would vanish from the landscape.

Use spam feedback to reorder results, but let me make up my own mind. I don’t need a nanny search engine.

Disclosure: While I consult for Buzzillions, this post is my personal opinion and does not reflect those of Buzzillions.