Image via Wikipedia

Overview:

IMHO, the current state of search is depressing. This is not a new realization for me. It is seven or eight years ago now that I first imagined a social search engine which would not rely solely on algorithms to determine the relative importance of search results but that would consider both machine and end-user feedback. This was in the early days of Nutch and I began researching the possibility of utilizing Nutch as the underlying core engine for such an endeavor, I rounded up some small-term investment capital, and so on. Unfortunately, this was also at the high peak of my struggle with Obsessive Compulsive Disorder (OCD) and my efforts eventually fell through.

Over the years I have watched as promising engine after promising engine has come along and in their turn failed to take the lead or even maintain their momentum. Years have passed and at each step of the way I have said, “It must just be around the corner…This is ages in technology time.” Even Google came out with SearchWiki, while not a perfect implementation it was a huge step in the right direction. For the last year or two I’ve been using Zakta and I’ve spent time on almost every other social search engine currently (or previously) available – yet I find that in the long-run they have all failed me.

So here I am so many years later longing for just such an engine. I’ve written on this blog about the topic before, but I will write again. In this post I will specifically propose the formation of an endeavor to create a social search engine, and I hope it will foster some interest in the community. I am not ready nor able to undertake such an endeavor myself – but I am interested in being part of such an endeavor.

Open Source: Ensuring Continuity

It is worth noting at this juncture that I’d intend for this project to be open source. Too many times I have lost the social search data I have accumulated because a specific engine has folded. My hope would be that the resultant project would be open source with commercial implementations and would provide a significant amount of data portability between engines, in case one engine should fold. We’ll talk more about the open source and portability aspects of the project later in this proposal.

What is Social Search?

Before we jump into a discussion about how to build a social search engine it is necessary first to define what is meant by social search. Unfortunately the term social search is used to delineate several different concepts which are very different from one another.

There are the real-time search engines which focus on aggregating information from various social media networks – and sometimes prioritizing links based on their popularity within a network. For example Topsy, the no-longer-real-time OneRiot, and the now-defunct Scoopler.

There are the engines which are focused on finding humans – e.g. allowing one to garner information about a person. Wink eventually became this sort of engine, Spokeo would be another example. They are essentially white pages on steroids.

Finally there is what I mean by social search – and I would use another term but there is no other term I am aware of which is so widely used to delineate this type of engine (and I want to ensure the widest possible audience). It is sometimes called a “human-powered search engine”^[1] Google and Wikimedia may have come closest by terming it a “Wiki” (SearchWiki and Wikia), but it seems to me that there is a need for an entirely new term that better and more precisely defines the idea…perhaps one result of this proposal and its aftermath will be just such a term.^[2]

Core Parameters

In this section I will delineate what I believe are the core required features for a social search engine. An engine which included these features I believe would be a 1.0 release. There is certainly room for numerous improvements, but this would define a baseline by which to measure the proposal’s progress. I am not infallible, and I am sure there are aspects of the baseline which should be edited, removed, or replaced – I am open to suggestions.

• Web Crawler – The engine must include a robust web crawler which can index the web, not just a subset of sites (e.g. Nutch).
• Interpretive Ability - The engine must be able to interpret a wide variety of file formats, minimizing the invisible web (e.g. Tika).
• Engine - The engine must be able to quickly query the aggregated web index and return results in an efficient manner (e.g. Nutch).
• Search Interface - The engine must include a powerful search interface for quickly and accurately returning relevant results (e.g. Solr).
• Scalability – The engine must be scalable to sustain worldwide utilization (e.g. Hadoop).
• Algorithms – In addition to the standard automated algorithms for page relevance the system must integrate human-based feedback including:
• Positive and negative votes on a per page basis.
• The ability to add and remove pages from query results.
• Influence of votes based on a calculation of user trustworthiness (merit).
• Promotion of results by administrative users.
• Custom Results – The results must be customized for the user. While the aggregate influence of users affects the results, the individual user is also able to customize results. One should see a search page which reflects the results one has chosen and not the results one has removed.
• Ability to annotate individual entries.
• Portability – The engine should define a standard format for user data which can be exported and imported between engines. This should include customized query results, annotations, votes, removed and added pages, etc. This will be available to the user for export/import at any time. While additional data may be maintained by individual engines, the basic customizations should be portable.

I’m sure I’m missing some essentials – please share with me whatever essentials I have forgotten that come to your mind.

Starting from Zero?

It is not necessary for this project to begin from nothing, significant portions of the endeavor have already been undertaken toward creating an open source search engine – largely by Apache’s Nutch project. The available code should be utilized and with customization could integrate social search features. This would allow some of the most significant aspects of the project to be offloaded to already existing projects.

Additionally, it might be hoped that companies and individuals who have previously created endeavors in this direction would open source their code. For example, Wikia was built on Nutch and the code – including the distributed crawler (GRUB) and UI (Wikia) was released into the open source world.^[3]

What We Need

Now the question becomes, “What do we need?” and more importantly, “Whom do we need?”

First off, we could use donated hosting. Perhaps one of the larger cloud-based hosting companies would consider offering us space for a period of time? I’m thinking here of someone like Rackspace, Amazon Web Services, or GoGrid.

Secondly, we’d need developers. I’m not a Java developer…though I’ve downloaded the code and am preparing to jump in. I also don’t have a ton of time – so depending on me to get the development done…well, it could take a while.

Thirdly, we’d need content curators…and I think this is key (and also one of the areas I love the most). We’d need people to edit the content and make the results awesome. These individuals would be “power users” whose influence on results would be more significant than the new user. With time individuals could increase their reputation, but this would seed us with a trusted core of individuals^[4] who would ensure that the results returned would be high quality right from the get-go for new users^[5]

Finally, we’d need some designers. I’m all for simplicity in search – but goodness knows most of us developers have very limited design abilities and an aesthetic touch here and there would be a huge boon to the endeavor.

Next Steps

At this juncture its all about gathering interest. Finding projects that have already begun the process, looking for old hidden open source code that may be of use, etc. Leave a comment if you’d like to be part of the discussion.

Appendixes

Current Open Source Search Engines

• DataparkSearch – GNU GPL, diverged from mnGoSearch in 2003, coded in C and CGI.
• Egothor – Open source, written in Java, currently under a complete from scratch rewrite for version 3.
• Grubng – Open source, distributed crawler..
• BeeSeek – Open source, P2P, focuses on user anonymity.
• Yioop! (SearchQuarry) – GNU GPLv3, documentation is very informative.
• Heritrix – Open source, by Archive.org for their web archives.
• Seeks Project – AGPLv3, P2P, fairly impressive project which attempts to take social search into consideration.
• OpenWebSpider – Open source, written in .NET, appears to be abandoned.
• Ex-Crawler – Open source, Java, impressive, last updated released 2010.
• Jumper Search – Open source, social search, website appears to be down, currently linking to SF.
• Open Search Server – Open source.

Related articles

• Could Bing Reach Its Potential if Owned by Apple or Facebook? (greatfinds.icrossing.com)
• Create Your Own Search Engine With Yioop (crenk.com)
• Do Search Engines Have a Future? (arnoldit.com)
• Your Future: Search, Social, and Content (greatfinds.icrossing.com)

1. ^[1] Or a human search engine, which becomes sadly entangled with engines meant for finding humans such as referenced previously. ↩
2. ^[2] A few other terms which might be appropriate are collaborative search engine, though this would have to be prefaced with “active” to distinguish it from passive user feedback aggregation (e.g. how long a user stayed at a site); curation search engine (giving the idea of content curation, but this is sometimes thought of in terms of archival); or crowd-sourced search engine (though this centers too much on democracy, whereas such engines would probably benefit from a meritocracy). ↩
3. ^[3] Unfortunately, I have been unable to find a copy of the Wikia UI code. ↩
4. ^[4] Taking a page from early Ask Jeeves history. ↩
5. ^[5] Obviously not necessarily in the long tail, but in the general topics. ↩

Alex Howard has an excellent article over at O’Reilly on the Community Health Data Initiative including some of the applications that have resulted from the availability of this data. Howard notes that it has been ten years since GPS data was open sourced – and everyone can see the tremendous ways in which that has boistered business (creating new industries) and innovation.

• WaveMaker – Build rich internet applications (RIA) using a WYSIWYG interface. Community edition is open source. Creates Java applications. Wikipedia Article.
• Visual Web GUI – Build RIA’s using visual development interface. There is a free/open source express edition with regular pricing beginning slightly under $350 for a license. Creates .NET applications. Can deploy to Windows Azure.
• SalesForce – The Force platform is the defacto standard PaaS. Significant free offering included with up to 100 users, etc. Also, free licenses for non-profits with majority price discount on additional licenses.
• nuBuilder – A open source project that allows for rapid development of web database applications. Wikipedia Article.
• BungeeConnect – Uses an Eclipse-based IDE.
• Web Fuser (Inuvia Technologies) – IDE and hosting. Hosting starts at $20/mo.
• WinDev – Free lite IDE for rapid development of JAVA/.NET applications. Wikipedia Article.
• Wolf Frameworks – Has a free starter plan (2 users, 100 MB storage, unlimited apps/entries). Wikipedia Article.
• LongJump – Pricing starts at $30/user/mo.
• WorkXpress – No pricing, thirty-day free trial. Does offer the ability to host with them, a third party, or your own. Claims to require no programming.
• SpringBase – Fairly impressive free account for those looking to create a small database application. Appears they no longer offer a free trial. Pricing starts at $99/year.
• TrackVia – Pricing is expensive ($249/mo., starts at $99/mo.). Online database platform
• DBstract – Offers free accounts and low-cost premium accounts ($20/mo.) for creating database applications/hosting.
• Caspio – Starts at $40/mo. Claims to require no programming. Wikipedia Article.
• Zoho – Free account for up to two users, $5/ea./mo. additional users.
• Hyrdo4GE – Still in closed beta.
• HyperBase – Part of HyperOffice. (thanks: Jean Churchill).
• MyTaskHelper – UI is pretty basic, but it is free.

Image via Wikipedia

For those who are Christians or who are interested in understanding Christianity, there are few sites on the internet more valuable in learning and growing than the Biblical Studies Foundation (bible.org). While the resources available to understand and practice Christianity are extensive generally, the availability and freedom with which the BSF makes its resources available is practically unparalleled. I have used the BSF site for years and continue to utilize it regularly and rave about its magnificent capabilities.

Let’s take a look at a few of the BSF’s many features:

• New English Translation (NET) – A brand new translation of the Old and New Testaments from the original manuscripts. The NET is readable and yet precise, but what really makes the translation stand apart is the 70k+/- footnotes that are throughout the text. These footnotes are not commentary on the text but rather explain the translators decisions, especially on controversial verses. They offer deep insight into the original texts and are an amazing aid to the bible student or translator.
• Book Commentaries - Contemporary commentaries written by sincere bible students/scholars are freely available on the BSF website. While there is great value in the two thousand years of commentary we have on Scripture, these commentaries offer an additional perspective including the latest manuscript and archaeological evidence, contemporary illustrations and applications, and so on while maintaining fidelity to the Scriptures.
• The Theology Program (TTP) – An extensive theological training program meant for churches to utilize in training lay individuals in theology. The course is indepth, practical, and understandable. Its meant to help those who want to push on in their theological understanding but cannot afford the expense or time commitments of a college education at this juncture in their lives.

These are only a few of so many wonderful things you will find at the BSF. I insist, you must visit!

• Open Source - I’m always a fan of open source projects, not just b/c I like a free lunch as much as the next guy but also because it allows for the project to continue on beyond the lifespan of a given individual or company.
• ASP.NET - Its only been within the last several years I’ve really begun messing around with LAMP, and for the longest time I loved ASP and then ASP.NET. Now I’ve been swung to the dark side recently, though I still find Microsoft‘s development tools to be leagues beyond the open source competition (for speed of development) and still prefer developing in a VB.NET-like syntax to C#, PHP, etc. But, this habit must die…b/c everyone else is going LAMP.
• Simplicity – Compared to Joomla or Drupal, DNN is a breeze. Within minutes of installing the application you can have a full featured site up and running.

That said, I’m now leaving the DNN community (I’ll get to what I’m moving to in a few moments). Here are the simple reasons why:

• Cost – While DNN itself is open source, the Microsoft ecosystem as a whole is much more oriented around cost-based. This especially holds true for the DNN third-party ecosystem of modules and skins. Both of these would have some commercial items in a similar LAMP based project, but there would be loads of free modules/skins. Not so of the DNN ecosystem.
• Development – Feature development in DNN seems to go at a much slower pace than equivalent open source projects (though this may change with the venture capital infusion DNN recently received). One significant example is the forums module which has been without an update for well over a year and has several show-stopping bugs in the current production version.
• Openness – While DNN is an OSS project, the sharing of news about what is happening internally as far as development as well as the ability to get the latest snapshot download to run on the bleeding edge is extremely limited.

So what am I moving to? Good question. Its not Drupal or Joomla. I find both of these overly convoluted (here come the haters). Instead I’m moving to WordPress. WordPress while initially designed as a blogging platform has extended itself significantly to include most functionality that a user could want from a CMS in the core install. Thousands of free extensions make up for whatever WordPress lacks at its core. The development pace is rapid and even minor versions include massive updates (e.g. 2.7 is awesome!). The skins/modules are free, free, free and if one module isn’t receiving development there are dozens others that are.

That said, I’m not abandoning DNN completely just yet. It works well enough for davemackey.net, ocddave.com, and a few other sites. At this juncture the cost to move them over to WordPress (in time and energy) is greater than the lost features (since these are essentially static content sites, they aren’t missing out on much). I plan to in the future – as the need arises.