Sharing ideas with the world

Monday, September 26, 2011

Information Overload

8:39 PM Posted by Deepak Nayal , 2 comments
[For the purpose of this article, data, information and content will be collectively referred to as information] 

We are all drowning in the ocean of information. Businesses and individuals are generating huge amounts of information everyday. This information can be commercial or non-commercial in nature, such as latest news, price quotes, marketing intelligence, bus routes or info on music shows near your house. We are reaching (at least envisioning) a stage where every single entity around us will start generating information (The Internet of Things).

With more information generated and larger data sets created we already are reaching a state of overload, which is leading to increase in information processing complexity, decrease in the average consumption time (or attention span) and increase in missing out on useful stuff. This problem is not just limited to people, but is also affecting organizations. The good news is that if technology can create this problem, it can solve it too (may be not completely though - at least in the near future).

This overall problem of information overload can be broadly divided into two aspects: Information Discovery and Information Consumption, as depicted in the diagram below. 

Information Discovery 
There is so much information getting generated around us that we miss out on many things. We are not able to discover all the information that we would like to or need to. In the early days of internet there were only a handful of sources that you would tap, and that was pretty much it. This obviously is not the case anymore. In addition to tapping their usual sources, users can and do also utilize other methods for discovering information - 
  • Aggregation - Instead of visiting every possible source of information that they can think of, users can get the required information from aggregators. These aggregators vary a lot in the kind of information they aggregate, which varies from news to movies. Google News, Techmeme, Rotten Tomatoes and Epinions are good examples of such [generic] aggregators, which collect information on a particular topic/area and share it with all users in the similar fashion. Services such as Quora and Hunch have taken aggregation to another level. Hunch asks its user a set of simple questions, answers to which help it understand these users better. Based on that understanding, it shares news and recommendations with these users. Quora asks its users to follow topics of their interest, and then shows them the questions raised by other users in those topics. These services leverage the interest graph to provide better aggregation customized for each individual. 
  • Social Discovery - People are always comfortable using tried and tested things, which is why social discovery is a major way of looking for relevant information. You check recommendations before you buy a book on, prefer watching videos online with highest ratings, look at seller's ratings before buying something from eBay, check out links shared by your friends and family on Facebook, and read the top stories of the day on Digg. Social discovery is a very powerful tool to find the relevant information, and has become even more powerful with the advent of social networking. Companies understand that, which is why the spend on social media marketing is increasing at a amazing rate. 
  • Search - Search is another powerful tool to discover information. People look for different kinds of information at different times and in different context, and a good and quick search is the best way to get to the right source. Google is the grand daddy of search and has been doing its job pretty well - barring the times when SEOs play smarter and help push some crap link higher in the search list. A good information search utility is not just useful for individuals, but for organizations as well - as database and content management software can only do so much. 
Even with all these tools/mechanisms it is still hard at times to find the relevant information easily - that relatively unknown song from the 70s which for some reason has been playing in your head all morning; or that research paper recommended by your professor that can help you complete your dissertation; or some good data on a niche market that you are exploring. Not much has changed in the way we discover information for years, but the quality of these methods surely has improved a lot. 

Information Consumption 
In spite of people missing on to a lot of useful information, a good amount useful and interesting stuff still reaches us, which leads to another problem - consumption. With more information getting generated, people face two kinds of problem; it becomes more difficult for them to comprehend that information, and the average time spent on processing also goes down. This is a particularly big problem with enterprise users. However, as with any other problem, there is solution to this one as well. There are ways in which information can be processed or represented in order to enable better consumption by users - 
  • Presentation - One of the most common ways of engaging audience is to present information in a comprehensive manner. This is why charts and graphs have always been preferred over numbers and text. Infographics are another great way of presenting a lot of information in a more comprehensive manner; Nicholas Felton's annual reports on his own life are a great example of the power of infographics. In addition to that, you can always explain things using videos. The more senses you use in learning about something, the more easily you understand it and the better you engage with it. With graphics the audience is only using their eyes, while with videos they use eyes and ears. Zappos started using videos for product demos, and witnessed "a sales impact of 6% to 30%". This is also the reason why iPad (or any good tablet) is a great tool for information consumption - because you are not just seeing and hearing information but also touching it. 
  • Summarization - Summarization is about taking a lot of information in, and then preparing a "executive summary" of sorts which can be consumed in lesser time. This feature is especially useful in fields where users have to deal with huge amount of documentation, such as legal and investment banking. The problem however is that there are not many services for this available out there. Topicmarks has been praised for its results in the past. Techcrunch Disrupt winner, Qwiki is trying to take information consumption to the next level. It summarizes the information requested and then presents it in a video format, which can even be interacted with on a tablet. If Qwiki is able to achieve its goal, it might become the default way to consume information. 
  • Analytics - Analytics can help us consume massive data sets, and is especially useful for enterprise customers. Presentation and summarization alone cannot solve the consumption problem for large data sets such as twitter feeds, click streams and log files. In addition to processing the information, analytics can also spot trends which can help organizations prepare for future. Companies expert in analytics are already helping organizations in many ways such as, developing better products and decreasing waste, and even helping governments prevent criminal activity. Earlier analytics was more of a reflection-on-the-past kind of activity, in which companies would crunch data that is days, weeks or even months old. Thanks to technologies such as Hadoop, organizations are now even capable of realtime analytics, processing the information on the fly and making faster decisions. 
Technology has come a long way in processing information, however we are still behind the curve because information is getting generated on an even faster rate. Entrepreneurs, engineers and mathematicians are working hard in helping us make the most out of this sea of information. It is definitely not an easy problem to solve, but surely an important one. After all, as Gordon Gekko, from the famous movie WallStreet, said - Information is the most valuable commodity.


  1. I think the game is completely about analytics now. I believe the discovery and consumption are maturing quite quickly but analytics has not kept up. with tons of info in their servers, web companies are yet not sure how to use it effectively. The data is not homogeneous and thus very difficult to manage. This is a bubbling volcano of wealth, companies like facebook who hardly make any money are banking on it completely. Put things in perspective, facebook already have 1/6th of worlds population as its user, still is not a profitable company and the key is in understanding that mountain of info.

  2. Analytics is a great field - one of my most favorite. It is definitely one of the most dynamic areas with loads of potential, especially with huge amounts and data and information generated.