OK this is a quick one… I’ve just spend 1hr fixing my LinkedIn profile. One very nice feature of this service, is that it can import your contacts from Gmail, do a string match against its database, and if some of your email contacts are already in LinkedIn, send them requests to join your professional network.

I just realize now, how easy it would have been if I had all the emails from my company going to Gmail: I would be able to easily leverage the proffesional network tracking capabilities of the LinkedIn service, keeping connected with all the people I exchanged email for business, without any effort from my part (and besides I wouldn’t have to deal with Outlook madness daily, rather do some Gmail Fu).

All these might seem self-intuitive, but look behind the lines: as many of the people I’ve met for the past couple of years (both online and in real space) we have been saying to each other, when data stay closed behind the firewalls, they are siloed and they cannot gain in value by being connected.

So put everything on the biggest existing cloud out there, the internet, and let it be connected by the duct tape called Web 2.0. A devil’s advocate might bring up the issue of privacy, which is a significant one: for example many of my email contacts might not want their address being parsed by LinkedIn, because we don’t know what LinkedIn does with them…

Now I am eager to see if I can evade IT, and fwd all my email to my Gmail account….

Imagine to be able to write code to process a large dataset, and immediately being able with a click of a button to run that code on a compute cluster, without worrying about setting up anything like job submission to a grid, resource allocation, compilation etc etc etc

Now stop imagining, because this is reality today. The details how this can happen is via your NetBeans IDE (Integrated Development Environment) and Amazon’s Elastic MapReduce. Leveraging the capability of this Amazon web service for provisioning Hadoop (the open source implementation of Google’s Map/Reduce parallel programming framework) compute clusters of any size on the fly, Karmasphere offers an amazing Netbeans plugin.

Developers can add their Amazon credentials within the IDE after installing the plugin, write their code, and perform parallel computing on as big cluster as they desire (or to be realistic, as big as the credit limit their card has). The plugin takes care of communicating with the Amazon Elastic MapReduce API and submitting the code for execution, while Amazon sets-up the cluster in minutes and pulls the desired data from  S3 storage.

The most impressive thing for me in this story, is the abstraction for all the layers that used to require an expert a few years back (see MPI and C), in order to perform high performance computing. And the second impressive thing is the democratization of access to large computing resources. Definitely infrastructures and software are in place in big corporations, allowing business analysts and researchers to write and execute code on big clusters without worrying about the details of cluster setup. But how about those outside those corporations ? And how about those that don’t want to code in C or C++ ?

Through Hadoop’s Streaming option though, anyone that can write a Perl script which will operate in a large dataset on a compute cluster, by following the most abstract programming model of Map / Reduce. With a laptop having an internet connection and a credit card, a Bionformatics researcher anywhere in the world can write code, and execute it without worrying for resource constraints, but only worrying about his or her application logic.

The latest issue of BioIT world magazine is dedicated to exploring, how are we going to survive the data tsunami coming with the next generation genome sequencing technologies.

Given the fact that I am bioinformatics engineer, sleeping, eating and drinking cloud computing, the following article caught my attention right away:

http://www.bio-itworld.com/BioIT_Article.aspx?id=94324

It is about GenomeQuest, a company that offers services covering all that follows completion of the sequencing of a genome (assembly, annotation etc) The most interesting part of this article, is that this company has an in-house cloud computing platform.

The best point from this article though which follows that, is that they made the infrastructure choice of an in-house cloud, which in turns allows them to scale up, by outsourcing to Amazon Web Services when business needs increase…

Now talk on scalability, and the power of the clouds and economies of scale in Bio-IT. Read the Big Switch by Nicholas Carr, and then you will be thinking like me on the issue of traditional data center VS cloud, to evolve or go the way of the Dodo (extinct).

I have been always amazed by the dynamics of the internet… a complex network, similar to the pathways in a living cell, power grids and social webs. It can explode, shrink, change dynamic, all by following purely non-linear dynamics.

A good example of this, is what happened yesterday to this blog… inactive for the past 3 months since I’ve started my new job, it just got 5000+ reads in the past couple of days, as the stats for WordPress’ dashboard show below:

All these reads went to a post from back in May, tittled  software jobs, PhDs and “over-qualification”.

Why did that happen ? By Googling around a bit, I’ve found that a Reddit power-user, BioGeek, posted a link to this post of mine. A lot of commentary followed up on Reddit and also many comments on the original post here on WordPress.

So I guess it all about riding on the shoulders of giants… In any case, this traffic to this blog lifted my spirit, and I decided to start posting to it again after a long absense.

Stay tuned for the updates, which will include knowledge, tips and tricks, opinions and trends, from the new thought space I’ve submerged myself too for the past 3 months; having to do with cloud computing, science as a service, big data and Hadoop, and everything distributed.

In this post, I’d like to talk a bit about the potential of the Vim (Vi Improved) editor, as far as using it for writing code… Huh? What does this have to do with the tittle of this post, anyone is wondering who is reading this line right now…

Before I write what is needed to justify the tittle above, let me quickly point you to what you can do with the Vim editor: you can see that structure of your code (methods, variables etc) and you can indent your code automatically (works with many languages, but think Ruby/Python)… then you can do many more, but with these two examples you get the idea. Have a look at the 2526 vim extensions here, and to pre-occupy you, there’s even an extension for querying databases and one for writing wordpress posts !

Coming back to to the tittle, imagine being a guy who is stuck with refactoring messy and boring Java code (me). So you get up every morning, sit in front of your computer and trying to find that bug, peeing through lines and lines of code, while scrolling down the bar on your editor… is this boring ? Yes. How do you spice it up ? Well, I used to switch every 5 minutes or so to the Firefox window with Twitter, and read the new updates from the people I follow, both to blink and clear my eyes and also to get my mind off from running in a loop when thinking on a certain bug… now, when I need a tiny break like that, I do a bit of Vim Fu every time. These little bits add up to a belt with a darker color eventually, moving me up the Fu hierarchy. So in software, there is always a learning opportunity, you just have have to discover it. When you are stuck with a boring project, similar to when you have a blatantly-tasting food, look for some spice to add to it…

PS 1 Another Vim goodie I should not forget to mention, is that you just move around your text without using the arrow keys, this without taking your fingers of the alphanumeric part of the keyboard… plus you can erase whole words / paragraphs / sections with a few key combinations…Try Vim and enjoy your time in front of the computer !

P.S. A nice, free online Vim book can be found here.

I recently wrote a comment to a post related to the problems of our energy and material hungry civilization, and how one man’s living style can make a huge difference. My ideas as seen below where I’ve copied that comment, have to do with how we are consuming resources from our physical environment, in order to produce more and more information (feed our global brain). For more on this, keep reading below….

….

To buy new socks when you get holes on your current ones, or buy baked bread, buys you thinking time (usually in front of a computer). It seems to me that in the industrial revolution we altered the physical world around us, in order to have it supporting us during the next step (which we’re taking currently). That support is getting your socks and bread ready, so that you can sit in front of the computer and take that next step, towards the inside-out information revolution. This of course doesn’t mean it is sustainable, since we get into the viscious cycle of adding more and more stress in the physical world around us (energy,materials), in order to feed our global brain.

Whether the solution is to take one small step back or a big one like you did, or just wait until we collapse under our own weight, is debatable. I think there is a hope of a sustainable energy and material hungry civilization, through efficient production of both these under newly discovered technologies resulting through the information revolution.

This post is just a quick ping (aka to the sound of submarine sonars), to give short sound of life to this blog. I haven’t written at all lately, part of it being working hard on the final aspects of my PhD research, while another part being the tons of reading I’ve been doing on exciting Web 2.0 and scalability stuff (Ruby on RAILs, MapReduce/Hadoop – for which I’ll be blogging in the near future).

For now, I want to wish a happy New Year to everybody out there… As a closing remark, I don’t know how many of you use wikipedia, but I spend a few minutes reading articles from it every day. So if you’re like me, and you want to keep this wonder of collective intelligence running, please consider giving a donation:

Best wishes to all !

….because up to now bioinformatics research with genomics datasets, has been happening like that: you download the data from a website of a big-iron institution (NCBI, TAIR), set them up locally, BLAST ‘em, MySQL’em, parse them with Perl script, and do all other sorts of un-imaginable things. Even though bioinformaticians might be un-aware of the term, part of the local processing that happens with the data is a mashup. This term translates to the combination of pieces of data from different sources, something akin to what has been happening on the web (see also Web 2.0 or programmable web). In no way this is close to the myriad of Web 2.0 mashups that exist out there, created using APIs offered openly by different servers. In this case different sets of data are brought together by the mashup developer, who also adds value to them through their re-combination (and reciprocally adds value to the providing server, through spreading out and offering a better view of their offered data).

While the big-iron bioinformatics institutions don’t quite live in a parallel universe from Web 2.0 (we have to credit the NCBI server for its GCI interface), there are light years away from the programmable web. That is both because of the technologies their are using (forget about Ruby on Rails and REST), but also because of the small number of institutions like NCBI offering APIs.

So why should a (bioinformatics) scientist learn web development ? Because this situation I am describing above will change. These bioinformatics institutions will adopt Web 2.0 at some point during the next years – I can bet you now that, OK maybe in 5 years, we will have an NCBI running a nice REST API backed by Rails or Django. But it might happen even earlier, when people take things at their own hands. And for that I refer you to Amazon Web Services, where bioinformaticians can build their own NCBI running on Rails and sell it to other Web 2.0-minded scientists, who understand the (added) value of an inter-operable web of data.

I believe FriendFeed is one of the best social networking sites that someone can draw benefits from the curation of knowledge through collective intelligence. In a quick summary, what you can do with this site is follow the consciousness stream of people within your online social network, as this stream is expressed by their online activity through bookmarking, blogging, photo-sharing etc (see this for list of aggregated services).

I personally have set search results from my FriendFeed network delivered to my Google reader. In this way, I get del.icio.us bookmarks, blog posts etc from people within my network that have keywords such as web 2.0, semantic web and so on. This doesn’t giv e you the option on discussing the chuncks of the aggregated consciousness stream (one of the strong points of FriendFeed), but still you get to get bookmarks or other pieces of information that you’d never dig out yourself (however well you know how to do searches with Google).

What I am trying to get to ? Well, I’m trying to present FriendFeed as a platform where the aggregation takes place, and datasets of value are created through human curation of sources from around the web. I am sure other sites exist that do something similar (Pownce maybe?), but I am not sure whether anyone offers such search options (choosing from an individual or a group of people’s stream, rooms where peers share their content etc), and especially to get a different RSS feed from each search term.

What we need is more platforms like this. In the best case, to have on open source implementation of the FriendFeed model  of operation, something akin to identi.ca, the open counterpart of Twitter which can be run on your own server. This is where people can flock together to create peer-to-peer communities, and this is where knowledge using the web 2.0 framework is aggregated, creating islands of knowledge on the net. There, simply someone can retrieve the pieces of the curated knowledge by searching the streams on topics of interest. In this way information collected (and given away for free?) in these separate communities becomes one’s building material for a second layer of aggregation, where collective intelligence is efficiently used to sift through the web, outsmarting any page-ranking algorithm.

And to conclude, here’s a good business model for FriendFeed (and any other communities that will implement the same operation model) :

- show the 100 most recent entries from an individual’s or a room’s stream

- if people want to mine the aggregated-curated knowledge (>100 entries) sell your data (has been done with Flickr, you upload as many photos as you want, but you see the last 100)

And what bussinness  you are building this way:  have you seen search engines aka Hakia, which try to build on collective intelligence? Friendfeed does and has that already, it only needs to figure out a smart way to sell it…

It happened that I stumbled upon this post by K21st, which triggered some of my thoughts on the upcoming singularity. The next two paragraphs are my comments to this writing. If you don’t want to go very far with computer-brain interfaces, but just want to think how slow are the computer interfaces (typing, reading) comparing to the tasks we are supposed to achieve on the multi-tasking, super information-intensive time we live, just skip to the last paragraph. Let me provide a couple of pointers though, for those who like novels I recommend Accelrando by Charles Stross, and for popular science here goes Kurzweil.

I believe that we’re still in the dark ages concerning brain function. Knowing via neuroimaging the brain area that gets activated when an arm moves, and plugging an electrode there that controls a robotic arm in a disabled person doesn’t mean much (except that it helps this person a lot!). The real “boom” in merging of brain and machine, will be when our knowledge lets us address the third point of your post, how information is stored in the brain. How we can change synapses and chemicals, so that we have 2-way flow of information: from the brain outwards, and from the outside environment to the brain.

This might be a totally different topic, but I strongly believe that a significant change on the machine interfaces that allows us to exchange information with each other, will create a huge difference. For this, see as an example slowness of typing, reading from the screen, visiting webpages etc. I think the bandwidth of the current networks is greatly underutilized, given that the Kbps of a DSL line stay idle while I write this blog post ! What would be the interface that allows me to assimilate information (maybe not by reading) while I put down my thoughts for this post ?

Today I was going through another round of pimping-up this blog, and was thinking of how to add some of my stream of consioussness displayed here (a stream expressed through online activity such as bookmarking, blog post reading, tweets etc). My first thought was Yahoo! Pipes, which lets you mashup and process feeds from around the web and then present them as a customized feed, picking up chunks from the various feeds that contain certain keywords for example.

But then epiphany happenned (!) and FriendFeed(FF) clicked in my head. As I have written before, FF allows you to aggregate feeds from a variety of social networking sites. Here is my online consciousness stream collected from FF, which is also publicly available. The nice thing with it is that you can type a keyword in the search box right above my public FF, and right away get a custom RSS with chunks of the aggregated feeds that contain this keyword. This custom feed is publicly available too, and here is one that contains pieces with the keyword “semantic”  and would be appropriate for the present blog.

So this makes Yahoo! Pipes obsolete for this kind of job. That’s because you can replicate the Pipes’ fucntionality by using FF with “impersonal” feeds, not bookmarks or photos or music, just text feeds (from weather, crime, news etc). Friendfeed. This allows you to aggregate plain RSS (for those who are in FF, use ‘blog’ radio wave button from add/edit services menu). One thing I’d suggest if you go ahead and do that, especially if you mashup a bunch of these plain text feeds, make up a new account on FF that you use specifically for this task. Reason for that is because if you already have a social network established, you don’t want your friends from FF getting all the RSS you’re piling up for the mashup.

Finally, the sweet thing with this approach is that you don’t need to write any code – well in Yahoo! Pipes you plug components together rather than writing, but still in with Pipes you have to engineer things a little. In my case, since I’m all up for two-way interaction and not passive stuff, the next follow-up hack on this will be to write a bit of Javascript so that you (the visitor of this blog) can interactively do that consoussness stream search from within SemanticLifeScience, with the keywords of your choice.

I have just stumbled upon this awesome link for 100 Semantic Web Research Papers … seems like there is a lot of good reading material there, and definitely it will fill up my time for the next few weeks. From a quick scroll through the list, I’ve picked out the following:

Embracing Web 3.0, by Ora Lassila

A survey of the Web Ontology landscape, by Taowei David Wanget al.

Semantic Web research trends and directions, by Jennifer Golbeck et al.

A definition of trust for computing with social networks, by Jennifer Golbeck

Task computing – the Semantic Web meets pervasive computing, by  Masuoka et al.

Semantic interoperability and information fluidity, by Guofei Jiang et al.

well, to be honest, there are 125 papers in this page. The ones I’ve picked above are the more “generic” ones I’d rather say. The remaining from the list deal with the more esoteric kind of stuff related to the semantic web, like trust, reasoning etc.

Happy reading !

I am currently in the process of looking around for jobs, in light of my graduation which (if I sit down to write my thesis consistently!) will be in the few months. One of the things I realized while deciding which employer’s doors to knock, is how much I was benefited since I started obsessively reading blogs and stuff around the web a couple of years back. And I will explain myself in the following few paragraphs…

For all of us that are into a PhD program, it means a lot of journal-article reading, classes that teach you all sorts of (in my opinion) esoteric stuff (like Multidimensional Scaling, Eigen-vector-inversible-matrices, Rolling Medians etc etc etc). Plus you have to “look the tennis ball in all 12 dimensions”, an expression I use to characterize detailed (and thus high quality) scientific research, which requires you to examine all aspects of a very-very focused domain you specialize during your doctoral. All those, don’t leave you too much time to see what’s happening out there in the real world, where the fruits of academic research are turned into useful products for society (see private sector and industry).

Of course to be able to look at the tennis ball in all 3 and 9 more imaginary dimensions, you have to acquire a lot of skills, which in the case of being say a bioinformatics student will be: Perl, Java, SQL, PL/SQL, SPARQL-RDF, Postgres, MySQL (schema design, query optimization, replication), Linux (server admin., .deb, .rpm), HTML, XML, Javascript, RSS, Apache Tomcat, Python, Xpath. XSLT, CSS …….

……….And the list goes on and on…….. (well, basically I copied all those above from my CV).

So I will come around now, to the web trend watching / blog reading issue I have mentioned in the beginning. And my point here, which also is what I am trying to say with this post, is that by reading your web stuff you get to see what happens out there (outside of the closed world of academia), so you can utilize those acquired skills of yours better. Cause if you get know for example what is and how works stuff like Web 2.0, web services and clouds, you realize that your RSS, Java and SQL/Linux skills can be of great use to a prospective employer.

Instead, most PhD’s don’t even have an idea about social networking or cloud computing, and they go around like crazy trying to find a postdoc. Well you might say, some might like to do a postdoc, for the research heck of it – but think about it: if you get to a postdoc job after your doctoral, how much coding are you going to be doing ? (referring to bioinformatics again) I believe a lot, and same is if you go work for a company as a developer (who will pay more?). Some people again might argue about publications that your get out of the postdoc, and then you can use them as credits to advance your academic career. Oh well, the way I see it with bioinformatics, there’s heavy coding wherever you go work, whether postdoc or company. The difference is that with the postdoc you get funded (most cases) by government and write some stuff that go into journals, and get some training to write proposals to ask for money from the funding agencies. Other than that, is the underlying nature of the job (coding) in either academia or industry the same ?

That is why I say read often, read a lot, read variety, so that you can see what is out there – get outside of the academic egg, and then make your own opinions – and some might match with all the ranting above. In any case it’s all a big experiment, like life itself, but if you’re happy with what you’re doing and what you have inside your head, all is good…

What is wrong with the current way science, dissemination of research findings and resulting benefits for society at large, take in the current Internet era ? … and why do we need things like Open Access, Science Commons, Open Science….?

The problem

One good way to define it, is by quoting from the Science Commons website:

“There are terabytes of research data being produced in laboratories around the world, but the best web search tools available can’t help us make sense of it. Why? Because more stands between basic research and meaningful discovery than the problem of search. Many scientists today work in relative isolation, left to follow blind alleys and duplicate existing research. Data is balkanized — trapped behind firewalls, locked up by contracts or lost in databases that can’t be accessed or integrated….”

“The consequences in many cases are no less than tragic. The time it takes to go from identifying a gene to developing a drug currently stands at 17 years — forever, for people suffering from disease.”

Three main factors should be considered when trying to understand the statements above:

• Copyrights for published data (Open Access)
• Open distributed collaboration (Open Science)
• Knowledge fragmentation and data standards (Science Commons)

Copyrights for published data

An excellent article by Peter Murray-Rust in Nature Precedings (January 2008) documents from personal experience and also using examples from others, what happens when you “hand” your article to a peer-reviewed journal for publication. The central idea in this article is that for many of the traditional, non-open access (subscriber only) access journals, whenever you publish an article you have to surrender copyright to the publishing house over the text, data, figures and supplemental material of your manuscript (by signing a “Copyright Transfer Agreement”). To demonstrate the consequences of this fact, Murray-Rust reports on a case of a student that posted on her blog some of the supplemental material from a manuscript, in order to comment on some conclusions she reached from the data. What followed her post was  notices for legal action from the journal and publishing house, based on the claim of copyright violations !

Doesn’t this journal’s action come in contrast with the fundamental spirit of science, which is being open with research results and sharing them in order to get or provide feedback from / to peers ? The way all journal copyright agreements are currently set up is not to serve the interests of the scientist or for the advancement of knowledge, but for protecting the profits of the publishing house. The first thing those agreements usually state, is pretty much that broadcasting of the article content on a medium that reaches the public is not allowed. So if we think about the simplest case, we all do an  infringement each time we put up on our presentations a graph from a published paper that is copyrighted !

Imagine yourself in a situation where after reading an article you see a pattern in the data, which the authors haven’t observed or wrote about in their article (or maybe they noticed but did not want to write about, for obvious reasons). You probably cannot publish a new article for just your observation. The alternative is to try sending a letter to the journal, but the question is whether they will publish it or not. Even if it gets through, the journal will not allow much print space for letters communicating back and forth. Another option is to contact the authors and let them know, or maybe also include the article’s graph with your observations during a presentation in a seminar. The journal’s lawyers probably will not get to see the copyright infringement, that you perform by broadcasting the graph through your Powerpoint slides within the conference room!  But how far can you get through these channels, and alternatively what would be the medium that will spread your word most ?

The answer is the world wide web, where publishing is free and gets to many, many people’s eyes and ears without any barriers. Especially by using a blog website allowing commentary, discussion can go on and on, unlike a journal pages where communication between authors and readers is restricted. Now thinking about  the real world example Murray-Rust reports in this paper, would you post chucks from the paper along with your comments on a blog, then sit around and wait for the journal to contact you with legal threats because you’ve posted on the web copyrighted material ?

While it seems logical for the text of an article to be copyright-able (because it is rather the art of the author, similarly to when writing a novel for example), the same is not true of the data. The data should belong to the public because they are facts measured from nature, and similarly to the patent law only artifacts made by man can be patented – but not any products of nature.

We see therefore how the copyright framework can stifle advancement of knowledge, since it allows discussion of scientific research to be performed only through limited channels. In contrast to Open Access, copyrighted articles do not allow  any user who has access to the Internet to link, read, post and dicsuss, data-mine and re-analyze the digital content of that article. Furthermore, Open Access secures equal opportunity, since everyone can have access to the latest research results and not only those nations or institutions who can afford the expensive subscriptions fees to journals.

Open distributed collaboration

As the corresponding article on Wikipedia puts it,  Open Research and Open Science are

“…research, conducted in the spirit of free and open source software…..built around a source code that is made public, the central theme of open research is to make clear accounts of the methodology, along with data and results extracted therefrom, freely available via the internet. This permits a massively distributed collaboration…..”

For no other reason,  open collaboration offers the advantage that a large group of minds can reach faster a solution, than an individual or a few closely collaborating peers. This is clearly demonstrated in the principles of Open Innovation, a business model encouraging search of skills further than the local talent pool within a company’s confinements. In the current era of Web 2.0, communication media such as blogs and social networking websites can be used by people of common goals and interests as high bandwidth channels, for the flow of ideas amongst each other. That is based on the fact that a sheer number of people are connected through the internet, and through the Web 2.0 sites they can express themselves and contribute their opinions on topics, something described already as collective intelligence. The practice of open source software development, and examples such as whole operating systems like the GNU/Linux developed by the self-assembly of individuals in online working groups, demonstrate the wisdom of the masses. Imagine therefore, how benefiting could be for a scientist working on a hard problem to harness this collective intelligence in order to reach a quicker and better solution.

The problem is that current academic system discourages open collaboration, mainly because promotion to tenure uses the number of peer-reviewed publications as evaluating metric. Researchers safeguard their results in order to secure their competitive advantage based on novelty of publication, in fear that openness might deprive them from that edge. There is currently no metric system that provides accreditation to the participating scientists of an online community, that might be working towards the solution of a difficult problem. A system which will credit open collaboration and account for individual contributions, will result to a more efficient discovery process by harnessing the untapped potential of large groups of scientists working together.

Knowledge fragmentation and data standards

This part problem comes from the fact that data are being locally deposited in the computers of each research group, while they are also stored in all sorts of formats. Probably we all have seen the “data available upon request from the authors” in research papers.  Most of us have also gone through the process of having to write to the authors in order to request the data from a publication. Following receipt, we find out that it takes quite some time to figure out the on-the-fly-just-designed-to-get-the-job-done data format most people use (I include myself with those people when I have deadlines), and how frustrating it can be when trying to built upon a published manuscript’s data.

The goal is to have directly accessibly and re-usable scientific data, by depositing on public repositories and following standardized formats. Ideally, this will enable integration of the various published results, but of course it would also require open access to articles’ contents. Just being able to play around by easily integrating data from various sets of published experiments (say for example studies on proteins that have been found to be related to cancer), would probably bring forward a completely new set of insights.

There is currently a whole range of technologies that are designed towards integration of information drawn from diverse sources, such as for example the semantic web, or more practical but less powerful like the XML standard. None of these can work though if research results are locked behind database firewalls, preventing development of applications which will automatically aggregate the data and sift through them for the discovery of interesting patterns. In my opinion this will be the real “boom” in bioinformatics, when we will be able to add value in the existing information by intelligently combining (mashup using computers) data-sets, something already being a very profitable industry outside academia (Google Maps mashups,  ProgrammableWeb).

Besides the experimental data from the published articles, equally important is for the text to be in machine-understandable format. A good example that demonstrates this is the following: suppose you are conducting research on one topic, and maybe there are 25, 50 or even 150 papers related to it. You can probably read all those in a few months and keep referring back to them (you cannot memorize all of the content!) while you advance in your experiments. But is there a more tedious thing than going back to a stack of papers – or an e-stack of PDF files, and trying to spot the section that you remember contains a clue to help you interpret a result you got from an experiment ?

That is where data standards and open-access comes to the rescue. Over at NCBI’s PubMed there is an open access repository of peer-reviewed articles, for which content is deposited in XML format. In a few words, Pubmed’s XML tags all sections of an article using computer-processable identifiers such as <Title>, <Abstract>, main text <Section>’s paragraphs, <Graph> etc. In this way the tedious finding of couple sections or graphs among 150 papers that contain results related to your work, becomes something your computer can do. Using a simple XML parser software, you can have every chunk of all the articles on your fingertips, and very easy perform operations like “bring me the graphs from articles that contain such and such keywords in their abstract”.

PubMed enables all that because it follows the XML standard and has the text of the articles open and free. But this is only one very small fraction of the articles published in peer-reviewed journals. For all the rest, you will have to go through your stack of papers and search to find the section of interest. We need open and machine processable literature, so that new knowledge can be generated by aggregation of the current findings. It is easy for somebody to see that this slow process of manually gathering knowledge, can hinder research in mission-critical areas such as drug research.

Conclusion

With the current publishing system it is difficult for the opinions of readers of an article to reach the authors, and even more difficult to echo it back to the wider audience that an academic article is intended for. The peer review process is slow, and only examines the results of scientific inquiry under the scope of the few minds participating in the process. Open collaboration can be the way for tackling difficult scientific problems by harnessing collective intelligence, but a new system is needed which will provide the necessary credit to those participating in online communities. Acceleration of research is possible when data are deposited in openly accessible repositories, in formats which can be processed by computers. Complex scientiifc problems can be solved, and difficult questions answered by building upon the increased value of the aggregated information.

References

Science 2.0: Great New Tool, or Great Risk?
(http://www.sciam.com/article.cfm?id=science-2-point-0-great-new-tool-or-great-risk)

Freeing the Dark Data in Scientific Experiments
(http://www.wired.com/science/discoveries/magazine/15-10/st_essay)

The Future of Science is Open
Part 1 (http://3quarksdaily.blogs.com/3quarksdaily/2006/10/the_future_of_s_1.html)
Part 2 (http://3quarksdaily.blogs.com/3quarksdaily/2006/11/the_future_of_s.html)
Part 3 (http://3quarksdaily.blogs.com/3quarksdaily/2007/01/the_future_of_s.html)

Open source drug discovery (http://economist.com/displaystory.cfm?story_id=2724420)

Another reason for opening access to research (http://www.bmj.com/cgi/content/full/333/7582/1306)

ProgrammableWeb Applications using open data from maps, markets, blogs adding value and some create a bussiness

Great part of the research within the bioinformatics field, involves inSilico experiments by assembling software pipelines for performing the subsequent steps of data analysis tasks. Until recently two approaches were followed for these computational experiments: first, for users with less technical experience the browser was the option for analyzing their data using algorithms accessible via the web interfaces. This involves the tedious task of copying and pasting data between browser windows, manually performing and documenting the steps of the analysis pipeline, while visiting one website after another. More computer-savvy users, circumvent this by installing on local machines or invoking remotely through web services the software implementing the bioinformatic algorithms, after scripting the analysis pipeline steps.

While the first approach consumes time, the second consumes resources. In between the two, lie the workflow design and enactment platforms, which allow techically oriented and not users to perform complex analyses, with minimal effort and resources. One such open-source platform is Taverna, which is implemented in Java and offers a user interface that allows assembly of analysis pipelines with simply plugging the output of one web service as input to the next. In Taverna, users can assemble workflows that invoke bioinformatics algorithms accessible through SOAP or REST interfaces. In addition, users can incorporate in their pipelines ready-made Java widgets for parsing and querying XML data, read and modify local files, execute system commands and, connect to relational databases, and furthermore perform virtually any data operation that could be scripted or wrapped in Java code.

So if you can plug it together by point-and-clicking, why should you script?

(besides the pleasure of coding….)

I’ve stumbled upon this post for the Open Laboratory of 2008. Toying with the idea of submitting one of my writings, and looking back in all the stuff I’ve written since I started putting my thoughts online a little less than a year back, I decided that the best writing is ironically not located here (!), but on GenBioWiki (that’s my graduate program’s flirt with Web 2.0 apps).

I’ve mentioned that writing I’ve did for GenBioWiki on the present blog, and the actual link for it on the wiki is here. So what I will be doing is adapting this article from the wiki, and re-posting it on SemanticLifeScience, and then I can go on with the submission.

I guess I’ll have to check a little the spelling, cause those guys from the Open Laboratory will put the selected blog posts on a book !

…is Shareaholic. It’s a Firefox pluggin, and as you can see in this picture it allows you to post in a bunch of Web 2.0 sites… I find Shareaholic to be so practical since it gives you everything in one button, without having to fill your Firefox toobar with 4-5 different buttons (which you’ll have to aim and click for each different site you want to post). Only drawback is that it opens a new tab to show you the post you just did (haven’t found yet how to turn this behavior off), which for some people might be good, for others bad. I tend to work frantically within my browser, and that’s why I do not want more than one buttons or new tabs popping up…

… The next best tool for me is FriendFeed. I will not get here into many details on how it works, I’ll just say that whatever gets marked/posted/commented via Shareaholic and the APIs of the various Web 2.0 sites it supports, eventually ends up via aggregated feeds in FriendFeed, for discussion and sharing in online rooms where similarly-minded people hang-out.

For those of you who are already messing around with FriendFeed, let me say that a nice hack is to create a private room with only yourself as a member, where you can re-share the chuncks of your feeds that were commented and discussed by other people. I find this to be extremely useful, in the sense that I daily go about my fruit-picking around the Web, gather stuff that I find interesting by posting on the various sites via Shareaholic, pull them as feeds over at FriendFeed, and then keep track of what other people say about the various thingies I’ve posted around the web…

This last example specifically, for me shows how one’s ideas expressed though collection of related/supporting information from around the Web, can be echoed back, augmented and verified by other people thinking alike. Maybe that’s what it is to harness some Collective Intelligence….

I just read this post on Programmable Web about Oracle teaming up with Amazon Web Services (AWS), to offer some of their products to run on virtual server instances…what this made me think is how many companies currently do most of their IT stuff with Oracle, and what does this mean for AWS: more business, if the management of companies buying Oracle’s stuff find that it’s more cost-effective, to go virtual on the cloud rather than setting up their own servers.

But also might mean that the demand for cloud computing will increase, since for startups and smaller companies it’s definitely better to delegate all the server purchase cost + maintance time, and buy computing as utility. I’m even tempted to think how many of the other software ventors will follow Oracle’s example and set up their products on the cloud.

Either way, this will drive the utility computing market up, so I expect to see more and more offerings matching those of AWS….

This is the cover letter I wrote for a job application at Google… I believe it captures all my thinking that took form the last couple of years, after jumping head first into the web 2.0 world of online apps. First I started exploring all this social and read/write type of web simply out of interest, but then I realized what web 2.0 can do from helping you to expand, cultivate, update your professional and personal ideospace, to increasing the usability of the world’s wide web of information.

“I believe innovation comes from crossing boundaries. The first time I did such a cross was five years ago, when I took the leap from molecular biology to bioinformatics and computer science. Following this transition I learned new ways of doing research, by learning to deal with larger datasets that can only be handled using computers, but also I realized that the same principle governs both fields: being able to find answers by searching information in the best way possible. This in particular is what Google started out doing, and still does, but now provides besides its ranking of web pages new tools that help us re-wire the web. Having used the mashup editor, the reader to aggregate and share blog content, and also seen the potential of the documents application for moving the desktop on the web, I decided instead of simply being a user, to become a contributor. I am determined to use my software and technical skills to play a part in the development of these tools or any other innovative projects currently happening at Google. I am ready to cross the boundaries once again, and use the lessons from years of scientific research in working for a company that evolves the web, providing users with the tools to make it a richer information medium.”

Now that’s a cover letter unlike the ones you usually see…. But Google is one kind of a company that urges people to explore, so I thought to let myself free and take a go at them with this letter. So I’ll be awaiting to see the outcome of this…. wish me luck !

This is a post adopted from a long email I wrote to a personal acquaintance of mine, where I went on to do some confessions on what I think about the nature of my current academic job. So here it goes:

” I did lots of introspection (and reading, and discussing with other geeks like me) this past semester, and lately my ideas finally started getting into full shape. I basically realized that I like to solve tricky problems. whether that has to do with analyzing results to find patterns, or reading about and understanding new methodologies and then applying them to my own data. But because I like to built stuff, I see best to make computer programs to accomplish this. And to make the long story short, I am repelled by the academic blah-blah-blah and lecturing, all the theorizing and writing papers, and in a sense making things on top of thin air… For me, it is better to build things (well even if you just type on a keyboard instead of using a hack & saw), things that you can hold with your own hands and see them become live and interact with the real world (even if they’re just operating inside a computer)… Academic papers and theses are just a bunch of ideas – even though they result from hard work – but it’s not a concrete thing, and somebody can use them only as part of their ideospace (see mental cloud, or space of ideas). But in the end, I think somebody can have an exhilarating experience by swimming through the ideospace of a programming language and its application towards the solution of a hard problem” .

So that was a paragraph of that email adapted. And I will close by saying about the importance of going after what fulfills you most. A very nice post related to this principle, that inspired me and I have read earlier this morning is here.

This debate stated so boldly on the tittle above, is not something that I’m going to solve in this post. But is something that has made me thinking a lot lately, partly for personal reasons while in the process of wrapping up a PhD, and trying to evaluate what I have learned as the years have passed (and what I want to do with my life next)…

So to get right into the point, I see a distinction in the bioinformatics field between scientists who go after testing hypotheses and bringing out new knowledge, and developers, who are basically engineers that built,break, hack etc. bytes which make computers do interesting stuff. What I want to emphasize though, is the catch for the ones that lie in the middle, not few cases of these observed within this field.

Many where the times that I fell (and still falling) into this “middle” trap myself, something I realize as I look in the past and the present of my work. Let me explain with a simple example what this trap is: suppose you want to use a piece of software that visualizes some data. Suppose also that this software was not made by Microshoot (you know which one I mean), but rather is a product of a small lab, presumably a result of a graduate student’s PhD project. Let’s say that this software is written well enough (i.e. a real program, not a collection of scripts), but still since it’s one grad. student’s creation and not Microshoot’s, it’ll take you some time until you get it working (see also related post from this blog).

And where will that time go? Probably you’ll need to do some *nix hacking to get the thing working (even if it comes with make/configure files, still you might need to set-up some libraries etc). Then add some time on top of that if you have to bring your data in the format appropriate for the software (often case in bioinformatics, with its babel tower of data formats). All-in-all the setup process will take 50% of the time you decided to allocate for trying your data with that software (and that 50% might be in any scale, from a day to weeks, depending on the scale of your endeavors). The remaining 50% will go to the scientific discovery / hypothesis testing part. On the other hand, a -scientist- (don’t want to use quotes cause it might seem I’m being ironic, but still want to emphasize the word) would go for something that works right out of the box, so that he/see can spent 100% of the time testing hypotheses with the data (that’s why people use Spotfire for analysis of microarray data).

The problem that originates here if you are on the 50-50 approach, is that you get to be dead in the middle. Why this is not good ? Well, personally I enjoy building stuff and would opt for spending 100% to develop the thing and forget the hypothesis testing. Or if I didn’t like development, I’d immerse myself into reading all the papers related to the hypothesis I want to test, curate knowledge in my head, and go for 100% there (spend some good bucks too for buying Spotfire).

I know many of you will tell me you use open source software developed by small labs and available for free, hack it and make it do you job. But think about how much time you spend on that. The time spend is often pretty significant, but it’s rather invested in setting it up the software on your machine, or doing a small hack that scratches the surface of what lies beneath the years of development that it took (a student probably) to build the software. Does the investment of time to set it up offer you something in learning ? Learn your system ? Maybe, but once you learn how to setup software on a *nix machine, I think it doesn’t offer anything to repeat it with any program you wanna experiment with.

I’ll close here, but before that I’ll underline my main point for this post: I believe everything you do must give you something in return, something that make you know more and be more experienced after you did it. This may sound like seeing things just black or white, but wouldn’t be better to spend your full time in either building the softare, or transparently using it to do your biological discoveries – so in the end you become a strong developer or a strong scientist ? I see lots of the middle-way in bioinformatics, and maybe that’s because we have the bulk quantity of papers published in the field being of average quality. Do we need segregation of the developers and the scientists ? Do we need journals that publish on bioinformatics software and rigorously review its quality, and journals that publish strong theories developed using large bioinformatic data from scientists that know their specialty ? Do we need scientists that are software developers ? The answer is yours.

I have been experimenting lately a bit, by going “under the hood” in Unix-type of operating systems, specifically Linux (openSuse), NetBSD and OpenBSD… why ? Because it is boring doing the same thing the whole time even if you learn and discover new stuff (see writing PhD thesis and data-mining relational databases).

So what did I learn: apart from the gory details, it boils down to that lots of hardware with still juice left after the years have passed, gets put aside because it becomes slow when using it with out-of-the-box operating systems. I’ll give you two examples for that, one being my 5-year old Powerbook G4 Titanium (800Mhz) which with OSX (and to be exact, not the latest version of OSX) was simply crawling. Yes – you would say, it’s a five year old computer, time to throw it away; but after installing OpenBSD to it, I got fast web browsing with Firefox, email with Thunderbird, pdf and document reading. And to bring a good example of how successful the resurrection was, when I use it to read blogs in Google Reader, or write posts through WordPress’ web interface (both heavyweight javascript), the Powerbook with OpenBSD just flies.

The second example is what I’ve seen recently in a Windows XP machine (well that’s the worst example ever!). The machine is a decent 1GB, Core Duo HP, only about 2 years old and its getting slower and slower. Why ? My suspicion is that with the patches and updates Windows install automatically through the network, it just makes the Window’s kernel a mess, with the accumulating mass of patches hogging away on the system’s resources. So that’s where the “2-year life cycle” of a computer comes from ? Without need to do any preaching to the choir, for those of you who have tasted a piece of Linux, you know that this machine is more than adequate for a production desktop with Linux or any other efficient Unix-type system.

Now the real part of the dark side, concerning my experimentation of compiling customized kernels for openSuse Linux and NetBSD. Basically, kernel compilation is editing a configuration file, which leads to compiling less code and consequentially removing bits and making smaller the core binary (the kernel image) that sits in your computer’s memory, and controls the hardware. Since Linux (and Windows) are intended to be installed on many different brands of machines with different hardware, their kernel comes pre-compiled with support for all of those. But if you don’t use infrared port or bluetooth with your laptop, do you want the bits for controlling this hardware just eating up your RAM, since they come as part of the pre-compiled kernel ? So what you do, is to edit the kernel compile configuration file appropriately, and then re-compile your kernel to shave off those bits and make the kernel image smaller.

My experimentation so far, has told me that compiling custom Linux kernels is a little more of a pain compared to NetBSD. That’s because Linux as a user-centric / supporting everything (joysticks, radio & tv cards etc etc) type of operating system, has a lot more as default in the configuration file of the generic kernel, that need to removed in order to tailor it to your specific hardware. The gain is significant though, in my case about 100MB of more available RAM with the custom-compiled kernel. On the other hand, NetBSD has a much simpler file for configuring your kernel’s compilation, and things you need to remove are RAID, PPP etc support (well you need to leave those if you’re compiling for a data center and not a laptop).

The other thing I observed when comparing openSuse Linux versus NetBSD, is that the latter has much smaller kernel that the former, with pretty much the same hardware support for an ordinary desktop pc. I haven’t drilled down on that yet, but maybe Berkeley BSD’s kernels are better written than the one put together by Linus ? To get down to earth again, I am now left with a Core 2 Duo – 2GB memory laptop, which has 1.7GB of memory free for my applications after 0.3GB are eaten by openSuse (did I mention that Windows Vista in the same machine eats 1.2GB by itself ?!!?). NetBSD runs only with only taking about 50MB (!) of RAM, and currently is installed on my old PIII with 256MB memory, making it a very happy computer….

Under Linux, I am running 4 virtual desktops (well not under KDE desktop manager which is a memory hog, but under Fluxbox window manager), with a couple of Firefox windows – each with a bunch of tabs – open in the first desktop, 4-5 x-terminals in the second, OpenOffice Impress presentation on the third, and Cytoscape with a gene interaction network (in the thousands of nodes) open on the fourth. The amazing thing is that I still have memory left to perform operations in the network in Cytoscape, and the machine never swaps data in the hard-disk. So I see this laptop staying with me for some more years to come, and when apps need more memory (i.e Firefox 7.0) or my data crunching needs grow bigger, I’ll just thin down the operating system (switch from Linux to NetBSD).

To conclude, it feels good to hack on the dark side I think I will stay here for a little bit, the next project being to transfer to the Core 2 Duo laptop, the Postgres database that hosts all my data-mining projects and runs on a Quad Xeon server. It will be an interesting thing to see whether a 1000$ present-day laptop, does better than a 4-year old / 10,000$ server…

In the meanwhile, don’t throw away your old computer please !

So I’ve been reading a bit lately around the web about Open Innovation, an interest spanned out of some lounging with the Wikinomics book by Don Tapscott. A prime example of this model of new economy is Innocentive, with outsourcing of brain power and seek for diverse expertise (problem solvers participating in Innocentive get cash). In my mind, the concept encompasses everything from open source operating systems (see Redhat, Novell to see how this is model of economy makes good bussiness), to open source drug development, and the whole breaking from the hierachical-closed company management systems.

So I was wandering whether this is a new thing, or it has been always happening. I got a hint that makes me believe the latter, from reading the Free as in Freedom book on Richard M. Stallman (FSF). Without any nervous looks around, and without cold sweat dripping down my forehead for violating any copyrights, here’s an excerpt from the first chapter of the book :

“….why companies like Xerox made it a policy to donate their machines and software programs to places where hackers typically congregated. If hackers improved the software, companies could borrow back the improvements, incorporating them into update versions for the commercial marketplace. In corporate terms, hackers were a leveragable community asset, an auxiliary research-and-development division available at minimal cost.”

It seems to me that it has been happening before, when the term open source did not exist, but the difference today is the upscaling of the phenomenon in the internet era. Plus cheap computers and network bandwidth for everyone. And I am wondering if we are in a transition phase, where big corporate organizations try to absorb / accept / adapt to the new model of Open Innovation, and share some of their IP, operations, seek of problem solvers with the commons, ripping the benefits of collective intelligence in return.

My speculation as I sit here writing this post, is that more and more companies will go for it, and my secret hope is for pharma, where I strongly believe collective intelligence will drive drug discovery as it drove all open source operating systems over the past 10 years.

For those of you liking to know more about Open Innovation, wanting to think how it will be like over the next 10 years, but are bored of reading blog posts about it and want to understand it while doing something more fun, I suggest this sci-fi book by Charles Stross.

(… or how to benefit from other people’s curation of information across the web)

So I have been brainstorming about how to write up a short article for the Biogang’s wikipage, with a topic for online collaborative communities in bioinformatics. I’m contemplating to gather a collection of examples across the web and present a brief description of each, but before going even further I thought I should have a look in my own online back-yard. So here came the present post, on how you can find information curated through human intelligence and not PageRank, gathered from a variety of sources and aggregated within a single focal website, a.k.a. Friendfeed (FF).

People over at FF, capture their daily web activity and stream of consioussness through a variety of ways, by sharing interesting articles from Google Reader, bookmarking at del.icio.us, posting short messages on Twitter and a variety of other Web 2.0 sites, as the two following pictures show. Some of these people I find interesting, and I subscribe to the records of their daily web activity…

Now the question that comes to mind, is that all these people have a diverse array of daily jobs, personal interests, expertise in topics. Most of our opinions, ideas, new knowledge on different topics comes from the web nowdays (at least for my FF peers, I find it difficult to imagine them curling up with hardbound volumes from the Annual Proceedings of some scientific Society). As these people move through the linked information space of the world’s largest (un-indexed) encyclopedia, they gather information that answer their questions, open a new window of interest, or simply amuse them.

Can I easily use their collection of answers, so that I can answer questions of my own on similar topics ? Yes, and that comes from searching through the stream of consciousness of the people I follow over at FF: here’s an example of what was curated over the web by my peers at FF, on a topic that is related to this blog:

Hmmm… Pierre has exactly posted an RSS link for how Web 2.0 can help with the information explosion, I’ll just click and read it, sure thing it’ll save me some Googling and spending time to search through the search engine’s results…

Anybody not convinced yet about Friendfeed ?

P.S. I will follow up with a post discussing the options for online collaboration, discussion over at FF.