Scaling the Manhattan Project

A couple of months ago someone I love was diagnosed with a brain tumour. Alongside with the very personal aspects of this, it’s got me thinking about the rate of progress being made at solving the hardest problems humanity faces, and what could be done to accelerate that.

I hope the work we are doing at Songkick is going to really matter. A lot of our vision for the future of live music is not visible yet, but I hope that by the time we are done we will make a meaningful contribution to music. I believe that increasing access to art and improving the general health of cultural industries is valuable work. I hope one day Songkick’s contribution will start to look like some of the culture centric companies I admire most - like IMDB, Kickstarter, Pitchfork.

But with this news about my loved one, I’ve been thinking a lot about paths not taken, and specifically the work I was doing during my Masters in Machine Learning. My professor had been working closely with a local hospital to build a data set of cancer biopsy images, which had been classified by the hospital’s pathology team into various grades. My project was to take this set of ‘training data’ and design a machine that could learn to classify biopsy images. Were this to be successful it could be applied as a pre-screening filter, saving time for overloaded oncology teams. There would have been another benefit – automation of classification might have helped to make the process of doing biopsies more scalable, enabling more people to have biopsy’s taken.

I made some progress, and developed a classifier that went some of the way to being able to pre-filter biopsy data. However, I didn’t take it even close to something that could be used in production by a hospital, and after my Masters did not continue working in the field. Over the past few months I have been trying to connect the news about my loved one’s diagnosis with my choice to leave the field. Asking myself why I didn’t keep contributing. Asking myself if it would have made any difference if I did. Asking myself what the most useful way for me to help is today.

Looking backward at my decision to leave academia, I found the sheer scale and complexity of the overall problem driving my research (cancer) unbelievably overwhelming. It felt impossible to contextualise how important or valuable what I was doing was in the space of what was being developed in the field more broadly. For example I found out late in my masters that some technology developed by the military for detecting enemy encampments in the desert was being applied to the same problem of medical image classification with great success and was way ahead of what I had done. There was no easy way to make a connection between my day to day challenges of tweaking machine learning algorithms and the bigger problem that was situated within. I work best when I can see the big picture and derive enormous motivation from affirming that the work I am doing has a high chance of being worthwhile.

A couple of months ago I read a brilliant piece by my friend Richard Price, co-founder of Academia.edu. Richard has been working extremely hard on the problem of better networking academics in the hope that this speeds up the pace of scientific discovery. It hasn’t been easy and but he has been working relentlessly on it since 2007 despite a general level of apathy from VCs for the space. The kind of entrepreneur who is on a mission.

Reading his essay was the first time I connected networked web services to the questions I have been asking myself since my loved one’s diagnosis. And it makes me think that there is something missing from the web that could make a difference.

4 themes that seem connected to me:

1. the step change in aggregate knowledge that comes from allowing anyone with access to the web to contribute to the sum of knowledge on a topic. Wikipedia is the obvious example of a networked crowd working together to produce a richer version of encyclopedic knowledge, but there are many others from open source software to the Goldcorp crowdsourced gold-mining example

2. the increasing ability of people to use social platforms to coalesce around issues they are passionate about. From Arab Spring to Occupy Wall Street

3. the increasing ‘siloization’ of research due to entrenched corporate interests, academic publication cycles and paper availability. Sergey Brin’s project to unite 

4. the increasing cross-departmental nature of scientific progress suggesting that the network of potential contributors around any given problem is expanding

I think there might be an opportunity to create a new internet platform at the scale of Wikipedia that enables humanity to collectively solve the biggest problems it faces. The internet equivalent of the Manhattan Project. For problems spanning Alzheimers to our energy issues. The characteristics of this platform might be:

- Focused on problems unsolved by humankind. Rather that how Wikipedia/Quora/Stack Overflow focus on aggregate knowledge on a topic, the service would focus on solving unsolved problems. A shift in focus from knowledge to progress

- A structure that enables huge problems to be broken down into sub-problems with the contribution of solving those sub-problems (build a classifier for breast cancer biopsy images) clearly related to the overall parent problem/s (treatment of breast cancer)

- Top down and bottom up. The motivation derived from top down problems ‘put a man on the moon’ cannot be underestimated, but we will also find solutions to problems that have not yet been contextualized against something else. This emergent reconciliation of solutions from different areas of scientific progress may end up being one of the most valuable features of the system

- A structure that enables anyone to contribute. From hobbiest hackers to amateur geologists. Similar to Wikipedia and others a key part of the thesis is the broader contribution the crowd can make

- A structure that recognizes and rewards individual contributions. It is critical in my mind that the best minds of our generation feel incredibly motivated to participate in solving components of these problems. As my friend Jeff Hammerbacher, once of Facebook, said “The best minds of my generation are thinking about how to make people click ads”. I see this needing to be a for profit entity that distributes both global recognition/reputation and financial reward to contributors.

As with web platforms in general, this would likely be focused on a niche to start. This first ‘mission’ would likely be characterized by 1. High importance to humanity, such that a solution would dramatically validate the platform, and that people could be mobilized around it more broadly e.g. Alzheimer's 2. Being in an area where a high percentage of potential (amateur) contributors have been largely marginalized.

I would greatly value being part of the distributed team of people working to help improve the prognosis of my loved one. I have some evenings and weekends. I am skilled in various areas that could be of help and I am incredibly motivated. I could help indirectly by raising money for a charity or research organisation working in the area, but I would love to contribute more directly if I could.

Is anyone making any progress on something like this? 

Separately if anyone knows someone working at the cutting edge of research into glioblastomas I would be very grateful to speak with you.