I mostly do science for the fun of it. I wake up and my brain chooses something fun to do in a way that feels random. This is perfectly valid and something I encourage. Indeed, a lot of great discoveries have happened this way!
But sometimes I have a lot of fun things to try, and I want to optimise my scientific choices. If you have experiment ideas that seem equally promising, how do you select which ones to start with?
One way might be:
List out all of your experiments.
Rank them on different factors.
For each experiment, average the rankings.
Start with the ones with the highest average ranking.
The initial guess is unlikely to be right, but it's better than guessing. A few months ago, it would have been unreasonable to do this because of the work it would take to rank many factors across experiments. But now we have LLMs. In my experience, they have at least been able to estimate costs well.
Here is a preliminary prototype of what I made to estimate the average ranking of each choice.
I fed the following factors for Claude to rank. I got it to rank interestingness from what I’ve written about before.
Cost
Most independent scientists are self-funded, myself included. Probably less than 5% of us with self-funded home labs spend more than $1000 a month. Most of the community is constrained by cost. Even for those who aren't, it is worth getting bang for your buck. Even though hardware costs are decreasing, the cost of living and real estate are increasing. Therefore, personal savings are useful to have in a world with intense debate around LLMs and jobs. All the more reason to keep costs low.
This gives us constraints to work with, since indeps have more reason than other labs to minimise cost. But constraints often give rise to creativity! Again, this may give us an edge.
Feasibility
An experiment is less feasible if it is long and complex to do. Oftentimes this applies to biological experiments because of contamination risk (although this is something I am trying to solve). If two experiments are equally interesting, then shouldn't you want to do the simpler experiment? A long and complex experiment is likely to give more uncertainties and errors than a simpler one. Uncertainties are not necessarily bad - I have talked about type B uncertainty in a previous post - but the experimenter needs to put effort into communicating both measurement and subjective uncertainties.
Interestingness
Interestingness can be defined however you'd like. This factor is likely to be the only one where an LLM can't really help, and you should rank based on your feelings. Alternatively, you can ask LLMs to proxy you from your previous writings. For me, things are interesting when there is beautiful mathematics involved. For example, the dripping water experiment uses methods from chaos theory.
Experimental uncertainty
When taking fine measurements, or using complex sensors, it is likely that there is going to be uncertainty involved. There is likely to be measurement uncertainty, such as from the calibration of your digital calliper or your thermal probes. There are also subjective uncertainties that are less obvious. For example, the well-mixedness assumption in a thermal experiment may or may not hold true. There may be turbulent effects in your fluids experiment that affect what you want to study. There may be unknown unknowns that you can't even quantify, but that you suspect. Given your equipment, it is prudent to minimise experimental uncertainty where you can.
Moral expected value
It would be bad if your science could be used for bad things, so experimenting with things that are obviously bad is bad. I did work on DIY biosecurity measures because I didn't think it would be harmful. There was a chance that the work might even have been useful.
That being said, most science is dual-purpose and initially feels morally neutral. In those cases, it is unclear what the long-term moral outcomes of those scientific actions are. Here, the experimenter is in a situation of moral uncertainty, which is a topic I don't think is written about enough. We usually talk about how to act in cases where we know what the right and wrong things are. But the decision theory for cases where right and wrong are unclear, or where moral frameworks are not obvious, is not that developed.
Contrarian-ness
Experiments that have the potential to disprove a well-known dogma or paradigm always feel strong. I have more conviction that if you’re going to spend time on stuff, then it’s probably worth trying to find something that would up-end literature in some way. Kuhn writes about this in The Structure of Scientific Revolutions.