The New Scientist has an article on seven unsolved dilemmas in the field of physics today. Just a hunch here, but for 3 of the 7, the reason these questions keep physicists up at night is that they are metaphysical questions that are beyond the scope of physics. It’s like the story about having a nice new hammer: everything starts to look like a nail.
For the first one, Why This Universe?, the question is why do we have these physical laws, and not some other set? It does in fact seem that other laws are imaginable, and this purportedly conflicts with the scientific paradigm. However, what allowed science to emerge in the West is the idea that the Universe is not necessary. If the physical world were a place of logical necessity, we would not need science in the modern sense because we could all be armchair physicists learning the secrets of the universe from our living rooms.
In fact what we see is that we must observe and experiment to narrow down the range of possibilities. The reason we have to do this is the physical world is contingent, it could be otherwise. Logic is never enough to determine the material world. To put this the modern way, the hypothesis is underdetermined by the data no matter how much data there is.
Some of the other questions touch on the limits of physical science, such as the nature of matter, and what is complexity, that are not really going to be solved by experiments. Experiments will certainly help, but the question is metaphysical by nature, and one cannot expect physical science to come up with a good answer. Physicists may be the ones who do this work, but strictly speaking, they will not be doing physics.
I can’t really blame physicists for being uninterested in philosophy, because modern philosophy is so bad. This is not to say that XXth century philosophy did not produce some really good arguments. The one about underdetermination of hypotheses is a characteristically XXth century argument that could not have been adequately formulated earlier. But the project as a whole is lost in the fen at the moment. Nonetheless, even a minimum of real philosophical sophistication would help these physicists sleep better at night.
Cross-posted to With Both Hands