The Beatles’ song Yesterday was written in what psychologists refer to as the “hypnagogic state”. This is the twilight zone between sleep and wakefulness, when we drowsily linger in a semi-conscious state, experiencing vivid mental images and sounds.

Waking up one morning in early 1965, Paul McCartney became aware of a long complex melody playing inside his head. He jumped straight out of bed, sat down at his piano and picked out the melody on the keys. He quickly found the chords to go with the melody and created some holding phrases (as songwriters call them, before they write proper lyrics) to fit the melody.

Finding it difficult to believe that such a beautiful melody could emerge spontaneously, McCartney suspected that he was subconsciously plagiarising another composition. As he recalled: “For about a month I went round to people in the music business and asked them whether they had ever heard it before … I thought if no one claimed after a few weeks, then I could have it.” But it turned out to be original.

Many great discoveries and inventions have emerged from the hypnagogic state. The physicist Niels Bohr effectively won the Noble prize while semi-conscious. Drifting off to sleep, he dreamt he saw the nucleus of the atom, with the electrons spinning around it, just like the solar system with the sun and planets – and in this way he “discovered” the structure of the atom.

The sweet spot

Research has shown that the hypnagogic state is a creative “sweet spot.” For example, in a 2021 study, participants in a hypnagogic state were three times more likely to discover the “hidden rule” that could solve a mathematical problem.

Psychologists associate creativity with qualities such as openness to experience and cognitive flexibility. Others have suggested that creativity arises from co-ordination between the cognitive control network of the brain (which deals with planning and problem solving) and the default mode network (which is associated with daydreaming and mind-wandering).

However, in my view, one of the most important theories of creativity is one of the oldest, put forward by the early British psychologist Frederic Myers in 1881. According to Myers, ideas and insights come as a sudden “uprush” from a subliminal mind.

As Myers saw it, our conscious mind is just a small segment of our overall mind, including not only what Sigmund Freud called the unconscious, but also wider and higher levels of consciousness. Ideas may gestate unconsciously for a long time before they emerge into conscious awareness.

This is why it often feels as if ideas come from beyond the mind, as if they are gifted to us. They can come from beyond our conscious mind.

The importance of relaxation

The hypnagogic state is so creative because, as we hover between sleep and wakefulness, the conscious mind is barely active. For a brief period, our mental boundaries are permeable, and there is a chance creative insights and ideas will flow through from the subliminal mind.

In a more general sense, this is why creativity is often associated with relaxation and idleness. When we relax, our conscious minds are usually less active. Often, when we are busy, our minds are full of chattering thoughts, so there is no space for creative insights to flow through.

This is also why meditation is strongly associated with creativity. Research shows that meditation promotes general creative qualities such as openness to experience and cognitive flexibility.

But perhaps even more importantly, meditation quietens and softens the conscious mind, so that we’re more liable to receive inspiration from beyond it. As I point out in my book The Leap, this is why there is a strong connection between spiritual awakening and creativity.

Nurturing the hypnogogic state

Research has found that around 80% of people have experienced the hypnagogic state, and that around a quarter of the population experience it regularly. It is slightly more common in women than men.

It is most likely to occur at the onset of sleep, but can also occur on waking up, or during the day if we become drowsy and zone out of normal consciousness.

Can we use the hypnagogic state to enhance our creativity? It’s certainly possible to linger in the hypnagogic state, as you probably know from Sunday morning lie-ins.

However, one of the difficulties is capturing the ideas that arise. In our drowsiness, we may not feel the impulse to record of our ideas. It’s tempting to tell ourselves before falling back to sleep, “This is such a good idea that it will definitely stick in my mind.” But when we wake up some time later, the idea is gone forever.

However, through mental training, there is no reason why we can’t build up a habit of recording our hypnagogic ideas. The best practice is to keep a pen and paper right on a bedside table. Or for a more contemporary variant, keep your phone beside the bed, with the recording app open.

In fact, this is a practice that Paul McCartney has always followed. He even trained himself to write in the dark for this purpose.

We can also use a technique of “conscious napping” to generate ideas. Whenever the great inventor Thomas Edison was stuck for a solution or new idea, he would allow himself to drift into unconsciousness, while holding a metal ball. As he fell asleep, the ball would clatter to the ground and wake him, when he would often find that a new insight had emerged.

More generally, we should use idleness as a way of cultivating creativity. Don’t think of napping or relaxing as a waste of time. Far from being unproductive, they may lead to the most inspired ideas and insights of our lives.

This article features references to books that have been included for editorial reasons, and may contain links to bookshop.org. If you click on one of the links and go on to buy something from bookshop.org The Conversation UK may earn a commission.