Interesting! It doesn’t to me. I don’t feel that meaning in the context of sentence that is contrasting ‘dark’ with ‘friendly’.

The only other time I can think of ‘sitting in a cave’ being used is by my parents before they would turn on the room light when they found me playing video games in the dark. Another context that is just about light and not size.

But I could believe that some people associate cave with small.