It has been suspected for some time that atmospheres of tidally locked worlds can undergo superrotation, which can do a surprisingly good job of evening out temperatures all around the world even with surface pressures as low as 100 millibars (see Simulations of the Atmospheres of Synchronously Rotating Terrestrial Planets Orbiting M Dwarfs: Conditions for Atmospheric Collapse and the Implications for Habitability from way back in 1997 for example). Atmospheres with a lot of CO2 (say, 1000 millibars or more) can even sustain liquid water on the dark side.
I'm also reasonably certain that surface windspeeds need not be nearly as bad as you might have heard, though I'm having difficulty tracking down some good references for that so take my word for it or not, as you like.
Leaving aside surface climate, there's an interesting page I bumped in to, Mercury Temperature vs. Latitude, Depth and Time, which includes a load of math which might be relevant to your interests. Mercury of course isn't locked into a 1:1 spin-orbit resonance, and it doesn't have an atmosphere so it has a better scope for losing heat, but if your world had a sufficiently thin atmosphere, or one that was very low on greehouse gases, you might still be able to make use of the same ideas. The author calculated that there are rings around the north and south of Mercury where, at a depth of less than a meter, the ambient temperature is similar to Earth's surface.
Such shapes will be quite different on your world especially if atmospheric heat transport is as effective as theorists suggests it could be. There's almost certainly a lot of potentially habitable subsurface real estate, if only you can explain how a natural environment could form there.
One problem you might consider is where did these caves come from? Caves on Earth tend to form in limestone, and formation of thick and widespread layers of limestone requires a history that includes oceans of water and a carbon cycle of some kind, which might in turn require volcanism. And that in turn brings up questions of why the sun-side of a formerly quite habitable and quite watery world is now hot... if it were a greenhouse effect, then that means a thick atmosphere and a temperature and probably oceanic night-side. After you've provided suitable materials, how did all these caves form in the first place? Changing sea levels might explain it, but that could leave you quite high and dry caves.
Lava tubes are another good option (and don't require a biological and geological history that produces limestone) but the total volume they can provide you looks more like an island habitat than a country, though that might not matter to you if you're happy to have very small, very scattered life-bearing regions in a dead, dry expanse of planet.
In either case, if enough light gets in to the cave for enough photosynthesizers to grow to feed an ecosystem, the hot dry winds outside are likely to dessicate your water sources, and the growing area might be too small to feed more than a food web of small invertebrates.