World Building is Entirely Unnecessary.
What justifies the time is writing with consistency. If you have a map with terrains, towns, plausible climates and climatic cycles, etc, then your writing will be consistent as your characters journey through it; you won't accidentally write in logically impossible distances or trip timing. Using Earth as an example, you won't write that the trip from Las Vegas to New York City took 8 hours to drive. It is 2529 miles, they'd have to drive 317 miles an hour! Of course you wouldn't make that mistake on EARTH, even if you aren't great at division you would immediately sense these are further apart than 8 hours, and consult a map and calculator.
But when you are writing and inventing everything yourself, it can be easy to forget what you told the reader about places W, X, Y, and Z, and accidentally create a physical paradox like the above. It takes one crew a day to walk the path W->X->Y, stopping at X on the way, because that was convenient to your plot at the time. Then later in the book you send them from Z->Y->X for supplies, and it takes them FIVE days to walk from Y to X. WTF? They walked from X->Y in less than ONE day the first time!
You might have character Joe talking about being born in W, and growing up with snow, ice, and bitterly cold winters. Then three months of writing and ten chapters later, you forget about that. Joe and your crew end up in W during winter, and you write it as pretty pleasant and mild. But your reader just read about the winters in W a few hours ago, goes back to read that again, and wonders a) Why Joe and nobody else isn't saying a damn thing about this pleasant winter, and b) Was Joe lying?
In such a case you have broken the reader's immersion, your world doesn't make sense because your writing is glaringly inconsistent.
But I said world building is entirely unnecessary!
You don't have to do this kind of work up front; you can take notes of what you say and build a map as you write along, and if necessary revise your story as you go. If you know some of the elements of world building (what makes deserts, or rain forests, mountain ranges, lakes, etc) then you won't make obvious mistakes, but all you really need to avoid is glaring errors that won't stand up to scrutiny. You don't have to talk about the geology of what makes a seashore cliff; you can just plug in a seashore cliff. Just Google "seashore cliff" and you will find a few dozen of them, pick one that best fits your story as a reference for writing your own. Describe actual nature and you won't create any glaring errors.
Of course world building can also include fantasy religions, cultures, rituals, animals, magic systems, and so on. Some of these are for entertainment value, others are developed in order to maintain consistency, and impose limitations on what can be done. Not only in magic, but perhaps politically and culturally as well. In many fantasy and scifi worlds the culture allows one character to kill another in front of witnesses and walk away, never to be pursued or even charged; there just is no law enforcement in such worlds other than vigilantism.
In other worlds more like our own, like the Mr. Robot series, law enforcement and how it works is central to the story, the hackers are breaking the law and in constant fear and danger of being caught.
World building can be an entertainment in and of itself, but (IMO) it is just a setting, not a story.
To me, I would rather write a story. I am a discovery writer, so I do my world-building on the fly, with a map I sketch as my story demands new places, towns, or features like rivers, oceans, forests, deserts, etc.
I also do several passes through my finished work, and along the way ensure my elements are consistent and there are no glaring errors.
The advantage of world building up front is your story maintains consistency and you don't have to invent settings on the fly. However, you can do these things on the fly without ever sitting down to invent your world. If your story is character-driven, like mine, the settings are not necessarily a big part of the story; and (for me) the main thing to keep straight is distances, travel times, and at times for the physicality (traveling or fighting) and dress of characters, whether it is freezing, sweltering, or fair weather, open terrain or forested, watered or dry, etc.
If setting is a major "character" of your story and often plays the villain (heroes versus nature, or culture, or politics) or has entertainment value (tourism for the reader, by terrain and/or cultures) then it should probably be planned and made consistent.
The time justification is in whether it is more efficient to design the setting and then write a story in it, or to write your story and design the setting on the fly, keeping it consistent with notes and sketches. That all depends on how important the world is in your story; and for many stories that focus on the emotions of characters (a relationship, falling in love, coming of age, dealing with loss, etc) the setting is not terribly important.
