Just a couple of small notes to round out your thinking to help you either drop the idea or to let you properly represent at least the not-the-fissile-material-itself aspects. Or to help you know what to obfuscate or have major hand waving written in for.
First, handling nuclear materials. As noted in one answer, the phrase disposable workers
is a BIG BIG thing. Take 35 guesses at what killed Madame Curie, the person most associated with early investigations into radioactive materials. Ah, just need 1 guess? And bear in mind she only used small samples. She didn't process tons a day for a career. Since your workers would, without changes to "history" on a fairly large scale, so I'm guessing not part of your desired writing plan... since they would do that, they would maybe have very short careers, longer in the mines, yes, but shortening at each step along the way. Since the "along the way" steps get more and more skilled, you'd see the early/swift massive breakdown in health, and outright deaths, definitively affecting your manufacturing silo with the quickest and most debilitating occurring with precisely the hardest workers to replace. Not really a "sentence the thieves to machining materials to a one in one hundred thousandth of an inch tolerance" kind of proposition here, so you'd need to figure a way around that...
On the other hand, but related, we go on and on about the massive economic cost and how it could/WOULD beggar nations. This both is and is NOT a direct concern. First, in the US: the bomb was actually only the second costliest weapon program for us during WWII. Second, not first. And we did both, in spades.
Germany had a program that languished more due to leadership being of the "Keystone Cop" variety. (Like that or not "Nazis-were-gods" folks... not saying you admired them, not AT ALL, but there was and is, a perception that these stumblebums were somehow amazingly on their game when they were really sitting on top of a very competent society and squandered, utterly, most of the actual advantages that gave them rather than capitalizing on them... successes came in spite of them, not because of them.) Add to the stumblebum aspect the utter waste of millions of talented (and stupid, lazy, and very not talented) people, not just in the death camps, or as very low-level slaves before declining to uselessness (to the Nazis), but also tens of millions more, the Slavs in, say, Ukraine, who'd've done almost anything to see Papa Joe dead and gone. (To get a scale of the waste, Alfried von Krupp und Etc. was the largest slaveholder in history owning/operating more than 10,000,000 slaves. That's from his Nuremberg trial. And Germany had many more. But you don't build atomic bombs out of an economy that drove itself in that ruinous direction rather than maintaining itself. But to the Nazi leadership and its underlings fighting to rise, or just to stay alive another day, everything the sun touched was their "commons" to rape and ruin because none of them owned it.)
Even with all that against them, what actually made their bomb project not succeed, more than anything, was that since the leadership had no real concept of it becoming a real thing. (One bomb to kill 150,000 of Papa Joe's soldiers and a few thousand tanks with one blast??? How could that actually be a real thing? There weren't even Norse gods or witchcraft involved so...) Without constant demand from the top, the venal, "kill to gain position" government and army chain of command ensured it would always be something to fail in and end up on a meat hook next to sides of beef (Yes, that was a thing for the Nazis.) for days while someone still had an interest in torturing you. Not really a program with prospects. So, no Bomb.
The British could have run a successful program, and did make important/critical contributions to the American project, people-wise and tech-wise. Why duplicate efforts? Seriously. And with the American effort absolutely impossible for any Axis bombing attack, to say nothing of on the ground military attacks, to reach, it was better to do things in the US. (The Japanese balloon carrying bombs that travelled 7,000+ miles were NOT directed at anything more definable than "the west coast of America" so... not directed at Hanford.)
The USSR also could have, but had no real reason to as conventional efforts were working very nicely and the US was supplying an extraordinary amount of material so their factories could concentrate on the fun things, tanks and rifles and bullets. So not much impetus: NOT "not much ability"...
So, back to the US effort. I'll just bring up one thing to give you the scale of it. It is the SCALE OF IT that matters to your writing. The scale will play out upon whatever economy and population you have, and that will give you cost. Not that cost matters: it is the resources in people mostly, and their skill sets that matters here as neither they, nor their skills, would still be in your economy. If it takes every blacksmith in existence in the nation, then who makes the rifles and bullets and so on?
You have surely heard of Hanford, the eventual factory of the bomb? Quiet, unpopulated western area... In the middle of 1944, it had 45,000 construction workers alone. Not fissile material workers, CONSTRUCTION workers. Think of that. 0 population to 45,000 construction workers, perhaps a soldier or two, actual program workers, oh, and ALL the support people backing them up three meals a day and clothing and so on.
And Hanford was just Hanford. It did not include Oak Ridge or any place in Arizona and New Mexico. Just Hanford.
How would that massive scale work out in doability in your world? Farming done with horses and men, not tractors and combines. Every machinist and so on taken for this project alone. No rifles or bullets except those produced by people who were homeless before your wars. Those people and all their support NOT in your armies themselves, or in support of them. I've read that the US had 16,000,000 men in uniform at the end of WWII out of something like 170-190,000,000 total population, yet no more than something in the 500,000 man range in actual combat, or close to actual combat. Huge armies take even larger support efforts, and could your societies to that and STILL support such a massive drain of people and talent and all their support?
All that is admittedly not likely to be thought of by the average reader. So maybe not a practical problem for you, the author. But... maybe it is.
By the way, fuel oil air bombs, "thermobaric" bombs, could have existed in 1850 with the skills and tech they had. They could have been delivered via balloon, PERHAPS, if someone developed plastics, lightweight plastics that is, much earlier than we did. If oil and its related products (kerosene, plastics) had a one or so generation headstart, maybe two... To give you an idea about balloons, don't think those pretty, pretty fairgrounds things. Think Roswell, balloons launched to monitor the atmosphere for nuclear explosions on the other side of the world, balloons that were 1,000 feet tall once at sufficiently low atmospheric pressure and commensurately sized around, large cylindrical balloons. Possible perhaps in your world. Those would have perhaps been able to carry significant kerosene and some machinery to disperse (mist) it and let that mist fall toward earth, then be ignited by the balloon suddenly flaring bindings, dropping lit materials down into the mist falling above an army or city...
We apparently have achieved 15 kiloton explosions with thermobaric weapons. 75% of the Hiroshima bomb's usual stated blast strength. The Russians/Soviets made one that achieved 39.9 kilotons, or twice the Hiroshima bomb's blast strength, though to be sure, not out of 1850's available kerosene... Admittedly, both from a far more technically advanced economy, but... good chance that's just because that's just the economy in place when people got to thinking on the subject, right? So why not that thinking in 1850-ish?
I'd suggest you go that route. I myself think of it in a fantasy setting. A boy who one day rides a dragon in such a world's wars remembers a frontier barfight in which a lantern is shattered and the fuel spread out into the air in the bar tent as someone swings it at people as a weapon itself, then the mist portion reach a fireplace and the blast killing the "bad guys" but only throwing the good guys, including the young boy, a hundred feet, say, (or twenty, whatever feels decent at the time, eh?) ripping through the canvas, fortunately not splattering any of them on a tent pole or boulder (outside), then the slamming punches of flame following out through those rips in the tent before the general blast vaporizes the tent and all concerned still in its immediate area. That boy, now a man, leading the dragon arm of the army he is in, having large more or less waterproof sacks made sized to the dragons who'd carry them, lantern fuel collected from the besieged capital (large) city, then the force (oh, "wing"... why not, right?) taking flight one night and wax-covered holes in the sacks eaten away by the kerosene, the contents sieving out into the air, falling a thousand feet breaking into a mist to a reasonably large extent though maybe most lumping back together into more of a rain, the riders, task done, flying off wondering what it was all for, then some noticing this man directing his dragon into the falling rain of kerosene above the center of the enemy camp and having his dragon breath the best blast of flame he's ever breathed... Death on a scale no such medieval society would have spent much time imagining. And sacrifice at the very center.
(I picture it via a young magic user as well, either way, and I hate dragon stories, so I spend more time on that avenue, mentally.)
Did the little bit above seem reasonable? Not Hemingway brilliantly written there, just... reasonable? In a medieval tech-level society without examples like successful atomic and fusion bombs to inspire them to find other ways of achieving something massive?
If it did, then why not in your world of 1850's tech instead of nuclear weapons? It just needs someone to notice that a mist of kerosene is incredibly more explosive than one would think at first. And that's not too much of a stretch as powders would easily have been a very present thing in such a society. Grain dust in huge storage silos, gunpowder in its making and storage, even powdered metal could exist as a noticeable thing in your world, and any powder from flour to gunpowder and beyond either direction can explode. Coal dust in mines (Benxi, Liaoning: 1,500 dead). Look up grain explosions. Flour would definitely make a mist as it fell, and not gather into rain-like "droplets."
I'm saying there are alternatives that can still give you horrendous and massive explosions to work with. Beats hand waving any day!
But if you need it to be nuclear, not much help.