Instead of making gunpowder's ingredients rare like @Xavon_Wrentaile suggested, you can make gunpowder harder to make and store.
If gunpowder is extremely volatile, to the point where just shaking it enough causes it to go off, gunpowder will be too dangerous to manufacture. It will constantly be exploding, destroying factories and driving up the price.
If it is very volatile, gunpowder will be way too dangerous to carry around. Even if the actual amount of it is small and the consequence of ignition is small, in combat this will be devastating:
Say you come across a swordsman who you end up angering. Being very fast, you run out of range of the sword and turn around to aim with your rifle. When loading the gunpowder into the barrel (we are talking old times here), you accidentally scrape the metal gunpowder canister against the metal barrel of the rifle. The tiny spark ignites all the gunpowder. While you were unharmed due to the small amount of gunpowder, you are now out of gunpowder. At this point, a sword or bow would have fared better against the swordsman.
Making gunpowder more volatile is pretty easy [citation needed]. Your first option is that gunpowder is volatile because it is a magical property. However, that is not the most realistic solution. By adding more Oxygen to the atmosphere, the ignition will happen more easily and the explosion will be more explode-y. Since we are talking about Napoleonic-era people, they have no idea how gasses work. However, many gunpowder-makers have found that by capturing air bubbling from the swamp nearby (methane), and filling all of the gunpowder barrels with this, explosions happen less often. Your people have no idea how gasses work, so they don't realize that by adding methane, they are replacing the oxygen with an explosive gas. While explosions will happen less often, they will be much more violent when they do. That will further increase the cost.
Bonus: By adding in the swamp gasses, they also introduce fowl-smelling compounds into the gunpowder mixture. The charcoal filters these compounds out, and when fired, the guns release a giant cloud of fowl-smelling smoke. Why would anyone want to use such a disgusting weapon anyway?
Edit:
Because of the magic tag, I can handwave gunpowder's volatility as being a magical property. However, I want the cause to be more realistic, so I justified its increased volatility as being a result of higher oxygen levels. If you think increased oxygen levels are not feasible for some reason, I will be happy to discuss it in the comments or chat. However, do not rant or complain to me about my use of the magic tag! The poster included that tag for a reason. If you have a people with the question, complain to the asker. If you have a problem with the magic tag, go complain on meta. If you have a problem with Worldbuilding, you shouldn't be on the site. And finally, if you have a problem with Stack Exchange, there is nothing you can do