"Magic" is not declared as such, but there is an energy that comes from every object, and even some things that are not physical, like a gust of wind. Some call it the entity's spirit, or even its soul. Mages can feel them, but they can't be seen.
Mages are people who can connect their own energy, their own soul, to the soul of others. When they do this they can influence the entity's soul, especially if their soul is much "firmer." Put another way, mages need to be very strong-willed and have great sense of self or they may find their own soul being manipulated by the other.
However, just because you can change something's soul doesn't mean you can change what it is. In fact, you can't directly influence much of anything. A fire, for instance. Its soul might be one of consuming, greed, and of passion. A mage may connect his soul and share his compassion, contentment, and patience. The outsider may see the fire slow, and maybe even start shrinking if it is able too. To the outsider, it may look like the mage put out the fire eventually. In reality, if the wind were too strong, the fire still may not have a choice and it may still grow and consume everything.
The danger of connecting to a soul that is simply too strong, like that of Earth itself, is great. The mages own soul is put in danger. But surely the Earth's soul has some qualities that would actually be helpful to a mage? Exactly what kind of danger might exist from connecting to a soul that is simply much stronger?
The problem I have is that you shouldn't be able to make yourself "better" (in society's, or your own, view) by just lightly absorbing someone/something else's qualities. I'm having trouble coming up with why your own individuality is so important that it is simply always a bad idea to let anything change yourself.
The goal is to provide a solution that "makes sense" and would be enough of a danger where mages would not try it unless they are either very stupid or incredibly desperate. Simply "losing your soul" has implications which I don't want - though death itself may occur.