If the scarcity of gold and the labour and time required to dig more of it out of the earth is what makes for the magical quantitative control of money, then why the more commonplace silver as gold's magical toady - the two making "sound money" and "hard money"?
And if silver, then why not bronze and copper?
Why not just control the quantity ourselves instead? Because that's all what's being done the whole time with the magical gold and silver: with gold and silver you're just saying that you would prefer the quantity of money be controlled by the rich people - that's all.
So you see, it is a question of who controls the quantity, not what backs the money.
Because when you are talking about what backs the money, you are talking about nothing but controlling its quantity.
And there is always a who.
So you cannot get away from that question - who controls the quantity - not with all the appra-ca-dabbra-alla-cazam let-the-magical-free-market-decide-and-it-will-all-sort-itself-out magic spells in the world.