I have tried many adhesives and do not recommend any of them, but two that other people have recommended (I have not used them because they’re not available where I live) are HH-66 and Stabond. They will stick to the fabric, but I don’t know if they’ll stick to a magnet. You may have better luck heat sealing a piece of fabric over the magnet to hold it in place.
Cheers!