I think that just changing the speakers will not improve the sound of your Bmw E34/E32. Changing the 5.25" speakers with 5.25" coax will not be an improvement neither because the BMW stereo system sends the right signal to the right speakers through the crossover inside the amp.
Each speaker works with in a specified frequency range, you cannot do anything you want if you just change the speakers. For a good improvement, you will have to remove the factory amp/crossover and use a new 4 x 60W (or more) RMS to drive a set of component system speakers.

2 way components speakers are a set of 2 tweeters and 2 woofers, used in conjunction with a passive crossover. The passive crossover takes the signal from the amp and direct it to the right speakers: tweeter signal and woofer signal.

Let's install a set of component speakers. Put the maximum money on the front speakers, the rear speakers are just to "fill in" so you sould use a pair of coax if you want. It is the same procedure for just changing speakers. The bigger speakers you can use without major modification are 6.5", they need a little bit of modification for the front and rear. I run 5.25" front and 6.5" rear component system JBL.

Procedure

Front speakers:
1. Front kick panels are easy, turn the clips 1/4 of a turn with a hex wrench. There are several small trim clips that you can remove with a screwdriver. Removing these will provide easy access to the speaker.
2. There is two way to connect the new tweeters: in the original location, remove the old one (could be hard to pull out but they will come) then glue the new one in place.
Or use the stock location of the midrange in the dashboard.

Rear speakers:
1. The rear speakers are harder. You need to use a needle noise pliers for some of the screws and a small Philips screwdriver to remove the cover. The rear speakers are into a bowl shaped black plastic. You don't need to remove the rear shelf, except if you use speakers bigger than 6.5".
3. Connect the factory wires to the speakers. Stop here if you are just changing the speakers.

For huge speakers: you've got to remove the shelf and the C pillar trim pieces too. Start by pulling up on the front of the seat part of the back seat. Then pull up and out to remove the seat back. You should see several trim fasteners holding the rear shelf in place remove those. The rear shelf should slide out. To remove the C pillar trim panels just grab them and gently pull them out. They're held in by tension fasteners. WARNING: the C pillar trim pieces area a hell to get back in place.

Passive crossover for new amplifier installation.
If you changed your whole system (bye, bye old oem amp) and installed a new amp, you need to install the passive crossover included with your component speakers.
The best location for the passive crossovers is in the trunk, at the fare back, behind the carpeting, you will find an area that is deeper than the rest of the trunk. This is the ideal spot for the two (if you have coax at the rear) or four passive crossovers. Ideal because the wiring harness of the stereo is right there. Check the wiring to connect the wires to the crossovers.
Just use double sided tape and place them in their new location.

Thanks Tim T.