Much of the assessment of the quality of your hay can be done the old-fashioned way: Break open a bale and scratch n' sniff! Good quality hay should be green rather than yellow or brown. (Keep in mind that some hays, particularly some varieties of clover, can cure to quite a dark color, and that this is not necessarily an indicator of mold growth.) It should have a high leaf content ("stemmy" hay is too mature) and few weeds; it should also smell pleasant and slightly sweet. There should be no visible mold (white or dark, matted patches in the hay) or other foreign material. Taking a handful of hay and squeezing it should not hurt your hand--prickly hay has been cut too late and has a low nutrient content. And if you drop a flake of hay from a height of a few feet, you should not see clouds of dust rising from it--dust is usually an indicator that the grower had the tines on his harvester set too low. As a general rule, the nicer you feel that the loose hay would be to sleep in, the better the quality!
To really determine the nutrient content of the hay, however, you'll need to do a hay analysis. Appearance is a poor indicator of nutritive value--even grass hays that appear very similar can vary in protein content by two to three times. Performing a hay analysis whenever you receive a shipment of hay is an excellent routine to establish, especially as the results may have a significant impact on the grains and supplements you choose to feed.
How much to feed? As a rough guideline, horses should consume 1% to 2% of their bodyweight each day in forage products--at least 50% of their total diets under all but the most extreme exercise programs. Though all of us prefer to feed by "eyeballing" amounts, the weight and size of a flake of hay can vary so much that it is worth weighing the flakes to determine how close you are to these guidelines. This can be done very simply by standing on a bathroom scale, with and without the flake of hay, and subtracting the difference.
When feeding hay, remember that, at heart, horses are grazing animals, programmed to chew on stemmy, fibrous plants for at least 12 hours a day. That urge to chew can be almost as compelling as a rodent's, so hay fulfills two functions in your barn: It provides nutrients (and keeps the digestive system in good health), but it also keeps horses busy (and thus not chewing the wood fences, stall doors, or their neighbors' tails!). An almost constant supply of small amounts of hay is far more beneficial than one or two large feedings a day because it mimics the horse's natural grazing habits. Make lots of good-quality hay the basis of your horses' diets, and you'll reap the benefits in terms of both health and contentment.
Disclaimer: Seek the advice of a qualified veterinarian before proceeding with any diagnosis, treatment, or therapy.