In genetics, a qualitative trait is one that's either/or: if you don't have the right gene, you don't have the trait. Quantitative genes are all about how much of the trait you have. Genes' effect on human height is quantitative, for instance. We all have height, but genes influence how much of it we have. The quantitative or qualitative genes influencing a particular trait are the genotype; the physical trait itself is called the phenotype.
Qualitative genes produce what are called discontinuous traits. For example, you only have Rh-positive blood -- A positive, B positive and so on -- if you have the genotype for Rh-positive blood. There's no in-between zone where your blood can be half-positive, half-negative. With quantitative genes, traits fall on a continuum. The genetic effect on blood pressure, for example, can set you up for low, medium or high blood pressure, depending on your exact mix of traits.
Complexity in genetics is a measure of how many genes are involved in producing a particular trait. Qualitative traits may be controlled by a single gene; quantitative traits are usually complex. A complex genetic disease is much harder to study as it's not controlled or influenced by one gene alone. Because multiple genes influence the trait, you get a continuum where the different genetic factors interact in multiple ways. The interactions of multiple qualitative genes can result in a quantitative trait.
Medical researchers study qualitative traits by studying the frequency of the related conditions. If you're researching a gene's effect on a qualitative trait -- a type of brain disorder, say -- you can compare the presence of the disorder in families with the gene to similar families that don't carry it. This gets complicated as most health problems are influenced by the environment too. As family members often live in the same environment, that has to be taken into account. If the trait is influenced by multiple genes, the analysis is more difficult.
Genetic tests and family studies are important for studying quantitative traits too. Unlike a qualitative trait, different family members may have some, none or a lot of whatever the trait gives people. If you're studying the effects of genes on blood pressure or height, for instance, you might look at what percentage of family members resemble each other in height, compared to the percentage of unrelated individuals. If the two percentages are significantly different, that's a sign of possible genetic influence.