Growing Rudbeckias from seed

I love Rudbeckias and grow several varieties from seed.  However, although they grow very nicely I seem to lose a lot of the plants when they get to about 6-9in. tall.  By this time they are in pots outside, so is it rain, too cold, too hot???  Those that survive grow into very healthy plants