Balloons fly early in the morning, right after sunrise and late in the day, right before sunset. This is when the wind is calmest since the sun is low in the sky.

The angle of the sun matters because the sun is the source of thermal activity which heats the earth unevenly. Sunlight falls directly on the equator, for example. The North Pole receives weaker, slanted rays of sunlight. Clouds may keep one area cool while another heats up. Water and land heat up at different rates. Hot air is lighter than cool air, so it rises. As hot air rises, cool air slides in to replace it. The result: wind. It isn't safe to fly during the daytime when different pockets of air are rising and falling.