A service animal should be under control at all times, but this does not only mean that they need to be harnessed, leashed, or tethered while in public places at all times. Sometimes these devices interfere with the service animal’s work or the person’s disability makes these devices difficult to use. In that case, the person must use voice, signal, or other effective means to maintain control of the animal (you can read more under Question 27 on the ADA FAQS page).
One example of this is a person who has a trained PTSD dog and has great difficulty entering unfamiliar spaces. They may have a dog that is trained to enter a space, check to see that no threats are there, and come back and signal that it is safe to enter. The dog must be off leash to do its job, but may be leashed at other times.
You can view the ADA website for more information on the laws regarding service dogs.