Do all types of muscle have endomysium covering individual cells?

Yes. The endomysium is a thin layer of connective tissue that surrounds each individual muscle fiber (muscle cell). All muscle tissues, including skeletal muscle, smooth muscle, and cardiac muscle, have an endomysium.