We have investigated the magnetic properties of B-site-ordered Sr2FeTaO6 double perovskite thin films grown on (111) SrTiO3 substrates by using pulsed-laser deposition. High degree of ordering of Fe and Ta ions has been confirmed using x-ray diffraction and atomically resolved high-angle annular dark-field scanning transmission electron microscopy. We have found clear signatures of magnetic frustration in {111} planes of local spin moment arising from next-nearest-neighbor interaction in triangular FeO6 sublattice below 50 K. It is also found that this behavior is suppressed by increasing disorder and eventually leads to antiferromagnetic state.