We perform magnetohydrodynamic simulations in full general relativity of uniformly rotating stars that are marginally unstable to collapse. These simulations model the direct collapse of supermassive stars (SMSs) to seed black holes that can grow to become the supermassive black holes at the centers of quasars and active galactic nuclei. They also crudely model the collapse of massive Population III stars to black holes, which could power a fraction of distant, long gamma-ray bursts. The initial stellar models we adopt are Γ=4/3 polytropes initially with a dynamically unimportant dipole magnetic field. We treat initial magnetic-field configurations either confined to the stellar interior or extending out from the stellar interior into the exterior. We find that the black hole formed following collapse has mass MBH≃0.9M (where M is the mass of the initial star) and dimensionless spin parameter aBH/MBH≃0.7. A massive, hot, magnetized torus surrounds the remnant black hole. At Δt∼400-550M≈2000−2700(M/106  M⊙)s following the gravitational wave peak amplitude, an incipient jet is launched. The disk lifetime is Δt∼105(M/106  M⊙)s, and the outgoing Poynting luminosity is LEM∼1051−52  ergs/s. If≳1%−10% of this power is converted into gamma rays, Swift and Fermi could potentially detect these events out to large redshifts z∼20. Thus, SMSs could be sources of ultra-long gamma-ray bursts (ULGRBs), and massive Population III stars could be the progenitors that power a fraction of the long GRBs observed at redshift z∼5-8. Gravitational waves are copiously emitted during the collapse and peak at ∼15(106  M⊙/M)  mHz [∼0.15(104  M⊙/M)  Hz], i.e., in the LISA (DECIGO/BBO) band; optimally oriented SMSs could be detectable by LISA (DECIGO/BBO) at z≲3 (z≲11). Hence, 104  M⊙ SMSs collapsing at z∼10 are promising multimessenger sources of coincident gravitational and electromagnetic waves.