Solar irradiance variability has been monitored almost exclusively from the Earth's perspective. We present a method to combine the unprecedented observations of the photospheric magnetic field and continuum intensity from outside the Sun-Earth line, which is being recorded by the Polarimetric and Helioseismic Imager on board the Solar Orbiter mission (SO/PHI), with solar observations recorded from the Earth's perspective to examine the solar irradiance variability from both perspectives simultaneously. Taking SO/PHI magnetograms and continuum intensity images from the cruise phase of the Solar Orbiter mission and concurrent observations from the Helioseismic and Magnetic Imager onboard the Solar Dynamics Observatory (SDO/HMI) as input into the SATIRE-S model, we successfully reconstructed the total solar irradiance variability as apparent from both perspectives. In later stages of the SO mission, the orbital plane will tilt in such a way as to bring the spacecraft away from the ecliptic to heliographic latitudes of up to 33°. The current study sets the template for the reconstruction of solar irradiance variability as seen from outside the ecliptic from data that SO/PHI is expected to collect from such positions. Such a reconstruction will be beneficial to factoring inclination into how the brightness variations of the Sun compare to those of other cool stars, whose rotation axes are randomly inclined.