The Extreme Ultraviolet Imager (EUI) on board the Solar Orbiter (SO) spacecraft observed small extreme ultraviolet (EUV) bursts, termed campfires, that have been proposed to be brightenings near the apexes of low-lying loops in the quiet-Sun atmosphere. The underlying magnetic processes driving these campfires are not understood. Aims: During the cruise phase of SO and at a distance of 0.523 AU from the Sun, the Polarimetric and Helioseismic Imager on Solar Orbiter (SO/PHI) observed a quiet-Sun region jointly with SO/EUI, offering the possibility to investigate the surface magnetic field dynamics underlying campfires at a spatial resolution of about 380 km. Methods: We used co-spatial and co-temporal data of the quiet-Sun network at disc centre acquired with the High Resolution Imager of SO/EUI at 17.4 nm (HRIEUV, cadence 2 s) and the High Resolution Telescope of SO/PHI at 617.3 nm (HRT, cadence 2.5 min). Campfires that are within the SO/PHI−SO/EUI common field of view were isolated and categorised according to the underlying magnetic activity. Results: In 71% of the 38 isolated events, campfires are confined between bipolar magnetic features, which seem to exhibit signatures of magnetic flux cancellation. The flux cancellation occurs either between the two main footpoints, or between one of the footpoints of the loop housing the campfire and a nearby opposite polarity patch. In one particularly clear-cut case, we detected the emergence of a small-scale magnetic loop in the internetwork followed soon afterwards by a campfire brightening adjacent to the location of the linear polarisation signal in the photosphere, that is to say near where the apex of the emerging loop lays. The rest of the events were observed over small scattered magnetic features, which could not be identified as magnetic footpoints of the campfire hosting loops. Conclusions: The majority of campfires could be driven by magnetic reconnection triggered at the footpoints, similar to the physical processes occurring in the burst-like EUV events discussed in the literature. About a quarter of all analysed campfires, however, are not associated to such magnetic activity in the photosphere, which implies that other heating mechanisms are energising these small-scale EUV brightenings.