This study investigates the characterization and mechanical performance of Stone Mastic Asphalt (SMA) mixtures modified with two types of polymers: styrene–butadiene–styrene (SBS) and high-molecular-weight polyethylene (PE). Neat asphalt cement PG 64-16 was modified using a higher content of SBS and PE at concentrations of 6%, 7%, and 8% by weight of asphalt through the dry blending method to produce Highly Modified Asphalts (HiMA). The physical and rheological properties of the modified binders were evaluated using penetration, softening point, rotational viscosity, and dynamic shear rheometer (DSR) tests. Also, their phase compatibility and morphological changes were evaluated using the storage stability testing and scanning electron microscopy (SEM) analysis. The mechanical performance of the corresponding SMA mixtures was assessed through Marshall stability and flow, moisture susceptibility, crack tolerance index (CT-index), resilient modulus, and rutting resistance tests. Also, a mechanistic durability analysis was conducted using the KENLAYER software. Results indicated that both polymers enhanced the binder’s stiffness and high-temperature performance, with SBS exhibiting greater overall improvements. SBS-modified binders displayed a relatively low softening point difference (ΔT) of 5.1 °C to 5.8 °C, indicating good thermal stability and uniform polymer dispersion. In contrast, PE-modified binders exhibited significantly higher ΔT values, reaching 13.5 °C with 8% PE content, indicating a greater tendency toward phase separation. Moreover, Marshall stability improved substantially, increasing by 43% for 8% SBS-modified mixes and 28% for 8% PE-modified mixes compared to the neat SMA mix. Flow number (FN) results indicated enhanced rutting resistance, with FN values increasing by 2.45 times for SBS mixes and 2.1 times for PE mixes at 8% polymer content. Additionally, moisture susceptibility was significantly improved, as evidenced by the tensile strength ratio (TSR) values of 97% with 8% SBS and 92% with 8% PE, compared to 81% for the neat mix. Resilient modules increased notably, with a 38% rise for 8% SBS mixes and a 24% rise for 8% PE mixes, reflecting enhanced stiffness and load-bearing capacity. Also, the CT-index significantly improved, reaching values of 154 for the 8% SBS mix and 127 for the 8% PE-modified mix, compared to 86 for the neat mix, indicating enhanced resistance to cracking. Finally, both polymer-modified mixes demonstrated improved durability, where the 8% SBS mix exhibited the longest design life (21.66 years) and the highest number of allowable load repetitions (5.42 × 106), followed by 8% PE (13.98 years and 3.50 × 106 repetitions).