They were shot by three Israeli Arabs close to the compound known to Jews as the Temple Mount and to Muslims as Haram al-Sharif (Noble Sanctuary).
Police chased the attackers into the site and shot them dead.
There has been a wave of stabbings, shootings and car-rammings of Israelis predominantly by Palestinians or Israeli Arabs since late 2015.
Police say the three men who carried out Friday's attack were aged between 19 and 29 and came from the northern Israeli city of Umm al-Fahm. Israel's Shin Bet security agency said that they were not previously known to the security services.
Police say the gunmen opened fire as they made their way from the Temple Mount/Haram al-Sharif towards Lions' Gate, an opening in the Old City walls about 100ft (30 metres) away.
The attackers were then pursued back to the compound, where they were killed.
Mobile phone footage showed at least one of the attackers in a confrontation with members of the security forces in the holy site before being shot.
The two policemen who died of their injuries in hospital were named as Advanced Staff Sergeant Major Kamil Shanan, aged 22, and Advanced Staff Sergeant Major Hail Sattawi, who was 30. They were Druze from Israel's northern Galilee region.
Photo by: ISRAEL FOREIGN MINISTRY (Sgt Maj Hail Sattawi (left) and Sgt Maj Kamil Shanan died of their wounds Two of the previous attackers were Jordanians).