In the deadliest day of violence in Gaza since the 2014 war, Palestinian officials say Israeli troops have killed 55 people and wounded 2,700.
Israeli PM Benjamin Netanyahu said his military was acting in self-defence against Gaza's Islamist rulers, Hamas, who he said wanted to destroy Israel.
The Palestinian Authority's leader condemned a "massacre". The UN spoke of "outrageous human rights violations".
The violence came as the US opened a controversial embassy in Jerusalem.
The move of the US embassy from Tel Aviv has incensed Palestinians, who claim eastern Jerusalem as the capital of a future Palestinian state.
They see the US move as backing Israeli control over the whole of the city, which Israel regards as its indivisible capital.
The day's events as they happened
What makes Jerusalem so holy?
What happened at the Gaza border?
Palestinians were demonstrating on Monday as they have been for six weeks as part of a protest, orchestrated by Hamas, called the "Great March of Return".
However, Monday's protests - and more planned for Tuesday - are the culmination, as they mark the anniversary of Israel's creation in 1948 and what Palestinians term the Nakba or Catastrophe, referring to the hundreds of thousands of their people who subsequently fled their homes or were displaced in the war that followed.
Monday also coincided with the dedication ceremony for the US embassy.
Israel said some 40,000 Palestinians had taken part in "violent riots" at 13 locations along the Gaza Strip security fence.
Palestinians hurled stones and incendiary devices, while the Israeli military used tear gas and live fire from snipers.
Mr Netanyahu defended his military, saying: "Every country has an obligation to defend its borders.
"The Hamas terrorist organisation declares its intention to destroy Israel and sends thousands to breach the border fence in order to achieve this goal. We will continue to act with determination to protect our sovereignty and our citizens."
An Israel Defense Forces (IDF) spokesman said soldiers fired on those engaged in "terrorist activity and not on demonstrators, who were dispersed by usual means such as tear gas and according to the rules of engagement".
Announcing three days of mourning, Palestinian Authority President Mahmoud Abbas said: "Today once again, the massacres against our people continue."
What has the international reaction been?
There has been a sometimes fiercely conflicting response:
White House spokesman Raj Shah said: "The responsibility for these tragic deaths rests squarely with Hamas... Hamas is intentionally and cynically provoking this response"
Kuwait drafted a UN Security Council statement calling for an independent inquiry into the violence - and expressing "outrage and sorrow" - but this was blocked by the US
EU foreign policy chief Federica Mogherini and the UK called for restraint
Germany said Israel had the right to defend itself but should do so proportionately
France's President Emmanuel Macron condemned violence by the Israeli military against the protesters
Turkey said the US shared responsibility with Israel for a "vile massacre" and that it was recalling its ambassadors from both the US and Israel
South Africa also recalled its ambassador to Israel, condemning "the indiscriminate and grave manner of the latest Israeli attack"
Was the violence linked to the opening of the embassy?
There were clashes between Israeli police and protesters who raised Palestinian flags outside the new embassy in Jerusalem and several protesters were detained.
Hamas had said the border protests were being stepped up for Monday and Tuesday anyway, but Mr Abbas's response to the US embassy move showed the anger among Palestinians.
He said: "We hear that they opened an embassy today. It is a settlement, not an embassy. A US settlement in East Jerusalem."
The atmosphere at the dedication ceremony was certainly in stark contrast to that at the Gaza border.
President Donald Trump sent a video message to the event, saying Israel had the "right to determine its own capital".
His daughter, Ivanka, unveiled the seal of the embassy, and her husband, Jared Kushner, said in his address: "When President Trump makes a promise he keeps it."
Mr Netanyahu said: "President Trump, by recognising history, you have made history."
Why is the embassy move so controversial?
The status of Jerusalem goes to the heart of the Israeli-Palestinian conflict.
Can the settlement issue be solved?
The alternatives to a two-state solution
Israeli sovereignty over Jerusalem is not recognised internationally and, according to the 1993 Israel-Palestinian peace accords, the final status of Jerusalem is meant to be discussed in the latter stages of peace talks.
Israel has occupied East Jerusalem since the 1967 Middle East war. It effectively annexed the sector, though this was not recognised by any countries until Mr Trump's declaration in December 2017.
Since 1967, Israel has built a dozen settlements, home to about 200,000 Jews, in East Jerusalem. These are considered illegal under international law, although Israel disputes this.
Why do evangelicals support Trump on Jerusalem?
Various countries once had embassies based in Jerusalem but many moved after Israel passed a law in 1980 formally making Jerusalem its capital.