With nearly all ballots counted, the governing centre-left coalition is marginally ahead of its centre-right Alliance rivals, with around 40% each.
The nationalist Sweden Democrats (SD) have won about 18% of the vote, up from 12.9% in the previous election.
A protracted battle to form a working coalition now looks a certainty.
Sweden uses a system of proportional representation, under which each party is allocated a number of seats in each constituency that is broadly in line with its share of the vote.
Both of the main blocs have refused to govern with the SD, although its leader said he was prepared to talk with all other parties
"We will increase our seats in parliament and we will gain huge influence over what happens in Sweden during the coming weeks, months and years," Jimmie Akesson told a party rally.
The ruling coalition, headed by outgoing PM Stefan Lofven, is made up of his Social Democrats and the Green Party, and is supported in parliament by the Left Party.
The centre-right Alliance is made up of four parties. Its candidate for PM is Ulf Kristersson, head of the Moderates. He said the ruling coalition had run its course and should resign.
But Mr Lofven said he would not quit. He told a party rally: "We have two weeks left until parliament opens. I will work on calmly, as prime minister, respecting voters and the Swedish electoral system."
Both the Social Democrats and the Moderates saw their percentage of the vote fall, as the SD and smaller parties picked up votes. However, even the SD performance was down on what some opinion polls had suggested.
Analysts suggest it may be easy for the centre-right bloc to form a working coalition, although it will take complicated negotiations.
Immigration has been a central issue of the campaign. The SD is seeking strong curbs on immigration. Mr Lofven has described it as "racist".
Who are the Sweden Democrats?
The SD was linked for years to neo-Nazis and other far-right groups, only entering parliament in 2010.
It has been working to rebrand itself, changing its logo from a flaming torch (similar to the one used by the UK far-right National Front) to a blue-and-yellow daisy, the colours of the Swedish flag.
Traditionally appealing to working-class men, it wants to attract more women and higher-income voters.
Mr Akesson says there is zero tolerance towards racism in the party and several members have been expelled.
However, the party has still been embroiled in various racism scandals.
One municipal candidate shared a song on Facebook with the lyrics "Swedes are white and the country is ours", according to a report in tabloid Aftonbladet.
What were the key issues?
Sweden's economy is booming but many voters are concerned that housing, healthcare and welfare services have come under pressure from a wave of immigration during the 2015 migrant crisis.
That year Sweden took in a record 163,000 asylum seekers - the highest such intake in the EU, per head of population.
Sweden's traditional parties have since hardened their tone to reflect concerns about integration.
Many voters are also concerned about violence. The SD links a rise in shootings to increased immigration, although official figures show no correlation.
Is Malmo the 'rape capital' of Europe?
The SD also wants to leave the European Union and has proposed a "Swexit" referendum. However, the powerful centrist parties all oppose such a vote, so it is unlikely to go ahead.
Aside from immigration, climate change is an issue many Swedes care about - particularly after a long, hot summer and severe forest fires.
During the heatwave, around 25,000 hectares of forest burned in wildfires. Support for the Green party, which had been struggling after various internal scandals, has crept up across the country.