It is hoped the deal will ease access to markets within the region and end problems arising from the fact several countries belong to multiple groups.
The deal also aims to strengthen the bloc's bargaining power when negotiating international deals.
Analysts say the agreement will help intra-regional trade and boost growth.
The three blocs which struck the deal were the Southern African Development Community (SADC), the East African Community (EAC) and the the Common Market for Eastern and Southern Africa (Comesa).
"The greatest enemy of Africa, the greatest source of weakness has been disunity and a low level of political and economic integration," said Ugandan President Yoweri Museveni at a meeting with the heads of state who chair the three trade blocs.
The agreement will also lend its backing to joint infrastructure and energy projects in the zone.
Six heads of state from 26 countries in Comesa, SADC and the EAC attended the meeting in the Ugandan capital, Kampala, to sign the agreement.
Many of the leaders and representatives consider the new pact a way of giving Africa a greater voice on the world stage.
"By coming together, the member states will have a strong voice in advancing our interests on the international scene," said South African President Kgalema Motlanthe.
Meanwhile, President Museveni said that it was a step in the right direction for a continent that suffered unfairly when it came to global trade.
President Motlanthe also called for developing countries to have positions within global institutions.
""While Africa and other developing countries had marginal influence over the decisions that have brought the international finance systems to the brink of collapse, unjustifiably, African countries will bear the brunt," he said.
"Development countries must be included in the governance of all international financing institutions to mitigate adverse effects on them," Mr Motlanthe added.
The three blocs are already well-established in their own right but cover varying swathes of land and numbers of people.
The SADC was first established as the Southern African Development Coordination Conference in 1980 in order to reduce independence on apartheid South Africa.
It was reincarnated as the SADC in 1992. It covers a population of some 248 million people and a zone whose cumulative GDP is $379bn in 2006.
The SADC's members include South Africa, Tanzania, Zambia and Zimbabwe.
Comesa was established in 1994 and replaced the Preferential Trade Area. It includes 398 million people and the area has a combined GDP of $286.7bn in 2006. Among its members are Zimbabwe, Zambia, Uganda and Sudan.
EAC is the smallest of the group in terms of GDP, and had a GDP of $46.6bn in 2006. Set up in 1967, disagreements between founding members Uganda, Kenya and Tanzania led to its collapse.
A treaty was signed for its re-establishment in 1999 and the new EAC was formed in 2000.