While searching for a suitable place for the monastery, Lama Tseweendorj arrived at a pleasant valley in the south of the Delgerkhaan mountain range in which the Khentii Province is now. At the foot of Mountain Munkh-Ulziit, he found a poor family's camp. Lama Tseweendorj met an old couple named Baldan and Zepelmaa, who invited him into their home. As he entered the old couple’s home, he was greeted with the auspicious sign of a pot of rice boiling over on the fire.  The Lama took some of the rice. He immediately understood that the rice and the significant names of the old couple were auspicious signs. He therefore selected this area as the place where the monastery would be built. In addition to the rice, they noticed that the mountains surrounding the valley in each of the four cardinal directions resembled important Buddhist protector deities. The construction of Baldan Bereeven began with the erection of a small yellow temple built with local materials. A statue of the god Manzushir, carved from a huge stone by the Lama Tseweendorj himself, was placed at the center of the temple. Construction continued with four storied building of the main temple "Tsogchin Dugan" and named “Bileg –Ulziit” near the first temple. The ground floor alone contained 64 angular pillars and was accessed through three different gates facing east, south, and west. Religious flags were hung at each corner and the roof ridge was richly decorated with gold plated Buddhist symbols such as Soyombo, Mirrors, Dug, Jantsan, Khorol, and the holy antelopes. The next building was constructed the theological department which included sections for astrology, philosophy, and medicine. The monastery soon became one of the most important Buddhist centers in Mongolia, inhabited by 5000-6000 monks. It took nearly 76 years to complete the building construction, was started in 1700s and finished around 1776. For centuries the Baldan Bereeven monastery was famous throughout the "Ar Khalkh" region for its many scholars, lamas, artwork, and cultural treasures. Behind the monastery special formation of rock which is called womb symbolizing rebirth.

Between 1937 and 1939, religious persecution became common and a political opposition began to form. Almost all monasteries, like Baldan Bereeven, have been destroyed during that time. At Baldan Bereeven, "The Yellow Temple," the entire surroundings were destroyed and only the ruins of the buildings remained. For over 60 years it was forbidden to visit the monastery.  In 1990s a democracy in Mongolia, a number of monks reestablished some part of the monastery along with the locals. The surrounding rocks of the monastery have been remained with its religious carvings which prove the history of that time.