In the 1930s about 700 Armenians lived in Germany. Archarchimandrite Grigor Shahlamian, who came to Germany in 1922 as a theology student, was the first Armenian clergyman to take up pastoral work in Germany in the 20th century. After completing his studies, he looked after the local Armenians from Berlin from 1926 to 1945 and then from Stuttgart until his death on February 21, 1952. Between 1945 and 1948 another Armenian clergyman, Archpriest Vahan Askarian, also served as parish priest in Stuttgart and Esslingen. After the death of Archarchimandrite Shahlamian, the Armenians remained without a permanent parish priest until 1965. Nevertheless, they were ecclesiastically organized under the jurisdiction of the Patriarchal Delegate for Western Europe, based in Paris, who occasionally sent an Armenian cleric from Paris to Germany at the request of the Armenians.

In 1965 the future primate of the diocese of the Armenian Church in Germany, Archbishop Karekin Bekdjian, then still an archimandrite, came to Germany as a student, where he stayed until 1972. During this time, with the permission of the Patriarchal Delegate for Western Europe and at the request of the local Armenians, he celebrated services in Cologne, Stuttgart, Hamburg, Frankfurt and in some other cities in Germany. Parallel to him and also afterwards, the Viennese archarchimandrite Mesrop Grigorian often visited Germany and celebrated church services mainly in Berlin. During this period, the Armenians began to form their own community structures in Germany. By order of the Catholicos of All Armenians, SH Wasken I. (1955-1994), Archimandrite Karekin Nersisian (today Supreme Patriarch and Catholicos of All Armenians) came to Germany in 1975 and took over the pastoral care of the Armenians in the Armenians until his return to Armenia in 1978 federal republic. After him Archimandrite Hagop Keledjian (today Archbishop, Primate of the Armenian Church in Uruguay) came to Germany, who looked after the local Armenian communities between 1978 and 1983.

On January 2, 1980 Wasken I. issued an official encyclical placing the Armenian parishes in Germany under the jurisdiction of the newly sent Patriarchal Delegate for Central Europe (Austria, Germany, Scandinavian countries) based in Vienna. Since then also Archarchimandrite Vazken Tatoyan (1980–1984, deceased), Bishop Davit Sahagian as visiting priest (1983–1985, deceased), Archparish Hagop Geuktchian (1986–1998, deceased), Archimandrite Komitas Hovnanyan (1987–1999, now in France) the pastoral work for the Armenians in Germany.

In 1992, the then supreme patriarch and Catholicos of All Armenians Wasken I founded an official encyclical in the Federal Republic of Germany, with its seat in Cologne, headed by Archbishop Karekin Bekdjian. Five other clergymen worked alongside him. In addition to the diocese, there are also other associations and communities in Germany, some cultural (about 20) and some religious (about 15). These latter are subordinate to the Diocese of the Armenian Church in Germany, which organizes their cooperation. A diocesan plan based on the canonical regulations and the democratic administration of the Armenian Church was developed.

After the retirement of Archbishop Karekin Bekdjian in January 2018, HH Archimandrite Serovpé Isakhanyan was elected the new Primate of the Armenian Diocese in Germany on Sunday, April 15, 2018, by the Extraordinary Assembly of Diocesan Delegates in Cologne. He was ordained bishop on May 12, 2019.