Estonian next generation digital Government architecture - from silos to proactive delivery
In 2020, Kristo Vaher the Estonian Government Chief Technology Officer released a discussion paper covering options for Estonia's future government architecture. The paper canvassed a number of areas of interest to researchers and policy makers interested in digital government. For the purposes of this discussion, the focus is upon the shift from siloed architecture to proactive service delivery.
Introduction
The transition from siloed government services to a more proactive, user-centric approach brings a range of challenges . Estonia's experiences in digital government transformation, highlights the complexities of (typically) decentralized service delivery and how governments can overcome these through innovative strategies.
Moving from siloed to proactive service delivery
It is noted that in the private sector, user engagement and retention are crucial for business success. However, in the public sector, and in this case the Estonian Government, services are often funded through a complex government budgeting process, without direct influence from the taxpayers. This can arguably lead to a dominant role for business stakeholders in project decisions, resulting in fragmented and siloed services across different government sectors. The causes for the fragmentation in other countries will inevitably differ.
Estonia’s digital government infrastructure is described as highly decentralised, with multiple ministries and agencies managing their own IT development, often with little coordination across sectors. While this autonomy has contributed to the success of Estonia’s digital government by allowing sectors to tailor solutions to their specific needs, it has also led to a fragmented user experience for citizens, who often find the services complicated and disjointed.
The challenges of fragmentation
The lack of standardisation and coordination has resulted in different sectors developing their own systems, leading to incompatibilities in both business processes and technology stacks. This fragmentation makes it difficult to transfer solutions across sectors, leading to the creation of silos. The complexity of user and permissions management further exacerbates the problem, making it challenging to reuse data or components across different domains.
Proactive services as a solution
In 2018, and in response to these challenges, the Estonian government approved a plan for what was described as 'proactive services'. The goal being to create user-friendly, seamless, and automatic life event services that reduce the need for citizens to repeatedly provide the same information to different government sectors. Unfortunately, the implementation of these proactive services have been subsequently hindered by a lack of standards and clear principles for design and construction.
Addressing the challenges and proposed solutions
To address these issues, the Estonian Government document discusses four key elements:
Conway’s Law: The importance of recognising how the structure of an organisation influences its systems, and that bad processes cannot be improved by technology alone. Flexible system design is noted as being crucial to accommodate organisational changes.
Domain-Driven Design: The focus on designing systems based on the domains they serve, rather than attempting to force a unified model across different sectors. This approach allows for more flexible and adaptive systems that better meet the needs of the organisation.
Business Process Modeling (BPM): Utilising tools for decoupled process management to better understand and optimize workflows across sectors.
Use of an interoperability catalogue: Establishing a centralized catalogue to facilitate data and service interoperability across government sectors.
Domain-Driven Design (DDD)
The paper posits that effective software development begins with a sound design and a thorough understanding of business processes. Arguably organisations focus too heavily on specific functions and technical requirements during the initial stages of developing an information system. However, this approach can lead to issues, as noted by Eric Evans, the author of "Domain-Driven Design," because it often violates Conway’s Law.
Domain-Driven Design (DDD) emphasizes designing systems based on the specific domains within an organisation. A domain represents a distinct area of knowledge, influence, function, or activity within the organization. Each employee typically manages one or more domains, such as inventory management or sales, each contributing to the overall business operations.
An illustrative example of DDD is a business that inherits a large stock of advanced electric car batteries. To manage this effectively, the business would hire an inventory manager, a marketing specialist, a sales manager, and possibly partner with a logistics company like DHL for delivery. Over time, many of these roles could be automated, transforming the business into a flexible, automated system. For instance, an inventory management system could replace the inventory manager, a marketing website could replace the marketing specialist, and so forth.
The flexibility offered by DDD allows businesses to adapt and scale efficiently, avoiding the pitfalls of monolithic systems that lack adaptability. Instead of building a single, all-encompassing system, DDD enables the creation of modular, domain-specific services that can evolve as the business needs change. This approach aligns with the principles outlined by Evans, advocating for systems driven by the unique domains of the process and organization rather than rigid, function-focused designs.
To lay out the scope of a project from a domain-driven perspective, it is recommended that these steps should be followed:
Domain Identification: List all the domains that will interact with the information system or service. A domain is a specific area of knowledge, function, or activity, such as a particular role within a company with distinct responsibilities.
Flow Mapping: Once domains are identified, outline the required flows—specific activities that the system should support. Each flow should have a clear beginning, middle, and end, and may span multiple domains.
Involve technical stakeholders: After mapping domains and flows, involve technical stakeholders, who are responsible for the system's backend services, and user experience (UX) designers, who focus on the interface and user interactions.
User Experience (UX): UX involves everything related to user interface design. In the context of an API, UX designers must understand the domains and flows to create an experience tailored to specific user needs.
Technical services: The backend services, which automate organisational routines, should ideally be aligned with individual domains to create a flexible system.
Testing: Tests should be designed to ensure that each flow in the system functions as expected, focusing on validating the core business functionalities.
Cross-Functional Requirements (CFRs): These are principles, standards, and requirements that span multiple domains and are essential for the system to meet business needs, laws, and regulations.
By following these steps, a project can be scoped effectively with input from business stakeholders, UX designers, and technical stakeholders, enabling the development of a domain-driven system.
The key benefits of this approach:
Agility: The approach aligns with Agile methodologies, making it easier to implement agile practices in software development.
Organisational clarity: It helps clarify business and information architecture, reflecting organisational processes.
Horizontal planning: Projects can be planned domain-by-domain and flow-by-flow.
Role automation: DDD emphasizes the automation of roles and fosters collaboration between business and engineering.
Maintainability: DDD supports the development of maintainable and decentralised service architectures.
For the public sector, different domains such as public servants, citizens, and businesses may need to be treated separately within the system, with distinct back-end services, user interfaces, and tests.
Additional Considerations:
If mapping domains and flows appears too complex, the "strangler pattern" methodology, commonly used to break down monolithic systems, may help simplify the process. (Note: The Strangler pattern is where an “old” or legacy system is put behind an intermediary facade. Over time the external replacement services for the legacy system are added behind the facade. The facade represents the functional entry points to the existing system. Calls to the old system pass then through the facade).
Business Process Modeling (BPM) workflow software is often developed from scratch, but established tools such as Camunda and Flowable could also be used instead. These tools allow for the separation of business flows from functional APIs, offering both engineers and business stakeholders a clear, visual representation of processes. This separation also facilitates the sharing and reuse of APIs, supporting a more distributed government technology architecture.
BPM serves as a domain for specific business processes, like a manager ensuring tasks are completed on time. Workflow tools can be reused across an organization for different flows, making them ideal for orchestrating proactive background services and enabling communication between different administrative sectors.
Key takeaways regarding BPM and workflow engines include:
Complex business processes may require multiple workflow engines to avoid creating a single point of failure.
Start with smaller projects to become familiar with BPM tools before wider implementation.
Workflow engines can provide real-time views for dashboards, aligning with Lean development principles.
BPM and Domain Driven Design are crucial for adopting modern software architecture in digital government.
BPM provides an accurate, up-to-date overview of business processes, reducing reliance on interpretations by analysts or engineers.
BPM helps decouple business process complexity from the functional codebase, encouraging reuse across sectors without forcing them to adopt complex flows.
In the context of digital government, many issues arise from the lack of transparency and interoperability between different sectors. Estonia’s experience highlights the challenges of low reuse of government services, fragmented infrastructure, and outdated systems. The Ministry of Economic Affairs and Communications attempted to address this with RIHA, a registry of government information systems, but this is still a work in progress.
As a consequence, the Estonian public sector engineering community proposed the creation of an Interoperability Catalogue to replace RIHA. This catalogue aims to improve transparency, re-usability, interoperability, and maintain up-to-date information across the digital government architecture. It should be a distributed system, enabling local IT development teams to share data while avoiding a central monolithic dependency.
The Interoperability Catalogue aims to streamline government technology by organising databases and technical components into distinct domains with clearly defined ownership responsibilities. These domain owners, typically business stakeholders or IT development teams, have the freedom to create services and data sets within their domains, ensuring the system remains current and effective.
The success of the Interoperability Catalogue hinges on its ability to include all relevant technical components and databases, making them as transparent as API documentation. In Estonia's experience, a key element is the "service manifest," an XML file describing each component's function, domain, and integration points. These manifests are submitted to the catalogue as part of the continuous integration/deployment (CI/CD) process, ensuring the catalogue is up-to-date and comprehensive.
If well implemented, an Interoperability Catalogue can enhance the reuse of public sector services, improve transparency, and potentially enable private sector companies to offer government services more efficiently. It could also provide a clear view of the costs associated with government services, and aid in better financial planning.
Data management is also noted as crucial for maintaining long-term data quality and supporting data-driven decision-making. All organisations must establish clear roles for data management, use governance tools, and ensure that data replication is controlled and that a single source of truth is maintained for each dataset.
The Interoperability Catalogue can also foster private sector innovation by making public sector services and components more accessible, promoting competition, and potentially reducing costs and improving service quality.
Finally, for governments transitioning from siloed architectures to cross-sector systems, it's important to align applicable regulations with this approach and involve multiple sectors in developing integrated services. Using Domain Driven Design and Business Process Modeling tools can help map out and build these services, even within a single organization. The key is to start piloting and testing new service development methods.
Conclusion
To improve digital service delivery, governments must transition from siloed to proactive service delivery models. The Estonian experience underscores the challenges of fragmentation in digital government services, where decentralised and sector-specific IT systems often lead to disjointed user experiences. Moving towards a more integrated and seamless service delivery requires recognising how organisational structure influences system design and adopting approaches like Domain-Driven Design (DDD) and Business Process Modeling (BPM).
Governments should establish an Interoperability Catalogue that organizes technical components and databases into well-defined domains, each with clear ownership and responsibilities. This ensures that systems remain flexible, up-to-date, and transparent, fostering the reuse of services and encouraging private sector involvement. Effective data management, supported by governance tools and clear roles, is also crucial to maintaining data quality and enabling data-driven decision-making.
For governments aiming to improve service delivery, the key is to pilot and adopt these modern architectural practices, aligning regulations and involving multiple sectors in the process. By doing so, they can enhance citizen experiences, reduce costs, and create more agile and sustainable digital infrastructures.
References
Grigalashvili, V. (2023). Digital Government and Digital Governance: Grand Concept. International Journal of Scientific and Management Research, 6(2).
Vaher, K. (2020). Next generation digital government architecture. Republic of Estonia GCIO Office.
Michael Urban (2018) notes in 'Abandoning Silos How innovative governments are collaborating horizontally to solve complex problems' how the New Zealand’s Data Exchange is a part of a larger government initiative called the social investment approach which is a data-driven, evidence-based “person-centred, integrated approach” that spans the whole-of-government.
The Data Exchange initiative emerged from the government’s realization that in many areas it lacked the data it needed to properly implement its social investment approach or that the data that it did possess was not being shared effectively.
For the social investment approach to be successful, the government and its partners needed to be able to cross-reference multiple datasets spread across and outside of government in order to better understand individuals’…