Scaling the InCommon Federation: Per-entity Metadata Service
Education and research organizations are increasingly dependent on federated identity management services to collaborate with each other. That increasing reliance is evident in the explosive growth of InCommon Federation metadata in the past 10 years.
In 2008, there were 200 entities registered in the InCommon metadata. As of December 2018, there were over 4,500 entities registered with InCommon. When combined with entities registered through the global eduGAIN interferation, there are now nearly 9,000 entities in the InCommon metadata aggregate.
While exciting, this tremendous growth exposed scaling issues in the way the federation aggregates and distributes metadata. With the aggregate model, metadata for all registered entities are collected into a single XML file. Participating services (identity providers (IdP) as well as service providers (SP) ) then retrieved and loaded the file into system memory on system start-up.
As the aggregate grew, IdPs and SPs needed more time to start up. The larger file also meant increased system memory requirements. By the time InCommon joined eduGAIN, the aggregate had grown large enough that resource-constrained deployers had developed difficulties to fully participate in the federation.
Anticipating the scaling problem, the InCommon Technical Advisory Committee chartered the Metadata Distribution Working Group to study possible solutions. Among other items, the working group recommended conducting a pilot study to explore feasibility of per-entity metadata based on the Metadata Query (MDQ) Protocol. It also to assess the need and applicability of hardware security modules to secure XML signing keys, a critical underpinning of trusted metadata.
In 2016, the Per-Entity Metadata Working Group formed to define requirements for a per-entity metadata distribution service and to outline its implementation and operational considerations. In parallel, InCommon operations carried out an extensive pilot study to develop and deploy an MDQ-based metadata distribution service aligned with the working group’s requirements. The goal: enable the federation to continue to scale while improving the availability and security of its metadata distribution mechanism.
That work is about to bear fruit. In spring 2019, InCommon will launch the Technology Preview of the InCommon Per-Entity Metadata Distribution Service. In parallel, InCommon will publish and review documentation and adoption guidance materials. The service is planned to go live and become generally available in summer of 2019.
The January edition of IAM Online Webinar featured a in-depth discussion of this exciting new service (New InCommon Metadata Service, Per-Entity Metadata Deployment). To learn more, be sure to check out the IAM Online recording and slides.