200 Million Certificates in 24 Hours

Let's Encrypt is a service provided by ISRG- Internet Research Security Group. They issue free Certificate Authority (CA) of two types; the Single Domain SSL and the Wildcard SSL. The Wildcard SSL covers domains that have multiple subdomains. Over 260 million websites currently use certificates supported by Let's Encrypt to enable secure HTTPS deployments. They are a non-profit organization and have contributors from around the globe.

The introduction of Let's Encrypt to the tech space made it easy for users to obtain access to encryption without even having to pay for it. Before their release in November 2014, obtaining a digital certificate was expensive, and websites used unencrypted HTTP connections. Let's Encrypt's solution was to provide free automated services to all and increase the adoption of HTTPS certificates globally.

The digital certificate issuer has issued over 1billion certificates so far, with over three million certificates issued daily. From published statistics made available, we culled that there are 45,891,744 live websites currently using Let's Encrypt.

The United States is the host country for the majority of its users. The countries with the most users are United States (15,346,397 users), United Kingdom (1,472,526 users), Russia (1,451,391 users), Germany (1,140,581 uses) and France (736,099 users). Popular alternative CA providers are GoDaddySSL, DigicertSSL, AplhaSSL, GeoTrustSSL and Global Sign SSL.

Insights from the annual report.

In their annual report last year, they announced the steps taken to address significant internet security challenges. The team would be addressing memory safety vulnerabilities in plans to make Let's Encrypt safer for users. To do this, they would be funding Daniel Stenberg (Curl and WolfSSL) in his work on Hyper, a safe HTTPS implementation written in RUST.

Issuing 200 million certificates in 24hours.

Let's Encrypt published an article on February 10, 2021, that discussed the plans put in motion to replace the certification of their users. This means they would be issuing over 200 million certificates in 24 hours, and they discussed the reasons for this and how they plan to achieve this.

The incident they experienced in February 2020 regarding a bug affecting compliance was a major trigger for this proposed task. The compliance checks were set up to run twice within a 30 day period. The last one took 8 hours before the certificate was issued. The bug, however, would prevent the second check and issue a certificate anyway. The bug was promptly fixed, but it led to them revoking and reissuing certificates to about 2.6% of their current users. This amounted to 3 million users.

This incident also triggered discussions of the possibility of larger-scale incidents affecting all 240 million domains currently in use and all 150 million active certificates. In view of this possibility, the certificates would have to be issued within 24hours, and this led them to increase the standard of their current infrastructure to accommodate this possibility.

The team identified all parts of their current infrastructure that would need to be changed to achieve this. If achieved, they would be the first platform issuing Certificate Authority to achieve this.

The first point was upgrading the database servers. The previous ones used to use dual Intel Xeon E5-2650 v4 CPU instances. An instance has 24 physical cores, 1 TB of memory, with 24 3.8TB SSD's all connected via SATA in a Raid 10 configuration.  The new database servers they replaced this with are DELL with dual AMD EPYC 7542 CPUs, 64 physical cores. Increasing the memory space will also increase speed.

They also received 25G fiber donated from Cisco to improve their network infrastructure, and new cryptographic signing modules (HSMs) donated by Thales that would increase performance. The HSMs have a 10 times increased performance when compared to the previous HSMs in use. They would enable them to perform the 600 million cryptographic signatures needed for the 200 million certificates in 24 hours. The bandwidth was increased by a hardware donated by Fortinet. The hardware would manage higher capacity connections needed.