Navid Khan.

Mozilla Broke Pushkit?

10 February 2021 444 views

One of the most common forms of cryptography today is public-key cryptography. This system needs a public key and a private key. Data is encrypted using the public key, and can be decoded using the private key. A common use of this protocol is encrypting internet traffic using SSL or TLS. (For example, configuring nginx to use HTTPS, which is HTTP over SSL/TLS). This allows us to encrypt data being transferred over a protocol that itself does not provide encryption.

A certificate is allows us to share a public key, along with information about the organisation that is responsible for it. A Certificate Authority (CA) is a trusted third party that has confirmed that the information contained in the certificate is accurate.

Hacker New's Certificate

Most software that supports SSL/TLS also maintains a list of CAs whose certificates they automatically trust. If the software encounters a certificate whose authorizing CA is not in their list, they might reject the connection, or ask the user to make a choice.

            $ curl -v -d '{ping: "pong"}'
            * SSL certificate problem: unable to get local issuer certificate
            * stopped the pause stream!
              0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
            * Closing connection 0
            curl: (60) SSL certificate problem: unable to get local issuer certificate
            More details here:

            curl failed to verify the legitimacy of the server and therefore could not
            establish a secure connection to it. To learn more about this situation and
            how to fix it, please visit the web page mentioned above.

Mozilla maintains an official repository of CA roots it trusts. Sometimes, other companies and organisations decide to use Mozilla's root store in their products.

When distributing binary and source code versions of Firefox, Thunderbird, and other Mozilla-related software products, Mozilla includes with such software a set of X.509v3 root certificates for various Certification Authorities (CAs). The included certificates have their "trust bits" set for various purposes, so that the software in question can use the CA certificates to anchor a chain of trust for certificates used by SSL servers and S/MIME email users without having to ask users for further permission or information.
Mozilla Root Source Policy

Here's where things get interesting. In 2017, a consensus proposal was reached among multiple browser makers for a graduated distrust of Symantec roots. The wiki here maintains a list of issues with this particular CA. It makes for an interesting read, and I would encourage you to visit the page.

  • - Issue B: Issuance of 1024-bit Certificate Expiring After Deadline (Dec 2013 - Jan 2014)
  • - Issue C: Unauthorized EV Issuance by RAs (January 2014 - February 2015)
  • - Issue D: Test Certificate Misissuance (April 2009 - September 2015)
  • - Issue E: Domain Validation Vulnerability (October 2015)
  • - Issue F: Symantec Audit Issues 2015 (December 2014 - November 2015)
  • - Issue H: SHA-1 Issuance After Deadline (January 2016)
  • - Issue J: SHA-1 Issuance After Deadline, Again (February 2016)
  • - Issue L: Cross-Signing the US Federal Bridge (2009 - July 2016)
  • - Issue N: Premature Manual Signing Using SHA-1 (July 2016)
  • - Issue P: UniCredit Sub CA Failing To Follow BRs (March - October 2016)
  • - Issue Q: Symantec Audit Issues 2016 (December 2015 - November 2016)
  • - STRUCK: Issue R: Insecure Issuance API (2013 or earlier - November 2016)
  • - Issue T: CrossCert Misissuances (January 2010 - January 2017)
  • - Issue V: GeoRoot Program Audit Issues (2013 or earlier - January 2017)
  • - Issue W: RA Program Audit Issues (2013 or earlier - January 2017)
  • - STRUCK: Issue X: Incomplete RA Program Remediation (February - March 2017)
  • - Issue Y: Under-audited or Unaudited Unconstrained Intermediates (December 2015 - April 2017)

The proposal published a timeline that would phase out CA:Symantec from Mozilla's root program.

  • - December 1st, 2017: Symantec to implement "Managed CA" proposal
  • - January 2018 (Firefox 58): Notices in the Developer Console will warn about Symantec certificates issued before 2016-06-01, to encourage site owners to migrate their TLS certs.
  • - May 2018 (Firefox 60): Websites will show an untrusted connection error if they have a TLS cert issued before 2016-06-01 that chains up to a Symantec root.
  • - October 2018 (Firefox 63): Removal/distrust of Symantec roots, with caveats described below.

As a result of distrust of CA:Symantec, a bunch of certificates would be removed from NSS.

  • - GeoTrust Primary Certification Authority
  • - GeoTrust Universal CA
  • - GeoTrust Universal CA 2
  • - Symantec Class 1 Public Primary Certification Authority - G4
  • - Symantec Class 1 Public Primary Certification Authority - G6
  • - Symantec Class 2 Public Primary Certification Authority - G4
  • - Symantec Class 2 Public Primary Certification Authority - G6
  • - thawte Primary Root CA
  • - thawte Primary Root CA - G2
  • - thawte Primary Root CA - G3
  • - VeriSign Class 1 Public PCA - G3
  • - VeriSign Class 2 Public PCA - G3
  • - VeriSign Class 3 Public Primary Certification Authority - G3
  • - VeriSign Class 3 Public Primary Certification Authority - G4
  • - VeriSign Class 3 Public Primary Certification Authority - G5
  • - VeriSign Universal Root Certification Authority

I knew little of Mozilla's root store or Symantec up until recently. I also did not know that ubuntu uses Mozilla's root store as well.

A week ago, all iOS users on Foxy Live, the app I have been building for EkAnek, stopped receiving VOIP calls. I send calls to our users using Apple's Pushkit, which mandates communication over HTTP2. Since Httparty, or Net::Http does not support HTTP2 yet, I use curl to make our requests to Apple's servers. These requests were failing, and as a result call notifications were not delviered. On investigation, I discovered the error messageSSL certificate problem: unable to get local issuer certificate error cURL produces when it fails to verify a certificate. My first instinct was to add -k to the request and ignore SSL verification, but Apple does not like that and rejects your request. I also frantically ran update-ca-certificates on the server to no effect, and tried updating the root CA store by copying in the file from a different machine. It looked like there was no easy solution to this problem, and I had to get some reading done. The reading is what lead to the discoveries I shared above.

By opening in a browser and inspecting the certificate provided, I learn that is signed by Apple IST CA 2 - G1 , which in turn is authorized by the root CA with the common name GeoTrust Global CA.

I learnt of /etc/ssl/certs/ca-certificates.crt and /etc/ca-certificates.conf. The first file holds all the CA certificates that we've activated in the second file. Lines that begin with a bang indicate certificates that have been deselected. By running awk -v cmd='openssl x509 -noout -subject' '/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt | less I am able to view a list of all authorities trusted by my OS.

            subject=C = PL, O = Unizeto Technologies S.A., OU = Certum Certification Authority, CN = Certum Trusted Network CA
            subject=C = PL, O = Unizeto Technologies S.A., OU = Certum Certification Authority, CN = Certum Trusted Network CA 2
            subject=C = EU, L = Madrid (see current address at, serialNumber = A82743287, O = AC Camerfirma S.A., CN = Chambers of Commerce Root - 2008
            subject=C = GB, ST = Greater Manchester, L = Salford, O = Comodo CA Limited, CN = AAA Certificate Services
            subject=O = "Cybertrust, Inc", CN = Cybertrust Global Root
            subject=C = DE, O = D-Trust GmbH, CN = D-TRUST Root Class 3 CA 2 2009
            subject=C = DE, O = D-Trust GmbH, CN = D-TRUST Root Class 3 CA 2 EV 2009
            subject=O = Digital Signature Trust Co., CN = DST Root CA X3

Grepping the response above for GeoTrust Global CA yields no results. Moving to /etc/ca-certificates.conf, I find the following lines


It looks like Ubuntu decided to update it's root CA store at some point, which told it to stop trusting GeoTrust certificates. This in turn made it difficult to connect to with ssl verification. The temporary solution I implemented to fix this was downloading api-push-apple-com-chain.pem from the certificate inspector in Firefox, and passing it as a CA certificate in the cURL command I am using with the --cacert flag. A solution could be editing the conf file to allow GeoTrust certificates, but I am unsure about how and when this file is automatically updated; and if changes are discarded. If you have better understanding of how this works on Ubuntu, I'd love to hear from you.

Debugging this issue was an interesting learning experience, and this was probably the first time I paid attention to SSL certificates and tried to understand how SSL works in general. certbot -d makes it possible for you to limit your understanding of SSL to the very basics, and lazy thinking made me do just that. There was little help available online around this issue, and searching for SSL certificate problem: unable to get local issuer certificate yields the same 2 - 3 answers of people stuck with incorrect php.ini configurations. I am surprised this change did not cause bigger problems, and by the lack of chatter on the internet. Surely a larger set of users were affected and are unable to connect to services.

Surprisingly, I am still able to access safely on Firefox's latest release, but not on the developer edition, where it rejects the certificate. While I am still learning to blog, and structure information in a manner that is accessible and easy to skim, I hope this helps people who end up with the same issue.

I also wonder what exploits or attacks this makes possible.