Release v0.19.1

Previously, if we rounded down the tree size to avoid downloading a
partial tile, but the log position was already within the partial tile
(which can happen with a brand new log and -start_at_end), we'd generate
a download batch where end < begin, which caused all sorts of problems.

CHANGELOG.md

@@ -0,0 +1,208 @@
+# Change Log
+
+## v0.19.1 (2025-05-07)
+- Fix panic when retrying failed log requests.
+- Properly log failed log requests.
+
+## v0.19.0 (2025-05-07) (RETRACTED)
+- Add support for static-ct-api logs, the next generation of CT logs.
+- Add support for downloading entries in parallel, to avoid backlogs when
+  monitoring fast-growing logs.
+- Use $EMAIL environment variable to determine sender of emails.
+- Remove submitct command.
+- Make certspotter more resilient to system crashes such as power failures.
+
+## v0.18.0 (2023-11-13)
+- Fix bug with downloading entries that did not materialize in practice
+  with any of the current logs.
+- Include `Message-ID` and `Date` in outbound emails.
+
+## v0.17.0 (2023-10-26)
+- Allow sendmail path to be configured with `$SENDMAIL_PATH`.
+- Minor improvements to documentation, efficiency.
+
+## v0.16.0 (2023-02-21)
+- Write malformed certs and failed healthchecks to filesystem so scripts
+  can access them.
+- Automatically execute scripts under `$CERTSPOTTER_CONFIG_DIR/hooks.d`
+  if it exists.
+- Automatically email addresses listed in `$CERTSPOTTER_CONFIG_DIR/email_recipients`
+  if it exists.
+
+## v0.15.1 (2023-02-09)
+- Fix some typos in help and error messages.
+- Allow version to be set via linker flag, to facilitate distro package building.
+
+## v0.15.0 (2023-02-08)
+- **Significant behavior change**: certspotter is now intended to run as
+  a daemon instead of a cron job.  Specifically, certspotter no longer
+  terminates unless it receives SIGTERM or SIGINT or there is a serious error.
+  You should remove certspotter from your crontab and arrange to run it as a
+  daemon, passing either the `-email` option or `-script` option to configure
+  how you want to be notified about certificates.
+
+  Reason for this change: although using cron made sense in the early days of
+  Certificate Transparency, certspotter now needs to run continuously to reliably
+  keep up with the high growth rate of contemporary CT logs, and to gracefully
+  handle the many transient errors that can arise when monitoring CT.
+  See <https://github.com/SSLMate/certspotter/issues/63> for background.
+
+- `-script` is now officially supported and can be used to execute
+  a command when a certificate is discovered or there is an error.  For details,
+  see the [certspotter-script(8) man page](man/certspotter-script.md).
+
+  Note the following changes from the experimental, undocumented `-script`
+  option found in previous versions:
+  - The script is also executed when there is an error.  Consult the `$EVENT`
+    variable to determine why the script was executed.
+  - The `$DNS_NAMES` and `$IP_ADDRESSES` variables have been removed because
+    the OS limits the size of environment variables and some certificates have
+    too many identifiers.  To determine a certificate's identifiers, you can
+    read the JSON file specified by the `$JSON_FILENAME` variable, as explained
+    in the [certspotter-script(8) man page](man/certspotter-script.md).
+  - The `$CERT_TYPE` variable has been removed because it is almost always
+    a serious mistake (that can make you miss malicious certificates) to treat
+    certificates and precertificates differently.  If you are currently
+    using this variable to skip precertificates, stop doing that because
+    precertificates imply the existence of a corresponding certificate that you
+    **might not** be separately notified about.  For more details, see
+    <https://github.com/SSLMate/certspotter/commit/cd2bb429fc2f4060a33ec8eb8b71a3eb12e9ba93>.
+  - New variable `$WATCH_ITEM` contains the first watch list item which
+    matched the certificate.
+
+- New `-email` option can be used to send an email when a certificate is
+  discovered or there is an error.  Your system must have a working `sendmail`
+  command.
+
+- (Behavior change) You must specify the `-stdout` option if you want discovered
+  certificates to be written to stdout.  This only makes sense when running
+  certspotter from the terminal; when running as a daemon you probably want to
+  use `-email` or `-script` instead.
+
+- Once a day, certspotter will send you a notification (per `-email` or
+  `-script`) if any problems are preventing it from detecting all certificates.
+  As in previous versions of certspotter, errors are written to stderr when they
+  occur, but since most errors are transient, you can now ignore stderr and rely
+  on the daily health check to notify you about any persistent problems that
+  require your attention.
+
+- certspotter now saves `.json` and `.txt` files alongside the `.pem` files
+  containing parsed representations of the certificate.
+
+- `.pem` files no longer have `.cert` or `.precert` in the filename.
+
+- certspotter will save its state periodically, and before terminating due to
+  SIGTERM or SIGINT, meaning it can resume monitoring without having to
+  re-download entries it has already processed.
+
+- The experimental "BygoneSSL" feature has been removed due to limited utility.
+
+- The `-num_workers` option has been removed.
+
+- The `-all_time` option has been removed. You can remove the certspotter state
+  directory if you want to re-download all entries.
+
+- The minimum supported Go version is now 1.19.
+
+## v0.14.0 (2022-06-13)
+- Switch to Go module versioning conventions.
+
+## v0.13 (2022-06-13)
+- Reduce minimum Go version to 1.17.
+- Update install instructions.
+
+## v0.12 (2022-06-07)
+- Retry failed log requests.  This should make certspotter resilient
+  to rate limiting by logs.
+- Add `-version` flag.
+- Eliminate unnecessary dependency. certspotter now depends only on
+  golang.org/x packages.
+- Switch to Go modules.
+
+## v0.11 (2021-08-17)
+- Add support for contacting logs via HTTP proxies;
+  just set the appropriate environment variable as documented at
+  <https://golang.org/pkg/net/http/#ProxyFromEnvironment>.
+- Work around RFC 6962 ambiguity related to consistency proofs
+  for empty trees.
+
+## v0.10 (2020-04-29)
+- Improve speed by processing logs in parallel
+- Add `-start_at_end` option to begin monitoring new logs at the end,
+  which significantly speeds up Cert Spotter, at the cost of missing
+  certificates that were added to a log before Cert Spotter starts
+  monitoring it
+- (Behavior change) Scan logs in their entirety the first time Cert
+  Spotter is run, unless `-start_at_end` specified (behavior change)
+- The log list is now retrieved from certspotter.org at startup instead
+  of being embedded in the source. This will allow Cert Spotter to react
+  more quickly to the frequent changes in logs.
+- (Behavior change) the `-logs` option now expects a JSON file in the v2
+  log list format. See <https://www.certificate-transparency.org/known-logs>
+  and <https://www.gstatic.com/ct/log_list/v2/log_list_schema.json>.
+- `-logs` now accepts an HTTPS URL in addition to a file path.
+- (Behavior change) the `-underwater` option has been removed. If you want
+  its behavior, specify `https://loglist.certspotter.org/underwater.json` to
+  the `-logs` option.
+
+## v0.9 (2018-04-19)
+- Add Cloudflare Nimbus logs
+- Remove Google Argon 2017 log
+- Remove WoSign and StartCom logs due to disqualification by Chromium
+  and extended downtime
+
+## v0.8 (2017-12-08)
+- Add Symantec Sirius log
+- Add DigiCert 2 log
+
+## v0.7 (2017-11-13)
+- Add Google Argon logs
+- Fix bug that caused crash on 32 bit architectures
+
+## v0.6 (2017-10-19)
+- Add Comodo Mammoth and Comodo Sabre logs
+- Minor bug fixes and improvements
+
+## v0.5 (2017-05-18)
+- Remove PuChuangSiDa 1 log due to excessive downtime and presumptive
+  disqualification from Chrome
+- Add Venafi Gen2 log
+- Improve monitoring robustness under certain pathological behavior
+  by logs
+- Minor documentation improvements
+
+## v0.4 (2017-04-03)
+- Add PuChuangSiDa 1 log
+- Remove Venafi log due to fork and disqualification from Chrome
+
+## v0.3 (2017-02-20)
+- Revise `-all_time` flag (behavior change):
+  - If `-all_time` is specified, scan the entirety of all logs, even
+    existing logs
+  - When a new log is added, scan it in its entirety even if `-all_time`
+    is not specified
+- Add new logs:
+  - Google Icarus
+  - Google Skydiver
+  - StartCom
+  - WoSign
+- Overhaul log processing and auditing logic:
+  - STHs are never deleted unless they can be verified
+  - Multiple unverified STHs can be queued per log, laying groundwork
+    for STH pollination support
+  - New state directory layout; current state directories will be
+    migrated, but migration will be removed in a future version
+  - Persist condensed Merkle Tree state between runs, instead of
+    reconstructing it from consistency proof every time
+- Use a lock file to prevent multiple instances of Cert Spotter from
+  running concurrently (which could clobber the state directory).
+- Minor bug fixes and improvements
+
+## v0.2 (2016-08-25)
+- Suppress duplicate identifiers in output.
+- Fix "EOF" error when running under Go 1.7.
+- Fix bug where hook script could fail silently.
+- Fix compilation under Go 1.5.
+
+## v0.1 (2016-07-27)
+- Initial release.

NEWS

@@ -1,61 +0,0 @@
-v0.9 (2018-04-19)
-  * Add Cloudflare Nimbus logs
-  * Remove Google Argon 2017 log
-  * Remove WoSign and StartCom logs due to disqualification by Chromium
-    and extended downtime
-
-v0.8 (2017-12-08)
-  * Add Symantec Sirius log
-  * Add DigiCert 2 log
-
-v0.7 (2017-11-13)
-  * Add Google Argon logs
-  * Fix bug that caused crash on 32 bit architectures
-
-v0.6 (2017-10-19)
-  * Add Comodo Mammoth and Comodo Sabre logs
-  * Minor bug fixes and improvements
-
-v0.5 (2017-05-18)
-  * Remove PuChuangSiDa 1 log due to excessive downtime and presumptive
-    disqualification from Chrome
-  * Add Venafi Gen2 log
-  * Improve monitoring robustness under certain pathological behavior
-    by logs
-  * Minor documentation improvements
-
-v0.4 (2017-04-03)
-  * Add PuChuangSiDa 1 log
-  * Remove Venafi log due to fork and disqualification from Chrome
-
-v0.3 (2017-02-20)
-  * Revise -all_time flag (behavior change):
-    - If -all_time is specified, scan the entirety of all logs, even
-      existing logs
-    - When a new log is added, scan it in its entirety even if -all_time
-      is not specified
-  * Add new logs:
-    - Google Icarus
-    - Google Skydiver
-    - StartCom
-    - WoSign
-  * Overhaul log processing and auditing logic:
-    - STHs are never deleted unless they can be verified
-    - Multiple unverified STHs can be queued per log, laying groundwork
-      for STH pollination support
-    - New state directory layout; current state directories will be
-      migrated, but migration will be removed in a future version
-    - Persist condensed Merkle Tree state between runs, instead of
-      reconstructing it from consistency proof every time
-  * Use a lock file to prevent multiple instances of Cert Spotter from
-    running concurrently (which could clobber the state directory).
-  * Minor bug fixes and improvements
-
-v0.2 (2016-08-25)
-  * Suppress duplicate identifiers in output.
-  * Fix "EOF" error when running under Go 1.7.
-  * Fix bug where hook script could fail silently.
-  * Fix compilation under Go 1.5.
-
-v0.1 (2016-07-27)
-  * Initial release.

README

@@ -1,132 +0,0 @@
-Cert Spotter is a Certificate Transparency log monitor from SSLMate that
-alerts you when a SSL/TLS certificate is issued for one of your domains.
-Cert Spotter is easier than other open source CT monitors, since it does
-not require a database.  It's also more robust, since it uses a special
-certificate parser that ensures it won't miss certificates.
-
-Cert Spotter is also available as a hosted service by SSLMate that
-requires zero setup and provides an easy web dashboard to centrally
-manage your certificates.  Visit <https://sslmate.com/certspotter>
-to sign up.
-
-You can use Cert Spotter to detect:
-
-* Certificates issued to attackers who have compromised a certificate
-  authority and want to impersonate your site.
-
-* Certificates issued to attackers who are using your infrastructure
-  to serve malware.
-
-* Certificates issued in violation of your corporate policy
-  or outside of your centralized certificate procurement process.
-
-* Certificates issued to your infrastructure providers without your
-  consent.
-
-
-USING CERT SPOTTER
-
-The easiest way to use Cert Spotter is to sign up for an account at
-<https://sslmate.com/certspotter>.  If you want to run Cert Spotter on
-your own server, follow these instructions.
-
-Cert Spotter requires Go version 1.5 or higher.
-
-1. Install Cert Spotter using go get:
-
-	go get software.sslmate.com/src/certspotter/cmd/certspotter
-
-2. Create a file called ~/.certspotter/watchlist listing the DNS names
-   you want to monitor, one per line.  To monitor an entire domain tree
-   (including the domain itself and all sub-domains) prefix the domain
-   name with a dot (e.g. ".example.com").  To monitor a single DNS name
-   only, do not prefix the name with a dot.
-
-3. Create a cron job to periodically run:
-
-	certspotter
-
-   When Cert Spotter detects a certificate for a name on your watchlist,
-   it writes a report to standard out, which the Cron daemon emails
-   to you.  Make sure you are able to receive emails sent by Cron.
-
-   Cert Spotter also saves a copy of matching certificates in
-   ~/.certspotter/certs.
-
-You can add and remove domains on your watchlist at any time.  However,
-the certspotter command only notifies you of certificates that were
-logged since adding a domain to the watchlist, unless you specify the
--all_time option, which requires scanning the entirety of every log
-and takes several hours to complete with a fast Internet connection.
-To examine preexisting certificates, it's better to use the Cert
-Spotter service <https://sslmate.com/certspotter>, the Cert Spotter
-API <https://sslmate.com/certspotter/api>, or a CT search engine such
-as <https://crt.sh>.
-
-
-COMMAND LINE FLAGS
-
-  -watchlist FILENAME
-	File containing identifiers to watch, one per line, as described
-	above (use - to read from stdin).  Default: ~/.certspotter/watchlist
-  -no_save
-	Do not save a copy of matching certificates.
-  -all_time
-	Scan for certificates from all time, not just those added since
-	the last run of Cert Spotter.  Unless this option is specified,
-	no certificates are scanned the first time Cert Spotter is run.
-  -logs FILENAME
-	JSON file containing logs to scan, in the format documented at
-	<https://www.certificate-transparency.org/known-logs>.
-	Default: use the logs trusted by Chromium.
-  -state_dir PATH
-	Directory for storing state. Default: ~/.certspotter
-  -verbose
-	Be verbose.
-
-
-WHAT CERTIFICATES ARE DETECTED BY CERT SPOTTER?
-
-Any certificate that is logged to a Certificate Transparency log trusted
-by Chromium will be detected by Cert Spotter.  All certificates issued
-after April 30, 2018 must be logged to such a log to be trusted by Chromium.
-
-Generally, certificate authorities will automatically submit certificates
-to logs so that they will work in Chromium.  In addition, certificates
-that are discovered during Internet-wide scans are submitted to Certificate
-Transparency logs.
-
-
-SECURITY
-
-Cert Spotter assumes an adversarial model in which an attacker produces
-a certificate that is accepted by at least some clients but goes
-undetected because of an encoding error that prevents CT monitors from
-understanding it.  To defend against this attack, Cert Spotter uses a
-special certificate parser that keeps the certificate unparsed except
-for the identifiers.  If one of the identifiers matches a domain on your
-watchlist, you will be notified, even if other parts of the certificate
-are unparsable.
-
-Cert Spotter takes special precautions to ensure identifiers are parsed
-correctly, and implements defenses against identifier-based attacks.
-For instance, if a DNS identifier contains a null byte, Cert Spotter
-interprets it as two identifiers: the complete identifier, and the
-identifier formed by truncating at the first null byte.  For example, a
-certificate for example.org\0.example.com will alert the owners of both
-example.org and example.com.  This defends against null prefix attacks
-<http://www.thoughtcrime.org/papers/null-prefix-attacks.pdf>.
-
-SSLMate continuously monitors CT logs to make sure every certificate's
-identifiers can be successfully parsed, and will release updates to
-Cert Spotter as necessary to fix parsing failures.
-
-Cert Spotter understands wildcard and redacted DNS names, and will alert
-you if a wildcard or redacted certificate might match an identifier on
-your watchlist.  For example, a watchlist entry for sub.example.com would
-match certificates for *.example.com or ?.example.com.
-
-Cert Spotter is not just a log monitor, but also a log auditor which
-checks that the log is obeying its append-only property.  A future
-release of Cert Spotter will support gossiping with other log monitors
-to ensure the log is presenting a single view.

README.md

@@ -0,0 +1,107 @@
+# Cert Spotter - Certificate Transparency Monitor
+
+**Cert Spotter** is a Certificate Transparency log monitor from SSLMate that
+alerts you when an SSL/TLS certificate is issued for one of your domains.
+Cert Spotter is easier to use than other open source CT monitors, since it does not require
+a database. It's also more robust, since it uses a special certificate parser
+that ensures it won't miss certificates.
+
+Cert Spotter is also available as a hosted service by SSLMate that
+requires zero setup and provides an easy web dashboard to centrally
+manage your certificates.  Visit <https://sslmate.com/certspotter>
+to sign up.
+
+You can use Cert Spotter to detect:
+
+ * Certificates issued to attackers who have compromised your DNS and
+   are redirecting your visitors to their malicious site.
+ * Certificates issued to attackers who have taken over an abandoned
+   sub-domain in order to serve malware under your name.
+ * Certificates issued to attackers who have compromised a certificate
+   authority and want to impersonate your site.
+ * Certificates issued in violation of your corporate policy
+   or outside of your centralized certificate procurement process.
+
+## Quickstart
+
+The following instructions require you to have [Go version 1.21 or higher](https://go.dev/dl/) installed.
+
+1. Install the certspotter command using the `go` command:
+
+   ```
+   go install software.sslmate.com/src/certspotter/cmd/certspotter@latest
+   ```
+
+2. Create a watch list file `$HOME/.certspotter/watchlist` containing the DNS names you want to monitor,
+   one per line.  To monitor an entire domain tree (including the domain itself
+   and all sub-domains) prefix the domain name with a dot (e.g. `.example.com`).
+   To monitor a single DNS name only, do not prefix the name with a dot.
+
+3. Place one or more email addresses in the `$HOME/.certspotter/email_recipients`
+   file (one per line), and/or place one or more executable scripts in the
+   `$HOME/.certspotter/hooks.d` directory.  certspotter will email the listed
+   addresses (requires your system to have a working sendmail command) and
+   execute the provided scripts when it detects a certificate for a domain on
+   your watch list.
+
+4. Configure your system to run `certspotter` as a daemon.  You may want to specify
+   the `-start_at_end` command line option to tell certspotter to start monitoring
+   new logs at the end instead of the beginning.  This saves significant bandwidth, but
+   you won't be notified about certificates which were logged before you started
+   using certspotter.
+
+## Documentation
+
+* Command line options and operational details: [certspotter(8) man page](man/certspotter.md)
+* The script interface: [certspotter-script(8) man page](man/certspotter-script.md)
+* [Change Log](CHANGELOG.md)
+
+## What certificates are detected by Cert Spotter?
+
+In the default configuration, any certificate that is logged to a Certificate
+Transparency log recognized by Google Chrome or Apple will be detected by
+Cert Spotter.  By default, Google Chrome and Apple only accept certificates that
+are logged, so any certificate that works in Chrome or Safari will be detected
+by Cert Spotter.
+
+Cert Spotter will monitor both traditional RFC6962 logs, and modern static-ct-api logs.
+
+## Security
+
+Cert Spotter assumes an adversarial model in which an attacker produces
+a certificate that is accepted by at least some clients but goes
+undetected because of an encoding error that prevents CT monitors from
+understanding it.  To defend against this attack, Cert Spotter uses a
+special certificate parser that keeps the certificate unparsed except
+for the identifiers.  If one of the identifiers matches a domain on your
+watchlist, you will be notified, even if other parts of the certificate
+are unparsable.
+
+Cert Spotter takes special precautions to ensure identifiers are parsed
+correctly, and implements defenses against identifier-based attacks.
+For instance, if a DNS identifier contains a null byte, Cert Spotter
+interprets it as two identifiers: the complete identifier, and the
+identifier formed by truncating at the first null byte.  For example, a
+certificate for `example.org\0.example.com` will alert the owners of both
+`example.org` and `example.com`.  This defends against [null prefix attacks](
+http://www.thoughtcrime.org/papers/null-prefix-attacks.pdf).
+
+SSLMate continuously monitors CT logs to make sure every certificate's
+identifiers can be successfully parsed, and will release updates to
+Cert Spotter as necessary to fix parsing failures.
+
+Cert Spotter understands wildcard DNS names, and will alert
+you if a wildcard certificate might match an identifier on
+your watchlist.  For example, a watchlist entry for `sub.example.com` would
+match certificates for `*.example.com`.
+
+Cert Spotter is not just a log monitor, but also a log auditor which
+checks that the log is obeying its append-only property.  A future
+release of Cert Spotter will support gossiping with other log monitors
+to ensure the log is presenting a single view.
+
+## Copyright
+
+Copyright © 2016-2025 Opsmate, Inc.
+
+Licensed under the [Mozilla Public License Version 2.0](LICENSE).

asn1.go

@@ -49,10 +49,11 @@ func decodeASN1String(value *asn1.RawValue) (string, error) {
 				return "", errors.New("Malformed UTF8String")
 			}
 			return string(value.Bytes), nil
-		} else if value.Tag == 19 || value.Tag == 22 || value.Tag == 20 {
+		} else if value.Tag == 19 || value.Tag == 22 || value.Tag == 20 || value.Tag == 26 {
 			// * PrintableString - subset of ASCII
 			// * IA5String - ASCII
 			// * TeletexString - 8 bit charset; not quite ISO-8859-1, but often treated as such
+			// * VisibleString - subset of ASCII
 
 			// Don't enforce character set rules. Allow any 8 bit character, since
 			// CAs routinely mess this up

auditing.go

@@ -1,213 +0,0 @@
-// Copyright (C) 2016 Opsmate, Inc.
-//
-// This Source Code Form is subject to the terms of the Mozilla
-// Public License, v. 2.0. If a copy of the MPL was not distributed
-// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
-//
-// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
-// See the Mozilla Public License for details.
-
-package certspotter
-
-import (
-	"bytes"
-	"crypto/sha256"
-	"encoding/json"
-	"errors"
-	"software.sslmate.com/src/certspotter/ct"
-)
-
-func reverseHashes(hashes []ct.MerkleTreeNode) {
-	for i := 0; i < len(hashes)/2; i++ {
-		j := len(hashes) - i - 1
-		hashes[i], hashes[j] = hashes[j], hashes[i]
-	}
-}
-
-func VerifyConsistencyProof(proof ct.ConsistencyProof, first *ct.SignedTreeHead, second *ct.SignedTreeHead) bool {
-	// TODO: make sure every hash in proof is right length? otherwise input to hashChildren is ambiguous
-	if second.TreeSize < first.TreeSize {
-		// Can't be consistent if tree got smaller
-		return false
-	}
-	if first.TreeSize == second.TreeSize {
-		if !(bytes.Equal(first.SHA256RootHash[:], second.SHA256RootHash[:]) && len(proof) == 0) {
-			return false
-		}
-		return true
-	}
-	if first.TreeSize == 0 {
-		// The purpose of the consistency proof is to ensure the append-only
-		// nature of the tree; i.e. that the first tree is a "prefix" of the
-		// second tree.  If the first tree is empty, then it's trivially a prefix
-		// of the second tree, so no proof is needed.
-		if len(proof) != 0 {
-			return false
-		}
-		return true
-	}
-	// Guaranteed that 0 < first.TreeSize < second.TreeSize
-
-	node := first.TreeSize - 1
-	lastNode := second.TreeSize - 1
-
-	// While we're the right child, everything is in both trees, so move one level up.
-	for node%2 == 1 {
-		node /= 2
-		lastNode /= 2
-	}
-
-	var newHash ct.MerkleTreeNode
-	var oldHash ct.MerkleTreeNode
-	if node > 0 {
-		if len(proof) == 0 {
-			return false
-		}
-		newHash = proof[0]
-		proof = proof[1:]
-	} else {
-		// The old tree was balanced, so we already know the first hash to use
-		newHash = first.SHA256RootHash[:]
-	}
-	oldHash = newHash
-
-	for node > 0 {
-		if node%2 == 1 {
-			// node is a right child; left sibling exists in both trees
-			if len(proof) == 0 {
-				return false
-			}
-			newHash = hashChildren(proof[0], newHash)
-			oldHash = hashChildren(proof[0], oldHash)
-			proof = proof[1:]
-		} else if node < lastNode {
-			// node is a left child; rigth sibling only exists in the new tree
-			if len(proof) == 0 {
-				return false
-			}
-			newHash = hashChildren(newHash, proof[0])
-			proof = proof[1:]
-		} // else node == lastNode: node is a left child with no sibling in either tree
-		node /= 2
-		lastNode /= 2
-	}
-
-	if !bytes.Equal(oldHash, first.SHA256RootHash[:]) {
-		return false
-	}
-
-	// If trees have different height, continue up the path to reach the new root
-	for lastNode > 0 {
-		if len(proof) == 0 {
-			return false
-		}
-		newHash = hashChildren(newHash, proof[0])
-		proof = proof[1:]
-		lastNode /= 2
-	}
-
-	if !bytes.Equal(newHash, second.SHA256RootHash[:]) {
-		return false
-	}
-
-	return true
-}
-
-func hashNothing() ct.MerkleTreeNode {
-	return sha256.New().Sum(nil)
-}
-
-func hashLeaf(leafBytes []byte) ct.MerkleTreeNode {
-	hasher := sha256.New()
-	hasher.Write([]byte{0x00})
-	hasher.Write(leafBytes)
-	return hasher.Sum(nil)
-}
-
-func hashChildren(left ct.MerkleTreeNode, right ct.MerkleTreeNode) ct.MerkleTreeNode {
-	hasher := sha256.New()
-	hasher.Write([]byte{0x01})
-	hasher.Write(left)
-	hasher.Write(right)
-	return hasher.Sum(nil)
-}
-
-type CollapsedMerkleTree struct {
-	nodes []ct.MerkleTreeNode
-	size  uint64
-}
-
-func calculateNumNodes(size uint64) int {
-	numNodes := 0
-	for size > 0 {
-		numNodes += int(size & 1)
-		size >>= 1
-	}
-	return numNodes
-}
-func EmptyCollapsedMerkleTree() *CollapsedMerkleTree {
-	return &CollapsedMerkleTree{}
-}
-func NewCollapsedMerkleTree(nodes []ct.MerkleTreeNode, size uint64) (*CollapsedMerkleTree, error) {
-	if len(nodes) != calculateNumNodes(size) {
-		return nil, errors.New("NewCollapsedMerkleTree: nodes has incorrect size")
-	}
-	return &CollapsedMerkleTree{nodes: nodes, size: size}, nil
-}
-func CloneCollapsedMerkleTree(source *CollapsedMerkleTree) *CollapsedMerkleTree {
-	nodes := make([]ct.MerkleTreeNode, len(source.nodes))
-	copy(nodes, source.nodes)
-	return &CollapsedMerkleTree{nodes: nodes, size: source.size}
-}
-
-func (tree *CollapsedMerkleTree) Add(hash ct.MerkleTreeNode) {
-	tree.nodes = append(tree.nodes, hash)
-	tree.size++
-	size := tree.size
-	for size%2 == 0 {
-		left, right := tree.nodes[len(tree.nodes)-2], tree.nodes[len(tree.nodes)-1]
-		tree.nodes = tree.nodes[:len(tree.nodes)-2]
-		tree.nodes = append(tree.nodes, hashChildren(left, right))
-		size /= 2
-	}
-}
-
-func (tree *CollapsedMerkleTree) CalculateRoot() ct.MerkleTreeNode {
-	if len(tree.nodes) == 0 {
-		return hashNothing()
-	}
-	i := len(tree.nodes) - 1
-	hash := tree.nodes[i]
-	for i > 0 {
-		i -= 1
-		hash = hashChildren(tree.nodes[i], hash)
-	}
-	return hash
-}
-
-func (tree *CollapsedMerkleTree) GetSize() uint64 {
-	return tree.size
-}
-
-func (tree *CollapsedMerkleTree) MarshalJSON() ([]byte, error) {
-	return json.Marshal(map[string]interface{}{
-		"nodes": tree.nodes,
-		"size":  tree.size,
-	})
-}
-
-func (tree *CollapsedMerkleTree) UnmarshalJSON(b []byte) error {
-	var rawTree struct {
-		Nodes []ct.MerkleTreeNode `json:"nodes"`
-		Size  uint64              `json:"size"`
-	}
-	if err := json.Unmarshal(b, &rawTree); err != nil {
-		return errors.New("Failed to unmarshal CollapsedMerkleTree: " + err.Error())
-	}
-	if len(rawTree.Nodes) != calculateNumNodes(rawTree.Size) {
-		return errors.New("Failed to unmarshal CollapsedMerkleTree: nodes has incorrect length")
-	}
-	tree.size = rawTree.Size
-	tree.nodes = rawTree.Nodes
-	return nil
-}

canonicalize.go

@@ -0,0 +1,92 @@
+// Copyright (C) 2019 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package certspotter
+
+import (
+	"encoding/asn1"
+)
+
+func canonicalizeRDNString(fromStr string) string {
+	from := []byte(fromStr)
+	to := []byte{}
+	inWhitespace := true
+	for _, ch := range from {
+		if ch == ' ' || ch == '\f' || ch == '\n' || ch == '\r' || ch == '\t' || ch == '\v' {
+			if !inWhitespace {
+				to = append(to, ' ')
+			}
+			inWhitespace = true
+		} else {
+			if ch >= 'A' && ch <= 'Z' {
+				to = append(to, ch+32) // convert to lowercase
+			} else {
+				to = append(to, ch)
+			}
+			inWhitespace = false
+		}
+	}
+	if inWhitespace && len(to) > 0 {
+		// whack off the space character that we appended
+		to = to[:len(to)-1]
+	}
+	return string(to)
+}
+
+func shouldCanonicalizeASN1String(value *asn1.RawValue) bool {
+	if !value.IsCompound && value.Class == 0 {
+		return value.Tag == 12 || value.Tag == 19 || value.Tag == 22 || value.Tag == 20 || value.Tag == 26 || value.Tag == 30 || value.Tag == 28
+	}
+	return false
+}
+
+func canonicalizeATV(oldATV AttributeTypeAndValue) (AttributeTypeAndValue, error) {
+	if shouldCanonicalizeASN1String(&oldATV.Value) {
+		str, err := decodeASN1String(&oldATV.Value)
+		if err != nil {
+			return AttributeTypeAndValue{}, err
+		}
+		str = canonicalizeRDNString(str)
+		return AttributeTypeAndValue{
+			Type: oldATV.Type,
+			Value: asn1.RawValue{
+				Class:      0,
+				Tag:        asn1.TagUTF8String,
+				IsCompound: false,
+				Bytes:      []byte(str),
+			},
+		}, nil
+	} else {
+		return oldATV, nil
+	}
+}
+
+func canonicalizeRDNSet(oldSet RelativeDistinguishedNameSET) (RelativeDistinguishedNameSET, error) {
+	newSet := make([]AttributeTypeAndValue, len(oldSet))
+	for i := range oldSet {
+		var err error
+		newSet[i], err = canonicalizeATV(oldSet[i])
+		if err != nil {
+			return nil, err
+		}
+	}
+	return newSet, nil
+}
+
+func CanonicalizeRDNSequence(oldSequence RDNSequence) (RDNSequence, error) {
+	newSequence := make([]RelativeDistinguishedNameSET, len(oldSequence))
+	for i := range oldSequence {
+		var err error
+		newSequence[i], err = canonicalizeRDNSet(oldSequence[i])
+		if err != nil {
+			return nil, err
+		}
+	}
+	return newSequence, nil
+}

canonicalize_test.go

@@ -0,0 +1,49 @@
+// Copyright (C) 2019 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package certspotter
+
+import (
+	"testing"
+)
+
+type stringCanonTest struct {
+	in  string
+	out string
+}
+
+var stringCanonTests = []stringCanonTest{
+	{"", ""},
+	{" ", ""},
+	{"  ", ""},
+	{"abc", "abc"},
+	{"aBc", "abc"},
+	{"ab c", "ab c"},
+	{"ab  c", "ab c"},
+	{"ab\n c", "ab c"},
+	{" ab c ", "ab c"},
+	{" ab  c ", "ab c"},
+	{" ab  c", "ab c"},
+	{"ab  c ", "ab c"},
+	{"abc ", "abc"},
+	{"abc  ", "abc"},
+	{"  abc  ", "abc"},
+	{"  abc ", "abc"},
+	{"  abc", "abc"},
+	{"  aBc de  f      g\n", "abc de f g"},
+}
+
+func TestCanonicalizeRDNString(t *testing.T) {
+	for i, test := range stringCanonTests {
+		ret := canonicalizeRDNString(test.in)
+		if test.out != ret {
+			t.Errorf("#%d: canonicalizeRDNString(%q) = %q, want %q", i, test.in, ret, test.out)
+		}
+	}
+}

cmd/certspotter/main.go

@@ -1,4 +1,4 @@
-// Copyright (C) 2016 Opsmate, Inc.
+// Copyright (C) 2016, 2023 Opsmate, Inc.
 //
 // This Source Code Form is subject to the terms of the Mozilla
 // Public License, v. 2.0. If a copy of the MPL was not distributed
@@ -11,185 +11,250 @@ package main
 
 import (
 	"bufio"
+	"context"
+	"errors"
 	"flag"
 	"fmt"
-	"io"
+	"io/fs"
 	"os"
+	"os/signal"
 	"path/filepath"
+	"runtime"
+	"runtime/debug"
 	"strings"
+	"syscall"
+	"time"
 
-	"golang.org/x/net/idna"
+	"software.sslmate.com/src/certspotter/loglist"
-
+	"software.sslmate.com/src/certspotter/monitor"
-	"software.sslmate.com/src/certspotter"
-	"software.sslmate.com/src/certspotter/cmd"
-	"software.sslmate.com/src/certspotter/ct"
 )
 
+var programName = os.Args[0]
+var Version = ""
+
+const defaultLogList = "https://loglist.certspotter.org/monitor.json"
+
+func certspotterVersion() string {
+	if Version != "" {
+		return Version + "?"
+	}
+	info, ok := debug.ReadBuildInfo()
+	if !ok {
+		return "unknown"
+	}
+	if strings.HasPrefix(info.Main.Version, "v") {
+		return info.Main.Version
+	}
+	var vcs, vcsRevision, vcsModified string
+	for _, s := range info.Settings {
+		switch s.Key {
+		case "vcs":
+			vcs = s.Value
+		case "vcs.revision":
+			vcsRevision = s.Value
+		case "vcs.modified":
+			vcsModified = s.Value
+		}
+	}
+	if vcs == "git" && vcsRevision != "" && vcsModified == "true" {
+		return vcsRevision + "+"
+	} else if vcs == "git" && vcsRevision != "" {
+		return vcsRevision
+	}
+	return "unknown"
+}
+
+func fileExists(filename string) bool {
+	_, err := os.Lstat(filename)
+	return err == nil
+}
+func homedir() string {
+	homedir, err := os.UserHomeDir()
+	if err != nil {
+		panic(fmt.Errorf("unable to determine home directory: %w", err))
+	}
+	return homedir
+}
 func defaultStateDir() string {
 	if envVar := os.Getenv("CERTSPOTTER_STATE_DIR"); envVar != "" {
 		return envVar
 	} else {
-		return cmd.DefaultStateDir("certspotter")
+		return filepath.Join(homedir(), ".certspotter")
 	}
 }
 func defaultConfigDir() string {
 	if envVar := os.Getenv("CERTSPOTTER_CONFIG_DIR"); envVar != "" {
 		return envVar
 	} else {
-		return cmd.DefaultConfigDir("certspotter")
+		return filepath.Join(homedir(), ".certspotter")
 	}
 }
-
+func defaultCacheDir() string {
-func trimTrailingDots(value string) string {
+	userCacheDir, err := os.UserCacheDir()
-	length := len(value)
-	for length > 0 && value[length-1] == '.' {
-		length--
-	}
-	return value[0:length]
-}
-
-var stateDir = flag.String("state_dir", defaultStateDir(), "Directory for storing state")
-var watchlistFilename = flag.String("watchlist", filepath.Join(defaultConfigDir(), "watchlist"), "File containing identifiers to watch (- for stdin)")
-
-type watchlistItem struct {
-	Domain       []string
-	AcceptSuffix bool
-}
-
-var watchlist []watchlistItem
-
-func parseWatchlistItem(str string) (watchlistItem, error) {
-	if str == "." { // "." as in root zone (matches everything)
-		return watchlistItem{
-			Domain:       []string{},
-			AcceptSuffix: true,
-		}, nil
-	} else {
-		acceptSuffix := false
-		if strings.HasPrefix(str, ".") {
-			acceptSuffix = true
-			str = str[1:]
-		}
-		asciiDomain, err := idna.ToASCII(strings.ToLower(trimTrailingDots(str)))
 	if err != nil {
-			return watchlistItem{}, fmt.Errorf("Invalid domain `%s': %s", str, err)
+		panic(fmt.Errorf("unable to determine user cache directory: %w", err))
 	}
-		return watchlistItem{
+	return filepath.Join(userCacheDir, "certspotter")
-			Domain:       strings.Split(asciiDomain, "."),
+}
-			AcceptSuffix: acceptSuffix,
+func defaultWatchListPath() string {
-		}, nil
+	return filepath.Join(defaultConfigDir(), "watchlist")
+}
+func defaultWatchListPathIfExists() string {
+	if fileExists(defaultWatchListPath()) {
+		return defaultWatchListPath()
+	} else {
+		return ""
 	}
 }
+func defaultScriptDir() string {
+	return filepath.Join(defaultConfigDir(), "hooks.d")
+}
+func defaultEmailFile() string {
+	return filepath.Join(defaultConfigDir(), "email_recipients")
+}
 
-func readWatchlist(reader io.Reader) ([]watchlistItem, error) {
+func simplifyError(err error) error {
-	items := []watchlistItem{}
+	var pathErr *fs.PathError
-	scanner := bufio.NewScanner(reader)
+	if errors.As(err, &pathErr) {
+		return pathErr.Err
+	}
+
+	return err
+}
+
+func readWatchListFile(filename string) (monitor.WatchList, error) {
+	file, err := os.Open(filename)
+	if err != nil {
+		return nil, simplifyError(err)
+	}
+	defer file.Close()
+	return monitor.ReadWatchList(file)
+}
+
+func readEmailFile(filename string) ([]string, error) {
+	file, err := os.Open(filename)
+	if err != nil {
+		return nil, simplifyError(err)
+	}
+	defer file.Close()
+
+	var emails []string
+	scanner := bufio.NewScanner(file)
 	for scanner.Scan() {
 		line := scanner.Text()
-		if line == "" || strings.HasPrefix(line, "#") {
+		if line == "" {
 			continue
 		}
-		item, err := parseWatchlistItem(line)
+		emails = append(emails, line)
-		if err != nil {
-			return nil, err
 	}
-		items = append(items, item)
+	return emails, err
-	}
-	return items, scanner.Err()
 }
 
-func dnsLabelMatches(certLabel string, watchLabel string) bool {
+func appendFunc(slice *[]string) func(string) error {
-	// For fail-safe behavior, if a label was unparsable, it matches everything.
+	return func(value string) error {
-	// Similarly, redacted labels match everything, since the label _might_ be
+		*slice = append(*slice, value)
-	// for a name we're interested in.
+		return nil
-
-	return certLabel == "*" ||
-		certLabel == "?" ||
-		certLabel == certspotter.UnparsableDNSLabelPlaceholder ||
-		certspotter.MatchesWildcard(watchLabel, certLabel)
-}
-
-func dnsNameMatches(dnsName []string, watchDomain []string, acceptSuffix bool) bool {
-	for len(dnsName) > 0 && len(watchDomain) > 0 {
-		certLabel := dnsName[len(dnsName)-1]
-		watchLabel := watchDomain[len(watchDomain)-1]
-
-		if !dnsLabelMatches(certLabel, watchLabel) {
-			return false
-		}
-
-		dnsName = dnsName[:len(dnsName)-1]
-		watchDomain = watchDomain[:len(watchDomain)-1]
-	}
-
-	return len(watchDomain) == 0 && (acceptSuffix || len(dnsName) == 0)
-}
-
-func dnsNameIsWatched(dnsName string) bool {
-	labels := strings.Split(dnsName, ".")
-	for _, item := range watchlist {
-		if dnsNameMatches(labels, item.Domain, item.AcceptSuffix) {
-			return true
-		}
-	}
-	return false
-}
-
-func anyDnsNameIsWatched(dnsNames []string) bool {
-	for _, dnsName := range dnsNames {
-		if dnsNameIsWatched(dnsName) {
-			return true
-		}
-	}
-	return false
-}
-
-func processEntry(scanner *certspotter.Scanner, entry *ct.LogEntry) {
-	info := certspotter.EntryInfo{
-		LogUri:    scanner.LogUri,
-		Entry:     entry,
-		IsPrecert: certspotter.IsPrecert(entry),
-		FullChain: certspotter.GetFullChain(entry),
-	}
-
-	info.CertInfo, info.ParseError = certspotter.MakeCertInfoFromLogEntry(entry)
-
-	if info.CertInfo != nil {
-		info.Identifiers, info.IdentifiersParseError = info.CertInfo.ParseIdentifiers()
-	}
-
-	// Fail safe behavior: if info.Identifiers is nil (which is caused by a
-	// parse error), report the certificate because we can't say for sure it
-	// doesn't match a domain we care about.  We try very hard to make sure
-	// parsing identifiers always succeeds, so false alarms should be rare.
-	if info.Identifiers == nil || anyDnsNameIsWatched(info.Identifiers.DNSNames) {
-		cmd.LogEntry(&info)
 	}
 }
 
 func main() {
+	loglist.UserAgent = fmt.Sprintf("certspotter/%s (%s; %s; %s)", certspotterVersion(), runtime.Version(), runtime.GOOS, runtime.GOARCH)
+
+	var flags struct {
+		batchSize   int // TODO-4: respect this option
+		email       []string
+		healthcheck time.Duration
+		logs        string
+		noSave      bool
+		script      string
+		startAtEnd  bool
+		stateDir    string
+		stdout      bool
+		verbose     bool
+		version     bool
+		watchlist   string
+	}
+	flag.IntVar(&flags.batchSize, "batch_size", 1000, "Max number of entries to request per call to get-entries (advanced)")
+	flag.Func("email", "Email address to contact when matching certificate is discovered (repeatable)", appendFunc(&flags.email))
+	flag.DurationVar(&flags.healthcheck, "healthcheck", 24*time.Hour, "How frequently to perform a health check")
+	flag.StringVar(&flags.logs, "logs", defaultLogList, "File path or URL of JSON list of logs to monitor")
+	flag.BoolVar(&flags.noSave, "no_save", false, "Do not save a copy of matching certificates in state directory")
+	flag.StringVar(&flags.script, "script", "", "Program to execute when a matching certificate is discovered")
+	flag.BoolVar(&flags.startAtEnd, "start_at_end", false, "Start monitoring new logs from the end rather than the beginning (saves considerable bandwidth)")
+	flag.StringVar(&flags.stateDir, "state_dir", defaultStateDir(), "Directory for storing log position and discovered certificates")
+	flag.BoolVar(&flags.stdout, "stdout", false, "Write matching certificates to stdout")
+	flag.BoolVar(&flags.verbose, "verbose", false, "Be verbose")
+	flag.BoolVar(&flags.version, "version", false, "Print version and exit")
+	flag.StringVar(&flags.watchlist, "watchlist", defaultWatchListPathIfExists(), "File containing domain names to watch")
 	flag.Parse()
 
-	if *watchlistFilename == "-" {
+	if flags.version {
-		var err error
+		fmt.Fprintf(os.Stdout, "certspotter version %s\n", certspotterVersion())
-		watchlist, err = readWatchlist(os.Stdin)
+		os.Exit(0)
-		if err != nil {
-			fmt.Fprintf(os.Stderr, "%s: (stdin): %s\n", os.Args[0], err)
-			os.Exit(1)
-		}
-	} else {
-		file, err := os.Open(*watchlistFilename)
-		if err != nil {
-			fmt.Fprintf(os.Stderr, "%s: %s: %s\n", os.Args[0], *watchlistFilename, err)
-			os.Exit(1)
-		}
-		defer file.Close()
-		watchlist, err = readWatchlist(file)
-		if err != nil {
-			fmt.Fprintf(os.Stderr, "%s: %s: %s\n", os.Args[0], *watchlistFilename, err)
-			os.Exit(1)
 	}
+	if flags.watchlist == "" {
+		fmt.Fprintf(os.Stderr, "%s: watch list not found: please create %s or specify alternative path using -watchlist\n", programName, defaultWatchListPath())
+		os.Exit(2)
 	}
 
-	os.Exit(cmd.Main(*stateDir, processEntry))
+	fsstate := &monitor.FilesystemState{
+		StateDir:  flags.stateDir,
+		CacheDir:  defaultCacheDir(),
+		SaveCerts: !flags.noSave,
+		Script:    flags.script,
+		ScriptDir: defaultScriptDir(),
+		Email:     flags.email,
+		Stdout:    flags.stdout,
+	}
+	config := &monitor.Config{
+		LogListSource:       flags.logs,
+		State:               fsstate,
+		StartAtEnd:          flags.startAtEnd,
+		Verbose:             flags.verbose,
+		HealthCheckInterval: flags.healthcheck,
+	}
+
+	emailFileExists := false
+	if emailRecipients, err := readEmailFile(defaultEmailFile()); err == nil {
+		emailFileExists = true
+		fsstate.Email = append(fsstate.Email, emailRecipients...)
+	} else if !errors.Is(err, fs.ErrNotExist) {
+		fmt.Fprintf(os.Stderr, "%s: error reading email recipients file %q: %s\n", programName, defaultEmailFile(), err)
+		os.Exit(1)
+	}
+
+	if len(fsstate.Email) == 0 && !emailFileExists && fsstate.Script == "" && !fileExists(fsstate.ScriptDir) && fsstate.Stdout == false {
+		fmt.Fprintf(os.Stderr, "%s: no notification methods were specified\n", programName)
+		fmt.Fprintf(os.Stderr, "Please specify at least one of the following notification methods:\n")
+		fmt.Fprintf(os.Stderr, " - Place one or more email addresses in %s (one address per line)\n", defaultEmailFile())
+		fmt.Fprintf(os.Stderr, " - Place one or more executable scripts in the %s directory\n", fsstate.ScriptDir)
+		fmt.Fprintf(os.Stderr, " - Specify an email address using the -email flag\n")
+		fmt.Fprintf(os.Stderr, " - Specify the path to an executable script using the -script flag\n")
+		fmt.Fprintf(os.Stderr, " - Specify the -stdout flag\n")
+		os.Exit(2)
+	}
+
+	if flags.watchlist == "-" {
+		watchlist, err := monitor.ReadWatchList(os.Stdin)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "%s: error reading watchlist from standard in: %s\n", programName, err)
+			os.Exit(1)
+		}
+		config.WatchList = watchlist
+	} else {
+		watchlist, err := readWatchListFile(flags.watchlist)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "%s: error reading watchlist from %q: %s\n", programName, flags.watchlist, err)
+			os.Exit(1)
+		}
+		config.WatchList = watchlist
+	}
+
+	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
+	defer stop()
+
+	if err := monitor.Run(ctx, config); err != nil && !errors.Is(err, context.Canceled) {
+		fmt.Fprintf(os.Stderr, "%s: %s\n", programName, err)
+		os.Exit(1)
+	}
 }

cmd/common.go

@@ -1,351 +0,0 @@
-// Copyright (C) 2016-2017 Opsmate, Inc.
-//
-// This Source Code Form is subject to the terms of the Mozilla
-// Public License, v. 2.0. If a copy of the MPL was not distributed
-// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
-//
-// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
-// See the Mozilla Public License for details.
-
-package cmd
-
-import (
-	"bytes"
-	"flag"
-	"fmt"
-	"log"
-	"os"
-	"os/user"
-	"path/filepath"
-	"sync"
-
-	"software.sslmate.com/src/certspotter"
-	"software.sslmate.com/src/certspotter/ct"
-)
-
-var batchSize = flag.Int("batch_size", 1000, "Max number of entries to request at per call to get-entries (advanced)")
-var numWorkers = flag.Int("num_workers", 2, "Number of concurrent matchers (advanced)")
-var script = flag.String("script", "", "Script to execute when a matching certificate is found")
-var logsFilename = flag.String("logs", "", "JSON file containing log information")
-var underwater = flag.Bool("underwater", false, "Monitor certificates from distrusted CAs instead of trusted CAs")
-var noSave = flag.Bool("no_save", false, "Do not save a copy of matching certificates")
-var verbose = flag.Bool("verbose", false, "Be verbose")
-var allTime = flag.Bool("all_time", false, "Scan certs from all time, not just since last scan")
-var state *State
-
-var printMutex sync.Mutex
-
-func homedir() string {
-	home := os.Getenv("HOME")
-	if home != "" {
-		return home
-	}
-	user, err := user.Current()
-	if err == nil {
-		return user.HomeDir
-	}
-	panic("Unable to determine home directory")
-}
-
-func DefaultStateDir(programName string) string {
-	return filepath.Join(homedir(), "."+programName)
-}
-
-func DefaultConfigDir(programName string) string {
-	return filepath.Join(homedir(), "."+programName)
-}
-
-func LogEntry(info *certspotter.EntryInfo) {
-	if !*noSave {
-		var alreadyPresent bool
-		var err error
-		alreadyPresent, info.Filename, err = state.SaveCert(info.IsPrecert, info.FullChain)
-		if err != nil {
-			log.Print(err)
-		}
-		if alreadyPresent {
-			return
-		}
-	}
-
-	if *script != "" {
-		if err := info.InvokeHookScript(*script); err != nil {
-			log.Print(err)
-		}
-	} else {
-		printMutex.Lock()
-		info.Write(os.Stdout)
-		fmt.Fprintf(os.Stdout, "\n")
-		printMutex.Unlock()
-	}
-}
-
-func loadLogList() ([]certspotter.LogInfo, error) {
-	if *logsFilename != "" {
-		var logFileObj certspotter.LogInfoFile
-		if err := readJSONFile(*logsFilename, &logFileObj); err != nil {
-			return nil, fmt.Errorf("Error reading logs file: %s: %s", *logsFilename, err)
-		}
-		return logFileObj.Logs, nil
-	} else if *underwater {
-		return certspotter.UnderwaterLogs, nil
-	} else {
-		return certspotter.DefaultLogs, nil
-	}
-}
-
-type logHandle struct {
-	scanner     *certspotter.Scanner
-	state       *LogState
-	tree        *certspotter.CollapsedMerkleTree
-	verifiedSTH *ct.SignedTreeHead
-}
-
-func makeLogHandle(logInfo *certspotter.LogInfo) (*logHandle, error) {
-	ctlog := new(logHandle)
-
-	logKey, err := logInfo.ParsedPublicKey()
-	if err != nil {
-		return nil, fmt.Errorf("Bad public key: %s", err)
-	}
-	ctlog.scanner = certspotter.NewScanner(logInfo.FullURI(), logInfo.ID(), logKey, &certspotter.ScannerOptions{
-		BatchSize:  *batchSize,
-		NumWorkers: *numWorkers,
-		Quiet:      !*verbose,
-	})
-
-	ctlog.state, err = state.OpenLogState(logInfo)
-	if err != nil {
-		return nil, fmt.Errorf("Error opening state directory: %s", err)
-	}
-	ctlog.tree, err = ctlog.state.GetTree()
-	if err != nil {
-		return nil, fmt.Errorf("Error loading tree: %s", err)
-	}
-	ctlog.verifiedSTH, err = ctlog.state.GetVerifiedSTH()
-	if err != nil {
-		return nil, fmt.Errorf("Error loading verified STH: %s", err)
-	}
-
-	if ctlog.tree == nil && ctlog.verifiedSTH == nil { // This branch can be removed eventually
-		legacySTH, err := state.GetLegacySTH(logInfo)
-		if err != nil {
-			return nil, fmt.Errorf("Error loading legacy STH: %s", err)
-		}
-		if legacySTH != nil {
-			log.Printf("Initializing log state from legacy state directory")
-			ctlog.tree, err = ctlog.scanner.MakeCollapsedMerkleTree(legacySTH)
-			if err != nil {
-				return nil, fmt.Errorf("Error reconstructing Merkle Tree for legacy STH: %s", err)
-			}
-			if err := ctlog.state.StoreTree(ctlog.tree); err != nil {
-				return nil, fmt.Errorf("Error storing tree: %s", err)
-			}
-			if err := ctlog.state.StoreVerifiedSTH(legacySTH); err != nil {
-				return nil, fmt.Errorf("Error storing verified STH: %s", err)
-			}
-			state.RemoveLegacySTH(logInfo)
-		}
-	}
-
-	return ctlog, nil
-}
-
-func (ctlog *logHandle) refresh() error {
-	if *verbose {
-		log.Printf("Retrieving latest STH from log")
-	}
-	latestSTH, err := ctlog.scanner.GetSTH()
-	if err != nil {
-		return fmt.Errorf("Error retrieving STH from log: %s", err)
-	}
-	if ctlog.verifiedSTH == nil {
-		if *verbose {
-			log.Printf("No existing STH is known; presuming latest STH (%d) is valid", latestSTH.TreeSize)
-		}
-		ctlog.verifiedSTH = latestSTH
-		if err := ctlog.state.StoreVerifiedSTH(ctlog.verifiedSTH); err != nil {
-			return fmt.Errorf("Error storing verified STH: %s", err)
-		}
-	} else {
-		if err := ctlog.state.StoreUnverifiedSTH(latestSTH); err != nil {
-			return fmt.Errorf("Error storing unverified STH: %s", err)
-		}
-	}
-	return nil
-}
-
-func (ctlog *logHandle) verifySTH(sth *ct.SignedTreeHead) error {
-	isValid, err := ctlog.scanner.CheckConsistency(ctlog.verifiedSTH, sth)
-	if err != nil {
-		return fmt.Errorf("Error fetching consistency proof: %s", err)
-	}
-	if !isValid {
-		return fmt.Errorf("Consistency proof between %d and %d is invalid", ctlog.verifiedSTH.TreeSize, sth.TreeSize)
-	}
-	return nil
-}
-
-func (ctlog *logHandle) audit() error {
-	sths, err := ctlog.state.GetUnverifiedSTHs()
-	if err != nil {
-		return fmt.Errorf("Error loading unverified STHs: %s", err)
-	}
-
-	for _, sth := range sths {
-		if *verbose {
-			log.Printf("Verifying consistency of STH %d (%x) with previously-verified STH %d (%x)", sth.TreeSize, sth.SHA256RootHash, ctlog.verifiedSTH.TreeSize, ctlog.verifiedSTH.SHA256RootHash)
-		}
-		if err := ctlog.verifySTH(sth); err != nil {
-			log.Printf("Unable to verify consistency of STH %d (%s) (if this error persists, it should be construed as misbehavior by the log): %s", sth.TreeSize, ctlog.state.UnverifiedSTHFilename(sth), err)
-			continue
-		}
-		if sth.TreeSize > ctlog.verifiedSTH.TreeSize {
-			if *verbose {
-				log.Printf("STH %d (%x) is now the latest verified STH", sth.TreeSize, sth.SHA256RootHash)
-			}
-			ctlog.verifiedSTH = sth
-			if err := ctlog.state.StoreVerifiedSTH(ctlog.verifiedSTH); err != nil {
-				return fmt.Errorf("Error storing verified STH: %s", err)
-			}
-		}
-		if err := ctlog.state.RemoveUnverifiedSTH(sth); err != nil {
-			return fmt.Errorf("Error removing redundant STH: %s", err)
-		}
-	}
-
-	return nil
-}
-
-func (ctlog *logHandle) scan(processCallback certspotter.ProcessCallback) error {
-	startIndex := int64(ctlog.tree.GetSize())
-	endIndex := int64(ctlog.verifiedSTH.TreeSize)
-
-	if endIndex > startIndex {
-		tree := certspotter.CloneCollapsedMerkleTree(ctlog.tree)
-
-		if err := ctlog.scanner.Scan(startIndex, endIndex, processCallback, tree); err != nil {
-			return fmt.Errorf("Error scanning log (if this error persists, it should be construed as misbehavior by the log): %s", err)
-		}
-
-		rootHash := tree.CalculateRoot()
-		if !bytes.Equal(rootHash, ctlog.verifiedSTH.SHA256RootHash[:]) {
-			return fmt.Errorf("Log has misbehaved: log entries at tree size %d do not correspond to signed tree root", ctlog.verifiedSTH.TreeSize)
-		}
-
-		ctlog.tree = tree
-		if err := ctlog.state.StoreTree(ctlog.tree); err != nil {
-			return fmt.Errorf("Error storing tree: %s", err)
-		}
-	}
-
-	return nil
-}
-
-func processLog(logInfo *certspotter.LogInfo, processCallback certspotter.ProcessCallback) int {
-	log.SetPrefix(os.Args[0] + ": " + logInfo.Url + ": ")
-
-	ctlog, err := makeLogHandle(logInfo)
-	if err != nil {
-		log.Printf("%s\n", err)
-		return 1
-	}
-
-	if err := ctlog.refresh(); err != nil {
-		log.Printf("%s\n", err)
-		return 1
-	}
-
-	if err := ctlog.audit(); err != nil {
-		log.Printf("%s\n", err)
-		return 1
-	}
-
-	if *allTime {
-		ctlog.tree = certspotter.EmptyCollapsedMerkleTree()
-		if *verbose {
-			log.Printf("Scanning all %d entries in the log because -all_time option specified", ctlog.verifiedSTH.TreeSize)
-		}
-	} else if ctlog.tree != nil {
-		if *verbose {
-			log.Printf("Existing log; scanning %d new entries since previous scan", ctlog.verifiedSTH.TreeSize-ctlog.tree.GetSize())
-		}
-	} else if state.IsFirstRun() {
-		ctlog.tree, err = ctlog.scanner.MakeCollapsedMerkleTree(ctlog.verifiedSTH)
-		if err != nil {
-			log.Printf("Error reconstructing Merkle Tree: %s", err)
-			return 1
-		}
-		if *verbose {
-			log.Printf("First run of Cert Spotter; not scanning %d existing entries because -all_time option not specified", ctlog.verifiedSTH.TreeSize)
-		}
-	} else {
-		ctlog.tree = certspotter.EmptyCollapsedMerkleTree()
-		if *verbose {
-			log.Printf("New log; scanning all %d entries in the log", ctlog.verifiedSTH.TreeSize)
-		}
-	}
-	if err := ctlog.state.StoreTree(ctlog.tree); err != nil {
-		log.Printf("Error storing tree: %s\n", err)
-		return 1
-	}
-
-	if err := ctlog.scan(processCallback); err != nil {
-		log.Printf("%s\n", err)
-		return 1
-	}
-
-	if *verbose {
-		log.Printf("Final log size = %d, final root hash = %x", ctlog.verifiedSTH.TreeSize, ctlog.verifiedSTH.SHA256RootHash)
-	}
-
-	return 0
-}
-
-func Main(statePath string, processCallback certspotter.ProcessCallback) int {
-	var err error
-
-	logs, err := loadLogList()
-	if err != nil {
-		fmt.Fprintf(os.Stderr, "%s: %s\n", os.Args[0], err)
-		return 1
-	}
-
-	state, err = OpenState(statePath)
-	if err != nil {
-		fmt.Fprintf(os.Stderr, "%s: %s\n", os.Args[0], err)
-		return 1
-	}
-	locked, err := state.Lock()
-	if err != nil {
-		fmt.Fprintf(os.Stderr, "%s: Error locking state directory: %s\n", os.Args[0], err)
-		return 1
-	}
-	if !locked {
-		var otherPidInfo string
-		if otherPid := state.LockingPid(); otherPid != 0 {
-			otherPidInfo = fmt.Sprintf(" (as process ID %d)", otherPid)
-		}
-		fmt.Fprintf(os.Stderr, "%s: Another instance of %s is already running%s; remove the file %s if this is not the case\n", os.Args[0], os.Args[0], otherPidInfo, state.LockFilename())
-		return 1
-	}
-
-	exitCode := 0
-	for i := range logs {
-		exitCode |= processLog(&logs[i], processCallback)
-	}
-
-	if state.IsFirstRun() && exitCode == 0 {
-		if err := state.WriteOnceFile(); err != nil {
-			fmt.Fprintf(os.Stderr, "%s: Error writing once file: %s\n", os.Args[0], err)
-			exitCode |= 1
-		}
-	}
-
-	if err := state.Unlock(); err != nil {
-		fmt.Fprintf(os.Stderr, "%s: Error unlocking state directory: %s\n", os.Args[0], err)
-		exitCode |= 1
-	}
-
-	return exitCode
-}

cmd/ctparsewatch/.gitignore

@@ -1 +0,0 @@
-/ctparsewatch

cmd/ctparsewatch/main.go

@@ -1,52 +0,0 @@
-// Copyright (C) 2016 Opsmate, Inc.
-//
-// This Source Code Form is subject to the terms of the Mozilla
-// Public License, v. 2.0. If a copy of the MPL was not distributed
-// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
-//
-// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
-// See the Mozilla Public License for details.
-
-package main
-
-import (
-	"flag"
-	"os"
-
-	"software.sslmate.com/src/certspotter"
-	"software.sslmate.com/src/certspotter/cmd"
-	"software.sslmate.com/src/certspotter/ct"
-)
-
-func DefaultStateDir() string {
-	if envVar := os.Getenv("CTPARSEWATCH_STATE_DIR"); envVar != "" {
-		return envVar
-	} else {
-		return cmd.DefaultStateDir("ctparsewatch")
-	}
-}
-
-var stateDir = flag.String("state_dir", DefaultStateDir(), "Directory for storing state")
-
-func processEntry(scanner *certspotter.Scanner, entry *ct.LogEntry) {
-	info := certspotter.EntryInfo{
-		LogUri:    scanner.LogUri,
-		Entry:     entry,
-		IsPrecert: certspotter.IsPrecert(entry),
-		FullChain: certspotter.GetFullChain(entry),
-	}
-
-	info.CertInfo, info.ParseError = certspotter.MakeCertInfoFromLogEntry(entry)
-	if info.CertInfo != nil {
-		info.Identifiers, info.IdentifiersParseError = info.CertInfo.ParseIdentifiers()
-	}
-
-	if info.HasParseErrors() {
-		cmd.LogEntry(&info)
-	}
-}
-
-func main() {
-	flag.Parse()
-	os.Exit(cmd.Main(*stateDir, processEntry))
-}

cmd/helpers.go

@@ -1,87 +0,0 @@
-// Copyright (C) 2017 Opsmate, Inc.
-//
-// This Source Code Form is subject to the terms of the Mozilla
-// Public License, v. 2.0. If a copy of the MPL was not distributed
-// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
-//
-// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
-// See the Mozilla Public License for details.
-
-package cmd
-
-import (
-	"crypto/sha256"
-	"encoding/hex"
-	"encoding/json"
-	"io/ioutil"
-	"os"
-
-	"software.sslmate.com/src/certspotter/ct"
-)
-
-func fileExists(path string) bool {
-	_, err := os.Lstat(path)
-	return err == nil
-}
-
-func writeFile(filename string, data []byte, perm os.FileMode) error {
-	tempname := filename + ".new"
-	if err := ioutil.WriteFile(tempname, data, perm); err != nil {
-		return err
-	}
-	if err := os.Rename(tempname, filename); err != nil {
-		os.Remove(tempname)
-		return err
-	}
-	return nil
-}
-
-func writeJSONFile(filename string, obj interface{}, perm os.FileMode) error {
-	tempname := filename + ".new"
-	f, err := os.OpenFile(tempname, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, perm)
-	if err != nil {
-		return err
-	}
-	if err := json.NewEncoder(f).Encode(obj); err != nil {
-		f.Close()
-		os.Remove(tempname)
-		return err
-	}
-	if err := f.Close(); err != nil {
-		os.Remove(tempname)
-		return err
-	}
-	if err := os.Rename(tempname, filename); err != nil {
-		os.Remove(tempname)
-		return err
-	}
-	return nil
-}
-
-func readJSONFile(filename string, obj interface{}) error {
-	bytes, err := ioutil.ReadFile(filename)
-	if err != nil {
-		return err
-	}
-	if err = json.Unmarshal(bytes, obj); err != nil {
-		return err
-	}
-	return nil
-}
-
-func readSTHFile(filename string) (*ct.SignedTreeHead, error) {
-	sth := new(ct.SignedTreeHead)
-	if err := readJSONFile(filename, sth); err != nil {
-		return nil, err
-	}
-	return sth, nil
-}
-
-func sha256sum(data []byte) []byte {
-	sum := sha256.Sum256(data)
-	return sum[:]
-}
-
-func sha256hex(data []byte) string {
-	return hex.EncodeToString(sha256sum(data))
-}

cmd/log_state.go

@@ -1,145 +0,0 @@
-// Copyright (C) 2017 Opsmate, Inc.
-//
-// This Source Code Form is subject to the terms of the Mozilla
-// Public License, v. 2.0. If a copy of the MPL was not distributed
-// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
-//
-// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
-// See the Mozilla Public License for details.
-
-package cmd
-
-import (
-	"crypto/sha256"
-	"encoding/base64"
-	"encoding/binary"
-	"fmt"
-	"os"
-	"path/filepath"
-	"strconv"
-	"strings"
-
-	"software.sslmate.com/src/certspotter"
-	"software.sslmate.com/src/certspotter/ct"
-)
-
-type LogState struct {
-	path string
-}
-
-// generate a filename that uniquely identifies the STH (within the context of a particular log)
-func sthFilename(sth *ct.SignedTreeHead) string {
-	hasher := sha256.New()
-	switch sth.Version {
-	case ct.V1:
-		binary.Write(hasher, binary.LittleEndian, sth.Timestamp)
-		binary.Write(hasher, binary.LittleEndian, sth.SHA256RootHash)
-	default:
-		panic(fmt.Sprintf("Unsupported STH version %d", sth.Version))
-	}
-	// For 6962-bis, we will need to handle a variable-length root hash, and include the signature in the filename hash (since signatures must be deterministic)
-	return strconv.FormatUint(sth.TreeSize, 10) + "-" + base64.RawURLEncoding.EncodeToString(hasher.Sum(nil)) + ".json"
-}
-
-func makeLogStateDir(logStatePath string) error {
-	if err := os.Mkdir(logStatePath, 0777); err != nil && !os.IsExist(err) {
-		return fmt.Errorf("%s: %s", logStatePath, err)
-	}
-	for _, subdir := range []string{"unverified_sths"} {
-		path := filepath.Join(logStatePath, subdir)
-		if err := os.Mkdir(path, 0777); err != nil && !os.IsExist(err) {
-			return fmt.Errorf("%s: %s", path, err)
-		}
-	}
-	return nil
-}
-
-func OpenLogState(logStatePath string) (*LogState, error) {
-	if err := makeLogStateDir(logStatePath); err != nil {
-		return nil, fmt.Errorf("Error creating log state directory: %s", err)
-	}
-	return &LogState{path: logStatePath}, nil
-}
-
-func (logState *LogState) VerifiedSTHFilename() string {
-	return filepath.Join(logState.path, "sth.json")
-}
-
-func (logState *LogState) GetVerifiedSTH() (*ct.SignedTreeHead, error) {
-	sth, err := readSTHFile(logState.VerifiedSTHFilename())
-	if err != nil {
-		if os.IsNotExist(err) {
-			return nil, nil
-		} else {
-			return nil, err
-		}
-	}
-	return sth, nil
-}
-
-func (logState *LogState) StoreVerifiedSTH(sth *ct.SignedTreeHead) error {
-	return writeJSONFile(logState.VerifiedSTHFilename(), sth, 0666)
-}
-
-func (logState *LogState) GetUnverifiedSTHs() ([]*ct.SignedTreeHead, error) {
-	dir, err := os.Open(filepath.Join(logState.path, "unverified_sths"))
-	if err != nil {
-		if os.IsNotExist(err) {
-			return []*ct.SignedTreeHead{}, nil
-		} else {
-			return nil, err
-		}
-	}
-	filenames, err := dir.Readdirnames(0)
-	if err != nil {
-		return nil, err
-	}
-
-	sths := make([]*ct.SignedTreeHead, 0, len(filenames))
-	for _, filename := range filenames {
-		if !strings.HasPrefix(filename, ".") {
-			sth, _ := readSTHFile(filepath.Join(dir.Name(), filename))
-			if sth != nil {
-				sths = append(sths, sth)
-			}
-		}
-	}
-	return sths, nil
-}
-
-func (logState *LogState) UnverifiedSTHFilename(sth *ct.SignedTreeHead) string {
-	return filepath.Join(logState.path, "unverified_sths", sthFilename(sth))
-}
-
-func (logState *LogState) StoreUnverifiedSTH(sth *ct.SignedTreeHead) error {
-	filename := logState.UnverifiedSTHFilename(sth)
-	if fileExists(filename) {
-		return nil
-	}
-	return writeJSONFile(filename, sth, 0666)
-}
-
-func (logState *LogState) RemoveUnverifiedSTH(sth *ct.SignedTreeHead) error {
-	filename := logState.UnverifiedSTHFilename(sth)
-	err := os.Remove(filepath.Join(filename))
-	if err != nil && !os.IsNotExist(err) {
-		return err
-	}
-	return nil
-}
-
-func (logState *LogState) GetTree() (*certspotter.CollapsedMerkleTree, error) {
-	tree := new(certspotter.CollapsedMerkleTree)
-	if err := readJSONFile(filepath.Join(logState.path, "tree.json"), tree); err != nil {
-		if os.IsNotExist(err) {
-			return nil, nil
-		} else {
-			return nil, err
-		}
-	}
-	return tree, nil
-}
-
-func (logState *LogState) StoreTree(tree *certspotter.CollapsedMerkleTree) error {
-	return writeJSONFile(filepath.Join(logState.path, "tree.json"), tree, 0666)
-}

cmd/state.go

@@ -1,220 +0,0 @@
-// Copyright (C) 2017 Opsmate, Inc.
-//
-// This Source Code Form is subject to the terms of the Mozilla
-// Public License, v. 2.0. If a copy of the MPL was not distributed
-// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
-//
-// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
-// See the Mozilla Public License for details.
-
-package cmd
-
-import (
-	"bytes"
-	"encoding/base64"
-	"encoding/pem"
-	"fmt"
-	"io/ioutil"
-	"log"
-	"os"
-	"path/filepath"
-	"strconv"
-	"strings"
-
-	"software.sslmate.com/src/certspotter"
-	"software.sslmate.com/src/certspotter/ct"
-)
-
-type State struct {
-	path string
-}
-
-func legacySTHFilename(logInfo *certspotter.LogInfo) string {
-	return strings.Replace(strings.Replace(logInfo.FullURI(), "://", "_", 1), "/", "_", -1)
-}
-
-func readVersionFile(statePath string) (int, error) {
-	versionFilePath := filepath.Join(statePath, "version")
-	versionBytes, err := ioutil.ReadFile(versionFilePath)
-	if err == nil {
-		version, err := strconv.Atoi(string(bytes.TrimSpace(versionBytes)))
-		if err != nil {
-			return -1, fmt.Errorf("%s: contains invalid integer: %s", versionFilePath, err)
-		}
-		if version < 0 {
-			return -1, fmt.Errorf("%s: contains negative integer", versionFilePath)
-		}
-		return version, nil
-	} else if os.IsNotExist(err) {
-		if fileExists(filepath.Join(statePath, "sths")) {
-			// Original version of certspotter had no version file.
-			// Infer version 0 if "sths" directory is present.
-			return 0, nil
-		}
-		return -1, nil
-	} else {
-		return -1, fmt.Errorf("%s: %s", versionFilePath, err)
-	}
-}
-
-func writeVersionFile(statePath string) error {
-	version := 1
-	versionString := fmt.Sprintf("%d\n", version)
-	versionFilePath := filepath.Join(statePath, "version")
-	if err := ioutil.WriteFile(versionFilePath, []byte(versionString), 0666); err != nil {
-		return fmt.Errorf("%s: %s\n", versionFilePath, err)
-	}
-	return nil
-}
-
-func makeStateDir(statePath string) error {
-	if err := os.Mkdir(statePath, 0777); err != nil && !os.IsExist(err) {
-		return fmt.Errorf("%s: %s", statePath, err)
-	}
-	for _, subdir := range []string{"certs", "logs"} {
-		path := filepath.Join(statePath, subdir)
-		if err := os.Mkdir(path, 0777); err != nil && !os.IsExist(err) {
-			return fmt.Errorf("%s: %s", path, err)
-		}
-	}
-	return nil
-}
-
-func OpenState(statePath string) (*State, error) {
-	version, err := readVersionFile(statePath)
-	if err != nil {
-		return nil, fmt.Errorf("Error reading version file: %s", err)
-	}
-
-	if version < 1 {
-		if err := makeStateDir(statePath); err != nil {
-			return nil, fmt.Errorf("Error creating state directory: %s", err)
-		}
-		if version == 0 {
-			log.Printf("Migrating state directory (%s) to new layout...", statePath)
-			if err := os.Rename(filepath.Join(statePath, "sths"), filepath.Join(statePath, "legacy_sths")); err != nil {
-				return nil, fmt.Errorf("Error migrating STHs directory: %s", err)
-			}
-			for _, subdir := range []string{"evidence", "legacy_sths"} {
-				os.Remove(filepath.Join(statePath, subdir))
-			}
-			if err := ioutil.WriteFile(filepath.Join(statePath, "once"), []byte{}, 0666); err != nil {
-				return nil, fmt.Errorf("Error creating once file: %s", err)
-			}
-		}
-		if err := writeVersionFile(statePath); err != nil {
-			return nil, fmt.Errorf("Error writing version file: %s", err)
-		}
-	} else if version > 1 {
-		return nil, fmt.Errorf("%s was created by a newer version of Cert Spotter; please remove this directory or upgrade Cert Spotter", statePath)
-	}
-
-	return &State{path: statePath}, nil
-}
-
-func (state *State) IsFirstRun() bool {
-	return !fileExists(filepath.Join(state.path, "once"))
-}
-
-func (state *State) WriteOnceFile() error {
-	if err := ioutil.WriteFile(filepath.Join(state.path, "once"), []byte{}, 0666); err != nil {
-		return fmt.Errorf("Error writing once file: %s", err)
-	}
-	return nil
-}
-
-func (state *State) SaveCert(isPrecert bool, certs [][]byte) (bool, string, error) {
-	if len(certs) == 0 {
-		return false, "", fmt.Errorf("Cannot write an empty certificate chain")
-	}
-
-	fingerprint := sha256hex(certs[0])
-	prefixPath := filepath.Join(state.path, "certs", fingerprint[0:2])
-	var filenameSuffix string
-	if isPrecert {
-		filenameSuffix = ".precert.pem"
-	} else {
-		filenameSuffix = ".cert.pem"
-	}
-	if err := os.Mkdir(prefixPath, 0777); err != nil && !os.IsExist(err) {
-		return false, "", fmt.Errorf("Failed to create prefix directory %s: %s", prefixPath, err)
-	}
-	path := filepath.Join(prefixPath, fingerprint+filenameSuffix)
-	file, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0666)
-	if err != nil {
-		if os.IsExist(err) {
-			return true, path, nil
-		} else {
-			return false, path, fmt.Errorf("Failed to open %s for writing: %s", path, err)
-		}
-	}
-	for _, cert := range certs {
-		if err := pem.Encode(file, &pem.Block{Type: "CERTIFICATE", Bytes: cert}); err != nil {
-			file.Close()
-			return false, path, fmt.Errorf("Error writing to %s: %s", path, err)
-		}
-	}
-	if err := file.Close(); err != nil {
-		return false, path, fmt.Errorf("Error writing to %s: %s", path, err)
-	}
-
-	return false, path, nil
-}
-
-func (state *State) OpenLogState(logInfo *certspotter.LogInfo) (*LogState, error) {
-	return OpenLogState(filepath.Join(state.path, "logs", base64.RawURLEncoding.EncodeToString(logInfo.ID())))
-}
-
-func (state *State) GetLegacySTH(logInfo *certspotter.LogInfo) (*ct.SignedTreeHead, error) {
-	sth, err := readSTHFile(filepath.Join(state.path, "legacy_sths", legacySTHFilename(logInfo)))
-	if err != nil {
-		if os.IsNotExist(err) {
-			return nil, nil
-		} else {
-			return nil, err
-		}
-	}
-	return sth, nil
-}
-func (state *State) RemoveLegacySTH(logInfo *certspotter.LogInfo) error {
-	err := os.Remove(filepath.Join(state.path, "legacy_sths", legacySTHFilename(logInfo)))
-	os.Remove(filepath.Join(state.path, "legacy_sths"))
-	return err
-}
-func (state *State) LockFilename() string {
-	return filepath.Join(state.path, "lock")
-}
-func (state *State) Lock() (bool, error) {
-	file, err := os.OpenFile(state.LockFilename(), os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0666)
-	if err != nil {
-		if os.IsExist(err) {
-			return false, nil
-		} else {
-			return false, err
-		}
-	}
-	if _, err := fmt.Fprintf(file, "%d\n", os.Getpid()); err != nil {
-		file.Close()
-		os.Remove(state.LockFilename())
-		return false, err
-	}
-	if err := file.Close(); err != nil {
-		os.Remove(state.LockFilename())
-		return false, err
-	}
-	return true, nil
-}
-func (state *State) Unlock() error {
-	return os.Remove(state.LockFilename())
-}
-func (state *State) LockingPid() int {
-	pidBytes, err := ioutil.ReadFile(state.LockFilename())
-	if err != nil {
-		return 0
-	}
-	pid, err := strconv.Atoi(string(bytes.TrimSpace(pidBytes)))
-	if err != nil {
-		return 0
-	}
-	return pid
-}

cmd/submitct/main.go

@@ -1,215 +0,0 @@
-// Copyright (C) 2017 Opsmate, Inc.
-//
-// This Source Code Form is subject to the terms of the Mozilla
-// Public License, v. 2.0. If a copy of the MPL was not distributed
-// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
-//
-// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
-// See the Mozilla Public License for details.
-
-package main
-
-import (
-	"software.sslmate.com/src/certspotter"
-	"software.sslmate.com/src/certspotter/ct"
-	"software.sslmate.com/src/certspotter/ct/client"
-
-	"bytes"
-	"crypto/sha256"
-	"encoding/pem"
-	"flag"
-	"fmt"
-	"io/ioutil"
-	"log"
-	"os"
-	"sync"
-	"sync/atomic"
-	"time"
-)
-
-var verbose = flag.Bool("v", false, "Enable verbose output")
-
-type Certificate struct {
-	Subject []byte
-	Issuer  []byte
-	Raw     []byte
-}
-
-func (cert *Certificate) Fingerprint() [32]byte {
-	return sha256.Sum256(cert.Raw)
-}
-
-func (cert *Certificate) CommonName() string {
-	subject, err := certspotter.ParseRDNSequence(cert.Subject)
-	if err != nil {
-		return "???"
-	}
-	cns, err := subject.ParseCNs()
-	if err != nil || len(cns) == 0 {
-		return "???"
-	}
-	return cns[0]
-}
-
-func parseCertificate(data []byte) (*Certificate, error) {
-	crt, err := certspotter.ParseCertificate(data)
-	if err != nil {
-		return nil, err
-	}
-
-	tbs, err := crt.ParseTBSCertificate()
-	if err != nil {
-		return nil, err
-	}
-
-	return &Certificate{
-		Subject: tbs.Subject.FullBytes,
-		Issuer:  tbs.Issuer.FullBytes,
-		Raw:     data,
-	}, nil
-}
-
-type Chain []*Certificate
-
-func (c Chain) GetRawCerts() [][]byte {
-	rawCerts := make([][]byte, len(c))
-	for i := range c {
-		rawCerts[i] = c[i].Raw
-	}
-	return rawCerts
-}
-
-type CertificateBunch struct {
-	byFingerprint map[[32]byte]*Certificate
-	bySubject     map[[32]byte]*Certificate
-}
-
-func MakeCertificateBunch() CertificateBunch {
-	return CertificateBunch{
-		byFingerprint: make(map[[32]byte]*Certificate),
-		bySubject:     make(map[[32]byte]*Certificate),
-	}
-}
-
-func (certs *CertificateBunch) Add(cert *Certificate) {
-	certs.byFingerprint[cert.Fingerprint()] = cert
-	certs.bySubject[sha256.Sum256(cert.Subject)] = cert
-}
-
-func (certs *CertificateBunch) FindBySubject(subject []byte) *Certificate {
-	return certs.bySubject[sha256.Sum256(subject)]
-}
-
-type Log struct {
-	info   certspotter.LogInfo
-	verify *ct.SignatureVerifier
-	client *client.LogClient
-}
-
-func (ctlog *Log) SubmitChain(chain Chain) (*ct.SignedCertificateTimestamp, error) {
-	rawCerts := chain.GetRawCerts()
-	sct, err := ctlog.client.AddChain(rawCerts)
-	if err != nil {
-		return nil, err
-	}
-
-	if err := certspotter.VerifyX509SCT(sct, rawCerts[0], ctlog.verify); err != nil {
-		return nil, fmt.Errorf("Bad SCT signature: %s", err)
-	}
-	return sct, nil
-}
-
-func buildChain(cert *Certificate, certs *CertificateBunch) Chain {
-	chain := make([]*Certificate, 0)
-	for len(chain) < 16 && cert != nil && !bytes.Equal(cert.Subject, cert.Issuer) {
-		chain = append(chain, cert)
-		cert = certs.FindBySubject(cert.Issuer)
-	}
-	return chain
-}
-
-func main() {
-	flag.Parse()
-	log.SetPrefix("submitct: ")
-
-	certsPem, err := ioutil.ReadAll(os.Stdin)
-	if err != nil {
-		log.Fatalf("Error reading stdin: %s", err)
-	}
-
-	logs := make([]Log, 0, len(certspotter.OpenLogs))
-	for _, loginfo := range certspotter.OpenLogs {
-		pubkey, err := loginfo.ParsedPublicKey()
-		if err != nil {
-			log.Fatalf("%s: Failed to parse log public key: %s", loginfo.Url, err)
-		}
-		verify, err := ct.NewSignatureVerifier(pubkey)
-		if err != nil {
-			log.Fatalf("%s: Failed to create signature verifier for log: %s", loginfo.Url, err)
-		}
-		logs = append(logs, Log{
-			info:   loginfo,
-			verify: verify,
-			client: client.New(loginfo.FullURI()),
-		})
-	}
-
-	certs := MakeCertificateBunch()
-	var parseErrors uint32
-	var submitErrors uint32
-
-	for len(certsPem) > 0 {
-		var pemBlock *pem.Block
-		pemBlock, certsPem = pem.Decode(certsPem)
-		if pemBlock == nil {
-			log.Fatalf("Invalid PEM read from stdin")
-		}
-		if pemBlock.Type != "CERTIFICATE" {
-			log.Printf("Ignoring non-certificate read from stdin")
-			continue
-		}
-
-		cert, err := parseCertificate(pemBlock.Bytes)
-		if err != nil {
-			log.Printf("Ignoring un-parseable certificate read from stdin: %s", err)
-			parseErrors++
-			continue
-		}
-		certs.Add(cert)
-	}
-
-	wg := sync.WaitGroup{}
-	for fingerprint, cert := range certs.byFingerprint {
-		cn := cert.CommonName()
-		chain := buildChain(cert, &certs)
-		if len(chain) == 0 {
-			continue
-		}
-		for _, ctlog := range logs {
-			wg.Add(1)
-			go func(fingerprint [32]byte, ctlog Log) {
-				sct, err := ctlog.SubmitChain(chain)
-				if err != nil {
-					log.Printf("%x (%s): %s: Submission Error: %s", fingerprint, cn, ctlog.info.Url, err)
-					atomic.AddUint32(&submitErrors, 1)
-				} else if *verbose {
-					timestamp := time.Unix(int64(sct.Timestamp)/1000, int64(sct.Timestamp%1000)*1000000)
-					log.Printf("%x (%s): %s: Submitted at %s", fingerprint, cn, ctlog.info.Url, timestamp)
-				}
-				wg.Done()
-			}(fingerprint, ctlog)
-		}
-	}
-	wg.Wait()
-
-	exitStatus := 0
-	if parseErrors > 0 {
-		log.Printf("%d certificates failed to parse and were ignored", parseErrors)
-		exitStatus |= 4
-	}
-	if submitErrors > 0 {
-		log.Printf("%d submission errors occurred", submitErrors)
-		exitStatus |= 8
-	}
-	os.Exit(exitStatus)
-}

ct/AUTHORS

@@ -1,24 +0,0 @@
-# This is the official list of benchmark authors for copyright purposes.
-# This file is distinct from the CONTRIBUTORS files.
-# See the latter for an explanation.
-#
-# Names should be added to this file as:
-#	Name or Organization <email address>
-# The email address is not required for organizations.
-#
-# Please keep the list sorted.
-
-Comodo CA Limited
-Ed Maste <emaste@freebsd.org>
-Fiaz Hossain <fiaz.hossain@salesforce.com>
-Google Inc.
-Jeff Trawick <trawick@gmail.com>
-Katriel Cohn-Gordon <katriel.cohn-gordon@cybersecurity.ox.ac.uk>
-Mark Schloesser <ms@mwcollect.org>
-NORDUnet A/S
-Nicholas Galbreath <nickg@client9.com>
-Oliver Weidner <Oliver.Weidner@gmail.com>
-Ruslan Kovalov <ruslan.kovalyov@gmail.com>
-Venafi, Inc.
-Vladimir Rutsky <vladimir@rutsky.org>
-Ximin Luo <infinity0@gmx.com>

ct/LICENSE

@@ -1,202 +0,0 @@
-
-                                 Apache License
-                           Version 2.0, January 2004
-                        http://www.apache.org/licenses/
-
-   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
-
-   1. Definitions.
-
-      "License" shall mean the terms and conditions for use, reproduction,
-      and distribution as defined by Sections 1 through 9 of this document.
-
-      "Licensor" shall mean the copyright owner or entity authorized by
-      the copyright owner that is granting the License.
-
-      "Legal Entity" shall mean the union of the acting entity and all
-      other entities that control, are controlled by, or are under common
-      control with that entity. For the purposes of this definition,
-      "control" means (i) the power, direct or indirect, to cause the
-      direction or management of such entity, whether by contract or
-      otherwise, or (ii) ownership of fifty percent (50%) or more of the
-      outstanding shares, or (iii) beneficial ownership of such entity.
-
-      "You" (or "Your") shall mean an individual or Legal Entity
-      exercising permissions granted by this License.
-
-      "Source" form shall mean the preferred form for making modifications,
-      including but not limited to software source code, documentation
-      source, and configuration files.
-
-      "Object" form shall mean any form resulting from mechanical
-      transformation or translation of a Source form, including but
-      not limited to compiled object code, generated documentation,
-      and conversions to other media types.
-
-      "Work" shall mean the work of authorship, whether in Source or
-      Object form, made available under the License, as indicated by a
-      copyright notice that is included in or attached to the work
-      (an example is provided in the Appendix below).
-
-      "Derivative Works" shall mean any work, whether in Source or Object
-      form, that is based on (or derived from) the Work and for which the
-      editorial revisions, annotations, elaborations, or other modifications
-      represent, as a whole, an original work of authorship. For the purposes
-      of this License, Derivative Works shall not include works that remain
-      separable from, or merely link (or bind by name) to the interfaces of,
-      the Work and Derivative Works thereof.
-
-      "Contribution" shall mean any work of authorship, including
-      the original version of the Work and any modifications or additions
-      to that Work or Derivative Works thereof, that is intentionally
-      submitted to Licensor for inclusion in the Work by the copyright owner
-      or by an individual or Legal Entity authorized to submit on behalf of
-      the copyright owner. For the purposes of this definition, "submitted"
-      means any form of electronic, verbal, or written communication sent
-      to the Licensor or its representatives, including but not limited to
-      communication on electronic mailing lists, source code control systems,
-      and issue tracking systems that are managed by, or on behalf of, the
-      Licensor for the purpose of discussing and improving the Work, but
-      excluding communication that is conspicuously marked or otherwise
-      designated in writing by the copyright owner as "Not a Contribution."
-
-      "Contributor" shall mean Licensor and any individual or Legal Entity
-      on behalf of whom a Contribution has been received by Licensor and
-      subsequently incorporated within the Work.
-
-   2. Grant of Copyright License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      copyright license to reproduce, prepare Derivative Works of,
-      publicly display, publicly perform, sublicense, and distribute the
-      Work and such Derivative Works in Source or Object form.
-
-   3. Grant of Patent License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      (except as stated in this section) patent license to make, have made,
-      use, offer to sell, sell, import, and otherwise transfer the Work,
-      where such license applies only to those patent claims licensable
-      by such Contributor that are necessarily infringed by their
-      Contribution(s) alone or by combination of their Contribution(s)
-      with the Work to which such Contribution(s) was submitted. If You
-      institute patent litigation against any entity (including a
-      cross-claim or counterclaim in a lawsuit) alleging that the Work
-      or a Contribution incorporated within the Work constitutes direct
-      or contributory patent infringement, then any patent licenses
-      granted to You under this License for that Work shall terminate
-      as of the date such litigation is filed.
-
-   4. Redistribution. You may reproduce and distribute copies of the
-      Work or Derivative Works thereof in any medium, with or without
-      modifications, and in Source or Object form, provided that You
-      meet the following conditions:
-
-      (a) You must give any other recipients of the Work or
-          Derivative Works a copy of this License; and
-
-      (b) You must cause any modified files to carry prominent notices
-          stating that You changed the files; and
-
-      (c) You must retain, in the Source form of any Derivative Works
-          that You distribute, all copyright, patent, trademark, and
-          attribution notices from the Source form of the Work,
-          excluding those notices that do not pertain to any part of
-          the Derivative Works; and
-
-      (d) If the Work includes a "NOTICE" text file as part of its
-          distribution, then any Derivative Works that You distribute must
-          include a readable copy of the attribution notices contained
-          within such NOTICE file, excluding those notices that do not
-          pertain to any part of the Derivative Works, in at least one
-          of the following places: within a NOTICE text file distributed
-          as part of the Derivative Works; within the Source form or
-          documentation, if provided along with the Derivative Works; or,
-          within a display generated by the Derivative Works, if and
-          wherever such third-party notices normally appear. The contents
-          of the NOTICE file are for informational purposes only and
-          do not modify the License. You may add Your own attribution
-          notices within Derivative Works that You distribute, alongside
-          or as an addendum to the NOTICE text from the Work, provided
-          that such additional attribution notices cannot be construed
-          as modifying the License.
-
-      You may add Your own copyright statement to Your modifications and
-      may provide additional or different license terms and conditions
-      for use, reproduction, or distribution of Your modifications, or
-      for any such Derivative Works as a whole, provided Your use,
-      reproduction, and distribution of the Work otherwise complies with
-      the conditions stated in this License.
-
-   5. Submission of Contributions. Unless You explicitly state otherwise,
-      any Contribution intentionally submitted for inclusion in the Work
-      by You to the Licensor shall be under the terms and conditions of
-      this License, without any additional terms or conditions.
-      Notwithstanding the above, nothing herein shall supersede or modify
-      the terms of any separate license agreement you may have executed
-      with Licensor regarding such Contributions.
-
-   6. Trademarks. This License does not grant permission to use the trade
-      names, trademarks, service marks, or product names of the Licensor,
-      except as required for reasonable and customary use in describing the
-      origin of the Work and reproducing the content of the NOTICE file.
-
-   7. Disclaimer of Warranty. Unless required by applicable law or
-      agreed to in writing, Licensor provides the Work (and each
-      Contributor provides its Contributions) on an "AS IS" BASIS,
-      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-      implied, including, without limitation, any warranties or conditions
-      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
-      PARTICULAR PURPOSE. You are solely responsible for determining the
-      appropriateness of using or redistributing the Work and assume any
-      risks associated with Your exercise of permissions under this License.
-
-   8. Limitation of Liability. In no event and under no legal theory,
-      whether in tort (including negligence), contract, or otherwise,
-      unless required by applicable law (such as deliberate and grossly
-      negligent acts) or agreed to in writing, shall any Contributor be
-      liable to You for damages, including any direct, indirect, special,
-      incidental, or consequential damages of any character arising as a
-      result of this License or out of the use or inability to use the
-      Work (including but not limited to damages for loss of goodwill,
-      work stoppage, computer failure or malfunction, or any and all
-      other commercial damages or losses), even if such Contributor
-      has been advised of the possibility of such damages.
-
-   9. Accepting Warranty or Additional Liability. While redistributing
-      the Work or Derivative Works thereof, You may choose to offer,
-      and charge a fee for, acceptance of support, warranty, indemnity,
-      or other liability obligations and/or rights consistent with this
-      License. However, in accepting such obligations, You may act only
-      on Your own behalf and on Your sole responsibility, not on behalf
-      of any other Contributor, and only if You agree to indemnify,
-      defend, and hold each Contributor harmless for any liability
-      incurred by, or claims asserted against, such Contributor by reason
-      of your accepting any such warranty or additional liability.
-
-   END OF TERMS AND CONDITIONS
-
-   APPENDIX: How to apply the Apache License to your work.
-
-      To apply the Apache License to your work, attach the following
-      boilerplate notice, with the fields enclosed by brackets "[]"
-      replaced with your own identifying information. (Don't include
-      the brackets!)  The text should be enclosed in the appropriate
-      comment syntax for the file format. We also recommend that a
-      file or class name and description of purpose be included on the
-      same "printed page" as the copyright notice for easier
-      identification within third-party archives.
-
-   Copyright [yyyy] [name of copyright owner]
-
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-
-       http://www.apache.org/licenses/LICENSE-2.0
-
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License.

ct/README

@@ -1,2 +0,0 @@
-The code in this directory is from https://github.com/google/certificate-transparency/tree/master/go
-See AUTHORS for the copyright holders, and LICENSE for the license.

ct/client/logclient.go

@@ -1,293 +0,0 @@
-// Package client is a CT log client implementation and contains types and code
-// for interacting with RFC6962-compliant CT Log instances.
-// See http://tools.ietf.org/html/rfc6962 for details
-package client
-
-import (
-	"bytes"
-	"crypto/sha256"
-	"crypto/tls"
-	"encoding/base64"
-	"encoding/json"
-	"errors"
-	"fmt"
-	"io"
-	"io/ioutil"
-	"net/http"
-	"net/url"
-	"time"
-
-	"github.com/mreiferson/go-httpclient"
-	"software.sslmate.com/src/certspotter/ct"
-)
-
-// URI paths for CT Log endpoints
-const (
-	GetSTHPath            = "/ct/v1/get-sth"
-	GetEntriesPath        = "/ct/v1/get-entries"
-	GetSTHConsistencyPath = "/ct/v1/get-sth-consistency"
-	GetProofByHashPath    = "/ct/v1/get-proof-by-hash"
-	AddChainPath          = "/ct/v1/add-chain"
-)
-
-// LogClient represents a client for a given CT Log instance
-type LogClient struct {
-	uri        string       // the base URI of the log. e.g. http://ct.googleapis/pilot
-	httpClient *http.Client // used to interact with the log via HTTP
-}
-
-//////////////////////////////////////////////////////////////////////////////////
-// JSON structures follow.
-// These represent the structures returned by the CT Log server.
-//////////////////////////////////////////////////////////////////////////////////
-
-// getSTHResponse respresents the JSON response to the get-sth CT method
-type getSTHResponse struct {
-	TreeSize          uint64 `json:"tree_size"`           // Number of certs in the current tree
-	Timestamp         uint64 `json:"timestamp"`           // Time that the tree was created
-	SHA256RootHash    []byte `json:"sha256_root_hash"`    // Root hash of the tree
-	TreeHeadSignature []byte `json:"tree_head_signature"` // Log signature for this STH
-}
-
-// base64LeafEntry respresents a Base64 encoded leaf entry
-type base64LeafEntry struct {
-	LeafInput []byte `json:"leaf_input"`
-	ExtraData []byte `json:"extra_data"`
-}
-
-// getEntriesReponse respresents the JSON response to the CT get-entries method
-type getEntriesResponse struct {
-	Entries []base64LeafEntry `json:"entries"` // the list of returned entries
-}
-
-// getConsistencyProofResponse represents the JSON response to the CT get-consistency-proof method
-type getConsistencyProofResponse struct {
-	Consistency [][]byte `json:"consistency"`
-}
-
-// getAuditProofResponse represents the JSON response to the CT get-proof-by-hash method
-type getAuditProofResponse struct {
-	LeafIndex uint64   `json:"leaf_index"`
-	AuditPath [][]byte `json:"audit_path"`
-}
-
-type addChainRequest struct {
-	Chain [][]byte `json:"chain"`
-}
-
-type addChainResponse struct {
-	SCTVersion uint8  `json:"sct_version"`
-	ID         []byte `json:"id"`
-	Timestamp  uint64 `json:"timestamp"`
-	Extensions []byte `json:"extensions"`
-	Signature  []byte `json:"signature"`
-}
-
-// New constructs a new LogClient instance.
-// |uri| is the base URI of the CT log instance to interact with, e.g.
-// http://ct.googleapis.com/pilot
-func New(uri string) *LogClient {
-	var c LogClient
-	c.uri = uri
-	transport := &httpclient.Transport{
-		ConnectTimeout:        10 * time.Second,
-		RequestTimeout:        60 * time.Second,
-		ResponseHeaderTimeout: 30 * time.Second,
-		MaxIdleConnsPerHost:   10,
-		DisableKeepAlives:     false,
-		TLSClientConfig: &tls.Config{
-			// We have to disable TLS certificate validation because because several logs
-			// (WoSign, StartCom, GDCA) use certificates that are not widely trusted.
-			// Since we verify that every response we receive from the log is signed
-			// by the log's CT public key (either directly, or indirectly via the Merkle Tree),
-			// TLS certificate validation is not actually necessary.  (We don't want to ship
-			// our own trust store because that adds undesired complexity and would require
-			// updating should a log ever change to a different CA.)
-			InsecureSkipVerify: true,
-		},
-	}
-	c.httpClient = &http.Client{Transport: transport}
-	return &c
-}
-
-// Makes a HTTP call to |uri|, and attempts to parse the response as a JSON
-// representation of the structure in |res|.
-// Returns a non-nil |error| if there was a problem.
-func (c *LogClient) fetchAndParse(uri string, respBody interface{}) error {
-	req, err := http.NewRequest("GET", uri, nil)
-	if err != nil {
-		return fmt.Errorf("GET %s: Sending request failed: %s", uri, err)
-	}
-	return c.doAndParse(req, respBody)
-}
-
-func (c *LogClient) postAndParse(uri string, body interface{}, respBody interface{}) error {
-	bodyReader, bodyWriter := io.Pipe()
-	go func() {
-		json.NewEncoder(bodyWriter).Encode(body)
-		bodyWriter.Close()
-	}()
-	req, err := http.NewRequest("POST", uri, bodyReader)
-	if err != nil {
-		return fmt.Errorf("POST %s: Sending request failed: %s", uri, err)
-	}
-	req.Header.Set("Content-Type", "application/json")
-	return c.doAndParse(req, respBody)
-}
-
-func (c *LogClient) doAndParse(req *http.Request, respBody interface{}) error {
-	//	req.Header.Set("Keep-Alive", "timeout=15, max=100")
-	resp, err := c.httpClient.Do(req)
-	var respBodyBytes []byte
-	if resp != nil {
-		respBodyBytes, err = ioutil.ReadAll(resp.Body)
-		resp.Body.Close()
-		if err != nil {
-			return fmt.Errorf("%s %s: Reading response failed: %s", req.Method, req.URL, err)
-		}
-	}
-	if err != nil {
-		return err
-	}
-	if resp.StatusCode/100 != 2 {
-		return fmt.Errorf("%s %s: %s (%s)", req.Method, req.URL, resp.Status, string(respBodyBytes))
-	}
-	if err = json.Unmarshal(respBodyBytes, &respBody); err != nil {
-		return fmt.Errorf("%s %s: Parsing response JSON failed: %s", req.Method, req.URL, err)
-	}
-	return nil
-}
-
-// GetSTH retrieves the current STH from the log.
-// Returns a populated SignedTreeHead, or a non-nil error.
-func (c *LogClient) GetSTH() (sth *ct.SignedTreeHead, err error) {
-	var resp getSTHResponse
-	if err = c.fetchAndParse(c.uri+GetSTHPath, &resp); err != nil {
-		return
-	}
-	sth = &ct.SignedTreeHead{
-		TreeSize:  resp.TreeSize,
-		Timestamp: resp.Timestamp,
-	}
-
-	if len(resp.SHA256RootHash) != sha256.Size {
-		return nil, fmt.Errorf("STH returned by server has invalid sha256_root_hash (expected length %d got %d)", sha256.Size, len(resp.SHA256RootHash))
-	}
-	copy(sth.SHA256RootHash[:], resp.SHA256RootHash)
-
-	ds, err := ct.UnmarshalDigitallySigned(bytes.NewReader(resp.TreeHeadSignature))
-	if err != nil {
-		return nil, err
-	}
-	// TODO(alcutter): Verify signature
-	sth.TreeHeadSignature = *ds
-	return
-}
-
-// GetEntries attempts to retrieve the entries in the sequence [|start|, |end|] from the CT
-// log server. (see section 4.6.)
-// Returns a slice of LeafInputs or a non-nil error.
-func (c *LogClient) GetEntries(start, end int64) ([]ct.LogEntry, error) {
-	if end < 0 {
-		return nil, errors.New("GetEntries: end should be >= 0")
-	}
-	if end < start {
-		return nil, errors.New("GetEntries: start should be <= end")
-	}
-	var resp getEntriesResponse
-	err := c.fetchAndParse(fmt.Sprintf("%s%s?start=%d&end=%d", c.uri, GetEntriesPath, start, end), &resp)
-	if err != nil {
-		return nil, err
-	}
-	entries := make([]ct.LogEntry, len(resp.Entries))
-	for index, entry := range resp.Entries {
-		leaf, err := ct.ReadMerkleTreeLeaf(bytes.NewBuffer(entry.LeafInput))
-		if err != nil {
-			return nil, fmt.Errorf("Reading Merkle Tree Leaf at index %d failed: %s", start+int64(index), err)
-		}
-		entries[index].LeafBytes = entry.LeafInput
-		entries[index].Leaf = *leaf
-
-		var chain []ct.ASN1Cert
-		switch leaf.TimestampedEntry.EntryType {
-		case ct.X509LogEntryType:
-			chain, err = ct.UnmarshalX509ChainArray(entry.ExtraData)
-
-		case ct.PrecertLogEntryType:
-			chain, err = ct.UnmarshalPrecertChainArray(entry.ExtraData)
-
-		default:
-			return nil, fmt.Errorf("Unknown entry type at index %d: %v", start+int64(index), leaf.TimestampedEntry.EntryType)
-		}
-		if err != nil {
-			return nil, fmt.Errorf("Parsing entry of type %d at index %d failed: %s", leaf.TimestampedEntry.EntryType, start+int64(index), err)
-		}
-		entries[index].Chain = chain
-		entries[index].Index = start + int64(index)
-	}
-	return entries, nil
-}
-
-// GetConsistencyProof retrieves a Merkle Consistency Proof between two STHs (|first| and |second|)
-// from the log.  Returns a slice of MerkleTreeNodes (a ct.ConsistencyProof) or a non-nil error.
-func (c *LogClient) GetConsistencyProof(first, second int64) (ct.ConsistencyProof, error) {
-	if second < 0 {
-		return nil, errors.New("GetConsistencyProof: second should be >= 0")
-	}
-	if second < first {
-		return nil, errors.New("GetConsistencyProof: first should be <= second")
-	}
-	var resp getConsistencyProofResponse
-	err := c.fetchAndParse(fmt.Sprintf("%s%s?first=%d&second=%d", c.uri, GetSTHConsistencyPath, first, second), &resp)
-	if err != nil {
-		return nil, err
-	}
-	nodes := make([]ct.MerkleTreeNode, len(resp.Consistency))
-	for index, nodeBytes := range resp.Consistency {
-		nodes[index] = nodeBytes
-	}
-	return nodes, nil
-}
-
-// GetAuditProof retrieves a Merkle Audit Proof (aka Inclusion Proof) for the given
-// |hash| based on the STH at |treeSize| from the log.  Returns a slice of MerkleTreeNodes
-// and the index of the leaf.
-func (c *LogClient) GetAuditProof(hash ct.MerkleTreeNode, treeSize uint64) (ct.AuditPath, uint64, error) {
-	var resp getAuditProofResponse
-	err := c.fetchAndParse(fmt.Sprintf("%s%s?hash=%s&tree_size=%d", c.uri, GetProofByHashPath, url.QueryEscape(base64.StdEncoding.EncodeToString(hash)), treeSize), &resp)
-	if err != nil {
-		return nil, 0, err
-	}
-	path := make([]ct.MerkleTreeNode, len(resp.AuditPath))
-	for index, nodeBytes := range resp.AuditPath {
-		path[index] = nodeBytes
-	}
-	return path, resp.LeafIndex, nil
-}
-
-func (c *LogClient) AddChain(chain [][]byte) (*ct.SignedCertificateTimestamp, error) {
-	req := addChainRequest{Chain: chain}
-
-	var resp addChainResponse
-	if err := c.postAndParse(c.uri+AddChainPath, &req, &resp); err != nil {
-		return nil, err
-	}
-
-	sct := &ct.SignedCertificateTimestamp{
-		SCTVersion: ct.Version(resp.SCTVersion),
-		Timestamp:  resp.Timestamp,
-		Extensions: resp.Extensions,
-	}
-
-	if len(resp.ID) != sha256.Size {
-		return nil, fmt.Errorf("SCT returned by server has invalid id (expected length %d got %d)", sha256.Size, len(resp.ID))
-	}
-	copy(sct.LogID[:], resp.ID)
-
-	ds, err := ct.UnmarshalDigitallySigned(bytes.NewReader(resp.Signature))
-	if err != nil {
-		return nil, err
-	}
-	sct.Signature = *ds
-	return sct, nil
-}

ct/serialization.go

@@ -1,462 +0,0 @@
-package ct
-
-import (
-	"bytes"
-	"container/list"
-	"crypto"
-	"encoding/binary"
-	"errors"
-	"fmt"
-	"io"
-)
-
-// Variable size structure prefix-header byte lengths
-const (
-	CertificateLengthBytes      = 3
-	PreCertificateLengthBytes   = 3
-	ExtensionsLengthBytes       = 2
-	CertificateChainLengthBytes = 3
-	SignatureLengthBytes        = 2
-)
-
-// Max lengths
-const (
-	MaxCertificateLength = (1 << 24) - 1
-	MaxExtensionsLength  = (1 << 16) - 1
-)
-
-func writeUint(w io.Writer, value uint64, numBytes int) error {
-	buf := make([]uint8, numBytes)
-	for i := 0; i < numBytes; i++ {
-		buf[numBytes-i-1] = uint8(value & 0xff)
-		value >>= 8
-	}
-	if value != 0 {
-		return errors.New("numBytes was insufficiently large to represent value")
-	}
-	if _, err := w.Write(buf); err != nil {
-		return err
-	}
-	return nil
-}
-
-func writeVarBytes(w io.Writer, value []byte, numLenBytes int) error {
-	if err := writeUint(w, uint64(len(value)), numLenBytes); err != nil {
-		return err
-	}
-	if _, err := w.Write(value); err != nil {
-		return err
-	}
-	return nil
-}
-
-func readUint(r io.Reader, numBytes int) (uint64, error) {
-	var l uint64
-	for i := 0; i < numBytes; i++ {
-		l <<= 8
-		var t uint8
-		if err := binary.Read(r, binary.BigEndian, &t); err != nil {
-			return 0, err
-		}
-		l |= uint64(t)
-	}
-	return l, nil
-}
-
-// Reads a variable length array of bytes from |r|. |numLenBytes| specifies the
-// number of (BigEndian) prefix-bytes which contain the length of the actual
-// array data bytes that follow.
-// Allocates an array to hold the contents and returns a slice view into it if
-// the read was successful, or an error otherwise.
-func readVarBytes(r io.Reader, numLenBytes int) ([]byte, error) {
-	switch {
-	case numLenBytes > 8:
-		return nil, fmt.Errorf("numLenBytes too large (%d)", numLenBytes)
-	case numLenBytes == 0:
-		return nil, errors.New("numLenBytes should be > 0")
-	}
-	l, err := readUint(r, numLenBytes)
-	if err != nil {
-		return nil, err
-	}
-	data := make([]byte, l)
-	if n, err := io.ReadFull(r, data); err != nil {
-		if err == io.EOF || err == io.ErrUnexpectedEOF {
-			return nil, fmt.Errorf("short read: expected %d but got %d", l, n)
-		}
-		return nil, err
-	}
-	return data, nil
-}
-
-// Reads a list of ASN1Cert types from |r|
-func readASN1CertList(r io.Reader, totalLenBytes int, elementLenBytes int) ([]ASN1Cert, error) {
-	listBytes, err := readVarBytes(r, totalLenBytes)
-	if err != nil {
-		return []ASN1Cert{}, err
-	}
-	list := list.New()
-	listReader := bytes.NewReader(listBytes)
-	var entry []byte
-	for err == nil {
-		entry, err = readVarBytes(listReader, elementLenBytes)
-		if err != nil {
-			if err != io.EOF {
-				return []ASN1Cert{}, err
-			}
-		} else {
-			list.PushBack(entry)
-		}
-	}
-	ret := make([]ASN1Cert, list.Len())
-	i := 0
-	for e := list.Front(); e != nil; e = e.Next() {
-		ret[i] = e.Value.([]byte)
-		i++
-	}
-	return ret, nil
-}
-
-// ReadTimestampedEntryInto parses the byte-stream representation of a
-// TimestampedEntry from |r| and populates the struct |t| with the data.  See
-// RFC section 3.4 for details on the format.
-// Returns a non-nil error if there was a problem.
-func ReadTimestampedEntryInto(r io.Reader, t *TimestampedEntry) error {
-	var err error
-	if err = binary.Read(r, binary.BigEndian, &t.Timestamp); err != nil {
-		return err
-	}
-	if err = binary.Read(r, binary.BigEndian, &t.EntryType); err != nil {
-		return err
-	}
-	switch t.EntryType {
-	case X509LogEntryType:
-		if t.X509Entry, err = readVarBytes(r, CertificateLengthBytes); err != nil {
-			return err
-		}
-	case PrecertLogEntryType:
-		if err := binary.Read(r, binary.BigEndian, &t.PrecertEntry.IssuerKeyHash); err != nil {
-			return err
-		}
-		if t.PrecertEntry.TBSCertificate, err = readVarBytes(r, PreCertificateLengthBytes); err != nil {
-			return err
-		}
-	default:
-		return fmt.Errorf("unknown EntryType: %d", t.EntryType)
-	}
-	t.Extensions, err = readVarBytes(r, ExtensionsLengthBytes)
-	return nil
-}
-
-// ReadMerkleTreeLeaf parses the byte-stream representation of a MerkleTreeLeaf
-// and returns a pointer to a new MerkleTreeLeaf structure containing the
-// parsed data.
-// See RFC section 3.4 for details on the format.
-// Returns a pointer to a new MerkleTreeLeaf or non-nil error if there was a
-// problem
-func ReadMerkleTreeLeaf(r io.Reader) (*MerkleTreeLeaf, error) {
-	var m MerkleTreeLeaf
-	if err := binary.Read(r, binary.BigEndian, &m.Version); err != nil {
-		return nil, err
-	}
-	if m.Version != V1 {
-		return nil, fmt.Errorf("unknown Version %d", m.Version)
-	}
-	if err := binary.Read(r, binary.BigEndian, &m.LeafType); err != nil {
-		return nil, err
-	}
-	if m.LeafType != TimestampedEntryLeafType {
-		return nil, fmt.Errorf("unknown LeafType %d", m.LeafType)
-	}
-	if err := ReadTimestampedEntryInto(r, &m.TimestampedEntry); err != nil {
-		return nil, err
-	}
-	return &m, nil
-}
-
-// UnmarshalX509ChainArray unmarshalls the contents of the "chain:" entry in a
-// GetEntries response in the case where the entry refers to an X509 leaf.
-func UnmarshalX509ChainArray(b []byte) ([]ASN1Cert, error) {
-	return readASN1CertList(bytes.NewReader(b), CertificateChainLengthBytes, CertificateLengthBytes)
-}
-
-// UnmarshalPrecertChainArray unmarshalls the contents of the "chain:" entry in
-// a GetEntries response in the case where the entry refers to a Precertificate
-// leaf.
-func UnmarshalPrecertChainArray(b []byte) ([]ASN1Cert, error) {
-	var chain []ASN1Cert
-
-	reader := bytes.NewReader(b)
-	// read the pre-cert entry:
-	precert, err := readVarBytes(reader, CertificateLengthBytes)
-	if err != nil {
-		return chain, err
-	}
-	chain = append(chain, precert)
-	// and then read and return the chain up to the root:
-	remainingChain, err := readASN1CertList(reader, CertificateChainLengthBytes, CertificateLengthBytes)
-	if err != nil {
-		return chain, err
-	}
-	chain = append(chain, remainingChain...)
-	return chain, nil
-}
-
-// UnmarshalDigitallySigned reconstructs a DigitallySigned structure from a Reader
-func UnmarshalDigitallySigned(r io.Reader) (*DigitallySigned, error) {
-	var h byte
-	if err := binary.Read(r, binary.BigEndian, &h); err != nil {
-		return nil, fmt.Errorf("failed to read HashAlgorithm: %v", err)
-	}
-
-	var s byte
-	if err := binary.Read(r, binary.BigEndian, &s); err != nil {
-		return nil, fmt.Errorf("failed to read SignatureAlgorithm: %v", err)
-	}
-
-	sig, err := readVarBytes(r, SignatureLengthBytes)
-	if err != nil {
-		return nil, fmt.Errorf("failed to read Signature bytes: %v", err)
-	}
-
-	return &DigitallySigned{
-		HashAlgorithm:      HashAlgorithm(h),
-		SignatureAlgorithm: SignatureAlgorithm(s),
-		Signature:          sig,
-	}, nil
-}
-
-// MarshalDigitallySigned marshalls a DigitallySigned structure into a byte array
-func MarshalDigitallySigned(ds DigitallySigned) ([]byte, error) {
-	var b bytes.Buffer
-	if err := b.WriteByte(byte(ds.HashAlgorithm)); err != nil {
-		return nil, fmt.Errorf("failed to write HashAlgorithm: %v", err)
-	}
-	if err := b.WriteByte(byte(ds.SignatureAlgorithm)); err != nil {
-		return nil, fmt.Errorf("failed to write SignatureAlgorithm: %v", err)
-	}
-	if err := writeVarBytes(&b, ds.Signature, SignatureLengthBytes); err != nil {
-		return nil, fmt.Errorf("failed to write HashAlgorithm: %v", err)
-	}
-	return b.Bytes(), nil
-}
-
-func checkCertificateFormat(cert ASN1Cert) error {
-	if len(cert) == 0 {
-		return errors.New("certificate is zero length")
-	}
-	if len(cert) > MaxCertificateLength {
-		return errors.New("certificate too large")
-	}
-	return nil
-}
-
-func checkExtensionsFormat(ext CTExtensions) error {
-	if len(ext) > MaxExtensionsLength {
-		return errors.New("extensions too large")
-	}
-	return nil
-}
-
-func serializeV1CertSCTSignatureInput(timestamp uint64, cert ASN1Cert, ext CTExtensions) ([]byte, error) {
-	if err := checkCertificateFormat(cert); err != nil {
-		return nil, err
-	}
-	if err := checkExtensionsFormat(ext); err != nil {
-		return nil, err
-	}
-	var buf bytes.Buffer
-	if err := binary.Write(&buf, binary.BigEndian, V1); err != nil {
-		return nil, err
-	}
-	if err := binary.Write(&buf, binary.BigEndian, CertificateTimestampSignatureType); err != nil {
-		return nil, err
-	}
-	if err := binary.Write(&buf, binary.BigEndian, timestamp); err != nil {
-		return nil, err
-	}
-	if err := binary.Write(&buf, binary.BigEndian, X509LogEntryType); err != nil {
-		return nil, err
-	}
-	if err := writeVarBytes(&buf, cert, CertificateLengthBytes); err != nil {
-		return nil, err
-	}
-	if err := writeVarBytes(&buf, ext, ExtensionsLengthBytes); err != nil {
-		return nil, err
-	}
-	return buf.Bytes(), nil
-}
-
-func serializeV1PrecertSCTSignatureInput(timestamp uint64, issuerKeyHash [issuerKeyHashLength]byte, tbs []byte, ext CTExtensions) ([]byte, error) {
-	if err := checkCertificateFormat(tbs); err != nil {
-		return nil, err
-	}
-	if err := checkExtensionsFormat(ext); err != nil {
-		return nil, err
-	}
-	var buf bytes.Buffer
-	if err := binary.Write(&buf, binary.BigEndian, V1); err != nil {
-		return nil, err
-	}
-	if err := binary.Write(&buf, binary.BigEndian, CertificateTimestampSignatureType); err != nil {
-		return nil, err
-	}
-	if err := binary.Write(&buf, binary.BigEndian, timestamp); err != nil {
-		return nil, err
-	}
-	if err := binary.Write(&buf, binary.BigEndian, PrecertLogEntryType); err != nil {
-		return nil, err
-	}
-	if _, err := buf.Write(issuerKeyHash[:]); err != nil {
-		return nil, err
-	}
-	if err := writeVarBytes(&buf, tbs, CertificateLengthBytes); err != nil {
-		return nil, err
-	}
-	if err := writeVarBytes(&buf, ext, ExtensionsLengthBytes); err != nil {
-		return nil, err
-	}
-	return buf.Bytes(), nil
-}
-
-func serializeV1SCTSignatureInput(sct SignedCertificateTimestamp, entry LogEntry) ([]byte, error) {
-	if sct.SCTVersion != V1 {
-		return nil, fmt.Errorf("unsupported SCT version, expected V1, but got %s", sct.SCTVersion)
-	}
-	if entry.Leaf.LeafType != TimestampedEntryLeafType {
-		return nil, fmt.Errorf("Unsupported leaf type %s", entry.Leaf.LeafType)
-	}
-	switch entry.Leaf.TimestampedEntry.EntryType {
-	case X509LogEntryType:
-		return serializeV1CertSCTSignatureInput(sct.Timestamp, entry.Leaf.TimestampedEntry.X509Entry, entry.Leaf.TimestampedEntry.Extensions)
-	case PrecertLogEntryType:
-		return serializeV1PrecertSCTSignatureInput(sct.Timestamp, entry.Leaf.TimestampedEntry.PrecertEntry.IssuerKeyHash,
-			entry.Leaf.TimestampedEntry.PrecertEntry.TBSCertificate,
-			entry.Leaf.TimestampedEntry.Extensions)
-	default:
-		return nil, fmt.Errorf("unknown TimestampedEntryLeafType %s", entry.Leaf.TimestampedEntry.EntryType)
-	}
-}
-
-// SerializeSCTSignatureInput serializes the passed in sct and log entry into
-// the correct format for signing.
-func SerializeSCTSignatureInput(sct SignedCertificateTimestamp, entry LogEntry) ([]byte, error) {
-	switch sct.SCTVersion {
-	case V1:
-		return serializeV1SCTSignatureInput(sct, entry)
-	default:
-		return nil, fmt.Errorf("unknown SCT version %d", sct.SCTVersion)
-	}
-}
-
-func serializeV1SCT(sct SignedCertificateTimestamp) ([]byte, error) {
-	if err := checkExtensionsFormat(sct.Extensions); err != nil {
-		return nil, err
-	}
-	var buf bytes.Buffer
-	if err := binary.Write(&buf, binary.BigEndian, V1); err != nil {
-		return nil, err
-	}
-	if err := binary.Write(&buf, binary.BigEndian, sct.LogID); err != nil {
-		return nil, err
-	}
-	if err := binary.Write(&buf, binary.BigEndian, sct.Timestamp); err != nil {
-		return nil, err
-	}
-	if err := writeVarBytes(&buf, sct.Extensions, ExtensionsLengthBytes); err != nil {
-		return nil, err
-	}
-	sig, err := MarshalDigitallySigned(sct.Signature)
-	if err != nil {
-		return nil, err
-	}
-	if err := binary.Write(&buf, binary.BigEndian, sig); err != nil {
-		return nil, err
-	}
-	return buf.Bytes(), nil
-}
-
-// SerializeSCT serializes the passed in sct into the format specified
-// by RFC6962 section 3.2
-func SerializeSCT(sct SignedCertificateTimestamp) ([]byte, error) {
-	switch sct.SCTVersion {
-	case V1:
-		return serializeV1SCT(sct)
-	default:
-		return nil, fmt.Errorf("unknown SCT version %d", sct.SCTVersion)
-	}
-}
-
-func deserializeSCTV1(r io.Reader, sct *SignedCertificateTimestamp) error {
-	if err := binary.Read(r, binary.BigEndian, &sct.LogID); err != nil {
-		return err
-	}
-	if err := binary.Read(r, binary.BigEndian, &sct.Timestamp); err != nil {
-		return err
-	}
-	ext, err := readVarBytes(r, ExtensionsLengthBytes)
-	if err != nil {
-		return err
-	}
-	sct.Extensions = ext
-	ds, err := UnmarshalDigitallySigned(r)
-	if err != nil {
-		return err
-	}
-	sct.Signature = *ds
-	return nil
-}
-
-func DeserializeSCT(r io.Reader) (*SignedCertificateTimestamp, error) {
-	var sct SignedCertificateTimestamp
-	if err := binary.Read(r, binary.BigEndian, &sct.SCTVersion); err != nil {
-		return nil, err
-	}
-	switch sct.SCTVersion {
-	case V1:
-		return &sct, deserializeSCTV1(r, &sct)
-	default:
-		return nil, fmt.Errorf("unknown SCT version %d", sct.SCTVersion)
-	}
-}
-
-func serializeV1STHSignatureInput(sth SignedTreeHead) ([]byte, error) {
-	if sth.Version != V1 {
-		return nil, fmt.Errorf("invalid STH version %d", sth.Version)
-	}
-	if sth.TreeSize < 0 {
-		return nil, fmt.Errorf("invalid tree size %d", sth.TreeSize)
-	}
-	if len(sth.SHA256RootHash) != crypto.SHA256.Size() {
-		return nil, fmt.Errorf("invalid TreeHash length, got %d expected %d", len(sth.SHA256RootHash), crypto.SHA256.Size())
-	}
-
-	var buf bytes.Buffer
-	if err := binary.Write(&buf, binary.BigEndian, V1); err != nil {
-		return nil, err
-	}
-	if err := binary.Write(&buf, binary.BigEndian, TreeHashSignatureType); err != nil {
-		return nil, err
-	}
-	if err := binary.Write(&buf, binary.BigEndian, sth.Timestamp); err != nil {
-		return nil, err
-	}
-	if err := binary.Write(&buf, binary.BigEndian, sth.TreeSize); err != nil {
-		return nil, err
-	}
-	if err := binary.Write(&buf, binary.BigEndian, sth.SHA256RootHash); err != nil {
-		return nil, err
-	}
-	return buf.Bytes(), nil
-}
-
-// SerializeSTHSignatureInput serializes the passed in sth into the correct
-// format for signing.
-func SerializeSTHSignatureInput(sth SignedTreeHead) ([]byte, error) {
-	switch sth.Version {
-	case V1:
-		return serializeV1STHSignatureInput(sth)
-	default:
-		return nil, fmt.Errorf("unsupported STH version %d", sth.Version)
-	}
-}

ct/signatures.go

@@ -1,110 +0,0 @@
-package ct
-
-import (
-	"crypto"
-	"crypto/ecdsa"
-	"crypto/rsa"
-	"crypto/sha256"
-	"crypto/x509"
-	"encoding/asn1"
-	"encoding/pem"
-	"errors"
-	"fmt"
-	"log"
-	"math/big"
-)
-
-// PublicKeyFromPEM parses a PEM formatted block and returns the public key contained within and any remaining unread bytes, or an error.
-func PublicKeyFromPEM(b []byte) (crypto.PublicKey, SHA256Hash, []byte, error) {
-	p, rest := pem.Decode(b)
-	if p == nil {
-		return nil, [sha256.Size]byte{}, rest, fmt.Errorf("no PEM block found in %s", string(b))
-	}
-	k, err := x509.ParsePKIXPublicKey(p.Bytes)
-	return k, sha256.Sum256(p.Bytes), rest, err
-}
-
-// SignatureVerifier can verify signatures on SCTs and STHs
-type SignatureVerifier struct {
-	pubKey crypto.PublicKey
-}
-
-// NewSignatureVerifier creates a new SignatureVerifier using the passed in PublicKey.
-func NewSignatureVerifier(pk crypto.PublicKey) (*SignatureVerifier, error) {
-	switch pkType := pk.(type) {
-	case *rsa.PublicKey:
-	case *ecdsa.PublicKey:
-	default:
-		return nil, fmt.Errorf("Unsupported public key type %v", pkType)
-	}
-
-	return &SignatureVerifier{
-		pubKey: pk,
-	}, nil
-}
-
-// verifySignature verifies that the passed in signature over data was created by our PublicKey.
-// Currently, only SHA256 is supported as a HashAlgorithm, and only ECDSA and RSA signatures are supported.
-func (s SignatureVerifier) verifySignature(data []byte, sig DigitallySigned) error {
-	if sig.HashAlgorithm != SHA256 {
-		return fmt.Errorf("unsupported HashAlgorithm in signature: %v", sig.HashAlgorithm)
-	}
-
-	hasherType := crypto.SHA256
-	hasher := hasherType.New()
-	if _, err := hasher.Write(data); err != nil {
-		return fmt.Errorf("failed to write to hasher: %v", err)
-	}
-	hash := hasher.Sum([]byte{})
-
-	switch sig.SignatureAlgorithm {
-	case RSA:
-		rsaKey, ok := s.pubKey.(*rsa.PublicKey)
-		if !ok {
-			return fmt.Errorf("cannot verify RSA signature with %T key", s.pubKey)
-		}
-		if err := rsa.VerifyPKCS1v15(rsaKey, hasherType, hash, sig.Signature); err != nil {
-			return fmt.Errorf("failed to verify rsa signature: %v", err)
-		}
-	case ECDSA:
-		ecdsaKey, ok := s.pubKey.(*ecdsa.PublicKey)
-		if !ok {
-			return fmt.Errorf("cannot verify ECDSA signature with %T key", s.pubKey)
-		}
-		var ecdsaSig struct {
-			R, S *big.Int
-		}
-		rest, err := asn1.Unmarshal(sig.Signature, &ecdsaSig)
-		if err != nil {
-			return fmt.Errorf("failed to unmarshal ECDSA signature: %v", err)
-		}
-		if len(rest) != 0 {
-			log.Printf("Garbage following signature %v", rest)
-		}
-
-		if !ecdsa.Verify(ecdsaKey, hash, ecdsaSig.R, ecdsaSig.S) {
-			return errors.New("failed to verify ecdsa signature")
-		}
-	default:
-		return fmt.Errorf("unsupported signature type %v", sig.SignatureAlgorithm)
-	}
-	return nil
-}
-
-// VerifySCTSignature verifies that the SCT's signature is valid for the given LogEntry
-func (s SignatureVerifier) VerifySCTSignature(sct SignedCertificateTimestamp, entry LogEntry) error {
-	sctData, err := SerializeSCTSignatureInput(sct, entry)
-	if err != nil {
-		return err
-	}
-	return s.verifySignature(sctData, sct.Signature)
-}
-
-// VerifySTHSignature verifies that the STH's signature is valid.
-func (s SignatureVerifier) VerifySTHSignature(sth SignedTreeHead) error {
-	sthData, err := SerializeSTHSignatureInput(sth)
-	if err != nil {
-		return err
-	}
-	return s.verifySignature(sthData, sth.TreeHeadSignature)
-}

ct/types.go

@@ -1,323 +0,0 @@
-package ct
-
-import (
-	"bytes"
-	"crypto/sha256"
-	"encoding/base64"
-	"encoding/json"
-	"fmt"
-)
-
-const (
-	issuerKeyHashLength = 32
-)
-
-///////////////////////////////////////////////////////////////////////////////
-// The following structures represent those outlined in the RFC6962 document:
-///////////////////////////////////////////////////////////////////////////////
-
-// LogEntryType represents the LogEntryType enum from section 3.1 of the RFC:
-//   enum { x509_entry(0), precert_entry(1), (65535) } LogEntryType;
-type LogEntryType uint16
-
-func (e LogEntryType) String() string {
-	switch e {
-	case X509LogEntryType:
-		return "X509LogEntryType"
-	case PrecertLogEntryType:
-		return "PrecertLogEntryType"
-	}
-	panic(fmt.Sprintf("No string defined for LogEntryType constant value %d", e))
-}
-
-// LogEntryType constants, see section 3.1 of RFC6962.
-const (
-	X509LogEntryType    LogEntryType = 0
-	PrecertLogEntryType LogEntryType = 1
-)
-
-// MerkleLeafType represents the MerkleLeafType enum from section 3.4 of the
-// RFC: enum { timestamped_entry(0), (255) } MerkleLeafType;
-type MerkleLeafType uint8
-
-func (m MerkleLeafType) String() string {
-	switch m {
-	case TimestampedEntryLeafType:
-		return "TimestampedEntryLeafType"
-	default:
-		return fmt.Sprintf("UnknownLeafType(%d)", m)
-	}
-}
-
-// MerkleLeafType constants, see section 3.4 of the RFC.
-const (
-	TimestampedEntryLeafType MerkleLeafType = 0 // Entry type for an SCT
-)
-
-// Version represents the Version enum from section 3.2 of the RFC:
-// enum { v1(0), (255) } Version;
-type Version uint8
-
-func (v Version) String() string {
-	switch v {
-	case V1:
-		return "V1"
-	default:
-		return fmt.Sprintf("UnknownVersion(%d)", v)
-	}
-}
-
-// CT Version constants, see section 3.2 of the RFC.
-const (
-	V1 Version = 0
-)
-
-// SignatureType differentiates STH signatures from SCT signatures, see RFC
-// section 3.2
-type SignatureType uint8
-
-func (st SignatureType) String() string {
-	switch st {
-	case CertificateTimestampSignatureType:
-		return "CertificateTimestamp"
-	case TreeHashSignatureType:
-		return "TreeHash"
-	default:
-		return fmt.Sprintf("UnknownSignatureType(%d)", st)
-	}
-}
-
-// SignatureType constants, see RFC section 3.2
-const (
-	CertificateTimestampSignatureType SignatureType = 0
-	TreeHashSignatureType             SignatureType = 1
-)
-
-// ASN1Cert type for holding the raw DER bytes of an ASN.1 Certificate
-// (section 3.1)
-type ASN1Cert []byte
-
-// PreCert represents a Precertificate (section 3.2)
-type PreCert struct {
-	IssuerKeyHash  [issuerKeyHashLength]byte
-	TBSCertificate []byte
-}
-
-// CTExtensions is a representation of the raw bytes of any CtExtension
-// structure (see section 3.2)
-type CTExtensions []byte
-
-// MerkleTreeNode represents an internal node in the CT tree
-type MerkleTreeNode []byte
-
-// ConsistencyProof represents a CT consistency proof (see sections 2.1.2 and
-// 4.4)
-type ConsistencyProof []MerkleTreeNode
-
-// AuditPath represents a CT inclusion proof (see sections 2.1.1 and 4.5)
-type AuditPath []MerkleTreeNode
-
-// LeafInput represents a serialized MerkleTreeLeaf structure
-type LeafInput []byte
-
-// HashAlgorithm from the DigitallySigned struct
-type HashAlgorithm byte
-
-// HashAlgorithm constants
-const (
-	None   HashAlgorithm = 0
-	MD5    HashAlgorithm = 1
-	SHA1   HashAlgorithm = 2
-	SHA224 HashAlgorithm = 3
-	SHA256 HashAlgorithm = 4
-	SHA384 HashAlgorithm = 5
-	SHA512 HashAlgorithm = 6
-)
-
-func (h HashAlgorithm) String() string {
-	switch h {
-	case None:
-		return "None"
-	case MD5:
-		return "MD5"
-	case SHA1:
-		return "SHA1"
-	case SHA224:
-		return "SHA224"
-	case SHA256:
-		return "SHA256"
-	case SHA384:
-		return "SHA384"
-	case SHA512:
-		return "SHA512"
-	default:
-		return fmt.Sprintf("UNKNOWN(%d)", h)
-	}
-}
-
-// SignatureAlgorithm from the the DigitallySigned struct
-type SignatureAlgorithm byte
-
-// SignatureAlgorithm constants
-const (
-	Anonymous SignatureAlgorithm = 0
-	RSA       SignatureAlgorithm = 1
-	DSA       SignatureAlgorithm = 2
-	ECDSA     SignatureAlgorithm = 3
-)
-
-func (s SignatureAlgorithm) String() string {
-	switch s {
-	case Anonymous:
-		return "Anonymous"
-	case RSA:
-		return "RSA"
-	case DSA:
-		return "DSA"
-	case ECDSA:
-		return "ECDSA"
-	default:
-		return fmt.Sprintf("UNKNOWN(%d)", s)
-	}
-}
-
-// DigitallySigned represents an RFC5246 DigitallySigned structure
-type DigitallySigned struct {
-	HashAlgorithm      HashAlgorithm
-	SignatureAlgorithm SignatureAlgorithm
-	Signature          []byte
-}
-
-// FromBase64String populates the DigitallySigned structure from the base64 data passed in.
-// Returns an error if the base64 data is invalid.
-func (d *DigitallySigned) FromBase64String(b64 string) error {
-	raw, err := base64.StdEncoding.DecodeString(b64)
-	if err != nil {
-		return fmt.Errorf("failed to unbase64 DigitallySigned: %v", err)
-	}
-	ds, err := UnmarshalDigitallySigned(bytes.NewReader(raw))
-	if err != nil {
-		return fmt.Errorf("failed to unmarshal DigitallySigned: %v", err)
-	}
-	*d = *ds
-	return nil
-}
-
-// Base64String returns the base64 representation of the DigitallySigned struct.
-func (d DigitallySigned) Base64String() (string, error) {
-	b, err := MarshalDigitallySigned(d)
-	if err != nil {
-		return "", err
-	}
-	return base64.StdEncoding.EncodeToString(b), nil
-}
-
-// MarshalJSON implements the json.Marshaller interface.
-func (d DigitallySigned) MarshalJSON() ([]byte, error) {
-	b64, err := d.Base64String()
-	if err != nil {
-		return []byte{}, err
-	}
-	return []byte(`"` + b64 + `"`), nil
-}
-
-// UnmarshalJSON implements the json.Unmarshaler interface.
-func (d *DigitallySigned) UnmarshalJSON(b []byte) error {
-	var content string
-	if err := json.Unmarshal(b, &content); err != nil {
-		return fmt.Errorf("failed to unmarshal DigitallySigned: %v", err)
-	}
-	return d.FromBase64String(content)
-}
-
-// LogEntry represents the contents of an entry in a CT log, see section 3.1.
-type LogEntry struct {
-	Index     int64
-	Leaf      MerkleTreeLeaf
-	Chain     []ASN1Cert
-	LeafBytes []byte
-}
-
-// SHA256Hash represents the output from the SHA256 hash function.
-type SHA256Hash [sha256.Size]byte
-
-// FromBase64String populates the SHA256 struct with the contents of the base64 data passed in.
-func (s *SHA256Hash) FromBase64String(b64 string) error {
-	bs, err := base64.StdEncoding.DecodeString(b64)
-	if err != nil {
-		return fmt.Errorf("failed to unbase64 LogID: %v", err)
-	}
-	if len(bs) != sha256.Size {
-		return fmt.Errorf("invalid SHA256 length, expected 32 but got %d", len(bs))
-	}
-	copy(s[:], bs)
-	return nil
-}
-
-// Base64String returns the base64 representation of this SHA256Hash.
-func (s SHA256Hash) Base64String() string {
-	return base64.StdEncoding.EncodeToString(s[:])
-}
-
-// MarshalJSON implements the json.Marshaller interface for SHA256Hash.
-func (s SHA256Hash) MarshalJSON() ([]byte, error) {
-	return []byte(`"` + s.Base64String() + `"`), nil
-}
-
-// UnmarshalJSON implements the json.Unmarshaller interface.
-func (s *SHA256Hash) UnmarshalJSON(b []byte) error {
-	var content string
-	if err := json.Unmarshal(b, &content); err != nil {
-		return fmt.Errorf("failed to unmarshal SHA256Hash: %v", err)
-	}
-	return s.FromBase64String(content)
-}
-
-// SignedTreeHead represents the structure returned by the get-sth CT method
-// after base64 decoding. See sections 3.5 and 4.3 in the RFC)
-type SignedTreeHead struct {
-	Version           Version         `json:"sth_version"`         // The version of the protocol to which the STH conforms
-	TreeSize          uint64          `json:"tree_size"`           // The number of entries in the new tree
-	Timestamp         uint64          `json:"timestamp"`           // The time at which the STH was created
-	SHA256RootHash    SHA256Hash      `json:"sha256_root_hash"`    // The root hash of the log's Merkle tree
-	TreeHeadSignature DigitallySigned `json:"tree_head_signature"` // The Log's signature for this STH (see RFC section 3.5)
-	LogID             SHA256Hash      `json:"log_id"`              // The SHA256 hash of the log's public key
-}
-
-// SignedCertificateTimestamp represents the structure returned by the
-// add-chain and add-pre-chain methods after base64 decoding. (see RFC sections
-// 3.2 ,4.1 and 4.2)
-type SignedCertificateTimestamp struct {
-	SCTVersion Version          `json:"sct_version"`    // The version of the protocol to which the SCT conforms
-	LogID      SHA256Hash       `json:"id"` // the SHA-256 hash of the log's public key, calculated over
-	// the DER encoding of the key represented as SubjectPublicKeyInfo.
-	Timestamp  uint64           `json:"timestamp"`          // Timestamp (in ms since unix epoc) at which the SCT was issued
-	Extensions CTExtensions     `json:"extensions"`    // For future extensions to the protocol
-	Signature  DigitallySigned  `json:"signature"` // The Log's signature for this SCT
-}
-
-func (s SignedCertificateTimestamp) String() string {
-	return fmt.Sprintf("{Version:%d LogId:%s Timestamp:%d Extensions:'%s' Signature:%v}", s.SCTVersion,
-		base64.StdEncoding.EncodeToString(s.LogID[:]),
-		s.Timestamp,
-		s.Extensions,
-		s.Signature)
-}
-
-// TimestampedEntry is part of the MerkleTreeLeaf structure.
-// See RFC section 3.4
-type TimestampedEntry struct {
-	Timestamp    uint64
-	EntryType    LogEntryType
-	X509Entry    ASN1Cert
-	PrecertEntry PreCert
-	Extensions   CTExtensions
-}
-
-// MerkleTreeLeaf represents the deserialized sructure of the hash input for the
-// leaves of a log's Merkle tree. See RFC section 3.4
-type MerkleTreeLeaf struct {
-	Version          Version          // the version of the protocol to which the MerkleTreeLeaf corresponds
-	LeafType         MerkleLeafType   // The type of the leaf input, currently only TimestampedEntry can exist
-	TimestampedEntry TimestampedEntry // The entry data itself
-}

ctclient/client.go

@@ -0,0 +1,108 @@
+// Copyright (C) 2025 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+// Package ctclient implements a client for monitoring RFC6962 and static-ct-api Certificate Transparency logs
+package ctclient
+
+import (
+	"context"
+	"crypto/tls"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"net"
+	"net/http"
+	"net/url"
+	"time"
+)
+
+// Create an HTTP client suitable for communicating with CT logs.  dialContext, if non-nil, is used for dialing.
+func NewHTTPClient(dialContext func(context.Context, string, string) (net.Conn, error)) *http.Client {
+	return &http.Client{
+		Transport: &http.Transport{
+			Proxy:                 http.ProxyFromEnvironment,
+			TLSHandshakeTimeout:   15 * time.Second,
+			ResponseHeaderTimeout: 30 * time.Second,
+			MaxIdleConnsPerHost:   10,
+			DisableKeepAlives:     false,
+			MaxIdleConns:          100,
+			IdleConnTimeout:       15 * time.Second,
+			ExpectContinueTimeout: 1 * time.Second,
+			TLSClientConfig: &tls.Config{
+				// We have to disable TLS certificate validation because because several logs
+				// (WoSign, StartCom, GDCA) use certificates that are not widely trusted.
+				// Since we verify that every response we receive from the log is signed
+				// by the log's CT public key (either directly, or indirectly via the Merkle Tree),
+				// TLS certificate validation is not actually necessary.  (We don't want to manage
+				// our own trust store because that adds undesired complexity and would require
+				// updating should a log ever change to a different CA.)
+				InsecureSkipVerify: true,
+			},
+			DialContext: dialContext,
+		},
+		CheckRedirect: func(*http.Request, []*http.Request) error {
+			return errors.New("redirects not followed")
+		},
+		Timeout: 60 * time.Second,
+	}
+}
+
+var defaultHTTPClient = NewHTTPClient(nil)
+
+func get(ctx context.Context, httpClient *http.Client, fullURL string) ([]byte, error) {
+	request, err := http.NewRequestWithContext(ctx, http.MethodGet, fullURL, nil)
+	if err != nil {
+		return nil, err
+	}
+	request.Header.Set("User-Agent", "") // Don't send a User-Agent to make life harder for malicious logs
+
+	if httpClient == nil {
+		httpClient = defaultHTTPClient
+	}
+
+	response, err := httpClient.Do(request)
+	if err != nil {
+		return nil, err
+	}
+
+	responseBody, err := io.ReadAll(response.Body)
+	response.Body.Close()
+	if err != nil {
+		return nil, fmt.Errorf("Get %q: error reading response: %w", fullURL, err)
+	}
+
+	if response.StatusCode != 200 {
+		return nil, fmt.Errorf("Get %q: %s (%q)", fullURL, response.Status, string(responseBody))
+	}
+
+	return responseBody, nil
+}
+
+func getJSON(ctx context.Context, httpClient *http.Client, fullURL string, response any) error {
+	responseBytes, err := get(ctx, httpClient, fullURL)
+	if err != nil {
+		return err
+	}
+	if err := json.Unmarshal(responseBytes, response); err != nil {
+		return fmt.Errorf("Get %q: error parsing response JSON: %w", fullURL, err)
+	}
+	return nil
+}
+
+func getRoots(ctx context.Context, httpClient *http.Client, logURL *url.URL) ([][]byte, error) {
+	fullURL := logURL.JoinPath("/ct/v1/get-roots").String()
+	var parsedResponse struct {
+		Certificates [][]byte `json:"certificates"`
+	}
+	if err := getJSON(ctx, httpClient, fullURL, &parsedResponse); err != nil {
+		return nil, err
+	}
+	return parsedResponse.Certificates, nil
+}

ctclient/log.go

@@ -0,0 +1,53 @@
+// Copyright (C) 2025 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package ctclient
+
+import (
+	"context"
+
+	"software.sslmate.com/src/certspotter/cttypes"
+	"software.sslmate.com/src/certspotter/merkletree"
+)
+
+type Log interface {
+	GetSTH(context.Context) (*cttypes.SignedTreeHead, string, error)
+	GetRoots(context.Context) ([][]byte, error)
+	GetEntries(ctx context.Context, startInclusive, endInclusive uint64) ([]Entry, error)
+	ReconstructTree(context.Context, *cttypes.SignedTreeHead) (*merkletree.CollapsedTree, error)
+}
+
+// IssuerGetter represents a source of issuer certificates.
+//
+// If a [Log] also implements IssuerGetter, then it is mandatory to provide
+// an IssuerGetter when using [Entry]s returned by the [Log].  The IssuerGetter
+// may be the Log itself, or your own implementation which retrieves issuers
+// from a different source, such as a cache.
+//
+// If a Log doesn't implement IssuerGetter, then you may pass a nil IssuerGetter
+// when using the Log's Entrys.
+type IssuerGetter interface {
+	GetIssuer(ctx context.Context, fingerprint *[32]byte) ([]byte, error)
+}
+
+type Entry interface {
+	LeafInput() []byte
+
+	// Returns error from IssuerGetter, otherwise infallible
+	ExtraData(context.Context, IssuerGetter) ([]byte, error)
+
+	// Returns an error if this is not a well-formed precert entry
+	Precertificate() (cttypes.ASN1Cert, error)
+
+	// Returns an error if this is not a well-formed x509 or precert entry
+	ChainFingerprints() ([][32]byte, error)
+
+	// Returns an error if this is not a well-formed x509 or precert entry, or if IssuerGetter failed
+	GetChain(context.Context, IssuerGetter) (cttypes.ASN1CertChain, error)
+}

ctclient/rfc6962.go

@@ -0,0 +1,219 @@
+// Copyright (C) 2025 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package ctclient
+
+import (
+	"context"
+	"crypto/sha256"
+	"fmt"
+	"net/http"
+	"net/url"
+	"slices"
+
+	"software.sslmate.com/src/certspotter/cttypes"
+	"software.sslmate.com/src/certspotter/merkletree"
+)
+
+type RFC6962Log struct {
+	URL        *url.URL
+	HTTPClient *http.Client // nil to use default client
+}
+
+type RFC6962LogEntry struct {
+	Leaf_input []byte `json:"leaf_input"`
+	Extra_data []byte `json:"extra_data"`
+}
+
+func (ctlog *RFC6962Log) GetSTH(ctx context.Context) (*cttypes.SignedTreeHead, string, error) {
+	fullURL := ctlog.URL.JoinPath("/ct/v1/get-sth").String()
+	sth := new(cttypes.SignedTreeHead)
+	if err := getJSON(ctx, ctlog.HTTPClient, fullURL, sth); err != nil {
+		return nil, fullURL, err
+	}
+	return sth, fullURL, nil
+}
+
+func (ctlog *RFC6962Log) GetRoots(ctx context.Context) ([][]byte, error) {
+	return getRoots(ctx, ctlog.HTTPClient, ctlog.URL)
+}
+
+func (ctlog *RFC6962Log) getEntries(ctx context.Context, startInclusive uint64, endInclusive uint64) ([]RFC6962LogEntry, error) {
+	fullURL := ctlog.URL.JoinPath("/ct/v1/get-entries").String()
+	fullURL += fmt.Sprintf("?start=%d&end=%d", startInclusive, endInclusive)
+
+	var parsedResponse struct {
+		Entries []RFC6962LogEntry `json:"entries"`
+	}
+	if err := getJSON(ctx, ctlog.HTTPClient, fullURL, &parsedResponse); err != nil {
+		return nil, err
+	}
+	if len(parsedResponse.Entries) == 0 {
+		return nil, fmt.Errorf("Get %q: zero entries returned", fullURL)
+	}
+	if uint64(len(parsedResponse.Entries)) > endInclusive-startInclusive+1 {
+		return nil, fmt.Errorf("Get %q: extraneous entries returned", fullURL)
+	}
+	return parsedResponse.Entries, nil
+}
+
+func (ctlog *RFC6962Log) GetEntries(ctx context.Context, startInclusive uint64, endInclusive uint64) ([]Entry, error) {
+	nativeEntries, err := ctlog.getEntries(ctx, startInclusive, endInclusive)
+	if err != nil {
+		return nil, err
+	}
+	entries := make([]Entry, len(nativeEntries))
+	for i := range nativeEntries {
+		entries[i] = &nativeEntries[i]
+	}
+	return entries, nil
+}
+
+type entryAndProofResponse struct {
+	LeafInput []byte            `json:"leaf_input"`
+	ExtraData []byte            `json:"extra_data"`
+	AuditPath []merkletree.Hash `json:"audit_path"`
+}
+
+func (ctlog *RFC6962Log) getEntryAndProof(ctx context.Context, leafIndex uint64, treeSize uint64) (*entryAndProofResponse, error) {
+	fullURL := ctlog.URL.JoinPath("/ct/v1/get-entry-and-proof").String()
+	fullURL += fmt.Sprintf("?leaf_index=%d&tree_size=%d", leafIndex, treeSize)
+	response := new(entryAndProofResponse)
+	if err := getJSON(ctx, ctlog.HTTPClient, fullURL, response); err != nil {
+		return nil, err
+	}
+	return response, nil
+}
+
+type proofResponse struct {
+	LeafIndex uint64            `json:"leaf_index"`
+	AuditPath []merkletree.Hash `json:"audit_path"`
+}
+
+func (ctlog *RFC6962Log) getProofByHash(ctx context.Context, hash *merkletree.Hash, treeSize uint64) (*proofResponse, error) {
+	fullURL := ctlog.URL.JoinPath("/ct/v1/get-proof-by-hash").String()
+	fullURL += fmt.Sprintf("?hash=%s&tree_size=%d", url.QueryEscape(hash.Base64String()), treeSize)
+	response := new(proofResponse)
+	if err := getJSON(ctx, ctlog.HTTPClient, fullURL, response); err != nil {
+		return nil, err
+	}
+	return response, nil
+}
+
+func (ctlog *RFC6962Log) reconstructTree(ctx context.Context, treeSize uint64) (*merkletree.CollapsedTree, error) {
+	if treeSize == 0 {
+		return new(merkletree.CollapsedTree), nil
+	}
+	if entryAndProof, err := ctlog.getEntryAndProof(ctx, treeSize-1, treeSize); err == nil {
+		tree := new(merkletree.CollapsedTree)
+		slices.Reverse(entryAndProof.AuditPath)
+		if err := tree.Init(entryAndProof.AuditPath, treeSize-1); err != nil {
+			return nil, fmt.Errorf("log returned invalid audit proof for entry %d to STH %d: %w", treeSize-1, treeSize, err)
+		}
+		tree.Add(merkletree.HashLeaf(entryAndProof.LeafInput))
+		return tree, nil
+	}
+	entries, err := ctlog.getEntries(ctx, treeSize-1, treeSize-1)
+	if err != nil {
+		return nil, err
+	}
+	leafHash := merkletree.HashLeaf(entries[0].Leaf_input)
+	tree := new(merkletree.CollapsedTree)
+	if treeSize > 1 {
+		response, err := ctlog.getProofByHash(ctx, &leafHash, treeSize)
+		if err != nil {
+			return nil, err
+		}
+		if response.LeafIndex != treeSize-1 {
+			// This can happen if the leaf hash is present in the tree in more than one place. Unfortunately, we can't reconstruct when tree if this happens. Fortunately, this is really unlikely, and most logs support get-entry-and-proof anyways.
+			return nil, fmt.Errorf("unable to reconstruct tree because leaf hash %s is present in tree at more than one index (need proof for index %d but get-proof-by-hash returned proof for index %d)", leafHash.Base64String(), treeSize-1, response.LeafIndex)
+		}
+		slices.Reverse(response.AuditPath)
+		if err := tree.Init(response.AuditPath, treeSize-1); err != nil {
+			return nil, fmt.Errorf("log returned invalid audit proof for hash %s to STH %d: %w", leafHash.Base64String(), treeSize, err)
+		}
+	}
+	tree.Add(leafHash)
+	return tree, nil
+}
+
+func (ctlog *RFC6962Log) ReconstructTree(ctx context.Context, sth *cttypes.SignedTreeHead) (*merkletree.CollapsedTree, error) {
+	tree, err := ctlog.reconstructTree(ctx, sth.TreeSize)
+	if err != nil {
+		return nil, err
+	}
+
+	if rootHash := tree.CalculateRoot(); rootHash != sth.RootHash {
+		return nil, fmt.Errorf("calculated root hash (%s) does not match STH (%s) at size %d", rootHash.Base64String(), sth.RootHash.Base64String(), sth.TreeSize)
+	}
+
+	return tree, nil
+}
+
+func (entry *RFC6962LogEntry) isX509() bool {
+	return len(entry.Leaf_input) >= 12 && entry.Leaf_input[0] == 0 && entry.Leaf_input[1] == 0 && entry.Leaf_input[10] == 0 && entry.Leaf_input[11] == 0
+}
+
+func (entry *RFC6962LogEntry) isPrecert() bool {
+	return len(entry.Leaf_input) >= 12 && entry.Leaf_input[0] == 0 && entry.Leaf_input[1] == 0 && entry.Leaf_input[10] == 0 && entry.Leaf_input[11] == 1
+}
+
+func (entry *RFC6962LogEntry) LeafInput() []byte {
+	return entry.Leaf_input
+}
+
+func (entry *RFC6962LogEntry) ExtraData(context.Context, IssuerGetter) ([]byte, error) {
+	return entry.Extra_data, nil
+}
+
+func (entry *RFC6962LogEntry) Precertificate() (cttypes.ASN1Cert, error) {
+	if !entry.isPrecert() {
+		return nil, fmt.Errorf("not a precertificate entry")
+	}
+	extraData, err := cttypes.ParseExtraDataForPrecertEntry(entry.Extra_data)
+	if err != nil {
+		return nil, fmt.Errorf("error parsing extra_data: %w", err)
+	}
+	return extraData.PreCertificate, nil
+}
+
+func (entry *RFC6962LogEntry) ChainFingerprints() ([][32]byte, error) {
+	chain, err := entry.parseChain()
+	if err != nil {
+		return nil, err
+	}
+	fingerprints := make([][32]byte, len(chain))
+	for i := range chain {
+		fingerprints[i] = sha256.Sum256(chain[i])
+	}
+	return fingerprints, nil
+}
+
+func (entry *RFC6962LogEntry) GetChain(context.Context, IssuerGetter) (cttypes.ASN1CertChain, error) {
+	return entry.parseChain()
+}
+
+func (entry *RFC6962LogEntry) parseChain() (cttypes.ASN1CertChain, error) {
+	switch {
+	case entry.isX509():
+		extraData, err := cttypes.ParseExtraDataForX509Entry(entry.Extra_data)
+		if err != nil {
+			return nil, fmt.Errorf("error parsing extra_data for X509 entry: %w", err)
+		}
+		return extraData, nil
+	case entry.isPrecert():
+		extraData, err := cttypes.ParseExtraDataForPrecertEntry(entry.Extra_data)
+		if err != nil {
+			return nil, fmt.Errorf("error parsing extra_data for precert entry: %w", err)
+		}
+		return extraData.PrecertificateChain, nil
+	default:
+		return nil, fmt.Errorf("unknown entry type")
+	}
+}

ctclient/static.go

@@ -0,0 +1,397 @@
+// Copyright (C) 2025 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package ctclient
+
+import (
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"errors"
+	"fmt"
+	"net/http"
+	"net/url"
+	"strconv"
+	"sync"
+
+	"golang.org/x/crypto/cryptobyte"
+
+	"software.sslmate.com/src/certspotter/cttypes"
+	"software.sslmate.com/src/certspotter/merkletree"
+)
+
+const (
+	staticTileHeight = 8
+	StaticTileWidth  = 1 << staticTileHeight
+)
+
+func staticSubtreeSize(level uint64) uint64 { return 1 << (level * staticTileHeight) }
+
+type StaticLog struct {
+	SubmissionURL *url.URL
+	MonitoringURL *url.URL
+	ID            cttypes.LogID
+	HTTPClient    *http.Client // nil to use default client
+}
+
+type StaticLogEntry struct {
+	timestampedEntry []byte
+	precertificate   []byte // nil iff x509 entry; non-nil iff precert entry
+	chain            [][32]byte
+}
+
+func (ctlog *StaticLog) GetSTH(ctx context.Context) (*cttypes.SignedTreeHead, string, error) {
+	fullURL := ctlog.MonitoringURL.JoinPath("/checkpoint").String()
+	responseBody, err := get(ctx, ctlog.HTTPClient, fullURL)
+	if err != nil {
+		return nil, fullURL, err
+	}
+	sth, err := cttypes.ParseCheckpoint(responseBody, ctlog.ID)
+	if err != nil {
+		return nil, fullURL, err
+	}
+	return sth, fullURL, nil
+}
+
+func (ctlog *StaticLog) GetRoots(ctx context.Context) ([][]byte, error) {
+	return getRoots(ctx, ctlog.HTTPClient, ctlog.SubmissionURL)
+}
+
+func (ctlog *StaticLog) getEntries(ctx context.Context, startInclusive uint64, endInclusive uint64) ([]StaticLogEntry, error) {
+	var (
+		tile       = startInclusive / StaticTileWidth
+		skip       = startInclusive % StaticTileWidth
+		tileWidth  = min(StaticTileWidth, endInclusive+1-tile*StaticTileWidth)
+		numEntries = tileWidth - skip
+	)
+
+	data, err := ctlog.getDataTile(ctx, tile, tileWidth)
+	if err != nil {
+		return nil, err
+	}
+	var skippedEntry StaticLogEntry
+	for i := range skip {
+		if rest, err := skippedEntry.parse(data); err != nil {
+			return nil, fmt.Errorf("error parsing skipped entry %d in tile %d: %w", i, tile, err)
+		} else {
+			data = rest
+		}
+	}
+	entries := make([]StaticLogEntry, numEntries)
+	for i := range numEntries {
+		if rest, err := entries[i].parse(data); err != nil {
+			return nil, fmt.Errorf("error parsing entry %d in tile %d: %w", skip+i, tile, err)
+		} else {
+			data = rest
+		}
+	}
+	return entries, nil
+}
+
+func (ctlog *StaticLog) GetEntries(ctx context.Context, startInclusive uint64, endInclusive uint64) ([]Entry, error) {
+	nativeEntries, err := ctlog.getEntries(ctx, startInclusive, endInclusive)
+	if err != nil {
+		return nil, err
+	}
+	entries := make([]Entry, len(nativeEntries))
+	for i := range nativeEntries {
+		entries[i] = &nativeEntries[i]
+	}
+	return entries, nil
+}
+
+func (ctlog *StaticLog) ReconstructTree(ctx context.Context, sth *cttypes.SignedTreeHead) (*merkletree.CollapsedTree, error) {
+	type job struct {
+		level  uint64
+		offset uint64
+		width  uint64
+		tree   *merkletree.CollapsedTree
+		err    error
+	}
+	var jobs []job
+	for level, size := uint64(0), sth.TreeSize; size > 0; level++ {
+		fullTiles := size / StaticTileWidth
+		remainder := size % StaticTileWidth
+		size = fullTiles
+		if remainder > 0 {
+			jobs = append(jobs, job{
+				level:  level,
+				offset: fullTiles,
+				width:  remainder,
+			})
+		}
+	}
+	var wg sync.WaitGroup
+	for i := range jobs {
+		job := &jobs[i]
+		wg.Add(1)
+		go func() {
+			defer wg.Done()
+			job.tree, job.err = ctlog.getTileCollapsedTree(ctx, job.level, job.offset, job.width)
+		}()
+	}
+	wg.Wait()
+
+	var errs []error
+	tree := new(merkletree.CollapsedTree)
+	for i := range jobs {
+		job := &jobs[len(jobs)-1-i]
+		if job.err != nil {
+			errs = append(errs, job.err)
+			continue
+		}
+		if err := tree.Append(*job.tree); err != nil {
+			panic(err)
+		}
+	}
+	if len(errs) > 0 {
+		return nil, errors.Join(errs...)
+	}
+
+	if rootHash := tree.CalculateRoot(); rootHash != sth.RootHash {
+		return nil, fmt.Errorf("calculated root hash (%s) does not match STH (%s) at size %d", rootHash.Base64String(), sth.RootHash.Base64String(), sth.TreeSize)
+	}
+
+	return tree, nil
+}
+
+func (ctlog *StaticLog) getDataTile(ctx context.Context, tile uint64, width uint64) ([]byte, error) {
+	if width == 0 || width > StaticTileWidth {
+		panic("width is out of range")
+	}
+	var partialErr error
+	if width < StaticTileWidth {
+		fullURL := ctlog.MonitoringURL.JoinPath(formatTilePath("data", tile, width)).String()
+		if data, err := get(ctx, ctlog.HTTPClient, fullURL); err != nil {
+			partialErr = err
+		} else {
+			return data, nil
+		}
+	}
+
+	fullURL := ctlog.MonitoringURL.JoinPath(formatTilePath("data", tile, 0)).String()
+	if data, err := get(ctx, ctlog.HTTPClient, fullURL); err != nil {
+		if partialErr != nil {
+			return nil, partialErr
+		} else {
+			return nil, err
+		}
+	} else {
+		return data, nil
+	}
+}
+
+// returned slice is numHashes*merkletree.HashLen bytes long
+func (ctlog *StaticLog) getTile(ctx context.Context, level uint64, tile uint64, numHashes uint64) ([]byte, error) {
+	if numHashes == 0 || numHashes > StaticTileWidth {
+		panic("numHashes is out of range")
+	}
+
+	var partialErr error
+	if numHashes < StaticTileWidth {
+		fullURL := ctlog.MonitoringURL.JoinPath(formatTilePath(strconv.FormatUint(level, 10), tile, numHashes)).String()
+		if data, err := get(ctx, ctlog.HTTPClient, fullURL); err != nil {
+			partialErr = err
+		} else if expectedLen := merkletree.HashLen * int(numHashes); len(data) != expectedLen {
+			return nil, fmt.Errorf("%s returned %d bytes instead of expected %d", fullURL, len(data), expectedLen)
+		} else {
+			return data, nil
+		}
+	}
+
+	fullURL := ctlog.MonitoringURL.JoinPath(formatTilePath(strconv.FormatUint(level, 10), tile, 0)).String()
+	if data, err := get(ctx, ctlog.HTTPClient, fullURL); err != nil {
+		if partialErr != nil {
+			return nil, partialErr
+		} else {
+			return nil, err
+		}
+	} else if expectedLen := merkletree.HashLen * StaticTileWidth; len(data) != expectedLen {
+		return nil, fmt.Errorf("%s returned %d bytes instead of expected %d", fullURL, len(data), expectedLen)
+	} else {
+		desiredLen := merkletree.HashLen * int(numHashes)
+		return data[:desiredLen], nil
+	}
+}
+
+func (ctlog *StaticLog) getTileCollapsedTree(ctx context.Context, level uint64, tile uint64, numHashes uint64) (*merkletree.CollapsedTree, error) {
+	data, err := ctlog.getTile(ctx, level, tile, numHashes)
+	if err != nil {
+		return nil, err
+	}
+	subtreeSize := staticSubtreeSize(level)
+	offset := staticSubtreeSize(level+1) * tile
+
+	tree := new(merkletree.CollapsedTree)
+	if err := tree.InitSubtree(offset, nil, 0); err != nil {
+		panic(err)
+	}
+	for i := uint64(0); i < numHashes; i++ {
+		hash := (merkletree.Hash)(data[i*merkletree.HashLen : (i+1)*merkletree.HashLen])
+		var subtree merkletree.CollapsedTree
+		if err := subtree.InitSubtree(offset+i*subtreeSize, []merkletree.Hash{hash}, subtreeSize); err != nil {
+			panic(err)
+		}
+		if err := tree.Append(subtree); err != nil {
+			panic(err)
+		}
+	}
+	return tree, nil
+}
+
+func (ctlog *StaticLog) GetIssuer(ctx context.Context, fingerprint *[32]byte) ([]byte, error) {
+	fullURL := ctlog.MonitoringURL.JoinPath("/issuer/" + hex.EncodeToString(fingerprint[:])).String()
+	data, err := get(ctx, ctlog.HTTPClient, fullURL)
+	if err != nil {
+		return nil, err
+	}
+	if gotFingerprint := sha256.Sum256(data); gotFingerprint != *fingerprint {
+		return nil, fmt.Errorf("%s returned incorrect data with fingerprint %x", fullURL, gotFingerprint[:])
+	}
+	return data, nil
+}
+
+func (entry *StaticLogEntry) parse(input []byte) ([]byte, error) {
+	var skipped cryptobyte.String
+	str := cryptobyte.String(input)
+
+	// TimestampedEntry.timestamp
+	if !str.Skip(8) {
+		return nil, fmt.Errorf("error reading timestamp")
+	}
+	// TimestampedEntry.entry_type
+	var entryType uint16
+	if !str.ReadUint16(&entryType) {
+		return nil, fmt.Errorf("error reading entry type")
+	}
+	// TimestampedEntry.signed_entry
+	if entryType == 0 {
+		if !str.ReadUint24LengthPrefixed(&skipped) {
+			return nil, fmt.Errorf("error reading certificate")
+		}
+	} else if entryType == 1 {
+		if !str.Skip(32) {
+			return nil, fmt.Errorf("error reading issuer_key_hash")
+		}
+		if !str.ReadUint24LengthPrefixed(&skipped) {
+			return nil, fmt.Errorf("error reading tbs_certificate")
+		}
+	} else {
+		return nil, fmt.Errorf("invalid entry type %d", entryType)
+	}
+
+	// TimestampedEntry.extensions
+	if !str.ReadUint16LengthPrefixed(&skipped) {
+		return nil, fmt.Errorf("error reading extensions")
+	}
+
+	timestampedEntryLen := len(input) - len(str)
+	entry.timestampedEntry = input[:timestampedEntryLen]
+
+	// precertificate
+	if entryType == 1 {
+		var precertificate cryptobyte.String
+		if !str.ReadUint24LengthPrefixed(&precertificate) {
+			return nil, fmt.Errorf("error reading precertificate")
+		}
+		entry.precertificate = precertificate
+	} else {
+		entry.precertificate = nil
+	}
+
+	// certificate_chain
+	var chainBytes cryptobyte.String
+	if !str.ReadUint16LengthPrefixed(&chainBytes) {
+		return nil, fmt.Errorf("error reading certificate_chain")
+	}
+	entry.chain = make([][32]byte, 0, len(chainBytes)/32)
+	for !chainBytes.Empty() {
+		var fingerprint [32]byte
+		if !chainBytes.CopyBytes(fingerprint[:]) {
+			return nil, fmt.Errorf("error reading fingerprint in certificate_chain")
+		}
+		entry.chain = append(entry.chain, fingerprint)
+	}
+
+	return str, nil
+}
+
+func (entry *StaticLogEntry) LeafInput() []byte {
+	return append([]byte{0, 0}, entry.timestampedEntry...)
+}
+
+func (entry *StaticLogEntry) ExtraData(ctx context.Context, issuerGetter IssuerGetter) ([]byte, error) {
+	b := cryptobyte.NewBuilder(nil)
+	if entry.precertificate != nil {
+		b.AddUint24LengthPrefixed(func(b *cryptobyte.Builder) {
+			b.AddBytes(entry.precertificate)
+		})
+	}
+	b.AddUint24LengthPrefixed(func(b *cryptobyte.Builder) {
+		for _, fingerprint := range entry.chain {
+			b.AddUint24LengthPrefixed(func(b *cryptobyte.Builder) {
+				cert, err := issuerGetter.GetIssuer(ctx, &fingerprint)
+				if err != nil {
+					panic(cryptobyte.BuildError{Err: fmt.Errorf("error getting issuer %x: %w", fingerprint, err)})
+				}
+				b.AddBytes(cert)
+			})
+		}
+	})
+	return b.Bytes()
+}
+
+func (entry *StaticLogEntry) Precertificate() (cttypes.ASN1Cert, error) {
+	if entry.precertificate == nil {
+		return nil, fmt.Errorf("not a precertificate entry")
+	}
+	return entry.precertificate, nil
+}
+
+func (entry *StaticLogEntry) ChainFingerprints() ([][32]byte, error) {
+	return entry.chain, nil
+}
+
+func (entry *StaticLogEntry) GetChain(ctx context.Context, issuerGetter IssuerGetter) (cttypes.ASN1CertChain, error) {
+	var (
+		chain = make(cttypes.ASN1CertChain, len(entry.chain))
+		errs  = make([]error, len(entry.chain))
+	)
+	var wg sync.WaitGroup
+	for i, fingerprint := range entry.chain {
+		wg.Add(1)
+		go func() {
+			defer wg.Done()
+			chain[i], errs[i] = issuerGetter.GetIssuer(ctx, &fingerprint)
+		}()
+	}
+	wg.Wait()
+	if err := errors.Join(errs...); err != nil {
+		return nil, err
+	}
+	return chain, nil
+}
+
+func formatTilePath(level string, tile uint64, partial uint64) string {
+	path := "tile/" + level + "/" + formatTileIndex(tile)
+	if partial != 0 {
+		path += fmt.Sprintf(".p/%d", partial)
+	}
+	return path
+}
+
+func formatTileIndex(tile uint64) string {
+	const base = 1000
+	str := fmt.Sprintf("%03d", tile%base)
+	for tile >= base {
+		tile = tile / base
+		str = fmt.Sprintf("x%03d/%s", tile%base, str)
+	}
+	return str
+}

ctclient/static_test.go

@@ -0,0 +1,38 @@
+// Copyright (C) 2025 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package ctclient
+
+import (
+	"testing"
+)
+
+func TestFormatTileIndex(t *testing.T) {
+	tests := []struct {
+		in  uint64
+		out string
+	}{
+		{0, "000"},
+		{1, "001"},
+		{12, "012"},
+		{105, "105"},
+		{1000, "x001/000"},
+		{1050, "x001/050"},
+		{52123, "x052/123"},
+		{999001, "x999/001"},
+		{1999001, "x001/x999/001"},
+		{15999001, "x015/x999/001"},
+	}
+	for i, test := range tests {
+		result := formatTileIndex(test.in)
+		if result != test.out {
+			t.Errorf("#%d: formatTileIndex(%q) = %q, want %q", i, test.in, result, test.out)
+		}
+	}
+}

ctcrypto/key.go

@@ -0,0 +1,67 @@
+// Copyright (C) 2025 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package ctcrypto
+
+import (
+	"bytes"
+	"crypto"
+	"crypto/ecdsa"
+	"crypto/rsa"
+	"crypto/x509"
+	"fmt"
+
+	"software.sslmate.com/src/certspotter/tlstypes"
+)
+
+type PublicKey []byte
+
+func (key PublicKey) Verify(input SignatureInput, signature tlstypes.DigitallySigned) error {
+	parsedKey, err := x509.ParsePKIXPublicKey(key)
+	if err != nil {
+		return fmt.Errorf("error parsing log key: %w", err)
+	}
+	switch key := parsedKey.(type) {
+	case *rsa.PublicKey:
+		if signature.Algorithm.Signature != tlstypes.RSA {
+			return fmt.Errorf("log key is RSA but this is not an RSA signature")
+		}
+		if signature.Algorithm.Hash != tlstypes.SHA256 {
+			return fmt.Errorf("unsupported hash algorithm %v (only SHA-256 is allowed in CT)", signature.Algorithm.Hash)
+		}
+		if rsa.VerifyPKCS1v15((*rsa.PublicKey)(key), crypto.SHA256, input[:], signature.Signature) != nil {
+			return fmt.Errorf("RSA signature is incorrect")
+		}
+		return nil
+
+	case *ecdsa.PublicKey:
+		if signature.Algorithm.Signature != tlstypes.ECDSA {
+			return fmt.Errorf("log key is ECDSA but this is not an ECDSA signature")
+		}
+		if signature.Algorithm.Hash != tlstypes.SHA256 {
+			return fmt.Errorf("unsupported hash algorithm %v (only SHA-256 is allowed in CT)", signature.Algorithm.Hash)
+		}
+		if !ecdsa.VerifyASN1((*ecdsa.PublicKey)(key), input[:], signature.Signature) {
+			return fmt.Errorf("ECDSA signature is incorrect")
+		}
+
+	default:
+		return fmt.Errorf("unsupported public key type %T (CT only allows RSA and ECDSA)", key)
+	}
+	return nil
+}
+
+func (key PublicKey) MarshalBinary() ([]byte, error) {
+	return bytes.Clone(key), nil
+}
+
+func (key *PublicKey) UnmarshalBinary(data []byte) error {
+	*key = bytes.Clone(data)
+	return nil
+}

ctcrypto/signatures.go

@@ -0,0 +1,55 @@
+// Copyright (C) 2025 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package ctcrypto
+
+import (
+	"crypto/sha256"
+	"golang.org/x/crypto/cryptobyte"
+
+	"software.sslmate.com/src/certspotter/cttypes"
+)
+
+type SignatureInput [32]byte
+
+func MakeSignatureInput(message []byte) SignatureInput {
+	return sha256.Sum256(message)
+}
+
+func SignatureInputForPrecertSCT(sct *cttypes.SignedCertificateTimestamp, precert cttypes.PreCert) SignatureInput {
+	var builder cryptobyte.Builder
+	builder.AddValue(sct.SCTVersion)
+	builder.AddValue(cttypes.CertificateTimestampSignatureType)
+	builder.AddUint64(sct.Timestamp)
+	builder.AddValue(cttypes.PrecertEntryType)
+	builder.AddValue(&precert)
+	builder.AddValue(sct.Extensions)
+	return MakeSignatureInput(builder.BytesOrPanic())
+}
+
+func SignatureInputForCertSCT(sct *cttypes.SignedCertificateTimestamp, cert cttypes.ASN1Cert) SignatureInput {
+	var builder cryptobyte.Builder
+	builder.AddValue(sct.SCTVersion)
+	builder.AddValue(cttypes.CertificateTimestampSignatureType)
+	builder.AddUint64(sct.Timestamp)
+	builder.AddValue(cttypes.X509EntryType)
+	builder.AddValue(cert)
+	builder.AddValue(sct.Extensions)
+	return MakeSignatureInput(builder.BytesOrPanic())
+}
+
+func SignatureInputForSTH(sth *cttypes.SignedTreeHead) SignatureInput {
+	var builder cryptobyte.Builder
+	builder.AddValue(cttypes.V1)
+	builder.AddValue(cttypes.TreeHashSignatureType)
+	builder.AddUint64(sth.Timestamp)
+	builder.AddUint64(sth.TreeSize)
+	builder.AddBytes(sth.RootHash[:])
+	return MakeSignatureInput(builder.BytesOrPanic())
+}

cttypes/certs.go

@@ -0,0 +1,121 @@
+// Copyright (C) 2025 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package cttypes
+
+import (
+	"fmt"
+	"golang.org/x/crypto/cryptobyte"
+)
+
+type TBSCertificate []byte
+
+type ASN1Cert []byte
+
+type ASN1CertChain []ASN1Cert
+
+// Corresponds to the PreCert structure in RFC 6962.  PreCert is a misnomer because this is really a TBSCertificate, not a precertificate.
+type PreCert struct {
+	IssuerKeyHash  [32]byte
+	TBSCertificate TBSCertificate
+}
+
+type PrecertChainEntry struct {
+	PreCertificate      ASN1Cert
+	PrecertificateChain ASN1CertChain
+}
+
+func (v *TBSCertificate) Unmarshal(s *cryptobyte.String) bool {
+	return s.ReadUint24LengthPrefixed((*cryptobyte.String)(v))
+}
+func (v TBSCertificate) Marshal(b *cryptobyte.Builder) error {
+	b.AddUint24LengthPrefixed(addBytesFunc(v))
+	return nil
+}
+
+func (v *ASN1Cert) Unmarshal(s *cryptobyte.String) bool {
+	return s.ReadUint24LengthPrefixed((*cryptobyte.String)(v))
+}
+func (v ASN1Cert) Marshal(b *cryptobyte.Builder) error {
+	b.AddUint24LengthPrefixed(addBytesFunc(v))
+	return nil
+}
+
+func (v *ASN1CertChain) Unmarshal(s *cryptobyte.String) bool {
+	chainBytes := new(cryptobyte.String)
+	if !s.ReadUint24LengthPrefixed(chainBytes) {
+		return false
+	}
+	*v = []ASN1Cert{}
+	for !chainBytes.Empty() {
+		var cert ASN1Cert
+		if !cert.Unmarshal(chainBytes) {
+			return false
+		}
+		*v = append(*v, cert)
+	}
+	return true
+}
+func (v ASN1CertChain) Marshal(b *cryptobyte.Builder) error {
+	b.AddUint24LengthPrefixed(func(b *cryptobyte.Builder) {
+		for _, cert := range v {
+			b.AddValue(cert)
+		}
+	})
+	return nil
+}
+
+func (precert *PreCert) Unmarshal(s *cryptobyte.String) error {
+	if !s.CopyBytes(precert.IssuerKeyHash[:]) {
+		return fmt.Errorf("error reading PreCert issuer_key_hash")
+	}
+	if !precert.TBSCertificate.Unmarshal(s) {
+		return fmt.Errorf("error reading PreCert tbs_certificate")
+	}
+	return nil
+}
+func (v *PreCert) Marshal(b *cryptobyte.Builder) error {
+	b.AddBytes(v.IssuerKeyHash[:])
+	b.AddValue(v.TBSCertificate)
+	return nil
+}
+
+func (entry *PrecertChainEntry) Unmarshal(s *cryptobyte.String) error {
+	if !entry.PreCertificate.Unmarshal(s) {
+		return fmt.Errorf("error reading PrecertChainEntry pre_certificate")
+	}
+	if !entry.PrecertificateChain.Unmarshal(s) {
+		return fmt.Errorf("error reading PrecertChainEntry preeertificate_chain")
+	}
+	return nil
+}
+
+func ParseExtraDataForX509Entry(extraData []byte) (ASN1CertChain, error) {
+	str := cryptobyte.String(extraData)
+	var chain ASN1CertChain
+	if !chain.Unmarshal(&str) {
+		return nil, fmt.Errorf("error reading ASN.1Cert chain")
+	}
+	if !str.Empty() {
+		return nil, fmt.Errorf("trailing garbage after ASN.1Cert chain")
+	}
+	return chain, nil
+}
+
+func ParseExtraDataForPrecertEntry(extraData []byte) (*PrecertChainEntry, error) {
+	str := cryptobyte.String(extraData)
+	entry := new(PrecertChainEntry)
+	if err := entry.Unmarshal(&str); err != nil {
+		return nil, err
+	}
+	if !str.Empty() {
+		return nil, fmt.Errorf("trailing garbage after PrecertChainEntry")
+	}
+	return entry, nil
+}

cttypes/checkpoint.go

@@ -0,0 +1,112 @@
+// Copyright (C) 2025 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package cttypes
+
+import (
+	"bytes"
+	"crypto/sha256"
+	"encoding/base64"
+	"encoding/binary"
+	"errors"
+	"fmt"
+	"strconv"
+	"strings"
+
+	"software.sslmate.com/src/certspotter/merkletree"
+	"software.sslmate.com/src/certspotter/tlstypes"
+)
+
+func chompLine(input []byte) (string, []byte, bool) {
+	newline := bytes.IndexByte(input, '\n')
+	if newline == -1 {
+		return "", nil, false
+	}
+	return string(input[:newline]), input[newline+1:], true
+}
+
+func makeCheckpointKeyID(origin string, logID LogID) [4]byte {
+	h := sha256.New()
+	h.Write([]byte(origin))
+	h.Write([]byte{'\n', 0x05})
+	h.Write(logID[:])
+
+	var digest [sha256.Size]byte
+	h.Sum(digest[:0])
+	return [4]byte(digest[:4])
+}
+
+func ParseCheckpoint(input []byte, logID LogID) (*SignedTreeHead, error) {
+	// origin
+	origin, input, _ := chompLine(input)
+
+	// tree size
+	sizeLine, input, _ := chompLine(input)
+	treeSize, err := strconv.ParseUint(sizeLine, 10, 64)
+	if err != nil {
+		return nil, fmt.Errorf("malformed tree size: %w", err)
+	}
+
+	// root hash
+	hashLine, input, _ := chompLine(input)
+	rootHash, err := base64.StdEncoding.DecodeString(hashLine)
+	if err != nil {
+		return nil, fmt.Errorf("malformed root hash: %w", err)
+	}
+	if len(rootHash) != merkletree.HashLen {
+		return nil, fmt.Errorf("root hash has wrong length (should be %d bytes long, not %d)", merkletree.HashLen, len(rootHash))
+	}
+
+	// 0 or more non-empty extension lines (ignored)
+	for {
+		line, rest, ok := chompLine(input)
+		if !ok {
+			return nil, errors.New("signed note ended prematurely")
+		}
+		input = rest
+		if len(line) == 0 {
+			break
+		}
+	}
+
+	// signature lines
+	signaturePrefix := "\u2014 " + origin + " "
+	keyID := makeCheckpointKeyID(origin, logID)
+	for {
+		signatureLine, rest, ok := chompLine(input)
+		if !ok {
+			return nil, errors.New("signed note is missing signature from the log")
+		}
+		input = rest
+		if !strings.HasPrefix(signatureLine, signaturePrefix) {
+			continue
+		}
+		signatureBytes, err := base64.StdEncoding.DecodeString(strings.TrimPrefix(signatureLine, signaturePrefix))
+		if err != nil {
+			return nil, fmt.Errorf("malformed signature: %w", err)
+		}
+		if !bytes.HasPrefix(signatureBytes, keyID[:]) {
+			continue
+		}
+		if len(signatureBytes) < 12 {
+			return nil, errors.New("malformed signature: too short")
+		}
+		timestamp := binary.BigEndian.Uint64(signatureBytes[4:12])
+		signature, err := tlstypes.ParseDigitallySigned(signatureBytes[12:])
+		if err != nil {
+			return nil, fmt.Errorf("malformed signature: %w", err)
+		}
+		return &SignedTreeHead{
+			TreeSize:  treeSize,
+			Timestamp: timestamp,
+			RootHash:  (merkletree.Hash)(rootHash),
+			Signature: *signature,
+		}, nil
+	}
+}

cttypes/checkpoint_test.go

@@ -0,0 +1,89 @@
+// Copyright (C) 2025 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package cttypes
+
+import (
+	"bytes"
+	"software.sslmate.com/src/certspotter/tlstypes"
+	"testing"
+)
+
+func TestParseCheckpoint(t *testing.T) {
+	logID := LogID{0x49, 0x4d, 0x90, 0x49, 0xb8, 0xaf, 0x3e, 0x5a, 0xca, 0xba, 0x99, 0x3e, 0x4c, 0x1f, 0x30, 0x56, 0x73, 0x7a, 0xa9, 0xf9, 0x6d, 0x00, 0xf7, 0xb0, 0xb9, 0xb2, 0x51, 0x06, 0xf7, 0xbe, 0x1a, 0x8f}
+	expected := SignedTreeHead{
+		TreeSize:  820495916,
+		Timestamp: 1719511711300,
+		RootHash:  [...]byte{0xc0, 0x3a, 0x6b, 0xe9, 0xf1, 0x1d, 0xc9, 0xcc, 0x20, 0x89, 0xe4, 0x45, 0xa7, 0x3f, 0x61, 0x41, 0xee, 0x82, 0xe8, 0x0d, 0x2d, 0x83, 0xa2, 0xb6, 0x65, 0x53, 0xd4, 0x96, 0xd7, 0x1e, 0xd2, 0x12},
+		Signature: tlstypes.DigitallySigned{
+			Algorithm: tlstypes.SignatureAndHashAlgorithm{
+				Hash:      tlstypes.SHA256,
+				Signature: tlstypes.ECDSA,
+			},
+			Signature: []byte{0x30, 0x46, 0x02, 0x21, 0x00, 0xd0, 0x0d, 0x31, 0x91, 0x50, 0x80, 0x62, 0xfa, 0xb0, 0xf9, 0xf7, 0x63, 0x61, 0x78, 0x95, 0x2b, 0x9c, 0x19, 0x22, 0x3a, 0x1d, 0x08, 0xc5, 0x68, 0x0e, 0xd0, 0x8b, 0x3b, 0x79, 0x18, 0x88, 0x86, 0x02, 0x21, 0x00, 0xd5, 0x74, 0xae, 0x8c, 0x1d, 0x1d, 0x7a, 0x4e, 0x80, 0x6c, 0x36, 0x46, 0x81, 0xb4, 0x7c, 0x91, 0x78, 0xc0, 0x3f, 0xdc, 0xc0, 0xab, 0xa5, 0x90, 0x40, 0x8d, 0x0e, 0xf6, 0x2c, 0x83, 0xa9, 0x34},
+		},
+	}
+
+	sthStrings := []string{
+		"sycamore.ct.letsencrypt.org/2024h2\n820495916\nwDpr6fEdycwgieRFpz9hQe6C6A0tg6K2ZVPUltce0hI=\n\n\u2014 sycamore.ct.letsencrypt.org/2024h2 PdkIagAAAZBa4n5EBAMASDBGAiEA0A0xkVCAYvqw+fdjYXiVK5wZIjodCMVoDtCLO3kYiIYCIQDVdK6MHR16ToBsNkaBtHyReMA/3MCrpZBAjQ72LIOpNA==\n",
+		"sycamore.ct.letsencrypt.org/2024h2\n820495916\nwDpr6fEdycwgieRFpz9hQe6C6A0tg6K2ZVPUltce0hI=\nextension line\n\n\u2014 sycamore.ct.letsencrypt.org/2024h2 PdkIagAAAZBa4n5EBAMASDBGAiEA0A0xkVCAYvqw+fdjYXiVK5wZIjodCMVoDtCLO3kYiIYCIQDVdK6MHR16ToBsNkaBtHyReMA/3MCrpZBAjQ72LIOpNA==\n",
+		"sycamore.ct.letsencrypt.org/2024h2\n820495916\nwDpr6fEdycwgieRFpz9hQe6C6A0tg6K2ZVPUltce0hI=\nextension line\nanother extension line\n\n\u2014 sycamore.ct.letsencrypt.org/2024h2 PdkIagAAAZBa4n5EBAMASDBGAiEA0A0xkVCAYvqw+fdjYXiVK5wZIjodCMVoDtCLO3kYiIYCIQDVdK6MHR16ToBsNkaBtHyReMA/3MCrpZBAjQ72LIOpNA==\n",
+		"sycamore.ct.letsencrypt.org/2024h2\n820495916\nwDpr6fEdycwgieRFpz9hQe6C6A0tg6K2ZVPUltce0hI=\n\n\u2014 sycamore.ct.letsencrypt.org/2024h2 PdkIagAAAZBa4n5EBAMASDBGAiEA0A0xkVCAYvqw+fdjYXiVK5wZIjodCMVoDtCLO3kYiIYCIQDVdK6MHR16ToBsNkaBtHyReMA/3MCrpZBAjQ72LIOpNA==\n\u2014 someoneelse.example c2lnbmF0dXJlCg==\n",
+		"sycamore.ct.letsencrypt.org/2024h2\n820495916\nwDpr6fEdycwgieRFpz9hQe6C6A0tg6K2ZVPUltce0hI=\n\n\u2014 someoneelse.example c2lnbmF0dXJlCg==\n\u2014 sycamore.ct.letsencrypt.org/2024h2 PdkIagAAAZBa4n5EBAMASDBGAiEA0A0xkVCAYvqw+fdjYXiVK5wZIjodCMVoDtCLO3kYiIYCIQDVdK6MHR16ToBsNkaBtHyReMA/3MCrpZBAjQ72LIOpNA==\n",
+		"sycamore.ct.letsencrypt.org/2024h2\n820495916\nwDpr6fEdycwgieRFpz9hQe6C6A0tg6K2ZVPUltce0hI=\nextension line\nanother extension line\n\n\u2014 sycamore.ct.letsencrypt.org/2024h2c2lnbmF0dXJlCg==\n\u2014 sycamore.ct.letsencrypt.org/2024h2 PdkIagAAAZBa4n5EBAMASDBGAiEA0A0xkVCAYvqw+fdjYXiVK5wZIjodCMVoDtCLO3kYiIYCIQDVdK6MHR16ToBsNkaBtHyReMA/3MCrpZBAjQ72LIOpNA==\n",
+	}
+
+	for i, str := range sthStrings {
+		parsed, err := ParseCheckpoint([]byte(str), logID)
+		if err != nil {
+			t.Errorf("%d: Unexpected error: %s", i, err)
+			return
+		}
+		if parsed.TreeSize != expected.TreeSize {
+			t.Errorf("%d: wrong tree size", i)
+			return
+		}
+		if parsed.Timestamp != expected.Timestamp {
+			t.Errorf("%d: wrong timestamp", i)
+			return
+		}
+		if !bytes.Equal(parsed.RootHash[:], expected.RootHash[:]) {
+			t.Errorf("%d: wrong root hash", i)
+			return
+		}
+		if !(parsed.Signature.Algorithm == expected.Signature.Algorithm && bytes.Equal(parsed.Signature.Signature, expected.Signature.Signature)) {
+			t.Errorf("%d: wrong signature", i)
+			return
+		}
+	}
+}
+
+func TestParseCheckpointFailure(t *testing.T) {
+	logID := LogID{0x49, 0x4d, 0x90, 0x49, 0xb8, 0xaf, 0x3e, 0x5a, 0xca, 0xba, 0x99, 0x3e, 0x4c, 0x1f, 0x30, 0x56, 0x73, 0x7a, 0xa9, 0xf9, 0x6d, 0x00, 0xf7, 0xb0, 0xb9, 0xb2, 0x51, 0x06, 0xf7, 0xbe, 0x1a, 0x8f}
+	sthStrings := []string{
+		"",
+		"\n",
+		"sycamore.ct.letsencrypt.org/2024h2\n820495916\nwDpr6fEdycwgieRFpz9hQe6C6A0tg6K2ZVPUltce0hI=\n",
+		"sycamore.ct.letsencrypt.org/2024h2\n820495916\nwDpr6fEdycwgieRFpz9hQe6C6A0tg6K2ZVPUltce0hI=\n\n",
+		"sycamore.ct.letsencrypt.org/2024h2\n820495916\nwDpr6fEdycwgieRFpz9hQe6C6A0tg6K2ZVPUltce0hI=\n\n\u2014 oak.ct.letsencrypt.org/2024h2 PdkIagAAAZBa4n5EBAMASDBGAiEA0A0xkVCAYvqw+fdjYXiVK5wZIjodCMVoDtCLO3kYiIYCIQDVdK6MHR16ToBsNkaBtHyReMA/3MCrpZBAjQ72LIOpNA==\n",
+		"sycamore.ct.letsencrypt.org/2024h2\n820495916\nwDpr6fEdycwgieRFpz9hQe6C6A0tg6K2ZVPUltce0hI=\n\n\u2014 sycamore.ct.letsencrypt.org/2024h2 bm90YXNpZ25hdHVyZQo=\n",
+		"sycamore.ct.letsencrypt.org/2024h2\n820495916\nwDpr6fEdycwgieRFpz9hQe6C6A0tg6K2ZVPUltce0hI=\n\n\u2014 sycamore.ct.letsencrypt.org/2024h2 notbase64!\n",
+		"sycamore.ct.letsencrypt.org/2024h2\n820495916!\nwDpr6fEdycwgieRFpz9hQe6C6A0tg6K2ZVPUltce0hI=\n\n\u2014 sycamore.ct.letsencrypt.org/2024h2 PdkIagAAAZBa4n5EBAMASDBGAiEA0A0xkVCAYvqw+fdjYXiVK5wZIjodCMVoDtCLO3kYiIYCIQDVdK6MHR16ToBsNkaBtHyReMA/3MCrpZBAjQ72LIOpNA==\n",
+		"sycamore.ct.letsencrypt.org/2024h2\n820495916\nwDpr6fEd!ycwgieRFpz9hQe6C6A0tg6K2ZVPUltce0hI=\n\n\u2014 sycamore.ct.letsencrypt.org/2024h2 PdkIagAAAZBa4n5EBAMASDBGAiEA0A0xkVCAYvqw+fdjYXiVK5wZIjodCMVoDtCLO3kYiIYCIQDVdK6MHR16ToBsNkaBtHyReMA/3MCrpZBAjQ72LIOpNA==\n",
+		"sycamore.ct.letsencrypt.org/2024h2\n820495916\nwDpr6fEdycwgieRFpz9hQe6C6A0tg6K2ZVPUltce0hI=\n\n- sycamore.ct.letsencrypt.org/2024h2 PdkIagAAAZBa4n5EBAMASDBGAiEA0A0xkVCAYvqw+fdjYXiVK5wZIjodCMVoDtCLO3kYiIYCIQDVdK6MHR16ToBsNkaBtHyReMA/3MCrpZBAjQ72LIOpNA==\n",
+	}
+
+	for i, str := range sthStrings {
+		_, err := ParseCheckpoint([]byte(str), logID)
+		if err == nil {
+			t.Errorf("%d: Unexpected success", i)
+			return
+		}
+	}
+}

cttypes/helpers.go

@@ -0,0 +1,20 @@
+// Copyright (C) 2025 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package cttypes
+
+import (
+	"golang.org/x/crypto/cryptobyte"
+)
+
+func addBytesFunc(v []byte) cryptobyte.BuilderContinuation {
+	return func(b *cryptobyte.Builder) {
+		b.AddBytes(v)
+	}
+}

cttypes/logid.go

@@ -0,0 +1,64 @@
+// Copyright (C) 2025 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package cttypes
+
+import (
+	"encoding/base64"
+	"fmt"
+	"golang.org/x/crypto/cryptobyte"
+)
+
+type LogID [32]byte
+
+func (v *LogID) Unmarshal(s *cryptobyte.String) bool {
+	return s.CopyBytes((*v)[:])
+}
+func (v LogID) Marshal(b *cryptobyte.Builder) error {
+	b.AddBytes(v[:])
+	return nil
+}
+
+func (id *LogID) UnmarshalBinary(bytes []byte) error {
+	if len(bytes) != len(*id) {
+		return fmt.Errorf("LogID has wrong length (should be %d, not %d)", len(*id), len(bytes))
+	}
+	*id = (LogID)(bytes)
+	return nil
+}
+func (id LogID) MarshalBinary() ([]byte, error) {
+	return id[:], nil
+}
+
+func (id *LogID) UnmarshalText(textData []byte) error {
+	if len(textData) != 44 {
+		return fmt.Errorf("LogID has wrong length (should be %d, not %d)", 44, len(textData))
+	}
+	var bytes [33]byte
+	if n, err := base64.StdEncoding.Decode(bytes[:], textData); err != nil {
+		return fmt.Errorf("LogID contains invalid base64: %w", err)
+	} else if n != 32 {
+		return fmt.Errorf("LogID has wrong length (should be %d bytes, not %d)", 32, n)
+	}
+	copy(id[:], bytes[:])
+	return nil
+}
+func (id LogID) MarshalText() ([]byte, error) {
+	encodedBytes := make([]byte, 44)
+	base64.StdEncoding.Encode(encodedBytes, id[:])
+	return encodedBytes, nil
+}
+
+func (id LogID) Base64String() string {
+	return base64.StdEncoding.EncodeToString(id[:])
+}
+
+func (id LogID) Base64URLString() string {
+	return base64.RawURLEncoding.EncodeToString(id[:])
+}

cttypes/merkleleaf.go

@@ -0,0 +1,213 @@
+// Copyright (C) 2025 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package cttypes
+
+import (
+	"fmt"
+	"golang.org/x/crypto/cryptobyte"
+	"software.sslmate.com/src/certspotter/merkletree"
+)
+
+type MerkleLeafType uint8
+
+const (
+	TimestampedEntryType MerkleLeafType = 0
+)
+
+type LogEntryType uint16
+
+const (
+	X509EntryType    LogEntryType = 0
+	PrecertEntryType LogEntryType = 1
+)
+
+type CTExtensions []byte
+
+type MerkleTreeLeaf struct {
+	Version          Version
+	LeafType         MerkleLeafType
+	TimestampedEntry *TimestampedEntry
+}
+
+type TimestampedEntry struct {
+	Timestamp           uint64
+	EntryType           LogEntryType
+	SignedEntryASN1Cert *ASN1Cert
+	SignedEntryPreCert  *PreCert
+	Extensions          CTExtensions
+}
+
+func (v *MerkleLeafType) Unmarshal(s *cryptobyte.String) bool {
+	return s.ReadUint8((*uint8)(v))
+}
+func (v MerkleLeafType) Marshal(b *cryptobyte.Builder) error {
+	b.AddUint8(uint8(v))
+	return nil
+}
+
+func (v *LogEntryType) Unmarshal(s *cryptobyte.String) bool {
+	return s.ReadUint16((*uint16)(v))
+}
+func (v LogEntryType) Marshal(b *cryptobyte.Builder) error {
+	b.AddUint16(uint16(v))
+	return nil
+}
+
+func (v *CTExtensions) Unmarshal(s *cryptobyte.String) bool {
+	return s.ReadUint16LengthPrefixed((*cryptobyte.String)(v))
+}
+func (v CTExtensions) Marshal(b *cryptobyte.Builder) error {
+	b.AddUint16LengthPrefixed(addBytesFunc(v))
+	return nil
+}
+
+func (leaf *MerkleTreeLeaf) Unmarshal(s *cryptobyte.String) error {
+	if !leaf.Version.Unmarshal(s) {
+		return fmt.Errorf("error reading MerkleTreeLeaf version")
+	}
+	if leaf.Version != V1 {
+		return fmt.Errorf("unsupported Version 0x%02x", leaf.Version)
+	}
+	if !leaf.LeafType.Unmarshal(s) {
+		return fmt.Errorf("error reading MerkleTreeLeaf leaf_type")
+	}
+	switch leaf.LeafType {
+	case TimestampedEntryType:
+		leaf.TimestampedEntry = new(TimestampedEntry)
+		if err := leaf.TimestampedEntry.Unmarshal(s); err != nil {
+			return err
+		}
+	default:
+		return fmt.Errorf("unrecognized MerkleLeafType 0x%02x", leaf.LeafType)
+	}
+	return nil
+}
+func (v *MerkleTreeLeaf) Marshal(b *cryptobyte.Builder) error {
+	b.AddValue(v.Version)
+	b.AddValue(v.LeafType)
+	switch v.LeafType {
+	case TimestampedEntryType:
+		b.AddValue(v.TimestampedEntry)
+	}
+	return nil
+}
+func (v *MerkleTreeLeaf) Bytes() ([]byte, error) {
+	var builder cryptobyte.Builder
+	builder.AddValue(v)
+	return builder.Bytes()
+}
+func (v *MerkleTreeLeaf) Hash() merkletree.Hash {
+	var builder cryptobyte.Builder
+	builder.AddValue(v)
+	return merkletree.HashLeaf(builder.BytesOrPanic())
+}
+
+func (entry *TimestampedEntry) Unmarshal(s *cryptobyte.String) error {
+	if !s.ReadUint64(&entry.Timestamp) {
+		return fmt.Errorf("error reading TimestampedEntry timestamp")
+	}
+	if !entry.EntryType.Unmarshal(s) {
+		return fmt.Errorf("error reading TimestampedEntry entry_type")
+	}
+	switch entry.EntryType {
+	case X509EntryType:
+		entry.SignedEntryASN1Cert = new(ASN1Cert)
+		if !entry.SignedEntryASN1Cert.Unmarshal(s) {
+			return fmt.Errorf("error reading TimestampedEntry signed_entry ASN.1Cert")
+		}
+	case PrecertEntryType:
+		entry.SignedEntryPreCert = new(PreCert)
+		if err := entry.SignedEntryPreCert.Unmarshal(s); err != nil {
+			return fmt.Errorf("error reading TimestampedEntryType signed_entry: %w", err)
+		}
+	default:
+		return fmt.Errorf("unrecognized TimestampedEntryType 0x%02x", entry.EntryType)
+	}
+	if !entry.Extensions.Unmarshal(s) {
+		return fmt.Errorf("error reading TimestampedEntry extensions")
+	}
+	return nil
+}
+func (v *TimestampedEntry) Marshal(b *cryptobyte.Builder) error {
+	b.AddUint64(v.Timestamp)
+	b.AddValue(v.EntryType)
+	switch v.EntryType {
+	case X509EntryType:
+		b.AddValue(v.SignedEntryASN1Cert)
+	case PrecertEntryType:
+		b.AddValue(v.SignedEntryPreCert)
+	}
+	b.AddValue(v.Extensions)
+	return nil
+}
+
+func ParseLeafInput(leafInput []byte) (*MerkleTreeLeaf, error) {
+	str := cryptobyte.String(leafInput)
+	leaf := new(MerkleTreeLeaf)
+	if err := leaf.Unmarshal(&str); err != nil {
+		return nil, err
+	}
+	if !str.Empty() {
+		return nil, fmt.Errorf("trailing garbage after MerkleTreeLeaf")
+	}
+	return leaf, nil
+}
+
+func MerkleTreeLeafForCert(timestamp uint64, extensions []byte, cert ASN1Cert) *MerkleTreeLeaf {
+	return &MerkleTreeLeaf{
+		Version:  V1,
+		LeafType: TimestampedEntryType,
+		TimestampedEntry: &TimestampedEntry{
+			Timestamp:           timestamp,
+			EntryType:           X509EntryType,
+			SignedEntryASN1Cert: &cert,
+			Extensions:          extensions,
+		},
+	}
+}
+
+func MerkleTreeLeafForCertSCT(sct *SignedCertificateTimestamp, cert ASN1Cert) *MerkleTreeLeaf {
+	return &MerkleTreeLeaf{
+		Version:  sct.SCTVersion,
+		LeafType: TimestampedEntryType,
+		TimestampedEntry: &TimestampedEntry{
+			Timestamp:           sct.Timestamp,
+			EntryType:           X509EntryType,
+			SignedEntryASN1Cert: &cert,
+			Extensions:          sct.Extensions,
+		},
+	}
+}
+
+func MerkleTreeLeafForPrecert(timestamp uint64, extensions []byte, precert PreCert) *MerkleTreeLeaf {
+	return &MerkleTreeLeaf{
+		Version:  V1,
+		LeafType: TimestampedEntryType,
+		TimestampedEntry: &TimestampedEntry{
+			Timestamp:          timestamp,
+			EntryType:          PrecertEntryType,
+			SignedEntryPreCert: &precert,
+			Extensions:         extensions,
+		},
+	}
+}
+
+func MerkleTreeLeafForPrecertSCT(sct *SignedCertificateTimestamp, precert PreCert) *MerkleTreeLeaf {
+	return &MerkleTreeLeaf{
+		Version:  sct.SCTVersion,
+		LeafType: TimestampedEntryType,
+		TimestampedEntry: &TimestampedEntry{
+			Timestamp:          sct.Timestamp,
+			EntryType:          PrecertEntryType,
+			SignedEntryPreCert: &precert,
+			Extensions:         sct.Extensions,
+		},
+	}
+}

cttypes/sct.go

@@ -0,0 +1,58 @@
+// Copyright (C) 2025 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package cttypes
+
+import (
+	"fmt"
+	"golang.org/x/crypto/cryptobyte"
+	"software.sslmate.com/src/certspotter/tlstypes"
+)
+
+type SignedCertificateTimestamp struct {
+	SCTVersion Version                  `json:"sct_version"`
+	ID         LogID                    `json:"id"`
+	Timestamp  uint64                   `json:"timestamp"`
+	Extensions CTExtensions             `json:"extensions"`
+	Signature  tlstypes.DigitallySigned `json:"signature"`
+}
+
+func (sct *SignedCertificateTimestamp) Unmarshal(s *cryptobyte.String) error {
+	if !sct.SCTVersion.Unmarshal(s) {
+		return fmt.Errorf("error reading SCT version")
+	}
+	if sct.SCTVersion != V1 {
+		return fmt.Errorf("unsupported SCT version 0x%02x", sct.SCTVersion)
+	}
+	if !sct.ID.Unmarshal(s) {
+		return fmt.Errorf("error reading SCT id")
+	}
+	if !s.ReadUint64(&sct.Timestamp) {
+		return fmt.Errorf("error reading SCT timestamp")
+	}
+	if !sct.Extensions.Unmarshal(s) {
+		return fmt.Errorf("error reading SCT extensions")
+	}
+	if !sct.Signature.Unmarshal(s) {
+		return fmt.Errorf("error reading SCT signature")
+	}
+	return nil
+}
+
+func ParseSignedCertificateTimestamp(data []byte) (*SignedCertificateTimestamp, error) {
+	str := cryptobyte.String(data)
+	sct := new(SignedCertificateTimestamp)
+	if err := sct.Unmarshal(&str); err != nil {
+		return nil, err
+	}
+	if !str.Empty() {
+		return nil, fmt.Errorf("trailing garbage after SignedCertificateTimestamp")
+	}
+	return sct, nil
+}

cttypes/signatures.go

@@ -0,0 +1,29 @@
+// Copyright (C) 2025 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package cttypes
+
+import (
+	"golang.org/x/crypto/cryptobyte"
+)
+
+type SignatureType uint8
+
+const (
+	CertificateTimestampSignatureType SignatureType = 0
+	TreeHashSignatureType             SignatureType = 1
+)
+
+func (v *SignatureType) Unmarshal(s *cryptobyte.String) bool {
+	return s.ReadUint8((*uint8)(v))
+}
+func (v SignatureType) Marshal(b *cryptobyte.Builder) error {
+	b.AddUint8(uint8(v))
+	return nil
+}

cttypes/sth.go

@@ -0,0 +1,33 @@
+// Copyright (C) 2025 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package cttypes
+
+import (
+	"software.sslmate.com/src/certspotter/merkletree"
+	"software.sslmate.com/src/certspotter/tlstypes"
+	"time"
+)
+
+type SignedTreeHead struct {
+	TreeSize  uint64                   `json:"tree_size"`
+	Timestamp uint64                   `json:"timestamp"`
+	RootHash  merkletree.Hash          `json:"sha256_root_hash"`
+	Signature tlstypes.DigitallySigned `json:"tree_head_signature"`
+}
+
+type GossipedSignedTreeHead struct {
+	SignedTreeHead
+	STHVersion Version `json:"sth_version"`
+	LogID      LogID   `json:"log_id"`
+}
+
+func (sth *SignedTreeHead) TimestampTime() time.Time {
+	return time.UnixMilli(int64(sth.Timestamp))
+}

cttypes/version.go

@@ -0,0 +1,29 @@
+// Copyright (C) 2025 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package cttypes
+
+import (
+	"golang.org/x/crypto/cryptobyte"
+)
+
+type Version uint8
+
+const (
+	V1 Version = 0
+)
+
+func (v Version) Marshal(b *cryptobyte.Builder) error {
+	b.AddUint8(uint8(v))
+	return nil
+}
+
+func (v *Version) Unmarshal(s *cryptobyte.String) bool {
+	return s.ReadUint8((*uint8)(v))
+}

go.mod

@@ -0,0 +1,13 @@
+module software.sslmate.com/src/certspotter
+
+go 1.24
+
+require (
+	golang.org/x/crypto v0.37.0
+	golang.org/x/net v0.39.0
+	golang.org/x/sync v0.13.0
+)
+
+require golang.org/x/text v0.24.0 // indirect
+
+retract v0.19.0 // Contains serious bugs.

go.sum

@@ -0,0 +1,8 @@
+golang.org/x/crypto v0.37.0 h1:kJNSjF/Xp7kU0iB2Z+9viTPMW4EqqsrywMXLJOOsXSE=
+golang.org/x/crypto v0.37.0/go.mod h1:vg+k43peMZ0pUMhYmVAWysMK35e6ioLh3wB8ZCAfbVc=
+golang.org/x/net v0.39.0 h1:ZCu7HMWDxpXpaiKdhzIfaltL9Lp31x/3fCP11bc6/fY=
+golang.org/x/net v0.39.0/go.mod h1:X7NRbYVEA+ewNkCNyJ513WmMdQ3BineSwVtN2zD/d+E=
+golang.org/x/sync v0.13.0 h1:AauUjRAJ9OSnvULf/ARrrVywoJDy0YS2AwQ98I37610=
+golang.org/x/sync v0.13.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA=
+golang.org/x/text v0.24.0 h1:dd5Bzh4yt5KYA8f9CJHCP4FB4D51c2c6JvN37xJJkJ0=
+golang.org/x/text v0.24.0/go.mod h1:L8rBsPeo2pSS+xqN0d5u2ikmjtmoJbDBT1b7nHvFCdU=

helpers.go

@@ -10,104 +10,9 @@
 package certspotter
 
 import (
-	"bytes"
-	"crypto/sha256"
-	"encoding/hex"
-	"encoding/json"
-	"fmt"
-	"io"
-	"io/ioutil"
 	"math/big"
-	"os"
-	"os/exec"
-	"strconv"
-	"strings"
-	"time"
-
-	"software.sslmate.com/src/certspotter/ct"
 )
 
-func ReadSTHFile(path string) (*ct.SignedTreeHead, error) {
-	content, err := ioutil.ReadFile(path)
-	if err != nil {
-		if os.IsNotExist(err) {
-			return nil, nil
-		}
-		return nil, err
-	}
-
-	var sth ct.SignedTreeHead
-	if err := json.Unmarshal(content, &sth); err != nil {
-		return nil, err
-	}
-
-	return &sth, nil
-}
-
-func WriteSTHFile(path string, sth *ct.SignedTreeHead) error {
-	sthJson, err := json.MarshalIndent(sth, "", "\t")
-	if err != nil {
-		return err
-	}
-	sthJson = append(sthJson, byte('\n'))
-	return ioutil.WriteFile(path, sthJson, 0666)
-}
-
-func WriteProofFile(path string, proof ct.ConsistencyProof) error {
-	proofJson, err := json.MarshalIndent(proof, "", "\t")
-	if err != nil {
-		return err
-	}
-	proofJson = append(proofJson, byte('\n'))
-	return ioutil.WriteFile(path, proofJson, 0666)
-}
-
-func IsPrecert(entry *ct.LogEntry) bool {
-	return entry.Leaf.TimestampedEntry.EntryType == ct.PrecertLogEntryType
-}
-
-func GetFullChain(entry *ct.LogEntry) [][]byte {
-	certs := make([][]byte, 0, len(entry.Chain)+1)
-
-	if entry.Leaf.TimestampedEntry.EntryType == ct.X509LogEntryType {
-		certs = append(certs, entry.Leaf.TimestampedEntry.X509Entry)
-	}
-	for _, cert := range entry.Chain {
-		certs = append(certs, cert)
-	}
-
-	return certs
-}
-
-func formatSerialNumber(serial *big.Int) string {
-	if serial != nil {
-		return fmt.Sprintf("%x", serial)
-	} else {
-		return ""
-	}
-}
-
-func sha256sum(data []byte) []byte {
-	sum := sha256.Sum256(data)
-	return sum[:]
-}
-
-func sha256hex(data []byte) string {
-	return hex.EncodeToString(sha256sum(data))
-}
-
-type EntryInfo struct {
-	LogUri                string
-	Entry                 *ct.LogEntry
-	IsPrecert             bool
-	FullChain             [][]byte // first entry is logged X509 cert or pre-cert
-	CertInfo              *CertInfo
-	ParseError            error // set iff CertInfo is nil
-	Identifiers           *Identifiers
-	IdentifiersParseError error
-	Filename              string
-}
-
 type CertInfo struct {
 	TBS *TBSCertificate
 
@@ -123,6 +28,7 @@ type CertInfo struct {
 	ValidityParseError     error
 	IsCA                   *bool
 	IsCAParseError         error
+	IsPreCert              bool
 }
 
 func MakeCertInfoFromTBS(tbs *TBSCertificate) *CertInfo {
@@ -134,6 +40,7 @@ func MakeCertInfoFromTBS(tbs *TBSCertificate) *CertInfo {
 	info.SerialNumber, info.SerialNumberParseError = tbs.ParseSerialNumber()
 	info.Validity, info.ValidityParseError = tbs.ParseValidity()
 	info.IsCA, info.IsCAParseError = tbs.ParseBasicConstraints()
+	info.IsPreCert = len(tbs.GetExtension(oidExtensionCTPoison)) > 0
 
 	return info
 }
@@ -154,212 +61,6 @@ func MakeCertInfoFromRawCert(certBytes []byte) (*CertInfo, error) {
 	return MakeCertInfoFromRawTBS(cert.GetRawTBSCertificate())
 }
 
-func MakeCertInfoFromLogEntry(entry *ct.LogEntry) (*CertInfo, error) {
-	switch entry.Leaf.TimestampedEntry.EntryType {
-	case ct.X509LogEntryType:
-		return MakeCertInfoFromRawCert(entry.Leaf.TimestampedEntry.X509Entry)
-
-	case ct.PrecertLogEntryType:
-		return MakeCertInfoFromRawTBS(entry.Leaf.TimestampedEntry.PrecertEntry.TBSCertificate)
-
-	default:
-		return nil, fmt.Errorf("MakeCertInfoFromCTEntry: unknown CT entry type (neither X509 nor precert)")
-	}
-}
-
-func (info *CertInfo) NotBefore() *time.Time {
-	if info.ValidityParseError == nil {
-		return &info.Validity.NotBefore
-	} else {
-		return nil
-	}
-}
-
-func (info *CertInfo) NotAfter() *time.Time {
-	if info.ValidityParseError == nil {
-		return &info.Validity.NotAfter
-	} else {
-		return nil
-	}
-}
-
-func (info *CertInfo) PubkeyHash() string {
-	return sha256hex(info.TBS.GetRawPublicKey())
-}
-
-func (info *CertInfo) PubkeyHashBytes() []byte {
-	return sha256sum(info.TBS.GetRawPublicKey())
-}
-
-func (info *CertInfo) Environ() []string {
-	env := make([]string, 0, 10)
-
-	env = append(env, "PUBKEY_HASH="+info.PubkeyHash())
-
-	if info.SerialNumberParseError != nil {
-		env = append(env, "SERIAL_PARSE_ERROR="+info.SerialNumberParseError.Error())
-	} else {
-		env = append(env, "SERIAL="+formatSerialNumber(info.SerialNumber))
-	}
-
-	if info.ValidityParseError != nil {
-		env = append(env, "VALIDITY_PARSE_ERROR="+info.ValidityParseError.Error())
-	} else {
-		env = append(env, "NOT_BEFORE="+info.Validity.NotBefore.String())
-		env = append(env, "NOT_BEFORE_UNIXTIME="+strconv.FormatInt(info.Validity.NotBefore.Unix(), 10))
-		env = append(env, "NOT_AFTER="+info.Validity.NotAfter.String())
-		env = append(env, "NOT_AFTER_UNIXTIME="+strconv.FormatInt(info.Validity.NotAfter.Unix(), 10))
-	}
-
-	if info.SubjectParseError != nil {
-		env = append(env, "SUBJECT_PARSE_ERROR="+info.SubjectParseError.Error())
-	} else {
-		env = append(env, "SUBJECT_DN="+info.Subject.String())
-	}
-
-	if info.IssuerParseError != nil {
-		env = append(env, "ISSUER_PARSE_ERROR="+info.IssuerParseError.Error())
-	} else {
-		env = append(env, "ISSUER_DN="+info.Issuer.String())
-	}
-
-	// TODO: include SANs in environment
-
-	return env
-}
-
-func (info *EntryInfo) HasParseErrors() bool {
-	return info.ParseError != nil ||
-		info.IdentifiersParseError != nil ||
-		info.CertInfo.SubjectParseError != nil ||
-		info.CertInfo.IssuerParseError != nil ||
-		info.CertInfo.SANsParseError != nil ||
-		info.CertInfo.SerialNumberParseError != nil ||
-		info.CertInfo.ValidityParseError != nil ||
-		info.CertInfo.IsCAParseError != nil
-}
-
-func (info *EntryInfo) Fingerprint() string {
-	if len(info.FullChain) > 0 {
-		return sha256hex(info.FullChain[0])
-	} else {
-		return ""
-	}
-}
-
-func (info *EntryInfo) FingerprintBytes() []byte {
-	if len(info.FullChain) > 0 {
-		return sha256sum(info.FullChain[0])
-	} else {
-		return []byte{}
-	}
-}
-
-func (info *EntryInfo) typeString() string {
-	if info.IsPrecert {
-		return "precert"
-	} else {
-		return "cert"
-	}
-}
-
-func (info *EntryInfo) typeFriendlyString() string {
-	if info.IsPrecert {
-		return "Pre-certificate"
-	} else {
-		return "Certificate"
-	}
-}
-
-func yesnoString(value bool) string {
-	if value {
-		return "yes"
-	} else {
-		return "no"
-	}
-}
-
-func (info *EntryInfo) Environ() []string {
-	env := []string{
-		"FINGERPRINT=" + info.Fingerprint(),
-		"CERT_TYPE=" + info.typeString(),
-		"CERT_PARSEABLE=" + yesnoString(info.ParseError == nil),
-		"LOG_URI=" + info.LogUri,
-		"ENTRY_INDEX=" + strconv.FormatInt(info.Entry.Index, 10),
-	}
-
-	if info.Filename != "" {
-		env = append(env, "CERT_FILENAME="+info.Filename)
-	}
-	if info.ParseError != nil {
-		env = append(env, "PARSE_ERROR="+info.ParseError.Error())
-	} else if info.CertInfo != nil {
-		certEnv := info.CertInfo.Environ()
-		env = append(env, certEnv...)
-	}
-	if info.IdentifiersParseError != nil {
-		env = append(env, "IDENTIFIERS_PARSE_ERROR="+info.IdentifiersParseError.Error())
-	} else if info.Identifiers != nil {
-		env = append(env, "DNS_NAMES="+info.Identifiers.dnsNamesString(","))
-		env = append(env, "IP_ADDRESSES="+info.Identifiers.ipAddrsString(","))
-	}
-
-	return env
-}
-
-func writeField(out io.Writer, name string, value interface{}, err error) {
-	if err == nil {
-		fmt.Fprintf(out, "\t%13s = %s\n", name, value)
-	} else {
-		fmt.Fprintf(out, "\t%13s = *** UNKNOWN (%s) ***\n", name, err)
-	}
-}
-
-func (info *EntryInfo) Write(out io.Writer) {
-	fingerprint := info.Fingerprint()
-	fmt.Fprintf(out, "%s:\n", fingerprint)
-	if info.IdentifiersParseError != nil {
-		writeField(out, "Identifiers", nil, info.IdentifiersParseError)
-	} else if info.Identifiers != nil {
-		for _, dnsName := range info.Identifiers.DNSNames {
-			writeField(out, "DNS Name", dnsName, nil)
-		}
-		for _, ipaddr := range info.Identifiers.IPAddrs {
-			writeField(out, "IP Address", ipaddr, nil)
-		}
-	}
-	if info.ParseError != nil {
-		writeField(out, "Parse Error", "*** "+info.ParseError.Error()+" ***", nil)
-	} else if info.CertInfo != nil {
-		writeField(out, "Pubkey", info.CertInfo.PubkeyHash(), nil)
-		writeField(out, "Issuer", info.CertInfo.Issuer, info.CertInfo.IssuerParseError)
-		writeField(out, "Not Before", info.CertInfo.NotBefore(), info.CertInfo.ValidityParseError)
-		writeField(out, "Not After", info.CertInfo.NotAfter(), info.CertInfo.ValidityParseError)
-	}
-	writeField(out, "Log Entry", fmt.Sprintf("%d @ %s (%s)", info.Entry.Index, info.LogUri, info.typeFriendlyString()), nil)
-	writeField(out, "crt.sh", "https://crt.sh/?sha256="+fingerprint, nil)
-	if info.Filename != "" {
-		writeField(out, "Filename", info.Filename, nil)
-	}
-}
-
-func (info *EntryInfo) InvokeHookScript(command string) error {
-	cmd := exec.Command(command)
-	cmd.Env = os.Environ()
-	infoEnv := info.Environ()
-	cmd.Env = append(cmd.Env, infoEnv...)
-	stderrBuffer := bytes.Buffer{}
-	cmd.Stderr = &stderrBuffer
-	if err := cmd.Run(); err != nil {
-		if _, isExitError := err.(*exec.ExitError); isExitError {
-			return fmt.Errorf("Script failed: %s: %s", command, strings.TrimSpace(stderrBuffer.String()))
-		} else {
-			return fmt.Errorf("Failed to execute script: %s: %s", command, err)
-		}
-	}
-	return nil
-}
-
 func MatchesWildcard(dnsName string, pattern string) bool {
 	for len(pattern) > 0 {
 		if pattern[0] == '*' {

loglist/helpers.go

@@ -0,0 +1,40 @@
+// Copyright (C) 2020 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package loglist
+
+import (
+	"time"
+)
+
+// Return all tiled and non-tiled logs from all operators
+func (list *List) AllLogs() []*Log {
+	logs := []*Log{}
+	for operator := range list.Operators {
+		for log := range list.Operators[operator].Logs {
+			logs = append(logs, &list.Operators[operator].Logs[log])
+		}
+		for log := range list.Operators[operator].TiledLogs {
+			logs = append(logs, &list.Operators[operator].TiledLogs[log])
+		}
+	}
+	return logs
+}
+
+func (log *Log) LogIDString() string {
+	return log.LogID.Base64String()
+}
+
+func (log *Log) AcceptsExpiration(expiration time.Time) bool {
+	return log.TemporalInterval == nil || withinInterval(expiration, log.TemporalInterval.StartInclusive, log.TemporalInterval.EndExclusive)
+}
+
+func withinInterval(expiration, startInclusive, endExclusive time.Time) bool {
+	return !expiration.Before(startInclusive) && expiration.Before(endExclusive)
+}

loglist/load.go

@@ -0,0 +1,118 @@
+// Copyright (C) 2020, 2023 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package loglist
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"strings"
+	"time"
+)
+
+var UserAgent = "certspotter"
+
+type ModificationToken struct {
+	etag     string
+	modified time.Time
+}
+
+var ErrNotModified = errors.New("loglist has not been modified")
+
+func newModificationToken(response *http.Response) *ModificationToken {
+	token := &ModificationToken{
+		etag: response.Header.Get("ETag"),
+	}
+	if t, err := time.Parse(http.TimeFormat, response.Header.Get("Last-Modified")); err == nil {
+		token.modified = t
+	}
+	return token
+}
+
+func (token *ModificationToken) setRequestHeaders(request *http.Request) {
+	if token.etag != "" {
+		request.Header.Set("If-None-Match", token.etag)
+	} else if !token.modified.IsZero() {
+		request.Header.Set("If-Modified-Since", token.modified.Format(http.TimeFormat))
+	}
+}
+
+func Load(ctx context.Context, urlOrFile string) (*List, error) {
+	list, _, err := LoadIfModified(ctx, urlOrFile, nil)
+	return list, err
+}
+
+func LoadIfModified(ctx context.Context, urlOrFile string, token *ModificationToken) (*List, *ModificationToken, error) {
+	if strings.HasPrefix(urlOrFile, "https://") {
+		return FetchIfModified(ctx, urlOrFile, token)
+	} else {
+		list, err := ReadFile(urlOrFile)
+		return list, nil, err
+	}
+}
+
+func Fetch(ctx context.Context, url string) (*List, error) {
+	list, _, err := FetchIfModified(ctx, url, nil)
+	return list, err
+}
+
+func FetchIfModified(ctx context.Context, url string, token *ModificationToken) (*List, *ModificationToken, error) {
+	request, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
+	if err != nil {
+		return nil, nil, err
+	}
+	request.Header.Set("User-Agent", UserAgent)
+	if token != nil {
+		token.setRequestHeaders(request)
+	}
+	response, err := http.DefaultClient.Do(request)
+	if err != nil {
+		return nil, nil, err
+	}
+	content, err := io.ReadAll(response.Body)
+	response.Body.Close()
+	if err != nil {
+		return nil, nil, err
+	}
+	if token != nil && response.StatusCode == http.StatusNotModified {
+		return nil, nil, ErrNotModified
+	}
+	if response.StatusCode != 200 {
+		return nil, nil, fmt.Errorf("%s: %s", url, response.Status)
+	}
+	list, err := Unmarshal(content)
+	if err != nil {
+		return nil, nil, fmt.Errorf("error parsing %s: %w", url, err)
+	}
+	return list, newModificationToken(response), err
+}
+
+func ReadFile(filename string) (*List, error) {
+	content, err := os.ReadFile(filename)
+	if err != nil {
+		return nil, err
+	}
+	return Unmarshal(content)
+}
+
+func Unmarshal(jsonBytes []byte) (*List, error) {
+	list := new(List)
+	if err := json.Unmarshal(jsonBytes, list); err != nil {
+		return nil, err
+	}
+	if err := list.Validate(); err != nil {
+		return nil, fmt.Errorf("Invalid log list: %s", err)
+	}
+	return list, nil
+}

loglist/schema.go

@@ -0,0 +1,120 @@
+// Copyright (C) 2020 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package loglist
+
+import (
+	"time"
+
+	"software.sslmate.com/src/certspotter/cttypes"
+)
+
+type List struct {
+	Version          string     `json:"version"`
+	LogListTimestamp time.Time  `json:"log_list_timestamp"` // Only present in v3 of schema
+	Operators        []Operator `json:"operators"`
+}
+
+type Operator struct {
+	Name      string   `json:"name"`
+	Email     []string `json:"email"`
+	Logs      []Log    `json:"logs"`
+	TiledLogs []Log    `json:"tiled_logs"`
+}
+
+type Log struct {
+	Key              []byte        `json:"key"`
+	LogID            cttypes.LogID `json:"log_id"`
+	MMD              int           `json:"mmd"`
+	URL              string        `json:"url,omitempty"`            // only for rfc6962 logs
+	SubmissionURL    string        `json:"submission_url,omitempty"` // only for static-ct-api logs
+	MonitoringURL    string        `json:"monitoring_url,omitempty"` // only for static-ct-api logs
+	Description      string        `json:"description"`
+	State            State         `json:"state"`
+	DNS              string        `json:"dns"`
+	LogType          LogType       `json:"log_type"`
+	TemporalInterval *struct {
+		StartInclusive time.Time `json:"start_inclusive"`
+		EndExclusive   time.Time `json:"end_exclusive"`
+	} `json:"temporal_interval"`
+
+	// certspotter-specific extensions
+	CertspotterDownloadSize int `json:"certspotter_download_size,omitempty"`
+	CertspotterDownloadJobs int `json:"certspotter_download_jobs,omitempty"`
+
+	// TODO: add previous_operators
+}
+
+func (log *Log) IsRFC6962() bool     { return log.URL != "" }
+func (log *Log) IsStaticCTAPI() bool { return log.SubmissionURL != "" && log.MonitoringURL != "" }
+
+// Return URL prefix for submission using the RFC6962 protocol
+func (log *Log) GetSubmissionURL() string {
+	if log.SubmissionURL != "" {
+		return log.SubmissionURL
+	} else {
+		return log.URL
+	}
+}
+
+// Return URL prefix for monitoring.
+// Since the protocol understood by the URL might be either RFC6962 or static-ct-api, this URL is
+// only useful for informational purposes.
+func (log *Log) GetMonitoringURL() string {
+	if log.MonitoringURL != "" {
+		return log.MonitoringURL
+	} else {
+		return log.URL
+	}
+}
+
+type State struct {
+	Pending *struct {
+		Timestamp time.Time `json:"timestamp"`
+	} `json:"pending"`
+
+	Qualified *struct {
+		Timestamp time.Time `json:"timestamp"`
+	} `json:"qualified"`
+
+	Usable *struct {
+		Timestamp time.Time `json:"timestamp"`
+	} `json:"usable"`
+
+	Readonly *struct {
+		Timestamp     time.Time `json:"timestamp"`
+		FinalTreeHead struct {
+			TreeSize       int64  `json:"tree_size"`
+			SHA256RootHash []byte `json:"sha256_root_hash"`
+		} `json:"final_tree_head"`
+	} `json:"readonly"`
+
+	Retired *struct {
+		Timestamp time.Time `json:"timestamp"`
+	} `json:"retired"`
+
+	Rejected *struct {
+		Timestamp time.Time `json:"timestamp"`
+	} `json:"rejected"`
+}
+
+func (state *State) IsApproved() bool {
+	return state.Qualified != nil || state.Usable != nil || state.Readonly != nil
+}
+
+func (state *State) WasApprovedAt(t time.Time) bool {
+	return state.Retired != nil && t.Before(state.Retired.Timestamp)
+}
+
+type LogType string
+
+const (
+	LogTypeProd = "prod"
+	LogTypeTest = "test"
+)

loglist/validate.go

@@ -0,0 +1,53 @@
+// Copyright (C) 2020 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package loglist
+
+import (
+	"crypto/sha256"
+	"fmt"
+)
+
+func (list *List) Validate() error {
+	for i := range list.Operators {
+		if err := list.Operators[i].Validate(); err != nil {
+			return fmt.Errorf("problem with %dth operator (%s): %w", i, list.Operators[i].Name, err)
+		}
+	}
+	return nil
+}
+
+func (operator *Operator) Validate() error {
+	for i := range operator.Logs {
+		if err := operator.Logs[i].Validate(); err != nil {
+			return fmt.Errorf("problem with %dth non-tiled log (%s): %w", i, operator.Logs[i].LogIDString(), err)
+		}
+	}
+	for i := range operator.TiledLogs {
+		if err := operator.TiledLogs[i].Validate(); err != nil {
+			return fmt.Errorf("problem with %dth tiled log (%s): %w", i, operator.TiledLogs[i].LogIDString(), err)
+		}
+	}
+	return nil
+}
+
+func (log *Log) Validate() error {
+	realLogID := sha256.Sum256(log.Key)
+	if log.LogID != realLogID {
+		return fmt.Errorf("log ID does not match log key")
+	}
+
+	if !log.IsRFC6962() && !log.IsStaticCTAPI() {
+		return fmt.Errorf("URL(s) not provided")
+	} else if log.IsRFC6962() && log.IsStaticCTAPI() {
+		return fmt.Errorf("inconsistent URLs provided")
+	}
+
+	return nil
+}

logs.go

@@ -1,217 +0,0 @@
-// Copyright (C) 2016 Opsmate, Inc.
-//
-// This Source Code Form is subject to the terms of the Mozilla
-// Public License, v. 2.0. If a copy of the MPL was not distributed
-// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
-//
-// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
-// See the Mozilla Public License for details.
-
-package certspotter
-
-import (
-	"crypto"
-	"crypto/sha256"
-	"crypto/x509"
-	"encoding/base64"
-	"time"
-)
-
-type LogInfoFile struct {
-	Logs []LogInfo `json:"logs"`
-}
-type LogInfo struct {
-	Description     string     `json:"description"`
-	Key             []byte     `json:"key"`
-	Url             string     `json:"url"`
-	MMD             int        `json:"maximum_merge_delay"`
-	CertExpiryBegin *time.Time `json:"cert_expiry_begin"`
-	CertExpiryEnd   *time.Time `json:"cert_expiry_end"`
-}
-
-func (info *LogInfo) FullURI() string {
-	return "https://" + info.Url
-}
-
-func (info *LogInfo) ParsedPublicKey() (crypto.PublicKey, error) {
-	if info.Key != nil {
-		return x509.ParsePKIXPublicKey(info.Key)
-	} else {
-		return nil, nil
-	}
-}
-
-func (info *LogInfo) ID() []byte {
-	sum := sha256.Sum256(info.Key)
-	return sum[:]
-}
-
-var DefaultLogs = []LogInfo{
-	{
-		Key: mustDecodeBase64("MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEfahLEimAoz2t01p3uMziiLOl/fHTDM0YDOhBRuiBARsV4UvxG2LdNgoIGLrtCzWE0J5APC2em4JlvR8EEEFMoA=="),
-		Url: "ct.googleapis.com/pilot",
-		MMD: 86400,
-	},
-	{
-		Key: mustDecodeBase64("MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE1/TMabLkDpCjiupacAlP7xNi0I1JYP8bQFAHDG1xhtolSY1l4QgNRzRrvSe8liE+NPWHdjGxfx3JhTsN9x8/6Q=="),
-		Url: "ct.googleapis.com/aviator",
-		MMD: 86400,
-	},
-	{
-		Key: mustDecodeBase64("MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEAkbFvhu7gkAW6MHSrBlpE1n4+HCFRkC5OLAjgqhkTH+/uzSfSl8ois8ZxAD2NgaTZe1M9akhYlrYkes4JECs6A=="),
-		Url: "ct1.digicert-ct.com/log",
-		MMD: 86400,
-	},
-	{
-		Key: mustDecodeBase64("MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEIFsYyDzBi7MxCAC/oJBXK7dHjG+1aLCOkHjpoHPqTyghLpzA9BYbqvnV16mAw04vUjyYASVGJCUoI3ctBcJAeg=="),
-		Url: "ct.googleapis.com/rocketeer",
-		MMD: 86400,
-	},
-	{
-		Key: mustDecodeBase64("MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEluqsHEYMG1XcDfy1lCdGV0JwOmkY4r87xNuroPS2bMBTP01CEDPwWJePa75y9CrsHEKqAy8afig1dpkIPSEUhg=="),
-		Url: "ct.ws.symantec.com",
-		MMD: 86400,
-	},
-	{
-		Key: mustDecodeBase64("MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE6pWeAv/u8TNtS4e8zf0ZF2L/lNPQWQc/Ai0ckP7IRzA78d0NuBEMXR2G3avTK0Zm+25ltzv9WWis36b4ztIYTQ=="),
-		Url: "vega.ws.symantec.com",
-		MMD: 86400,
-	},
-	{
-		Key: mustDecodeBase64("MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAv7UIYZopMgTTJWPp2IXhhuAf1l6a9zM7gBvntj5fLaFm9pVKhKYhVnno94XuXeN8EsDgiSIJIj66FpUGvai5samyetZhLocRuXhAiXXbDNyQ4KR51tVebtEq2zT0mT9liTtGwiksFQccyUsaVPhsHq9gJ2IKZdWauVA2Fm5x9h8B9xKn/L/2IaMpkIYtd967TNTP/dLPgixN1PLCLaypvurDGSVDsuWabA3FHKWL9z8wr7kBkbdpEhLlg2H+NAC+9nGKx+tQkuhZ/hWR65aX+CNUPy2OB9/u2rNPyDydb988LENXoUcMkQT0dU3aiYGkFAY0uZjD2vH97TM20xYtNQIDAQAB"),
-		Url: "ctserver.cnnic.cn",
-		MMD: 86400,
-	},
-	{
-		Key: mustDecodeBase64("MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAETtK8v7MICve56qTHHDhhBOuV4IlUaESxZryCfk9QbG9co/CqPvTsgPDbCpp6oFtyAHwlDhnvr7JijXRD9Cb2FA=="),
-		Url: "ct.googleapis.com/icarus",
-		MMD: 86400,
-	},
-	{
-		Key: mustDecodeBase64("MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEEmyGDvYXsRJsNyXSrYc9DjHsIa2xzb4UR7ZxVoV6mrc9iZB7xjI6+NrOiwH+P/xxkRmOFG6Jel20q37hTh58rA=="),
-		Url: "ct.googleapis.com/skydiver",
-		MMD: 86400,
-	},
-	{
-		Key: mustDecodeBase64("MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEjicnerZVCXTrbEuUhGW85BXx6lrYfA43zro/bAna5ymW00VQb94etBzSg4j/KS/Oqf/fNN51D8DMGA2ULvw3AQ=="),
-		Url: "ctlog-gen2.api.venafi.com",
-		MMD: 86400,
-	},
-	{
-		Key: mustDecodeBase64("MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE7+R9dC4VFbbpuyOL+yy14ceAmEf7QGlo/EmtYU6DRzwat43f/3swtLr/L8ugFOOt1YU/RFmMjGCL17ixv66MZw=="),
-		Url: "mammoth.ct.comodo.com",
-		MMD: 86400,
-	},
-	{
-		Key: mustDecodeBase64("MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE8m/SiQ8/xfiHHqtls9m7FyOMBg4JVZY9CgiixXGz0akvKD6DEL8S0ERmFe9U4ZiA0M4kbT5nmuk3I85Sk4bagA=="),
-		Url: "sabre.ct.comodo.com",
-		MMD: 86400,
-	},
-	{
-		Key: mustDecodeBase64("MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE0gBVBa3VR7QZu82V+ynXWD14JM3ORp37MtRxTmACJV5ZPtfUA7htQ2hofuigZQs+bnFZkje+qejxoyvk2Q1VaA=="),
-		Url: "ct.googleapis.com/logs/argon2018",
-		MMD: 86400,
-		CertExpiryBegin: makeTime(1514764800),
-		CertExpiryEnd: makeTime(1546300800),
-	},
-	{
-		Key: mustDecodeBase64("MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEI3MQm+HzXvaYa2mVlhB4zknbtAT8cSxakmBoJcBKGqGwYS0bhxSpuvABM1kdBTDpQhXnVdcq+LSiukXJRpGHVg=="),
-		Url: "ct.googleapis.com/logs/argon2019",
-		MMD: 86400,
-		CertExpiryBegin: makeTime(1546300800),
-		CertExpiryEnd: makeTime(1577836800),
-	},
-	{
-		Key: mustDecodeBase64("MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE6Tx2p1yKY4015NyIYvdrk36es0uAc1zA4PQ+TGRY+3ZjUTIYY9Wyu+3q/147JG4vNVKLtDWarZwVqGkg6lAYzA=="),
-		Url: "ct.googleapis.com/logs/argon2020",
-		MMD: 86400,
-		CertExpiryBegin: makeTime(1577836800),
-		CertExpiryEnd: makeTime(1609459200),
-	},
-	{
-		Key: mustDecodeBase64("MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAETeBmZOrzZKo4xYktx9gI2chEce3cw/tbr5xkoQlmhB18aKfsxD+MnILgGNl0FOm0eYGilFVi85wLRIOhK8lxKw=="),
-		Url: "ct.googleapis.com/logs/argon2021",
-		MMD: 86400,
-		CertExpiryBegin: makeTime(1609459200),
-		CertExpiryEnd: makeTime(1640995200),
-	},
-	{
-		Key: mustDecodeBase64("MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEowJkhCK7JewN47zCyYl93UXQ7uYVhY/Z5xcbE4Dq7bKFN61qxdglnfr0tPNuFiglN+qjN2Syxwv9UeXBBfQOtQ=="),
-		Url: "sirius.ws.symantec.com",
-		MMD: 86400,
-	},
-	{
-		Key: mustDecodeBase64("MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEzF05L2a4TH/BLgOhNKPoioYCrkoRxvcmajeb8Dj4XQmNY+gxa4Zmz3mzJTwe33i0qMVp+rfwgnliQ/bM/oFmhA=="),
-		Url: "ct2.digicert-ct.com/log",
-		MMD: 86400,
-	},
-	{
-		Key: mustDecodeBase64("MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEAsVpWvrH3Ke0VRaMg9ZQoQjb5g/xh1z3DDa6IuxY5DyPsk6brlvrUNXZzoIg0DcvFiAn2kd6xmu4Obk5XA/nRg=="),
-		Url: "ct.cloudflare.com/logs/nimbus2018",
-		MMD: 86400,
-		CertExpiryBegin: makeTime(1514764800),
-		CertExpiryEnd: makeTime(1546300800),
-	},
-	{
-		Key: mustDecodeBase64("MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEkZHz1v5r8a9LmXSMegYZAg4UW+Ug56GtNfJTDNFZuubEJYgWf4FcC5D+ZkYwttXTDSo4OkanG9b3AI4swIQ28g=="),
-		Url: "ct.cloudflare.com/logs/nimbus2019",
-		MMD: 86400,
-		CertExpiryBegin: makeTime(1546300800),
-		CertExpiryEnd: makeTime(1577836800),
-	},
-	{
-		Key: mustDecodeBase64("MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE01EAhx4o0zPQrXTcYjgCt4MVFsT0Pwjzb1RwrM0lhWDlxAYPP6/gyMCXNkOn/7KFsjL7rwk78tHMpY8rXn8AYg=="),
-		Url: "ct.cloudflare.com/logs/nimbus2020",
-		MMD: 86400,
-		CertExpiryBegin: makeTime(1577836800),
-		CertExpiryEnd: makeTime(1609459200),
-	},
-	{
-		Key: mustDecodeBase64("MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAExpon7ipsqehIeU1bmpog9TFo4Pk8+9oN8OYHl1Q2JGVXnkVFnuuvPgSo2Ep+6vLffNLcmEbxOucz03sFiematg=="),
-		Url: "ct.cloudflare.com/logs/nimbus2021",
-		MMD: 86400,
-		CertExpiryBegin: makeTime(1609459200),
-		CertExpiryEnd: makeTime(1640995200),
-	},
-}
-
-// Logs which monitor certs from distrusted roots
-var UnderwaterLogs = []LogInfo{
-	{
-		Description: "Google 'Submariner' log",
-		Key:         mustDecodeBase64("MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEOfifIGLUV1Voou9JLfA5LZreRLSUMOCeeic8q3Dw0fpRkGMWV0Gtq20fgHQweQJeLVmEByQj9p81uIW4QkWkTw=="),
-		Url:         "ct.googleapis.com/submariner",
-		MMD:         86400,
-	},
-}
-
-// Logs which accept submissions from anyone
-var OpenLogs = []LogInfo{
-	{
-		Key: mustDecodeBase64("MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEfahLEimAoz2t01p3uMziiLOl/fHTDM0YDOhBRuiBARsV4UvxG2LdNgoIGLrtCzWE0J5APC2em4JlvR8EEEFMoA=="),
-		Url: "ct.googleapis.com/pilot",
-		MMD: 86400,
-	},
-	{
-		Key: mustDecodeBase64("MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEIFsYyDzBi7MxCAC/oJBXK7dHjG+1aLCOkHjpoHPqTyghLpzA9BYbqvnV16mAw04vUjyYASVGJCUoI3ctBcJAeg=="),
-		Url: "ct.googleapis.com/rocketeer",
-		MMD: 86400,
-	},
-	{
-		Key: mustDecodeBase64("MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAELPXCMfVjQ2oWSgrewu4fIW4Sfh3lco90CwKZ061pvAI1eflh6c8ACE90pKM0muBDHCN+j0HV7scco4KKQPqq4A=="),
-		Url: "dodo.ct.comodo.com",
-		MMD: 86400,
-	},
-}
-
-func mustDecodeBase64(str string) []byte {
-	bytes, err := base64.StdEncoding.DecodeString(str)
-	if err != nil {
-		panic("MustDecodeBase64: " + err.Error())
-	}
-	return bytes
-}
-func makeTime(seconds int64) *time.Time {
-	t := time.Unix(seconds, 0)
-	return &t
-}

man/.gitignore

@@ -0,0 +1 @@
+*.8

man/Makefile

@@ -0,0 +1,9 @@
+all: certspotter-script.8 certspotter.8
+
+%.8: %.md
+	lowdown -s -Tman \
+		-M title:$(basename $(notdir $@)) \
+		-M section:$(subst .,,$(suffix $@)) \
+		-M date:$(if $(SOURCE_DATE_EPOCH),$(shell date -I -u -d "@$(SOURCE_DATE_EPOCH)"),$(shell date -I -u)) \
+		-o $@ $<
+

man/certspotter-script.md

@@ -0,0 +1,262 @@
+# NAME
+
+**certspotter-script** - Certificate Transparency Log Monitor (hook script)
+
+# DESCRIPTION
+
+**certspotter-script** is *any* program that is executed by **certspotter(8)**
+when it needs to notify you about an event, such as detecting a certificate for
+a domain on your watch list.
+
+Scripts are placed in the `$CERTSPOTTER_CONFIG_DIR/hooks.d` directory
+(`~/.certspotter/hooks.d` by default), or specified on the command line
+using the `-script` argument.
+
+# ENVIRONMENT
+
+## Event information
+
+The following environment variables are set for all types of events:
+
+`EVENT`
+
+:   One of the following values, indicating the type of event:
+
+      * `discovered_cert` - certspotter has discovered a certificate for a
+      domain on your watch list.
+
+      * `malformed_cert` - certspotter can't determine if a certificate
+      matches your watch list because the certificate or the log entry
+      is malformed.
+
+      * `error` - a problem is preventing certspotter from monitoring all
+      logs.
+
+    Additional event types may be defined in the future, so your script should
+    be able to handle unknown values.
+
+`SUMMARY`
+
+:   A short human-readable string describing the event.  This is the same string
+    used in the subject line of emails sent by certspotter.
+
+
+## Discovered certificate information
+
+The following environment variables are set for `discovered_cert` events:
+
+`WATCH_ITEM`
+
+:    The item from your watch list which matches this certificate.
+     (If more than one item matches, the first one is used.)
+
+`LOG_URI`
+
+:    The URI of the log containing the certificate.
+
+`ENTRY_INDEX`
+
+:    The index of the log entry containing the certificate.
+
+`TBS_SHA256`
+
+:    The hex-encoded SHA-256 digest of the TBSCertificate, as defined in RFC 6962 Section 3.2.
+     Certificates and their corresponding precertificates have the same `TBS_SHA256` value.
+
+`CERT_SHA256`
+
+:    The hex-encoded SHA-256 digest (sometimes called fingerprint) of the certificate.
+     The digest is computed over the ASN.1 DER encoding. 
+
+`PUBKEY_SHA256`
+
+:    The hex-encoded SHA-256 digest of the certificate's Subject Public Key Info.
+
+`CERT_FILENAME`
+
+:    Path to a file containing the PEM-encoded certificate chain.  Not set if `-no_save` was used.
+
+`JSON_FILENAME`
+
+:    Path to a JSON file containing additional information about the certificate.  See below for the format of the JSON file.
+     Not set if `-no_save` was used.
+
+`TEXT_FILENAME`
+
+:    Path to a text file containing information about the certificate.  This file contains the same text that
+     certspotter uses in emails.  You should not attempt to parse this file because its format may change in the future.
+     Not set if `-no_save` was used.
+
+`NOT_BEFORE`, `NOT_BEFORE_UNIXTIME`, `NOT_BEFORE_RFC3339`
+
+:    The not before time of the certificate, in a human-readable format, seconds since the UNIX epoch, and RFC3339, respectively.  These variables may be unset if there was a parse error, in which case `VALIDITY_PARSE_ERROR` is set.
+
+`NOT_AFTER`, `NOT_AFTER_UNIXTIME`, `NOT_AFTER_RFC3339`
+
+:    The not after (expiration) time of the certificate, in a human-readable format, seconds since the UNIX epoch, and RFC3339, respectively.  These variables may be unset if there was a parse error, in which case `VALIDITY_PARSE_ERROR` is set.
+
+`VALIDITY_PARSE_ERROR`
+
+:    Error parsing not before and not after, if any.  If this variable is set, then the `NOT_BEFORE` and `NOT_AFTER` family of variables are unset.
+
+`SUBJECT_DN`
+
+:    The distinguished name of the certificate's subject.  This variable may be unset if there was a parse error, in which case `SUBJECT_PARSE_ERROR` is set.
+
+`SUBJECT_PARSE_ERROR`
+
+:    Error parsing the subject, if any.  If this variable is set, then `SUBJECT_DN` is unset.
+
+`ISSUER_DN`
+
+:    The distinguished name of the certificate's issuer.  This variable may be unset if there was a parse error, in which case `ISSUER_PARSE_ERROR` is set.
+
+`ISSUER_PARSE_ERROR`
+
+:    Error parsing the issuer, if any.  If this variable is set, then `ISSUER_DN` is unset.
+
+`SERIAL`
+
+:    The hex-encoded serial number of the certificate.  Prefixed with a minus (-) sign if negative.  This variable may be unset if there was a parse error, in which case `SERIAL_PARSE_ERROR` is set.
+
+`SERIAL_PARSE_ERROR`
+
+:    Error parsing the serial number, if any.  If this variable is set, then `SERIAL` is unset.
+
+`CHAIN_ERROR`
+
+:    Error building or verifying the certificate chain, if any.  If this variable is set, then the certificate chain in `CERT_FILENAME` may be incomplete or invalid.
+
+## Malformed certificate information
+
+The following environment variables are set for `malformed_cert` events:
+
+`LOG_URI`
+
+:    The URI of the log containing the malformed certificate.
+
+`ENTRY_INDEX`
+
+:    The index of the log entry containing the malformed certificate.
+
+`LEAF_HASH`
+
+:    The base64-encoded Merkle hash of the leaf containing the malformed certificate.
+
+`PARSE_ERROR`
+
+:    A human-readable string describing why the certificate is malformed.
+
+`ENTRY_FILENAME`
+
+:    Path to a file containing the JSON log entry.  The file contains a JSON object with two fields, `leaf_input` and `extra_data`, as described in RFC 6962 Section 4.6.
+
+`TEXT_FILENAME`
+
+:    Path to a text file containing a description of the malformed certificate.  This file contains the same text that certspotter uses in emails.
+
+## Error information
+
+The following environment variables are set for `error` events:
+
+`TEXT_FILENAME`
+
+:    Path to a text file containing a description of the error.  This file contains the same text that certspotter uses in emails.
+
+# JSON FILE FORMAT
+
+Unless `-no_save` is used, certspotter saves a JSON file for every discovered certificate
+under `$CERTSPOTTER_STATE_DIR`, and puts the path to the file in `$JSON_FILENAME`.  Your
+script can read the JSON file, such as with the jq(1) command, to get additional information
+about the certificate which isn't appropriate for environment variables.
+
+The JSON file contains an object with the following fields:
+
+`tbs_sha256`
+
+:    A string containing the hex-encoded SHA-256 digest of the TBSCertificate, as defined in RFC 6962 Section 3.2.
+     Certificates and their corresponding precertificates have the same `tbs_sha256` value.
+
+`pubkey_sha256`
+
+:    A string containing the hex-encoded SHA-256 digest of the certificate's Subject Public Key Info.
+
+`dns_names`
+
+:    An array of strings containing the DNS names for which the
+     certificate is valid, taken from both the DNS subject alternative names
+     (SANs) and the subject common name (CN). Internationalized domain names
+     are encoded in Punycode.
+
+`ip_addresses`
+
+:    An array of strings containing the IP addresses for which the certificate is valid,
+     taken from both the IP subject alternative names (SANs) and the subject common name (CN).
+
+`not_before`
+
+:    A string containing the not before time of the certificate in RFC3339 format.
+     Null if there was an error parsing the certificate's validity.
+
+`not_after`
+
+:    A string containing the not after (expiration) time of the certificate in RFC3339 format.
+     Null if there was an error parsing the certificate's validity.
+
+Additional fields will be added in the future based on user feedback. Please open
+an issue at <https://github.com/SSLMate/certspotter> if you have a use case for another field.
+
+# EXAMPLES
+
+Example environment variables for a `discovered_cert` event:
+
+```
+CERT_FILENAME=/home/andrew/.certspotter/certs/3c/3cdc83b3932c194fcdf17aa2bf1abc34e8438b293c3d5c70693e175b38ff128a.pem
+CERT_SHA256=3cdc83b3932c194fcdf17aa2bf1abc34e8438b293c3d5c70693e175b38ff128a
+ENTRY_INDEX=6464843
+EVENT=discovered_cert
+ISSUER_DN=C=GB, ST=Greater Manchester, L=Salford, O=Sectigo Limited, CN=Sectigo RSA Domain Validation Secure Server CA
+JSON_FILENAME=/usr2/andrew/.certspotter/certs/3c/3cdc83b3932c194fcdf17aa2bf1abc34e8438b293c3d5c70693e175b38ff128a.v1.json
+LOG_URI=https://ct.cloudflare.com/logs/nimbus2024/
+NOT_AFTER='2024-01-26 03:47:26 +0000 UTC'
+NOT_AFTER_RFC3339=2024-01-26T03:47:26Z
+NOT_AFTER_UNIXTIME=1706240846
+NOT_BEFORE='2023-01-31 03:47:26 +0000 UTC'
+NOT_BEFORE_RFC3339=2023-01-31T03:47:26Z
+NOT_BEFORE_UNIXTIME=1675136846
+PUBKEY_SHA256=33ac1d9b9e56005ccac045eac2398b3e9dd6b3f5b66ae6260f2d478c7c0d82c8
+SERIAL=c170fbf3bf27481e5c351a4db6f2dc5f
+SUBJECT_DN=CN=sslmate.com
+SUMMARY='certificate discovered for .sslmate.com'
+TBS_SHA256=2388ee81c6f45cffc73e68a35fa8921e839e20acc9a98e8e6dcaea07cbfbdef8
+TEXT_FILENAME=/usr2/andrew/.certspotter/certs/3c/3cdc83b3932c194fcdf17aa2bf1abc34e8438b293c3d5c70693e175b38ff128a.txt
+WATCH_ITEM=.sslmate.com
+```
+
+Example JSON file for a discovered certificate:
+
+```
+{
+  "dns_names": [
+    "sslmate.com",
+    "www.sslmate.com"
+  ],
+  "ip_addresses": [],
+  "not_after": "2024-01-26T03:47:26Z",
+  "not_before": "2023-01-31T03:47:26Z",
+  "pubkey_sha256": "33ac1d9b9e56005ccac045eac2398b3e9dd6b3f5b66ae6260f2d478c7c0d82c8",
+  "tbs_sha256": "2388ee81c6f45cffc73e68a35fa8921e839e20acc9a98e8e6dcaea07cbfbdef8"
+}
+```
+
+# SEE ALSO
+
+certspotter(8)
+
+# COPYRIGHT
+
+Copyright (c) 2016-2025 Opsmate, Inc.
+
+# BUGS
+
+Report bugs to <https://github.com/SSLMate/certspotter>.

man/certspotter.md

@@ -0,0 +1,242 @@
+# NAME
+
+**certspotter** - Certificate Transparency Log Monitor
+
+# SYNOPSIS
+
+**certspotter** [`-start_at_end`] [`-watchlist` *FILENAME*] [`-email` *ADDRESS*] `...`
+
+# DESCRIPTION
+
+**Cert Spotter** is a Certificate Transparency log monitor from SSLMate that
+alerts you when a SSL/TLS certificate is issued for one of your domains.
+Cert Spotter is easier to use than other open source CT monitors, since it does not require
+a database. It's also more robust, since it uses a special certificate parser
+that ensures it won't miss certificates.
+
+Cert Spotter is also available as a hosted service by SSLMate,
+<https://sslmate.com/certspotter>.
+
+You can use Cert Spotter to detect:
+
+ * Certificates issued to attackers who have compromised your DNS and
+   are redirecting your visitors to their malicious site.
+ * Certificates issued to attackers who have taken over an abandoned
+   sub-domain in order to serve malware under your name.
+ * Certificates issued to attackers who have compromised a certificate
+   authority and want to impersonate your site.
+ * Certificates issued in violation of your corporate policy
+   or outside of your centralized certificate procurement process.
+
+# OPTIONS
+
+-batch_size *NUMBER*
+
+:   Maximum number of entries to request per call to get-entries.
+    You should not generally need to change this. Defaults to 1000.
+
+-email *ADDRESS*
+
+:   Email address to contact when a matching certificate is discovered, or
+    an error occurs.  You can specify this option more than once to email
+    multiple addresses.  Your system must have a working sendmail(1) command.
+
+    Regardless of the `-email` option, certspotter also emails any address listed
+    in `$CERTSPOTTER_CONFIG_DIR/email_recipients` file
+    (`~/.certspotter/email_recipients` by default).  (One address per line,
+    blank lines are ignored.)  This file is read only at startup, so you
+    must restart certspotter if you change it.
+
+-healthcheck *INTERVAL*
+
+:   Perform a health check at the given interval (default: "24h") as described
+    below.  *INTERVAL* must be a decimal number followed by "h" for hours or
+    "m" for minutes.
+
+-logs *ADDRESS*
+
+:   Filename or HTTPS URL of a v2 or v3 JSON log list containing logs to monitor.
+    The schema for this file can be found at <https://www.gstatic.com/ct/log_list/v3/log_list_schema.json>.
+    Defaults to <https://loglist.certspotter.org/monitor.json>, which includes
+    the union of active logs recognized by Chrome and Apple.  certspotter periodically
+    reloads the log list in case it has changed.
+
+-no\_save
+
+:   Do not save a copy of matching certificates. Note that enabling this option
+    will cause you to receive duplicate notifications, since certspotter will
+    have no way of knowing if you've been previously notified about a certificate.
+
+-script *COMMAND*
+
+:   Command to execute when a matching certificate is found or an error occurs. See
+    certspotter-script(8) for information about the interface to scripts.
+
+    Regardless of the `-script` option, certspotter also executes any executable
+    file in the `$CERTSPOTTER_CONFIG_DIR/hooks.d` directory
+    (`~/.certspotter/hooks.d` by default).
+
+-start\_at\_end
+
+:   Start monitoring logs from the end rather than the beginning.
+
+    **WARNING**: monitoring from the beginning guarantees detection of all
+    certificates, but requires downloading hundreds of millions of
+    certificates, which takes days.
+
+-state\_dir *PATH*
+
+:   Directory for storing state. Defaults to `$CERTSPOTTER_STATE_DIR`, which is
+    "~/.certspotter" by default.
+
+-stdout
+
+:   Write matching certificates and errors to stdout.
+
+-verbose
+
+:   Be verbose.
+
+-version
+
+:   Print version and exit.
+
+-watchlist *PATH*
+
+:   File containing DNS names to monitor, one per line.  To monitor an entire
+    domain namespace (including the domain itself and all sub-domains) prefix
+    the domain name with a dot (e.g. ".example.com").  To monitor a single DNS
+    name only, do not prefix the name with a dot.
+    
+    Defaults to `$CERTSPOTTER_CONFIG_DIR/watchlist`, which is
+    "~/.certspotter/watchlist" by default.
+    Specify `-` to read the watch list from stdin.
+    
+    certspotter reads the watch list only when starting up, so you must restart
+    certspotter if you change it.
+
+# NOTIFICATIONS
+
+When certspotter detects a certificate matching your watchlist, or encounters
+an error that is preventing it from discovering certificates, it notifies you
+as follows:
+
+* Emails any address specified by the `-email` command line flag.
+
+* Emails any address listed in the `$CERTSPOTTER_CONFIG_DIR/email_recipients`
+  file (`~/.certspotter/email_recipients` by default).  (One address per line,
+  blank lines are ignored.)  This file is read only at startup, so you
+  must restart certspotter if you change it.
+
+* Executes the script specified by the `-script` command line flag.
+
+* Executes every executable file in the `$CERTSPOTTER_CONFIG_DIR/hooks.d`
+ directory (`~/.certspotter/hooks.d` by default).
+
+* Writes the notification to standard out if the `-stdout` flag was specified.
+
+Sending email requires a working sendmail(1) command.  For details about
+the script interface, see certspotter-script(8).
+
+# OPERATION
+
+certspotter continuously monitors all browser-recognized Certificate
+Transparency logs looking for certificates which are valid for any domain
+on your watch list. When certspotter detects a matching certificate, it
+emails you, executes a script, and/or writes a report to standard out,
+as described above.
+
+certspotter also saves a copy of matching certificates in
+`$CERTSPOTTER_STATE_DIR/certs` ("~/.certspotter/certs" by default)
+unless you specify the `-no_save` option.
+
+When certspotter has not previously monitored a log, it can either start
+monitoring the log from the beginning, or seek to the end of the log and
+start monitoring from there. Monitoring from the beginning guarantees
+detection of all certificates, but requires downloading hundreds of
+millions of certificates, which takes days. The default behavior is to
+monitor from the beginning. To start monitoring new logs from the end,
+specify the `-start_at_end` option.
+
+If certspotter has previously monitored a log, it resumes monitoring
+the log from the previous position.  This means that if you add
+a domain to your watch list, certspotter will not detect any certificates
+that were logged prior to the addition.  To detect such certificates,
+you must delete `$CERTSPOTTER_STATE_DIR/logs`, which will cause certspotter
+to restart monitoring from the very beginning of each log (provided
+`-start_at_end` is not specified).  This will cause certspotter to download
+hundreds of millions of certificates, which takes days.  To find preexisting
+certificates, it's faster to use the Cert Spotter service
+<https://sslmate.com/certspotter>, SSLMate's Certificate Transparency Search
+API <https://sslmate.com/ct_search_api>, or a CT search engine such as
+<https://crt.sh>.
+
+# ERROR HANDLING
+
+When certspotter encounters a problem with the local system (e.g. failure
+to write a file or execute a script), it prints a message to stderr and
+exits with a non-zero status.
+
+When certspotter encounters a problem monitoring a log, it prints a message
+to stderr and continues running.  It will try monitoring the log again later;
+most log errors are transient.
+
+Every 24 hours (unless overridden by `-healthcheck`), certspotter performs the
+following health checks:
+
+ * Ensure that the log list has been successfully retrieved at least once
+   since the previous health check.
+ * Ensure that every log has been successfully contacted at least once
+   since the previous health check.
+ * Ensure that certspotter is not falling behind monitoring any logs.
+
+If any health check fails, certspotter notifies you by email, script, and/or
+standard out, as described above.
+
+Health check failures should be rare, and you should take them seriously because it means
+certspotter might not detect all certificates.  It might also be an indication
+of CT log misbehavior.  Consult certspotter's stderr output for details, and if
+you need help, file an issue at <https://github.com/SSLMate/certspotter>.
+
+# EXIT STATUS
+
+certspotter exits 0 when it receives `SIGTERM` or `SIGINT`,
+and non-zero when a serious error occurs.
+
+# ENVIRONMENT
+
+`CERTSPOTTER_STATE_DIR`
+
+:   Directory for storing state. Overridden by `-state_dir`. Defaults to
+    `~/.certspotter`.
+
+`CERTSPOTTER_CONFIG_DIR`
+
+:   Directory from which any configuration, such as the watch list, is read.
+    Defaults to `~/.certspotter`.
+
+`EMAIL`
+
+:   Email address from which to send emails. If not set, certspotter lets sendmail pick
+    the address.
+
+`HTTPS_PROXY`
+
+:   URL of proxy server for making HTTPS requests.  `http://`, `https://`, and
+    `socks5://` URLs are supported.  By default, no proxy server is used.
+
+`SENDMAIL_PATH`
+
+:   Path to the sendmail binary used for sending emails. Defaults to `/usr/sbin/sendmail`.
+
+# SEE ALSO
+
+certspotter-script(8)
+
+# COPYRIGHT
+
+Copyright (c) 2016-2025 Opsmate, Inc.
+
+# BUGS
+
+Report bugs to <https://github.com/SSLMate/certspotter>.

merkletree/collapsed_tree.go

@@ -0,0 +1,182 @@
+// Copyright (C) 2022 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package merkletree
+
+import (
+	"encoding/json"
+	"fmt"
+	"math/bits"
+	"slices"
+)
+
+// CollapsedTree is an efficient representation of a Merkle (sub)tree that permits appending
+// nodes and calculating the root hash.
+type CollapsedTree struct {
+	offset uint64
+	nodes  []Hash
+	size   uint64
+}
+
+func calculateNumNodes(size uint64) int {
+	return bits.OnesCount64(size)
+}
+
+// TODO: phase out this function
+func EmptyCollapsedTree() *CollapsedTree {
+	return &CollapsedTree{nodes: []Hash{}, size: 0}
+}
+
+// TODO: phase out this function
+func NewCollapsedTree(nodes []Hash, size uint64) (*CollapsedTree, error) {
+	tree := new(CollapsedTree)
+	if err := tree.Init(nodes, size); err != nil {
+		return nil, err
+	}
+	return tree, nil
+}
+
+func (tree CollapsedTree) Equal(other CollapsedTree) bool {
+	return tree.offset == other.offset && tree.size == other.size && slices.Equal(tree.nodes, other.nodes)
+}
+
+func (tree CollapsedTree) Clone() CollapsedTree {
+	return CollapsedTree{
+		offset: tree.offset,
+		nodes:  slices.Clone(tree.nodes),
+		size:   tree.size,
+	}
+}
+
+// Add a new leaf hash to the end of the tree.
+// Returns an error if and only if the new tree would be too large for the subtree offset.
+// Always returns a nil error if tree.Offset() == 0.
+func (tree *CollapsedTree) Add(hash Hash) error {
+	if tree.offset > 0 {
+		maxSize := uint64(1) << bits.TrailingZeros64(tree.offset)
+		if tree.size+1 > maxSize {
+			return fmt.Errorf("subtree at offset %d is already at maximum size %d", tree.offset, maxSize)
+		}
+	}
+	tree.nodes = append(tree.nodes, hash)
+	tree.size++
+	tree.collapse()
+	return nil
+}
+
+func (tree *CollapsedTree) Append(other CollapsedTree) error {
+	if tree.offset+tree.size != other.offset {
+		return fmt.Errorf("subtree at offset %d cannot be appended to subtree ending at offset %d", other.offset, tree.offset+tree.size)
+	}
+	if tree.offset > 0 {
+		newSize := tree.size + other.size
+		maxSize := uint64(1) << bits.TrailingZeros64(tree.offset)
+		if newSize > maxSize {
+			return fmt.Errorf("size of new subtree (%d) would exceed maximum size %d for a subtree at offset %d", newSize, maxSize, tree.offset)
+		}
+	}
+	if tree.size > 0 {
+		maxSize := uint64(1) << bits.TrailingZeros64(tree.size)
+		if other.size > maxSize {
+			return fmt.Errorf("tree of size %d is too large to append to a tree of size %d (maximum size is %d)", other.size, tree.size, maxSize)
+		}
+	}
+
+	tree.nodes = append(tree.nodes, other.nodes...)
+	tree.size += other.size
+	tree.collapse()
+	return nil
+}
+
+func (tree *CollapsedTree) collapse() {
+	numNodes := calculateNumNodes(tree.size)
+	for len(tree.nodes) > numNodes {
+		left, right := tree.nodes[len(tree.nodes)-2], tree.nodes[len(tree.nodes)-1]
+		tree.nodes = tree.nodes[:len(tree.nodes)-2]
+		tree.nodes = append(tree.nodes, HashChildren(left, right))
+	}
+}
+
+func (tree CollapsedTree) CalculateRoot() Hash {
+	if len(tree.nodes) == 0 {
+		return HashNothing()
+	}
+	i := len(tree.nodes) - 1
+	hash := tree.nodes[i]
+	for i > 0 {
+		i -= 1
+		hash = HashChildren(tree.nodes[i], hash)
+	}
+	return hash
+}
+
+// Return the subtree offset (0 if this represents an entire tree)
+func (tree CollapsedTree) Offset() uint64 {
+	return tree.offset
+}
+
+// Return a non-nil slice containing the nodes.  The slice
+// must not be modified.
+func (tree CollapsedTree) Nodes() []Hash {
+	if tree.nodes == nil {
+		return []Hash{}
+	} else {
+		return tree.nodes
+	}
+}
+
+// Return the number of leaf nodes in the tree.
+func (tree CollapsedTree) Size() uint64 {
+	return tree.size
+}
+
+type collapsedTreeMessage struct {
+	Offset uint64 `json:"offset,omitempty"`
+	Nodes  []Hash `json:"nodes"` // never nil
+	Size   uint64 `json:"size"`
+}
+
+func (tree CollapsedTree) MarshalJSON() ([]byte, error) {
+	return json.Marshal(collapsedTreeMessage{
+		Offset: tree.offset,
+		Nodes:  tree.Nodes(),
+		Size:   tree.size,
+	})
+}
+
+func (tree *CollapsedTree) UnmarshalJSON(b []byte) error {
+	var rawTree collapsedTreeMessage
+	if err := json.Unmarshal(b, &rawTree); err != nil {
+		return fmt.Errorf("error unmarshalling Collapsed Merkle Tree: %w", err)
+	}
+	if err := tree.InitSubtree(rawTree.Offset, rawTree.Nodes, rawTree.Size); err != nil {
+		return fmt.Errorf("error unmarshalling Collapsed Merkle Tree: %w", err)
+	}
+	return nil
+}
+
+func (tree *CollapsedTree) Init(nodes []Hash, size uint64) error {
+	if len(nodes) != calculateNumNodes(size) {
+		return fmt.Errorf("nodes has wrong length (should be %d, not %d)", calculateNumNodes(size), len(nodes))
+	}
+	tree.size = size
+	tree.nodes = nodes
+	return nil
+}
+
+func (tree *CollapsedTree) InitSubtree(offset uint64, nodes []Hash, size uint64) error {
+	if offset > 0 {
+		maxSize := uint64(1) << bits.TrailingZeros64(offset)
+		if size > maxSize {
+			return fmt.Errorf("subtree size (%d) is too large for offset %d (maximum size is %d)", size, offset, maxSize)
+		}
+	}
+	tree.offset = offset
+	return tree.Init(nodes, size)
+}

merkletree/fragment.go

@@ -0,0 +1,122 @@
+// Copyright (C) 2024 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package merkletree
+
+import (
+	"encoding/json"
+	"fmt"
+	"slices"
+)
+
+// FragmentedCollapsedTree represents a sequence of non-overlapping subtrees
+type FragmentedCollapsedTree struct {
+	subtrees []CollapsedTree // sorted by offset
+}
+
+func (tree *FragmentedCollapsedTree) AddHash(position uint64, hash Hash) error {
+	return tree.Add(CollapsedTree{
+		offset: position,
+		nodes:  []Hash{hash},
+		size:   1,
+	})
+}
+
+func (tree *FragmentedCollapsedTree) Add(subtree CollapsedTree) error {
+	if subtree.size == 0 {
+		return nil
+	}
+	i := len(tree.subtrees)
+	for i > 0 && tree.subtrees[i-1].offset > subtree.offset {
+		i--
+	}
+	if i > 0 && tree.subtrees[i-1].offset+tree.subtrees[i-1].size > subtree.offset {
+		return fmt.Errorf("overlaps with subtree ending at %d", tree.subtrees[i-1].offset+tree.subtrees[i-1].size)
+	}
+	if i < len(tree.subtrees) && subtree.offset+subtree.size > tree.subtrees[i].offset {
+		return fmt.Errorf("overlaps with subtree starting at %d", tree.subtrees[i].offset)
+	}
+	if i == 0 || tree.subtrees[i-1].Append(subtree) != nil {
+		tree.subtrees = slices.Insert(tree.subtrees, i, subtree)
+		i++
+	}
+	for i < len(tree.subtrees) && tree.subtrees[i-1].Append(tree.subtrees[i]) == nil {
+		tree.subtrees = slices.Delete(tree.subtrees, i, i+1)
+	}
+	return nil
+}
+
+func (tree *FragmentedCollapsedTree) Merge(other FragmentedCollapsedTree) error {
+	// TODO: try to make this linear time instead of quadratic; it should be possible since the subtrees are sorted by offset
+	for _, subtree := range other.subtrees {
+		if err := tree.Add(subtree); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func (tree FragmentedCollapsedTree) Gaps(yield func(uint64, uint64) bool) {
+	var prevEnd uint64
+	for i := range tree.subtrees {
+		if prevEnd != tree.subtrees[i].offset {
+			if !yield(prevEnd, tree.subtrees[i].offset) {
+				return
+			}
+		}
+		prevEnd = tree.subtrees[i].offset + tree.subtrees[i].size
+	}
+	yield(prevEnd, 0)
+}
+
+func (tree FragmentedCollapsedTree) NumSubtrees() int {
+	return len(tree.subtrees)
+}
+
+func (tree FragmentedCollapsedTree) Subtree(i int) CollapsedTree {
+	return tree.subtrees[i]
+}
+
+func (tree FragmentedCollapsedTree) Subtrees() []CollapsedTree {
+	if tree.subtrees == nil {
+		return []CollapsedTree{}
+	} else {
+		return tree.subtrees
+	}
+}
+
+// Return true iff the tree contains at least the first n nodes (without any gaps)
+func (tree FragmentedCollapsedTree) ContainsFirstN(n uint64) bool {
+	return len(tree.subtrees) >= 1 && tree.subtrees[0].offset == 0 && tree.subtrees[0].size >= n
+}
+
+func (tree *FragmentedCollapsedTree) Init(subtrees []CollapsedTree) error {
+	for i := 1; i < len(subtrees); i++ {
+		if subtrees[i-1].offset+subtrees[i-1].size > subtrees[i].offset {
+			return fmt.Errorf("subtrees %d and %d overlap", i-1, i)
+		}
+	}
+	tree.subtrees = subtrees
+	return nil
+}
+
+func (tree FragmentedCollapsedTree) MarshalJSON() ([]byte, error) {
+	return json.Marshal(tree.Subtrees())
+}
+
+func (tree *FragmentedCollapsedTree) UnmarshalJSON(b []byte) error {
+	var subtrees []CollapsedTree
+	if err := json.Unmarshal(b, &subtrees); err != nil {
+		return fmt.Errorf("error unmarshaling Fragmented Collapsed Merkle Tree: %w", err)
+	}
+	if err := tree.Init(subtrees); err != nil {
+		return fmt.Errorf("error unmarshaling Fragmented Collapsed Merkle Tree: %w", err)
+	}
+	return nil
+}

merkletree/hash.go

@@ -0,0 +1,72 @@
+// Copyright (C) 2022 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package merkletree
+
+import (
+	"crypto/sha256"
+	"encoding/base64"
+	"encoding/json"
+	"fmt"
+)
+
+const HashLen = 32
+
+type Hash [HashLen]byte
+
+func (h Hash) Base64String() string {
+	return base64.StdEncoding.EncodeToString(h[:])
+}
+
+func (h Hash) MarshalJSON() ([]byte, error) {
+	return json.Marshal(h[:])
+}
+
+func (h Hash) MarshalBinary() ([]byte, error) {
+	return h[:], nil
+}
+
+func (h *Hash) UnmarshalJSON(b []byte) error {
+	var hashBytes []byte
+	if err := json.Unmarshal(b, &hashBytes); err != nil {
+		return err
+	}
+	return h.UnmarshalBinary(hashBytes)
+}
+
+func (h *Hash) UnmarshalBinary(hashBytes []byte) error {
+	if len(hashBytes) != HashLen {
+		return fmt.Errorf("Merkle Tree hash has wrong length (should be %d bytes long, not %d)", HashLen, len(hashBytes))
+	}
+	copy(h[:], hashBytes)
+	return nil
+}
+
+func HashNothing() Hash {
+	return sha256.Sum256(nil)
+}
+
+func HashLeaf(leafBytes []byte) Hash {
+	var hash Hash
+	hasher := sha256.New()
+	hasher.Write([]byte{0x00})
+	hasher.Write(leafBytes)
+	hasher.Sum(hash[:0])
+	return hash
+}
+
+func HashChildren(left Hash, right Hash) Hash {
+	var hash Hash
+	hasher := sha256.New()
+	hasher.Write([]byte{0x01})
+	hasher.Write(left[:])
+	hasher.Write(right[:])
+	hasher.Sum(hash[:0])
+	return hash
+}

monitor/config.go

@@ -0,0 +1,23 @@
+// Copyright (C) 2023 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package monitor
+
+import (
+	"time"
+)
+
+type Config struct {
+	LogListSource       string
+	State               StateProvider
+	StartAtEnd          bool
+	WatchList           WatchList
+	Verbose             bool
+	HealthCheckInterval time.Duration
+}

monitor/daemon.go

@@ -0,0 +1,168 @@
+// Copyright (C) 2023 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package monitor
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"golang.org/x/sync/errgroup"
+	"log"
+	insecurerand "math/rand"
+	"software.sslmate.com/src/certspotter/loglist"
+	"time"
+)
+
+const (
+	reloadLogListIntervalMin = 30 * time.Minute
+	reloadLogListIntervalMax = 90 * time.Minute
+)
+
+func randomDuration(min, max time.Duration) time.Duration {
+	return min + time.Duration(insecurerand.Int63n(int64(max-min+1)))
+}
+
+func reloadLogListInterval() time.Duration {
+	return randomDuration(reloadLogListIntervalMin, reloadLogListIntervalMax)
+}
+
+type task struct {
+	log  *loglist.Log
+	stop context.CancelFunc
+}
+
+type daemon struct {
+	config         *Config
+	taskgroup      *errgroup.Group
+	tasks          map[LogID]task
+	logsLoadedAt   time.Time
+	logListToken   *loglist.ModificationToken
+	logListError   string
+	logListErrorAt time.Time
+}
+
+func (daemon *daemon) healthCheck(ctx context.Context) error {
+	if time.Since(daemon.logsLoadedAt) >= daemon.config.HealthCheckInterval {
+		info := &StaleLogListInfo{
+			Source:        daemon.config.LogListSource,
+			LastSuccess:   daemon.logsLoadedAt,
+			LastError:     daemon.logListError,
+			LastErrorTime: daemon.logListErrorAt,
+		}
+		if err := daemon.config.State.NotifyHealthCheckFailure(ctx, nil, info); err != nil {
+			return fmt.Errorf("error notifying about stale log list: %w", err)
+		}
+	}
+
+	for _, task := range daemon.tasks {
+		if err := healthCheckLog(ctx, daemon.config, task.log); err != nil {
+			return fmt.Errorf("error checking health of log %q: %w", task.log.GetMonitoringURL(), err)
+		}
+	}
+	return nil
+}
+
+func (daemon *daemon) startTask(ctx context.Context, ctlog *loglist.Log) task {
+	ctx, cancel := context.WithCancel(ctx)
+	daemon.taskgroup.Go(func() error {
+		defer cancel()
+		err := monitorLogContinously(ctx, daemon.config, ctlog)
+		if daemon.config.Verbose {
+			log.Printf("task for log %s stopped with error %s", ctlog.GetMonitoringURL(), err)
+		}
+		if ctx.Err() == context.Canceled && errors.Is(err, context.Canceled) {
+			return nil
+		} else {
+			return fmt.Errorf("error while monitoring %s: %w", ctlog.GetMonitoringURL(), err)
+		}
+	})
+	return task{log: ctlog, stop: cancel}
+}
+
+func (daemon *daemon) loadLogList(ctx context.Context) error {
+	newLogList, newToken, err := getLogList(ctx, daemon.config.LogListSource, daemon.logListToken)
+	if errors.Is(err, loglist.ErrNotModified) {
+		return nil
+	} else if err != nil {
+		return err
+	}
+
+	if daemon.config.Verbose {
+		log.Printf("fetched %d logs from %q", len(newLogList), daemon.config.LogListSource)
+	}
+
+	for logID, task := range daemon.tasks {
+		if _, exists := newLogList[logID]; exists {
+			continue
+		}
+		if daemon.config.Verbose {
+			log.Printf("stopping task for log %s", logID.Base64String())
+		}
+		task.stop()
+		delete(daemon.tasks, logID)
+	}
+	for logID, ctlog := range newLogList {
+		if _, isRunning := daemon.tasks[logID]; isRunning {
+			continue
+		}
+		if daemon.config.Verbose {
+			log.Printf("starting task for log %s (%s)", logID.Base64String(), ctlog.GetMonitoringURL())
+		}
+		daemon.tasks[logID] = daemon.startTask(ctx, ctlog)
+	}
+	daemon.logsLoadedAt = time.Now()
+	daemon.logListToken = newToken
+	return nil
+}
+
+func (daemon *daemon) run(ctx context.Context) error {
+	if err := daemon.config.State.Prepare(ctx); err != nil {
+		return fmt.Errorf("error preparing state: %w", err)
+	}
+
+	if err := daemon.loadLogList(ctx); err != nil {
+		return fmt.Errorf("error loading log list: %w", err)
+	}
+
+	reloadLogListTicker := time.NewTicker(reloadLogListInterval())
+	defer reloadLogListTicker.Stop()
+
+	healthCheckTicker := time.NewTicker(daemon.config.HealthCheckInterval)
+	defer healthCheckTicker.Stop()
+
+	for ctx.Err() == nil {
+		select {
+		case <-ctx.Done():
+		case <-reloadLogListTicker.C:
+			if err := daemon.loadLogList(ctx); err != nil {
+				daemon.logListError = err.Error()
+				daemon.logListErrorAt = time.Now()
+				recordError(ctx, daemon.config, nil, fmt.Errorf("error reloading log list (will try again later): %w", err))
+			}
+			reloadLogListTicker.Reset(reloadLogListInterval())
+		case <-healthCheckTicker.C:
+			if err := daemon.healthCheck(ctx); err != nil {
+				return err
+			}
+		}
+	}
+	return ctx.Err()
+}
+
+func Run(ctx context.Context, config *Config) error {
+	group, ctx := errgroup.WithContext(ctx)
+	daemon := &daemon{
+		config:    config,
+		taskgroup: group,
+		tasks:     make(map[LogID]task),
+	}
+	group.Go(func() error { return daemon.run(ctx) })
+	return group.Wait()
+}

monitor/discoveredcert.go

@@ -0,0 +1,187 @@
+// Copyright (C) 2023 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package monitor
+
+import (
+	"bytes"
+	"encoding/hex"
+	"encoding/pem"
+	"fmt"
+	"strings"
+	"time"
+
+	"software.sslmate.com/src/certspotter"
+	"software.sslmate.com/src/certspotter/cttypes"
+)
+
+type DiscoveredCert struct {
+	WatchItem    WatchItem
+	LogEntry     *LogEntry
+	Info         *certspotter.CertInfo
+	Chain        []cttypes.ASN1Cert // first entry is the leaf certificate or precertificate
+	ChainError   error              // any error generating or validating Chain; if non-nil, Chain may be partial or incorrect
+	TBSSHA256    [32]byte           // computed over Info.TBS.Raw
+	SHA256       [32]byte           // computed over Chain[0]
+	PubkeySHA256 [32]byte           // computed over Info.TBS.PublicKey.FullBytes
+	Identifiers  *certspotter.Identifiers
+}
+
+type certPaths struct {
+	certPath string
+	jsonPath string
+	textPath string
+}
+
+func (cert *DiscoveredCert) pemChain() []byte {
+	var buffer bytes.Buffer
+	if cert.ChainError != nil {
+		fmt.Fprintln(&buffer, "Warning: this chain may be incomplete or invalid: ", cert.ChainError)
+	}
+	for _, certBytes := range cert.Chain {
+		if err := pem.Encode(&buffer, &pem.Block{
+			Type:  "CERTIFICATE",
+			Bytes: certBytes,
+		}); err != nil {
+			panic(fmt.Errorf("encoding certificate as PEM failed unexpectedly: %w", err))
+		}
+	}
+	return buffer.Bytes()
+}
+
+func (cert *DiscoveredCert) json() any {
+	object := map[string]any{
+		"tbs_sha256":    hex.EncodeToString(cert.TBSSHA256[:]),
+		"pubkey_sha256": hex.EncodeToString(cert.PubkeySHA256[:]),
+		"dns_names":     cert.Identifiers.DNSNames,
+		"ip_addresses":  cert.Identifiers.IPAddrs,
+	}
+
+	if cert.Info.ValidityParseError == nil {
+		object["not_before"] = cert.Info.Validity.NotBefore
+		object["not_after"] = cert.Info.Validity.NotAfter
+	} else {
+		object["not_before"] = nil
+		object["not_after"] = nil
+	}
+
+	return object
+}
+
+func writeCertFiles(cert *DiscoveredCert, paths *certPaths) error {
+	if err := writeFile(paths.certPath, cert.pemChain(), 0666); err != nil {
+		return err
+	}
+	if err := writeJSONFile(paths.jsonPath, cert.json(), 0666); err != nil {
+		return err
+	}
+	if err := writeTextFile(paths.textPath, certNotificationText(cert, paths), 0666); err != nil {
+		return err
+	}
+	return nil
+}
+
+func certNotificationEnviron(cert *DiscoveredCert, paths *certPaths) []string {
+	env := []string{
+		"EVENT=discovered_cert",
+		"SUMMARY=" + certNotificationSummary(cert),
+		"CERT_PARSEABLE=yes", // backwards compat with pre-0.15.0; not documented
+		"LOG_URI=" + cert.LogEntry.Log.GetMonitoringURL(),
+		"ENTRY_INDEX=" + fmt.Sprint(cert.LogEntry.Index),
+		"WATCH_ITEM=" + cert.WatchItem.String(),
+		"TBS_SHA256=" + hex.EncodeToString(cert.TBSSHA256[:]),
+		"CERT_SHA256=" + hex.EncodeToString(cert.SHA256[:]),
+		"FINGERPRINT=" + hex.EncodeToString(cert.SHA256[:]), // backwards compat with pre-0.15.0; not documented
+		"PUBKEY_SHA256=" + hex.EncodeToString(cert.PubkeySHA256[:]),
+		"PUBKEY_HASH=" + hex.EncodeToString(cert.PubkeySHA256[:]), // backwards compat with pre-0.15.0; not documented
+	}
+
+	if paths != nil {
+		env = append(env, "CERT_FILENAME="+paths.certPath)
+		env = append(env, "JSON_FILENAME="+paths.jsonPath)
+		env = append(env, "TEXT_FILENAME="+paths.textPath)
+	}
+
+	if cert.Info.ValidityParseError == nil {
+		env = append(env, "NOT_BEFORE="+cert.Info.Validity.NotBefore.String())
+		env = append(env, "NOT_BEFORE_UNIXTIME="+fmt.Sprint(cert.Info.Validity.NotBefore.Unix()))
+		env = append(env, "NOT_BEFORE_RFC3339="+cert.Info.Validity.NotBefore.Format(time.RFC3339))
+		env = append(env, "NOT_AFTER="+cert.Info.Validity.NotAfter.String())
+		env = append(env, "NOT_AFTER_UNIXTIME="+fmt.Sprint(cert.Info.Validity.NotAfter.Unix()))
+		env = append(env, "NOT_AFTER_RFC3339="+cert.Info.Validity.NotAfter.Format(time.RFC3339))
+	} else {
+		env = append(env, "VALIDITY_PARSE_ERROR="+cert.Info.ValidityParseError.Error())
+	}
+
+	if cert.Info.SubjectParseError == nil {
+		env = append(env, "SUBJECT_DN="+cert.Info.Subject.String())
+	} else {
+		env = append(env, "SUBJECT_PARSE_ERROR="+cert.Info.SubjectParseError.Error())
+	}
+
+	if cert.Info.IssuerParseError == nil {
+		env = append(env, "ISSUER_DN="+cert.Info.Issuer.String())
+	} else {
+		env = append(env, "ISSUER_PARSE_ERROR="+cert.Info.IssuerParseError.Error())
+	}
+
+	if cert.Info.SerialNumberParseError == nil {
+		env = append(env, "SERIAL="+fmt.Sprintf("%x", cert.Info.SerialNumber))
+	} else {
+		env = append(env, "SERIAL_PARSE_ERROR="+cert.Info.SerialNumberParseError.Error())
+	}
+
+	if cert.ChainError != nil {
+		env = append(env, "CHAIN_ERROR="+cert.ChainError.Error())
+	}
+
+	return env
+}
+
+func certNotificationText(cert *DiscoveredCert, paths *certPaths) string {
+	// TODO-4: improve the output: include WatchItem, indicate hash algorithm used for fingerprints, ... (look at SSLMate email for inspiration)
+
+	text := new(strings.Builder)
+	writeField := func(name string, value any) { fmt.Fprintf(text, "\t%13s = %s\n", name, value) }
+
+	fmt.Fprintf(text, "%x:\n", cert.SHA256)
+	for _, dnsName := range cert.Identifiers.DNSNames {
+		writeField("DNS Name", dnsName)
+	}
+	for _, ipaddr := range cert.Identifiers.IPAddrs {
+		writeField("IP Address", ipaddr)
+	}
+	writeField("Pubkey", hex.EncodeToString(cert.PubkeySHA256[:]))
+	if cert.Info.IssuerParseError == nil {
+		writeField("Issuer", cert.Info.Issuer)
+	} else {
+		writeField("Issuer", fmt.Sprintf("[unable to parse: %s]", cert.Info.IssuerParseError))
+	}
+	if cert.Info.ValidityParseError == nil {
+		writeField("Not Before", cert.Info.Validity.NotBefore)
+		writeField("Not After", cert.Info.Validity.NotAfter)
+	} else {
+		writeField("Not Before", fmt.Sprintf("[unable to parse: %s]", cert.Info.ValidityParseError))
+		writeField("Not After", fmt.Sprintf("[unable to parse: %s]", cert.Info.ValidityParseError))
+	}
+	writeField("Log Entry", fmt.Sprintf("%d @ %s", cert.LogEntry.Index, cert.LogEntry.Log.GetMonitoringURL()))
+	writeField("crt.sh", "https://crt.sh/?sha256="+hex.EncodeToString(cert.SHA256[:]))
+	if cert.ChainError != nil {
+		writeField("Error Building Chain", cert.ChainError.Error())
+	}
+	if paths != nil {
+		writeField("Filename", paths.certPath)
+	}
+
+	return text.String()
+}
+
+func certNotificationSummary(cert *DiscoveredCert) string {
+	return fmt.Sprintf("Certificate Discovered for %s", cert.WatchItem)
+}

monitor/errors.go

@@ -0,0 +1,28 @@
+// Copyright (C) 2023 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package monitor
+
+import (
+	"context"
+	"log"
+
+	"software.sslmate.com/src/certspotter/loglist"
+)
+
+func recordError(ctx context.Context, config *Config, ctlog *loglist.Log, errToRecord error) {
+	if err := config.State.NotifyError(ctx, ctlog, errToRecord); err != nil {
+		log.Print("unable to notify about error: ", err)
+		if ctlog == nil {
+			log.Print(errToRecord)
+		} else {
+			log.Print(ctlog.GetMonitoringURL(), ": ", errToRecord)
+		}
+	}
+}

monitor/fileutils.go

@@ -0,0 +1,71 @@
+// Copyright (C) 2017, 2023 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package monitor
+
+import (
+	"crypto/rand"
+	"encoding/hex"
+	"encoding/json"
+	"fmt"
+	"os"
+)
+
+func randomFileSuffix() string {
+	var randomBytes [12]byte
+	if _, err := rand.Read(randomBytes[:]); err != nil {
+		panic(err)
+	}
+	return hex.EncodeToString(randomBytes[:])
+}
+
+func writeSyncFile(filename string, data []byte, perm os.FileMode) error {
+	f, err := os.OpenFile(filename, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, perm)
+	if err != nil {
+		return err
+	}
+	_, err = f.Write(data)
+	if err2 := f.Sync(); err2 != nil && err == nil {
+		err = err2
+	}
+	if err2 := f.Close(); err2 != nil && err == nil {
+		err = err2
+	}
+	return err
+}
+
+func writeFile(filename string, data []byte, perm os.FileMode) error {
+	tempname := filename + ".tmp." + randomFileSuffix()
+	if err := writeSyncFile(tempname, data, perm); err != nil {
+		return fmt.Errorf("error writing %s: %w", filename, err)
+	}
+	if err := os.Rename(tempname, filename); err != nil {
+		os.Remove(tempname)
+		return fmt.Errorf("error writing %s: %w", filename, err)
+	}
+	return nil
+}
+
+func writeTextFile(filename string, fileText string, perm os.FileMode) error {
+	return writeFile(filename, []byte(fileText), perm)
+}
+
+func writeJSONFile(filename string, data any, perm os.FileMode) error {
+	fileBytes, err := json.Marshal(data)
+	if err != nil {
+		return err
+	}
+	fileBytes = append(fileBytes, '\n')
+	return writeFile(filename, fileBytes, perm)
+}
+
+func fileExists(filename string) bool {
+	_, err := os.Lstat(filename)
+	return err == nil
+}

monitor/fsstate.go

@@ -0,0 +1,257 @@
+// Copyright (C) 2024 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package monitor
+
+import (
+	"context"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io/fs"
+	"log"
+	"os"
+	"path/filepath"
+	"strings"
+
+	"software.sslmate.com/src/certspotter/cttypes"
+	"software.sslmate.com/src/certspotter/loglist"
+	"software.sslmate.com/src/certspotter/merkletree"
+)
+
+type FilesystemState struct {
+	StateDir  string
+	CacheDir  string
+	SaveCerts bool
+	Script    string
+	ScriptDir string
+	Email     []string
+	Stdout    bool
+}
+
+func (s *FilesystemState) logStateDir(logID LogID) string {
+	return filepath.Join(s.StateDir, "logs", logID.Base64URLString())
+}
+
+func (s *FilesystemState) Prepare(ctx context.Context) error {
+	if err := prepareStateDir(s.StateDir); err != nil {
+		return err
+	}
+	if err := prepareCacheDir(s.CacheDir); err != nil {
+		return err
+	}
+	return nil
+}
+
+func (s *FilesystemState) PrepareLog(ctx context.Context, logID LogID) error {
+	var (
+		stateDirPath        = s.logStateDir(logID)
+		sthsDirPath         = filepath.Join(stateDirPath, "unverified_sths")
+		malformedDirPath    = filepath.Join(stateDirPath, "malformed_entries")
+		healthchecksDirPath = filepath.Join(stateDirPath, "healthchecks")
+	)
+	for _, dirPath := range []string{stateDirPath, sthsDirPath, malformedDirPath, healthchecksDirPath} {
+		if err := os.Mkdir(dirPath, 0777); err != nil && !errors.Is(err, fs.ErrExist) {
+			return err
+		}
+	}
+	return nil
+}
+
+func (s *FilesystemState) LoadLogState(ctx context.Context, logID LogID) (*LogState, error) {
+	filePath := filepath.Join(s.logStateDir(logID), "state.json")
+	fileBytes, err := os.ReadFile(filePath)
+	if errors.Is(err, fs.ErrNotExist) {
+		return nil, nil
+	} else if err != nil {
+		return nil, err
+	}
+	state := new(LogState)
+	if err := json.Unmarshal(fileBytes, state); err != nil {
+		return nil, fmt.Errorf("error parsing %s: %w", filePath, err)
+	}
+	return state, nil
+}
+
+func (s *FilesystemState) StoreLogState(ctx context.Context, logID LogID, state *LogState) error {
+	filePath := filepath.Join(s.logStateDir(logID), "state.json")
+	return writeJSONFile(filePath, state, 0666)
+}
+
+func (s *FilesystemState) StoreSTH(ctx context.Context, logID LogID, sth *cttypes.SignedTreeHead) error {
+	sthsDirPath := filepath.Join(s.logStateDir(logID), "unverified_sths")
+	return storeSTHInDir(sthsDirPath, sth)
+}
+
+func (s *FilesystemState) LoadSTHs(ctx context.Context, logID LogID) ([]*StoredSTH, error) {
+	sthsDirPath := filepath.Join(s.logStateDir(logID), "unverified_sths")
+	return loadSTHsFromDir(sthsDirPath)
+}
+
+func (s *FilesystemState) RemoveSTH(ctx context.Context, logID LogID, sth *cttypes.SignedTreeHead) error {
+	sthsDirPath := filepath.Join(s.logStateDir(logID), "unverified_sths")
+	return removeSTHFromDir(sthsDirPath, sth)
+}
+
+func (s *FilesystemState) StoreIssuer(ctx context.Context, fingerprint *[32]byte, issuer []byte) error {
+	filePath := filepath.Join(s.CacheDir, "issuers", hex.EncodeToString(fingerprint[:]))
+	return writeFile(filePath, issuer, 0666)
+}
+
+func (s *FilesystemState) LoadIssuer(ctx context.Context, fingerprint *[32]byte) ([]byte, error) {
+	filePath := filepath.Join(s.CacheDir, "issuers", hex.EncodeToString(fingerprint[:]))
+	issuer, err := os.ReadFile(filePath)
+	if errors.Is(err, fs.ErrNotExist) {
+		return nil, nil
+	} else if err != nil {
+		return nil, err
+	} else {
+		return issuer, err
+	}
+}
+
+func (s *FilesystemState) NotifyCert(ctx context.Context, cert *DiscoveredCert) error {
+	var notifiedPath string
+	var paths *certPaths
+	if s.SaveCerts {
+		hexFingerprint := hex.EncodeToString(cert.SHA256[:])
+		prefixPath := filepath.Join(s.StateDir, "certs", hexFingerprint[0:2])
+		var (
+			notifiedFilename      = "." + hexFingerprint + ".notified"
+			certFilename          = hexFingerprint + ".pem"
+			jsonFilename          = hexFingerprint + ".v1.json"
+			textFilename          = hexFingerprint + ".txt"
+			legacyCertFilename    = hexFingerprint + ".cert.pem"
+			legacyPrecertFilename = hexFingerprint + ".precert.pem"
+		)
+
+		for _, filename := range []string{notifiedFilename, legacyCertFilename, legacyPrecertFilename} {
+			if fileExists(filepath.Join(prefixPath, filename)) {
+				return nil
+			}
+		}
+
+		if err := os.Mkdir(prefixPath, 0777); err != nil && !errors.Is(err, fs.ErrExist) {
+			return fmt.Errorf("error creating directory in which to save certificate %x: %w", cert.SHA256, err)
+		}
+
+		notifiedPath = filepath.Join(prefixPath, notifiedFilename)
+		paths = &certPaths{
+			certPath: filepath.Join(prefixPath, certFilename),
+			jsonPath: filepath.Join(prefixPath, jsonFilename),
+			textPath: filepath.Join(prefixPath, textFilename),
+		}
+		if err := writeCertFiles(cert, paths); err != nil {
+			return fmt.Errorf("error saving certificate %x: %w", cert.SHA256, err)
+		}
+	} else {
+		// TODO-4: save cert to temporary files, and defer their unlinking
+	}
+
+	if err := s.notify(ctx, &notification{
+		summary: certNotificationSummary(cert),
+		environ: certNotificationEnviron(cert, paths),
+		text:    certNotificationText(cert, paths),
+	}); err != nil {
+		return fmt.Errorf("error notifying about discovered certificate for %s (%x): %w", cert.WatchItem, cert.SHA256, err)
+	}
+
+	if notifiedPath != "" {
+		if err := os.WriteFile(notifiedPath, nil, 0666); err != nil {
+			return fmt.Errorf("error saving certificate %x: %w", cert.SHA256, err)
+		}
+	}
+
+	return nil
+}
+
+func (s *FilesystemState) NotifyMalformedEntry(ctx context.Context, entry *LogEntry, parseError error) error {
+	var (
+		dirPath   = filepath.Join(s.logStateDir(entry.Log.LogID), "malformed_entries")
+		entryPath = filepath.Join(dirPath, fmt.Sprintf("%d.json", entry.Index))
+		textPath  = filepath.Join(dirPath, fmt.Sprintf("%d.txt", entry.Index))
+	)
+
+	summary := fmt.Sprintf("Unable to Parse Entry %d in %s", entry.Index, entry.Log.GetMonitoringURL())
+	leafHash := merkletree.HashLeaf(entry.LeafInput())
+
+	text := new(strings.Builder)
+	writeField := func(name string, value any) { fmt.Fprintf(text, "\t%13s = %s\n", name, value) }
+	fmt.Fprintf(text, "Unable to determine if log entry matches your watchlist. Please file a bug report at https://github.com/SSLMate/certspotter/issues/new with the following details:\n")
+	writeField("Log Entry", fmt.Sprintf("%d @ %s", entry.Index, entry.Log.GetMonitoringURL()))
+	writeField("Leaf Hash", leafHash.Base64String())
+	writeField("Error", parseError.Error())
+
+	if err := writeJSONFile(entryPath, entry.Entry, 0666); err != nil {
+		return fmt.Errorf("error saving JSON file: %w", err)
+	}
+	if err := writeTextFile(textPath, text.String(), 0666); err != nil {
+		return fmt.Errorf("error saving texT file: %w", err)
+	}
+
+	environ := []string{
+		"EVENT=malformed_cert",
+		"SUMMARY=" + summary,
+		"LOG_URI=" + entry.Log.GetMonitoringURL(),
+		"ENTRY_INDEX=" + fmt.Sprint(entry.Index),
+		"LEAF_HASH=" + leafHash.Base64String(),
+		"PARSE_ERROR=" + parseError.Error(),
+		"ENTRY_FILENAME=" + entryPath,
+		"TEXT_FILENAME=" + textPath,
+		"CERT_PARSEABLE=no", // backwards compat with pre-0.15.0; not documented
+	}
+
+	if err := s.notify(ctx, &notification{
+		environ: environ,
+		summary: summary,
+		text:    text.String(),
+	}); err != nil {
+		return err
+	}
+	return nil
+}
+
+func (s *FilesystemState) healthCheckDir(ctlog *loglist.Log) string {
+	if ctlog == nil {
+		return filepath.Join(s.StateDir, "healthchecks")
+	} else {
+		return filepath.Join(s.logStateDir(ctlog.LogID), "healthchecks")
+	}
+}
+
+func (s *FilesystemState) NotifyHealthCheckFailure(ctx context.Context, ctlog *loglist.Log, info HealthCheckFailure) error {
+	textPath := filepath.Join(s.healthCheckDir(ctlog), healthCheckFilename())
+	environ := []string{
+		"EVENT=error",
+		"SUMMARY=" + info.Summary(),
+		"TEXT_FILENAME=" + textPath,
+	}
+	text := info.Text()
+	if err := writeTextFile(textPath, text, 0666); err != nil {
+		return fmt.Errorf("error saving text file: %w", err)
+	}
+	if err := s.notify(ctx, &notification{
+		environ: environ,
+		summary: info.Summary(),
+		text:    text,
+	}); err != nil {
+		return err
+	}
+	return nil
+}
+
+func (s *FilesystemState) NotifyError(ctx context.Context, ctlog *loglist.Log, err error) error {
+	if ctlog == nil {
+		log.Print(err)
+	} else {
+		log.Print(ctlog.GetMonitoringURL(), ": ", err)
+	}
+	return nil
+}

monitor/healthcheck.go

@@ -0,0 +1,138 @@
+// Copyright (C) 2023 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package monitor
+
+import (
+	"context"
+	"fmt"
+	"strings"
+	"time"
+
+	"software.sslmate.com/src/certspotter/cttypes"
+	"software.sslmate.com/src/certspotter/loglist"
+)
+
+func healthCheckFilename() string {
+	return time.Now().UTC().Format(time.RFC3339) + ".txt"
+}
+
+func healthCheckLog(ctx context.Context, config *Config, ctlog *loglist.Log) error {
+	state, err := config.State.LoadLogState(ctx, ctlog.LogID)
+	if err != nil {
+		return fmt.Errorf("error loading log state: %w", err)
+	} else if state == nil {
+		return nil
+	}
+
+	if time.Since(state.LastSuccess) < config.HealthCheckInterval {
+		return nil
+	}
+
+	sths, err := config.State.LoadSTHs(ctx, ctlog.LogID)
+	if err != nil {
+		return fmt.Errorf("error loading STHs: %w", err)
+	}
+
+	if len(sths) == 0 {
+		info := &StaleSTHInfo{
+			Log:         ctlog,
+			LastSuccess: state.LastSuccess,
+			LatestSTH:   state.VerifiedSTH,
+		}
+		if err := config.State.NotifyHealthCheckFailure(ctx, ctlog, info); err != nil {
+			return fmt.Errorf("error notifying about stale STH: %w", err)
+		}
+	} else {
+		info := &BacklogInfo{
+			Log:       ctlog,
+			LatestSTH: sths[len(sths)-1],
+			Position:  state.DownloadPosition.Size(),
+		}
+		if err := config.State.NotifyHealthCheckFailure(ctx, ctlog, info); err != nil {
+			return fmt.Errorf("error notifying about backlog: %w", err)
+		}
+	}
+
+	return nil
+}
+
+type HealthCheckFailure interface {
+	Summary() string
+	Text() string
+}
+
+type StaleSTHInfo struct {
+	Log         *loglist.Log
+	LastSuccess time.Time
+	LatestSTH   *cttypes.SignedTreeHead // may be nil
+}
+
+type BacklogInfo struct {
+	Log       *loglist.Log
+	LatestSTH *StoredSTH
+	Position  uint64
+}
+
+type StaleLogListInfo struct {
+	Source        string
+	LastSuccess   time.Time
+	LastError     string
+	LastErrorTime time.Time
+}
+
+func (e *BacklogInfo) Backlog() uint64 {
+	return e.LatestSTH.TreeSize - e.Position
+}
+
+func (e *StaleSTHInfo) Summary() string {
+	return fmt.Sprintf("Unable to contact %s since %s", e.Log.GetMonitoringURL(), e.LastSuccess)
+}
+func (e *BacklogInfo) Summary() string {
+	return fmt.Sprintf("Backlog of size %d from %s", e.Backlog(), e.Log.GetMonitoringURL())
+}
+func (e *StaleLogListInfo) Summary() string {
+	return fmt.Sprintf("Unable to retrieve log list since %s", e.LastSuccess)
+}
+
+func (e *StaleSTHInfo) Text() string {
+	text := new(strings.Builder)
+	fmt.Fprintf(text, "certspotter has been unable to contact %s since %s. Consequentially, certspotter may fail to notify you about certificates in this log.\n", e.Log.GetMonitoringURL(), e.LastSuccess)
+	fmt.Fprintf(text, "\n")
+	fmt.Fprintf(text, "For details, see certspotter's stderr output.\n")
+	fmt.Fprintf(text, "\n")
+	if e.LatestSTH != nil {
+		fmt.Fprintf(text, "Latest known log size = %d\n", e.LatestSTH.TreeSize)
+	} else {
+		fmt.Fprintf(text, "Latest known log size = none\n")
+	}
+	return text.String()
+}
+func (e *BacklogInfo) Text() string {
+	text := new(strings.Builder)
+	fmt.Fprintf(text, "certspotter has been unable to download entries from %s in a timely manner. Consequentially, certspotter may be slow to notify you about certificates in this log.\n", e.Log.GetMonitoringURL())
+	fmt.Fprintf(text, "\n")
+	fmt.Fprintf(text, "For more details, see certspotter's stderr output.\n")
+	fmt.Fprintf(text, "\n")
+	fmt.Fprintf(text, "Current log size = %d (as of %s)\n", e.LatestSTH.TreeSize, e.LatestSTH.StoredAt)
+	fmt.Fprintf(text, "Current position = %d\n", e.Position)
+	fmt.Fprintf(text, "         Backlog = %d\n", e.Backlog())
+	return text.String()
+}
+func (e *StaleLogListInfo) Text() string {
+	text := new(strings.Builder)
+	fmt.Fprintf(text, "certspotter has been unable to retrieve the log list from %s since %s.\n", e.Source, e.LastSuccess)
+	fmt.Fprintf(text, "\n")
+	fmt.Fprintf(text, "Last error (at %s): %s\n", e.LastErrorTime, e.LastError)
+	fmt.Fprintf(text, "\n")
+	fmt.Fprintf(text, "Consequentially, certspotter may not be monitoring all logs, and might fail to detect certificates.\n")
+	return text.String()
+}
+
+// TODO-3: make the errors more actionable

monitor/loglist.go

@@ -0,0 +1,45 @@
+// Copyright (C) 2023 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package monitor
+
+import (
+	"context"
+	"fmt"
+	"software.sslmate.com/src/certspotter/cttypes"
+	"software.sslmate.com/src/certspotter/loglist"
+)
+
+type LogID = cttypes.LogID
+
+func getLogList(ctx context.Context, source string, token *loglist.ModificationToken) (map[LogID]*loglist.Log, *loglist.ModificationToken, error) {
+	list, newToken, err := loglist.LoadIfModified(ctx, source, token)
+	if err != nil {
+		return nil, nil, err
+	}
+
+	logs := make(map[LogID]*loglist.Log)
+	for operatorIndex := range list.Operators {
+		for logIndex := range list.Operators[operatorIndex].Logs {
+			log := &list.Operators[operatorIndex].Logs[logIndex]
+			if _, exists := logs[log.LogID]; exists {
+				return nil, nil, fmt.Errorf("log list contains more than one entry with ID %s", log.LogID.Base64String())
+			}
+			logs[log.LogID] = log
+		}
+		for logIndex := range list.Operators[operatorIndex].TiledLogs {
+			log := &list.Operators[operatorIndex].TiledLogs[logIndex]
+			if _, exists := logs[log.LogID]; exists {
+				return nil, nil, fmt.Errorf("log list contains more than one entry with ID %s", log.LogID.Base64String())
+			}
+			logs[log.LogID] = log
+		}
+	}
+	return logs, newToken, nil
+}

monitor/mailutils.go

@@ -0,0 +1,34 @@
+// Copyright (C) 2023 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package monitor
+
+import (
+	"crypto/rand"
+	"encoding/hex"
+	"os"
+)
+
+const mailDateFormat = "Mon, 2 Jan 2006 15:04:05 -0700"
+
+func generateMessageID() string {
+	var randomBytes [16]byte
+	if _, err := rand.Read(randomBytes[:]); err != nil {
+		panic(err)
+	}
+	return hex.EncodeToString(randomBytes[:]) + "@selfhosted.certspotter.org"
+}
+
+func sendmailPath() string {
+	if envVar := os.Getenv("SENDMAIL_PATH"); envVar != "" {
+		return envVar
+	} else {
+		return "/usr/sbin/sendmail"
+	}
+}

monitor/monitor.go

@@ -0,0 +1,532 @@
+// Copyright (C) 2025 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package monitor
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"golang.org/x/sync/errgroup"
+	"log"
+	mathrand "math/rand/v2"
+	"net/url"
+	"slices"
+	"time"
+
+	"software.sslmate.com/src/certspotter/ctclient"
+	"software.sslmate.com/src/certspotter/ctcrypto"
+	"software.sslmate.com/src/certspotter/cttypes"
+	"software.sslmate.com/src/certspotter/loglist"
+	"software.sslmate.com/src/certspotter/merkletree"
+	"software.sslmate.com/src/certspotter/sequencer"
+)
+
+const (
+	getSTHInterval = 5 * time.Minute
+)
+
+func downloadJobSize(ctlog *loglist.Log) uint64 {
+	if ctlog.IsStaticCTAPI() {
+		return ctclient.StaticTileWidth
+	} else if ctlog.CertspotterDownloadSize != 0 {
+		return uint64(ctlog.CertspotterDownloadSize)
+	} else {
+		return 1000
+	}
+}
+
+func downloadWorkers(ctlog *loglist.Log) int {
+	if ctlog.CertspotterDownloadJobs != 0 {
+		return ctlog.CertspotterDownloadJobs
+	} else {
+		return 1
+	}
+}
+
+type verifyEntriesError struct {
+	sth             *cttypes.SignedTreeHead
+	entriesRootHash merkletree.Hash
+}
+
+func (e *verifyEntriesError) Error() string {
+	return fmt.Sprintf("error verifying at tree size %d: the STH root hash (%x) does not match the entries returned by the log (%x)", e.sth.TreeSize, e.sth.RootHash, e.entriesRootHash)
+}
+
+func withRetry(ctx context.Context, config *Config, ctlog *loglist.Log, maxRetries int, f func() error) error {
+	minSleep := 1 * time.Second
+	numRetries := 0
+	for ctx.Err() == nil {
+		err := f()
+		if err == nil || errors.Is(err, context.Canceled) {
+			return err
+		}
+		if maxRetries != -1 && numRetries >= maxRetries {
+			return fmt.Errorf("%w (retried %d times)", err, numRetries)
+		}
+		recordError(ctx, config, ctlog, err)
+		sleepTime := minSleep + mathrand.N(minSleep)
+		if err := sleep(ctx, sleepTime); err != nil {
+			return err
+		}
+		minSleep = min(minSleep*2, 5*time.Minute)
+		numRetries++
+	}
+	return ctx.Err()
+}
+
+func getEntriesFull(ctx context.Context, client ctclient.Log, startInclusive, endExclusive uint64) ([]ctclient.Entry, error) {
+	allEntries := make([]ctclient.Entry, 0, endExclusive-startInclusive)
+	for startInclusive < endExclusive {
+		entries, err := client.GetEntries(ctx, startInclusive, endExclusive-1)
+		if err != nil {
+			return nil, err
+		}
+		allEntries = append(allEntries, entries...)
+		startInclusive += uint64(len(entries))
+	}
+	return allEntries, nil
+}
+
+func getAndVerifySTH(ctx context.Context, ctlog *loglist.Log, client ctclient.Log) (*cttypes.SignedTreeHead, string, error) {
+	sth, url, err := client.GetSTH(ctx)
+	if err != nil {
+		return nil, "", err
+	}
+	if err := ctcrypto.PublicKey(ctlog.Key).Verify(ctcrypto.SignatureInputForSTH(sth), sth.Signature); err != nil {
+		return nil, "", fmt.Errorf("STH has invalid signature: %w", err)
+	}
+	return sth, url, nil
+}
+
+type logClient struct {
+	config *Config
+	log    *loglist.Log
+	client ctclient.Log
+}
+
+func (client *logClient) GetSTH(ctx context.Context) (sth *cttypes.SignedTreeHead, url string, err error) {
+	err = withRetry(ctx, client.config, client.log, -1, func() error {
+		sth, url, err = getAndVerifySTH(ctx, client.log, client.client)
+		return err
+	})
+	return
+}
+func (client *logClient) GetRoots(ctx context.Context) (roots [][]byte, err error) {
+	err = withRetry(ctx, client.config, client.log, -1, func() error {
+		roots, err = client.client.GetRoots(ctx)
+		return err
+	})
+	return
+}
+func (client *logClient) GetEntries(ctx context.Context, startInclusive, endInclusive uint64) (entries []ctclient.Entry, err error) {
+	err = withRetry(ctx, client.config, client.log, -1, func() error {
+		entries, err = client.client.GetEntries(ctx, startInclusive, endInclusive)
+		return err
+	})
+	return
+}
+func (client *logClient) ReconstructTree(ctx context.Context, sth *cttypes.SignedTreeHead) (tree *merkletree.CollapsedTree, err error) {
+	err = withRetry(ctx, client.config, client.log, -1, func() error {
+		tree, err = client.client.ReconstructTree(ctx, sth)
+		return err
+	})
+	return
+}
+
+type issuerGetter struct {
+	config    *Config
+	log       *loglist.Log
+	logGetter ctclient.IssuerGetter
+}
+
+func (ig *issuerGetter) GetIssuer(ctx context.Context, fingerprint *[32]byte) ([]byte, error) {
+	if issuer, err := ig.config.State.LoadIssuer(ctx, fingerprint); err != nil {
+		log.Printf("error loading cached issuer %x (issuer will be retrieved from log instead): %s", *fingerprint, err)
+	} else if issuer != nil {
+		return issuer, nil
+	}
+
+	var issuer []byte
+	if err := withRetry(ctx, ig.config, ig.log, 7, func() error {
+		var err error
+		issuer, err = ig.logGetter.GetIssuer(ctx, fingerprint)
+		return err
+	}); err != nil {
+		return nil, err
+	}
+
+	if err := ig.config.State.StoreIssuer(ctx, fingerprint, issuer); err != nil {
+		log.Printf("error caching issuer %x (issuer will be re-retrieved from log in the future): %s", *fingerprint, err)
+	}
+
+	return issuer, nil
+}
+
+func newLogClient(config *Config, ctlog *loglist.Log) (ctclient.Log, ctclient.IssuerGetter, error) {
+	switch {
+	case ctlog.IsRFC6962():
+		logURL, err := url.Parse(ctlog.URL)
+		if err != nil {
+			return nil, nil, fmt.Errorf("log has invalid URL: %w", err)
+		}
+		return &logClient{
+			config: config,
+			log:    ctlog,
+			client: &ctclient.RFC6962Log{URL: logURL},
+		}, nil, nil
+	case ctlog.IsStaticCTAPI():
+		submissionURL, err := url.Parse(ctlog.SubmissionURL)
+		if err != nil {
+			return nil, nil, fmt.Errorf("log has invalid submission URL: %w", err)
+		}
+		monitoringURL, err := url.Parse(ctlog.MonitoringURL)
+		if err != nil {
+			return nil, nil, fmt.Errorf("log has invalid monitoring URL: %w", err)
+		}
+		client := &ctclient.StaticLog{
+			SubmissionURL: submissionURL,
+			MonitoringURL: monitoringURL,
+			ID:            ctlog.LogID,
+		}
+		return &logClient{
+				config: config,
+				log:    ctlog,
+				client: client,
+			}, &issuerGetter{
+				config:    config,
+				log:       ctlog,
+				logGetter: client,
+			}, nil
+	default:
+		return nil, nil, fmt.Errorf("log uses unknown protocol")
+	}
+}
+
+func monitorLogContinously(ctx context.Context, config *Config, ctlog *loglist.Log) (returnedErr error) {
+	client, issuerGetter, err := newLogClient(config, ctlog)
+	if err != nil {
+		return err
+	}
+	if err := config.State.PrepareLog(ctx, ctlog.LogID); err != nil {
+		return fmt.Errorf("error preparing state: %w", err)
+	}
+	state, err := config.State.LoadLogState(ctx, ctlog.LogID)
+	if err != nil {
+		return fmt.Errorf("error loading log state: %w", err)
+	}
+	if state == nil {
+		if config.StartAtEnd {
+			sth, _, err := client.GetSTH(ctx)
+			if err != nil {
+				return err
+			}
+			tree, err := client.ReconstructTree(ctx, sth)
+			if err != nil {
+				return err
+			}
+			state = &LogState{
+				DownloadPosition: tree,
+				VerifiedPosition: tree,
+				VerifiedSTH:      sth,
+				LastSuccess:      time.Now(),
+			}
+		} else {
+			state = &LogState{
+				DownloadPosition: merkletree.EmptyCollapsedTree(),
+				VerifiedPosition: merkletree.EmptyCollapsedTree(),
+				VerifiedSTH:      nil,
+				LastSuccess:      time.Now(),
+			}
+		}
+		if config.Verbose {
+			log.Printf("brand new log %s (starting from %d)", ctlog.GetMonitoringURL(), state.DownloadPosition.Size())
+		}
+		if err := config.State.StoreLogState(ctx, ctlog.LogID, state); err != nil {
+			return fmt.Errorf("error storing log state: %w", err)
+		}
+	}
+
+	defer func() {
+		if config.Verbose {
+			log.Printf("saving state in defer for %s", ctlog.GetMonitoringURL())
+		}
+		storeCtx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
+		defer cancel()
+		if err := config.State.StoreLogState(storeCtx, ctlog.LogID, state); err != nil && returnedErr == nil {
+			returnedErr = fmt.Errorf("error storing log state: %w", err)
+		}
+	}()
+
+retry:
+	position := state.DownloadPosition.Size()
+
+	// generateBatchesWorker ==> downloadWorker ==> processWorker ==> saveStateWorker
+
+	batches := make(chan *batch, downloadWorkers(ctlog))
+	processedBatches := sequencer.New[batch](0, uint64(downloadWorkers(ctlog))*10)
+
+	group, gctx := errgroup.WithContext(ctx)
+	group.Go(func() error { return getSTHWorker(gctx, config, ctlog, client) })
+	group.Go(func() error { return generateBatchesWorker(gctx, config, ctlog, position, batches) })
+	for range downloadWorkers(ctlog) {
+		downloadedBatches := make(chan *batch, 1)
+		group.Go(func() error { return downloadWorker(gctx, config, ctlog, client, batches, downloadedBatches) })
+		group.Go(func() error {
+			return processWorker(gctx, config, ctlog, issuerGetter, downloadedBatches, processedBatches)
+		})
+	}
+	group.Go(func() error { return saveStateWorker(gctx, config, ctlog, state, processedBatches) })
+
+	err = group.Wait()
+	if verifyErr := (*verifyEntriesError)(nil); errors.As(err, &verifyErr) {
+		recordError(ctx, config, ctlog, verifyErr)
+		state.rewindDownloadPosition()
+		if err := config.State.StoreLogState(ctx, ctlog.LogID, state); err != nil {
+			return fmt.Errorf("error storing log state: %w", err)
+		}
+		if err := sleep(ctx, 5*time.Minute); err != nil {
+			return err
+		}
+		goto retry
+	}
+	return err
+}
+
+func getSTHWorker(ctx context.Context, config *Config, ctlog *loglist.Log, client ctclient.Log) error {
+	for ctx.Err() == nil {
+		sth, _, err := client.GetSTH(ctx)
+		if err != nil {
+			return err
+		}
+		if err := config.State.StoreSTH(ctx, ctlog.LogID, sth); err != nil {
+			return fmt.Errorf("error storing STH: %w", err)
+		}
+		if err := sleep(ctx, getSTHInterval); err != nil {
+			return err
+		}
+	}
+	return ctx.Err()
+}
+
+type batch struct {
+	number     uint64
+	begin, end uint64
+	sths       []*StoredSTH     // STHs with sizes in range [begin,end], sorted by TreeSize
+	entries    []ctclient.Entry // in range [begin,end)
+}
+
+func generateBatchesWorker(ctx context.Context, config *Config, ctlog *loglist.Log, position uint64, batches chan<- *batch) error {
+	ticker := time.NewTicker(15 * time.Second)
+	var number uint64
+	for ctx.Err() == nil {
+		sths, err := config.State.LoadSTHs(ctx, ctlog.LogID)
+		if err != nil {
+			return fmt.Errorf("error loading STHs: %w", err)
+		}
+		for len(sths) > 0 && sths[0].TreeSize < position {
+			// TODO-4: audit sths[0] against log's verified STH
+			if err := config.State.RemoveSTH(ctx, ctlog.LogID, &sths[0].SignedTreeHead); err != nil {
+				return fmt.Errorf("error removing STH: %w", err)
+			}
+			sths = sths[1:]
+		}
+		position, number, err = generateBatches(ctx, ctlog, position, number, sths, batches)
+		if err != nil {
+			return err
+		}
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		case <-ticker.C:
+		}
+	}
+	return ctx.Err()
+}
+
+// return the time at which the right-most tile indicated by sths was discovered
+func tileDiscoveryTime(sths []*StoredSTH) time.Time {
+	largestSTH, sths := sths[len(sths)-1], sths[:len(sths)-1]
+	tileNumber := largestSTH.TreeSize / ctclient.StaticTileWidth
+	storedAt := largestSTH.StoredAt
+	for _, sth := range slices.Backward(sths) {
+		if sth.TreeSize/ctclient.StaticTileWidth != tileNumber {
+			break
+		}
+		if sth.StoredAt.Before(storedAt) {
+			storedAt = sth.StoredAt
+		}
+	}
+	return storedAt
+}
+
+func generateBatches(ctx context.Context, ctlog *loglist.Log, position uint64, number uint64, sths []*StoredSTH, batches chan<- *batch) (uint64, uint64, error) {
+	downloadJobSize := downloadJobSize(ctlog)
+	if len(sths) == 0 {
+		return position, number, nil
+	}
+	largestSTH := sths[len(sths)-1]
+	treeSize := largestSTH.TreeSize
+	if ctlog.IsStaticCTAPI() && time.Since(tileDiscoveryTime(sths)) < 5*time.Minute {
+		// Round down to the tile boundary to avoid downloading a partial tile that was recently discovered
+		// In a future invocation of this function, either enough time will have passed that this code path will be skipped, or the log will have grown and treeSize will be rounded to a larger tile boundary
+		treeSize -= treeSize % ctclient.StaticTileWidth
+		if treeSize < position {
+			// This can arise with a brand new log when config.StartAtEnd is true
+			return position, number, nil
+		}
+	}
+	for {
+		batch := &batch{
+			number: number,
+			begin:  position,
+			end:    min(treeSize, (position/downloadJobSize+1)*downloadJobSize),
+		}
+		for len(sths) > 0 && sths[0].TreeSize <= batch.end {
+			batch.sths = append(batch.sths, sths[0])
+			sths = sths[1:]
+		}
+		select {
+		case <-ctx.Done():
+			return position, number, ctx.Err()
+		default:
+		}
+		select {
+		case <-ctx.Done():
+			return position, number, ctx.Err()
+		case batches <- batch:
+		}
+		number++
+		if position == batch.end {
+			break
+		}
+		position = batch.end
+	}
+	return position, number, nil
+}
+
+func downloadWorker(ctx context.Context, config *Config, ctlog *loglist.Log, client ctclient.Log, batchesIn <-chan *batch, batchesOut chan<- *batch) error {
+	for {
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		default:
+		}
+		var batch *batch
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		case batch = <-batchesIn:
+		}
+
+		entries, err := getEntriesFull(ctx, client, batch.begin, batch.end)
+		if err != nil {
+			return err
+		}
+		batch.entries = entries
+
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		default:
+		}
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		case batchesOut <- batch:
+		}
+	}
+}
+
+func processWorker(ctx context.Context, config *Config, ctlog *loglist.Log, issuerGetter ctclient.IssuerGetter, batchesIn <-chan *batch, batchesOut *sequencer.Channel[batch]) error {
+	for {
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		default:
+		}
+		var batch *batch
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		case batch = <-batchesIn:
+		}
+		for offset, entry := range batch.entries {
+			index := batch.begin + uint64(offset)
+			if err := processLogEntry(ctx, config, issuerGetter, &LogEntry{
+				Entry: entry,
+				Index: index,
+				Log:   ctlog,
+			}); err != nil {
+				return fmt.Errorf("error processing entry %d: %w", index, err)
+			}
+		}
+		if err := batchesOut.Add(ctx, batch.number, batch); err != nil {
+			return err
+		}
+	}
+}
+
+func saveStateWorker(ctx context.Context, config *Config, ctlog *loglist.Log, state *LogState, batchesIn *sequencer.Channel[batch]) error {
+	for {
+		batch, err := batchesIn.Next(ctx)
+		if err != nil {
+			return err
+		}
+		if batch.begin != state.DownloadPosition.Size() {
+			panic(fmt.Errorf("saveStateWorker: expected batch to start at %d but got %d instead", state.DownloadPosition.Size(), batch.begin))
+		}
+		rootHash := state.DownloadPosition.CalculateRoot()
+		for {
+			for len(batch.sths) > 0 && batch.sths[0].TreeSize == state.DownloadPosition.Size() {
+				sth := batch.sths[0]
+				batch.sths = batch.sths[1:]
+				if sth.RootHash != rootHash {
+					return &verifyEntriesError{
+						sth:             &sth.SignedTreeHead,
+						entriesRootHash: rootHash,
+					}
+				}
+				state.advanceVerifiedPosition()
+				state.LastSuccess = sth.StoredAt
+				state.VerifiedSTH = &sth.SignedTreeHead
+				if err := config.State.StoreLogState(ctx, ctlog.LogID, state); err != nil {
+					return fmt.Errorf("error storing log state: %w", err)
+				}
+				// don't remove the STH until state has been durably stored
+				if err := config.State.RemoveSTH(ctx, ctlog.LogID, &sth.SignedTreeHead); err != nil {
+					return fmt.Errorf("error removing verified STH: %w", err)
+				}
+			}
+			if len(batch.entries) == 0 {
+				break
+			}
+			entry := batch.entries[0]
+			batch.entries = batch.entries[1:]
+			leafHash := merkletree.HashLeaf(entry.LeafInput())
+			state.DownloadPosition.Add(leafHash)
+			rootHash = state.DownloadPosition.CalculateRoot()
+		}
+
+		if err := config.State.StoreLogState(ctx, ctlog.LogID, state); err != nil {
+			return fmt.Errorf("error storing log state: %w", err)
+		}
+	}
+}
+
+func sleep(ctx context.Context, duration time.Duration) error {
+	timer := time.NewTimer(duration)
+	defer timer.Stop()
+	select {
+	case <-ctx.Done():
+		return ctx.Err()
+	case <-timer.C:
+		return nil
+	}
+}

monitor/notify.go

@@ -0,0 +1,156 @@
+// Copyright (C) 2023 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package monitor
+
+import (
+	"bytes"
+	"context"
+	"errors"
+	"fmt"
+	"io/fs"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strings"
+	"sync"
+	"time"
+)
+
+var stdoutMu sync.Mutex
+
+type notification struct {
+	environ []string
+	summary string
+	text    string
+}
+
+func (s *FilesystemState) notify(ctx context.Context, notif *notification) error {
+	if s.Stdout {
+		writeToStdout(notif)
+	}
+
+	if len(s.Email) > 0 {
+		if err := sendEmail(ctx, s.Email, notif); err != nil {
+			return err
+		}
+	}
+
+	if s.Script != "" {
+		if err := execScript(ctx, s.Script, notif); err != nil {
+			return err
+		}
+	}
+
+	if s.ScriptDir != "" {
+		if err := execScriptDir(ctx, s.ScriptDir, notif); err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+func writeToStdout(notif *notification) {
+	stdoutMu.Lock()
+	defer stdoutMu.Unlock()
+	os.Stdout.WriteString(notif.text + "\n")
+}
+
+func sendEmail(ctx context.Context, to []string, notif *notification) error {
+	stdin := new(bytes.Buffer)
+	stderr := new(bytes.Buffer)
+
+	from := os.Getenv("EMAIL")
+
+	if from != "" {
+		fmt.Fprintf(stdin, "From: %s\n", from)
+	}
+	fmt.Fprintf(stdin, "To: %s\n", strings.Join(to, ", "))
+	fmt.Fprintf(stdin, "Subject: [certspotter] %s\n", notif.summary)
+	fmt.Fprintf(stdin, "Date: %s\n", time.Now().Format(mailDateFormat))
+	fmt.Fprintf(stdin, "Message-ID: <%s>\n", generateMessageID())
+	fmt.Fprintf(stdin, "Mime-Version: 1.0\n")
+	fmt.Fprintf(stdin, "Content-Type: text/plain; charset=US-ASCII\n")
+	fmt.Fprintf(stdin, "X-Mailer: certspotter\n")
+	fmt.Fprintf(stdin, "\n")
+	fmt.Fprint(stdin, notif.text)
+
+	args := []string{"-i"}
+	if from != "" {
+		args = append(args, "-f", from)
+	}
+	args = append(args, "--")
+	args = append(args, to...)
+
+	sendmail := exec.CommandContext(ctx, sendmailPath(), args...)
+	sendmail.Stdin = stdin
+	sendmail.Stderr = stderr
+
+	if err := sendmail.Run(); err == nil {
+		return nil
+	} else if ctx.Err() != nil {
+		return ctx.Err()
+	} else if exitErr, isExitError := err.(*exec.ExitError); isExitError && exitErr.Exited() {
+		return fmt.Errorf("error sending email to %v: sendmail failed with exit code %d and error %q", to, exitErr.ExitCode(), strings.TrimSpace(stderr.String()))
+	} else {
+		return fmt.Errorf("error sending email to %v: %w", to, err)
+	}
+}
+
+func execScript(ctx context.Context, scriptName string, notif *notification) error {
+	stderr := new(bytes.Buffer)
+
+	cmd := exec.CommandContext(ctx, scriptName)
+	cmd.Env = os.Environ()
+	cmd.Env = append(cmd.Env, notif.environ...)
+	cmd.Stderr = stderr
+
+	if err := cmd.Run(); err == nil {
+		return nil
+	} else if ctx.Err() != nil {
+		return ctx.Err()
+	} else if exitErr, isExitError := err.(*exec.ExitError); isExitError && exitErr.Exited() {
+		return fmt.Errorf("script %q exited with code %d and error %q", scriptName, exitErr.ExitCode(), strings.TrimSpace(stderr.String()))
+	} else if isExitError {
+		return fmt.Errorf("script %q terminated by signal with error %q", scriptName, strings.TrimSpace(stderr.String()))
+	} else {
+		return fmt.Errorf("error executing script: %w", err)
+	}
+}
+
+func execScriptDir(ctx context.Context, dirPath string, notif *notification) error {
+	dirents, err := os.ReadDir(dirPath)
+	if errors.Is(err, fs.ErrNotExist) {
+		return nil
+	} else if err != nil {
+		return fmt.Errorf("error executing scripts in directory %q: %w", dirPath, err)
+	}
+	for _, dirent := range dirents {
+		if strings.HasPrefix(dirent.Name(), ".") {
+			continue
+		}
+		scriptPath := filepath.Join(dirPath, dirent.Name())
+		info, err := os.Stat(scriptPath)
+		if errors.Is(err, fs.ErrNotExist) {
+			continue
+		} else if err != nil {
+			return fmt.Errorf("error executing %q in directory %q: %w", dirent.Name(), dirPath, err)
+		} else if info.Mode().IsRegular() && isExecutable(info.Mode()) {
+			if err := execScript(ctx, scriptPath, notif); err != nil {
+				return err
+			}
+		}
+	}
+	return nil
+}
+
+func isExecutable(mode os.FileMode) bool {
+	return mode&0111 != 0
+}

monitor/process.go

@@ -0,0 +1,138 @@
+// Copyright (C) 2025 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package monitor
+
+import (
+	"context"
+	"crypto/sha256"
+	"errors"
+	"fmt"
+
+	"software.sslmate.com/src/certspotter"
+	"software.sslmate.com/src/certspotter/ctclient"
+	"software.sslmate.com/src/certspotter/cttypes"
+	"software.sslmate.com/src/certspotter/loglist"
+)
+
+type LogEntry struct {
+	ctclient.Entry
+	Index uint64
+	Log   *loglist.Log
+}
+
+func processLogEntry(ctx context.Context, config *Config, issuerGetter ctclient.IssuerGetter, entry *LogEntry) error {
+	leaf, err := cttypes.ParseLeafInput(entry.LeafInput())
+	if err != nil {
+		return processMalformedLogEntry(ctx, config, entry, fmt.Errorf("error parsing Merkle Tree Leaf: %w", err))
+	}
+	switch leaf.TimestampedEntry.EntryType {
+	case cttypes.X509EntryType:
+		return processX509LogEntry(ctx, config, issuerGetter, entry, leaf.TimestampedEntry.SignedEntryASN1Cert)
+	case cttypes.PrecertEntryType:
+		return processPrecertLogEntry(ctx, config, issuerGetter, entry, leaf.TimestampedEntry.SignedEntryPreCert)
+	default:
+		return processMalformedLogEntry(ctx, config, entry, fmt.Errorf("unknown log entry type %d", leaf.TimestampedEntry.EntryType))
+	}
+}
+
+func processX509LogEntry(ctx context.Context, config *Config, issuerGetter ctclient.IssuerGetter, entry *LogEntry, cert *cttypes.ASN1Cert) error {
+	certInfo, err := certspotter.MakeCertInfoFromRawCert(*cert)
+	if err != nil {
+		return processMalformedLogEntry(ctx, config, entry, fmt.Errorf("error parsing X.509 certificate: %w", err))
+	}
+	if precertTBS, err := certspotter.ReconstructPrecertTBS(certInfo.TBS); err == nil {
+		certInfo.TBS = precertTBS
+	} else {
+		return processMalformedLogEntry(ctx, config, entry, fmt.Errorf("error reconstructing precertificate TBSCertificate: %w", err))
+	}
+
+	getChain := func(ctx context.Context) ([]cttypes.ASN1Cert, error) {
+		var (
+			chain = []cttypes.ASN1Cert{*cert}
+			errs  = []error{}
+		)
+		if issuers, err := entry.GetChain(ctx, issuerGetter); err == nil {
+			chain = append(chain, issuers...)
+		} else {
+			errs = append(errs, err)
+		}
+		return chain, errors.Join(errs...)
+	}
+	return processCertificate(ctx, config, entry, certInfo, getChain)
+}
+
+func processPrecertLogEntry(ctx context.Context, config *Config, issuerGetter ctclient.IssuerGetter, entry *LogEntry, precert *cttypes.PreCert) error {
+	certInfo, err := certspotter.MakeCertInfoFromRawTBS(precert.TBSCertificate)
+	if err != nil {
+		return processMalformedLogEntry(ctx, config, entry, fmt.Errorf("error parsing precert TBSCertificate: %w", err))
+	}
+	precertBytes, err := entry.Precertificate()
+	if err != nil {
+		return processMalformedLogEntry(ctx, config, entry, fmt.Errorf("error getting precert entry's precertificate: %w", err))
+	}
+
+	getChain := func(ctx context.Context) ([]cttypes.ASN1Cert, error) {
+		var (
+			chain = []cttypes.ASN1Cert{precertBytes}
+			errs  = []error{}
+		)
+		if issuers, err := entry.GetChain(ctx, issuerGetter); err == nil {
+			chain = append(chain, issuers...)
+		} else {
+			errs = append(errs, err)
+		}
+		if _, err := certspotter.ValidatePrecert(precertBytes, precert.TBSCertificate); err != nil {
+			errs = append(errs, fmt.Errorf("precertificate in extra_data does not match TBSCertificate in leaf_input: %w", err))
+		}
+		return chain, errors.Join(errs...)
+	}
+	return processCertificate(ctx, config, entry, certInfo, getChain)
+}
+
+func processCertificate(ctx context.Context, config *Config, entry *LogEntry, certInfo *certspotter.CertInfo, getChain func(context.Context) ([]cttypes.ASN1Cert, error)) error {
+	identifiers, err := certInfo.ParseIdentifiers()
+	if err != nil {
+		return processMalformedLogEntry(ctx, config, entry, err)
+	}
+	matched, watchItem := config.WatchList.Matches(identifiers)
+	if !matched {
+		return nil
+	}
+
+	chain, chainErr := getChain(ctx)
+	if errors.Is(chainErr, context.Canceled) {
+		return chainErr
+	}
+
+	cert := &DiscoveredCert{
+		WatchItem:    watchItem,
+		LogEntry:     entry,
+		Info:         certInfo,
+		Chain:        chain,
+		ChainError:   chainErr,
+		TBSSHA256:    sha256.Sum256(certInfo.TBS.Raw),
+		SHA256:       sha256.Sum256(chain[0]),
+		PubkeySHA256: sha256.Sum256(certInfo.TBS.PublicKey.FullBytes),
+		Identifiers:  identifiers,
+	}
+
+	if err := config.State.NotifyCert(ctx, cert); err != nil {
+		return fmt.Errorf("error notifying about certificate %x: %w", cert.SHA256, err)
+	}
+
+	return nil
+}
+
+func processMalformedLogEntry(ctx context.Context, config *Config, entry *LogEntry, parseError error) error {
+	if err := config.State.NotifyMalformedEntry(ctx, entry, parseError); err != nil {
+		return fmt.Errorf("error notifying about malformed log entry %d in %s (%q): %w", entry.Index, entry.Log.GetMonitoringURL(), parseError, err)
+	}
+	return nil
+}

monitor/state.go

@@ -0,0 +1,88 @@
+// Copyright (C) 2024 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package monitor
+
+import (
+	"context"
+	"time"
+
+	"software.sslmate.com/src/certspotter/cttypes"
+	"software.sslmate.com/src/certspotter/loglist"
+	"software.sslmate.com/src/certspotter/merkletree"
+)
+
+type LogState struct {
+	DownloadPosition *merkletree.CollapsedTree `json:"download_position"`
+	VerifiedPosition *merkletree.CollapsedTree `json:"verified_position"`
+	VerifiedSTH      *cttypes.SignedTreeHead   `json:"verified_sth"`
+	LastSuccess      time.Time                 `json:"last_success"`
+}
+
+func (state *LogState) rewindDownloadPosition() {
+	position := state.VerifiedPosition.Clone()
+	state.DownloadPosition = &position
+}
+
+func (state *LogState) advanceVerifiedPosition() {
+	position := state.DownloadPosition.Clone()
+	state.VerifiedPosition = &position
+}
+
+// Methods are safe to call concurrently.
+type StateProvider interface {
+	// Initialize the state.  Called before any other method in this interface.
+	// Idempotent: returns nil if the state is already initialized.
+	Prepare(context.Context) error
+
+	// Initialize the state for the given log.  Called before any other method
+	// with the log ID.  Idempotent: returns nil if log state already initialized.
+	PrepareLog(context.Context, LogID) error
+
+	// Store log state for retrieval by LoadLogState.
+	StoreLogState(context.Context, LogID, *LogState) error
+
+	// Load log state that was previously stored with StoreLogState.
+	// Returns nil, nil if StoreLogState has not been called yet for this log.
+	LoadLogState(context.Context, LogID) (*LogState, error)
+
+	// Store STH for retrieval by LoadSTHs.  If an STH with the same
+	// timestamp and root hash is already stored, this STH can be ignored.
+	StoreSTH(context.Context, LogID, *cttypes.SignedTreeHead) error
+
+	// Load all STHs for this log previously stored with StoreSTH.
+	// The returned slice must be sorted by tree size.
+	LoadSTHs(context.Context, LogID) ([]*StoredSTH, error)
+
+	// Remove an STH so it is no longer returned by LoadSTHs.
+	RemoveSTH(context.Context, LogID, *cttypes.SignedTreeHead) error
+
+	// Store a DER-encoded issuer certificate with the given fingerprint for
+	// retrieval by LoadIssuer.  Returns nil if the issuer has already been stored.
+	StoreIssuer(context.Context, *[32]byte, []byte) error
+
+	// Retrieve a DER-encoded issuer certificate previously stored with StoreIssuer.
+	// Returns nil, nil if this issuer certificate has not been stored.
+	LoadIssuer(context.Context, *[32]byte) ([]byte, error)
+
+	// Called when a certificate matching the watch list is discovered.
+	NotifyCert(context.Context, *DiscoveredCert) error
+
+	// Called when certspotter fails to parse a log entry.
+	NotifyMalformedEntry(context.Context, *LogEntry, error) error
+
+	// Called when a health check fails.  The log is nil if the
+	// feailure is not associated with a log.
+	NotifyHealthCheckFailure(context.Context, *loglist.Log, HealthCheckFailure) error
+
+	// Called when a non-fatal error occurs.  The log is nil if the error is
+	// not associated with a log.  Note that most errors are transient, and
+	// certspotter will retry the failed operation later.
+	NotifyError(context.Context, *loglist.Log, error) error
+}

monitor/statedir.go

@@ -0,0 +1,167 @@
+// Copyright (C) 2023 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package monitor
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io/fs"
+	"os"
+	"path/filepath"
+	"software.sslmate.com/src/certspotter/cttypes"
+	"software.sslmate.com/src/certspotter/merkletree"
+	"strconv"
+	"strings"
+	"time"
+)
+
+func readVersion(stateDir string) (int, error) {
+	path := filepath.Join(stateDir, "version")
+
+	fileBytes, err := os.ReadFile(path)
+	if errors.Is(err, fs.ErrNotExist) {
+		if fileExists(filepath.Join(stateDir, "evidence")) {
+			return 0, nil
+		} else {
+			return -1, nil
+		}
+	} else if err != nil {
+		return -1, err
+	}
+
+	version, err := strconv.Atoi(strings.TrimSpace(string(fileBytes)))
+	if err != nil {
+		return -1, fmt.Errorf("version file %q is malformed: %w", path, err)
+	}
+
+	return version, nil
+}
+
+func writeVersion(stateDir string) error {
+	return writeFile(filepath.Join(stateDir, "version"), []byte{'2', '\n'}, 0666)
+}
+
+func migrateLogStateDirV1(dir string) error {
+	var sth cttypes.SignedTreeHead
+	var tree merkletree.CollapsedTree
+
+	sthPath := filepath.Join(dir, "sth.json")
+	sthData, err := os.ReadFile(sthPath)
+	if errors.Is(err, fs.ErrNotExist) {
+		return nil
+	} else if err != nil {
+		return err
+	}
+
+	treePath := filepath.Join(dir, "tree.json")
+	treeData, err := os.ReadFile(treePath)
+	if errors.Is(err, fs.ErrNotExist) {
+		return nil
+	} else if err != nil {
+		return err
+	}
+
+	if err := json.Unmarshal(sthData, &sth); err != nil {
+		return fmt.Errorf("error unmarshaling %s: %w", sthPath, err)
+	}
+	if err := json.Unmarshal(treeData, &tree); err != nil {
+		return fmt.Errorf("error unmarshaling %s: %w", treePath, err)
+	}
+
+	stateFile := LogState{
+		DownloadPosition: &tree,
+		VerifiedPosition: &tree,
+		VerifiedSTH:      &sth,
+		LastSuccess:      time.Now(),
+	}
+	if err := writeJSONFile(filepath.Join(dir, "state.json"), stateFile, 0666); err != nil {
+		return err
+	}
+
+	if err := os.Remove(sthPath); err != nil {
+		return err
+	}
+	if err := os.Remove(treePath); err != nil {
+		return err
+	}
+	return nil
+}
+
+func migrateStateDirV1(stateDir string) error {
+	if lockfile := filepath.Join(stateDir, "lock"); fileExists(lockfile) {
+		return fmt.Errorf("directory is locked by another instance of certspotter; remove %s if this is not the case", lockfile)
+	}
+
+	if logDirs, err := os.ReadDir(filepath.Join(stateDir, "logs")); err == nil {
+		for _, logDir := range logDirs {
+			if strings.HasPrefix(logDir.Name(), ".") || !logDir.IsDir() {
+				continue
+			}
+			if err := migrateLogStateDirV1(filepath.Join(stateDir, "logs", logDir.Name())); err != nil {
+				return fmt.Errorf("error migrating log state: %w", err)
+			}
+		}
+	} else if !errors.Is(err, fs.ErrNotExist) {
+		return err
+	}
+
+	if err := writeVersion(stateDir); err != nil {
+		return err
+	}
+
+	if err := os.Remove(filepath.Join(stateDir, "once")); err != nil && !errors.Is(err, fs.ErrNotExist) {
+		return err
+	}
+
+	return nil
+}
+
+func prepareStateDir(stateDir string) error {
+	if err := os.Mkdir(stateDir, 0777); err != nil && !errors.Is(err, fs.ErrExist) {
+		return err
+	}
+
+	if version, err := readVersion(stateDir); err != nil {
+		return err
+	} else if version == -1 {
+		if err := writeVersion(stateDir); err != nil {
+			return err
+		}
+	} else if version == 0 {
+		return fmt.Errorf("%s was created by a very old version of certspotter; run any version of certspotter after 0.2 and before 0.15.0 to upgrade this directory, or remove it to start from scratch", stateDir)
+	} else if version == 1 {
+		if err := migrateStateDirV1(stateDir); err != nil {
+			return err
+		}
+	} else if version > 2 {
+		return fmt.Errorf("%s was created by a newer version of certspotter; upgrade to the latest version of certspotter or remove this directory to start from scratch", stateDir)
+	}
+
+	for _, subdir := range []string{"certs", "logs", "healthchecks"} {
+		if err := os.Mkdir(filepath.Join(stateDir, subdir), 0777); err != nil && !errors.Is(err, fs.ErrExist) {
+			return err
+		}
+	}
+
+	return nil
+}
+
+func prepareCacheDir(cacheDir string) error {
+	if err := os.MkdirAll(cacheDir, 0777); err != nil {
+		return err
+	}
+	for _, subdir := range []string{"issuers"} {
+		if err := os.Mkdir(filepath.Join(cacheDir, subdir), 0777); err != nil && !errors.Is(err, fs.ErrExist) {
+			return err
+		}
+	}
+	return nil
+}

monitor/sthdir.go

@@ -0,0 +1,109 @@
+// Copyright (C) 2017, 2023 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package monitor
+
+import (
+	"cmp"
+	"crypto/sha256"
+	"encoding/base64"
+	"encoding/binary"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"io/fs"
+	"os"
+	"path/filepath"
+	"slices"
+	"software.sslmate.com/src/certspotter/cttypes"
+	"strconv"
+	"strings"
+	"time"
+)
+
+type StoredSTH struct {
+	cttypes.SignedTreeHead
+	StoredAt time.Time `json:"stored_at"` // time at which the STH was first stored
+}
+
+func loadSTHsFromDir(dirPath string) ([]*StoredSTH, error) {
+	entries, err := os.ReadDir(dirPath)
+	if errors.Is(err, fs.ErrNotExist) {
+		return []*StoredSTH{}, nil
+	} else if err != nil {
+		return nil, err
+	}
+	sths := make([]*StoredSTH, 0, len(entries))
+	for _, entry := range entries {
+		filename := entry.Name()
+		if strings.HasPrefix(filename, ".") || !strings.HasSuffix(filename, ".json") {
+			continue
+		}
+		sth, err := readSTHFile(filepath.Join(dirPath, filename))
+		if err != nil {
+			return nil, err
+		}
+		sths = append(sths, sth)
+	}
+	slices.SortFunc(sths, func(a, b *StoredSTH) int { return cmp.Compare(a.TreeSize, b.TreeSize) })
+	return sths, nil
+}
+
+func readSTHFile(filePath string) (*StoredSTH, error) {
+	file, err := os.Open(filePath)
+	if err != nil {
+		return nil, err
+	}
+	defer file.Close()
+
+	info, err := file.Stat()
+	if err != nil {
+		return nil, err
+	}
+	sth := &StoredSTH{
+		StoredAt: info.ModTime(),
+	}
+
+	fileBytes, err := io.ReadAll(file)
+	if err != nil {
+		return nil, fmt.Errorf("error reading %s: %w", filePath, err)
+	}
+	if err := json.Unmarshal(fileBytes, &sth.SignedTreeHead); err != nil {
+		return nil, fmt.Errorf("error parsing %s: %w", filePath, err)
+	}
+	return sth, nil
+}
+
+func storeSTHInDir(dirPath string, sth *cttypes.SignedTreeHead) error {
+	filePath := filepath.Join(dirPath, sthFilename(sth))
+	if fileExists(filePath) {
+		// If the file already exists, we don't want its mtime to change
+		// because StoredSTH.StoredAt needs to be the time the STH was *first* stored.
+		return nil
+	}
+	return writeJSONFile(filePath, sth, 0666)
+}
+
+func removeSTHFromDir(dirPath string, sth *cttypes.SignedTreeHead) error {
+	filePath := filepath.Join(dirPath, sthFilename(sth))
+	err := os.Remove(filePath)
+	if err != nil && !errors.Is(err, fs.ErrNotExist) {
+		return err
+	}
+	return nil
+}
+
+// generate a filename that uniquely identifies the STH (within the context of a particular log)
+func sthFilename(sth *cttypes.SignedTreeHead) string {
+	hasher := sha256.New()
+	binary.Write(hasher, binary.LittleEndian, sth.Timestamp)
+	hasher.Write(sth.RootHash[:])
+	return strconv.FormatUint(sth.TreeSize, 10) + "-" + base64.RawURLEncoding.EncodeToString(hasher.Sum(nil)) + ".json"
+}

monitor/watchlist.go

@@ -0,0 +1,135 @@
+// Copyright (C) 2016, 2023 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package monitor
+
+import (
+	"bufio"
+	"fmt"
+	"golang.org/x/net/idna"
+	"io"
+	"software.sslmate.com/src/certspotter"
+	"strings"
+)
+
+type WatchItem struct {
+	domain       []string
+	acceptSuffix bool
+}
+
+type WatchList []WatchItem
+
+func ParseWatchItem(str string) (WatchItem, error) {
+	fields := strings.Fields(str)
+	if len(fields) == 0 {
+		return WatchItem{}, fmt.Errorf("empty domain")
+	}
+	domain := fields[0]
+
+	for _, field := range fields[1:] {
+		switch {
+		case strings.HasPrefix(field, "valid_at:"):
+			// Ignore for backwards compatibility
+		default:
+			return WatchItem{}, fmt.Errorf("unknown parameter %q", field)
+		}
+	}
+
+	if domain == "." {
+		// "." as in root zone -> matches everything
+		return WatchItem{
+			domain:       []string{},
+			acceptSuffix: true,
+		}, nil
+	}
+
+	acceptSuffix := false
+	if strings.HasPrefix(domain, ".") {
+		acceptSuffix = true
+		domain = domain[1:]
+	}
+
+	asciiDomain, err := idna.ToASCII(strings.ToLower(strings.TrimRight(domain, ".")))
+	if err != nil {
+		return WatchItem{}, fmt.Errorf("invalid domain %q (%w)", domain, err)
+	}
+	return WatchItem{
+		domain:       strings.Split(asciiDomain, "."),
+		acceptSuffix: acceptSuffix,
+	}, nil
+}
+
+func ReadWatchList(reader io.Reader) (WatchList, error) {
+	items := make(WatchList, 0, 50)
+	scanner := bufio.NewScanner(reader)
+	lineNo := 0
+	for scanner.Scan() {
+		line := scanner.Text()
+		lineNo++
+		if line == "" || strings.HasPrefix(line, "#") {
+			continue
+		}
+		item, err := ParseWatchItem(line)
+		if err != nil {
+			return nil, fmt.Errorf("%w on line %d", err, lineNo)
+		}
+		items = append(items, item)
+	}
+	return items, scanner.Err()
+}
+
+func (item WatchItem) String() string {
+	if item.acceptSuffix {
+		return "." + strings.Join(item.domain, ".")
+	} else {
+		return strings.Join(item.domain, ".")
+	}
+}
+
+func (item WatchItem) matchesDNSName(dnsName []string) bool {
+	watchDomain := item.domain
+	for len(dnsName) > 0 && len(watchDomain) > 0 {
+		certLabel := dnsName[len(dnsName)-1]
+		watchLabel := watchDomain[len(watchDomain)-1]
+
+		if !dnsLabelMatches(certLabel, watchLabel) {
+			return false
+		}
+
+		dnsName = dnsName[:len(dnsName)-1]
+		watchDomain = watchDomain[:len(watchDomain)-1]
+	}
+	return len(watchDomain) == 0 && (item.acceptSuffix || len(dnsName) == 0)
+}
+
+func dnsLabelMatches(certLabel string, watchLabel string) bool {
+	// For fail-safe behavior, if a label was unparsable, it matches everything.
+	// Similarly, redacted labels match everything, since the label _might_ be
+	// for a name we're interested in.
+
+	return certLabel == "*" ||
+		certLabel == "?" ||
+		certLabel == certspotter.UnparsableDNSLabelPlaceholder ||
+		certspotter.MatchesWildcard(watchLabel, certLabel)
+}
+
+func (list WatchList) Matches(identifiers *certspotter.Identifiers) (bool, WatchItem) {
+	dnsNames := make([][]string, len(identifiers.DNSNames))
+	for i, dnsName := range identifiers.DNSNames {
+		dnsNames[i] = strings.Split(dnsName, ".")
+	}
+	for _, item := range list {
+		for _, dnsName := range dnsNames {
+			if item.matchesDNSName(dnsName) {
+				return true, item
+			}
+		}
+	}
+	return false, WatchItem{}
+}

scanner.go

@@ -1,321 +0,0 @@
-// Copyright (C) 2016 Opsmate, Inc.
-//
-// This Source Code Form is subject to the terms of the Mozilla
-// Public License, v. 2.0. If a copy of the MPL was not distributed
-// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
-//
-// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
-// See the Mozilla Public License for details.
-//
-// This file contains code from https://github.com/google/certificate-transparency/tree/master/go
-// See ct/AUTHORS and ct/LICENSE for copyright and license information.
-
-package certspotter
-
-import (
-	//	"container/list"
-	"bytes"
-	"crypto"
-	"errors"
-	"fmt"
-	"log"
-	"sync"
-	"sync/atomic"
-	"time"
-
-	"software.sslmate.com/src/certspotter/ct"
-	"software.sslmate.com/src/certspotter/ct/client"
-)
-
-type ProcessCallback func(*Scanner, *ct.LogEntry)
-
-const (
-	FETCH_RETRIES    = 10
-	FETCH_RETRY_WAIT = 1
-)
-
-// ScannerOptions holds configuration options for the Scanner
-type ScannerOptions struct {
-	// Number of entries to request in one batch from the Log
-	BatchSize int
-
-	// Number of concurrent proecssors to run
-	NumWorkers int
-
-	// Don't print any status messages to stdout
-	Quiet bool
-}
-
-// Creates a new ScannerOptions struct with sensible defaults
-func DefaultScannerOptions() *ScannerOptions {
-	return &ScannerOptions{
-		BatchSize:  1000,
-		NumWorkers: 1,
-		Quiet:      false,
-	}
-}
-
-// Scanner is a tool to scan all the entries in a CT Log.
-type Scanner struct {
-	// Base URI of CT log
-	LogUri string
-
-	// Public key of the log
-	publicKey crypto.PublicKey
-	LogId     []byte
-
-	// Client used to talk to the CT log instance
-	logClient *client.LogClient
-
-	// Configuration options for this Scanner instance
-	opts ScannerOptions
-}
-
-// fetchRange represents a range of certs to fetch from a CT log
-type fetchRange struct {
-	start int64
-	end   int64
-}
-
-// Worker function to process certs.
-// Accepts ct.LogEntries over the |entries| channel, and invokes processCert on them.
-// Returns true over the |done| channel when the |entries| channel is closed.
-func (s *Scanner) processerJob(id int, certsProcessed *int64, entries <-chan ct.LogEntry, processCert ProcessCallback, wg *sync.WaitGroup) {
-	for entry := range entries {
-		atomic.AddInt64(certsProcessed, 1)
-		processCert(s, &entry)
-	}
-	wg.Done()
-}
-
-func (s *Scanner) fetch(r fetchRange, entries chan<- ct.LogEntry, tree *CollapsedMerkleTree) error {
-	success := false
-	retries := FETCH_RETRIES
-	retryWait := FETCH_RETRY_WAIT
-	for !success {
-		s.Log(fmt.Sprintf("Fetching entries %d to %d", r.start, r.end))
-		logEntries, err := s.logClient.GetEntries(r.start, r.end)
-		if err != nil {
-			if retries == 0 {
-				s.Warn(fmt.Sprintf("Problem fetching entries %d to %d from log: %s", r.start, r.end, err.Error()))
-				return err
-			} else {
-				s.Log(fmt.Sprintf("Problem fetching entries %d to %d from log (will retry): %s", r.start, r.end, err.Error()))
-				time.Sleep(time.Duration(retryWait) * time.Second)
-				retries--
-				retryWait *= 2
-				continue
-			}
-		}
-		retries = FETCH_RETRIES
-		retryWait = FETCH_RETRY_WAIT
-		for _, logEntry := range logEntries {
-			if tree != nil {
-				tree.Add(hashLeaf(logEntry.LeafBytes))
-			}
-			logEntry.Index = r.start
-			entries <- logEntry
-			r.start++
-		}
-		if r.start > r.end {
-			// Only complete if we actually got all the leaves we were
-			// expecting -- Logs MAY return fewer than the number of
-			// leaves requested.
-			success = true
-		}
-	}
-	return nil
-}
-
-// Worker function for fetcher jobs.
-// Accepts cert ranges to fetch over the |ranges| channel, and if the fetch is
-// successful sends the individual LeafInputs out into the
-// |entries| channel for the processors to chew on.
-// Will retry failed attempts to retrieve ranges indefinitely.
-// Sends true over the |done| channel when the |ranges| channel is closed.
-/* disabled becuase error handling is broken
-func (s *Scanner) fetcherJob(id int, ranges <-chan fetchRange, entries chan<- ct.LogEntry, wg *sync.WaitGroup) {
-	for r := range ranges {
-		s.fetch(r, entries, nil)
-	}
-	wg.Done()
-}
-*/
-
-// Returns the smaller of |a| and |b|
-func min(a int64, b int64) int64 {
-	if a < b {
-		return a
-	} else {
-		return b
-	}
-}
-
-// Returns the larger of |a| and |b|
-func max(a int64, b int64) int64 {
-	if a > b {
-		return a
-	} else {
-		return b
-	}
-}
-
-// Pretty prints the passed in number of |seconds| into a more human readable
-// string.
-func humanTime(seconds int) string {
-	nanos := time.Duration(seconds) * time.Second
-	hours := int(nanos / (time.Hour))
-	nanos %= time.Hour
-	minutes := int(nanos / time.Minute)
-	nanos %= time.Minute
-	seconds = int(nanos / time.Second)
-	s := ""
-	if hours > 0 {
-		s += fmt.Sprintf("%d hours ", hours)
-	}
-	if minutes > 0 {
-		s += fmt.Sprintf("%d minutes ", minutes)
-	}
-	if seconds > 0 {
-		s += fmt.Sprintf("%d seconds ", seconds)
-	}
-	return s
-}
-
-func (s Scanner) Log(msg string) {
-	if !s.opts.Quiet {
-		log.Print(msg)
-	}
-}
-
-func (s Scanner) Warn(msg string) {
-	log.Print(msg)
-}
-
-func (s *Scanner) GetSTH() (*ct.SignedTreeHead, error) {
-	latestSth, err := s.logClient.GetSTH()
-	if err != nil {
-		return nil, err
-	}
-	if s.publicKey != nil {
-		verifier, err := ct.NewSignatureVerifier(s.publicKey)
-		if err != nil {
-			return nil, err
-		}
-		if err := verifier.VerifySTHSignature(*latestSth); err != nil {
-			return nil, errors.New("STH signature is invalid: " + err.Error())
-		}
-	}
-	copy(latestSth.LogID[:], s.LogId)
-	return latestSth, nil
-}
-
-func (s *Scanner) CheckConsistency(first *ct.SignedTreeHead, second *ct.SignedTreeHead) (bool, error) {
-	if first.TreeSize < second.TreeSize {
-		proof, err := s.logClient.GetConsistencyProof(int64(first.TreeSize), int64(second.TreeSize))
-		if err != nil {
-			return false, err
-		}
-		return VerifyConsistencyProof(proof, first, second), nil
-	} else if first.TreeSize > second.TreeSize {
-		proof, err := s.logClient.GetConsistencyProof(int64(second.TreeSize), int64(first.TreeSize))
-		if err != nil {
-			return false, err
-		}
-		return VerifyConsistencyProof(proof, second, first), nil
-	} else {
-		// There is no need to ask the server for a consistency proof if the trees
-		// are the same size, and the DigiCert log returns a 400 error if we try.
-		return bytes.Equal(first.SHA256RootHash[:], second.SHA256RootHash[:]), nil
-	}
-}
-
-func (s *Scanner) MakeCollapsedMerkleTree(sth *ct.SignedTreeHead) (*CollapsedMerkleTree, error) {
-	if sth.TreeSize == 0 {
-		return &CollapsedMerkleTree{}, nil
-	}
-
-	entries, err := s.logClient.GetEntries(int64(sth.TreeSize-1), int64(sth.TreeSize-1))
-	if err != nil {
-		return nil, err
-	}
-	if len(entries) == 0 {
-		return nil, fmt.Errorf("Log did not return entry %d", sth.TreeSize-1)
-	}
-	leafHash := hashLeaf(entries[0].LeafBytes)
-
-	var tree *CollapsedMerkleTree
-	if sth.TreeSize > 1 {
-		auditPath, _, err := s.logClient.GetAuditProof(leafHash, sth.TreeSize)
-		if err != nil {
-			return nil, err
-		}
-		reverseHashes(auditPath)
-		tree, err = NewCollapsedMerkleTree(auditPath, sth.TreeSize-1)
-		if err != nil {
-			return nil, fmt.Errorf("Error returned bad audit proof for %x to %d", leafHash, sth.TreeSize)
-		}
-	} else {
-		tree = EmptyCollapsedMerkleTree()
-	}
-
-	tree.Add(leafHash)
-	if !bytes.Equal(tree.CalculateRoot(), sth.SHA256RootHash[:]) {
-		return nil, fmt.Errorf("Calculated root hash does not match signed tree head at size %d", sth.TreeSize)
-	}
-
-	return tree, nil
-}
-
-func (s *Scanner) Scan(startIndex int64, endIndex int64, processCert ProcessCallback, tree *CollapsedMerkleTree) error {
-	s.Log("Starting scan...")
-
-	certsProcessed := new(int64)
-	startTime := time.Now()
-	/* TODO: only launch ticker goroutine if in verbose mode; kill the goroutine when the scanner finishes
-	ticker := time.NewTicker(time.Second)
-	go func() {
-		for range ticker.C {
-			throughput := float64(s.certsProcessed) / time.Since(startTime).Seconds()
-			remainingCerts := int64(endIndex) - int64(startIndex) - s.certsProcessed
-			remainingSeconds := int(float64(remainingCerts) / throughput)
-			remainingString := humanTime(remainingSeconds)
-			s.Log(fmt.Sprintf("Processed: %d certs (to index %d). Throughput: %3.2f ETA: %s", s.certsProcessed,
-				startIndex+int64(s.certsProcessed), throughput, remainingString))
-		}
-	}()
-	*/
-
-	// Start processor workers
-	jobs := make(chan ct.LogEntry, 100)
-	var processorWG sync.WaitGroup
-	for w := 0; w < s.opts.NumWorkers; w++ {
-		processorWG.Add(1)
-		go s.processerJob(w, certsProcessed, jobs, processCert, &processorWG)
-	}
-
-	for start := startIndex; start < int64(endIndex); {
-		end := min(start+int64(s.opts.BatchSize), int64(endIndex)) - 1
-		if err := s.fetch(fetchRange{start, end}, jobs, tree); err != nil {
-			return err
-		}
-		start = end + 1
-	}
-	close(jobs)
-	processorWG.Wait()
-	s.Log(fmt.Sprintf("Completed %d certs in %s", *certsProcessed, humanTime(int(time.Since(startTime).Seconds()))))
-
-	return nil
-}
-
-// Creates a new Scanner instance using |client| to talk to the log, and taking
-// configuration options from |opts|.
-func NewScanner(logUri string, logId []byte, publicKey crypto.PublicKey, opts *ScannerOptions) *Scanner {
-	var scanner Scanner
-	scanner.LogUri = logUri
-	scanner.LogId = logId
-	scanner.publicKey = publicKey
-	scanner.logClient = client.New(logUri)
-	scanner.opts = *opts
-	return &scanner
-}

sct.go

@@ -1,46 +0,0 @@
-// Copyright (C) 2017 Opsmate, Inc.
-//
-// This Source Code Form is subject to the terms of the Mozilla
-// Public License, v. 2.0. If a copy of the MPL was not distributed
-// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
-//
-// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
-// See the Mozilla Public License for details.
-
-package certspotter
-
-import (
-	"software.sslmate.com/src/certspotter/ct"
-)
-
-func VerifyX509SCT(sct *ct.SignedCertificateTimestamp, cert []byte, verify *ct.SignatureVerifier) error {
-	entry := ct.LogEntry{
-		Leaf: ct.MerkleTreeLeaf{
-			Version:  0,
-			LeafType: ct.TimestampedEntryLeafType,
-			TimestampedEntry: ct.TimestampedEntry{
-				Timestamp:  sct.Timestamp,
-				EntryType:  ct.X509LogEntryType,
-				X509Entry:  cert,
-				Extensions: sct.Extensions,
-			},
-		},
-	}
-	return verify.VerifySCTSignature(*sct, entry)
-}
-
-func VerifyPrecertSCT(sct *ct.SignedCertificateTimestamp, precert ct.PreCert, verify *ct.SignatureVerifier) error {
-	entry := ct.LogEntry{
-		Leaf: ct.MerkleTreeLeaf{
-			Version:  0,
-			LeafType: ct.TimestampedEntryLeafType,
-			TimestampedEntry: ct.TimestampedEntry{
-				Timestamp:  sct.Timestamp,
-				EntryType:  ct.PrecertLogEntryType,
-				PrecertEntry:  precert,
-				Extensions: sct.Extensions,
-			},
-		},
-	}
-	return verify.VerifySCTSignature(*sct, entry)
-}

sequencer/sequencer.go

@@ -0,0 +1,128 @@
+// Copyright (C) 2025 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package sequencer
+
+import (
+	"context"
+	"slices"
+	"sync"
+)
+
+type seqWriter struct {
+	seqNbr uint64
+	ready  chan<- struct{}
+}
+
+// Channel[T] is a multi-producer, single-consumer channel of items with monotonicaly-increasing sequence numbers.
+// Items can be sent in any order, but they are always received in order of their sequence number.
+type Channel[T any] struct {
+	mu          sync.Mutex
+	next        uint64
+	buf         []*T
+	writers     []seqWriter
+	readWaiting bool
+	readReady   chan struct{}
+}
+
+func New[T any](initialSequenceNumber uint64, capacity uint64) *Channel[T] {
+	return &Channel[T]{
+		buf:       make([]*T, capacity),
+		next:      initialSequenceNumber,
+		readReady: make(chan struct{}, 1),
+	}
+}
+
+func (seq *Channel[T]) parkWriter(seqNbr uint64) <-chan struct{} {
+	ready := make(chan struct{})
+	seq.writers = append(seq.writers, seqWriter{seqNbr: seqNbr, ready: ready})
+	return ready
+}
+
+func (seq *Channel[T]) signalWriter(seqNbr uint64) {
+	for i := range seq.writers {
+		if seq.writers[i].seqNbr == seqNbr {
+			close(seq.writers[i].ready)
+			seq.writers = slices.Delete(seq.writers, i, i+1)
+			return
+		}
+	}
+}
+
+func (seq *Channel[T]) forgetWriter(seqNbr uint64) {
+	for i := range seq.writers {
+		if seq.writers[i].seqNbr == seqNbr {
+			seq.writers = slices.Delete(seq.writers, i, i+1)
+			return
+		}
+	}
+}
+
+func (seq *Channel[T]) Cap() uint64 {
+	return uint64(len(seq.buf))
+}
+
+func (seq *Channel[T]) index(seqNbr uint64) int {
+	return int(seqNbr % seq.Cap())
+}
+
+// Send an item with the given sequence number.  Blocks if the channel does not have capacity for the item.
+// It is undefined behavior to send a sequence number that has previously been sent.
+func (seq *Channel[T]) Add(ctx context.Context, sequenceNumber uint64, item *T) error {
+	seq.mu.Lock()
+	if sequenceNumber >= seq.next+seq.Cap() {
+		ready := seq.parkWriter(sequenceNumber)
+		seq.mu.Unlock()
+		select {
+		case <-ctx.Done():
+			seq.mu.Lock()
+			seq.forgetWriter(sequenceNumber)
+			seq.mu.Unlock()
+			return ctx.Err()
+		case <-ready:
+		}
+		seq.mu.Lock()
+	}
+	seq.buf[seq.index(sequenceNumber)] = item
+	if sequenceNumber == seq.next && seq.readWaiting {
+		seq.readReady <- struct{}{}
+	}
+	seq.mu.Unlock()
+	return nil
+}
+
+// Return the item with the next sequence number, blocking if necessary.
+// Not safe to call concurrently with other Next calls.
+func (seq *Channel[T]) Next(ctx context.Context) (*T, error) {
+	seq.mu.Lock()
+	if seq.buf[seq.index(seq.next)] == nil {
+		seq.readWaiting = true
+		seq.mu.Unlock()
+		select {
+		case <-ctx.Done():
+			seq.mu.Lock()
+			select {
+			case <-seq.readReady:
+			default:
+			}
+			seq.readWaiting = false
+			seq.mu.Unlock()
+			return nil, ctx.Err()
+		case <-seq.readReady:
+		}
+		seq.mu.Lock()
+		seq.readWaiting = false
+	}
+	item := seq.buf[seq.index(seq.next)]
+	seq.buf[seq.index(seq.next)] = nil
+	seq.signalWriter(seq.next + seq.Cap())
+	seq.next++
+	seq.mu.Unlock()
+	return item, nil
+}

sequencer/sequencer_test.go

@@ -0,0 +1,149 @@
+// Copyright (C) 2025 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package sequencer
+
+import (
+	"context"
+	"fmt"
+	mathrand "math/rand/v2"
+	"testing"
+	"time"
+)
+
+func TestSequencerBasic(t *testing.T) {
+	ctx := context.Background()
+	seq := New[uint64](0, 100)
+	go func() {
+		for i := range uint64(10_000) {
+			err := seq.Add(ctx, i, &i)
+			if err != nil {
+				panic(fmt.Sprintf("%d: seq.Add returned unexpected error %v", i, err))
+			}
+		}
+	}()
+
+	for i := range uint64(10_000) {
+		next, err := seq.Next(ctx)
+		if err != nil {
+			t.Fatalf("%d: seq.Next returned unexpected error %v", i, err)
+		}
+		if *next != i {
+			t.Fatalf("%d: got unexpected value %d", i, *next)
+		}
+	}
+}
+
+func TestSequencerNonZeroStart(t *testing.T) {
+	ctx := context.Background()
+	seq := New[uint64](10, 100)
+	go func() {
+		for i := range uint64(10_000) {
+			err := seq.Add(ctx, i+10, &i)
+			if err != nil {
+				panic(fmt.Sprintf("%d: seq.Add returned unexpected error %v", i, err))
+			}
+		}
+	}()
+
+	for i := range uint64(10_000) {
+		next, err := seq.Next(ctx)
+		if err != nil {
+			t.Fatalf("%d: seq.Next returned unexpected error %v", i, err)
+		}
+		if *next != i {
+			t.Fatalf("%d: got unexpected value %d", i, *next)
+		}
+	}
+}
+
+func TestSequencerCapacity1(t *testing.T) {
+	ctx := context.Background()
+	seq := New[uint64](0, 1)
+	go func() {
+		for i := range uint64(10_000) {
+			err := seq.Add(ctx, i, &i)
+			if err != nil {
+				panic(fmt.Sprintf("%d: seq.Add returned unexpected error %v", i, err))
+			}
+		}
+	}()
+
+	for i := range uint64(10_000) {
+		next, err := seq.Next(ctx)
+		if err != nil {
+			t.Fatalf("%d: seq.Next returned unexpected error %v", i, err)
+		}
+		if *next != i {
+			t.Fatalf("%d: got unexpected value %d", i, *next)
+		}
+	}
+}
+
+func TestSequencerTimeout(t *testing.T) {
+	ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
+	defer cancel()
+	seq := New[uint64](0, 10_000)
+	go func() {
+		var i uint64
+		for {
+			newI := i
+			err := seq.Add(ctx, i, &newI)
+			if err != nil {
+				break
+			}
+			i++
+		}
+	}()
+
+	var i uint64
+	for {
+		next, err := seq.Next(ctx)
+		if err != nil {
+			break
+		}
+		if *next != i {
+			t.Fatalf("%d: got unexpected value %d", i, *next)
+		}
+		i++
+	}
+}
+
+func TestSequencerOutOfOrder(t *testing.T) {
+	ctx := context.Background()
+	seq := New[uint64](0, 100)
+	ch := make(chan uint64)
+	go func() {
+		for i := range uint64(10_000) {
+			ch <- i
+		}
+	}()
+	for range 4 {
+		go func() {
+			for i := range ch {
+				time.Sleep(mathrand.N(10 * time.Millisecond))
+				//t.Logf("seq.Add %d", i)
+				err := seq.Add(ctx, i, &i)
+				if err != nil {
+					panic(fmt.Sprintf("%d: seq.Add returned unexpected error %v", i, err))
+				}
+			}
+		}()
+	}
+	for i := range uint64(10_000) {
+		next, err := seq.Next(ctx)
+		if err != nil {
+			t.Fatalf("%d: seq.Next returned unexpected error %v", i, err)
+		}
+		if *next != i {
+			t.Fatalf("%d: got unexpected value %d", i, *next)
+		}
+		//t.Logf("seq.Next %d", i)
+	}
+}

tlstypes/signatures.go

@@ -0,0 +1,134 @@
+// Copyright (C) 2025 Opsmate, Inc.
+//
+// This Source Code Form is subject to the terms of the Mozilla
+// Public License, v. 2.0. If a copy of the MPL was not distributed
+// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+//
+// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
+// See the Mozilla Public License for details.
+
+package tlstypes
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"golang.org/x/crypto/cryptobyte"
+)
+
+type HashAlgorithm uint8
+
+const (
+	SHA224 HashAlgorithm = 3
+	SHA256 HashAlgorithm = 4
+	SHA384 HashAlgorithm = 5
+	SHA512 HashAlgorithm = 6
+)
+
+type SignatureAlgorithm uint8
+
+const (
+	RSA   SignatureAlgorithm = 1
+	ECDSA SignatureAlgorithm = 3
+)
+
+type SignatureAndHashAlgorithm struct {
+	Hash      HashAlgorithm
+	Signature SignatureAlgorithm
+}
+
+type DigitallySigned struct {
+	Algorithm SignatureAndHashAlgorithm
+	Signature []byte
+}
+
+func (v HashAlgorithm) Marshal(b *cryptobyte.Builder) error {
+	b.AddUint8(uint8(v))
+	return nil
+}
+func (v *HashAlgorithm) Unmarshal(s *cryptobyte.String) bool {
+	return s.ReadUint8((*uint8)(v))
+}
+
+func (v SignatureAlgorithm) Marshal(b *cryptobyte.Builder) error {
+	b.AddUint8(uint8(v))
+	return nil
+}
+func (v *SignatureAlgorithm) Unmarshal(s *cryptobyte.String) bool {
+	return s.ReadUint8((*uint8)(v))
+}
+
+func (v SignatureAndHashAlgorithm) Marshal(b *cryptobyte.Builder) error {
+	b.AddValue(v.Hash)
+	b.AddValue(v.Signature)
+	return nil
+}
+func (v *SignatureAndHashAlgorithm) Unmarshal(s *cryptobyte.String) bool {
+	return v.Hash.Unmarshal(s) && v.Signature.Unmarshal(s)
+}
+
+func (v DigitallySigned) Marshal(b *cryptobyte.Builder) error {
+	b.AddValue(v.Algorithm)
+	b.AddUint16LengthPrefixed(func(b *cryptobyte.Builder) { b.AddBytes(v.Signature) })
+	return nil
+}
+func (v *DigitallySigned) Unmarshal(s *cryptobyte.String) bool {
+	return v.Algorithm.Unmarshal(s) && s.ReadUint16LengthPrefixed((*cryptobyte.String)(&v.Signature))
+}
+
+func (v DigitallySigned) Bytes() []byte {
+	b := cryptobyte.NewBuilder(make([]byte, 0, 4+len(v.Signature)))
+	b.AddValue(v)
+	return b.BytesOrPanic()
+}
+
+func (v DigitallySigned) MarshalBinary() ([]byte, error) {
+	b := cryptobyte.NewBuilder(make([]byte, 0, 4+len(v.Signature)))
+	b.AddValue(v)
+	return b.Bytes()
+}
+
+func (v *DigitallySigned) UnmarshalBinary(data []byte) error {
+	str := cryptobyte.String(bytes.Clone(data))
+	if !v.Unmarshal(&str) {
+		return fmt.Errorf("DigitallySigned bytes are malformed")
+	}
+	if !str.Empty() {
+		return fmt.Errorf("trailing bytes after DigitallySigned")
+	}
+	return nil
+}
+
+func (v DigitallySigned) MarshalJSON() ([]byte, error) {
+	b := cryptobyte.NewBuilder(make([]byte, 0, 4+len(v.Signature)))
+	b.AddValue(v)
+	if bytes, err := b.Bytes(); err != nil {
+		return nil, err
+	} else {
+		return json.Marshal(bytes)
+	}
+}
+func (v *DigitallySigned) UnmarshalJSON(data []byte) error {
+	str := new(cryptobyte.String)
+	if err := json.Unmarshal(data, (*[]byte)(str)); err != nil {
+		return fmt.Errorf("unable to unmarshal DigitallySigned JSON: %w", err)
+	}
+	if !v.Unmarshal(str) {
+		return fmt.Errorf("DigitallySigned bytes are malformed")
+	}
+	if !str.Empty() {
+		return fmt.Errorf("trailing bytes after DigitallySigned")
+	}
+	return nil
+}
+func ParseDigitallySigned(bytes []byte) (*DigitallySigned, error) {
+	ds := new(DigitallySigned)
+	str := cryptobyte.String(bytes)
+	if !ds.Unmarshal(&str) {
+		return nil, fmt.Errorf("DigitallySigned bytes are malformed")
+	}
+	if !str.Empty() {
+		return nil, fmt.Errorf("trailing bytes after DigitallySigned")
+	}
+	return ds, nil
+}

x509.go

@@ -11,6 +11,7 @@ package certspotter
 
 import (
 	"bytes"
+	"crypto/x509/pkix"
 	"encoding/asn1"
 	"errors"
 	"fmt"
@@ -83,6 +84,10 @@ func ParseRDNSequence(rdnsBytes []byte) (RDNSequence, error) {
 	return rdns, nil
 }
 
+func MarshalRDNSequence(rdns RDNSequence) ([]byte, error) {
+	return asn1.Marshal(rdns)
+}
+
 type TBSCertificate struct {
 	Raw asn1.RawContent
 
@@ -315,7 +320,7 @@ func (tbs *TBSCertificate) ParseSubjectAltNames() ([]SubjectAltName, error) {
 
 	for _, sanExt := range tbs.GetExtension(oidExtensionSubjectAltName) {
 		var err error
-		sans, err = parseSANExtension(sans, sanExt.Value)
+		sans, err = ParseSANExtension(sans, sanExt.Value)
 		if err != nil {
 			return nil, err
 		}
@@ -352,7 +357,27 @@ func (cert *Certificate) ParseTBSCertificate() (*TBSCertificate, error) {
 	return ParseTBSCertificate(cert.GetRawTBSCertificate())
 }
 
-func parseSANExtension(sans []SubjectAltName, value []byte) ([]SubjectAltName, error) {
+func (cert *Certificate) ParseSignatureAlgorithm() (*pkix.AlgorithmIdentifier, error) {
+	signatureAlgorithm := new(pkix.AlgorithmIdentifier)
+	if rest, err := asn1.Unmarshal(cert.SignatureAlgorithm.FullBytes, signatureAlgorithm); err != nil {
+		return nil, errors.New("failed to parse signature algorithm: " + err.Error())
+	} else if len(rest) > 0 {
+		return nil, fmt.Errorf("trailing data after signature algorithm: %v", rest)
+	}
+	return signatureAlgorithm, nil
+}
+
+func (cert *Certificate) ParseSignatureValue() ([]byte, error) {
+	var signatureValue asn1.BitString
+	if rest, err := asn1.Unmarshal(cert.SignatureValue.FullBytes, &signatureValue); err != nil {
+		return nil, errors.New("failed to parse signature value: " + err.Error())
+	} else if len(rest) > 0 {
+		return nil, fmt.Errorf("trailing data after signature value: %v", rest)
+	}
+	return signatureValue.RightAlign(), nil
+}
+
+func ParseSANExtension(sans []SubjectAltName, value []byte) ([]SubjectAltName, error) {
 	var seq asn1.RawValue
 	if rest, err := asn1.Unmarshal(value, &seq); err != nil {
 		return nil, errors.New("failed to parse subjectAltName extension: " + err.Error())