Commit Graph

102 Commits

Author SHA1 Message Date
Andrew Ayer 5fae49a971 Simplify some code 2023-10-29 07:45:23 -04:00
chayleaf 74fb03b579 make sendmail path configurable using an env var
Closes: #80
2023-10-24 11:32:59 -04:00
guoguangwu 197414cfef chore: remove refs to deprecated io/ioutil 2023-10-10 10:17:05 +08:00
Andrew Ayer cd1b7a80ca Read email addresses from $CERTSPOTTER_CONFIG_DIR/email_recipients 2023-02-20 10:14:10 -05:00
Andrew Ayer 8b7cef7f61 Factor out some code into a helper function 2023-02-20 10:05:04 -05:00
Andrew Ayer fd0a2a4d44 Execute scripts under $CERTSPOTTER_CONFIG_DIR/hooks.d, if it exists 2023-02-20 10:02:48 -05:00
Andrew Ayer d08ad53464 Make error message for missing watch list more friendly 2023-02-20 09:47:15 -05:00
Andrew Ayer 7d910d5521 When version is set via ldflag, append a "?" to indicate uncertainty 2023-02-09 11:29:30 -05:00
Faidon Liambotis 1a5f581c07 certspotterVersion: also allow -X=main.Version
debug.ReadBuildInfo() doesn't work for the Debian package, where there
is no git repository (at best; could be the Debian packaging one). The
go module doesn't have a version either.

Add a quick shortcut at the top, that sets the version to main.Version
if it's non-empty -- which is by default, so no change in behavior.

Effectively, this allows one to use the standard
-ldflags=-X=main.Version to set the version, like we intend to do in the
Debian package.
2023-02-09 11:28:04 -05:00
Andrew Ayer 6a6f74414a Fix typos
Closes: #3
2023-02-09 11:27:48 -05:00
Andrew Ayer 52949d8ea3 Apply gofmt 2023-02-06 09:18:53 -05:00
Andrew Ayer 2a24abaa31 Make health check interval configurable 2023-02-06 09:18:37 -05:00
Andrew Ayer 2f2ad094db Set User-Agent header when fetching log list 2023-02-03 15:24:55 -05:00
Andrew Ayer 6bb03865fb Modernize loglist fetching, add context support 2023-02-03 14:55:09 -05:00
Andrew Ayer 29ed939006 Remove old code 2023-02-03 14:47:47 -05:00
Andrew Ayer 209cdb181b Convert to a daemon and make many other improvements
Specifically, certspotter no longer terminates unless it receives SIGTERM
or SIGINT or there is a serious error.

Although using cron made sense in the early days of Certificate
Transparency, certspotter now needs to run continuously to reliably keep
up with the high growth rate of contemporary CT logs, and to gracefully
handle the many transient errors that can arise when monitoring CT.

Closes: #63
Closes: #37
Closes: #32 (presumably by eliminating $DNS_NAMES and $IP_ADDRESSES)
Closes: #21 (with $WATCH_ITEM)
Closes: #25
2023-02-03 14:12:03 -05:00
Andrew Ayer e3835dea53 Add some comments about the nature of various errors 2023-02-03 13:59:40 -05:00
Andrew Ayer 936a1ca8ed Remove ctparsewatch 2023-01-22 13:54:43 -05:00
Andrew Ayer 33ebbdfd07 Use os.UserHomeDir to determine home directory
Go provides this since Go 1.12; no need to reinvent the wheel.
2022-10-08 18:17:35 -04:00
Andrew Ayer 039339154f Move retry logic into LogClient
This allows retry logic to be used for all requests, not just get-entries

Also add context arguments
2022-06-02 10:02:32 -04:00
Andrew Ayer c0e79476ae Add .gitignore file 2022-02-28 08:11:13 -05:00
Andrew Ayer c9aaa2782f Add -version flag 2021-10-12 10:28:34 -04:00
Andrew Ayer a147970db8 Use ct.SHA256Hash for log ID rather than []byte 2021-04-30 17:04:16 -04:00
Andrew Ayer 2cccf67601 Avoid leaving a file open for longer than necessary 2020-10-06 19:27:58 -04:00
Daniel Peukert 6d5e2395a1
Fix missing Printf 2020-05-01 00:25:39 +02:00
Andrew Ayer 185445e158 Retrieve log list from certspotter.org at startup instead of embedding in source
The list of logs changes far too frequently (with annual shards and operators
dropping out of the ecosystem) to continue embedding in the source code.

Breaking change: the -logs option now expects a
JSON file in the v2 log list format, as documented at
<https://www.certificate-transparency.org/known-logs> and
<https://www.gstatic.com/ct/log_list/v2/log_list_schema.json>.

You can now specify an HTTPS URL to -logs in addition to a file path.

Breaking change: the -underwater option has been removed; if you want
this behavior then specify https://loglist.certspotter.org/underwater.json
as your log list.
2020-04-29 11:51:50 -04:00
Andrew Ayer 30d171343a Add -start_at_end option to begin monitoring logs at the end
When Cert Spotter starts monitoring a log that it has never monitored before,
it can either start monitoring it from the beginning, or seek to the end and
start monitoring there.

Monitoring from the beginning guarantees detection of all certificates, but
requires downloading hundreds of millions of certificates, which takes days.

With the new -start_at_end option, you can save significant time by
starting at the end.  You will miss certificates that were added to a
log before Cert Spotter starts monitoring it, but you can always use the
Cert Spotter API <https://sslmate.com/certspotter/api> or crt.sh to find them.

Previously, the -start_at_end behavior was implied the first time you
ever ran Cert Spotter.  This is no longer the case.
2019-12-03 11:12:40 -05:00
Andrew Ayer 86785d89d7 Process logs in parallel 2019-12-02 15:19:35 -05:00
Andrew Ayer c2099d6d49 Manually prefix all log messages with log URL
(Instead of using log.SetPrefix)

This will let us process logs in parallel.
2019-12-02 15:03:34 -05:00
Ian Foster 6991be261c changed bygonessl behavior 2018-07-19 16:12:17 -07:00
Ian Foster 1b4943c198 rename issued_before to valid_at 2018-07-13 11:11:58 -07:00
Ian Foster e5fd2e9efc Initial BygoneSSL support 2018-07-04 19:03:57 -07:00
Andrew Ayer bc255f43d5 Add functions to verify SCTs 2017-12-17 09:51:46 -08:00
Andrew Ayer 06c253a0ea Continue processing a log even if an STH failed to verify
It may still be possible to audit other STHs, and to scan new entries
up to the latest verified STH.  This allows Cert Spotter to continue
to make forward progress even if a log is persistently skewed (as the
DigiCert has been lately).

Also, rework some code to be simpler and less redundant.
2017-04-27 10:48:48 -07:00
Andrew Ayer 8b9c08b984 submitct: apply gofmt 2017-02-05 10:09:42 -08:00
Andrew Ayer 732a660767 submitct: reorganize code 2017-02-05 10:09:19 -08:00
Andrew Ayer b94d850dbe submitct: use maps for efficiency
This will make submitct work better with lots of certificates as input.
2017-02-05 10:07:30 -08:00
Andrew Ayer 779230cbc8 submitct: include cert index in log entries 2017-02-04 16:09:56 -08:00
Andrew Ayer e7a9ba6e7d Add submitct program for submitting certs 2017-02-04 16:05:51 -08:00
Andrew Ayer e8c4f10e97 Use a lock file to prevent certspotter from running concurrently 2017-01-10 10:50:41 -08:00
Andrew Ayer 2f0833ac9c Apply gofmt 2017-01-08 10:17:00 -08:00
Andrew Ayer 9f3e9968ec Rename verified_sth.json to just sth.json 2017-01-08 10:15:41 -08:00
Andrew Ayer 7adbc6cffe Add .json extension to log state filenames
Since they are JSON.
2017-01-08 10:14:44 -08:00
Andrew Ayer b88de6b320 Rename "position" to "tree" in log state 2017-01-06 14:49:42 -08:00
Andrew Ayer 671814cd9b CollapsedMerkleTree: rename numLeaves to size 2017-01-06 14:41:51 -08:00
Andrew Ayer 9ceedea9ef Rename MerkleTreeBuilder to CollapsedMerkleTree 2017-01-06 14:39:08 -08:00
Andrew Ayer d8b1877e8d Improve filenames of unverified STHs
Include the tree size in plain decimal, since it's more user-friendly.

Don't include tree size in hash (redundant now that we're storing it
outside of hash) or version (implied by signature).
2017-01-06 12:51:10 -08:00
Andrew Ayer 1719aa5d8e Set log ID in STHs that we download
This will facilitate STH pollination.
2017-01-06 12:50:21 -08:00
Andrew Ayer 0eb6d199a4 Improve the name of a function 2017-01-06 12:24:09 -08:00
Andrew Ayer c8f0a0f9e8 Only write once file if run was 100% successful
Otherwise, if a single log was unreachable, we'd be force to download
all of it on the next run.
2017-01-06 12:23:20 -08:00