Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
// Copyright (C) 2016-2017 Opsmate, Inc.
2016-05-04 20:53:48 +02:00
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
2016-02-05 03:45:37 +01:00
package cmd
import (
2016-07-28 20:55:46 +02:00
"bytes"
2016-02-05 03:45:37 +01:00
"flag"
"fmt"
"log"
"os"
2016-02-05 05:16:25 +01:00
"os/user"
"path/filepath"
2016-07-28 20:55:46 +02:00
"sync"
2016-02-05 03:45:37 +01:00
2016-05-04 21:19:59 +02:00
"software.sslmate.com/src/certspotter"
"software.sslmate.com/src/certspotter/ct"
2016-02-05 03:45:37 +01:00
)
2016-07-27 02:02:56 +02:00
var batchSize = flag . Int ( "batch_size" , 1000 , "Max number of entries to request at per call to get-entries (advanced)" )
var numWorkers = flag . Int ( "num_workers" , 2 , "Number of concurrent matchers (advanced)" )
2016-02-05 03:45:37 +01:00
var script = flag . String ( "script" , "" , "Script to execute when a matching certificate is found" )
2016-07-27 02:02:56 +02:00
var logsFilename = flag . String ( "logs" , "" , "JSON file containing log information" )
2016-02-23 00:14:17 +01:00
var underwater = flag . Bool ( "underwater" , false , "Monitor certificates from distrusted CAs instead of trusted CAs" )
2016-02-05 05:16:25 +01:00
var noSave = flag . Bool ( "no_save" , false , "Do not save a copy of matching certificates" )
2016-02-05 03:45:37 +01:00
var verbose = flag . Bool ( "verbose" , false , "Be verbose" )
2016-02-17 23:54:25 +01:00
var allTime = flag . Bool ( "all_time" , false , "Scan certs from all time, not just since last scan" )
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
var state * State
2016-02-05 03:45:37 +01:00
var printMutex sync . Mutex
2016-07-28 20:55:46 +02:00
func homedir ( ) string {
2016-02-05 05:16:25 +01:00
home := os . Getenv ( "HOME" )
if home != "" {
return home
}
user , err := user . Current ( )
if err == nil {
return user . HomeDir
}
panic ( "Unable to determine home directory" )
}
2016-07-28 20:55:46 +02:00
func DefaultStateDir ( programName string ) string {
return filepath . Join ( homedir ( ) , "." + programName )
2016-02-05 05:16:25 +01:00
}
2016-07-28 20:55:46 +02:00
func DefaultConfigDir ( programName string ) string {
return filepath . Join ( homedir ( ) , "." + programName )
2016-05-12 20:30:59 +02:00
}
2016-07-28 20:55:46 +02:00
func LogEntry ( info * certspotter . EntryInfo ) {
2016-02-05 05:16:25 +01:00
if ! * noSave {
2016-02-05 17:20:12 +01:00
var alreadyPresent bool
var err error
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
alreadyPresent , info . Filename , err = state . SaveCert ( info . IsPrecert , info . FullChain )
2016-02-05 03:45:37 +01:00
if err != nil {
log . Print ( err )
}
if alreadyPresent {
return
}
}
if * script != "" {
2016-02-09 19:28:52 +01:00
if err := info . InvokeHookScript ( * script ) ; err != nil {
2016-02-05 03:45:37 +01:00
log . Print ( err )
}
} else {
printMutex . Lock ( )
2016-02-09 19:28:52 +01:00
info . Write ( os . Stdout )
2016-02-05 03:45:37 +01:00
fmt . Fprintf ( os . Stdout , "\n" )
printMutex . Unlock ( )
}
}
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
func loadLogList ( ) ( [ ] certspotter . LogInfo , error ) {
if * logsFilename != "" {
var logFileObj certspotter . LogInfoFile
if err := readJSONFile ( * logsFilename , & logFileObj ) ; err != nil {
return nil , fmt . Errorf ( "Error reading logs file: %s: %s" , * logsFilename , err )
}
return logFileObj . Logs , nil
} else if * underwater {
return certspotter . UnderwaterLogs , nil
} else {
return certspotter . DefaultLogs , nil
}
2016-02-05 05:16:25 +01:00
}
2016-02-05 03:45:37 +01:00
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
type logHandle struct {
scanner * certspotter . Scanner
state * LogState
2017-01-06 23:39:08 +01:00
position * certspotter . CollapsedMerkleTree
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
verifiedSTH * ct . SignedTreeHead
}
2016-02-18 21:40:21 +01:00
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
func makeLogHandle ( logInfo * certspotter . LogInfo ) ( * logHandle , error ) {
ctlog := new ( logHandle )
2016-02-18 21:40:21 +01:00
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
logKey , err := logInfo . ParsedPublicKey ( )
if err != nil {
return nil , fmt . Errorf ( "Bad public key: %s" , err )
}
2017-01-06 21:50:21 +01:00
ctlog . scanner = certspotter . NewScanner ( logInfo . FullURI ( ) , logInfo . ID ( ) , logKey , & certspotter . ScannerOptions {
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
BatchSize : * batchSize ,
NumWorkers : * numWorkers ,
Quiet : ! * verbose ,
} )
ctlog . state , err = state . OpenLogState ( logInfo )
if err != nil {
return nil , fmt . Errorf ( "Error opening state directory: %s" , err )
}
ctlog . position , err = ctlog . state . GetLogPosition ( )
if err != nil {
return nil , fmt . Errorf ( "Error loading log position: %s" , err )
}
ctlog . verifiedSTH , err = ctlog . state . GetVerifiedSTH ( )
if err != nil {
return nil , fmt . Errorf ( "Error loading verified STH: %s" , err )
2016-02-18 21:40:21 +01:00
}
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
if ctlog . position == nil && ctlog . verifiedSTH == nil { // This branch can be removed eventually
legacySTH , err := state . GetLegacySTH ( logInfo ) ;
if err != nil {
return nil , fmt . Errorf ( "Error loading legacy STH: %s" , err )
}
if legacySTH != nil {
2017-01-06 19:31:34 +01:00
log . Printf ( "Initializing log state from legacy state directory" )
2017-01-06 23:39:08 +01:00
ctlog . position , err = ctlog . scanner . MakeCollapsedMerkleTree ( legacySTH )
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
if err != nil {
return nil , fmt . Errorf ( "Error reconstructing Merkle Tree for legacy STH: %s" , err )
}
if err := ctlog . state . StoreLogPosition ( ctlog . position ) ; err != nil {
return nil , fmt . Errorf ( "Error storing log position: %s" , err )
}
if err := ctlog . state . StoreVerifiedSTH ( legacySTH ) ; err != nil {
return nil , fmt . Errorf ( "Error storing verified STH: %s" , err )
}
state . RemoveLegacySTH ( logInfo )
}
2016-04-06 17:10:06 +02:00
}
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
return ctlog , nil
}
func ( ctlog * logHandle ) refresh ( ) error {
2017-01-06 19:31:34 +01:00
if * verbose {
log . Printf ( "Retrieving latest STH from log" )
}
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
latestSTH , err := ctlog . scanner . GetSTH ( )
if err != nil {
return fmt . Errorf ( "Error retrieving STH from log: %s" , err )
}
if ctlog . verifiedSTH == nil {
2017-01-06 19:31:34 +01:00
if * verbose {
log . Printf ( "No existing STH is known; presuming latest STH (%d) is valid" , latestSTH . TreeSize )
}
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
ctlog . verifiedSTH = latestSTH
if err := ctlog . state . StoreVerifiedSTH ( ctlog . verifiedSTH ) ; err != nil {
return fmt . Errorf ( "Error storing verified STH: %s" , err )
}
} else {
if err := ctlog . state . StoreUnverifiedSTH ( latestSTH ) ; err != nil {
return fmt . Errorf ( "Error storing unverified STH: %s" , err )
}
}
return nil
2016-02-18 21:40:21 +01:00
}
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
func ( ctlog * logHandle ) audit ( ) error {
sths , err := ctlog . state . GetUnverifiedSTHs ( )
if err != nil {
return fmt . Errorf ( "Error loading unverified STHs: %s" , err )
}
for _ , sth := range sths {
2017-01-06 19:31:34 +01:00
if * verbose {
log . Printf ( "Verifying consistency between STH %d (%x) and STH %d (%x)" , sth . TreeSize , sth . SHA256RootHash , ctlog . verifiedSTH . TreeSize , ctlog . verifiedSTH . SHA256RootHash )
}
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
if sth . TreeSize > ctlog . verifiedSTH . TreeSize {
2017-01-06 06:06:37 +01:00
isValid , err := ctlog . scanner . CheckConsistency ( ctlog . verifiedSTH , sth )
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
if err != nil {
return fmt . Errorf ( "Error fetching consistency proof between %d and %d (if this error persists, it should be construed as misbehavior by the log): %s" , ctlog . verifiedSTH . TreeSize , sth . TreeSize , err )
}
if ! isValid {
return fmt . Errorf ( "Log has misbehaved: STH in '%s' is not consistent with STH in '%s'" , ctlog . state . VerifiedSTHFilename ( ) , ctlog . state . UnverifiedSTHFilename ( sth ) )
}
2017-01-06 19:31:34 +01:00
if * verbose {
log . Printf ( "STH %d (%x) is now the latest verified STH" , sth . TreeSize , sth . SHA256RootHash )
}
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
ctlog . verifiedSTH = sth
if err := ctlog . state . StoreVerifiedSTH ( ctlog . verifiedSTH ) ; err != nil {
return fmt . Errorf ( "Error storing verified STH: %s" , err )
}
} else if sth . TreeSize < ctlog . verifiedSTH . TreeSize {
2017-01-06 06:06:37 +01:00
isValid , err := ctlog . scanner . CheckConsistency ( sth , ctlog . verifiedSTH )
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
if err != nil {
return fmt . Errorf ( "Error fetching consistency proof between %d and %d (if this error persists, it should be construed as misbehavior by the log): %s" , ctlog . verifiedSTH . TreeSize , sth . TreeSize , err )
}
if ! isValid {
return fmt . Errorf ( "Log has misbehaved: STH in '%s' is not consistent with STH in '%s'" , ctlog . state . VerifiedSTHFilename ( ) , ctlog . state . UnverifiedSTHFilename ( sth ) )
}
} else {
if ! bytes . Equal ( sth . SHA256RootHash [ : ] , ctlog . verifiedSTH . SHA256RootHash [ : ] ) {
return fmt . Errorf ( "Log has misbehaved: STH in '%s' is not consistent with STH in '%s'" , ctlog . state . VerifiedSTHFilename ( ) , ctlog . state . UnverifiedSTHFilename ( sth ) )
}
}
if err := ctlog . state . RemoveUnverifiedSTH ( sth ) ; err != nil {
return fmt . Errorf ( "Error removing redundant STH: %s" , err )
}
}
return nil
2016-11-15 21:23:24 +01:00
}
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
func ( ctlog * logHandle ) scan ( processCallback certspotter . ProcessCallback ) error {
startIndex := int64 ( ctlog . position . GetNumLeaves ( ) )
endIndex := int64 ( ctlog . verifiedSTH . TreeSize )
2016-02-05 03:45:37 +01:00
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
if endIndex > startIndex {
2017-01-06 23:39:08 +01:00
tree := certspotter . CloneCollapsedMerkleTree ( ctlog . position )
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
2017-01-06 23:39:08 +01:00
if err := ctlog . scanner . Scan ( startIndex , endIndex , processCallback , tree ) ; err != nil {
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
return fmt . Errorf ( "Error scanning log (if this error persists, it should be construed as misbehavior by the log): %s" , err )
2016-02-05 05:16:25 +01:00
}
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
2017-01-06 23:39:08 +01:00
rootHash := tree . CalculateRoot ( )
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
if ! bytes . Equal ( rootHash , ctlog . verifiedSTH . SHA256RootHash [ : ] ) {
return fmt . Errorf ( "Log has misbehaved: log entries at tree size %d do not correspond to signed tree root" , ctlog . verifiedSTH . TreeSize )
2016-02-05 05:16:25 +01:00
}
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
2017-01-06 23:39:08 +01:00
ctlog . position = tree
2017-01-06 21:19:53 +01:00
if err := ctlog . state . StoreLogPosition ( ctlog . position ) ; err != nil {
return fmt . Errorf ( "Error storing log position: %s" , err )
}
2016-02-05 03:45:37 +01:00
}
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
return nil
}
func processLog ( logInfo * certspotter . LogInfo , processCallback certspotter . ProcessCallback ) int {
log . SetPrefix ( os . Args [ 0 ] + ": " + logInfo . Url + ": " )
2016-11-15 21:23:24 +01:00
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
ctlog , err := makeLogHandle ( logInfo )
if err != nil {
log . Printf ( "%s\n" , err )
2016-06-03 16:21:08 +02:00
return 1
2016-02-05 03:45:37 +01:00
}
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
if err := ctlog . refresh ( ) ; err != nil {
log . Printf ( "%s\n" , err )
return 1
2016-02-05 03:45:37 +01:00
}
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
if err := ctlog . audit ( ) ; err != nil {
log . Printf ( "%s\n" , err )
return 1
}
2016-02-05 05:16:25 +01:00
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
if * allTime {
2017-01-06 23:39:08 +01:00
ctlog . position = certspotter . EmptyCollapsedMerkleTree ( )
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
if * verbose {
log . Printf ( "Scanning all %d entries in the log because -all_time option specified" , ctlog . verifiedSTH . TreeSize )
2016-02-05 05:16:25 +01:00
}
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
} else if ctlog . position != nil {
if * verbose {
log . Printf ( "Existing log; scanning %d new entries since previous scan" , ctlog . verifiedSTH . TreeSize - ctlog . position . GetNumLeaves ( ) )
2016-02-05 05:16:25 +01:00
}
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
} else if state . IsFirstRun ( ) {
2017-01-06 23:39:08 +01:00
ctlog . position , err = ctlog . scanner . MakeCollapsedMerkleTree ( ctlog . verifiedSTH )
2016-02-05 05:16:25 +01:00
if err != nil {
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
log . Printf ( "Error reconstructing Merkle Tree: %s" , err )
return 1
2016-02-05 05:16:25 +01:00
}
2016-02-22 23:58:11 +01:00
if * verbose {
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
log . Printf ( "First run of Cert Spotter; not scanning %d existing entries because -all_time option not specified" , ctlog . verifiedSTH . TreeSize )
2016-02-22 23:58:11 +01:00
}
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
} else {
2017-01-06 23:39:08 +01:00
ctlog . position = certspotter . EmptyCollapsedMerkleTree ( )
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
if * verbose {
log . Printf ( "New log; scanning all %d entries in the log" , ctlog . verifiedSTH . TreeSize )
2016-02-17 23:54:25 +01:00
}
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
}
2017-01-06 21:19:53 +01:00
if err := ctlog . state . StoreLogPosition ( ctlog . position ) ; err != nil {
log . Printf ( "Error storing log position: %s\n" , err )
return 1
}
2016-02-17 23:54:25 +01:00
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
if err := ctlog . scan ( processCallback ) ; err != nil {
log . Printf ( "%s\n" , err )
return 1
}
2016-02-17 23:54:25 +01:00
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
if * verbose {
log . Printf ( "Final log size = %d, final root hash = %x" , ctlog . verifiedSTH . TreeSize , ctlog . verifiedSTH . SHA256RootHash )
}
2016-02-17 23:54:25 +01:00
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
return 0
}
2016-02-05 05:16:25 +01:00
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
func Main ( statePath string , processCallback certspotter . ProcessCallback ) int {
var err error
2016-02-22 23:58:11 +01:00
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
state , err = OpenState ( statePath )
if err != nil {
fmt . Fprintf ( os . Stderr , "%s: %s\n" , os . Args [ 0 ] , err )
return 1
2016-02-05 03:45:37 +01:00
}
2016-02-05 05:16:25 +01:00
Overhaul log processing and auditing
1. Instead of storing a single STH per log, we now store one verified
STH and any number of unverified STHs. When we process a log, we verify
each unverified STH using a consistency proof with the verified STH,
and only delete it if it successfully verifies. We set the verified
STH to the largest STH which we've successfully verified.
This has two important benefits. First, we never ever delete an STH
unless we can successfully verify it (previously, we would forget about
an STH under certain error conditions). Second, it lays the groundwork
for STH pollination. Upon reception of an STH, we can simply drop it in
the log's unverified_sths directory (assuming the signature is valid),
and Cert Spotter will audit it.
There is no more "evidence" directory; if a consistency proof fails,
the STHs will already be present elsewhere in the state directory.
2. We now persist a MerkleTreeBuilder between each run of Cert Spotter,
instead of rebuilding it every time from the consistency proof. This is
not intrinsically better, but it makes the code simpler considering we
can now fetch numerous consistency proofs per run.
3. To accommodate the above changes, the state directory has a brand
new layout. The state directory is now versioned, and Cert Spotter
will automatically migrate old state directories to the new layout.
This migration logic will be removed in a future Cert Spotter release.
As a bonus, the code is generally cleaner now :-)
2017-01-06 05:46:42 +01:00
logs , err := loadLogList ( )
if err != nil {
fmt . Fprintf ( os . Stderr , "%s: %s\n" , os . Args [ 0 ] , err )
return 1
}
exitCode := 0
for i := range logs {
exitCode |= processLog ( & logs [ i ] , processCallback )
}
2017-01-06 21:23:20 +01:00
if state . IsFirstRun ( ) && exitCode == 0 {
2017-01-06 21:24:09 +01:00
if err := state . WriteOnceFile ( ) ; err != nil {
fmt . Fprintf ( os . Stderr , "%s: Error writing once file: %s\n" , os . Args [ 0 ] , err )
2017-01-06 21:23:20 +01:00
exitCode |= 1
}
2016-11-15 21:23:24 +01:00
}
2016-06-03 16:21:08 +02:00
return exitCode
2016-02-05 03:45:37 +01:00
}