boundary

Commit Graph

Author	SHA1	Message	Date
Jim	a16033b73a	fix (kms): move oplog DEK into its own internal kms (#3665 ) * fix (auth/ldap): fix which DEK is used for oplog during authenticate * chore: update go-kms-wrapping dep for new extras/kms changes The latest version supports setting a table name prefix for a kms which is needed to fix our oplog DEK problem * chore: update extras/kms dep This latest version supports an optional TableNamePrefix which is required to fix our existing oplog DEK issue. * fix (oplog): schema oplog DEK fix These changes will allow use to store the oplog DEK in it's own set of tables, so they won't get deleted when a scope is deleted * fix (kms): move oplog DEK into its own internal kms * tests: update tests to be compatible with oplog fix * tests: remove migration tests which are incompatible with oplog fix These tests no longer work because they are missing the required schema changes for the oplog fix	3 years ago
hashicorp-copywrite[bot]	29da0bcb92	[COMPLIANCE] License changes (#3567 ) * Adding explicit MPL license for sub-package. This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository. * Adding explicit MPL license for sub-package This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository. * Updating the license from MPL to Business Source License Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at www.hashicorp.com/licensing-faq, and details of the license at www.hashicorp.com/bsl. * Update copyright file headers to BUS-1.1 * Rerun make gen This will pick up the last of the license changes * Revert "Rerun make gen" This reverts commit `dbe0968bff`. * Remove copywrite from generation script The current version of copywrite is not sophisticated enough to only apply the copywrite changes to certain subdirectories, so to avoid accidentally relicensing our api and sdk modules, we will stop running the copywrite script until such time that it can do the right thing. * License protobuf API as MPL The protobuf API files are used to generate the SDK, since we want the SDK to be MPL, the protobuf files must also be MPL. This extends to all protobuf files whose sources end up in the SDK directory, including: - internal/proto/controller/api - internal/proto/controller/custom_options - internal/proto/plugin It also includes the generated service directory in internal/gen/controller/api/services, since they are generated from the protobuf sources used for the SDK. * Change website license file name This makes the naming scheme consistent with the rest of the repository. * Add note on licensing to README * Set copywrite license to BUSL * Revert "Add note on licensing to README" This reverts commit `4d865a91a7`. --------- Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com> Co-authored-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>	3 years ago
Johan Brandhorst-Satzkorn	3c29308673	chore: Add license headers to all files	3 years ago
Johan Brandhorst-Satzkorn	edd323b73a	Key Rotation/Destruction (#2477 ) (#2607 ) * Key Rotation/Destruction (#2477) * fix: Correct kms.CreateKeys comment This method does not return anything. * feat(kms): Add new ListKeys method The new ListKeys method wraps the underlying KMS wrapper, returning all the keys in the scope specified. * feat(scopes): Add new scope actions for key management Adds list-keys, rotate-keys and revoke-keys actions for scopes. * feat(scopes): Add ListKeys API endpoint This new endpoint lists all keys in a scope * feat(api): Add ListKeys to Scopes client This has to be a custom function because it is a custom action and doesn't map well to any of the existing templates, or generalises well in a way that would make it reasonable to create a new template. * feat(cli): Add new scopes list-keys command * add RotateKeys function to the kms repository * Add Rotate Keys Endpoint to Scopes Api (#2360) * add rotate keys endpoint * add tests for RotateKeys endpoint * remove that one comment I forgot about * addressing more PR comments * Add Rotate Keys CLI Command (#2395) * add cli command and client * add bats tests * fix some info text and pr comments * write a new rotate keys description * Add Missing DEK Foreign Keys (#2408) * add missing fks * mark key as nullable in gorm * update tests to use a new wrapper with a real key_id * fix formatting in test and add keys for auth_token test * replace key values * revert postgres_40_01_test.go * style: reformat sql for readability * refactor: change type of key_id columns to kms_private_id * Store key_id and scope_id with oplog entries, re-type columns referencing data key version (#2431) * fix(db): Correct types of data key version referencers * fix(oplog): Fix missing error handling * feat(oplog): Store key_id and scope_id with oplog entries This migration truncates all existing oplog entries, as migrating existing data would have been complex and likely unnecessary. * Update view references * Always delete oplog entries when dropping keys * Fix rebase issues * Fix sql tests * Add schema for new kms destruction jobs (#2479) * feat: Add schema for new kms destruction jobs The schema lets us manage destruction jobs per-table while providing a guarantee that only one run is running at a time. * Apply suggestions from code review Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com> * Fix table reference and tests Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com> * update domain, api, and cli to all include key versions in outputs (#2472) * update domain, api, and cli to all include key versions in outputs * fix go sum * add migration to fix immutable error * fix final test issue as well as make gen issue * rebase and fix * address johan pr comments * make gen, didn't realize comments were included * update migrations based on new migration 55 file 03 * Move migrations to 56 * feat: Add key version rewrapping function registry (#2478) The new RegisterTableRewrapFn can be used by a domain to register a callback to use when rewrapping data in a specific table name. * feat(kms): Add scopeId to RewrapFn (#2539) This makes it much easier for rewrap functions to look up the correct wrapper and limit the number of rows searched. * Update Root Cert Proto Definition (#2542) * update root cert proto definition to properly identify public_key as the table primary key * add the test fix as well * Rewrap Functions for Encrypted Tables (#2532) * add oidc rewrap * add argon2 config rewrap * add auth token rewrapping * add worker auth cert rewrapping * check the err * add username password credential rewrapping * add ssh private key credential rewrapping * add vault client certificate rewrap function * add host catalog secret rewrap function * cleanup a little * add host vault token rewrap function * add session credential rewrap function * add session rewrap function * add worker auth rewrap function * update test to include full assert gamut * add session rewrap function TEST * bring all tests up to current standards, make more readable, and include full assert gamut * rework all rewrap functions to utilize scope id * update comments, queries, sql refs, etc, in response to PR comments * forgot a comment * remove hmac updates and fix a couple comments again * Rewrapping Registered Tables Test (#2555) * add the registered table test as well as the two missing rewrap functions * address pr comments, add db r/w conversions, cmp.diff, sql comment, remove (now) faulty immutable test * Add ability to list key version destruction jobs, recurring jobs and destroying key versions (#2498) * feat(kms): Add ability to list data key version destruction jobs * feat(scopes): Add ListKeyVersionDestructionJobs * feat(cli): Add list-key-version-destruction-jobs * feat(kms): Add recurring job that performs table rewrapping The new recurring job runs for each job table with a registered rewrapping callback. It attempts to become the running run and start its rewrapping, gracefully handling sudden interruptions by resuming it's work if it finds evidence of sudden interruption. * feat(kms): Add recurring job for destroying key versions The new job monitors the database for data key version destruction jobs that have finished rewrapping all its data, performing the final data key version revocation. * feat(kms): Add DestroyKeyVersion DestroyKeyVersion immediately destroys a key version if it is a root key version or a data key version which encrypts no data. If the data key version encrypts data which needs to be rewrapped, it queues an asynchronous job to complete the rewrapping operation. * feat(scopes): Add DestroyKeyVersion API endpoint * feat(cli): Add scopes destroy-key-version CLI command * Ensure all secret updates include key ID (#2556) * feat(all): Ensure all secret updates include key ID When updating a secret, it is paramount that the key ID is also updated, as it is the only means we have of ensuring that we only destroy keys once there is no more data associated with the key. * Remove fix to session repository for now This breaks the current private key derivation method * add new encrypt/decrypt funcs and made necessary proto changes (#2557) * add new encrypt/decrypt funcs and made necessary proto changes * address all PR comments * fix test panic * add param checking to rewraps * fix(schema): Correctly organize migrations again * Review comments part 1 * Review comments part 2 * Review comments part 3 * Store cert private key, encrypt tofu token (#2583) * fix(session): store cert private key, encrypt tofu token Previously, we would derive a session certificate private key from the session key used in each project, and not store the key. We would also encrypt the tofu token with the database key but not store a reference to this key in the database. This change fixes this by replacing our key derivation with a key generation step, and instead store the generated key, encrypted, in the database. We also ensure that any new tofu tokens are encrypted with the same key. To handle existing sessions, we lazily rewrite the sessions whenever a user lists them. There is a minor risk that a user could end up destroying the database key that encrypts the tofu token before that session has been rewrapped, but this is going to be rare and temporary. * feat(session): Allow the random source for secrets to be configured * Review comments * Minor fixes * More minor fixes * More small fixes Co-authored-by: Danielle Miu <dani.miu@hashicorp.com> Co-authored-by: Danielle <29378233+DanielleMiu@users.noreply.github.com> Co-authored-by: Michael Gaffney <mike@gaffney.cc> Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com> * fix(session): Unset session key ID when scope is deleted Unsetting the key ID allows the session to live beyond the lifetime of the scope, while allowing the scope to cascade delete its keys. * fix(migrations): Rewrite migration 56 tests These tests can no longer use the normal test helpers as they try to use the new oplog table structure, so we have to manually write all the interactions. * fix(migrations): Rename migration 58 file name This was the name used in the release branch, update it to the new migration number. * fix(e2e): Add some comments and debug to kms tests * fix(session): Handle empty project and user IDs in read When a project or user is deleted, the session is automatically canceled and the respective field on the session is unset. LookupSession did not handle this case gracefully, and would error on decryption in this state. Skip decrypting sessions that do not have either a user or project to avoid this error. The decrypted values are not used for a canceled session. * Fix worker test * Increase test timeout Co-authored-by: Danielle Miu <dani.miu@hashicorp.com> Co-authored-by: Danielle <29378233+DanielleMiu@users.noreply.github.com> Co-authored-by: Michael Gaffney <mike@gaffney.cc> Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com>	3 years ago
Johan Brandhorst-Satzkorn	f57454b6b9	Rewrite interface{} to any (#2535 ) * chore: Rewrite interface{} to any * Add ignore revs	3 years ago
Hugo Vieira	bed6e1a0ff	feat(server): Reload database on SIGHUP If the database url is updated on Boundary's config and then a SIGHUP triggered, we now connect to the new database, ensure it's OK to use and use that one going forwards instead. This allows a controller to be deployed in a more dynamic infrastructure where, for example, database credentials could change, without having to restart the controller. In this case only a database url config change and a SIGHUP signal to all controllers would be required. This uses an atomic.Pointer to store wrapped DB objects. This allows atomically swapping to the new object. A timeout is introduced so that if the object is swapped after the existing one has been looked up it isn't immediately invalidated. The timeout is a maximum; a ticker once a second that looks at stats to determine how many database connections are still in use and then closes the driver as soon as none are in use. This could be done with mutexes to avoid checking stats in a loop, but that carries its own risk: if a write lock is attempted to be acquired while a query is holding a read lock, the write lock will block, and any subsequent read locks will also block since write locks take priority. As a result, if a query is long-running or timing out, all other queries in the system will become blocked. Our default user request timeout is 90 seconds and some jobs have no timeout at all, so this is not a purely theoretical concern, even if it's unlikely. Refs: #2435 Co-authored-by: Jeff Mitchell <jeffrey.mitchell@gmail.com>	4 years ago
Jim	7aa7deafa3	chore: update go-dbw dependencies (#2298 ) * chore: update go-dbw dependencies This update included an update of the upstream gorm dependency. In the latest update, gorm appears to have become more sensitive when a slice is passed into operations. The ptr to slice must now be a ptr to slice of ptrs. This change required an update to how boundary handles slices of PrincipalRole, which now must return a ptr to PrincipalRole. There were also some tests that needed to be updated because of this change. * fixup! chore: update go-dbw dependencies	4 years ago
Timothy Messier	fd5d15af27	feat(config): Add database config options for idle connection management (#2176 ) This adds two new config options for the `database` stanza: - `max_idle_connections` - `max_idle_time` These correspond to settings for the underlying `sql.DB` instance. They can be used to have finer control over the controller's database connection pool. For example this can be used to mitigate the high connection overhead when connecting to postgres by setting a high `max_idle_connections` and either setting `max_idle_time` to zero or to a high duration. Alternatively, if using a connection pool like pgpool-II or pgbouncer, `max_idle_connections` can be set to 0. See: https://pkg.go.dev/database/sql#DB.SetMaxIdleConns https://pkg.go.dev/database/sql#DB.SetConnMaxIdleTime https://www.pgpool.net/mediawiki/index.php/Main_Page https://www.pgbouncer.org/	4 years ago
Timothy Messier	f3c90df300	chore(db): Silence db logs by default for tests This removes a lot of log clutter from normal test execution with `go test -v`. There are a number of test that expect no results from the database or other error cases from the database. The previous default log level of dbw.Error would cause these cases to generate logs in the success case, creating noise in the test output. The new default for tests is dbw.Silent. This can be overriden in two ways. A test can be updated to pass in the db.WithTestLogLevel option, which allows for targeting specific tests to include additional output. This may make sense for tests that normally should not cause errors, but would report an error if it fails. Alternatively a new environment variable can be set, DEFAULT_TEST_LOG_LEVEL. This allows for adjusting the log level without changing any test file. The environment variable should be set to the string value of the log level, and is treated as case insensitive, ie: export DEFAULT_TEST_LOG_LEVEL="error"	4 years ago
Jim	70c5da1048	feature (workers): add repository CreateWorker(...) (#2105 )	4 years ago
Johan Brandhorst-Satzkorn	7f9b294a7e	refact(all): Use testing.TB for test helpers Our test helper packages were not possible to use with benchmarks. This refactor switches to testing.TB to allow both tests and benchmarks to use the helpers.	4 years ago
Jeff Mitchell	7eb29261b2	Update to go-kms-wrapping version 2, and plugin-based KMS (#1901 )	4 years ago
Jim	591ec476e7	refactor: Update internal/db and oplog to use go-dbw package for database operations. (#1785 ) * chore: Add new dbw dependency * refactor: Convert to use dbw.DB * refactor (db): Change GetTicket(...) signature * feature (db tests): Added TestDeleteWhere(...) Adding this allows us to eliminate a leak of the gorm library * refactor: Add ctx to ScanRows(...) signature This is required to refactor all the deprecated errors pkg calls. This refactor required adding a context to quite a few call chains and in particular the job scheduler bits * feature (db/tests): Add support for test opt: WithTestLogLevel(...) This test only option allows callers to specify a log level for the underlying db package (where supported/applicable) when calling TestSetup(...) * refactor (db): Simplify Db type Only store the underlying DB and create the RW on demand when needed. * refactor (oplog): Port to using dbw pkg for db operations * refactor (db): Remove gormDB(...) func * tests (db): Add unit tests for TestDeleteWhere(...) * chore: make fmt changes to formatting	4 years ago
Jim	c047c4d89d	refactor (db): Remove the internal/db test tables from the migrations (#1675 )	4 years ago
Timothy Messier	8caaf3e076	refactor(schema): Add support for migration editions This refactors the schema package and subpackages to support editions. An Edition is a collection of sql migrations. A given boundary binary can have multiple migration editions. The `schema.Manager` is updated to apply the necessary migrations in the correct order, leveraging an Edition's priority to determine the order in which the separate Editions are applied. The sql migration files are now nested under a directory for the edition. This also switches from the generated code to using embed, which eliminates the need for the `make migrations` target and the generated code. As such this commit removes the target and generated code. See: https://pkg.go.dev/embed	5 years ago
Jim	8d6dee09a9	refact: Add db.DB wrapper and refact all test fixtures to use it. (#1535 ) * refact: Changes required for gorm v2 * refact: Add db.DB wrapper and refact all test fixtures to use it Stop leaking the gorm abstraction within our unit tests. * refact: Migrate direct calls to pq to pgx driver The refactor will remove the remaining direct dependencies on the pq driver which has been deprecated in favor of the pgx driver. It includes the addition of common.SqlOpen which is a wrapper around sql.Open, which is needed to deal with the driver name changing from "postgres" to "pgx" but in our code base the dialect and driver are synonyms.	5 years ago
Jim	136ac00b49	refact: Changes required for gorm v2 (#1528 )	5 years ago
Timothy Messier	75388f727a	test: Refactor to use dbtest package instead of docker This swaps the test initialization to use the dbtest package so each test will get a fresh database created from a template database, instead of a completely new docker container.	5 years ago
Louis Ruch	feb5d7385d	Louis job scheduler (#1197 ) * Adds job repo * Adds scheduler * Adds plugin table which is a place holder for future expansion of the scheduler. Currently all jobs that are created are linked to the pi_system plugin which are jobs that are held in a local registeredJobs map populated at controller startup.	5 years ago
Todd Knight	1f9fcad1cd	Adding Migrate Command (#872 )	5 years ago
Todd Knight	ec6151d174	Organize DB Schema Migration Code and DB Init Checks (#842 ) Create Schema Manager and propagate contexts into the DB calls downstream. Add checks at controller startup and database init for correct schema version and schema migration dirty bit being set.	5 years ago
Jeff Mitchell	813d21565f	Allow authorize-session to be invoked with target name (#737 )	6 years ago
Jeff Mitchell	c1a1f8bb1b	Separate out docker dep into package (#482 ) This also then removes support for docker-based containing starting (for now) from platforms other than windows/darwin/linux as there are build issues on several.	6 years ago
Jim	9570897032	basic keys mgmt repo (#264 )	6 years ago
Jeff Mitchell	6661117d4c	The name. The name. The name!	6 years ago
Todd Knight	f69d6b9952	Creating Auth Token Repo (#125 )	6 years ago
Todd Knight	bb4a185412	Pushing test db cleanup up to where the database objects are created using t.Cleanup. (#166 )	6 years ago
Jim	980a55a4e3	Added DeleteScope() with tests and TestVerifyOplog() for easier testing (#49 )	6 years ago
Jim	ee46266893	ErrRecordNotFound from LookupWhere() when no rows found (#59 )	6 years ago
Jeff Mitchell	f8b456f3a7	Add TestController (#50 ) This can be used in tests to get access to a controller that behaves just like a dev controller but doesn't require the normal Command bits (UI, etc). This can be enhanced over time as needed; for now it's pretty simple but it's good enough to be used for API tests. A simple project create/read API test is added as well, which serves to show the programmatic model (as well as test those functions at the API level).	6 years ago
Michael Gaffney	8d7f24268c	Return url of test database (#48 ) It is helpful to connect to a database when tests have failed.	6 years ago
Jeff Mitchell	7f2aa5ca8f	Add httpfs DB migrations and generate the input (#35 )	6 years ago
Jim	c43e05be2e	provide db.Open() and db.Migrate() as part of icu-64 (#34 )	6 years ago
Jim	27b4d7a034	basic internal db interfaces and Gorm implementation * db reader and writer interfaces which should allow us to encapsulate the vast majority of database code in this package. * GormReadWriter implementation of the Reader and Writer interfaces	6 years ago

34 Commits (deeac5c457e86c6c2bf6db50f00a385ecbf4a360)