boundary

Commit Graph

Author	SHA1	Message	Date
Jeff Mitchell	b76b24a4ad	Move prefixes for many packages into the globals package (#3069 ) This is a prerequisite for some enhancements to grant validation	3 years ago
Johan Brandhorst-Satzkorn	3c29308673	chore: Add license headers to all files	3 years ago
Jim	38b12394fd	refactor (kms): add GetWrapperer for RewrapFn parameter (#2701 ) * refactor (kms): add GetWrapperer for RewrapFn parameter * tests (kms): add MockWrapper and MockGetWrapperer Make it a bit easier to write unhappy path tests which depend on a Wrapper or GetWrapperer	3 years ago
Johan Brandhorst-Satzkorn	edd323b73a	Key Rotation/Destruction (#2477 ) (#2607 ) * Key Rotation/Destruction (#2477) * fix: Correct kms.CreateKeys comment This method does not return anything. * feat(kms): Add new ListKeys method The new ListKeys method wraps the underlying KMS wrapper, returning all the keys in the scope specified. * feat(scopes): Add new scope actions for key management Adds list-keys, rotate-keys and revoke-keys actions for scopes. * feat(scopes): Add ListKeys API endpoint This new endpoint lists all keys in a scope * feat(api): Add ListKeys to Scopes client This has to be a custom function because it is a custom action and doesn't map well to any of the existing templates, or generalises well in a way that would make it reasonable to create a new template. * feat(cli): Add new scopes list-keys command * add RotateKeys function to the kms repository * Add Rotate Keys Endpoint to Scopes Api (#2360) * add rotate keys endpoint * add tests for RotateKeys endpoint * remove that one comment I forgot about * addressing more PR comments * Add Rotate Keys CLI Command (#2395) * add cli command and client * add bats tests * fix some info text and pr comments * write a new rotate keys description * Add Missing DEK Foreign Keys (#2408) * add missing fks * mark key as nullable in gorm * update tests to use a new wrapper with a real key_id * fix formatting in test and add keys for auth_token test * replace key values * revert postgres_40_01_test.go * style: reformat sql for readability * refactor: change type of key_id columns to kms_private_id * Store key_id and scope_id with oplog entries, re-type columns referencing data key version (#2431) * fix(db): Correct types of data key version referencers * fix(oplog): Fix missing error handling * feat(oplog): Store key_id and scope_id with oplog entries This migration truncates all existing oplog entries, as migrating existing data would have been complex and likely unnecessary. * Update view references * Always delete oplog entries when dropping keys * Fix rebase issues * Fix sql tests * Add schema for new kms destruction jobs (#2479) * feat: Add schema for new kms destruction jobs The schema lets us manage destruction jobs per-table while providing a guarantee that only one run is running at a time. * Apply suggestions from code review Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com> * Fix table reference and tests Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com> * update domain, api, and cli to all include key versions in outputs (#2472) * update domain, api, and cli to all include key versions in outputs * fix go sum * add migration to fix immutable error * fix final test issue as well as make gen issue * rebase and fix * address johan pr comments * make gen, didn't realize comments were included * update migrations based on new migration 55 file 03 * Move migrations to 56 * feat: Add key version rewrapping function registry (#2478) The new RegisterTableRewrapFn can be used by a domain to register a callback to use when rewrapping data in a specific table name. * feat(kms): Add scopeId to RewrapFn (#2539) This makes it much easier for rewrap functions to look up the correct wrapper and limit the number of rows searched. * Update Root Cert Proto Definition (#2542) * update root cert proto definition to properly identify public_key as the table primary key * add the test fix as well * Rewrap Functions for Encrypted Tables (#2532) * add oidc rewrap * add argon2 config rewrap * add auth token rewrapping * add worker auth cert rewrapping * check the err * add username password credential rewrapping * add ssh private key credential rewrapping * add vault client certificate rewrap function * add host catalog secret rewrap function * cleanup a little * add host vault token rewrap function * add session credential rewrap function * add session rewrap function * add worker auth rewrap function * update test to include full assert gamut * add session rewrap function TEST * bring all tests up to current standards, make more readable, and include full assert gamut * rework all rewrap functions to utilize scope id * update comments, queries, sql refs, etc, in response to PR comments * forgot a comment * remove hmac updates and fix a couple comments again * Rewrapping Registered Tables Test (#2555) * add the registered table test as well as the two missing rewrap functions * address pr comments, add db r/w conversions, cmp.diff, sql comment, remove (now) faulty immutable test * Add ability to list key version destruction jobs, recurring jobs and destroying key versions (#2498) * feat(kms): Add ability to list data key version destruction jobs * feat(scopes): Add ListKeyVersionDestructionJobs * feat(cli): Add list-key-version-destruction-jobs * feat(kms): Add recurring job that performs table rewrapping The new recurring job runs for each job table with a registered rewrapping callback. It attempts to become the running run and start its rewrapping, gracefully handling sudden interruptions by resuming it's work if it finds evidence of sudden interruption. * feat(kms): Add recurring job for destroying key versions The new job monitors the database for data key version destruction jobs that have finished rewrapping all its data, performing the final data key version revocation. * feat(kms): Add DestroyKeyVersion DestroyKeyVersion immediately destroys a key version if it is a root key version or a data key version which encrypts no data. If the data key version encrypts data which needs to be rewrapped, it queues an asynchronous job to complete the rewrapping operation. * feat(scopes): Add DestroyKeyVersion API endpoint * feat(cli): Add scopes destroy-key-version CLI command * Ensure all secret updates include key ID (#2556) * feat(all): Ensure all secret updates include key ID When updating a secret, it is paramount that the key ID is also updated, as it is the only means we have of ensuring that we only destroy keys once there is no more data associated with the key. * Remove fix to session repository for now This breaks the current private key derivation method * add new encrypt/decrypt funcs and made necessary proto changes (#2557) * add new encrypt/decrypt funcs and made necessary proto changes * address all PR comments * fix test panic * add param checking to rewraps * fix(schema): Correctly organize migrations again * Review comments part 1 * Review comments part 2 * Review comments part 3 * Store cert private key, encrypt tofu token (#2583) * fix(session): store cert private key, encrypt tofu token Previously, we would derive a session certificate private key from the session key used in each project, and not store the key. We would also encrypt the tofu token with the database key but not store a reference to this key in the database. This change fixes this by replacing our key derivation with a key generation step, and instead store the generated key, encrypted, in the database. We also ensure that any new tofu tokens are encrypted with the same key. To handle existing sessions, we lazily rewrite the sessions whenever a user lists them. There is a minor risk that a user could end up destroying the database key that encrypts the tofu token before that session has been rewrapped, but this is going to be rare and temporary. * feat(session): Allow the random source for secrets to be configured * Review comments * Minor fixes * More minor fixes * More small fixes Co-authored-by: Danielle Miu <dani.miu@hashicorp.com> Co-authored-by: Danielle <29378233+DanielleMiu@users.noreply.github.com> Co-authored-by: Michael Gaffney <mike@gaffney.cc> Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com> * fix(session): Unset session key ID when scope is deleted Unsetting the key ID allows the session to live beyond the lifetime of the scope, while allowing the scope to cascade delete its keys. * fix(migrations): Rewrite migration 56 tests These tests can no longer use the normal test helpers as they try to use the new oplog table structure, so we have to manually write all the interactions. * fix(migrations): Rename migration 58 file name This was the name used in the release branch, update it to the new migration number. * fix(e2e): Add some comments and debug to kms tests * fix(session): Handle empty project and user IDs in read When a project or user is deleted, the session is automatically canceled and the respective field on the session is unset. LookupSession did not handle this case gracefully, and would error on decryption in this state. Skip decrypting sessions that do not have either a user or project to avoid this error. The decrypted values are not used for a canceled session. * Fix worker test * Increase test timeout Co-authored-by: Danielle Miu <dani.miu@hashicorp.com> Co-authored-by: Danielle <29378233+DanielleMiu@users.noreply.github.com> Co-authored-by: Michael Gaffney <mike@gaffney.cc> Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com>	3 years ago
Johan Brandhorst-Satzkorn	f57454b6b9	Rewrite interface{} to any (#2535 ) * chore: Rewrite interface{} to any * Add ignore revs	3 years ago
Johan Brandhorst-Satzkorn	aef9073fa6	Upgrade to Go 1.19 (#2347 )	4 years ago
Damian Debkowski	27e9f775be	feat: add initial migration hook	4 years ago
Jeff Mitchell	7364d84835	Add mechanism for plugin to normalize attributes (#2376 ) * Add mechanism for plugin to normalize attributes This introduces a new normalization call to the plugin RPC service. If implemented, the plugin will be given the attributes prior to saving them in a create/update call, and can perform any normalization needed, e.g. ensuring that a parameter passed in is represented in a canonical way. As an example, the CLI allows a user of the AWS plugin to pass in a string, but canonically it is a string slice, and should be returned that way as well. Not by chance, this is the exact behavior of the testing implementation of this function. This does not update any individual plugin, but rather sets the controller-side logic up for plugins to support this functionality.	4 years ago
Johan Brandhorst-Satzkorn	222a935223	Update grpc-gateway dependency (#2311 ) * chore(deps): Update grpc-gateway dependency This fixes an issue which would allow an attacker to trigger a (recovered) panic in the gateway handler. * fix(server): Use cmp.Diff instead of assert.Equal Testify is not great at handling protobuf types, and was for some reason returning a diff here when in reality there is none. Switch to go-cmp for the comparison to remove the error.	4 years ago
Jeff Mitchell	8c56a5648d	Migrate plugin host/set/catalog prefixes to typed (#2256 ) * Migrate plugin host/set/catalog prefixes to typed * Add credential store typing too	4 years ago
Johan Brandhorst-Satzkorn	f69fbf2255	feat(proto): Format protobuf files with buf (#2033 ) Formats all protobuf files with buf format. This ensures a consistent style across all our protobuf files, similar to gofumpt.	4 years ago
Johan Brandhorst-Satzkorn	7f9b294a7e	refact(all): Use testing.TB for test helpers Our test helper packages were not possible to use with benchmarks. This refactor switches to testing.TB to allow both tests and benchmarks to use the helpers.	4 years ago
Johan Brandhorst-Satzkorn	d51ce41db8	feat(hostsets): Classify all proto fields	4 years ago
Johan Brandhorst-Satzkorn	8fa3e8dee0	feat(hostcatalogs): Classify all proto fields	4 years ago
Johan Brandhorst-Satzkorn	5440195cce	feat(hosts): Classify host proto fields	4 years ago
Jim	addbfee593	chore: upgrade gofumpt to v0.3.1 (#2028 )	4 years ago
Johan Brandhorst-Satzkorn	b1d6a1da9a	Switch to buf for protobuf generation (#1944 ) * Install buf with go install This reduces the impact of using buf on our dependency closure, which was significant. It also avoids potential issues with building buf given our own internal dependency constraints. This has caused issues (including crashes!) in the past. See https://github.com/cockroachdb/cockroach/issues/67216#issuecomment-874176186 for more discussion on using tool dependencies generally, and https://github.com/bufbuild/buf/issues/52 for an example of where this sort of thing caused an issue for a user. * Upgrade buf version We should migrate to v1 sooner rather than later, as it has some breaking changes from v0. * Change buf lint/breaking commands These commands were moved up one level for v1 * Migrate to buf.yaml to v1 * Remove unused imports These were uncovered by using "buf lint". * Use buf v1.3.1 * Depend on googleapis and grpc-gateway via BSR * Switch to buf for proto generation * Remove third party dependency * Remove unused tools and scripts * Move local protos up one level * Temporarily don't fail on breaking changes * Generate CI config * Fix generation of controller.json * Replace use of deprecated flag * Remove unused import, regenerate files * Remove temporary skipping over breaking check	4 years ago
Jeff Mitchell	7eb29261b2	Update to go-kms-wrapping version 2, and plugin-based KMS (#1901 )	4 years ago
Louis Ruch	4ed4575802	chore(fmt): make fmt	4 years ago
Todd	7ee1f74388	fix(hostsets/plugin): Only delete preferred endpoints when there are preferred endpoints to delete. (#1865 )	4 years ago
Justen Walker	6166d2240f	bug(plugin): Fix incorrect use of WithIpAddrs in Repository.Endpoints (#1849 ) When creating the options for the preferencer, this used WithIpAddrs for both the host addresses and also DNS names. This causes errors when a host contains DNS names that are not IP addresses (ie: whenever dns names are present). This change will: - Replace WithIpAddrs with WithDNSNames option for DNS Names when constructing options for the preferencer. - Add a test which includes a DNS name to prevent regressions Fixes #1848	4 years ago
Louis Ruch	aaaf0241ff	bug(host): Fix sync job failure blocking other set syncs (#1841 ) During the plugin_host_set_sync job, a single set failing to sync can cause all following syncs to fail.	4 years ago
Haotian	fa00a06bef	feat(host): Add host set information to static host (#1828 ) * feat(host): Add host set information to static host Updates LookupHost and ListHost API to include sets a static host is part of, as like with plugin hosts. * test: updated TestList_Plugin and TestCrud to be more consistent under race conditions	4 years ago
Louis Ruch	c4506ec743	feat(scheduler): Add support for run now option	4 years ago
Jim	591ec476e7	refactor: Update internal/db and oplog to use go-dbw package for database operations. (#1785 ) * chore: Add new dbw dependency * refactor: Convert to use dbw.DB * refactor (db): Change GetTicket(...) signature * feature (db tests): Added TestDeleteWhere(...) Adding this allows us to eliminate a leak of the gorm library * refactor: Add ctx to ScanRows(...) signature This is required to refactor all the deprecated errors pkg calls. This refactor required adding a context to quite a few call chains and in particular the job scheduler bits * feature (db/tests): Add support for test opt: WithTestLogLevel(...) This test only option allows callers to specify a log level for the underlying db package (where supported/applicable) when calling TestSetup(...) * refactor (db): Simplify Db type Only store the underlying DB and create the RW on demand when needed. * refactor (oplog): Port to using dbw pkg for db operations * refactor (db): Remove gormDB(...) func * tests (db): Add unit tests for TestDeleteWhere(...) * chore: make fmt changes to formatting	4 years ago
Chris Marchesi	bf18555d5e	internal/host/plugin: remove custom queries for secrets (#1676 ) * internal/host/plugin: remove custom queries for secrets This removes the custom queries for host catalog secrets, now that #1658 has been merged. It also does a lot of cleanup surrounding the related oplog entries, using db.WithOplog natively. * Simplify conflict actions by replacing SetColumnValues with SetColumns * Remove unused oplog method	4 years ago
Chris Marchesi	84febd9983	internal/host/plugin: ensure UpdateCatalog triggers updates for sets (#1736 ) * internal/host/plugin: ensure UpdateCatalog triggers updates for sets This ensures that attribute updates to a host catalog triggers an update to any of its downstream host sets. This ensures that updates to elements that affect a host set, such as the region a host catalog is configured for, take effect immediately instead of requiring a wait of long as 10m after an update. * Add WithNextRunIn to test sync job This is to ensure it doesn't run right away when added. This ensures that we can accurately test when the job is set to run immediately. * Update internal/host/plugin/repository_host_catalog.go Co-authored-by: Todd <T.Alan.Knight@gmail.com> * Change TestRepository_UpdateCatalog_SyncSets to check next run time, not wait for run * Remove explicit version increment and instead rely on implicit increment * Fix lanuage for rows updated check when flagging host sets for sync * remove errant non-zero check when looping over the thing we're checking * remove scheduler start line from TestRepository_UpdateCatalog_SyncSets Co-authored-by: Todd <T.Alan.Knight@gmail.com>	4 years ago
Jeff Mitchell	5417cc5cb0	Fix setting nil attributes on host sets (#1746 ) This is the analogue of #1743 but for host sets	4 years ago
Jeff Mitchell	1c39691023	Fix setting host catalog attributes to nil (#1743 ) Internally attributes are not allowed to be the null value. However, as currently implemented, this then breaks our normal model of sending an explicit `null` in the JSON to reset/clear a field. This is a problem because the CLI doesn't have a way to say "set this but to empty" without a lot of hoops, and causes some issues for API users and TF as well. This changes PatchBytes and PatchStruct to return an empty value (or marshaled empty value in the case of PatchBytes) if the source is null or empty. In the host catalog update function we then always set this value if it's in the db mask (removing the nil check). Together this means a null value ends up coming out as an empty value (and thus valid for the DB) from PatchStruct/PatchBytes, and our API translation function already returns nothing for attributes if the length of the fields are zero.	4 years ago
Chris Marchesi	9e976cb4ae	internal/host/plugin: Fix TestUpdateSet (#1735 ) Now asserts on error on the last lookup check and removes the plugin lookup since we're not asserting on that.	4 years ago
Todd	18de3653f5	Set sync job still syncs when plugin reports no hosts in sets (#1726 ) * Allow the set sync job to sync when no hosts are returned by the sync job.	4 years ago
Jeff Mitchell	866bfceefc	Add secrets_hmac to host catalog output (#1712 ) Co-authored-by: Jim <jlambert@hashicorp.com>	4 years ago
Jeff Mitchell	8110e0eab3	Increment version of host during updates (#1707 ) Currently host does not use a version, which means it's not protected against race conditions of either itself or its value objects. This adds and plumbs through version to ensure that if a host or its value objects change that they are not going to be affected by a race condition.	4 years ago
Jeff Mitchell	b3fd23863f	Add external ID to host output (#1705 ) This also fixes a bug if no hosts were returned where the job would run over and over in a loop, and some other cleanup.	4 years ago
Todd	78dc6cf370	OnConflict constraints for adding host ip and dns names in upsertHosts (#1700 ) * Ignore constraint violations when adding ip addresses and dns addresses that already exist.	4 years ago
Todd	5d2b0e94aa	Update sync interval seconds field. (#1698 ) * Update sync interval seconds field. * If the sync interval was updated, schedule the next sync job wakeup immediately. * Calculate the next sync time for the scheduler should wake up if we only updated the sync interval.	4 years ago
Todd	b16ce175e5	Fix secrets clearing bug in update. Changes to update count behavior for catalogs (#1691 ) * Fix secret clearing bug in update. Always return 1 catalog updated if we can find a catalog w/ a matching version.	4 years ago
Todd	5cfcc41f00	Fix bug for adding preferred endpoint when non existed prior. (#1692 )	4 years ago
Todd Knight	f41bceda83	Merge from main and `make gen`	4 years ago
Todd	61f9d1f6af	Add update set handler and sdk functionality (#1684 )	4 years ago
Todd	29b4537d3b	Reference PreferredEndpoints by top level struct name (#1688 ) * Reference PreferredEndpoints by top level struct name instead of embedded proto GetPreferredEndpoint method. Reorganize the sql files a little. * Make PreferredEndpoints replayable. Test refactoring to remove variables that persist across test calls. DeleteItems instead of trying to use WithWhere.	4 years ago
Jeff Mitchell	30e9f944a2	Add sync interval to host sets (#1680 ) This adds a parameter to host sets allowing the sync interval to be user-specified on a per-host-set basis. A value of -1 disables syncing; a value of null uses the current default.	4 years ago
Todd Knight	dba7c6b531	Fix value of op constant for syncSets	4 years ago
Chris Marchesi	c2cd6febca	internal/host/plugin: add UpdateSet (#1686 ) * internal/host/plugin: add UpdateSet This adds the UpdateSet repository function. UpdateSet runs the OnUpdateSet plugin call within the transaction as per the changes in #1683. The plugin call returns the updated set, the hosts in the set, the rows updated, and any error. * Ensure that attributes can be cleared by supplying a full null * move version increment to the end and track perferred endpoint update separately * move plugin call to end of transaction * only clone returned set once, and move host fetch call out of transaction * Request a sync when attributes are updated * Add test to assert that old preferred endpoints are being removed correctly	4 years ago
Todd	2229d1a819	Call OnCreate or OnUpdate on the plugin after db constraints are checked (#1683 ) * Only call the Plugin at create/update time after the db has been partially updated. This allows more constraints to be checked prior to calling the plugin.	4 years ago
Todd	050553ffcf	Add update catalog support for plugin based catalogs in handler and go sdk (#1673 ) * Create catalog service handler for updating plugin. Add tests for handler and run make fmt. Silently drop attributes where the value is nil when creating the catalog or set.	4 years ago
Chris Marchesi	157f1a091b	internal/host/plugin: add update catalog test for empty/same description (#1672 ) This is to mainly get coverage on the empty description case, but it follows the same logic as the "update name to same" test.	4 years ago
Chris Marchesi	0670e272e3	internal/host/plugin: Add UpdateCatalog (#1634 ) * internal/host/plugin: Add UpdateCatalog This adds support for updating plugin-based host catalogs. The workflow is: * The existing catalog is fetched. * Any attribute changes are patched against it, along with other changes such as name and description. * Both the existing catalog, and the state of the catalog as it will exist post-update, along with any persisted secrets, are sent to the plugin via an OnUpdateCatalog request. Upon success, the following is done: * The record is updated if non-secret updates need to be made. * Secrets are updated if they still exist. * If an empty secret set was sent back, secrets are instead deleted. * If the secrets field was instead null, secret update is skipped altogether. There has also been a lot of helper code added to the tests to help make the update tests more readable and reduce boilerplate. While the create tests have not been updated to use these new helpers, it might be a good idea to do so for consistency. * move secret/persisted state closer to plugin call * Update internal/host/plugin/repository_host_catalog.go Co-authored-by: Todd <T.Alan.Knight@gmail.com> * Update internal/host/plugin/repository_host_catalog_test.go Co-authored-by: Todd <T.Alan.Knight@gmail.com> * move all test helpers for UpdateCatalog inline * WIP - transparent attributes struct/wire conversion * Revert "WIP - transparent attributes struct/wire conversion" This reverts commit `31af1b1b7e`. * deal with multiple attribute masks * Note TODO for upsertQuery for HostCatalogSecret * add version check guard * Update internal/host/plugin/host_catalog_secret.go Co-authored-by: Todd <T.Alan.Knight@gmail.com> * remove redundant JIRA reference * don't separate number of updated catalogs and secrets * increment version when secrets is the only thing being updated * add another test for updating both catalog and secrets at once * correct comment for mock plugin manager section * add first-part path matching for secrets * update plugin lookup error to return errors.Internal * only add name/description to plugin request if they are non-empty * update duplicate tests to use two project scopes * remove unused test checks * return plugin record along with host catalog in update * move away from using passed in catalog after patching newCatalog * use t.Cleanup() instead of defer Co-authored-by: Todd <T.Alan.Knight@gmail.com>	4 years ago
Chris Marchesi	165cc5ef1b	fix ineffectual assignment to err in DeleteCatalog (#1664 ) Was being caught by my linters. Looks like the error wasn't being checked after getCatalog.	4 years ago
Chris Marchesi	1bc9a07316	populate correct field for host catalog secret (#1661 )	4 years ago

1 2

73 Commits (b6d58d0c42ea197f040c54e468ec284ec0023054)