131 Commits (08ecadb2670c81555ae48e80093ced08c12e2eac)

Author	SHA1	Message	Date
Jeff Mitchell	fff5f9506f	Migrate proxy logic to the API package (#4216) ... This allows arbitrary programs to have an easy mechanism to make connections through Boundary using Go. Internal cluster tests and test helpers have been migrated to use this package, as well as the `connect` command. --------- Co-authored-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>	2 years ago
Timothy Messier	a77c88ae5d	feat(ratelimit): Increase default values (#4129) ... This increases the default rate limits to: - 1,500 reqs / 30 seconds for list actions in total - 1,500 reqs / 30 seconds for list actions per IP address - 150 reqs / 30 seconds for list actions per auth token And for all other actions: - 30,000 reqs / 30 seconds for list actions in total - 30,000 reqs / 30 seconds for list actions per IP address - 3,000 reqs / 30 seconds for list actions per auth token This seems to be a reasonable default for a small to medium size cluster based on some scale testing. (cherry picked from commit 237c8b6084)	3 years ago
Timothy Messier	6e233c879f	fix(ratelimit): Round up when setting Retry-After header ... Previously this would truncate to the nearest integer, which could result in clients retrying before the quota has reset, and getting rate limited again. This also includes an update from go-rate that will round up when reporting the reset time for the RateLimit header. Note that in the case of a 503, the Retry-After header was already being set correctly. See: https://github.com/hashicorp/go-rate/pull/8	3 years ago
Timothy Messier	9438be4506	feat(ratelimier): Support disabling rate limiting	3 years ago
Timothy Messier	0bc423ebd8	feat(controller): Reload api rate limits on SIGHUP	3 years ago
Sorawis Nilparuk (Bo)	45eb3b7b51	support remote-dev format for cluster ID (#3637)	3 years ago
Jeff Mitchell	20ac42c851	Add some useful dev flags (#3865)	3 years ago
Jeff Mitchell	b64eadedbf	Fix flaky TestServer_ReloadInitialUpstreams (#3758) ... With the KMS changes this should never have worked at all (because the different databases means the credentials conveyed after the KMS-based authentication would never be valid). However, it was sometimes passing because of a race condition: the worker could be found in the database before the new credentials were returned to the worker, causing the worker to start at the KMS step with the second controller. This changes the test to use the same database for two controllers, but to only ever have one running at a time, and uses a sequence of two changed (or unchanged, depending) status updates from the worker to ensure that the reload was successful.	3 years ago
Jim	0d19892e03	fix (events): convert multierror to stdlib error (#3743) ... * fix (events): convert multierror to stdlib error multierror errors will be converted to a stdlib error (because multierror doesn't support json marshaling). * refact (errors): convert from multierror to stdlib errors.Join(...) * refact: convert from pkg/errors to stdlib error wrapping	3 years ago
Irena Rindos	7bd2d2c913	feat(worker): allow hcp cluster id to be sourced from env var (#3709) ... * feat(worker): allow hcp cluster id to be sourced from env var	3 years ago
Irena Rindos	9733a250fa	refact(event): move observability/event to event (#3704)	3 years ago
Johan Brandhorst-Satzkorn	f89515a19c	all: avoid mutating global state in parallel tests (#3687) ... The SetupSuiteTargetFilters globally mutates the function used for logic inside the target handler service. This means that we could get inconsistent logical behavior in certain tests that were run in parallel, since the logic can change at runtime. Ensure tests that make use of this function are never run in parallel with other tests.	3 years ago
hashicorp-copywrite[bot]	29da0bcb92	[COMPLIANCE] License changes (#3567) ... * Adding explicit MPL license for sub-package. This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository. * Adding explicit MPL license for sub-package This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository. * Updating the license from MPL to Business Source License Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at www.hashicorp.com/licensing-faq, and details of the license at www.hashicorp.com/bsl. * Update copyright file headers to BUS-1.1 * Rerun make gen This will pick up the last of the license changes * Revert "Rerun make gen" This reverts commit dbe0968bff. * Remove copywrite from generation script The current version of copywrite is not sophisticated enough to only apply the copywrite changes to certain subdirectories, so to avoid accidentally relicensing our api and sdk modules, we will stop running the copywrite script until such time that it can do the right thing. * License protobuf API as MPL The protobuf API files are used to generate the SDK, since we want the SDK to be MPL, the protobuf files must also be MPL. This extends to all protobuf files whose sources end up in the SDK directory, including: - internal/proto/controller/api - internal/proto/controller/custom_options - internal/proto/plugin It also includes the generated service directory in internal/gen/controller/api/services, since they are generated from the protobuf sources used for the SDK. * Change website license file name This makes the naming scheme consistent with the rest of the repository. * Add note on licensing to README * Set copywrite license to BUSL * Revert "Add note on licensing to README" This reverts commit 4d865a91a7. --------- Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com> Co-authored-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>	3 years ago
Jeff Mitchell	8322916fb4	Remove all deprecated Err/Wrap/E calls (#3286)	3 years ago
Louis Ruch	798a1f952a	feat(storage): Add support for storage plugins	3 years ago
Jeff Mitchell	77fd51fb12	Add more logging to worker auth rotation and test flags ... This makes it much easier to figure out when and why rotation is happening and control it for tests/debugging. Remove cert skew Cherry-picked from release/0.12.x	3 years ago
Jim	667ea285be	feature (boundary dev): add support for dev ldap auth method (#3192)	3 years ago
Jeff Mitchell	163ce184b8	KMS-PKI Workers (#3101) ... This adds a new mechanism for worker authentication that uses the third-party arbitration of the original KMS flow by leveraging recently-added capabilities in nodeenrollment to support a registration wrapper. This allows for workers to be registered to the system with just a name and a matching wrapper, while at the same time allowing these workers to be used for private Vault access and multi-hop as they contain per-worker encryption keys and support rotation. Nodes using this mode will generate and rotate credentials in-memory (using another recently added nodeenrollment capability) so will have a new set of keys on every startup. This has the side effect of ensuring that there is never a conflict or re-use of these credentials. Note that the normal test worker flow was using the now-deprecated KMS method; there is still a test that uses this (explicitly) but all other tests using test workers that did not specify a specific PKI flow now actually use this mechanism. This means there is fairly decent test coverage in a general sense, plus the specific tests updated/added for this feature. This also adds a new KMS type that can be used to have distinct KMSes for upstream authentication vs. accepting from downstream.	3 years ago
Jeff Mitchell	b7d810e475	Fix function signature (#3070)	3 years ago
Jeff Mitchell	b2bab45c13	Port over changes (#3061) (#3062)	3 years ago
Eng Zer Jun	6403b0f389	test: use `t.TempDir` to create temporary test directory (#2922) ... This commit replaces `os.MkdirTemp` with `t.TempDir` in tests. The directory created by `t.TempDir` is automatically removed when the test and all its subtests [complete. Prior to this commit, temporary directory created using `os.MkdirTemp` needs to be removed manually by calling `os.RemoveAll`, which is omitted in some tests. The error handling boilerplate e.g. defer func() { if err := os.RemoveAll(dir); err != nil { t.Fatal(err) } } is also tedious, but `t.TempDir` handles this for us nicely. Reference: https://pkg.go.dev/testing#T.TempDir Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	3 years ago
Johan Brandhorst-Satzkorn	3c29308673	chore: Add license headers to all files	3 years ago
Irena Rindos	a030c15403	Support HCPBInt in hcpb_cluster_id field (#2880) ... * initial commit	3 years ago
Jeff Mitchell	19180af0eb	Fix target port handling (#2846) ... * Fix target port handling This fixes two issues that compounded on each other (see the Changelog update for more information): * The verification logic for hosts was not correct for update operations (in multiple ways) which meant that a host could be updated after creation to have a port. Targets had a previously fixed bug where they did not require a default port, which meant that ports could be used from hosts * A recent (unreleased) change had prioritized any port coming from the host over the default port, which would mean there was no way if a host had a port specified to use it in multiple targets. This fixes the update verification logic, and strikes a middle ground between breaking things and not by allowing existing addresses with ports to be used with targets but ignoring that port, instead requiring targets to have default port set at authorize time (currently there is backwards compat that does not require this due to the original optional port bug). * Update CHANGELOG.md Co-authored-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>	3 years ago
Irena Rindos	af24972386	warn if controller and worker name are the same (#2773)	3 years ago
Hugo Vieira	f85da3ee2b	feat(cmd): Create target using address on boundary dev / database init ... Creates a target using an address for boundary dev and when first init-ing boundary (boundary database init). It also makes this newly created target the default one (ttcp_1234567890). The target using host sources is now a secondary target (ttcp_0987654321).	3 years ago
Irena Rindos	9135cc7668	In absence of ingress filter, use directly connected worker (#2757) ... * directly connected ingress filtering	3 years ago
Johan Brandhorst-Satzkorn	df02501dcc	Create new "previous-root" key purpose for root key migrations (#2639) (#2648) ... * Update go-kms-wrapping dependency * Apply changes suggested by Jeff * Fix listener test * Add note about config fix to CHANGELOG * Review comments Co-authored-by: Dan Heath <76443935+Dan-Heath@users.noreply.github.com> Co-authored-by: Dan Heath <76443935+Dan-Heath@users.noreply.github.com>	4 years ago
Jeff Mitchell	0c96c6ff6c	Split grace period into multiple config values (#2578) (#2603) ... Co-authored-by: Todd <github@quaddi.com>	4 years ago
Michael Li	067530cdbf	fix(test): Retry websocket connection due to flaky failure (#2584) ... We have observed several unit tests occasionally fail when trying to read a handshake due to the following error: failed to read protobuf message: failed to get reader: received close frame: status = StatusProtocolError and reason = "received header with unexpected rsv bits set: false:false:true" After an investigation, it's still unclear how this happens. We are opting to just retry to mitigate the flakiness.	4 years ago
Johan Brandhorst-Satzkorn	f57454b6b9	Rewrite interface{} to any (#2535) ... * chore: Rewrite interface{} to any * Add ignore revs	4 years ago
Timothy Messier	d896f37493	Support reading multiple configuration files(#2504) ... feat(config): Support multiple files This updates the three subcommands that take a `-config` flag to to support multiple config files via a repeated `-config` flag. It updates the following subcommands: - `server` - `database init` - `database migrate` Note that the multiple file support only works if all config files are HCL, not JSON. Also note that the config files are concatenated together. There is no attempt to merge config files, so having the same stanza in multiple files can yield unexpected results, with only one of the stanzas being used. When used with these limitations understood, this provides a way to take different actions for different kinds of dynamic configuration changes. An example would be when using consul-template to render the configuration files for boundary. With the multi file support consul-template can render one config file for the controller stanza, and provide boundary with a dynamic database credential via vault. Then when the credential is rotated, consul-template can send boundary a SIGHUP to switch to the new credential without a restart. A separate template could be used to render a config for a stanza that does not support SIGHUP, such as the events stanza that gets its values via consul's KV. If these values change, consul-template can restart boundary so the new configuration is used. Co-authored-by: Timothy Messier <tim.messier@gmail.com> Co-authored-by: Jeff Mitchell <jeffrey.mitchell@gmail.com>	4 years ago
Todd	06fd086004	Add health endpoint for worker (#2442) ... * Return worker process information in the health endpoint. Experimental: The worker_info and returned worker_process_info fields are experimental and can change or be removed without notice. When a user passes in "worker_info=1" or "worker_info=true" and a worker is running the returned json will be returned with a key named "worker_process_info" which value is a json object with information about the worker process.	4 years ago
Irena Rindos	d951e1ebc1	Worker graceful shutdown (#2455) ... * add graceful shutdown to worker	4 years ago
Jeff Mitchell	9002930d65	Adapt shared lock skip logic to dbswap branch (#2437)	4 years ago
Hugo Vieira	bed6e1a0ff	feat(server): Reload database on SIGHUP ... If the database url is updated on Boundary's config and then a SIGHUP triggered, we now connect to the new database, ensure it's OK to use and use that one going forwards instead. This allows a controller to be deployed in a more dynamic infrastructure where, for example, database credentials could change, without having to restart the controller. In this case only a database url config change and a SIGHUP signal to all controllers would be required. This uses an atomic.Pointer to store wrapped DB objects. This allows atomically swapping to the new object. A timeout is introduced so that if the object is swapped after the existing one has been looked up it isn't immediately invalidated. The timeout is a maximum; a ticker once a second that looks at stats to determine how many database connections are still in use and then closes the driver as soon as none are in use. This could be done with mutexes to avoid checking stats in a loop, but that carries its own risk: if a write lock is attempted to be acquired while a query is holding a read lock, the write lock will block, and any subsequent read locks will also block since write locks take priority. As a result, if a query is long-running or timing out, all other queries in the system will become blocked. Our default user request timeout is 90 seconds and some jobs have no timeout at all, so this is not a purely theoretical concern, even if it's unlikely. Refs: #2435 Co-authored-by: Jeff Mitchell <jeffrey.mitchell@gmail.com>	4 years ago
Hugo Vieira	8ab9ffbcac	refact(cmd): Encapsulate some functionality on Command and Server ... This commit extracts 3 separate pieces of functionality into their own function/method: - Extracts the database opening with the intention to have a function that performs all that functionality but returns the created database object rather than setting it to the Server structure. - Extracts database validation logic, to provide a common code path for upcoming functionality. - Expose schema manager on the Command struct as well as providing common functions to acquire shared lock/close.	4 years ago
Hugo Vieira	621f8a9ff2	fix(schema): Database connection not cleanly closed ... This commit introduces closing the Schema Manager's underlying driver connection. We were only releasing the shared lock but not actually cleaning up the db connection and relied on the fact that the process would exit to do that for us.	4 years ago
Timothy Messier	047a66e87d	fix(worker): Improper reload when running as controller and worker (#2438) ... In cases where the server was started with both a controller and worker stanza, the reload of the config was not properly setting the initial upstream value on the new config. In this case the value is derived from other config values and not necessarily set directly. This now properly initializes the new config so that initial upstream is set in the same manner as the first load. Blame: 5d0cdf680f	4 years ago
Jeff Mitchell	16c9e9b592	Add ability to skip shared lock acquisition	4 years ago
Jeff Mitchell	01fb949d0b	Add controller-led worker auth flow (#2413) ... This adds an action to the API that allows pulling out an activation token that can be given to a worker in its config (directly or via env or file). The worker can pass this along with its normal request in place of its normal nonce and the server will automatically accept it. Co-authored-by: Irena Rindos <irenarindos@users.noreply.github.com>	4 years ago
Timothy Messier	5d0cdf680f	feat(worker): Support reloading of initial upstreams on SIGHUP (#2417) ... On a SIGHUP, if the worker's `initial_upstreams` config option has changed, it will use the new values as the address to send its status. Any previously discovered upstream addresses from previous status requests are dropped. Once a successful status request is made to the new addresses, the worker will resume its normal behavior of using the addresses from the response to the status request. This allows a worker to be deployed in a more stable way in a dynamic environment where controller IPs could change. In particular this helps with a case where all controllers are replaced with new controllers running on new IPs. To illustrate with an example, suppose there is an instance group of controllers instances. And there are worker instances that have their config rendered from consul-template that sets `initial_upstreams` to the IPs of the controller instances. If all controller instances are restarted or replaced at the same time and assigned new IPs, a new config will be rendered for the worker. Previously the only way for the worker to use the new value would be for the worker process to be restarted. Otherwise the worker would only know about the previous IPs and would fail to connect to the new controllers. However, a restart would mean any active sessions and session connections on the worker instance would be terminated. With this update, consul-template can send a SIGHUP to the worker, the worker will connect to the new controllers and resume normal status reporting, allowing the sessions and session connections to remain. The same principles would apply for other examples of dynamic environments such as nomad and kubernetes. See: https://github.com/hashicorp/consul-template https://www.nomadproject.io/docs/job-specification/template	4 years ago
Louis Ruch	dcccaf1bcf	bug(cmd): Ensure RunShutdownFuncs is called when an early error occurs (#2333) ... * bug(cmd): Ensure RunShutdownFuncs is called when an early error occurs	4 years ago
Jeff Mitchell	8800ec9c94	Reorganize logic to allow a poison pill (#2317)	4 years ago
Louis Ruch	fc25178827	bug(server): Start ops listener even if controller is nil (#2314)	4 years ago
Jeff Mitchell	5117cf9548	Add some safety checks in server command (#2288)	4 years ago
Jeff Mitchell	7ab7c8eaf4	Have fallback eventing use json format (#2216) ... * Have fallback eventing use json format The server will initialize a logger via InitFallbackLogger(...) after setting up the server's logging (and honoring the related cmd flags). This fallback logger will be used if when no eventer is initialized. If no fallback logger is initialized, then a event.defaultLogger() will be used if no eventer is defined. Co-authored-by: Jim <jlambert@hashicorp.com>	4 years ago
Timothy Messier	fd5d15af27	feat(config): Add database config options for idle connection management (#2176) ... This adds two new config options for the `database` stanza: - `max_idle_connections` - `max_idle_time` These correspond to settings for the underlying `sql.DB` instance. They can be used to have finer control over the controller's database connection pool. For example this can be used to mitigate the high connection overhead when connecting to postgres by setting a high `max_idle_connections` and either setting `max_idle_time` to zero or to a high duration. Alternatively, if using a connection pool like pgpool-II or pgbouncer, `max_idle_connections` can be set to 0. See: https://pkg.go.dev/database/sql#DB.SetMaxIdleConns https://pkg.go.dev/database/sql#DB.SetConnMaxIdleTime https://www.pgpool.net/mediawiki/index.php/Main_Page https://www.pgbouncer.org/	4 years ago
Jeff Mitchell	a589c32269	Minor capitalization change	4 years ago
Jeff Mitchell	32d04b9ce7	Allow a worker to use KMS auth but accept PKI for proxy (#2206) ... This essentially worked for non-dev mode before but would create storage directories that weren't actually used or needed, and it didn't work in dev mode because it made assumptions about the presence of a worker auth kms or not. This fixes those behaviors.	4 years ago