2661 Commits (69fc3a47d07969c3e29e1fcd6a9b6d818052d47a)

Author	SHA1	Message	Date
Michael Gaffney	e086fd3007	test(batch): Fix data races in tests ... Fixes: 9b984b0ab6	1 year ago
Johan Brandhorst-Satzkorn	a2be306eb4	More managed groups limit fixes (#5245) ... * internal/handlers/managed_groups: ensure membership lookup is unlimited Previously, we would only show the first 10,000 members in the group, giving the impression that a complete list was performed. Now we list all members. * internal/handlers/accounts: ensure group lookup is unlimited Previously, we would only show the first 10,000 managed group memberships. Now we list all group memberships of an account.	1 year ago
Johan Brandhorst-Satzkorn	d830bdd416	internal/auth/oidc: ensure all managed groups are tested on auth (#5242) ... Previously, we would list the managed groups with an implied filter of db.DefaultLimit (10,000), which would incorrectly remove a user from managed group memberships if there were more than 10,000 managed groups in an auth method. Explicitly set the list limit to unlimited to ensure all managed groups are updated appropriately.	1 year ago
Elim Tsiagbey	514bcc19f6	feat(gcp): add GCP plugin (#5229) ... Add GCP as a supported plugin for dynamic host catalogs	1 year ago
Hugo	f227020b67	feat(scheduler): Use job table to select jobs to run ... This commit changes the logic around selecting jobs to run. Before this, Boundary used a view "job_jobs_to_run" which was a collation of all jobs that were not running and scheduled to run. When jobs were requested to run, Boundary would look at this view, grab a certain amount of jobs and insert them into the "job_run" table. Importantly, while the amount of jobs to be scheduled at once defaulted to 1 in the scheduler package, this was entirely overriden in the controller package to -1 (no limit, run all jobs that the query would return). With this commit, Boundary now looks at the "job" table directly for jobs to run, regardless of job run status. This was done to improve DB performance as the view (which queried all "job_run" rows) was causing significant DB contention. Additionally, the option to limit job runs is now removed since it was already hardcoded to -1 and not user-configurable anyways. Should the need arise to implement this limit again, this will have to be done in a different way since the new query does not account for running jobs and limiting the rows returned after jobs are already running would yield a no-op (the new query inserts nothing into "job_run" if any given job is already running).	1 year ago
Hugo	27c432a748	feat(scheduler): Drop completed job runs as they complete ... Before this change, upon the completion of a job run, the scheduler would update a job_run entry with the completed status as well as the number of objects affected by the run. Additionally, we then had a cleaner job whose scope was deleting these completed entries from job_run. This commit changes this design to drop the cleaner job the and rolls its functionality directly into the scheduler. Now, when any given job run is complete, the scheduler will automatically delete it from the job_run table itself. In addition, the completed status has been removed as a valid status as it is not set anymore. This change follows a decision that given the data in these job_run entries is already being deleted by the cleaner job, updating its state when a job completes just to have it deleted is not efficient. Additionally, no use-case for this data was surfaced.	1 year ago
dani	62c41191ff	[fix] allow IPv6 controllers, workers, and targets (#5221) ... Co-authored-by: Damian Debkowski <damian.debkowski@hashicorp.com>	1 year ago
Michael Gaffney	9529491fa6	feat(session): Delete terminated sessions in batches ... This changes the periodic job that deletes terminated sessions to delete the those sessions in batches. Prior to this, a single SQL DELETE command was used to delete all qualifying terminated sessions. In systems under heavy load, the statement could timeout leading to a situation where no qualifying terminated sessions were ever being deleted.	1 year ago
Michael Gaffney	9b984b0ab6	feat(batch): Add batching for SQL jobs ... This adds a new `batch` package under the `scheduler` package.	1 year ago
Michael Gaffney	8c291a2934	feat(scheduler): Add retries to job status ... This adds retries as an attribute to JobStatus allowing a job to indicate liveness to the scheduler when work has not progressed. A job could use this and the statusThreshold passed into the Run method to break work up into batches that would otherwise timeout if all the work was attempted at once, for example a long running SQL delete command.	1 year ago
Michael Gaffney	9354a4ebcf	refactor(scheduler): Add statusThreshold parameter to job.Run ... The scheduler monitors running jobs to make sure they are not hung. If the status of a running job does not change after a duration (the interruptThreshold in the scheduler), the scheduler kills the job. This change passes the scheduler's interruptThreshold to the job when it calls Run() on the job.	1 year ago
Michael Gaffney	50ab600519	docs(scheduler): Clarify Job and JobStatus ... This adds more details to the Job interface and the JobStatus struct to clarify how the scheduler uses them.	1 year ago
Irena Rindos	36158056d9	feat(workers): add VerifyKnownWorkers repo function (#5189)	1 year ago
Irena Rindos	dfde64e648	chore(server): fix const ops (#5190)	1 year ago
Jim	ffd0d2a7d3	feat (cache): impl. soft-delete of users in cache (#5173)	1 year ago
Irena Rindos	7798906e4e	chore: update version and add Next to changelog post-release (#5179) ... * chore: update version and add Next to changelog post-release	1 year ago
Cameron Perera	2e3fdb718c	bump ui commit to 0ff91f9 (#5172)	1 year ago
Lisbet Alvarez	987cdb3b64	bump ui commit to 857bf01 (#5170)	1 year ago
Johan Brandhorst-Satzkorn	ab922b5786	internal/clientcache/cmd/cache: restart cache on version mismatch (#5165) ... Restart the client cache if it is found to be a version below the version of the CLI during client cache hook execution.	1 year ago
Cameron Perera	f656509739	bump ui commit to 0d55812 (#5158)	1 year ago
Cameron Perera	a7105868d1	bump ui commit to 64e568bd (#5150)	1 year ago
Timothy Messier	21230e1c61	db: Re-order migrations for next release (#5146) ... Refs: 37bf59243a Refs: 3609658edd	1 year ago
Cameron Perera	d9055bd316	bump ui commit to d3c789c66c (#5147)	1 year ago
Michael Gaffney	9119effa9e	fix(session): Fix query for orphaned connections (#5138)	1 year ago
Timothy Messier	3609658edd	perf(session): Avoid select when canceling session (#5140) ... When a resource relating to a session is deleted, notably a target, host, or auth_token, the corresponding sessions are marked as canceled via a database trigger that calls the cancel_session function. This function was performing a select as part of the insert call to ensure that it would not attempt to insert a session_sate of canceled if it was an invalid state transition. However, since this function was first introduced, there has been additional constraints added that would prevent an invalid state transition. In addition, performing a select as part of an insert increases the chances for contention with other queries that interact with the session_state table, especially since a large number of sessions and session_state rows can be included in a single transaction when things like auth_tokens or targets get deleted. Now, the cancel_function will perform the insert, and catch any of the potential violations, allowing the query to continue without needing to perform the select, thus reduction the potential for contention.	1 year ago
Irena Rindos	10a33eb7a0	Bump boundary version & update changelog (#5143) ... * Bump version to 0.18	1 year ago
Hugo	ec2d5d4285	feat(host/plugin): Support empty host catalog secrets	1 year ago
Hugo	a5237cc171	feat(host/plugin): Use plugin client pkg func on job_set_sync	1 year ago
Hugo	5b5aa484da	feat(host/plugin): Extract plugin client functionality to pkg func	1 year ago
Hugo	ee9f9fb4a1	feat(worker): Support running AWS host plugin	1 year ago
Hugo	60dc0ec3ff	feat(host/plugin): Always send plugin information on DHC requests	1 year ago
Hugo	c3d03f69d8	feat(cmd): Support plugin host catalog worker filter	1 year ago
Hugo	b8046a3ef3	feat(handlers/host_catalogs): Support plugin host catalog worker filter	1 year ago
Hugo	7449fd6f2a	feat(host/plugin): Implement worker filter	1 year ago
Hugo	22b54c8b97	fix(event): Prevent eventer data races by removing shallow object copy (#5139) ... This issue was surfaced through other changes (PR 5137). Given that the eventer works in a singleton pattern (see sysEventer global variable), performing this shallow object copy can cause a data race because the copy operation reads the underlying eventer data structures. At the same time, given the global nature of this eventer, other goroutines can be writing to those same data structures. Since we're already using a singleton pattern in this package, and given that the current copy operation is shallow, copying the object doesn't seem necessary, so this commit changes `StandardWriter` to build the `logAdapter` with the `e` eventer directly instead of copying it.	1 year ago
Ryan Derr	de141e34a9	Worker Connection Failure Add Controller Target (#5109) ... * ICU-11692 Add Upstream Controller Addrs When Throwing Controller Connection Failure * Reference Consistant Load Variable For Client Connection	1 year ago
Timothy Messier	37bf59243a	refact(session): Replace state start_time/end_time with time range (#5134) ... This replaces the start_time, end_time, and previous_end_time columns on the session_state table with a time range column that records when the state was active. This allowed for dropping several constraints on the session_state table that were used to ensure that states for a session did not overlap. Instead this can be ensured with a single exclusion constraint. By reducing the number of constraints, it improves the performance for a number of queries involved with session state. Most notably when resolving delete cascades of resources related to sessions, such as targets, where as part of the delete transaction large numbers of sessions get canceled. Similarly, when sessions get deleted, and the corresponding session_state rows are deleted, these constraints all must be checked during the transaction. The domain layer continues to expose StartTime and EndTime fields on session.State. However it drops PreviousEndTime, which was never used by anything calling the domain layer. See: https://www.postgresql.org/docs/current/ddl-constraints.html#DDL-CONSTRAINTS-EXCLUSION https://www.postgresql.org/docs/current/rangetypes.html	1 year ago
Jeff Mitchell	dc74e725d4	Some clarity updates to comments in ACL code (#5136)	1 year ago
Timothy Messier	632d048f7b	lint: Fix linter errors ... (cherry picked from commit 51a2b2035f)	1 year ago
Timothy Messier	a92c88dfb5	refact(db): Replace get_deleted_tables function with a view ... This function was returning the set of deletion tables. A view seems better suited for this task since it would allow for applying additional filters of the result set. This was particularly necessary to easily make changes to some sqltests due to switching the delete trigger for the session table. (cherry picked from commit 32d4163989)	1 year ago
Timothy Messier	33037b4887	perf(db): Use statement trigger for marking deleted sessions ... When sessions are deleted, a trigger is used to insert records into the session_deleted table. This table is utilized by the sessions list endpoint when using a refresh token to inform a client of the sessions that have been deleted since the last request. We delete sessions in bulk via a controller job to delete sessions that have been terminated over an hour ago, which results in the trigger running a large number of separate insert statements while processing the delete statement. This changes the trigger to run once for the delete statement, instead of for each row, resulting in a single bulk insert statement to the session_deleted table. This new trigger function also avoids the use of `on conflict`. When testing this function, while the single statement was still faster than running multiple inserts, the `on conflict` still added significant overhead, even when there were no conflicts. It should be safe to perform the insert without the `on conflict`, since the same ID should never be deleted more than once if it is successfully deleted. (cherry picked from commit 3569b36c6e)	1 year ago
Timothy Messier	091ed6bacb	perf(db): Reorder index columns for primary keys ... When processing a controller API request, system uses a query to fetch the grants for the requesting user. This query requires pulling together information from several tables, and is currently performing many sequential scans to do so. For a number of the tables involved, there are indexes from multi-column primary keys, however, due to the order of the columns in the index, the postgres planner would need to do a full scan of the index, which can be less efficient than a sequential scan, so instead it will perform a sequential scan of the table. This recreates the primary keys while swapping the order of the columns in the primary key definition, and thus the order in the index. By doing so, the planner will not need to perform a full index scan, and will be more likely to use the index when executing the grants query. (cherry picked from commit 5577b3c358)	1 year ago
Timothy Messier	c9578b441a	perf(db): Add indexes on foreign key to improve delete performance ... This adds indexes for a few categories: 1. Foreign keys on the session table. These are set to `null` when the referenced row is deleted. These indexes will help to more efficiently set these `null` values in the case where a target is deleted, or and auth token is deleted. 2. Foreign keys from other tables to the session table. These are either set to `null` or cascade deleted when a session is deleted. These indexes help with these update/deletes when a session is deleted. 3. A multi-column index on session_state. This helps with the query that is used to delete all terminated sessions that have been terminated for over an hour. (cherry picked from commit 263a96e9dd)	1 year ago
Jeff Mitchell	62d4f8453b	Don't explode grants from the DB (#5104) ... This removes the Cartesian product in the DB for GrantsForUser in favor of returning the actual grant scope information and dealing with it in application code. (cherry picked from commit 913cd53682)	1 year ago
Irena Rindos	2945692bf4	Client cache API paging (#5107) ... * internal/clientcache: add -force-reset-schema flag * clientcache: stream list pages directly to DB --------- Co-authored-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com> (cherry picked from commit ccea8a4ce5)	1 year ago
Jeff Mitchell	3a4ee032d9	Paginate targets in cache (#5101) ... * Update Go API to support pagination * Implement pagination of targets in cache (cherry picked from commit cd25fad42f)	1 year ago
Jim	2a87dcad48	fix (cache/db): add indexes for desktop queries (#5091) ... (cherry picked from commit bfdbae1f04)	1 year ago
Jim	85afd56266	fix (cache): support for DB debug logging (#5087) ... (cherry picked from commit a7fe48b8d8)	1 year ago
Jim	863359aa26	test (cache/search): fix NewTestServer(...) storage (#5081) ... Fix ensures that the TestServer returned uses the correct directory for writing its sqlite files, which was created via t.TempDir() (cherry picked from commit 85f91043a1)	1 year ago
Jeff Mitchell	829ea02e6f	Ensure only one refresh is running for a user/resource at a time (#5054) ... This ensures we don't have competing update processes running. When doing the inline RefreshForSearch, a specific error will be returned so we can hint in the API that we're in the middle of a refresh. (cherry picked from commit 54776091e3)	1 year ago