606 Commits (efe9fe780d4094de4ac5041e7b15c24ae6104b9c)

Author	SHA1	Message	Date
Johan Brandhorst-Satzkorn	ec0a3f964b	Worker status refactor (#5417) ... * oss: feat(proto): statistics rpc added to server coordination service * oss: feat(session): add update statistics status service * oss: feat(handlers): add Statistics method to worker service server * oss: feat(worker): add statistics ticking * oss: internal/proto: define the new RoutingInfo RPC * oss: internal/server: repository changes for RoutingInfo RPC * oss: internal/daemon: add RoutingInfo handler to worker service The new RoutingInfo RPC is used to get information from the worker that is used to make routing decisions on the controller. It also handles the initial request from the worker. * oss: feat(db): server_worker_session_info table * oss: feat(proto): session info rpc for server coordination service * oss: feat(server): worker session info request data type * oss: feat(worker): session info ticker * oss: feat(handler): server coordination handler for session info * oss: feat(session): utilize session info request time for session cleanup * oss: proto: fix some gotags comments Some of these fields were missing comments, while others had incorrectly formatted comments. * oss: worker: send connection status to server This is required for the correct tracking of orphaned connections * oss: internal/daemon/worker: add function to wait for Statistics RPC * oss: internal/daemon/worker: add routing info RPC The new RoutingInfo RPC replaces part of the Status RPC, namely the part which is used by the controller and worker to perform decisions around the establishment of new sessions and new worker connections. * oss: internal/daemon/worker: remove unnecessary error check * oss: internal/daemon/worker: lock access to config and upstreams The new confLock protects access to the conf value and to the addressReceivers slice. * oss: worker: replace Status RPC with RoutingInfo * oss: worker: deprecate the Status RPC and related types The Status RPC and related messages have been replaced by the RoutingInfo/SessionInfo/Statistics RPCs. The controller will continue to support the Status RPC until two new minor versions have been released. * oss: internal/event: add default deny for new worker RPCs * oss: internal/daemon/worker: allow RPC interval to be set The new SessionInfo/RoutingInfo/Statistics RPCs run at long intervals, which make some of our cluster tests take a long time to complete. This allows their interval to be configured in tests. * oss: all: use TestWorkerRPCInterval in slow tests This should speed up these tests significantly. * oss: internal/tests/cluster: reset sys eventer in test --------- Co-authored-by: Damian Debkowski <damian.debkowski@hashicorp.com>	1 year ago
Timothy Messier	71af25f584	fix(db): Remove begin when assembling migrations (#5148) ... When building the set of migrations to run as part of executing the `database migrate` or `database init`, the migrations are collected and combined to run within a database transaction. As such the `begin` and `commit` statements in the migration files should get removed from the final set of statements that get executed. However, when the migrations files were updated to include copyright and license headers, this broke the logic that would strip out the `begin` statement. While this does not cause any issue with executing the statements in a single transaction, it does result in noise in the database server's logs, with log messages like: BEGIN WARNING: 25001: there is already a transaction in progress BEGIN LOCATION: BeginTransactionBlock, xact.c:3778 This commit fixes the logic to correctly strip the `begin` for files that have the header lines.	1 year ago
Hugo	846c849589	Worker Domain Refactor (#5338) ... * feat(server): Refactor LookupWorkerByName to be a test method * feat(server): Split worker tags table * refact(worker): Export api and config tags * refact(server): Return worker id from UpsertWorkerStatus * refact(server): Create ListWorkerStorageBucketUpdates domain function * refact(server): Use dedicated HCPb-worker list func * refact(server): Create ConnectionRoute domain function * refact(handlers): Move worker selection logic to internal/server package This commit moves all the logic to select a worker to handle a session to the internal/server package. Most functions were moved to this package with no changes, however, there are a couple of notable exceptions: - Given that we don't use GRPC errors in repos, all errors were adjusted to use the internal/errors package instead. - StorageBucketFilterCredIdFn is now used to prevent the internal/server package from importing internal/storage/plugin, which would cause a dep cycle. Callers of SelectSessionWorkers must implement this func which is responsible for returning a storage bucket's worker filter as well as its associated credential id. - authorizeSessionWithWorkerFilter now receives an internal/server.(*Repository) object when called. This is transparent to SelectSessionWorkers callers as it is handled internally. * feat(server): Add FilteredWithFeatures fn and new server options * refact(server): create new list and lookup worker queries * test(server): Add WorkerList unit tests * feat(server): Implement WorkerList randomness using crypto/rand pkg * chore(db): add index to server_worker table * refact(server): prefix test method with Test * refact(server): rename Downstreamers interface to Graph * chore(server): unexport and rename method * chore: add doc string to WorkerAddress (#5420) --------- Co-authored-by: Irena Rindos <irenarindos@users.noreply.github.com> Co-authored-by: irenarindos <irena.rindos@gmail.com>	1 year ago
Hugo	f227020b67	feat(scheduler): Use job table to select jobs to run ... This commit changes the logic around selecting jobs to run. Before this, Boundary used a view "job_jobs_to_run" which was a collation of all jobs that were not running and scheduled to run. When jobs were requested to run, Boundary would look at this view, grab a certain amount of jobs and insert them into the "job_run" table. Importantly, while the amount of jobs to be scheduled at once defaulted to 1 in the scheduler package, this was entirely overriden in the controller package to -1 (no limit, run all jobs that the query would return). With this commit, Boundary now looks at the "job" table directly for jobs to run, regardless of job run status. This was done to improve DB performance as the view (which queried all "job_run" rows) was causing significant DB contention. Additionally, the option to limit job runs is now removed since it was already hardcoded to -1 and not user-configurable anyways. Should the need arise to implement this limit again, this will have to be done in a different way since the new query does not account for running jobs and limiting the rows returned after jobs are already running would yield a no-op (the new query inserts nothing into "job_run" if any given job is already running).	1 year ago
Hugo	27c432a748	feat(scheduler): Drop completed job runs as they complete ... Before this change, upon the completion of a job run, the scheduler would update a job_run entry with the completed status as well as the number of objects affected by the run. Additionally, we then had a cleaner job whose scope was deleting these completed entries from job_run. This commit changes this design to drop the cleaner job the and rolls its functionality directly into the scheduler. Now, when any given job run is complete, the scheduler will automatically delete it from the job_run table itself. In addition, the completed status has been removed as a valid status as it is not set anymore. This change follows a decision that given the data in these job_run entries is already being deleted by the cleaner job, updating its state when a job completes just to have it deleted is not efficient. Additionally, no use-case for this data was surfaced.	1 year ago
Michael Gaffney	9529491fa6	feat(session): Delete terminated sessions in batches ... This changes the periodic job that deletes terminated sessions to delete the those sessions in batches. Prior to this, a single SQL DELETE command was used to delete all qualifying terminated sessions. In systems under heavy load, the statement could timeout leading to a situation where no qualifying terminated sessions were ever being deleted.	1 year ago
Michael Gaffney	8c291a2934	feat(scheduler): Add retries to job status ... This adds retries as an attribute to JobStatus allowing a job to indicate liveness to the scheduler when work has not progressed. A job could use this and the statusThreshold passed into the Run method to break work up into batches that would otherwise timeout if all the work was attempted at once, for example a long running SQL delete command.	1 year ago
Jim	ffd0d2a7d3	feat (cache): impl. soft-delete of users in cache (#5173)	1 year ago
Timothy Messier	21230e1c61	db: Re-order migrations for next release (#5146) ... Refs: 37bf59243a Refs: 3609658edd	1 year ago
Timothy Messier	3609658edd	perf(session): Avoid select when canceling session (#5140) ... When a resource relating to a session is deleted, notably a target, host, or auth_token, the corresponding sessions are marked as canceled via a database trigger that calls the cancel_session function. This function was performing a select as part of the insert call to ensure that it would not attempt to insert a session_sate of canceled if it was an invalid state transition. However, since this function was first introduced, there has been additional constraints added that would prevent an invalid state transition. In addition, performing a select as part of an insert increases the chances for contention with other queries that interact with the session_state table, especially since a large number of sessions and session_state rows can be included in a single transaction when things like auth_tokens or targets get deleted. Now, the cancel_function will perform the insert, and catch any of the potential violations, allowing the query to continue without needing to perform the select, thus reduction the potential for contention.	1 year ago
Hugo	60dc0ec3ff	feat(host/plugin): Always send plugin information on DHC requests	1 year ago
Hugo	7449fd6f2a	feat(host/plugin): Implement worker filter	1 year ago
Timothy Messier	37bf59243a	refact(session): Replace state start_time/end_time with time range (#5134) ... This replaces the start_time, end_time, and previous_end_time columns on the session_state table with a time range column that records when the state was active. This allowed for dropping several constraints on the session_state table that were used to ensure that states for a session did not overlap. Instead this can be ensured with a single exclusion constraint. By reducing the number of constraints, it improves the performance for a number of queries involved with session state. Most notably when resolving delete cascades of resources related to sessions, such as targets, where as part of the delete transaction large numbers of sessions get canceled. Similarly, when sessions get deleted, and the corresponding session_state rows are deleted, these constraints all must be checked during the transaction. The domain layer continues to expose StartTime and EndTime fields on session.State. However it drops PreviousEndTime, which was never used by anything calling the domain layer. See: https://www.postgresql.org/docs/current/ddl-constraints.html#DDL-CONSTRAINTS-EXCLUSION https://www.postgresql.org/docs/current/rangetypes.html	1 year ago
Timothy Messier	a92c88dfb5	refact(db): Replace get_deleted_tables function with a view ... This function was returning the set of deletion tables. A view seems better suited for this task since it would allow for applying additional filters of the result set. This was particularly necessary to easily make changes to some sqltests due to switching the delete trigger for the session table. (cherry picked from commit 32d4163989)	1 year ago
Timothy Messier	33037b4887	perf(db): Use statement trigger for marking deleted sessions ... When sessions are deleted, a trigger is used to insert records into the session_deleted table. This table is utilized by the sessions list endpoint when using a refresh token to inform a client of the sessions that have been deleted since the last request. We delete sessions in bulk via a controller job to delete sessions that have been terminated over an hour ago, which results in the trigger running a large number of separate insert statements while processing the delete statement. This changes the trigger to run once for the delete statement, instead of for each row, resulting in a single bulk insert statement to the session_deleted table. This new trigger function also avoids the use of `on conflict`. When testing this function, while the single statement was still faster than running multiple inserts, the `on conflict` still added significant overhead, even when there were no conflicts. It should be safe to perform the insert without the `on conflict`, since the same ID should never be deleted more than once if it is successfully deleted. (cherry picked from commit 3569b36c6e)	1 year ago
Timothy Messier	091ed6bacb	perf(db): Reorder index columns for primary keys ... When processing a controller API request, system uses a query to fetch the grants for the requesting user. This query requires pulling together information from several tables, and is currently performing many sequential scans to do so. For a number of the tables involved, there are indexes from multi-column primary keys, however, due to the order of the columns in the index, the postgres planner would need to do a full scan of the index, which can be less efficient than a sequential scan, so instead it will perform a sequential scan of the table. This recreates the primary keys while swapping the order of the columns in the primary key definition, and thus the order in the index. By doing so, the planner will not need to perform a full index scan, and will be more likely to use the index when executing the grants query. (cherry picked from commit 5577b3c358)	1 year ago
Timothy Messier	c9578b441a	perf(db): Add indexes on foreign key to improve delete performance ... This adds indexes for a few categories: 1. Foreign keys on the session table. These are set to `null` when the referenced row is deleted. These indexes will help to more efficiently set these `null` values in the case where a target is deleted, or and auth token is deleted. 2. Foreign keys from other tables to the session table. These are either set to `null` or cascade deleted when a session is deleted. These indexes help with these update/deletes when a session is deleted. 3. A multi-column index on session_state. This helps with the query that is used to delete all terminated sessions that have been terminated for over an hour. (cherry picked from commit 263a96e9dd)	1 year ago
dani	557591967a	add tests for sql ip address columns to ensure ipv6 capabiltiy (#5062)	1 year ago
Louis Ruch	a3788ab978	fix(policy): Cascade delete policy when scope is deleted (#5014)	2 years ago
Damian Debkowski	5707a64b20	fix(db): remove storage_plugin_storage_bucket_secret-rewrapping-job table (#4993)	2 years ago
Jim	28b465e4b8	refact: to use go-dbw CreateItems, DeleteItems (#4974) ... * refact: to use go-dbw CreateItems, DeleteItems Latest version of go-dbw updated the signature of CreateItems(...) and DeleteItems(...) on a happy note this latest version of go-dbw actually does batch inserts and updates for this functions. * all: regenerate protos with new protoc-gen-go version	2 years ago
Irena Rindos	2191aa1d03	refact(session connection): remove session connection state table (#4617) ... * refact(session connection): remove session connection state table	2 years ago
Damian Debkowski	1be0e5d413	fix: pass along storage bucket credential id for sb update job (#4947)	2 years ago
Damian Debkowski	dc21e714a5	feat(db): tracking storage bucket by worker	2 years ago
Irena Rindos	08f391c5d6	workerauth: satisfy NodeIdLoader interface (#4870) ... * workerauth: satisfy NodeIdLoader interface	2 years ago
Michael Li	2467f9107a	test(db): Fix skip command (#4860)	2 years ago
Michael Li	e18e3ab783	test: Fix tests due to time (#4855) ... * test(billing): Calculate time deltas from a fixed time Some time calculations are impacted when using the current day vs. the start of the month. * test(db): Conditionally skip tests at end of month Certain tests fail when run on the last day of the month because of edge cases in the setup of test data (for example, if today is May 31 and NZ time is June 1). This adds conditional logic to skip those tests when run on the last day of a month. * CR: typo	2 years ago
Louis Ruch	0cfca44cf5	fix(vault): Fixes credential_vault_credential_private view ... The worker_filter column was dropped from the credential_vault_credential_private view in the 56/02_add_data_key_foreign_key_references.up.sql migration. Any credential renewal/revocations that relied on the worker filter would not successfully renew.	2 years ago
Louis Ruch	c376d06355	feat: Add support for correlationId	2 years ago
Elim Tsiagbey	79917f74b8	feat(worker): Worker Local Storage State DB Column ... - Add new `local_storage_state` column to `server_worker` table which keeps track of the state of the local disk space. - The valid values for `local_storage_state` are: -- `available`: The worker local storage state is at an acceptable state -- `lowStorage`: The worker is below the minimum threshold for local storage -- `criticallyLowStorage`: The worker local storage state is below the critical minimum threshold for local storage -- `outOfStorage`: The worker is out of local disk space -- `notConfigured`: The worker does not have a local storage path configured -- `unknown`: The default local storage state of a worker. Used when the local storage state of a worker is not yet known	2 years ago
Michael Li	9eb608feeb	chore: Update protobuf to 1.33 (#4485) ... Release: https://github.com/protocolbuffers/protobuf-go/releases/tag/v1.33.0 CVE: https://nvd.nist.gov/vuln/detail/CVE-2024-24786	2 years ago
Todd	5a17c9cd39	Resolve aliases prior to subtype attribute transforming interceptor (#4379) ... * Resolve aliases prior to subtype interceptor	2 years ago
Todd	e848e76b48	Create Alias Domain Object and Repo (#4275) ... * Create alias domain object resource * Add paginated listing * fixing ratelimit related panics * Address reviewer comments * remove NewAlias value checks	2 years ago
Todd	7e2038bfcc	Add alias schema changes (#4230) ... * Adding alias resource * Update website/content/docs/install-boundary/system-requirements.mdx --------- Co-authored-by: Irena Rindos <irenarindos@users.noreply.github.com>	2 years ago
Timothy Messier	111dc87ead	feat(billing): Fix end time comparision for all view	2 years ago
Timothy Messier	e6c3795f91	refact(db): Renumber migrations for billing	2 years ago
Timothy Messier	0f765e7149	feat(timestamp): Support -/+infinity for Valuer ... When converting a timestamp.Timestamp for use in SQL, this now correctly converts the NegativeInfinity and PositiveInfinity constants into the corresponding `-infinity` and `infinity` SQL constants.	2 years ago
Timothy Messier	5405a275ab	feat(census): Update census tables to support additional metrics	2 years ago
Timothy Messier	d84eb01516	feat(billing): Add sql functions for monthly active user counts ... This adds two sql functions that can be used to access the views to retrieve monthly active user counts. The functions ensure that any time calculates are performed in UTC, so that the month bounds align with the start of the month in UTC. 1. hcp_billing_monthly_active_users_last_2_months() will return monthly active users for the current and previous month. 2. hcp_billing_monthly_active_users_all() will return monthly active users for all months. This function can also be provided a start time and or end time to filter the results to a specific range.	2 years ago
Timothy Messier	324c474f5d	feat(billing): Add views for monthly active user counts	2 years ago
Timothy Messier	bbb43e20a0	test(sqltest): Remove impossible assertion ... Now that there is an auth token fact table in the data warehouse, it is not possible to insert an auth_token with an old public_id even after that auth_token was deleted, since it now causes a constraint violation on the data warehouse table.	2 years ago
Timothy Messier	9514a76289	test(sqltest): Fix tests due to new auth_token insert trigger ... With the addition of an after insert trigger to the auth_token table, entries are added to the wh_user_dimension table when auth token's are inserted with the 'token issued' status. This caused a number of tests to fail that assumed the user dimension table would be empty prior to inserting sessions. This introduces a number of changes to the tests: - In the test setup, auth tokens are not created with an explicit expiration time and status. The expiration time ensures that the tokens are 'valid' for the test. The status allows test to make assertions on tokens in different statuses, and allow the status to change during the test. - Tests make the correct assertions on the number of entries in the user dimension table after test setup. - User dimension tests modify a user after the standard setup to ensure that a new user dimension is added when a session is inserted.	2 years ago
Timothy Messier	9e0d96b887	feat(dw): Update auth token fact when auth token is deleted ... This adds a function and after delete trigger to the auth_token table to ensure that a corresponding auth token fact is updated with the deleted time and sets the auth token valid time range. The data warehouse will only be updated if the token that was deleted is in the 'token issued' status.	2 years ago
Timothy Messier	b938dd6706	feat(dw): Insert/update auth token fact when auth token updated ... This adds a function and update trigger to auth_token to ensure that: 1. If the auth token is updated to the issued state, a auth token fact is added to the warehouse. 2. If the auth token's approximate last access time is updated, the corresponding auth token fact is updated with the new time.	2 years ago
Timothy Messier	585b344da6	feat(auth_token): Set approximate_last_access_time when token is issued	2 years ago
Timothy Messier	07b3a7a19e	feat(dw): Add auth token fact after inserting issued auth token ... This adds a trigger and function to ensure that an auth token fact is created when a new auth token is created with the 'token issued' status.	2 years ago
Timothy Messier	eb626bb54c	refact(wh): Use auth token id for user dimension ... Previously the function used to upsert into the user dimension required both an auth token id, and a user id. However, the user id can be determined from the auth token, in fact it already was via the whx_user_dimension_source view. This refactor removes the user id as an argument, since it is not needed and will not be directly available when this function is used when populating the new auth token accumulating fact.	2 years ago
Timothy Messier	03d1cc4572	feat(dw): Add auth token accumulating fact table	2 years ago
Timothy Messier	005f3d0843	chore(sqltest): Make it possible to enable/disable db container logs ... When running the sqltests, sometimes it is helpful to see the full database container logs, especially when there is an error in a migration or database setup. However, it also makes it difficult to see the test results. This adds a new variable to the makefile to easily toggle the output of these logs on or off. To run the tests without printing the dtabase logs use: DB_LOGS=0 make test To enable the logs use: DB_LOGS=1 make test By default the logs are not printed.	2 years ago
Timothy Messier	e51d0e0b4f	feat(vault): Remove database notice when setting credential type ... This notice just generated noise, particuarlly in some of the sqltests.	2 years ago