boundary

Commit Graph

Author	SHA1	Message	Date
Michael Gaffney	e086fd3007	test(batch): Fix data races in tests Fixes: `9b984b0ab6`	1 year ago
Hugo	f227020b67	feat(scheduler): Use job table to select jobs to run This commit changes the logic around selecting jobs to run. Before this, Boundary used a view "job_jobs_to_run" which was a collation of all jobs that were not running and scheduled to run. When jobs were requested to run, Boundary would look at this view, grab a certain amount of jobs and insert them into the "job_run" table. Importantly, while the amount of jobs to be scheduled at once defaulted to 1 in the scheduler package, this was entirely overriden in the controller package to -1 (no limit, run all jobs that the query would return). With this commit, Boundary now looks at the "job" table directly for jobs to run, regardless of job run status. This was done to improve DB performance as the view (which queried all "job_run" rows) was causing significant DB contention. Additionally, the option to limit job runs is now removed since it was already hardcoded to -1 and not user-configurable anyways. Should the need arise to implement this limit again, this will have to be done in a different way since the new query does not account for running jobs and limiting the rows returned after jobs are already running would yield a no-op (the new query inserts nothing into "job_run" if any given job is already running).	1 year ago
Hugo	27c432a748	feat(scheduler): Drop completed job runs as they complete Before this change, upon the completion of a job run, the scheduler would update a job_run entry with the completed status as well as the number of objects affected by the run. Additionally, we then had a cleaner job whose scope was deleting these completed entries from job_run. This commit changes this design to drop the cleaner job the and rolls its functionality directly into the scheduler. Now, when any given job run is complete, the scheduler will automatically delete it from the job_run table itself. In addition, the completed status has been removed as a valid status as it is not set anymore. This change follows a decision that given the data in these job_run entries is already being deleted by the cleaner job, updating its state when a job completes just to have it deleted is not efficient. Additionally, no use-case for this data was surfaced.	1 year ago
Michael Gaffney	9b984b0ab6	feat(batch): Add batching for SQL jobs This adds a new `batch` package under the `scheduler` package.	2 years ago
Michael Gaffney	8c291a2934	feat(scheduler): Add retries to job status This adds retries as an attribute to JobStatus allowing a job to indicate liveness to the scheduler when work has not progressed. A job could use this and the statusThreshold passed into the Run method to break work up into batches that would otherwise timeout if all the work was attempted at once, for example a long running SQL delete command.	2 years ago
Michael Gaffney	9354a4ebcf	refactor(scheduler): Add statusThreshold parameter to job.Run The scheduler monitors running jobs to make sure they are not hung. If the status of a running job does not change after a duration (the interruptThreshold in the scheduler), the scheduler kills the job. This change passes the scheduler's interruptThreshold to the job when it calls Run() on the job.	2 years ago
Michael Gaffney	50ab600519	docs(scheduler): Clarify Job and JobStatus This adds more details to the Job interface and the JobStatus struct to clarify how the scheduler uses them.	2 years ago
Jim	28b465e4b8	refact: to use go-dbw CreateItems, DeleteItems (#4974 ) * refact: to use go-dbw CreateItems, DeleteItems Latest version of go-dbw updated the signature of CreateItems(...) and DeleteItems(...) on a happy note this latest version of go-dbw actually does batch inserts and updates for this functions. * all: regenerate protos with new protoc-gen-go version	2 years ago
Michael Li	9eb608feeb	chore: Update protobuf to 1.33 (#4485 ) Release: https://github.com/protocolbuffers/protobuf-go/releases/tag/v1.33.0 CVE: https://nvd.nist.gov/vuln/detail/CVE-2024-24786	2 years ago
Jeff Mitchell	25bce9e8ff	Fix linter warnings	2 years ago
Johan Brandhorst-Satzkorn	d9c6de4e8b	deps: update runc dependency (#4349 ) Fixes GHSA-xr7r-f8xq-vfvv	2 years ago
Jim	6a37c93d77	fix: properly handle rows.Err() (#4261 ) * fix (auth): properly handle rows.Err() * fix (auth/ldap): properly handle rows.Err() * fix (auth/oidc): properly handle rows.Err() * fix (auth/password): properly handle rows.Err() * fix (authtoken): properly handle rows.Err() * fix (credential): properly handle rows.Err() * fix (credential/static): properly handle rows.Err() * fix (credential/vault): properly handle rows.Err() * fix (db): properly handle rows.Err() * fix (db/schema): properly handle rows.Err() * fix (db/migrations): properly handle rows.Err() * fix (host): properly handle rows.Err() * fix (host/plugin): properly handle rows.Err() * fix (host/static): properly handle rows.Err() * fix (iam): properly handle rows.Err() * fix (kms): properly handle rows.Err() * fix (oplog): properly handle rows.Err() * fix (pagination/purge): properly handle rows.Err() * fix (plugin): properly handle rows.Err() * fix (job/scheduler): properly handle rows.Err() * fix (server): properly handle rows.Err() * fix (session): properly handle rows.Err() * fix (target): properly handle rows.Err()	2 years ago
Irena Rindos	9733a250fa	refact(event): move observability/event to event (#3704 )	3 years ago
Jim	d5411a26f6	chore: update protobuf dep (#3669 )	3 years ago
hashicorp-copywrite[bot]	29da0bcb92	[COMPLIANCE] License changes (#3567 ) * Adding explicit MPL license for sub-package. This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository. * Adding explicit MPL license for sub-package This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository. * Updating the license from MPL to Business Source License Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at www.hashicorp.com/licensing-faq, and details of the license at www.hashicorp.com/bsl. * Update copyright file headers to BUS-1.1 * Rerun make gen This will pick up the last of the license changes * Revert "Rerun make gen" This reverts commit `dbe0968bff`. * Remove copywrite from generation script The current version of copywrite is not sophisticated enough to only apply the copywrite changes to certain subdirectories, so to avoid accidentally relicensing our api and sdk modules, we will stop running the copywrite script until such time that it can do the right thing. * License protobuf API as MPL The protobuf API files are used to generate the SDK, since we want the SDK to be MPL, the protobuf files must also be MPL. This extends to all protobuf files whose sources end up in the SDK directory, including: - internal/proto/controller/api - internal/proto/controller/custom_options - internal/proto/plugin It also includes the generated service directory in internal/gen/controller/api/services, since they are generated from the protobuf sources used for the SDK. * Change website license file name This makes the naming scheme consistent with the rest of the repository. * Add note on licensing to README * Set copywrite license to BUSL * Revert "Add note on licensing to README" This reverts commit `4d865a91a7`. --------- Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com> Co-authored-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>	3 years ago
Jeff Mitchell	8322916fb4	Remove all deprecated Err/Wrap/E calls (#3286 )	3 years ago
Jim	63a2b62b85	chore (deps): update GKW and protobuf; and run "make gen" (#3111 )	3 years ago
Johan Brandhorst-Satzkorn	3c29308673	chore: Add license headers to all files	3 years ago
Johan Brandhorst-Satzkorn	55259029f1	Add job run cleaner (#2866 ) * chore: remove duplicate import * feat: Add job run cleaner The new job cleans job runs older than a configured interval. Configure the job to clean jobs older than 1 minute.	3 years ago
Johan Brandhorst-Satzkorn	f57454b6b9	Rewrite interface{} to any (#2535 ) * chore: Rewrite interface{} to any * Add ignore revs	4 years ago
Louis Ruch	c73a13aa3e	scheduler: log details when starting scheduler (#2459 )	4 years ago
Damian Debkowski	2db6bee3e2	fix: return an error for vault tokens that will expire before the scheduler can run (#2444 )	4 years ago
Louis Ruch	5812a42ba3	feat(scheduler): set intervals from config (#2443 )	4 years ago
Johan Brandhorst-Satzkorn	aef9073fa6	Upgrade to Go 1.19 (#2347 )	4 years ago
Johan Brandhorst-Satzkorn	222a935223	Update grpc-gateway dependency (#2311 ) * chore(deps): Update grpc-gateway dependency This fixes an issue which would allow an attacker to trigger a (recovered) panic in the gateway handler. * fix(server): Use cmp.Diff instead of assert.Equal Testify is not great at handling protobuf types, and was for some reason returning a diff here when in reality there is none. Switch to go-cmp for the comparison to remove the error.	4 years ago
Louis Ruch	4cacd0a605	bug(scheduler): Fix controller id length check (#2226 ) * bug(scheduler): Fix controller id length check The job_run table has a foreign key to the server_controller private_id. While this column is called a prviate_id it does not implement the wt_private_id postgres domain type, and therefore does not have a minimum length of 10 characters.	4 years ago
Todd	19b549c44a	Rename package servers to server (#2222 )	4 years ago
Jeff Mitchell	704d68848c	Merge remote-tracking branch 'origin/main' into llb-byow	4 years ago
Louis Ruch	9663404f1a	feat(scheduler): Support unlimited job runs (#2173 )	4 years ago
Todd	2ecd550e27	Add fields to server_worker to be a boundary resource (#2083 ) * Remove the server_id from session table, change it to worker_id on session_connection This is step 1 in updating the worker table. The worker is becoming a public resource so it must have a public id. We drop the server_id column on the session_connection table and add the worker_id column so we can use the wt_public_key type. Since db migrations are applied while the controller is offline, it should be safe to assume all session connections are not in a running state, thus allowing us to null out all the worker ids in the session_connection table. For now we don't have very good enforcement of the wt_public_id contraints, since the id is currently defined in the worker config, but this will soon change and the worker's id will be generated by the controller. * Updating server_id references to controller_id. * Adding worker tag db writing test. * Move the server table back to the top. * Update comment for the worker object. * Update comment for the worker table. * Update the newWorkerId call to take a context and fix its error. * Correct comment for the wt_network_address domain.	4 years ago
irenarindos	8579a248d4	refactor(servers): minor sql adjustments to worker and controller tables	4 years ago
irenarindos	9e27605213	refact(servers): Split server table into worker and controller	4 years ago
Johan Brandhorst-Satzkorn	7f9b294a7e	refact(all): Use testing.TB for test helpers Our test helper packages were not possible to use with benchmarks. This refactor switches to testing.TB to allow both tests and benchmarks to use the helpers.	4 years ago
Timothy Messier	4599e64cab	feat(event): Remove BOUNDARY_DEVELOPER_ENABLE_EVENTS env var	4 years ago
Johan Brandhorst-Satzkorn	b1d6a1da9a	Switch to buf for protobuf generation (#1944 ) * Install buf with go install This reduces the impact of using buf on our dependency closure, which was significant. It also avoids potential issues with building buf given our own internal dependency constraints. This has caused issues (including crashes!) in the past. See https://github.com/cockroachdb/cockroach/issues/67216#issuecomment-874176186 for more discussion on using tool dependencies generally, and https://github.com/bufbuild/buf/issues/52 for an example of where this sort of thing caused an issue for a user. * Upgrade buf version We should migrate to v1 sooner rather than later, as it has some breaking changes from v0. * Change buf lint/breaking commands These commands were moved up one level for v1 * Migrate to buf.yaml to v1 * Remove unused imports These were uncovered by using "buf lint". * Use buf v1.3.1 * Depend on googleapis and grpc-gateway via BSR * Switch to buf for proto generation * Remove third party dependency * Remove unused tools and scripts * Move local protos up one level * Temporarily don't fail on breaking changes * Generate CI config * Fix generation of controller.json * Replace use of deprecated flag * Remove unused import, regenerate files * Remove temporary skipping over breaking check	4 years ago
Jeff Mitchell	7eb29261b2	Update to go-kms-wrapping version 2, and plugin-based KMS (#1901 )	4 years ago
Louis Ruch	f8b27645d9	fix(scheduler): Update job run query to not error on running conflict (#1914 )	4 years ago
Louis Ruch	8cab9a4315	feat(scheduler): Update job description on conflict (#1852 )	4 years ago
Louis Ruch	c4506ec743	feat(scheduler): Add support for run now option	4 years ago
Jim	591ec476e7	refactor: Update internal/db and oplog to use go-dbw package for database operations. (#1785 ) * chore: Add new dbw dependency * refactor: Convert to use dbw.DB * refactor (db): Change GetTicket(...) signature * feature (db tests): Added TestDeleteWhere(...) Adding this allows us to eliminate a leak of the gorm library * refactor: Add ctx to ScanRows(...) signature This is required to refactor all the deprecated errors pkg calls. This refactor required adding a context to quite a few call chains and in particular the job scheduler bits * feature (db/tests): Add support for test opt: WithTestLogLevel(...) This test only option allows callers to specify a log level for the underlying db package (where supported/applicable) when calling TestSetup(...) * refactor (db): Simplify Db type Only store the underlying DB and create the RW on demand when needed. * refactor (oplog): Port to using dbw pkg for db operations * refactor (db): Remove gormDB(...) func * tests (db): Add unit tests for TestDeleteWhere(...) * chore: make fmt changes to formatting	4 years ago
Jeff Mitchell	158ab09952	Merge branch 'main' into plugin-hostcatalogs	5 years ago
Jim	8d6dee09a9	refact: Add db.DB wrapper and refact all test fixtures to use it. (#1535 ) * refact: Changes required for gorm v2 * refact: Add db.DB wrapper and refact all test fixtures to use it Stop leaking the gorm abstraction within our unit tests. * refact: Migrate direct calls to pq to pgx driver The refactor will remove the remaining direct dependencies on the pq driver which has been deprecated in favor of the pgx driver. It includes the addition of common.SqlOpen which is a wrapper around sql.Open, which is needed to deal with the driver name changing from "postgres" to "pgx" but in our code base the dialect and driver are synonyms.	5 years ago
Jim	136ac00b49	refact: Changes required for gorm v2 (#1528 )	5 years ago
Todd Knight	843d126428	Merge branch 'main' into plugin-hostcatalogs	5 years ago
Louis Ruch	1c40100edd	tests(scheduler): fix unit test deadlock (#1550 ) If a new job is started as the test successfully compeltes and begins the scheduler shutdown process. The new job will deadlock waiting on channel reads.	5 years ago
Jeff Mitchell	f9c620a5d5	Merge branch 'main' into plugin-hostcatalogs	5 years ago
Jeff Mitchell	7fc712de44	Bump protoc	5 years ago
Todd Knight	db9e1106a9	feat(host): Add host plugin related schemas and tests (#1475 ) * feat(host): Add host plugin related schemas and tests This adds the schema files for host plugins, plugin based catalogs and plugin based sets. This also adds the protos and tests which utilize gorm to ensure the schemas enforce the constraints we expect. Along with this change was a migration which moved catalog names to the base type table so we can ensure all catalogs in a single scope, regardless of subtype, have a unique name. * Updated plugin schemas to do the following: - Add plugin_executable table - plugin table now also has scope_id and name w/ uniqueness constraints - host_plugin no longer has id_prefix column - host_plugin separates name and plugin_name - plugin_version table renamed version to symantic_version column. * Fix typo in delete_plugin_subtype trigger. Add update test for host plugins. * Update id prefixes. Fix public_id typos in protos and plugin_executable name in sql diagrams. * Created internal/plugin and moved base plugin structs to it and moved host plugin subdirectory to it. * Created internal/plugin and moved base plugin structs to it and moved host plugin subdirectory to it. * Add first pass at enums for OS and Arch supported by plugin executables. * Capitalizing the const enums for OS and Architectures. * Add comments to sql files. * Reviewer comments. Remove unsupported OSes and Archs from SQL and go constants. Add missing tests. * Be more permissive in the values the db accepts. The go logic can restrict which values it will allow getting populated into the db. * Removing constraints on which values can be placed inside operating_system and architecture columns. * Updating test helper functions to only create a single catalog or set. * Move files to new migration version. * Fixing job/plugin test. * Update table names, named a number of constraints, remove plugin version and executable tables. * Add constraints for plugin name and id prefix. * Rename constraint, remove unhelpful comment block. * Adding semantic version to plugin_host and set attributes to not null. * Remove semantic versioning from all locations. * Remove semantic_version from plugin_host table comment. * Fix op copy/paste error. Don't use deprecated errors methods. Update constraint names. Add docs. * Apply suggestions from code review Fixing constraint names. Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com>	5 years ago
Louis Ruch	61722659d1	test(scheduler) Remove un-needed test causing intermittent failures (#1501 )	5 years ago
Jim	477b541bf0	Events: ongoing clean up of eventing (#1448 )	5 years ago

1 2

63 Commits (efe9fe780d4094de4ac5041e7b15c24ae6104b9c)