Commit Graph

119 Commits

Author SHA1 Message Date
1c4daa99a9 Add e2e tests for exec env createOrUpdate
This also adds a Nomad client to the e2e_tests that can be used to
query Nomad and validate that certain actions happened in Nomad
correctly.
2021-06-03 13:21:49 +00:00
1be744f2d4 Explicitly set task groups network when networkAccess is false
Previously, updating an environment from with to without network
access would leave the network resource in the task group as they
were before.
2021-06-03 13:21:49 +00:00
b990df7b9d Add route to create or update execution environments 2021-06-03 13:21:49 +00:00
3d395f0a38 Set network_mode to bridge to overwrite old setting
Previously, the network_mode was only set when creating a job with
network_access = false. This results in Nomad leaving this setting
as is when updating the job to use network. Thus a job would have
had the mapped ports in the Nomad UI, but the Docker network_mode
would still be 'none'.
2021-06-03 13:21:49 +00:00
5f35ba30a2 Remove data race in the runner length function 2021-06-03 11:38:40 +02:00
02b3f52a11 Add ability to copy files to and delete files from runner 2021-06-02 14:54:54 +02:00
242d0175a2 Add tests and end-to-end tests for websocket execution
For unit tests, this mocks the runners Execute method with a
customizable function that operates on the request, streams and exit
channel to simulate a real execution.

End-to-end tests are moved to the tests/e2e_tests folder. The tests
folder allows us to have shared helper functions for all tests in a
separate package (tests) that is not included in the non-test build.

This also adds one second of delay before each end-to-end test case by
using the TestSetup method of suite. By slowing down test execution,
this gives Nomad time to create new allocations when a test requested a
runner. Another solution could be to increase the scale of the job to
have enough allocations for all end-to-end tests.

Co-authored-by: Maximilian Paß <maximilian.pass@student.hpi.uni-potsdam.de>
2021-05-31 12:32:51 +02:00
3afcdeaba8 Execute commands in runner via WebSocket
This enables executing commands in runners and forwarding input and
output between the runner and the websocket to the client.

Co-authored-by: Maximilian Paß <maximilian.pass@student.hpi.uni-potsdam.de>
2021-05-31 12:32:51 +02:00
892f902377 Add deps dependency to e2e-test make target
Without the deps dependency the e2e-test target would sometimes
fail in the CI due to missing entries in the go.sum file.
2021-05-28 09:13:59 +02:00
6084b00e23 Exit MonitorEvaluation once stream is closed
As we pass the context to the Nomad API event stream, they close
the event stream once the passed context is cancelled. We use this
to exit our receive loop on the event stream once the stream is closed,
instead of having to check the context manually.
2021-05-28 06:52:38 +00:00
a891d72c4f Update go.sum 2021-05-28 06:52:38 +00:00
f228a3e599 Add Nomad job registration with monitoring afterwards
Once a Nomad job is registered, we listen to the Nomad event stream
and return once we find the evaluation to complete.
2021-05-28 06:52:38 +00:00
4c3cc0cc4c Move Nomad job creation logic to environment manager 2021-05-28 06:52:38 +00:00
619cd40fb6 Refactor EntityStore interface to multiple interfaces of specific type 2021-05-27 09:56:07 +00:00
52636b04cf Allow lint step to fail
This way we can use the linter in all branches already and gradually
adjust our code to the new linter.
2021-05-27 08:18:00 +00:00
ea7eff0062 Remove deprecated golint 2021-05-27 08:18:00 +00:00
bb1f5edea8 Add golangci-lint configuration
We enable more linters to enforce better code.
2021-05-27 08:18:00 +00:00
e9906107c5 Do not use cached test results
When running the `test` and `e2e-test` target with make, this prevents
`go test` from using cached test results. Rerunning the tests every time
allows for easy detection of flaky tests.
2021-05-27 08:22:41 +02:00
ca2c989778 Make git directory static in Makefile
We used a git command to find the location of the users git directory.
This resulted in warnings in the CI where git was not installed.
For now we make the git directory static to .git to avoid this as
the directory is mostly located there.
2021-05-26 15:05:29 +00:00
fb1c117a2e Update dependencies to avoid vulns 2021-05-26 15:05:29 +00:00
1b49d396f3 Separatly test dependencies and Docker image.
This way, we don't enter all the steps in the CI if the dependencies
appear to be vulnerable.
2021-05-26 15:05:29 +00:00
8d7aa2ea3c Add trivy vulnerability scanner and custom docker-make image
Trivy is used in the CI after building our Docker image. It scans
the Docker image and our dependencies for known vulnerabilities.
The docker-make image is a simple docker:latest containing make.
2021-05-26 15:05:29 +00:00
a48804bd19 Add websocket schema for tunneling connections to the runner 2021-05-26 08:18:22 +02:00
fd89139a33 Specify missing response codes in API definition 2021-05-20 11:52:49 +02:00
5bb3c6745e Add tests for job store, Len and Sample method of pool 2021-05-19 14:52:03 +02:00
17c9839fac Refactor runners and websocket tests 2021-05-19 14:52:03 +02:00
fe2ec4df35 Incorporate review comments 2021-05-19 14:52:03 +02:00
0590f31200 Refactor setup of runner manager test suite 2021-05-19 14:52:03 +02:00
5d5f175f52 Implement http.Hijack interface in HTTPLoggingMiddleware 2021-05-19 14:52:03 +02:00
83ea552cf7 Refactor interfaces to use a runner manager and an environment manager.
See https://gitlab.hpi.de/codeocean/codemoon/poseidon/-/issues/44.
2021-05-19 14:52:03 +02:00
0d697bfd67 Merge e2e and deploy-review Nomad job for CI
This way we don't have to manage two separate job files.
2021-05-18 12:10:39 +00:00
24b7f1a2fa Create demo job when deploying for review 2021-05-18 12:10:39 +00:00
b141a6870e Use Nomad namespace in CI 2021-05-18 12:10:39 +00:00
6da9080bce Add option to configure Nomad namespace 2021-05-18 12:10:39 +00:00
d83e0e4548 Adjust README to use make targets 2021-05-18 06:16:09 +00:00
3b8e757ba8 Use make in CI 2021-05-18 06:16:09 +00:00
676d5f1e8d Add Makefile 2021-05-18 06:16:09 +00:00
1f921b2ce2 Resolve merge conflicts 2021-05-12 12:02:12 +02:00
cee5eee622 Rename job scale functions to be consistent in naming 2021-05-12 10:46:15 +02:00
a2a9ed367a Add tests for api client creation 2021-05-12 10:46:11 +02:00
9879b152e5 Add tests for Nomad job registration 2021-05-12 07:52:22 +00:00
dfd34144f8 Store default Nomad job in apiClient to cache it 2021-05-12 07:52:22 +00:00
9a5d982dfc Parse default job on client side 2021-05-12 07:52:22 +00:00
1aaad5c5fe Add default Nomad job and job creation 2021-05-12 07:52:22 +00:00
5a5ab8f02b Implement merge request review comments 2021-05-12 09:01:25 +02:00
ed735f284f Add tests for websocket connection
Co-authored-by: Konrad Hanff <konrad.hanff@student.hpi.uni-potsdam.de>
2021-05-12 09:01:25 +02:00
465577fea6 Add basic websocket structure and request upgrader 2021-05-12 09:01:25 +02:00
e07e8216f2 Deploy review only manually 2021-05-11 13:52:02 +00:00
deb4a8b116 Build Docker image and deploy API only if tests succeed 2021-05-11 13:52:02 +00:00
c455353925 Increase timeout for golang ci lint 2021-05-11 14:29:26 +02:00