Commit Graph

867 Commits

Author SHA1 Message Date
sirkrypt0
b990df7b9d Add route to create or update execution environments 2021-06-03 13:21:49 +00:00
sirkrypt0
3d395f0a38 Set network_mode to bridge to overwrite old setting
Previously, the network_mode was only set when creating a job with
network_access = false. This results in Nomad leaving this setting
as is when updating the job to use network. Thus a job would have
had the mapped ports in the Nomad UI, but the Docker network_mode
would still be 'none'.
2021-06-03 13:21:49 +00:00
Maximilian Paß
5f35ba30a2 Remove data race in the runner length function 2021-06-03 11:38:40 +02:00
Jan-Eric Hellenberg
02b3f52a11 Add ability to copy files to and delete files from runner 2021-06-02 14:54:54 +02:00
Konrad Hanff
242d0175a2 Add tests and end-to-end tests for websocket execution
For unit tests, this mocks the runners Execute method with a
customizable function that operates on the request, streams and exit
channel to simulate a real execution.

End-to-end tests are moved to the tests/e2e_tests folder. The tests
folder allows us to have shared helper functions for all tests in a
separate package (tests) that is not included in the non-test build.

This also adds one second of delay before each end-to-end test case by
using the TestSetup method of suite. By slowing down test execution,
this gives Nomad time to create new allocations when a test requested a
runner. Another solution could be to increase the scale of the job to
have enough allocations for all end-to-end tests.

Co-authored-by: Maximilian Paß <maximilian.pass@student.hpi.uni-potsdam.de>
2021-05-31 12:32:51 +02:00
Konrad Hanff
3afcdeaba8 Execute commands in runner via WebSocket
This enables executing commands in runners and forwarding input and
output between the runner and the websocket to the client.

Co-authored-by: Maximilian Paß <maximilian.pass@student.hpi.uni-potsdam.de>
2021-05-31 12:32:51 +02:00
sirkrypt0
892f902377 Add deps dependency to e2e-test make target
Without the deps dependency the e2e-test target would sometimes
fail in the CI due to missing entries in the go.sum file.
2021-05-28 09:13:59 +02:00
sirkrypt0
6084b00e23 Exit MonitorEvaluation once stream is closed
As we pass the context to the Nomad API event stream, they close
the event stream once the passed context is cancelled. We use this
to exit our receive loop on the event stream once the stream is closed,
instead of having to check the context manually.
2021-05-28 06:52:38 +00:00
sirkrypt0
a891d72c4f Update go.sum 2021-05-28 06:52:38 +00:00
sirkrypt0
f228a3e599 Add Nomad job registration with monitoring afterwards
Once a Nomad job is registered, we listen to the Nomad event stream
and return once we find the evaluation to complete.
2021-05-28 06:52:38 +00:00
sirkrypt0
4c3cc0cc4c Move Nomad job creation logic to environment manager 2021-05-28 06:52:38 +00:00
Jan-Eric Hellenberg
619cd40fb6 Refactor EntityStore interface to multiple interfaces of specific type 2021-05-27 09:56:07 +00:00
sirkrypt0
52636b04cf Allow lint step to fail
This way we can use the linter in all branches already and gradually
adjust our code to the new linter.
2021-05-27 08:18:00 +00:00
sirkrypt0
ea7eff0062 Remove deprecated golint 2021-05-27 08:18:00 +00:00
sirkrypt0
bb1f5edea8 Add golangci-lint configuration
We enable more linters to enforce better code.
2021-05-27 08:18:00 +00:00
Konrad Hanff
e9906107c5 Do not use cached test results
When running the `test` and `e2e-test` target with make, this prevents
`go test` from using cached test results. Rerunning the tests every time
allows for easy detection of flaky tests.
2021-05-27 08:22:41 +02:00
sirkrypt0
ca2c989778 Make git directory static in Makefile
We used a git command to find the location of the users git directory.
This resulted in warnings in the CI where git was not installed.
For now we make the git directory static to .git to avoid this as
the directory is mostly located there.
2021-05-26 15:05:29 +00:00
sirkrypt0
fb1c117a2e Update dependencies to avoid vulns 2021-05-26 15:05:29 +00:00
sirkrypt0
1b49d396f3 Separatly test dependencies and Docker image.
This way, we don't enter all the steps in the CI if the dependencies
appear to be vulnerable.
2021-05-26 15:05:29 +00:00
sirkrypt0
8d7aa2ea3c Add trivy vulnerability scanner and custom docker-make image
Trivy is used in the CI after building our Docker image. It scans
the Docker image and our dependencies for known vulnerabilities.
The docker-make image is a simple docker:latest containing make.
2021-05-26 15:05:29 +00:00
Jan-Eric Hellenberg
a48804bd19 Add websocket schema for tunneling connections to the runner 2021-05-26 08:18:22 +02:00
Jan-Eric Hellenberg
fd89139a33 Specify missing response codes in API definition 2021-05-20 11:52:49 +02:00
Jan-Eric Hellenberg
5bb3c6745e Add tests for job store, Len and Sample method of pool 2021-05-19 14:52:03 +02:00
Jan-Eric Hellenberg
17c9839fac Refactor runners and websocket tests 2021-05-19 14:52:03 +02:00
Jan-Eric Hellenberg
fe2ec4df35 Incorporate review comments 2021-05-19 14:52:03 +02:00
Jan-Eric Hellenberg
0590f31200 Refactor setup of runner manager test suite 2021-05-19 14:52:03 +02:00
Jan-Eric Hellenberg
5d5f175f52 Implement http.Hijack interface in HTTPLoggingMiddleware 2021-05-19 14:52:03 +02:00
Jan-Eric Hellenberg
83ea552cf7 Refactor interfaces to use a runner manager and an environment manager.
See https://gitlab.hpi.de/codeocean/codemoon/poseidon/-/issues/44.
2021-05-19 14:52:03 +02:00
sirkrypt0
0d697bfd67 Merge e2e and deploy-review Nomad job for CI
This way we don't have to manage two separate job files.
2021-05-18 12:10:39 +00:00
sirkrypto
24b7f1a2fa Create demo job when deploying for review 2021-05-18 12:10:39 +00:00
sirkrypt0
b141a6870e Use Nomad namespace in CI 2021-05-18 12:10:39 +00:00
sirkrypt0
6da9080bce Add option to configure Nomad namespace 2021-05-18 12:10:39 +00:00
sirkrypt0
d83e0e4548 Adjust README to use make targets 2021-05-18 06:16:09 +00:00
sirkrypt0
3b8e757ba8 Use make in CI 2021-05-18 06:16:09 +00:00
sirkrypt0
676d5f1e8d Add Makefile 2021-05-18 06:16:09 +00:00
Konrad Hanff
1f921b2ce2 Resolve merge conflicts 2021-05-12 12:02:12 +02:00
sirkrypt0
cee5eee622 Rename job scale functions to be consistent in naming 2021-05-12 10:46:15 +02:00
sirkrypt0
a2a9ed367a Add tests for api client creation 2021-05-12 10:46:11 +02:00
sirkrypt0
9879b152e5 Add tests for Nomad job registration 2021-05-12 07:52:22 +00:00
sirkrypt0
dfd34144f8 Store default Nomad job in apiClient to cache it 2021-05-12 07:52:22 +00:00
sirkrypt0
9a5d982dfc Parse default job on client side 2021-05-12 07:52:22 +00:00
sirkrypt0
1aaad5c5fe Add default Nomad job and job creation 2021-05-12 07:52:22 +00:00
Maximilian Paß
5a5ab8f02b Implement merge request review comments 2021-05-12 09:01:25 +02:00
Maximilian Paß
ed735f284f Add tests for websocket connection
Co-authored-by: Konrad Hanff <konrad.hanff@student.hpi.uni-potsdam.de>
2021-05-12 09:01:25 +02:00
Maximilian Paß
465577fea6 Add basic websocket structure and request upgrader 2021-05-12 09:01:25 +02:00
sirkrypt0
e07e8216f2 Deploy review only manually 2021-05-11 13:52:02 +00:00
sirkrypt0
deb4a8b116 Build Docker image and deploy API only if tests succeed 2021-05-11 13:52:02 +00:00
Jan-Eric Hellenberg
c455353925 Increase timeout for golang ci lint 2021-05-11 14:29:26 +02:00
Jan-Eric Hellenberg
fde334a610 Perform a few renamings 2021-05-11 14:26:05 +02:00
Jan-Eric Hellenberg
54df1e8ec8 Move api querier to own file 2021-05-11 14:26:05 +02:00