Commit Graph

70 Commits

Author SHA1 Message Date
ec3b2a93db Fix Golangci-lint configuration 2024-05-07 14:48:17 +02:00
ab938bfc22 Refactor MemoryLeakTestSuite
as we identified two issues where the goroutine count from before differs from after the test.

1) It seemed like a Go runtime specific Goroutine appeared in rare cases before the test. To avoid this, we introduced a short timeout before looking up the Goroutines.
Another solution might be to do the lookup twice and check if the count matches.

2) A Goroutine that periodically monitors some storage unexpectedly got closed in rare cases. As we could not identify the cause for this, we removed the leaking Goroutines by properly cleaning up.
2024-02-28 11:52:51 +01:00
221a6ff1b2 Watchdog: Verify Server TLS Certificate 2024-01-24 17:21:00 +01:00
b48c7fe8b6 Configure Systemd Watchdog
that monitors the reachability of Poseidon and automatically restarts Poseidon if required.
2024-01-24 17:21:00 +01:00
eaddc65989 Configure Systemd Socket Activation
as new way for Poseidon to accept connections. This should reduce our issues caused by deployments.
2024-01-15 16:05:35 +00:00
4ce124fdab Fix flaky TestNomadStderrFifoIsRemoved
that expected only one allocation per job. However, in practice, a job can be rescheduled on multiple allocations.
2023-11-17 23:38:06 +00:00
41eb2627ae Handle stopping goroutines of the MemoryLeakTestSuite.
We observed a race condition that sometimes goroutines are just closing but still measured by the goroutine Lookup of the following test.
We try to avoid this by adding `runtime.Gosched()` at the start of each test.
2023-11-16 12:10:57 +01:00
2ea24911de Fix inconsistent calculation of goroutines.
We observed that sometimes the `pprof.Lookup("goroutine")` resulted in a different goroutine count than `runtime.NumGoroutine()`. It might be due to a race condition, but our usage of both made it impossible to debug.
2023-11-16 12:10:57 +01:00
6b69a2d732 Refactor Nomad Recovery
from an approach that loaded the runners only once at the startup
to a method that will be repeated i.e. if the Nomad Event Stream connection interrupts.
2023-10-31 15:49:56 +01:00
3abd4d9a3d Refactor all tests to use the MemoryLeakTestSuite. 2023-09-11 13:44:29 +02:00
a01bd0fa7e Provide Memory Leak Test Suite
by adding an assertion about the number of Goroutines to the unit tests.
2023-09-11 13:44:29 +02:00
0fd6e42487 Add regression e2e test for incomplete debug message.
See #325.
2023-08-14 11:37:51 +02:00
5b64725faa Fix golangci-lint errors
that appeared due to the new version v1.53.1.
2023-06-04 11:54:42 +01:00
0218f91cd4 Disable AWS e2e tests. 2023-03-14 23:42:19 +01:00
e0419c2e58 Fix Sentry Debug Regex
that was ignoring composed messages including a newline.
Also, add regression test.
2023-03-14 23:42:19 +01:00
7dadc5dfe9 Refactor Nomad Command Generation.
- Abstracting from the exec form while generating.
- Removal of single quotes (usage of only double-quotes).
- Bash-nesting using escaping of special characters.
2023-03-14 23:42:19 +01:00
0b7f71f3dc #190 Add recovery e2e tests. 2022-11-26 13:33:44 +00:00
0c6c48c3cf #190 Add unit tests for runner recovery. 2022-11-26 13:33:44 +00:00
4c25473c9e Hide Nomad specific environment variables
from the user environment.
2022-11-04 15:15:43 +01:00
b3eee17846 Support protected directories
by setting the sticky bit to all explicitly requested directories.
2022-10-29 19:11:05 +02:00
939f36dac6 Fix e2e test exhausted cpu
by decreasing the reservation amount.
2022-10-26 01:27:33 +02:00
3e9606d1b9 Fix flaky e2e tests because of no idle runners
at startup.
2022-10-14 21:40:09 +01:00
195f88177e Add Content-Length and Content-Disposition Header
for GetFileContent route.
2022-10-05 12:11:47 +01:00
847e5cda65 Extend ls2json reader
by also parsing the link target, permissions, group and owner.
2022-10-05 12:11:47 +01:00
3469d0ce77 Specify http not found exit code
by replacing it with StatusGone (410) for a missing runner and StatusFailedDependency (424) for missing or not accessible files.
2022-10-05 12:11:47 +01:00
152b77afe5 Add listing of runners file system. 2022-10-05 12:11:47 +01:00
f2b25566dd #136 Copy files back from Nomad runner. 2022-10-05 12:11:47 +01:00
1a5a49d7c8 Explicitly switch user for code execution.
Co-authored-by: Maximilian Pass <maximilian.pass@student.hpi.uni-potsdam.de>
2022-09-24 23:09:23 +02:00
7d58d2b99c CI Debug Print AWS functions 2022-08-01 16:52:34 +02:00
0094776a88 CI Debug WIP 2022-08-01 14:39:12 +02:00
d0ad9e03a3 Suppress request noctx linter. 2022-08-01 14:35:52 +02:00
ed5205236f #166 Fix AWS JAVA ignoring shell commands in combination with a Makefile. 2022-06-21 13:24:01 +02:00
b0916a2485 #166 Adjust outdated make command pattern
as with #147 a new unset command part is introduced.

Also add a regression e2e test that would have caught this issue.
2022-06-21 13:24:01 +02:00
2f066a9fb4 E2e test environment variables. 2022-06-10 18:10:28 +02:00
0f8a1fa25a Specify AWS Functions as list
to conform with the yaml standard of list definition.
2022-06-08 09:01:46 +02:00
97a2311a74 Fix CodeClimate duplication smells
by extracting one http helper method and
increasing the threshold for duplication issues.
2022-06-07 16:59:02 +02:00
a41659eed4 Enable memory oversubscription (#102)
* Enable memory oversubscription

* Fix and add e2e test
2022-03-18 08:31:27 +01:00
4ffbb712ed Parametrize e2e tests to also check AWS environments.
- Fix destroy runner after timeout.
- Add file deletion
2022-02-28 14:54:40 +01:00
f3929cbf50 Fix webSocket error
that was introduced by closing the WebSocket connection without stopping the inputLoop.
2021-12-16 15:05:47 +01:00
f45d28d4a4 Fix flaky TestOutputToStdout test 2021-12-12 10:27:03 +01:00
825ebdd3e6 Add forcePull option
* Add forcePull option
for pulling the image when the execution environment gets updated

* Apply suggestions from code review

Co-authored-by: Sebastian Serth <MrSerth@users.noreply.github.com>

* Add unit tests

* Clean up and implement option two

Co-authored-by: Sebastian Serth <MrSerth@users.noreply.github.com>
2021-12-09 14:54:14 +01:00
24b22f2c40 Fix racing condition
when creating a Nomad job in the e2e tests
2021-10-22 16:23:51 +02:00
34d4bb7ea0 Implement routes to list, get and delete execution environments
* #9 Implement routes to list, get and delete execution environments.
A refactoring was required to introduce the ExecutionEnvironment interface.

* Fix MR comments, linting issues and bug that lead to e2e test failure

* Add e2e tests

* Add unit tests
2021-10-21 10:33:52 +02:00
9b106f4cd8 Fix linting issues
An update of golangci-lint yielded new linting issues. This commit
fixes them.
2021-08-05 13:40:48 +02:00
c8c5357b8c Rename module for GitHub 2021-07-30 16:43:05 +02:00
6a60b6cd89 Add config option to enable (m)TLS between Poseidon and Nomad 2021-07-29 09:43:21 +00:00
91537a7364 Use test docker image in e2e tests
The TestCreateOrUpdateEnvironment function would previously use
the python:latest Docker image in its execution environment request.
However, this lead to pull rate limiting by Docker Hub in our CI.
2021-07-27 15:26:53 +00:00
8b26ecbe5f Restructure project
We previously didn't really had any structure in our project apart
from creating a new folder for each package in our project root.
Now that we have accumulated some packages, we use the well-known
Golang project layout in order to clearly communicate our intent
with packages. See https://github.com/golang-standards/project-layout
2021-07-21 12:55:35 +02:00
2f1383b743 Add tests for returning mapped ports of runners 2021-07-21 08:22:10 +02:00
c7606f3d5f Fix a lot of linting issues
After we introduced the linter we haven't really touched the old code.
This commit now fixes all linting issue that exist right now.
2021-07-13 08:59:25 +02:00