Compare commits

..

205 Commits

Author SHA1 Message Date
Matthew Penner
d739948989
feat: add ability to mount generated passwd files to containers (#197)
This PR will add an option to mount:
- `/etc/group`
- `/etc/passwd`

Signed-off-by: Matthew Penner <me@matthewp.io>
2024-07-24 15:12:42 -06:00
Matthew Penner
ac260bd5ee
ci: update to go1.22.5, drop go1.21
Signed-off-by: Matthew Penner <me@matthewp.io>
2024-07-24 13:14:25 -06:00
Matthew Penner
2f4a0d7262
nix: update flake.lock
Flake lock file updates:

• Updated input 'flake-parts':
    'github:hercules-ci/flake-parts/b253292d9c0a5ead9bc98c4e9a26c6312e27d69f?narHash=sha256-a0NYyp%2Bh9hlb7ddVz4LUn1vT/PLwqfrWYcHMvFB1xYg%3D' (2024-02-01)
  → 'github:hercules-ci/flake-parts/9227223f6d922fee3c7b190b2cc238a99527bbb7?narHash=sha256-pQMhCCHyQGRzdfAkdJ4cIWiw%2BJNuWsTX7f0ZYSyz0VY%3D' (2024-07-03)
• Updated input 'flake-parts/nixpkgs-lib':
    'github:NixOS/nixpkgs/97b17f32362e475016f942bbdfda4a4a72a8a652?dir=lib&narHash=sha256-UcsnCG6wx%2B%2B23yeER4Hg18CXWbgNpqNXcHIo5/1Y%2Bhc%3D' (2024-01-29)
  → '5daf051448.tar.gz?narHash=sha256-Fm2rDDs86sHy0/1jxTOKB1118Q0O3Uc7EC0iXvXKpbI%3D' (2024-07-01)
• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/a4d4fe8c5002202493e87ec8dbc91335ff55552c?narHash=sha256-ZL2TrjVsiFNKOYwYQozpbvQSwvtV/3Me7Zwhmdsfyu4%3D' (2024-02-15)
  → 'github:NixOS/nixpkgs/68c9ed8bbed9dfce253cc91560bf9043297ef2fe?narHash=sha256-Tybxt65eyOARf285hMHIJ2uul8SULjFZbT9ZaEeUnP8%3D' (2024-07-21)
• Updated input 'treefmt-nix':
    'github:numtide/treefmt-nix/ac599dab59a66304eb511af07b3883114f061b9d?narHash=sha256-qQF0fEkHlnxHcrKIMRzOETnRBksUK048MXkX0SOmxvA%3D' (2024-02-07)
  → 'github:numtide/treefmt-nix/8db8970be1fb8be9c845af7ebec53b699fe7e009?narHash=sha256-6Pqa0bi5nV74IZcENKYRToRNM5obo1EQ%2B3ihtunJ014%3D' (2024-07-23)
2024-07-24 13:10:35 -06:00
EpicPlayerA10
1d8b383682
fix: only count hard-links once when calculating filesystem usage (#181) 2024-06-29 12:56:51 -06:00
Arnaud Lier
934bf2493d
fix: properly use base2 (1024, *bibyte) when calculating memory limits (#190) 2024-06-29 12:34:20 -06:00
Daniel Barton
29e4425e21
fix: overhaul docker container termination signals (#192)
Fixes https://github.com/pterodactyl/panel/issues/4783

Requires https://github.com/pterodactyl/panel/pull/5132 to work
2024-06-29 12:31:36 -06:00
Daniel Barton
5a15612754
chore: show the actual location in "config not found" error (#179) 2024-06-29 12:26:59 -06:00
Daniel Barton
ad1ae862a9
fix: user-defined labels not being passed to environment (#191) 2024-06-29 12:25:47 -06:00
Matthew Penner
3114a3b82e
Update README.md
Signed-off-by: Matthew Penner <me@matthewp.io>
2024-06-22 19:21:54 -06:00
Matthew Penner
500f217514
Update CHANGELOG.md 2024-05-13 16:01:33 -06:00
Daniel Barton
9ffbcdcdb1
cmd: handle relative paths to the config file (#180) 2024-05-13 15:53:33 -06:00
Daniel Barton
9b341db2db
server(filesystem): fix sort position of directories (#188) 2024-05-13 15:52:10 -06:00
Matthew Penner
71c5338549
Update CHANGELOG.md 2024-05-07 22:12:59 -06:00
Matthew Penner
326f115f5b
Update README.md 2024-05-07 22:12:55 -06:00
Geri
06614de99d
server(filesystem): handle individual compressed files (#184) 2024-05-07 22:06:15 -06:00
Matthew Penner
2b0e35360b
cmd(configure): fix panel url not being set
Fixes https://github.com/pterodactyl/panel/issues/5087
2024-05-07 22:01:12 -06:00
Matthew Penner
202f2229a9
Update README.md 2024-05-03 10:09:21 -06:00
Matthew Penner
baf1f0b5cd
Update CHANGELOG.md 2024-05-03 10:09:18 -06:00
Daniel Barton
ec54371b86
cmd(diagnostics): fix Content-Type and accept more status codes (#186) 2024-04-16 15:15:19 -06:00
Matthew Penner
1d5090957b
ci: use go 1.21.9 and 1.22.2 2024-04-10 17:16:28 -06:00
Matthew Penner
5415f8ae07
config: prevent programmatic updates to specific fields 2024-04-10 15:27:54 -06:00
Matthew Penner
617fbcbf27
router(download): validate that backup_uuid is actually a uuid 2024-04-10 15:22:39 -06:00
Matthew Penner
c152e36101
downloader: move internal subnet validation into http Transport 2024-04-10 15:22:09 -06:00
Matthew Penner
5b0422d756
go: update dependencies 2024-04-10 15:21:37 -06:00
Matthew Penner
f1c5bbd42d
server(filesystem): allow decompressing individual files
Implements logic for handling a file that is compressed but isn't an archive.

Fixes https://github.com/pterodactyl/panel/issues/5034

Signed-off-by: Matthew Penner <me@matthewp.io>
2024-03-17 14:58:30 -06:00
Matthew Penner
1c5ddcd20c
server(filesystem): handle writing empty files
Fixes https://github.com/pterodactyl/panel/issues/5038

Signed-off-by: Matthew Penner <me@matthewp.io>
2024-03-17 14:58:13 -06:00
Matthew Penner
a877305202
Update README.md 2024-03-16 14:02:13 -06:00
Matthew Penner
1f77d2256b
Update CHANGELOG.md
Signed-off-by: Matthew Penner <me@matthewp.io>
2024-03-14 14:01:13 -06:00
Matthew Penner
ac9bd1d95e
server(filesystem): fix archives in subdirectories
When creating an archive starting from a subdirectory instead of the
root of a server's filesystem, the walker would treat the paths as if
they where relative to the parent of the subdirectory, instead of as
descendants of the subdirectory.

Fixes https://github.com/pterodactyl/panel/issues/5030
2024-03-14 13:46:36 -06:00
Matthew Penner
979df34392
server(filesystem): fix archive creation with ignore
Signed-off-by: Matthew Penner <me@matthewp.io>
2024-03-13 20:29:09 -06:00
Matthew Penner
8f129931d5
Update CHANGELOG.md
Signed-off-by: Matthew Penner <me@matthewp.io>
2024-03-13 15:19:24 -06:00
Matthew Penner
2931430eb8
server(filesystem): fix archive skipping directories
Fixes https://github.com/pterodactyl/panel/issues/5027

Signed-off-by: Matthew Penner <me@matthewp.io>
2024-03-13 15:09:24 -06:00
Matthew Penner
99b9924a4a
router(server): fix panic on server deletion
Fixes https://github.com/pterodactyl/panel/issues/5028

Signed-off-by: Matthew Penner <me@matthewp.io>
2024-03-13 14:46:17 -06:00
Matthew Penner
d649bb1116
Update README.md
Signed-off-by: Matthew Penner <me@matthewp.io>
2024-03-13 12:27:57 -06:00
Matthew Penner
1477b7034b
Update CHANGELOG.md
Signed-off-by: Matthew Penner <me@matthewp.io>
2024-03-13 12:27:36 -06:00
Matthew Penner
d1c0ca5260
server(filesystem): rebuild everything imaginable
This wonderfully large commit replaces basically everything under the
`server/filesystem` package, re-implementing essentially everything.

This is related to
https://github.com/pterodactyl/wings/security/advisories/GHSA-494h-9924-xww9

If any vulnerabilities related to symlinks persist after this commit, I
will be very upset.

Signed-off-by: Matthew Penner <me@matthewp.io>
2024-03-13 12:27:29 -06:00
Matthew Penner
27f3e76c77
ci: update to go 1.21.8 and 1.22.1
Signed-off-by: Matthew Penner <me@matthewp.io>
2024-03-12 21:43:50 -06:00
Matthew Penner
eadbe920fe
config: fix docker Go API deprecation
Signed-off-by: Matthew Penner <me@matthewp.io>
2024-03-12 21:42:58 -06:00
Matthew Penner
3e804b81fe
go: update dependencies
Signed-off-by: Matthew Penner <me@matthewp.io>
2024-03-12 16:25:54 -06:00
Matthew Penner
f68965e7c9
Update README.md 2023-12-29 22:12:22 -07:00
Matthew Penner
accc833e87
ci(push): only upload artifacts built with go 1.20.12 2023-12-19 14:44:57 -07:00
Matthew Penner
d4bfdd4548
ci: actions/upload-artifact v3 -> v4 2023-12-19 14:42:11 -07:00
Matthew Penner
2641080007
ci: update go to 1.20.12 and 1.21.5 2023-12-19 14:38:48 -07:00
Matthew Penner
10c58d3dc0
go: update dependencies 2023-12-19 14:38:21 -07:00
Matthew Penner
9496b1f7e5
go: update dependencies 2023-12-02 21:19:07 -07:00
Matthew Penner
2f1b67ed35
ci: update go to 1.20.11 and 1.21.4 2023-12-02 21:13:52 -07:00
Matthew Penner
579278b4de
sftp: log ip on inbound connection failure 2023-12-02 21:11:24 -07:00
Daniel Barton
d30ab7b9bd
dockerfile: update to go 1.20.10 (#175) 2023-10-17 20:49:39 -06:00
Daniel Barton
d1fd0465e4
server(filesystem): fix file names when extracting archives (#165) 2023-10-17 20:47:36 -06:00
Matthew Penner
79eb8e1365
Update CHANGELOG.md 2023-10-10 12:53:51 -06:00
Matthew Penner
2cb201d202
go: update dependencies 2023-10-10 12:46:36 -06:00
Matthew Penner
fc1ffc8cd3
workflows: update actions, use go 1.20.10 and 1.21.3 2023-10-10 12:43:02 -06:00
guangwu
48c55af373
chore: slice loop replace (#173) 2023-10-02 17:24:17 -06:00
guangwu
7a59d0929c
chore: remove unnecessary use of fmt.Sprintf (#174) 2023-10-02 17:23:45 -06:00
Matthew Penner
9b5eaf44df
Update README.md 2023-10-02 17:05:34 -06:00
Matthew Penner
438e5fdbe9
Update CHANGELOG.md 2023-07-24 22:48:10 -06:00
Matthew Penner
a866493d0a
go: update dependencies 2023-07-24 13:47:53 -06:00
Matthew Penner
c9d92f7bac
ci: ubuntu-22.04, go1.19.11, go1.20.6 2023-07-15 14:40:45 -06:00
Matthew Penner
aa8ffdfcf7
go: update dependencies
This updates the docker client to include the fix added with
https://github.com/moby/moby/pull/45962 which solves a breaking change
to Go due to CVE-2023-29406.
2023-07-15 14:38:31 -06:00
Matthew Penner
8d7e23f542
Update README.md 2023-06-28 20:25:06 -06:00
Matthew Penner
bd26d6eefd
Update README.md 2023-05-12 23:13:09 -06:00
Matthew Penner
9441d2a523
Update CHANGELOG.md 2023-05-10 13:28:00 -06:00
Matthew Penner
4d51de71c2
server(filesystem): fix test failure 2023-05-10 12:59:22 -06:00
Matthew Penner
4b66a222cd
ci: update go (1.19.7 -> 1.19.9, 1.20.2 -> 1.20.4) 2023-05-10 12:37:35 -06:00
Matthew Penner
b665c943a2
server(install): remove privileges from install container 2023-05-10 12:35:46 -06:00
Matthew Penner
a50e4ce9d1
go: update dependencies 2023-05-08 13:52:15 -06:00
Matthew Penner
c76d68bc96
Update README.md 2023-03-11 14:12:46 -07:00
Matthew Penner
02cb64e31b
Update CHANGELOG.md 2023-03-07 15:57:12 -07:00
Matthew Penner
639ad76be3
go: update dependencies 2023-03-07 15:36:35 -07:00
Matthew Penner
a373bf8eda
system: fix sink pool test 2023-03-07 15:36:20 -07:00
Matthew Penner
74b1c46b7f
server(filesystem): fix archive test 2023-03-07 15:29:29 -07:00
Matthew Penner
5424c6718e
ci: update to go 1.19 and 1.20 2023-03-07 15:20:49 -07:00
Matthew Penner
43b3496f00
server(filesystem): fix archiver path matching
Closes https://github.com/pterodactyl/panel/issues/4630
2023-03-07 15:19:09 -07:00
dependabot[bot]
38c69ebfda
build(deps): bump golang.org/x/net (#163)
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.0.0-20221004154528-8021a29435af to 0.7.0.
- [Release notes](https://github.com/golang/net/releases)
- [Commits](https://github.com/golang/net/commits/v0.7.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-02-27 19:54:18 -07:00
dependabot[bot]
234e11b28b
build(deps): bump golang.org/x/text from 0.3.7 to 0.3.8 (#161)
Bumps [golang.org/x/text](https://github.com/golang/text) from 0.3.7 to 0.3.8.
- [Release notes](https://github.com/golang/text/releases)
- [Commits](https://github.com/golang/text/compare/v0.3.7...v0.3.8)

---
updated-dependencies:
- dependency-name: golang.org/x/text
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-02-27 13:08:10 -07:00
Jakob Schrettenbrunner
ec6d6d83ea
config: add option to disable panel config updates (#162) 2023-02-27 13:05:36 -07:00
Matthew Penner
4d9fee383f
Update CHANGELOG.md 2023-02-08 14:21:02 -07:00
Matthew Penner
429ac62dba
server(filesystem): Delete tweaks 2023-02-08 14:20:13 -07:00
Matthew Penner
020abec6f2
Update CHANGELOG.md 2023-02-07 19:15:11 -07:00
Matthew Penner
dac9685298
server(filesystem): SafePath tweaks 2023-02-07 19:14:35 -07:00
Matthew Penner
519d38f238
Update README.md 2023-02-06 11:00:25 -07:00
Matthew Penner
1d17233d6d
Update README.md 2023-02-06 10:25:20 -07:00
Matthew Penner
774c0af0b0
Update CHANGELOG.md 2023-01-30 18:33:30 -07:00
Alexander Trost
71fbd9271e
activity: fix ip validity check (#159) 2023-01-30 09:09:36 -07:00
Matthew Penner
2d640209e5
backup: fix restore erroring due to closed reader 2023-01-29 17:06:49 -07:00
Matthew Penner
304fd91283
Update CHANGELOG.md 2023-01-27 12:14:22 -07:00
Matthew Penner
18de96d7b8
activity: fix IP parsing, drop all columns with malformed ips 2023-01-24 14:36:18 -07:00
Matthew Penner
a36cab1783
router(transfer): throw error if server fails to stop 2023-01-24 12:36:02 -07:00
Matthew Penner
6e0c095bb8
router(transfer): decrease WaitForStop timeout 2023-01-24 12:34:05 -07:00
Matthew Penner
14eea3b1e4
router: close body once finished 2023-01-24 12:33:42 -07:00
Matthew Penner
1bc77dc969
system: close Docker client once finished 2023-01-24 12:33:20 -07:00
Matthew Penner
b8715d1d4f
ci: update push-artifact action 2023-01-24 12:32:55 -07:00
Matthew Penner
13d3490bcf
server(filesystem): fix Writefile being broken 2023-01-17 18:44:56 -07:00
Matthew Penner
e9b8b11fec
Ensure files are closed after they are done being used 2023-01-17 18:34:08 -07:00
Matthew Penner
43b7aa2536
sftp: disable insecure protocols 2023-01-17 11:50:06 -07:00
Matthew Penner
9b8b3c90fb
environment(docker): improve logging and stacks 2023-01-17 11:47:27 -07:00
Matthew Penner
e74d8e3501
ci: update go versions 2023-01-13 11:07:09 -07:00
Matthew Penner
4b3bd2ff47
ci(docker): fix latest tag 2022-12-04 18:37:53 -07:00
Matthew Penner
e652d2df84
ci: cleanup 2022-12-04 17:39:55 -07:00
Matthew Penner
e4d790ea40
ci: remove use of deprecated ::set-output 2022-12-04 17:29:36 -07:00
Matthew Penner
5641e45059
ci: overhaul workflows 2022-12-04 17:24:33 -07:00
Matthew Penner
9a718b699f
Update CHANGELOG.md 2022-12-04 15:23:29 -07:00
Matthew Penner
92efdb1981
remote: use POST for server tranfer status 2022-12-01 11:50:56 -07:00
Matthew Penner
43227bf24d
server(transfer): fix dead-lock while uploading archive
Previously we waited for both the request and multipart writer
to "complete", before handing any errors.  This lead to a problem
where if the request returns before all the data has been read,
the upload would become stuck and keep the server in a transferring
state when the transfer should've been aborted.

Closes https://github.com/pterodactyl/panel/issues/4578
2022-11-29 14:04:27 -07:00
Matthew Penner
105f0150f6
Update CHANGELOG.md 2022-11-22 13:49:10 -07:00
Matthew Penner
aeec51632e
Update README.md 2022-11-22 13:48:56 -07:00
Dane Everitt
ff50d0e5bd
Cleanup request error handling; properly handle os.ErrNotExist errors (#150) 2022-11-22 11:18:27 -07:00
Matthew Penner
9226ccae31
system: more detailed system information 2022-11-21 16:01:14 -07:00
Matthew Penner
2fd0edbff9
environment(docker): fix timeout when sending a stop signal
Previously, Docker would terminate the container when it's stop
configuration was configured to send a signal to the container.
This was due to Docker's API wanting the value as a duration string
(`1s`) rather than a number, so our value of `-1` was being formatted
to `0s` rather than `-1s` like we needed.

Closes https://github.com/pterodactyl/panel/issues/4555
2022-11-21 15:06:38 -07:00
Matthew Penner
1457470fff
environment(docker): fix outgoing ip not changing
Fixes an issue where when a server has it's primary allocation
changed, the outgoing ip is not updated on the Docker network.

The only downside of this change is old networks are not cleaned up.

Closes https://github.com/pterodactyl/panel/issues/4547
2022-11-21 14:59:53 -07:00
Matthew Penner
da94f750ad
server(install): update installation status request
The new request includes a `reinstall` option to denote
whether the installation failed for the first time, or
during a reinstall.

ref https://github.com/pterodactyl/panel/issues/1994
2022-11-21 14:57:44 -07:00
Matthew Penner
51cb6dfa42
ci(release): fix wrong directory name 2022-11-14 20:33:06 -07:00
Matthew Penner
47f94b8358
Update CHANGELOG.md 2022-11-14 20:26:20 -07:00
Matthew Penner
57e7eb714c
Streaming Transfers (#153) 2022-11-14 18:25:01 -07:00
Matthew Penner
4781eeaedc
ci: fix arm64 failures 2022-11-06 14:39:02 -07:00
Matthew Penner
a5a8200eb8
ci: cleanup 2022-11-06 13:55:49 -07:00
Matthew Penner
eb4df39d14
server(filesystem): fix inaccurate archive progress (#145) 2022-11-06 13:38:30 -07:00
Matthew Penner
3337362955
environment(docker): fix podman compatibility (#151) 2022-11-06 13:33:01 -07:00
Matthew Penner
f577f5521f
server(filesystem): fix panic with archive Progress 2022-11-04 11:24:19 -06:00
Cubxity
5a760a0dcc
Add customizable container labels (#146) 2022-10-16 17:17:27 -06:00
Matthew Penner
058f643e65
chore: reformat code 2022-10-06 09:58:42 -06:00
Matthew Penner
6c7065592d
ci: remove binary compression 2022-10-05 21:27:33 -06:00
Matthew Penner
3f481e9540
chore: run gofumpt 2022-10-05 17:24:11 -06:00
Matthew Penner
984bd10cf2
Update CHANGELOG.md 2022-10-04 20:43:05 -06:00
Matthew Penner
f5a64a0d7f
docker: use go1.18 2022-10-04 20:42:48 -06:00
Matthew Penner
6fb61261b0
server(transfers): track progress of archive creation and extraction (#143) 2022-10-04 20:35:48 -06:00
Matthew Penner
3edec80efa
go: update minimum to 1.18, update dependencies (#144) 2022-10-04 20:13:27 -06:00
argetlam-coder
0637eebefe
docker: add configuration for user namespace remapping (#121) 2022-10-04 18:12:13 -06:00
Daniel Barton
e98d249cf7
Add configuration for trusted proxies (#141) 2022-10-04 18:11:07 -06:00
Matthew Penner
b20bf6deab
ci: pin go-version to 1.18.7 2022-10-04 13:22:46 -06:00
Matthew Penner
1b268b5625
Update FUNDING.yml 2022-09-26 11:23:01 -06:00
Matthew Penner
7245791214
backups(s3): send uploaded parts in complete request 2022-09-26 11:14:57 -06:00
Matthew Penner
02cbf2df5b
Update README.md 2022-09-25 19:16:29 -06:00
Matthew Penner
b6edf3acf9
environment(docker): set outgoing ip correctly (#135)
Closes https://github.com/pterodactyl/panel/issues/3841
2022-09-25 18:49:48 -06:00
PotatoMaaan
c686992e85
backups: add an option to change gzip compression level (#128) 2022-09-25 18:47:09 -06:00
camjac251
c736c24118
it's to its (#123) 2022-09-25 13:34:28 -06:00
Chance Callahan
9dfc651a91
rpm: update to 1.7.0 (#140) 2022-09-25 13:25:53 -06:00
Matthew Penner
ad26022c30
parser(yaml): fix issues
Closes https://github.com/pterodactyl/panel/issues/4236
Closes https://github.com/pterodactyl/wings/pull/139
2022-09-21 11:50:55 -06:00
DaneEveritt
83861a6dec
Update CHANGELOG.md 2022-07-24 19:43:43 -04:00
DaneEveritt
231e24aa33
Support new metadata from panel for servers 2022-07-24 17:16:45 -04:00
DaneEveritt
e3ab241d7f
Track file upload activity 2022-07-24 17:12:47 -04:00
DaneEveritt
c18e844689
Support more rapid insertion; ignore issues with i/o 2022-07-24 16:58:03 -04:00
DaneEveritt
8cee18a92b
Save activity in a background routine to speed things along; cap query time at 3 seconds 2022-07-24 16:27:25 -04:00
DaneEveritt
f952efd9c7
Don't try to store nil for the metadata 2022-07-24 16:27:05 -04:00
DaneEveritt
21cf66b2b4
Use single connection in pool to avoid simultaneous write lock issues 2022-07-24 16:26:52 -04:00
DaneEveritt
251f91a08e
Fix crons to actually run correctly using the configuration values 2022-07-24 15:59:17 -04:00
DaneEveritt
4634c93182
Add cron to handle parsing SFTP events 2022-07-24 14:40:06 -04:00
DaneEveritt
8a867ccc44
Switch to gorm for activity logging 2022-07-24 11:43:48 -04:00
DaneEveritt
61baccb1a3
Push draft of sftp reconcilation details 2022-07-24 10:28:42 -04:00
DaneEveritt
7bd11c1c28
Switch to SQLite for activity tracking 2022-07-10 16:51:11 -04:00
DaneEveritt
e1e7916790
Handle ErrKeyNotFound as a non-error 2022-07-10 14:53:54 -04:00
DaneEveritt
f28e06267c
Better tracking of SFTP events 2022-07-10 14:30:32 -04:00
DaneEveritt
59fbd2bcea
Add initial niaeve implementation of SFTP logging
This will end up flooding the activity logs due to the way SFTP works, we'll need to have an intermediate step in Wings that batches events every 10 seconds or so and submits them as a single "event" for activity.
2022-07-09 19:37:42 -04:00
Michael
204a4375fc
Make the Docker network MTU configurable (#130) 2022-07-09 18:08:52 -04:00
DaneEveritt
dda7d10d37
Use the natural panel event names 2022-07-09 17:52:59 -04:00
DaneEveritt
ed330fa6be
Squashed commit of the following:
commit f5baab4e88
Author: DaneEveritt <dane@daneeveritt.com>
Date:   Sat Jul 9 17:50:53 2022 -0400

    Finalize activity event sending logic and cron config

commit 9830387f21
Author: DaneEveritt <dane@daneeveritt.com>
Date:   Sat Jul 9 16:26:13 2022 -0400

    Send power events in a more usable format

commit 49f3a61d16
Author: DaneEveritt <dane@daneeveritt.com>
Date:   Sat Jul 9 15:47:24 2022 -0400

    Configure cron to actually send to endpoint

commit 28137c4c14
Author: DaneEveritt <dane@daneeveritt.com>
Date:   Sat Jul 9 15:42:29 2022 -0400

    Copy the body buffer otherwise subsequent backoff attempts will not have a buffer to send

commit 20e44bdc55
Author: DaneEveritt <dane@daneeveritt.com>
Date:   Sat Jul 9 14:38:41 2022 -0400

    Add internal logic to process activity events and send them to the panel

commit 0380488cd2
Author: DaneEveritt <dane@daneeveritt.com>
Date:   Mon Jul 4 17:55:17 2022 -0400

    Track power events

commit 9eab08b92f
Author: DaneEveritt <dane@daneeveritt.com>
Date:   Mon Jul 4 17:36:03 2022 -0400

    Initial logic to support logging activity on Wings to send back to the panel
2022-07-09 17:51:19 -04:00
DaneEveritt
9864a0fe34
Update README.md 2022-07-09 12:01:15 -04:00
Michael (Parker) Parker
214baf83fb
Fix/arm64 docker (#133)
* fix: arm64 docker builds

Don't hardcode amd64 platform for the Wings binary.

* update docker file

don't specify buildplatform
remove upx as it causes arm64 failures
remove goos as the build is on linux hosts.

Co-authored-by: softwarenoob <admin@softwarenoob.com>
2022-07-03 11:09:07 -04:00
Dane Everitt
41fc1973d1
Update README.md 2022-06-24 10:51:58 -04:00
DaneEveritt
a51ce6f4ac
Update README.md 2022-06-16 20:37:01 -04:00
DaneEveritt
cec51f11f0
Update CHANGELOG.md 2022-05-31 14:36:24 -04:00
DaneEveritt
b1be2081eb
Better archive detection logic; try to use reflection as last ditch effort if unmatched
closes pterodactyl/panel#4101
2022-05-30 18:42:31 -04:00
DaneEveritt
203a2091a0
Use the correct CPU period when throttling servers; closes pterodactyl/panel#4102 2022-05-30 17:45:41 -04:00
DaneEveritt
7fa7cc313f
Fix permissions not being checked correctly for admins 2022-05-29 21:48:49 -04:00
DaneEveritt
f390784973
Include error in log output if one occurs during move 2022-05-21 17:01:12 -04:00
DaneEveritt
5df1acd10e
We don't return public keys 2022-05-15 16:41:26 -04:00
DaneEveritt
1927a59cd0
Send key correctly; don't retry 4xx errors 2022-05-15 16:17:06 -04:00
DaneEveritt
5bcf4164fb
Add support for public key based auth 2022-05-15 16:01:52 -04:00
DaneEveritt
37e4d57cdf
Don't include files and folders with identical name prefixes when archiving; closes pterodactyl/panel#3946 2022-05-12 18:00:55 -04:00
DaneEveritt
7ededdb9a2
Update CHANGELOG.md 2022-05-12 17:57:26 -04:00
DaneEveritt
1d197714df
Fix faulty handling of named pipes; closes pterodactyl/panel#4059 2022-05-07 15:53:08 -04:00
DaneEveritt
6c98a955e3
Only set cpu limits if specified; closes pterodactyl/panel#3988 2022-05-07 15:23:56 -04:00
Matthew Penner
8bd1ebe360
go: update dependencies 2022-03-25 10:04:57 -06:00
Matthew Penner
93664fd112
router: add additional fields to remote file pull 2022-02-23 15:03:15 -07:00
Matthew Penner
3a738e44d6
run gofumpt 2022-02-23 15:02:19 -07:00
Noah van der Aa
067ca5bb60
Actually enforce upload file size limit (#122) 2022-02-21 14:59:28 -08:00
Dane Everitt
f85509a0c7 Support a custom tmp directory location 2022-02-13 11:59:53 -05:00
Dane Everitt
225a89be72 Update CHANGELOG.md 2022-02-05 12:41:53 -05:00
Dane Everitt
5d1d3cc9e6 Fix panic conditions 2022-02-05 12:11:00 -05:00
Dane Everitt
9f985ae044 Check for error before prefix; fixes abandoned routine; closes pterodactyl/panel#3911
Due to the order of the previous logic in ScanReader, an error not caused by EOF would effectively get ignored since an error will always be returned with `isPrefix` equal to false, thus triggering the first break, and error checking is not performed beyond that point.

Thus, canceling an installation process for a server while this process was running would hang the routine and cause the loop to run endlessly, even with a canceled context.
2022-02-05 11:56:17 -05:00
Dane Everitt
1372eba84e Remove unused function 2022-02-05 11:14:48 -05:00
Dane Everitt
879dcd8df5 Don't trigger a panic condition decoding event stats; closes pterodactyl/panel#3941 2022-02-05 11:06:11 -05:00
Dane Everitt
72476c61ec Simplify the event bus system; address pterodactyl/panel#3903
If my debugging is correct, this should address pterodactyl/panel#3903 in its entirety by addressing a few areas where it was possible for a channel to lock up and cause everything to block
2022-02-02 21:03:53 -05:00
Dane Everitt
0f2e9fcc0b Move the sink pool to be a shared tool 2022-02-02 19:16:34 -05:00
Dane Everitt
5c3e2c2c94 Fix failing test 2022-01-31 19:33:32 -05:00
Dane Everitt
7051feee01 Add additional debug points to server start process 2022-01-31 19:30:07 -05:00
Dane Everitt
cd67e5fdb9 Fix logic for context based environment stopping
Uses dual contexts to handle stopping using a timed context, and also terminating the entire process loop if the parent context gets canceled.
2022-01-31 19:09:08 -05:00
Dane Everitt
84bbefdadc Pass a context through to the start/stop/terminate actions 2022-01-31 18:40:15 -05:00
Dane Everitt
6a4178648f Return context cancelations as a locker locked error 2022-01-31 18:39:41 -05:00
Dane Everitt
1e52ffef64 Fix panic condition when no response is returned 2022-01-31 18:37:02 -05:00
Dane Everitt
0f9f80c181 Improve support for block/mutex contention in pprof 2022-01-30 21:02:18 -05:00
Dane Everitt
4b702052c7 Update CHANGELOG.md 2022-01-30 20:27:26 -05:00
Dane Everitt
7ee6c48fb0 Use a more efficient logging format for containers
JSON has a huge amount of overhead from Docker when we're trying to process large amounts of log data. It makes more sense to just use a better format.
2022-01-30 19:51:23 -05:00
Dane Everitt
2b2b5200eb Rewrite console throttling logic; drop complex timer usage and use a very simple throttle
This also removes server process termination logic when a server is breaching the output limits. It simply continues to efficiently throttle the console output.
2022-01-30 19:31:04 -05:00
Dane Everitt
fb73d5dbbf Always run pprof when running debug through makefile 2022-01-30 15:11:17 -05:00
Dane Everitt
fd7ec2aaac Generate normal and debug artifacts 2022-01-30 15:06:56 -05:00
Dane Everitt
c3df8d2309 Add support for proper use of pprof 2022-01-30 14:50:37 -05:00
Dane Everitt
1965e68a78 Include debug symbols in non-release binaries 2022-01-30 14:05:55 -05:00
Dane Everitt
3208b8579b Add test coverage 2022-01-30 13:58:36 -05:00
Dane Everitt
c4ee82c4dc Code cleanup, providing better commentary to decisions 2022-01-30 12:56:25 -05:00
Dane Everitt
0ec0fffa4d Handle future scenarios where we forgot to add a listener 2022-01-30 11:58:53 -05:00
Dane Everitt
57daf0889a Cleanup logic for updating stats to avoid calling mutex outside of file 2022-01-30 11:55:59 -05:00
Dane Everitt
d7c7155802 Make the powerlocker logic a little more idiomatic 2022-01-30 11:46:27 -05:00
Dane Everitt
11ae5e69ed Improve performance of console output watcher; work directly with bytes rather than string conversions 2022-01-30 11:28:06 -05:00
Dane Everitt
fab88a380e Use buffered channels and ring-buffer logic when processing console data
This change fixes pterodactyl/panel#3921 by implementing logic to drop the oldest message in a channel and push the newest message onto the channel when the channel buffer is full.

This is distinctly different than the previous implementation which just dropped the newest messages, leading to confusing behavior on the client side when a large amount of data was sent over the connection.

Up to 10ms per channel is allowed for blocking before falling back to the drop logic.
2022-01-30 10:55:45 -05:00
139 changed files with 9316 additions and 5074 deletions

1
.envrc Normal file
View File

@ -0,0 +1 @@
use flake

1
.github/FUNDING.yaml vendored Normal file
View File

@ -0,0 +1 @@
github: [matthewpi]

2
.github/FUNDING.yml vendored
View File

@ -1,2 +0,0 @@
github: [ DaneEveritt ]
custom: [ "https://paypal.me/PterodactylSoftware" ]

View File

@ -1,71 +0,0 @@
name: Run Tests
on:
push:
branches:
- develop
pull_request:
branches:
- develop
jobs:
build:
strategy:
fail-fast: false
matrix:
os: [ ubuntu-20.04 ]
go: [ '^1.17' ]
goos: [ linux ]
goarch: [ amd64, arm64 ]
runs-on: ${{ matrix.os }}
steps:
- name: Code Checkout
uses: actions/checkout@v2
- name: Setup Go v${{ matrix.go }}
uses: actions/setup-go@v2
with:
go-version: ${{ matrix.go }}
- name: Print Environment
id: env
run: |
printf "Go Executable Path: $(which go)\n"
printf "Go Version: $(go version)\n"
printf "\n\nGo Environment:\n\n"
go env
printf "\n\nSystem Environment:\n\n"
env
printf "Git Version: $(git version)\n\n"
echo "::set-output name=version_tag::${GITHUB_REF/refs\/tags\//}"
echo "::set-output name=short_sha::$(git rev-parse --short HEAD)"
echo "::set-output name=go_cache::$(go env GOCACHE)"
echo "::set-output name=go_mod_cache::$(go env GOMODCACHE)"
- name: Build Cache
uses: actions/cache@v2
with:
key: ${{ runner.os }}-go${{ matrix.go }}-${{ hashFiles('**/go.sum') }}
restore-keys: |
${{ runner.os }}-go${{ matrix.go }}-
path: |
${{ steps.env.outputs.go_cache }}
${{ steps.env.outputs.go_mod_cache }}
- name: Get Dependencies
run: |
go get -v -t -d ./...
- name: Build
env:
GOOS: ${{ matrix.goos }}
GOARCH: ${{ matrix.goarch }}
CGO_ENABLED: 0
SRC_PATH: github.com/pterodactyl/wings
run: |
go build -v -trimpath -ldflags="-s -w -X ${SRC_PATH}/system.Version=dev-${GIT_COMMIT:0:7}" -o build/wings_${{ matrix.goos }}_${{ matrix.goarch }} wings.go
upx build/wings_${{ matrix.goos }}_${{ matrix.goarch }}
chmod +x build/wings_${{ matrix.goos }}_${{ matrix.goarch }}
- name: Tests
run: go test ./...
- name: Tests (Race)
run: go test -race ./...
- name: Upload Artifact
uses: actions/upload-artifact@v2
if: ${{ github.ref == 'refs/heads/develop' || github.event_name == 'pull_request' }}
with:
name: wings_${{ matrix.goos }}_${{ matrix.goarch }}
path: build/wings_${{ matrix.goos }}_${{ matrix.goarch }}

View File

@ -1,4 +1,5 @@
name: CodeQL name: CodeQL
on: on:
push: push:
branches: branches:
@ -7,24 +8,35 @@ on:
branches: branches:
- develop - develop
schedule: schedule:
- cron: '0 9 * * 4' - cron: "0 9 * * 4"
jobs: jobs:
analyze: analyze:
name: Analyze name: Analyze
runs-on: ubuntu-latest runs-on: ubuntu-22.04
permissions: permissions:
actions: read actions: read
contents: read contents: read
security-events: write security-events: write
strategy: strategy:
fail-fast: false fail-fast: false
matrix: matrix:
language: [ 'go' ] language:
- go
steps: steps:
- uses: actions/checkout@v2 - name: Code Checkout
uses: actions/checkout@v4
- name: Initialize CodeQL - name: Initialize CodeQL
uses: github/codeql-action/init@v1 uses: github/codeql-action/init@v2
with: with:
languages: ${{ matrix.language }} languages: ${{ matrix.language }}
- uses: github/codeql-action/autobuild@v1
- uses: github/codeql-action/analyze@v1 - name: Autobuild
uses: github/codeql-action/autobuild@v2
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v2

78
.github/workflows/docker.yaml vendored Normal file
View File

@ -0,0 +1,78 @@
name: Docker
on:
push:
branches:
- develop
release:
types:
- published
jobs:
build-and-push:
name: Build and Push
runs-on: ubuntu-22.04
# Always run against a tag, even if the commit into the tag has [docker skip] within the commit message.
if: "!contains(github.ref, 'develop') || (!contains(github.event.head_commit.message, 'skip docker') && !contains(github.event.head_commit.message, 'docker skip'))"
steps:
- name: Code checkout
uses: actions/checkout@v4
- name: Docker metadata
id: docker_meta
uses: docker/metadata-action@v5
with:
images: ghcr.io/pterodactyl/wings
flavor: |
latest=false
tags: |
type=raw,value=latest,enable=${{ github.event_name == 'release' && github.event.action == 'published' && github.event.release.prerelease == false }}
type=ref,event=tag
type=ref,event=branch
- name: Setup QEMU
uses: docker/setup-qemu-action@v3
- name: Setup Docker buildx
uses: docker/setup-buildx-action@v3
- name: Login to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.REGISTRY_TOKEN }}
- name: Get Build Information
id: build_info
run: |
echo "version_tag=${GITHUB_REF/refs\/tags\/v/}" >> $GITHUB_OUTPUT
echo "short_sha=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
- name: Build and Push (tag)
uses: docker/build-push-action@v5
if: "github.event_name == 'release' && github.event.action == 'published'"
with:
context: .
file: ./Dockerfile
push: true
platforms: linux/amd64,linux/arm64
build-args: |
VERSION=${{ steps.build_info.outputs.version_tag }}
labels: ${{ steps.docker_meta.outputs.labels }}
tags: ${{ steps.docker_meta.outputs.tags }}
- name: Build and Push (develop)
uses: docker/build-push-action@v5
if: "github.event_name == 'push' && contains(github.ref, 'develop')"
with:
context: .
file: ./Dockerfile
push: ${{ github.event_name != 'pull_request' }}
platforms: linux/amd64,linux/arm64
build-args: |
VERSION=dev-${{ steps.build_info.outputs.short_sha }}
labels: ${{ steps.docker_meta.outputs.labels }}
tags: ${{ steps.docker_meta.outputs.tags }}
cache-from: type=gha
cache-to: type=gha,mode=max

View File

@ -1,58 +0,0 @@
name: Publish Docker Image
on:
push:
branches:
- develop
tags:
- 'v*'
jobs:
push:
name: Push
runs-on: ubuntu-20.04
# Always run against a tag, even if the commit into the tag has [docker skip] within the commit message.
if: "!contains(github.ref, 'develop') || (!contains(github.event.head_commit.message, 'skip docker') && !contains(github.event.head_commit.message, 'docker skip'))"
steps:
- name: Code Checkout
uses: actions/checkout@v2
- name: Docker Meta
id: docker_meta
uses: crazy-max/ghaction-docker-meta@v1
with:
images: ghcr.io/pterodactyl/wings
- name: Set up QEMU
uses: docker/setup-qemu-action@v1
- name: Install buildx
uses: docker/setup-buildx-action@v1
with:
version: v0.5.1
- name: Login to GitHub Container Registry
uses: docker/login-action@v1
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.REGISTRY_TOKEN }}
- name: Get Build Information
id: build_info
run: |
echo "::set-output name=version_tag::${GITHUB_REF/refs\/tags\/v/}"
echo "::set-output name=short_sha::$(git rev-parse --short HEAD)"
- name: Release Production Build
uses: docker/build-push-action@v2
if: "!contains(github.ref, 'develop')"
with:
build-args: |
VERSION=${{ steps.build_info.outputs.version_tag }}
labels: ${{ steps.docker_meta.outputs.labels }}
platforms: linux/amd64,linux/arm64
push: true
tags: ${{ steps.docker_meta.outputs.tags }}
- name: Release Development Build
uses: docker/build-push-action@v2
if: "contains(github.ref, 'develop')"
with:
build-args: |
VERSION=dev-${{ steps.build_info.outputs.short_sha }}
labels: ${{ steps.docker_meta.outputs.labels }}
platforms: linux/amd64,linux/arm64
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.docker_meta.outputs.tags }}

75
.github/workflows/push.yaml vendored Normal file
View File

@ -0,0 +1,75 @@
name: Push
on:
push:
branches:
- develop
pull_request:
branches:
- develop
jobs:
build-and-test:
name: Build and Test
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-22.04]
go: ["1.22.5"]
goos: [linux]
goarch: [amd64, arm64]
steps:
- name: Setup Go
uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go }}
- name: Code checkout
uses: actions/checkout@v4
- name: go mod download
env:
CGO_ENABLED: 0
run: |
go mod download
- name: Build
env:
GOOS: ${{ matrix.goos }}
GOARCH: ${{ matrix.goarch }}
CGO_ENABLED: 0
SRC_PATH: github.com/pterodactyl/wings
run: |
go build -v -trimpath -ldflags="-s -w -X ${SRC_PATH}/system.Version=dev-${GIT_COMMIT:0:7}" -o dist/wings ${SRC_PATH}
go build -v -trimpath -ldflags="-X ${SRC_PATH}/system.Version=dev-${GIT_COMMIT:0:7}" -o dist/wings_debug ${SRC_PATH}
chmod 755 dist/*
- name: go test
if: ${{ matrix.goarch == 'amd64' }}
env:
CGO_ENABLED: 0
run: |
go test $(go list ./...)
- name: go test -race
if: ${{ matrix.goarch == 'amd64' }}
env:
CGO_ENABLED: 1
run: |
go test -race $(go list ./...)
- name: Upload Release Artifact
uses: actions/upload-artifact@v4
if: ${{ (github.ref == 'refs/heads/develop' || github.event_name == 'pull_request') && matrix.go == '1.22.5' }}
with:
name: wings_linux_${{ matrix.goarch }}
path: dist/wings
- name: Upload Debug Artifact
uses: actions/upload-artifact@v4
if: ${{ (github.ref == 'refs/heads/develop' || github.event_name == 'pull_request') && matrix.go == '1.22.5' }}
with:
name: wings_linux_${{ matrix.goarch }}_debug
path: dist/wings_debug

View File

@ -1,41 +1,47 @@
name: Create Release name: Release
on: on:
push: push:
tags: tags:
- 'v*' - "v*"
jobs: jobs:
release: release:
runs-on: ubuntu-20.04 name: Release
runs-on: ubuntu-22.04
steps: steps:
- name: Code Checkout - name: Code Checkout
uses: actions/checkout@v2 uses: actions/checkout@v4
- uses: actions/setup-go@v2
- name: Setup Go
uses: actions/setup-go@v5
with: with:
go-version: '^1.17' go-version: "1.22.5"
- name: Build
- name: Build release binaries
env: env:
CGO_ENABLED: 0
REF: ${{ github.ref }} REF: ${{ github.ref }}
run: | run: |
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags="-s -w -X github.com/pterodactyl/wings/system.Version=${REF:11}" -o build/wings_linux_amd64 -v wings.go GOARCH=amd64 go build -o dist/wings_linux_amd64 -v -trimpath -ldflags="-s -w -X github.com/pterodactyl/wings/system.Version=${REF:11}" github.com/pterodactyl/wings
CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -ldflags="-s -w -X github.com/pterodactyl/wings/system.Version=${REF:11}" -o build/wings_linux_arm64 -v wings.go chmod 755 dist/wings_linux_amd64
- name: Test GOARCH=arm64 go build -o dist/wings_linux_arm64 -v -trimpath -ldflags="-s -w -X github.com/pterodactyl/wings/system.Version=${REF:11}" github.com/pterodactyl/wings
run: go test ./... chmod 755 dist/wings_linux_arm64
- name: Compress binary and make it executable
run: |
upx build/wings_linux_amd64 && chmod +x build/wings_linux_amd64
upx build/wings_linux_arm64 && chmod +x build/wings_linux_arm64
- name: Extract changelog - name: Extract changelog
env: env:
REF: ${{ github.ref }} REF: ${{ github.ref }}
run: | run: |
sed -n "/^## ${REF:10}/,/^## /{/^## /b;p}" CHANGELOG.md > ./RELEASE_CHANGELOG sed -n "/^## ${REF:10}/,/^## /{/^## /b;p}" CHANGELOG.md > ./RELEASE_CHANGELOG
echo ::set-output name=version_name::`sed -nr "s/^## (${REF:10} .*)$/\1/p" CHANGELOG.md`
- name: Create checksum and add to changelog - name: Create checksum and add to changelog
run: | run: |
SUM=`cd build && sha256sum wings_linux_amd64` SUM=`cd dist && sha256sum wings_linux_amd64`
SUM2=`cd build && sha256sum wings_linux_arm64` SUM2=`cd dist && sha256sum wings_linux_arm64`
echo -e "\n#### SHA256 Checksum\n\`\`\`\n$SUM\n$SUM2\n\`\`\`\n" >> ./RELEASE_CHANGELOG echo -e "\n#### SHA256 Checksum\n\`\`\`\n$SUM\n$SUM2\n\`\`\`\n" >> ./RELEASE_CHANGELOG
echo -e "$SUM\n$SUM2" > checksums.txt echo -e "$SUM\n$SUM2" > checksums.txt
- name: Create release branch - name: Create release branch
env: env:
REF: ${{ github.ref }} REF: ${{ github.ref }}
@ -49,35 +55,37 @@ jobs:
git add system/const.go git add system/const.go
git commit -m "bump version for release" git commit -m "bump version for release"
git push git push
- name: Create Release
- name: Create release
id: create_release id: create_release
uses: actions/create-release@v1 uses: softprops/action-gh-release@v1
env: env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with: with:
tag_name: ${{ github.ref }}
release_name: ${{ steps.extract_changelog.outputs.version_name }}
body_path: ./RELEASE_CHANGELOG
draft: true draft: true
prerelease: ${{ contains(github.ref, 'beta') || contains(github.ref, 'alpha') }} prerelease: ${{ contains(github.ref, 'rc') || contains(github.ref, 'beta') || contains(github.ref, 'alpha') }}
- name: Upload amd64 Binary body_path: ./RELEASE_CHANGELOG
- name: Upload amd64 binary
uses: actions/upload-release-asset@v1 uses: actions/upload-release-asset@v1
env: env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with: with:
upload_url: ${{ steps.create_release.outputs.upload_url }} upload_url: ${{ steps.create_release.outputs.upload_url }}
asset_path: build/wings_linux_amd64 asset_path: dist/wings_linux_amd64
asset_name: wings_linux_amd64 asset_name: wings_linux_amd64
asset_content_type: application/octet-stream asset_content_type: application/octet-stream
- name: Upload arm64 Binary
- name: Upload arm64 binary
uses: actions/upload-release-asset@v1 uses: actions/upload-release-asset@v1
env: env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with: with:
upload_url: ${{ steps.create_release.outputs.upload_url }} upload_url: ${{ steps.create_release.outputs.upload_url }}
asset_path: build/wings_linux_arm64 asset_path: dist/wings_linux_arm64
asset_name: wings_linux_arm64 asset_name: wings_linux_arm64
asset_content_type: application/octet-stream asset_content_type: application/octet-stream
- name: Upload checksum - name: Upload checksum
uses: actions/upload-release-asset@v1 uses: actions/upload-release-asset@v1
env: env:

3
.gitignore vendored
View File

@ -23,6 +23,8 @@
# ignore configuration file # ignore configuration file
/config.yml /config.yml
/config*.yml /config*.yml
/config.yaml
/config*.yaml
# Ignore Vagrant stuff # Ignore Vagrant stuff
/.vagrant /.vagrant
@ -49,3 +51,4 @@ debug
.DS_Store .DS_Store
*.pprof *.pprof
*.pdf *.pdf
pprof.*

View File

@ -1,5 +1,221 @@
# Changelog # Changelog
## v1.11.14
### Added
* Support relative file paths for the Wings config ([#180](https://github.com/pterodactyl/wings/pull/180))
### Fixed
* Folders not being sorted before files properly ([#5078](https://github.com/pterodactyl/panel/issues/5078)
## v1.11.13
### Fixed
* Auto-configure not working ([#5087](https://github.com/pterodactyl/panel/issues/5087))
* Individual files unable to be decompressed ([#5034](https://github.com/pterodactyl/panel/issues/5034))
## v1.11.12
### Fixed
* Arbitrary File Write/Read ([GHSA-gqmf-jqgv-v8fw](https://github.com/pterodactyl/wings/security/advisories/GHSA-gqmf-jqgv-v8fw))
* Server-side Request Forgery (SSRF) during remote file pull ([GHSA-qq22-jj8x-4wwv](https://github.com/pterodactyl/wings/security/advisories/GHSA-qq22-jj8x-4wwv))
* Invalid `Content-Type` being used with the `wings diagnostics` command ([#186](https://github.com/pterodactyl/wings/pull/186))
## v1.11.11
### Fixed
* Backups missing content when a `.pteroignore` file is used
* Archives originating from a subdirectory not containing any files ([#5030](https://github.com/pterodactyl/panel/issues/5030))
## v1.11.10
### Fixed
* Archives randomly ignoring files and directories ([#5027](https://github.com/pterodactyl/panel/issues/5027))
* Crash when deleting or transferring a server ([#5028](https://github.com/pterodactyl/panel/issues/5028))
## v1.11.9
### Changed
* Release binaries are now built with Go 1.21.8
* Updated Go dependencies
### Fixed
* [CVE-2024-27102](https://www.cve.org/CVERecord?id=CVE-2024-27102)
## v1.11.8
### Changed
* Release binaries are now built with Go 1.20.10 (resolves [CVE-2023-44487](https://www.cve.org/CVERecord?id=CVE-2023-44487))
* Updated Go dependencies
## v1.11.7
### Changed
* Updated Go dependencies (this resolves an issue related to `http: invalid Host header` with Docker)
* Wings is now built with go1.19.11
## v1.11.6
### Fixed
* CVE-2023-32080
## v1.11.5
### Added
* Added a config option to disable Wings config.yml updates from the Panel (https://github.com/pterodactyl/wings/commit/ec6d6d83ea3eb14995c24f001233e85b37ffb87b)
### Changed
* Wings is now built with Go 1.19.7
### Fixed
* Fixed archives containing partially matched file names (https://github.com/pterodactyl/wings/commit/43b3496f0001cec231c80af1f9a9b3417d04e8d4)
## v1.11.4
### Fixed
* CVE-2023-25168
## v1.11.3
### Fixed
* CVE-2023-25152
## v1.11.2
### Fixed
* Backups being restored from remote storage (s3) erroring out due to a closed stream.
* Fix IP validation logic for activity logs filtering out valid IPs instead of invalid IPs
## v1.11.1
### Changed
* Release binaries are now built with Go 1.18.10
* Timeout when stopping a server before a transfer begins has been reduced to 15 seconds from 1 minute
* Removed insecure SSH protocols for use with the SFTP server
### Fixed
* Unnecessary Docker client connections being left open, causing a slow leak of file descriptors
* Files being left open in parts of the server's filesystem, causing a leak of file descriptors
* IPv6 addresses being corrupted by flawed port stripping logic for activity logs, old entries with malformed IPs will be deleted from the local SQLite database automatically
* A server that times out while being stopped at the beginning of a transfer no longer causes the server to become stuck in a transferring state
## v1.11.0
### Added (since 1.7.2)
* More detailed information returned by the `/api/system` endpoint when using the `?v=2` query parameter.
### Changed (since 1.7.2)
* Send re-installation status separately from installation status.
* Wings release versions will now follow the major and minor version of the Panel.
* Transfers no longer buffer to disk, instead they are fully streamed with only a small amount of memory used for buffering.
* Release binaries are no longer compressed with UPX.
* Use `POST` instead of `GET` for sending the status of a transfer to the Panel.
### Fixed (since 1.7.2)
* Fixed servers outgoing IP not being updated whenever a server's primary allocation is changed when using the Force Outgoing IP option.
* Fixed servers being terminated rather than gracefully stopped when a signal is used to stop the container rather than a command.
* Fixed file not found errors being treated as an internal error, they are now treated as a 404.
* Wings can be run with Podman instead of Docker, this is still experimental and not recommended for production use.
* Archive progress is now reported correctly.
* Labels for containers can now be set by the Panel.
* Fixed servers becoming deadlocked when the target node of a transfer goes offline.
## v1.11.0-rc.2
### Added
* More detailed information returned by the `/api/system` endpoint when using the `?v=2` query parameter.
### Changed
* Send reinstallation status separately from installation status.
### Fixed
* Fixed servers outgoing IP not being updated whenever a server's primary allocation is changed when using the Force Outgoing IP option.
* Fixed servers being terminated rather than gracefully stopped when a signal is used to stop the container rather than a command.
* Fixed file not found errors being treated as an internal error, they are now treated as a 404.
## v1.11.0-rc.1
### Changed
* Wings release versions will now follow the major and minor version of the panel.
* Transfers no longer buffer to disk, instead they are fully streamed with only a small amount of memory used for buffering.
* Release binaries are no longer compressed with UPX.
### Fixed
* Wings can be run with podman instead of Docker, this is still experimental and not recommended for production use.
* Archive progress is now reported correctly.
* Labels for containers can now be set by the Panel.
## v1.7.5
### Fixed
* CVE-2023-32080
## v1.7.4
### Fixed
* CVE-2023-25168
## v1.7.3
### Fixed
* CVE-2023-25152
## v1.7.2
### Fixed
* The S3 backup driver now supports Cloudflare R2
### Added
* During a server transfer, there is a new "Archiving" status that outputs the progress of creating the server transfer archive.
* Adds a configuration option to control the list of trusted proxies that can be used to determine the client IP address.
* Adds a configuration option to control the Docker username space setting when Wings creates containers.
### Changed
* Releases are now built using `Go 1.18` — the minimum version required to build Wings is now `Go 1.18`.
## v1.7.1
### Fixed
* YAML parser has been updated to fix some strange issues
### Added
* Added `Force Outgoing IP` option for servers to ensure outgoing traffic uses the server's IP address
* Adds an option to control the level of gzip compression for backups
## v1.7.0
### Fixed
* Fixes multi-platform support for Wings' Docker image.
### Added
* Adds support for tracking of SFTP actions, power actions, server commands, and file uploads by utilizing a local SQLite database and processing events before sending them to the Panel.
* Adds support for configuring the MTU on the `pterodactyl0` network.
## v1.6.4
### Fixed
* Fixes a bug causing CPU limiting to not be properly applied to servers.
* Fixes a bug causing zip archives to decompress without taking into account nested folder structures.
## v1.6.3
### Fixed
* Fixes SFTP authentication failing for administrative users due to a permissions adjustment on the Panel.
## v1.6.2
### Fixed
* Fixes file upload size not being properly enforced.
* Fixes a bug that prevented listing a directory when it contained a named pipe. Also added a check to prevent attempting to read a named pipe directly.
* Fixes a bug with the archiver logic that would include folders that had the same name prefix. (for example, requesting only `map` would also include `map2` and `map3`)
* Requests to the Panel that return a client error (4xx response code) no longer trigger an exponential backoff, they immediately stop the request.
### Changed
* CPU limit fields are only set on the Docker container if they have been specified for the server — otherwise they are left empty.
### Added
* Added the ability to define the location of the temporary folder used by Wings — defaults to `/tmp/pterodactyl`.
* Adds the ability to authenticate for SFTP using public keys (requires `Panel@1.8.0`).
## v1.6.1
### Fixed
* Fixes error that would sometimes occur when starting a server that would cause the temporary power action lock to never be released due to a blocked channel.
* Fixes a bug causing the CPU usage of Wings to get stuck at 100% when a server is deleted while the installation process is running.
### Changed
* Cleans up a lot of the logic for handling events between the server and environment process to make it easier to make modifications to down the road.
* Cleans up logic handling the `StopAndWait` logic for stopping a server gracefully before terminating the process if it does not respond.
## v1.6.0
### Fixed
* Internal logic for processing a server start event has been adjusted to attach to the Docker container before attempting to start the container. This should fix issues where a server would get stuck after pulling the container image.
* Fixes a bug in the console output that was dropping console lines when a large number of lines were sent at once.
### Changed
* Removed the console throttle logic that would terminate a server instance that was sending too much data. This logic has been replaced with simpler logic that only throttles the console, it does not try to terminate the server. In addition, this change has reduced the number of go-routines needed by the application and dramatically simplified internal logic.
* Removed the `--profiler` flag and replaced it with `--pprof` which will start an internal server listening on `localhost:6060` allowing you to use Go's standard `pprof` tooling.
* Replaced the `json` log driver for Docker containers with `local` to reduce the amount of overhead when it comes to streaming logs from instances.
## v1.5.6 ## v1.5.6
### Fixed ### Fixed
* Rewrote handler logic for the power actions lock to hopefully address issues people have been having when a server crashes and they're unable to start it again until restarting Wings. * Rewrote handler logic for the power actions lock to hopefully address issues people have been having when a server crashes and they're unable to start it again until restarting Wings.

View File

@ -1,19 +1,18 @@
# Stage 1 (Build) # Stage 1 (Build)
FROM --platform=$BUILDPLATFORM golang:1.17-alpine AS builder FROM golang:1.22.5-alpine AS builder
ARG VERSION ARG VERSION
RUN apk add --update --no-cache git make upx RUN apk add --update --no-cache git make
WORKDIR /app/ WORKDIR /app/
COPY go.mod go.sum /app/ COPY go.mod go.sum /app/
RUN go mod download RUN go mod download
COPY . /app/ COPY . /app/
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build \ RUN CGO_ENABLED=0 go build \
-ldflags="-s -w -X github.com/pterodactyl/wings/system.Version=$VERSION" \ -ldflags="-s -w -X github.com/pterodactyl/wings/system.Version=$VERSION" \
-v \ -v \
-trimpath \ -trimpath \
-o wings \ -o wings \
wings.go wings.go
RUN upx wings
RUN echo "ID=\"distroless\"" > /etc/os-release RUN echo "ID=\"distroless\"" > /etc/os-release
# Stage 2 (Final) # Stage 2 (Final)

View File

@ -5,8 +5,8 @@ build:
GOOS=linux GOARCH=arm64 go build -ldflags="-s -w" -gcflags "all=-trimpath=$(pwd)" -o build/wings_linux_arm64 -v wings.go GOOS=linux GOARCH=arm64 go build -ldflags="-s -w" -gcflags "all=-trimpath=$(pwd)" -o build/wings_linux_arm64 -v wings.go
debug: debug:
go build -ldflags="-X github.com/pterodactyl/wings/system.Version=$(GIT_HEAD)" -race go build -ldflags="-X github.com/pterodactyl/wings/system.Version=$(GIT_HEAD)"
sudo ./wings --debug --ignore-certificate-errors --config config.yml sudo ./wings --debug --ignore-certificate-errors --config config.yml --pprof --pprof-block-rate 1
# Runs a remotly debuggable session for Wings allowing an IDE to connect and target # Runs a remotly debuggable session for Wings allowing an IDE to connect and target
# different breakpoints. # different breakpoints.
@ -14,9 +14,6 @@ rmdebug:
go build -gcflags "all=-N -l" -ldflags="-X github.com/pterodactyl/wings/system.Version=$(GIT_HEAD)" -race go build -gcflags "all=-N -l" -ldflags="-X github.com/pterodactyl/wings/system.Version=$(GIT_HEAD)" -race
sudo dlv --listen=:2345 --headless=true --api-version=2 --accept-multiclient exec ./wings -- --debug --ignore-certificate-errors --config config.yml sudo dlv --listen=:2345 --headless=true --api-version=2 --accept-multiclient exec ./wings -- --debug --ignore-certificate-errors --config config.yml
compress:
upx --brute build/wings_*
cross-build: clean build compress cross-build: clean build compress
clean: clean:

View File

@ -5,6 +5,7 @@
[![Go Report Card](https://goreportcard.com/badge/github.com/pterodactyl/wings)](https://goreportcard.com/report/github.com/pterodactyl/wings) [![Go Report Card](https://goreportcard.com/badge/github.com/pterodactyl/wings)](https://goreportcard.com/report/github.com/pterodactyl/wings)
# Pterodactyl Wings # Pterodactyl Wings
Wings is Pterodactyl's server control plane, built for the rapidly changing gaming industry and designed to be Wings is Pterodactyl's server control plane, built for the rapidly changing gaming industry and designed to be
highly performant and secure. Wings provides an HTTP API allowing you to interface directly with running server highly performant and secure. Wings provides an HTTP API allowing you to interface directly with running server
instances, fetch server logs, generate backups, and control all aspects of the server lifecycle. instances, fetch server logs, generate backups, and control all aspects of the server lifecycle.
@ -13,36 +14,29 @@ In addition, Wings ships with a built-in SFTP server allowing your system to rem
dependencies, and allowing users to authenticate with the same credentials they would normally use to access the Panel. dependencies, and allowing users to authenticate with the same credentials they would normally use to access the Panel.
## Sponsors ## Sponsors
I would like to extend my sincere thanks to the following sponsors for helping find Pterodactyl's developement.
[Interested in becoming a sponsor?](https://github.com/sponsors/DaneEveritt)
| Company | About | I would like to extend my sincere thanks to the following sponsors for helping fund Pterodactyl's development.
| ------- | ----- | [Interested in becoming a sponsor?](https://github.com/sponsors/matthewpi)
| [**WISP**](https://wisp.gg) | Extra features. |
| [**MixmlHosting**](https://mixmlhosting.com) | MixmlHosting provides high quality Virtual Private Servers along with game servers, all at a affordable price. | | Company | About |
| [**BisectHosting**](https://www.bisecthosting.com/) | BisectHosting provides Minecraft, Valheim and other server hosting services with the highest reliability and lightning fast support since 2012. | |--------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [**Bloom.host**](https://bloom.host) | Bloom.host offers dedicated core VPS and Minecraft hosting with Ryzen 9 processors. With owned-hardware, we offer truly unbeatable prices on high-performance hosting. | | [**Aussie Server Hosts**](https://aussieserverhosts.com/) | No frills Australian Owned and operated High Performance Server hosting for some of the most demanding games serving Australia and New Zealand. |
| [**MineStrator**](https://minestrator.com/) | Looking for a French highend hosting company for you minecraft server? More than 14,000 members on our discord, trust us. | | [**CodeNode LLC**](https://codenode.gg/) | Looking for simplicity? Well, look no further! CodeNode has got you covered with everything you need at the rock-bottom price of $1.75 per GB, including dedicated IPs in Dallas, Texas, and Amsterdam, Netherlands. We're not just good, we're the best in the game! |
| [**DedicatedMC**](https://dedicatedmc.io/) | DedicatedMC provides Raw Power hosting at affordable pricing, making sure to never compromise on your performance and giving you the best performance money can buy. | | [**BisectHosting**](https://www.bisecthosting.com/) | BisectHosting provides Minecraft, Valheim and other server hosting services with the highest reliability and lightning fast support since 2012. |
| [**Skynode**](https://www.skynode.pro/) | Skynode provides blazing fast game servers along with a top-notch user experience. Whatever our clients are looking for, we're able to provide it! | | [**MineStrator**](https://minestrator.com/) | Looking for the most highend French hosting company for your minecraft server? More than 24,000 members on our discord trust us. Give us a try! |
| [**XCORE**](https://xcore-server.de/) | XCORE offers High-End Servers for hosting and gaming since 2012. Fast, excellent and well-known for eSports Gaming. | | [**HostEZ**](https://hostez.io) | US & EU Rust & Minecraft Hosting. DDoS Protected bare metal, VPS and colocation with low latency, high uptime and maximum availability. EZ! |
| [**RoyaleHosting**](https://royalehosting.net/) | Build your dreams and deploy them with RoyaleHostings reliable servers and network. Easy to use, provisioned in a couple of minutes. | | [**Blueprint**](https://blueprint.zip/?pterodactyl=true) | Create and install Pterodactyl addons and themes with the growing Blueprint framework - the package-manager for Pterodactyl. Use multiple modifications at once without worrying about conflicts and make use of the large extension ecosystem. |
| [**Spill Hosting**](https://spillhosting.no/) | Spill Hosting is a Norwegian hosting service, which aims for inexpensive services on quality servers. Premium i9-9900K processors will run your game like a dream. | | [**indifferent broccoli**](https://indifferentbroccoli.com/) | indifferent broccoli is a game server hosting and rental company. With us, you get top-notch computer power for your gaming sessions. We destroy lag, latency, and complexity--letting you focus on the fun stuff. |
| [**DeinServerHost**](https://deinserverhost.de/) | DeinServerHost offers Dedicated, vps and Gameservers for many popular Games like Minecraft and Rust in Germany since 2013. |
| [**HostBend**](https://hostbend.com/) | HostBend offers a variety of solutions for developers, students, and others who have a tight budget but don't want to compromise quality and support. |
| [**Capitol Hosting Solutions**](https://chs.gg/) | CHS is *the* budget friendly hosting company for Australian and American gamers, offering a variety of plans from Web Hosting to Game Servers; Custom Solutions too! |
| [**ByteAnia**](https://byteania.com/?utm_source=pterodactyl) | ByteAnia offers the best performing and most affordable **Ryzen 5000 Series hosting** on the market for *unbeatable prices*! |
| [**Aussie Server Hosts**](https://aussieserverhosts.com/) | No frills Australian Owned and operated High Performance Server hosting for some of the most demanding games serving Australia and New Zealand. |
| [**VibeGAMES**](https://vibegames.net/) | VibeGAMES is a game server provider that specializes in DDOS protection for the games we offer. We have multiple locations in the US, Brazil, France, Germany, Singapore, Australia and South Africa.|
| [**RocketNode**](https://rocketnode.net) | RocketNode is a VPS and Game Server provider that offers the best performing VPS and Game hosting Solutions at affordable prices! |
## Documentation ## Documentation
* [Panel Documentation](https://pterodactyl.io/panel/1.0/getting_started.html) * [Panel Documentation](https://pterodactyl.io/panel/1.0/getting_started.html)
* [Wings Documentation](https://pterodactyl.io/wings/1.0/installing.html) * [Wings Documentation](https://pterodactyl.io/wings/1.0/installing.html)
* [Community Guides](https://pterodactyl.io/community/about.html) * [Community Guides](https://pterodactyl.io/community/about.html)
* Or, get additional help [via Discord](https://discord.gg/pterodactyl) * Or, get additional help [via Discord](https://discord.gg/pterodactyl)
## Reporting Issues ## Reporting Issues
Please use the [pterodactyl/panel](https://github.com/pterodactyl/panel) repository to report any issues or make Please use the [pterodactyl/panel](https://github.com/pterodactyl/panel) repository to report any issues or make
feature requests for Wings. In addition, the [security policy](https://github.com/pterodactyl/panel/security/policy) listed feature requests for Wings. In addition, the [security policy](https://github.com/pterodactyl/panel/security/policy) listed
within that repository also applies to Wings. within that repository also applies to Wings.

View File

@ -155,6 +155,9 @@ func configureCmdRun(cmd *cobra.Command, args []string) {
panic(err) panic(err)
} }
// Manually specify the Panel URL as it won't be decoded from JSON.
cfg.PanelLocation = configureArgs.PanelURL
if err = config.WriteToDisk(cfg); err != nil { if err = config.WriteToDisk(cfg); err != nil {
panic(err) panic(err)
} }

View File

@ -58,14 +58,14 @@ func newDiagnosticsCommand() *cobra.Command {
return command return command
} }
// diagnosticsCmdRun collects diagnostics about wings, it's configuration and the node. // diagnosticsCmdRun collects diagnostics about wings, its configuration and the node.
// We collect: // We collect:
// - wings and docker versions // - wings and docker versions
// - relevant parts of daemon configuration // - relevant parts of daemon configuration
// - the docker debug output // - the docker debug output
// - running docker containers // - running docker containers
// - logs // - logs
func diagnosticsCmdRun(cmd *cobra.Command, args []string) { func diagnosticsCmdRun(*cobra.Command, []string) {
questions := []*survey.Question{ questions := []*survey.Question{
{ {
Name: "IncludeEndpoints", Name: "IncludeEndpoints",
@ -229,8 +229,8 @@ func uploadToHastebin(hbUrl, content string) (string, error) {
return "", err return "", err
} }
u.Path = path.Join(u.Path, "documents") u.Path = path.Join(u.Path, "documents")
res, err := http.Post(u.String(), "plain/text", r) res, err := http.Post(u.String(), "text/plain", r)
if err != nil || res.StatusCode != 200 { if err != nil || res.StatusCode < 200 || res.StatusCode >= 300 {
fmt.Println("Failed to upload report to ", u.String(), err) fmt.Println("Failed to upload report to ", u.String(), err)
return "", err return "", err
} }

View File

@ -7,11 +7,12 @@ import (
"fmt" "fmt"
log2 "log" log2 "log"
"net/http" "net/http"
_ "net/http/pprof"
"os" "os"
"path" "path"
"path/filepath" "path/filepath"
"runtime"
"strconv" "strconv"
"strings"
"time" "time"
"github.com/NYTimes/logrotate" "github.com/NYTimes/logrotate"
@ -20,13 +21,14 @@ import (
"github.com/docker/docker/client" "github.com/docker/docker/client"
"github.com/gammazero/workerpool" "github.com/gammazero/workerpool"
"github.com/mitchellh/colorstring" "github.com/mitchellh/colorstring"
"github.com/pkg/profile"
"github.com/spf13/cobra" "github.com/spf13/cobra"
"golang.org/x/crypto/acme" "golang.org/x/crypto/acme"
"golang.org/x/crypto/acme/autocert" "golang.org/x/crypto/acme/autocert"
"github.com/pterodactyl/wings/config" "github.com/pterodactyl/wings/config"
"github.com/pterodactyl/wings/environment" "github.com/pterodactyl/wings/environment"
"github.com/pterodactyl/wings/internal/cron"
"github.com/pterodactyl/wings/internal/database"
"github.com/pterodactyl/wings/loggers/cli" "github.com/pterodactyl/wings/loggers/cli"
"github.com/pterodactyl/wings/remote" "github.com/pterodactyl/wings/remote"
"github.com/pterodactyl/wings/router" "github.com/pterodactyl/wings/router"
@ -75,8 +77,10 @@ func init() {
rootCommand.PersistentFlags().BoolVar(&debug, "debug", false, "pass in order to run wings in debug mode") rootCommand.PersistentFlags().BoolVar(&debug, "debug", false, "pass in order to run wings in debug mode")
// Flags specifically used when running the API. // Flags specifically used when running the API.
rootCommand.Flags().String("profiler", "", "the profiler to run for this instance") rootCommand.Flags().Bool("pprof", false, "if the pprof profiler should be enabled. The profiler will bind to localhost:6060 by default")
rootCommand.Flags().Bool("auto-tls", false, "pass in order to have wings generate and manage it's own SSL certificates using Let's Encrypt") rootCommand.Flags().Int("pprof-block-rate", 0, "enables block profile support, may have performance impacts")
rootCommand.Flags().Int("pprof-port", 6060, "If provided with --pprof, the port it will run on")
rootCommand.Flags().Bool("auto-tls", false, "pass in order to have wings generate and manage its own SSL certificates using Let's Encrypt")
rootCommand.Flags().String("tls-hostname", "", "required with --auto-tls, the FQDN for the generated SSL certificate") rootCommand.Flags().String("tls-hostname", "", "required with --auto-tls, the FQDN for the generated SSL certificate")
rootCommand.Flags().Bool("ignore-certificate-errors", false, "ignore certificate verification errors when executing API calls") rootCommand.Flags().Bool("ignore-certificate-errors", false, "ignore certificate verification errors when executing API calls")
@ -86,25 +90,6 @@ func init() {
} }
func rootCmdRun(cmd *cobra.Command, _ []string) { func rootCmdRun(cmd *cobra.Command, _ []string) {
switch cmd.Flag("profiler").Value.String() {
case "cpu":
defer profile.Start(profile.CPUProfile).Stop()
case "mem":
defer profile.Start(profile.MemProfile).Stop()
case "alloc":
defer profile.Start(profile.MemProfile, profile.MemProfileAllocs).Stop()
case "heap":
defer profile.Start(profile.MemProfile, profile.MemProfileHeap).Stop()
case "routines":
defer profile.Start(profile.GoroutineProfile).Stop()
case "mutex":
defer profile.Start(profile.MutexProfile).Stop()
case "threads":
defer profile.Start(profile.ThreadcreationProfile).Stop()
case "block":
defer profile.Start(profile.BlockProfile).Stop()
}
printLogo() printLogo()
log.Debug("running in debug mode") log.Debug("running in debug mode")
log.WithField("config_file", configPath).Info("loading configuration from file") log.WithField("config_file", configPath).Info("loading configuration from file")
@ -124,10 +109,12 @@ func rootCmdRun(cmd *cobra.Command, _ []string) {
log.WithField("error", err).Fatal("failed to configure system directories for pterodactyl") log.WithField("error", err).Fatal("failed to configure system directories for pterodactyl")
return return
} }
log.WithField("username", config.Get().System.User).Info("checking for pterodactyl system user")
if err := config.EnsurePterodactylUser(); err != nil { if err := config.EnsurePterodactylUser(); err != nil {
log.WithField("error", err).Fatal("failed to create pterodactyl system user") log.WithField("error", err).Fatal("failed to create pterodactyl system user")
} }
if err := config.ConfigurePasswd(); err != nil {
log.WithField("error", err).Fatal("failed to configure container passwd file")
}
log.WithFields(log.Fields{ log.WithFields(log.Fields{
"username": config.Get().System.Username, "username": config.Get().System.Username,
"uid": config.Get().System.User.Uid, "uid": config.Get().System.User.Uid,
@ -146,6 +133,10 @@ func rootCmdRun(cmd *cobra.Command, _ []string) {
}), }),
) )
if err := database.Initialize(); err != nil {
log.WithField("error", err).Fatal("failed to initialize database")
}
manager, err := server.NewManager(cmd.Context(), pclient) manager, err := server.NewManager(cmd.Context(), pclient)
if err != nil { if err != nil {
log.WithField("error", err).Fatal("failed to load server configurations") log.WithField("error", err).Fatal("failed to load server configurations")
@ -172,7 +163,7 @@ func rootCmdRun(cmd *cobra.Command, _ []string) {
ticker := time.NewTicker(time.Minute) ticker := time.NewTicker(time.Minute)
// Every minute, write the current server states to the disk to allow for a more // Every minute, write the current server states to the disk to allow for a more
// seamless hard-reboot process in which wings will re-sync server states based // seamless hard-reboot process in which wings will re-sync server states based
// on it's last tracked state. // on its last tracked state.
go func() { go func() {
for { for {
select { select {
@ -275,6 +266,13 @@ func rootCmdRun(cmd *cobra.Command, _ []string) {
} }
}() }()
if s, err := cron.Scheduler(cmd.Context(), manager); err != nil {
log.WithField("error", err).Fatal("failed to initialize cron system")
} else {
log.WithField("subsystem", "cron").Info("starting cron processes")
s.StartAsync()
}
go func() { go func() {
// Run the SFTP server. // Run the SFTP server.
if err := sftp.New(manager).Run(); err != nil { if err := sftp.New(manager).Run(); err != nil {
@ -325,6 +323,20 @@ func rootCmdRun(cmd *cobra.Command, _ []string) {
TLSConfig: config.DefaultTLSConfig, TLSConfig: config.DefaultTLSConfig,
} }
profile, _ := cmd.Flags().GetBool("pprof")
if profile {
if r, _ := cmd.Flags().GetInt("pprof-block-rate"); r > 0 {
runtime.SetBlockProfileRate(r)
}
// Catch at least 1% of mutex contention issues.
runtime.SetMutexProfileFraction(100)
profilePort, _ := cmd.Flags().GetInt("pprof-port")
go func() {
http.ListenAndServe(fmt.Sprintf("localhost:%d", profilePort), nil)
}()
}
// Check if the server should run with TLS but using autocert. // Check if the server should run with TLS but using autocert.
if autotls { if autotls {
m := autocert.Manager{ m := autocert.Manager{
@ -352,7 +364,7 @@ func rootCmdRun(cmd *cobra.Command, _ []string) {
return return
} }
// Check if main http server should run with TLS. Otherwise reset the TLS // Check if main http server should run with TLS. Otherwise, reset the TLS
// config on the server and then serve it over normal HTTP. // config on the server and then serve it over normal HTTP.
if api.Ssl.Enabled { if api.Ssl.Enabled {
if err := s.ListenAndServeTLS(api.Ssl.CertificateFile, api.Ssl.KeyFile); err != nil { if err := s.ListenAndServeTLS(api.Ssl.CertificateFile, api.Ssl.KeyFile); err != nil {
@ -369,13 +381,14 @@ func rootCmdRun(cmd *cobra.Command, _ []string) {
// Reads the configuration from the disk and then sets up the global singleton // Reads the configuration from the disk and then sets up the global singleton
// with all the configuration values. // with all the configuration values.
func initConfig() { func initConfig() {
if !strings.HasPrefix(configPath, "/") { if !filepath.IsAbs(configPath) {
d, err := os.Getwd() d, err := filepath.Abs(configPath)
if err != nil { if err != nil {
log2.Fatalf("cmd/root: could not determine directory: %s", err) log2.Fatalf("cmd/root: failed to get path to config file: %s", err)
} }
configPath = path.Clean(path.Join(d, configPath)) configPath = d
} }
err := config.FromFile(configPath) err := config.FromFile(configPath)
if err != nil { if err != nil {
if errors.Is(err, os.ErrNotExist) { if errors.Is(err, os.ErrNotExist) {
@ -430,18 +443,18 @@ in all copies or substantial portions of the Software.%s`), system.Version, time
} }
func exitWithConfigurationNotice() { func exitWithConfigurationNotice() {
fmt.Print(colorstring.Color(` fmt.Printf(colorstring.Color(`
[_red_][white][bold]Error: Configuration File Not Found[reset] [_red_][white][bold]Error: Configuration File Not Found[reset]
Wings was not able to locate your configuration file, and therefore is not Wings was not able to locate your configuration file, and therefore is not
able to complete its boot process. Please ensure you have copied your instance able to complete its boot process. Please ensure you have copied your instance
configuration file into the default location below. configuration file into the default location below.
Default Location: /etc/pterodactyl/config.yml Default Location: %s
[yellow]This is not a bug with this software. Please do not make a bug report [yellow]This is not a bug with this software. Please do not make a bug report
for this issue, it will be closed.[reset] for this issue, it will be closed.[reset]
`)) `), config.DefaultLocation)
os.Exit(1) os.Exit(1)
} }

View File

@ -12,14 +12,16 @@ import (
"regexp" "regexp"
"strings" "strings"
"sync" "sync"
"sync/atomic"
"text/template" "text/template"
"time" "time"
"emperror.dev/errors" "emperror.dev/errors"
"github.com/acobaugh/osrelease"
"github.com/apex/log" "github.com/apex/log"
"github.com/cobaugh/osrelease"
"github.com/creasty/defaults" "github.com/creasty/defaults"
"github.com/gbrlsnchs/jwt/v3" "github.com/gbrlsnchs/jwt/v3"
"golang.org/x/sys/unix"
"gopkg.in/yaml.v2" "gopkg.in/yaml.v2"
"github.com/pterodactyl/wings/system" "github.com/pterodactyl/wings/system"
@ -87,10 +89,13 @@ type ApiConfiguration struct {
// Determines if functionality for allowing remote download of files into server directories // Determines if functionality for allowing remote download of files into server directories
// is enabled on this instance. If set to "true" remote downloads will not be possible for // is enabled on this instance. If set to "true" remote downloads will not be possible for
// servers. // servers.
DisableRemoteDownload bool `json:"disable_remote_download" yaml:"disable_remote_download"` DisableRemoteDownload bool `json:"-" yaml:"disable_remote_download"`
// The maximum size for files uploaded through the Panel in bytes. // The maximum size for files uploaded through the Panel in MB.
UploadLimit int `default:"100" json:"upload_limit" yaml:"upload_limit"` UploadLimit int64 `default:"100" json:"upload_limit" yaml:"upload_limit"`
// A list of IP address of proxies that may send a X-Forwarded-For header to set the true clients IP
TrustedProxies []string `json:"trusted_proxies" yaml:"trusted_proxies"`
} }
// RemoteQueryConfiguration defines the configuration settings for remote requests // RemoteQueryConfiguration defines the configuration settings for remote requests
@ -118,19 +123,23 @@ type RemoteQueryConfiguration struct {
// SystemConfiguration defines basic system configuration settings. // SystemConfiguration defines basic system configuration settings.
type SystemConfiguration struct { type SystemConfiguration struct {
// The root directory where all of the pterodactyl data is stored at. // The root directory where all of the pterodactyl data is stored at.
RootDirectory string `default:"/var/lib/pterodactyl" yaml:"root_directory"` RootDirectory string `default:"/var/lib/pterodactyl" json:"-" yaml:"root_directory"`
// Directory where logs for server installations and other wings events are logged. // Directory where logs for server installations and other wings events are logged.
LogDirectory string `default:"/var/log/pterodactyl" yaml:"log_directory"` LogDirectory string `default:"/var/log/pterodactyl" json:"-" yaml:"log_directory"`
// Directory where the server data is stored at. // Directory where the server data is stored at.
Data string `default:"/var/lib/pterodactyl/volumes" yaml:"data"` Data string `default:"/var/lib/pterodactyl/volumes" json:"-" yaml:"data"`
// Directory where server archives for transferring will be stored. // Directory where server archives for transferring will be stored.
ArchiveDirectory string `default:"/var/lib/pterodactyl/archives" yaml:"archive_directory"` ArchiveDirectory string `default:"/var/lib/pterodactyl/archives" json:"-" yaml:"archive_directory"`
// Directory where local backups will be stored on the machine. // Directory where local backups will be stored on the machine.
BackupDirectory string `default:"/var/lib/pterodactyl/backups" yaml:"backup_directory"` BackupDirectory string `default:"/var/lib/pterodactyl/backups" json:"-" yaml:"backup_directory"`
// TmpDirectory specifies where temporary files for Pterodactyl installation processes
// should be created. This supports environments running docker-in-docker.
TmpDirectory string `default:"/tmp/pterodactyl" json:"-" yaml:"tmp_directory"`
// The user that should own all of the server files, and be used for containers. // The user that should own all of the server files, and be used for containers.
Username string `default:"pterodactyl" yaml:"username"` Username string `default:"pterodactyl" yaml:"username"`
@ -145,9 +154,42 @@ type SystemConfiguration struct {
// Definitions for the user that gets created to ensure that we can quickly access // Definitions for the user that gets created to ensure that we can quickly access
// this information without constantly having to do a system lookup. // this information without constantly having to do a system lookup.
User struct { User struct {
Uid int // Rootless controls settings related to rootless container daemons.
Gid int Rootless struct {
} // Enabled controls whether rootless containers are enabled.
Enabled bool `yaml:"enabled" default:"false"`
// ContainerUID controls the UID of the user inside the container.
// This should likely be set to 0 so the container runs as the user
// running Wings.
ContainerUID int `yaml:"container_uid" default:"0"`
// ContainerGID controls the GID of the user inside the container.
// This should likely be set to 0 so the container runs as the user
// running Wings.
ContainerGID int `yaml:"container_gid" default:"0"`
} `yaml:"rootless"`
Uid int `yaml:"uid"`
Gid int `yaml:"gid"`
} `yaml:"user"`
// Passwd controls the mounting of a generated passwd files into containers started by Wings.
Passwd struct {
// Enable controls whether generated passwd files should be mounted into containers.
//
// By default this option is disabled and Wings will not mount any additional passwd
// files into containers.
Enable bool `yaml:"enabled" default:"false"`
// Directory is the directory on disk where the generated files will be stored.
// This directory may be temporary as it will be re-created whenever Wings is started.
//
// This path **WILL** be both written to by Wings and mounted into containers created by
// Wings. If you are running Wings itself in a container, this path will need to be mounted
// into the Wings container as the exact path on the host, which should match the value
// specified here. If you are using SELinux, you will need to make sure this file has the
// correct SELinux context in order for containers to use it.
Directory string `yaml:"directory" default:"/run/wings/etc"`
} `yaml:"passwd"`
// The amount of time in seconds that can elapse before a server's disk space calculation is // The amount of time in seconds that can elapse before a server's disk space calculation is
// considered stale and a re-check should occur. DANGER: setting this value too low can seriously // considered stale and a re-check should occur. DANGER: setting this value too low can seriously
@ -159,6 +201,15 @@ type SystemConfiguration struct {
// disk usage is not a concern. // disk usage is not a concern.
DiskCheckInterval int64 `default:"150" yaml:"disk_check_interval"` DiskCheckInterval int64 `default:"150" yaml:"disk_check_interval"`
// ActivitySendInterval is the amount of time that should ellapse between aggregated server activity
// being sent to the Panel. By default this will send activity collected over the last minute. Keep
// in mind that only a fixed number of activity log entries, defined by ActivitySendCount, will be sent
// in each run.
ActivitySendInterval int `default:"60" yaml:"activity_send_interval"`
// ActivitySendCount is the number of activity events to send per batch.
ActivitySendCount int `default:"100" yaml:"activity_send_count"`
// If set to true, file permissions for a server will be checked when the process is // If set to true, file permissions for a server will be checked when the process is
// booted. This can cause boot delays if the server has a large amount of files. In most // booted. This can cause boot delays if the server has a large amount of files. In most
// cases disabling this should not have any major impact unless external processes are // cases disabling this should not have any major impact unless external processes are
@ -179,6 +230,8 @@ type SystemConfiguration struct {
Backups Backups `yaml:"backups"` Backups Backups `yaml:"backups"`
Transfers Transfers `yaml:"transfers"` Transfers Transfers `yaml:"transfers"`
OpenatMode string `default:"auto" yaml:"openat_mode"`
} }
type CrashDetection struct { type CrashDetection struct {
@ -206,6 +259,15 @@ type Backups struct {
// //
// Defaults to 0 (unlimited) // Defaults to 0 (unlimited)
WriteLimit int `default:"0" yaml:"write_limit"` WriteLimit int `default:"0" yaml:"write_limit"`
// CompressionLevel determines how much backups created by wings should be compressed.
//
// "none" -> no compression will be applied
// "best_speed" -> uses gzip level 1 for fast speed
// "best_compression" -> uses gzip level 9 for minimal disk space useage
//
// Defaults to "best_speed" (level 1)
CompressionLevel string `default:"best_speed" yaml:"compression_level"`
} }
type Transfers struct { type Transfers struct {
@ -222,26 +284,14 @@ type ConsoleThrottles struct {
// Whether or not the throttler is enabled for this instance. // Whether or not the throttler is enabled for this instance.
Enabled bool `json:"enabled" yaml:"enabled" default:"true"` Enabled bool `json:"enabled" yaml:"enabled" default:"true"`
// The total number of lines that can be output in a given LineResetInterval period before // The total number of lines that can be output in a given Period period before
// a warning is triggered and counted against the server. // a warning is triggered and counted against the server.
Lines uint64 `json:"lines" yaml:"lines" default:"2000"` Lines uint64 `json:"lines" yaml:"lines" default:"2000"`
// The total number of throttle activations that can accumulate before a server is considered
// to be breaching and will be stopped. This value is decremented by one every DecayInterval.
MaximumTriggerCount uint64 `json:"maximum_trigger_count" yaml:"maximum_trigger_count" default:"5"`
// The amount of time after which the number of lines processed is reset to 0. This runs in // The amount of time after which the number of lines processed is reset to 0. This runs in
// a constant loop and is not affected by the current console output volumes. By default, this // a constant loop and is not affected by the current console output volumes. By default, this
// will reset the processed line count back to 0 every 100ms. // will reset the processed line count back to 0 every 100ms.
LineResetInterval uint64 `json:"line_reset_interval" yaml:"line_reset_interval" default:"100"` Period uint64 `json:"line_reset_interval" yaml:"line_reset_interval" default:"100"`
// The amount of time in milliseconds that must pass without an output warning being triggered
// before a throttle activation is decremented.
DecayInterval uint64 `json:"decay_interval" yaml:"decay_interval" default:"10000"`
// The amount of time that a server is allowed to be stopping for before it is terminated
// forcefully if it triggers output throttles.
StopGracePeriod uint `json:"stop_grace_period" yaml:"stop_grace_period" default:"15"`
} }
type Configuration struct { type Configuration struct {
@ -275,7 +325,7 @@ type Configuration struct {
// The location where the panel is running that this daemon should connect to // The location where the panel is running that this daemon should connect to
// to collect data and send events. // to collect data and send events.
PanelLocation string `json:"remote" yaml:"remote"` PanelLocation string `json:"-" yaml:"remote"`
RemoteQuery RemoteQueryConfiguration `json:"remote_query" yaml:"remote_query"` RemoteQuery RemoteQueryConfiguration `json:"remote_query" yaml:"remote_query"`
// AllowedMounts is a list of allowed host-system mount points. // AllowedMounts is a list of allowed host-system mount points.
@ -292,6 +342,9 @@ type Configuration struct {
// is only required by users running Wings without SSL certificates and using internal IP // is only required by users running Wings without SSL certificates and using internal IP
// addresses in order to connect. Most users should NOT enable this setting. // addresses in order to connect. Most users should NOT enable this setting.
AllowCORSPrivateNetwork bool `json:"allow_cors_private_network" yaml:"allow_cors_private_network"` AllowCORSPrivateNetwork bool `json:"allow_cors_private_network" yaml:"allow_cors_private_network"`
// IgnorePanelConfigUpdates causes confiuration updates that are sent by the panel to be ignored.
IgnorePanelConfigUpdates bool `json:"ignore_panel_config_updates" yaml:"ignore_panel_config_updates"`
} }
// NewAtPath creates a new struct and set the path where it should be stored. // NewAtPath creates a new struct and set the path where it should be stored.
@ -412,6 +465,19 @@ func EnsurePterodactylUser() error {
return nil return nil
} }
if _config.System.User.Rootless.Enabled {
log.Info("rootless mode is enabled, skipping user creation...")
u, err := user.Current()
if err != nil {
return err
}
_config.System.Username = u.Username
_config.System.User.Uid = system.MustInt(u.Uid)
_config.System.User.Gid = system.MustInt(u.Gid)
return nil
}
log.WithField("username", _config.System.Username).Info("checking for pterodactyl system user")
u, err := user.Lookup(_config.System.Username) u, err := user.Lookup(_config.System.Username)
// If an error is returned but it isn't the unknown user error just abort // If an error is returned but it isn't the unknown user error just abort
// the process entirely. If we did find a user, return it immediately. // the process entirely. If we did find a user, return it immediately.
@ -450,6 +516,37 @@ func EnsurePterodactylUser() error {
return nil return nil
} }
// ConfigurePasswd generates required passwd files for use with containers started by Wings.
func ConfigurePasswd() error {
passwd := _config.System.Passwd
if !passwd.Enable {
return nil
}
v := []byte(fmt.Sprintf(
`root:x:0:
container:x:%d:
nogroup:x:65534:`,
_config.System.User.Gid,
))
if err := os.WriteFile(filepath.Join(passwd.Directory, "group"), v, 0o644); err != nil {
return fmt.Errorf("failed to write file to %s/group: %v", passwd.Directory, err)
}
v = []byte(fmt.Sprintf(
`root:x:0:0::/root:/bin/sh
container:x:%d:%d::/home/container:/bin/sh
nobody:x:65534:65534::/var/empty:/bin/sh
`,
_config.System.User.Uid,
_config.System.User.Gid,
))
if err := os.WriteFile(filepath.Join(passwd.Directory, "passwd"), v, 0o644); err != nil {
return fmt.Errorf("failed to write file to %s/passwd: %v", passwd.Directory, err)
}
return nil
}
// FromFile reads the configuration from the provided file and stores it in the // FromFile reads the configuration from the provided file and stores it in the
// global singleton for this instance. // global singleton for this instance.
func FromFile(path string) error { func FromFile(path string) error {
@ -514,6 +611,13 @@ func ConfigureDirectories() error {
return err return err
} }
if _config.System.Passwd.Enable {
log.WithField("path", _config.System.Passwd.Directory).Debug("ensuring passwd directory exists")
if err := os.MkdirAll(_config.System.Passwd.Directory, 0o755); err != nil {
return err
}
}
return nil return nil
} }
@ -628,3 +732,36 @@ func getSystemName() (string, error) {
} }
return release["ID"], nil return release["ID"], nil
} }
var (
openat2 atomic.Bool
openat2Set atomic.Bool
)
func UseOpenat2() bool {
if openat2Set.Load() {
return openat2.Load()
}
defer openat2Set.Store(true)
c := Get()
openatMode := c.System.OpenatMode
switch openatMode {
case "openat2":
openat2.Store(true)
return true
case "openat":
openat2.Store(false)
return false
default:
fd, err := unix.Openat2(unix.AT_FDCWD, "/", &unix.OpenHow{})
if err != nil {
log.WithError(err).Warn("error occurred while checking for openat2 support, falling back to openat")
openat2.Store(false)
return false
}
_ = unix.Close(fd)
openat2.Store(true)
return true
}
}

View File

@ -4,7 +4,8 @@ import (
"encoding/base64" "encoding/base64"
"sort" "sort"
"github.com/docker/docker/api/types" "github.com/docker/docker/api/types/container"
"github.com/docker/docker/api/types/registry"
"github.com/goccy/go-json" "github.com/goccy/go-json"
) )
@ -36,6 +37,7 @@ type DockerNetworkConfiguration struct {
Mode string `default:"pterodactyl_nw" yaml:"network_mode"` Mode string `default:"pterodactyl_nw" yaml:"network_mode"`
IsInternal bool `default:"false" yaml:"is_internal"` IsInternal bool `default:"false" yaml:"is_internal"`
EnableICC bool `default:"true" yaml:"enable_icc"` EnableICC bool `default:"true" yaml:"enable_icc"`
NetworkMTU int64 `default:"1500" yaml:"network_mtu"`
Interfaces dockerNetworkInterfaces `yaml:"interfaces"` Interfaces dockerNetworkInterfaces `yaml:"interfaces"`
} }
@ -77,6 +79,30 @@ type DockerConfiguration struct {
Overhead Overhead `json:"overhead" yaml:"overhead"` Overhead Overhead `json:"overhead" yaml:"overhead"`
UsePerformantInspect bool `default:"true" json:"use_performant_inspect" yaml:"use_performant_inspect"` UsePerformantInspect bool `default:"true" json:"use_performant_inspect" yaml:"use_performant_inspect"`
// Sets the user namespace mode for the container when user namespace remapping option is
// enabled.
//
// If the value is blank, the daemon's user namespace remapping configuration is used,
// if the value is "host", then the pterodactyl containers are started with user namespace
// remapping disabled
UsernsMode string `default:"" json:"userns_mode" yaml:"userns_mode"`
LogConfig struct {
Type string `default:"local" json:"type" yaml:"type"`
Config map[string]string `default:"{\"max-size\":\"5m\",\"max-file\":\"1\",\"compress\":\"false\",\"mode\":\"non-blocking\"}" json:"config" yaml:"config"`
} `json:"log_config" yaml:"log_config"`
}
func (c DockerConfiguration) ContainerLogConfig() container.LogConfig {
if c.LogConfig.Type == "" {
return container.LogConfig{}
}
return container.LogConfig{
Type: c.LogConfig.Type,
Config: c.LogConfig.Config,
}
} }
// RegistryConfiguration defines the authentication credentials for a given // RegistryConfiguration defines the authentication credentials for a given
@ -89,7 +115,7 @@ type RegistryConfiguration struct {
// Base64 returns the authentication for a given registry as a base64 encoded // Base64 returns the authentication for a given registry as a base64 encoded
// string value. // string value.
func (c RegistryConfiguration) Base64() (string, error) { func (c RegistryConfiguration) Base64() (string, error) {
b, err := json.Marshal(types.AuthConfig{ b, err := json.Marshal(registry.AuthConfig{
Username: c.Username, Username: c.Username,
Password: c.Password, Password: c.Password,
}) })

View File

@ -12,6 +12,11 @@ import (
// Defines the allocations available for a given server. When using the Docker environment // Defines the allocations available for a given server. When using the Docker environment
// driver these correspond to mappings for the container that allow external connections. // driver these correspond to mappings for the container that allow external connections.
type Allocations struct { type Allocations struct {
// ForceOutgoingIP causes a dedicated bridge network to be created for the
// server with a special option, causing Docker to SNAT outgoing traffic to
// the DefaultMapping's IP. This is important to servers which rely on external
// services that check the IP of the server (Source Engine servers, for example).
ForceOutgoingIP bool `json:"force_outgoing_ip"`
// Defines the default allocation that should be used for this server. This is // Defines the default allocation that should be used for this server. This is
// what will be used for {SERVER_IP} and {SERVER_PORT} when modifying configuration // what will be used for {SERVER_IP} and {SERVER_PORT} when modifying configuration
// files or the startup arguments for a server. // files or the startup arguments for a server.

View File

@ -8,6 +8,7 @@ type Settings struct {
Mounts []Mount Mounts []Mount
Allocations Allocations Allocations Allocations
Limits Limits Limits Limits
Labels map[string]string
} }
// Defines the actual configuration struct for the environment with all of the settings // Defines the actual configuration struct for the environment with all of the settings
@ -68,6 +69,14 @@ func (c *Configuration) Mounts() []Mount {
return c.settings.Mounts return c.settings.Mounts
} }
// Labels returns the container labels associated with this instance.
func (c *Configuration) Labels() map[string]string {
c.mu.RLock()
defer c.mu.RUnlock()
return c.settings.Labels
}
// Returns the environment variables associated with this instance. // Returns the environment variables associated with this instance.
func (c *Configuration) EnvironmentVariables() []string { func (c *Configuration) EnvironmentVariables() []string {
c.mu.RLock() c.mu.RLock()

View File

@ -41,12 +41,12 @@ func ConfigureDocker(ctx context.Context) error {
nw := config.Get().Docker.Network nw := config.Get().Docker.Network
resource, err := cli.NetworkInspect(ctx, nw.Name, types.NetworkInspectOptions{}) resource, err := cli.NetworkInspect(ctx, nw.Name, types.NetworkInspectOptions{})
if err != nil { if err != nil {
if client.IsErrNotFound(err) { if !client.IsErrNotFound(err) {
log.Info("creating missing pterodactyl0 interface, this could take a few seconds...") return err
if err := createDockerNetwork(ctx, cli); err != nil { }
return err
} log.Info("creating missing pterodactyl0 interface, this could take a few seconds...")
} else { if err := createDockerNetwork(ctx, cli); err != nil {
return err return err
} }
} }
@ -92,7 +92,7 @@ func createDockerNetwork(ctx context.Context, cli *client.Client) error {
"com.docker.network.bridge.enable_ip_masquerade": "true", "com.docker.network.bridge.enable_ip_masquerade": "true",
"com.docker.network.bridge.host_binding_ipv4": "0.0.0.0", "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
"com.docker.network.bridge.name": "pterodactyl0", "com.docker.network.bridge.name": "pterodactyl0",
"com.docker.network.driver.mtu": "1500", "com.docker.network.driver.mtu": strconv.FormatInt(nw.NetworkMTU, 10),
}, },
}) })
if err != nil { if err != nil {

View File

@ -14,6 +14,7 @@ import (
"github.com/docker/docker/client" "github.com/docker/docker/client"
"github.com/docker/docker/errdefs" "github.com/docker/docker/errdefs"
"github.com/goccy/go-json" "github.com/goccy/go-json"
"github.com/pterodactyl/wings/config" "github.com/pterodactyl/wings/config"
) )
@ -73,6 +74,9 @@ func (e *Environment) ContainerInspect(ctx context.Context) (types.ContainerJSON
res, err := e.client.HTTPClient().Do(req) res, err := e.client.HTTPClient().Do(req)
if err != nil { if err != nil {
if res == nil {
return st, errdefs.Unknown(err)
}
return st, errdefs.FromStatusCode(err, res.StatusCode) return st, errdefs.FromStatusCode(err, res.StatusCode)
} }
@ -113,4 +117,4 @@ func parseErrorFromResponse(res *http.Response, body []byte) error {
} }
return errors.Wrap(errors.New(emsg), "Error response from daemon") return errors.Wrap(errors.New(emsg), "Error response from daemon")
} }

View File

@ -16,7 +16,6 @@ import (
"github.com/docker/docker/api/types/container" "github.com/docker/docker/api/types/container"
"github.com/docker/docker/api/types/mount" "github.com/docker/docker/api/types/mount"
"github.com/docker/docker/client" "github.com/docker/docker/client"
"github.com/docker/docker/daemon/logger/jsonfilelog"
"github.com/pterodactyl/wings/config" "github.com/pterodactyl/wings/config"
"github.com/pterodactyl/wings/environment" "github.com/pterodactyl/wings/environment"
@ -38,23 +37,17 @@ func (nw noopWriter) Write(b []byte) (int, error) {
} }
// Attach attaches to the docker container itself and ensures that we can pipe // Attach attaches to the docker container itself and ensures that we can pipe
// data in and out of the process stream. This should not be used for reading // data in and out of the process stream. This should always be called before
// console data as you *will* miss important output at the beginning because of // you have started the container, but after you've ensured it exists.
// the time delay with attaching to the output.
// //
// Calling this function will poll resources for the container in the background // Calling this function will poll resources for the container in the background
// until the provided context is canceled by the caller. Failure to cancel said // until the container is stopped. The context provided to this function is used
// context will cause background memory leaks as the goroutine will not exit. // for the purposes of attaching to the container, a second context is created
// within the function for managing polling.
func (e *Environment) Attach(ctx context.Context) error { func (e *Environment) Attach(ctx context.Context) error {
if e.IsAttached() { if e.IsAttached() {
return nil return nil
} }
e.log().Debug("not attached to container, continuing with attach...")
if err := e.followOutput(); err != nil {
return err
}
e.log().Debug("following container output")
opts := types.ContainerAttachOptions{ opts := types.ContainerAttachOptions{
Stdin: true, Stdin: true,
@ -64,13 +57,11 @@ func (e *Environment) Attach(ctx context.Context) error {
} }
// Set the stream again with the container. // Set the stream again with the container.
e.log().Debug("attempting to attach...")
if st, err := e.client.ContainerAttach(ctx, e.Id, opts); err != nil { if st, err := e.client.ContainerAttach(ctx, e.Id, opts); err != nil {
return err return errors.WrapIf(err, "environment/docker: error while attaching to container")
} else { } else {
e.SetStream(&st) e.SetStream(&st)
} }
e.log().Debug("attached!")
go func() { go func() {
// Don't use the context provided to the function, that'll cause the polling to // Don't use the context provided to the function, that'll cause the polling to
@ -94,20 +85,13 @@ func (e *Environment) Attach(ctx context.Context) error {
} }
}() }()
// Block the completion of this routine until the container is no longer running. This allows if err := system.ScanReader(e.stream.Reader, func(v []byte) {
// the pollResources function to run until it needs to be stopped. Because the container e.logCallbackMx.Lock()
// can be polled for resource usage, even when stopped, we need to have this logic present defer e.logCallbackMx.Unlock()
// in order to cancel the context and therefore stop the routine that is spawned. e.logCallback(v)
// }); err != nil && err != io.EOF {
// For now, DO NOT use client#ContainerWait from the Docker package. There is a nasty log.WithField("error", err).WithField("container_id", e.Id).Warn("error processing scanner line in console output")
// bug causing containers to hang on deletion and cause servers to lock up on the system. return
//
// This weird code isn't intuitive, but it keeps the function from ending until the container
// is stopped and therefore the stream reader ends up closed.
// @see https://github.com/moby/moby/issues/41827
c := new(noopWriter)
if _, err := io.Copy(c, e.stream.Reader); err != nil {
e.log().WithField("error", err).Error("could not copy from environment stream to noop writer")
} }
}() }()
@ -151,13 +135,15 @@ func (e *Environment) InSituUpdate() error {
// currently available for it. If the container already exists it will be // currently available for it. If the container already exists it will be
// returned. // returned.
func (e *Environment) Create() error { func (e *Environment) Create() error {
ctx := context.Background()
// If the container already exists don't hit the user with an error, just return // If the container already exists don't hit the user with an error, just return
// the current information about it which is what we would do when creating the // the current information about it which is what we would do when creating the
// container anyways. // container anyways.
if _, err := e.ContainerInspect(context.Background()); err == nil { if _, err := e.ContainerInspect(ctx); err == nil {
return nil return nil
} else if !client.IsErrNotFound(err) { } else if !client.IsErrNotFound(err) {
return errors.Wrap(err, "environment/docker: failed to inspect container") return errors.WrapIf(err, "environment/docker: failed to inspect container")
} }
// Try to pull the requested image before creating the container. // Try to pull the requested image before creating the container.
@ -165,21 +151,30 @@ func (e *Environment) Create() error {
return errors.WithStackIf(err) return errors.WithStackIf(err)
} }
cfg := config.Get()
a := e.Configuration.Allocations() a := e.Configuration.Allocations()
evs := e.Configuration.EnvironmentVariables() evs := e.Configuration.EnvironmentVariables()
for i, v := range evs { for i, v := range evs {
// Convert 127.0.0.1 to the pterodactyl0 network interface if the environment is Docker // Convert 127.0.0.1 to the pterodactyl0 network interface if the environment is Docker
// so that the server operates as expected. // so that the server operates as expected.
if v == "SERVER_IP=127.0.0.1" { if v == "SERVER_IP=127.0.0.1" {
evs[i] = "SERVER_IP=" + config.Get().Docker.Network.Interface evs[i] = "SERVER_IP=" + cfg.Docker.Network.Interface
} }
} }
// Merge user-provided labels with system labels
confLabels := e.Configuration.Labels()
labels := make(map[string]string, 2+len(confLabels))
for key := range confLabels {
labels[key] = confLabels[key]
}
labels["Service"] = "Pterodactyl"
labels["ContainerType"] = "server_process"
conf := &container.Config{ conf := &container.Config{
Hostname: e.Id, Hostname: e.Id,
Domainname: config.Get().Docker.Domainname, Domainname: cfg.Docker.Domainname,
User: strconv.Itoa(config.Get().System.User.Uid) + ":" + strconv.Itoa(config.Get().System.User.Gid),
AttachStdin: true, AttachStdin: true,
AttachStdout: true, AttachStdout: true,
AttachStderr: true, AttachStderr: true,
@ -188,13 +183,44 @@ func (e *Environment) Create() error {
ExposedPorts: a.Exposed(), ExposedPorts: a.Exposed(),
Image: strings.TrimPrefix(e.meta.Image, "~"), Image: strings.TrimPrefix(e.meta.Image, "~"),
Env: e.Configuration.EnvironmentVariables(), Env: e.Configuration.EnvironmentVariables(),
Labels: map[string]string{ Labels: labels,
"Service": "Pterodactyl",
"ContainerType": "server_process",
},
} }
tmpfsSize := strconv.Itoa(int(config.Get().Docker.TmpfsSize)) // Set the user running the container properly depending on what mode we are operating in.
if cfg.System.User.Rootless.Enabled {
conf.User = fmt.Sprintf("%d:%d", cfg.System.User.Rootless.ContainerUID, cfg.System.User.Rootless.ContainerGID)
} else {
conf.User = strconv.Itoa(cfg.System.User.Uid) + ":" + strconv.Itoa(cfg.System.User.Gid)
}
networkMode := container.NetworkMode(cfg.Docker.Network.Mode)
if a.ForceOutgoingIP {
e.log().Debug("environment/docker: forcing outgoing IP address")
networkName := "ip-" + strings.ReplaceAll(strings.ReplaceAll(a.DefaultMapping.Ip, ".", "-"), ":", "-")
networkMode = container.NetworkMode(networkName)
if _, err := e.client.NetworkInspect(ctx, networkName, types.NetworkInspectOptions{}); err != nil {
if !client.IsErrNotFound(err) {
return err
}
if _, err := e.client.NetworkCreate(ctx, networkName, types.NetworkCreate{
Driver: "bridge",
EnableIPv6: false,
Internal: false,
Attachable: false,
Ingress: false,
ConfigOnly: false,
Options: map[string]string{
"encryption": "false",
"com.docker.network.bridge.default_bridge": "false",
"com.docker.network.host_ipv4": a.DefaultMapping.Ip,
},
}); err != nil {
return err
}
}
}
hostConf := &container.HostConfig{ hostConf := &container.HostConfig{
PortBindings: a.DockerBindings(), PortBindings: a.DockerBindings(),
@ -206,27 +232,20 @@ func (e *Environment) Create() error {
// Configure the /tmp folder mapping in containers. This is necessary for some // Configure the /tmp folder mapping in containers. This is necessary for some
// games that need to make use of it for downloads and other installation processes. // games that need to make use of it for downloads and other installation processes.
Tmpfs: map[string]string{ Tmpfs: map[string]string{
"/tmp": "rw,exec,nosuid,size=" + tmpfsSize + "M", "/tmp": "rw,exec,nosuid,size=" + strconv.Itoa(int(cfg.Docker.TmpfsSize)) + "M",
}, },
// Define resource limits for the container based on the data passed through // Define resource limits for the container based on the data passed through
// from the Panel. // from the Panel.
Resources: e.Configuration.Limits().AsContainerResources(), Resources: e.Configuration.Limits().AsContainerResources(),
DNS: config.Get().Docker.Network.Dns, DNS: cfg.Docker.Network.Dns,
// Configure logging for the container to make it easier on the Daemon to grab // Configure logging for the container to make it easier on the Daemon to grab
// the server output. Ensure that we don't use too much space on the host machine // the server output. Ensure that we don't use too much space on the host machine
// since we only need it for the last few hundred lines of output and don't care // since we only need it for the last few hundred lines of output and don't care
// about anything else in it. // about anything else in it.
LogConfig: container.LogConfig{ LogConfig: cfg.Docker.ContainerLogConfig(),
Type: jsonfilelog.Name,
Config: map[string]string{
"max-size": "5m",
"max-file": "1",
"compress": "false",
},
},
SecurityOpt: []string{"no-new-privileges"}, SecurityOpt: []string{"no-new-privileges"},
ReadonlyRootfs: true, ReadonlyRootfs: true,
@ -234,10 +253,11 @@ func (e *Environment) Create() error {
"setpcap", "mknod", "audit_write", "net_raw", "dac_override", "setpcap", "mknod", "audit_write", "net_raw", "dac_override",
"fowner", "fsetid", "net_bind_service", "sys_chroot", "setfcap", "fowner", "fsetid", "net_bind_service", "sys_chroot", "setfcap",
}, },
NetworkMode: container.NetworkMode(config.Get().Docker.Network.Mode), NetworkMode: networkMode,
UsernsMode: container.UsernsMode(cfg.Docker.UsernsMode),
} }
if _, err := e.client.ContainerCreate(context.Background(), conf, hostConf, nil, nil, e.Id); err != nil { if _, err := e.client.ContainerCreate(ctx, conf, hostConf, nil, nil, e.Id); err != nil {
return errors.Wrap(err, "environment/docker: failed to create container") return errors.Wrap(err, "environment/docker: failed to create container")
} }
@ -315,59 +335,6 @@ func (e *Environment) Readlog(lines int) ([]string, error) {
return out, nil return out, nil
} }
// Attaches to the log for the container. This avoids us missing crucial output
// that happens in the split seconds before the code moves from 'Starting' to
// 'Attaching' on the process.
func (e *Environment) followOutput() error {
if exists, err := e.Exists(); !exists {
if err != nil {
return err
}
return errors.New(fmt.Sprintf("no such container: %s", e.Id))
}
opts := types.ContainerLogsOptions{
ShowStderr: true,
ShowStdout: true,
Follow: true,
Since: time.Now().Format(time.RFC3339),
}
reader, err := e.client.ContainerLogs(context.Background(), e.Id, opts)
if err != nil {
return err
}
go e.scanOutput(reader)
return nil
}
func (e *Environment) scanOutput(reader io.ReadCloser) {
defer reader.Close()
if err := system.ScanReader(reader, func(v []byte) {
e.logCallbackMx.Lock()
defer e.logCallbackMx.Unlock()
e.logCallback(v)
}); err != nil && err != io.EOF {
log.WithField("error", err).WithField("container_id", e.Id).Warn("error processing scanner line in console output")
return
}
// Return here if the server is offline or currently stopping.
if e.State() == environment.ProcessStoppingState || e.State() == environment.ProcessOfflineState {
return
}
// Close the current reader before starting a new one, the defer will still run,
// but it will do nothing if we already closed the stream.
_ = reader.Close()
// Start following the output of the server again.
go e.followOutput()
}
// Pulls the image from Docker. If there is an error while pulling the image // Pulls the image from Docker. If there is an error while pulling the image
// from the source but the image already exists locally, we will report that // from the source but the image already exists locally, we will report that
// error to the logger but continue with the process. // error to the logger but continue with the process.
@ -483,21 +450,3 @@ func (e *Environment) convertMounts() []mount.Mount {
return out return out
} }
func (e *Environment) resources() container.Resources {
l := e.Configuration.Limits()
pids := l.ProcessLimit()
return container.Resources{
Memory: l.BoundedMemoryLimit(),
MemoryReservation: l.MemoryLimit * 1_000_000,
MemorySwap: l.ConvertedSwap(),
CPUQuota: l.ConvertedCpuLimit(),
CPUPeriod: 100_000,
CPUShares: 1024,
BlkioWeight: l.IoWeight,
OomKillDisable: &l.OOMDisabled,
CpusetCpus: l.Threads,
PidsLimit: &pids,
}
}

View File

@ -10,6 +10,7 @@ import (
"github.com/apex/log" "github.com/apex/log"
"github.com/docker/docker/api/types" "github.com/docker/docker/api/types"
"github.com/docker/docker/client" "github.com/docker/docker/client"
"github.com/pterodactyl/wings/environment" "github.com/pterodactyl/wings/environment"
"github.com/pterodactyl/wings/events" "github.com/pterodactyl/wings/events"
"github.com/pterodactyl/wings/remote" "github.com/pterodactyl/wings/remote"
@ -26,8 +27,7 @@ type Metadata struct {
var _ environment.ProcessEnvironment = (*Environment)(nil) var _ environment.ProcessEnvironment = (*Environment)(nil)
type Environment struct { type Environment struct {
mu sync.RWMutex mu sync.RWMutex
eventMu sync.Once
// The public identifier for this environment. In this case it is the Docker container // The public identifier for this environment. In this case it is the Docker container
// name that will be used for all instances created under it. // name that will be used for all instances created under it.
@ -73,6 +73,7 @@ func New(id string, m *Metadata, c *environment.Configuration) (*Environment, er
meta: m, meta: m,
client: cli, client: cli,
st: system.NewAtomicString(environment.ProcessOfflineState), st: system.NewAtomicString(environment.ProcessOfflineState),
emitter: events.NewBus(),
} }
return e, nil return e, nil
@ -86,34 +87,33 @@ func (e *Environment) Type() string {
return "docker" return "docker"
} }
// Set if this process is currently attached to the process. // SetStream sets the current stream value from the Docker client. If a nil
// value is provided we assume that the stream is no longer operational and the
// instance is effectively offline.
func (e *Environment) SetStream(s *types.HijackedResponse) { func (e *Environment) SetStream(s *types.HijackedResponse) {
e.mu.Lock() e.mu.Lock()
defer e.mu.Unlock()
e.stream = s e.stream = s
e.mu.Unlock()
} }
// Determine if the this process is currently attached to the container. // IsAttached determines if this process is currently attached to the
// container instance by checking if the stream is nil or not.
func (e *Environment) IsAttached() bool { func (e *Environment) IsAttached() bool {
e.mu.RLock() e.mu.RLock()
defer e.mu.RUnlock() defer e.mu.RUnlock()
return e.stream != nil return e.stream != nil
} }
// Events returns an event bus for the environment.
func (e *Environment) Events() *events.Bus { func (e *Environment) Events() *events.Bus {
e.eventMu.Do(func() {
e.emitter = events.NewBus()
})
return e.emitter return e.emitter
} }
// Determines if the container exists in this environment. The ID passed through should be the // Exists determines if the container exists in this environment. The ID passed
// server UUID since containers are created utilizing the server UUID as the name and docker // through should be the server UUID since containers are created utilizing the
// will work fine when using the container name as the lookup parameter in addition to the longer // server UUID as the name and docker will work fine when using the container
// ID auto-assigned when the container is created. // name as the lookup parameter in addition to the longer ID auto-assigned when
// the container is created.
func (e *Environment) Exists() (bool, error) { func (e *Environment) Exists() (bool, error) {
_, err := e.ContainerInspect(context.Background()) _, err := e.ContainerInspect(context.Background())
if err != nil { if err != nil {
@ -122,10 +122,8 @@ func (e *Environment) Exists() (bool, error) {
if client.IsErrNotFound(err) { if client.IsErrNotFound(err) {
return false, nil return false, nil
} }
return false, err return false, err
} }
return true, nil return true, nil
} }
@ -146,7 +144,7 @@ func (e *Environment) IsRunning(ctx context.Context) (bool, error) {
return c.State.Running, nil return c.State.Running, nil
} }
// Determine the container exit state and return the exit code and whether or not // ExitState returns the container exit state, the exit code and whether or not
// the container was killed by the OOM killer. // the container was killed by the OOM killer.
func (e *Environment) ExitState() (uint32, bool, error) { func (e *Environment) ExitState() (uint32, bool, error) {
c, err := e.ContainerInspect(context.Background()) c, err := e.ContainerInspect(context.Background())
@ -163,15 +161,13 @@ func (e *Environment) ExitState() (uint32, bool, error) {
if client.IsErrNotFound(err) { if client.IsErrNotFound(err) {
return 1, false, nil return 1, false, nil
} }
return 0, false, errors.WrapIf(err, "environment/docker: failed to inspect container")
return 0, false, err
} }
return uint32(c.State.ExitCode), c.State.OOMKilled, nil return uint32(c.State.ExitCode), c.State.OOMKilled, nil
} }
// Returns the environment configuration allowing a process to make modifications of the // Config returns the environment configuration allowing a process to make
// environment on the fly. // modifications of the environment on the fly.
func (e *Environment) Config() *environment.Configuration { func (e *Environment) Config() *environment.Configuration {
e.mu.RLock() e.mu.RLock()
defer e.mu.RUnlock() defer e.mu.RUnlock()
@ -179,12 +175,11 @@ func (e *Environment) Config() *environment.Configuration {
return e.Configuration return e.Configuration
} }
// Sets the stop configuration for the environment. // SetStopConfiguration sets the stop configuration for the environment.
func (e *Environment) SetStopConfiguration(c remote.ProcessStopConfiguration) { func (e *Environment) SetStopConfiguration(c remote.ProcessStopConfiguration) {
e.mu.Lock() e.mu.Lock()
defer e.mu.Unlock()
e.meta.Stop = c e.meta.Stop = c
e.mu.Unlock()
} }
func (e *Environment) SetImage(i string) { func (e *Environment) SetImage(i string) {

View File

@ -4,7 +4,6 @@ import (
"context" "context"
"os" "os"
"strings" "strings"
"syscall"
"time" "time"
"emperror.dev/errors" "emperror.dev/errors"
@ -39,7 +38,7 @@ func (e *Environment) OnBeforeStart(ctx context.Context) error {
// //
// This won't actually run an installation process however, it is just here to ensure the // This won't actually run an installation process however, it is just here to ensure the
// environment gets created properly if it is missing and the server is started. We're making // environment gets created properly if it is missing and the server is started. We're making
// an assumption that all of the files will still exist at this point. // an assumption that all the files will still exist at this point.
if err := e.Create(); err != nil { if err := e.Create(); err != nil {
return err return err
} }
@ -103,23 +102,28 @@ func (e *Environment) Start(ctx context.Context) error {
// exists on the system, and rebuild the container if that is required for server booting to // exists on the system, and rebuild the container if that is required for server booting to
// occur. // occur.
if err := e.OnBeforeStart(ctx); err != nil { if err := e.OnBeforeStart(ctx); err != nil {
return errors.WithStackIf(err) return errors.WrapIf(err, "environment/docker: failed to run pre-boot process")
} }
// If we cannot start & attach to the container in 30 seconds something has gone // If we cannot start & attach to the container in 30 seconds something has gone
// quite sideways and we should stop trying to avoid a hanging situation. // quite sideways, and we should stop trying to avoid a hanging situation.
actx, cancel := context.WithTimeout(ctx, time.Second*30) actx, cancel := context.WithTimeout(ctx, time.Second*30)
defer cancel() defer cancel()
// You must attach to the instance _before_ you start the container. If you do this
// in the opposite order you'll enter a deadlock condition where we're attached to
// the instance successfully, but the container has already stopped and you'll get
// the entire program into a very confusing state.
//
// By explicitly attaching to the instance before we start it, we can immediately
// react to errors/output stopping/etc. when starting.
if err := e.Attach(actx); err != nil { if err := e.Attach(actx); err != nil {
return err return errors.WrapIf(err, "environment/docker: failed to attach to container")
} }
e.log().Debug("attempting to start container...")
if err := e.client.ContainerStart(actx, e.Id, types.ContainerStartOptions{}); err != nil { if err := e.client.ContainerStart(actx, e.Id, types.ContainerStartOptions{}); err != nil {
return errors.WrapIf(err, "environment/docker: failed to start container") return errors.WrapIf(err, "environment/docker: failed to start container")
} }
e.log().Debug("started container!")
// No errors, good to continue through. // No errors, good to continue through.
sawError = false sawError = false
@ -133,49 +137,60 @@ func (e *Environment) Start(ctx context.Context) error {
// You most likely want to be using WaitForStop() rather than this function, // You most likely want to be using WaitForStop() rather than this function,
// since this will return as soon as the command is sent, rather than waiting // since this will return as soon as the command is sent, rather than waiting
// for the process to be completed stopped. // for the process to be completed stopped.
// func (e *Environment) Stop(ctx context.Context) error {
// TODO: pass context through from the server instance.
func (e *Environment) Stop() error {
e.mu.RLock() e.mu.RLock()
s := e.meta.Stop s := e.meta.Stop
e.mu.RUnlock() e.mu.RUnlock()
// A native "stop" as the Type field value will just skip over all of this
// logic and end up only executing the container stop command (which may or
// may not work as expected).
if s.Type == "" || s.Type == remote.ProcessStopSignal {
if s.Type == "" {
log.WithField("container_id", e.Id).Warn("no stop configuration detected for environment, using termination procedure")
}
signal := os.Kill
// Handle a few common cases, otherwise just fall through and just pass along
// the os.Kill signal to the process.
switch strings.ToUpper(s.Value) {
case "SIGABRT":
signal = syscall.SIGABRT
case "SIGINT":
signal = syscall.SIGINT
case "SIGTERM":
signal = syscall.SIGTERM
}
return e.Terminate(signal)
}
// If the process is already offline don't switch it back to stopping. Just leave it how // If the process is already offline don't switch it back to stopping. Just leave it how
// it is and continue through to the stop handling for the process. // it is and continue through to the stop handling for the process.
if e.st.Load() != environment.ProcessOfflineState { if e.st.Load() != environment.ProcessOfflineState {
e.SetState(environment.ProcessStoppingState) e.SetState(environment.ProcessStoppingState)
} }
// Handle signal based actions
if s.Type == remote.ProcessStopSignal {
log.WithField("signal_value", s.Value).Debug("stopping server using signal")
// Handle some common signals - Default to SIGKILL
signal := "SIGKILL"
switch strings.ToUpper(s.Value) {
case "SIGABRT":
signal = "SIGABRT"
case "SIGINT", "C":
signal = "SIGINT"
case "SIGTERM":
signal = "SIGTERM"
case "SIGKILL":
signal = "SIGKILL"
default:
log.Info("Unrecognised signal requested, defaulting to SIGKILL")
}
return e.SignalContainer(ctx, signal)
}
// Handle command based stops
// Only attempt to send the stop command to the instance if we are actually attached to // Only attempt to send the stop command to the instance if we are actually attached to
// the instance. If we are not for some reason, just send the container stop event. // the instance. If we are not for some reason, just send the container stop event.
if e.IsAttached() && s.Type == remote.ProcessStopCommand { if e.IsAttached() && s.Type == remote.ProcessStopCommand {
return e.SendCommand(s.Value) return e.SendCommand(s.Value)
} }
t := time.Second * 30 if s.Type == "" {
if err := e.client.ContainerStop(context.Background(), e.Id, &t); err != nil { log.WithField("container_id", e.Id).Warn("no stop configuration detected for environment, using native docker stop")
}
// Fallback to a native docker stop. As we aren't passing a signal to ContainerStop docker will
// attempt to stop the container using the default stop signal, SIGTERM, unless
// another signal was specified in the Dockerfile
//
// Using a negative timeout here will allow the container to stop gracefully,
// rather than forcefully terminating it. Value is in seconds, but -1 is
// treated as indefinitely.
timeout := -1
if err := e.client.ContainerStop(ctx, e.Id, container.StopOptions{Timeout: &timeout}); err != nil {
// If the container does not exist just mark the process as stopped and return without // If the container does not exist just mark the process as stopped and return without
// an error. // an error.
if client.IsErrNotFound(err) { if client.IsErrNotFound(err) {
@ -193,54 +208,75 @@ func (e *Environment) Stop() error {
// command. If the server does not stop after seconds have passed, an error will // command. If the server does not stop after seconds have passed, an error will
// be returned, or the instance will be terminated forcefully depending on the // be returned, or the instance will be terminated forcefully depending on the
// value of the second argument. // value of the second argument.
func (e *Environment) WaitForStop(seconds uint, terminate bool) error { //
if err := e.Stop(); err != nil { // Calls to Environment.Terminate() in this function use the context passed
return err // through since we don't want to prevent termination of the server instance
// just because the context.WithTimeout() has expired.
func (e *Environment) WaitForStop(ctx context.Context, duration time.Duration, terminate bool) error {
tctx, cancel := context.WithTimeout(context.Background(), duration)
defer cancel()
// If the parent context is canceled, abort the timed context for termination.
go func() {
select {
case <-ctx.Done():
cancel()
case <-tctx.Done():
// When the timed context is canceled, terminate this routine since we no longer
// need to worry about the parent routine being canceled.
break
}
}()
doTermination := func(s string) error {
e.log().WithField("step", s).WithField("duration", duration).Warn("container stop did not complete in time, terminating process...")
return e.Terminate(ctx, "SIGKILL")
} }
ctx, cancel := context.WithTimeout(context.Background(), time.Duration(seconds)*time.Second) // We pass through the timed context for this stop action so that if one of the
defer cancel() // internal docker calls fails to ever finish before we've exhausted the time limit
// the resources get cleaned up, and the exection is stopped.
if err := e.Stop(tctx); err != nil {
if terminate && errors.Is(err, context.DeadlineExceeded) {
return doTermination("stop")
}
return err
}
// Block the return of this function until the container as been marked as no // Block the return of this function until the container as been marked as no
// longer running. If this wait does not end by the time seconds have passed, // longer running. If this wait does not end by the time seconds have passed,
// attempt to terminate the container, or return an error. // attempt to terminate the container, or return an error.
ok, errChan := e.client.ContainerWait(ctx, e.Id, container.WaitConditionNotRunning) ok, errChan := e.client.ContainerWait(tctx, e.Id, container.WaitConditionNotRunning)
select { select {
case <-ctx.Done(): case <-ctx.Done():
if ctxErr := ctx.Err(); ctxErr != nil { if err := ctx.Err(); err != nil {
if terminate { if terminate {
log.WithField("container_id", e.Id).Info("server did not stop in time, executing process termination") return doTermination("parent-context")
return e.Terminate(os.Kill)
} }
return err
return ctxErr
} }
case err := <-errChan: case err := <-errChan:
// If the error stems from the container not existing there is no point in wasting // If the error stems from the container not existing there is no point in wasting
// CPU time to then try and terminate it. // CPU time to then try and terminate it.
if err != nil && !client.IsErrNotFound(err) { if err == nil || client.IsErrNotFound(err) {
if terminate { return nil
l := log.WithField("container_id", e.Id)
if errors.Is(err, context.DeadlineExceeded) {
l.Warn("deadline exceeded for container stop; terminating process")
} else {
l.WithField("error", err).Warn("error while waiting for container stop; terminating process")
}
return e.Terminate(os.Kill)
}
return errors.WrapIf(err, "environment/docker: error waiting on container to enter \"not-running\" state")
} }
if terminate {
if !errors.Is(err, context.DeadlineExceeded) {
e.log().WithField("error", err).Warn("error while waiting for container stop; terminating process")
}
return doTermination("wait")
}
return errors.WrapIf(err, "environment/docker: error waiting on container to enter \"not-running\" state")
case <-ok: case <-ok:
} }
return nil return nil
} }
// Terminate forcefully terminates the container using the signal provided. // Sends the specified signal to the container in an attempt to stop it.
func (e *Environment) Terminate(signal os.Signal) error { func (e *Environment) SignalContainer(ctx context.Context, signal string) error {
c, err := e.ContainerInspect(context.Background()) c, err := e.ContainerInspect(ctx)
if err != nil { if err != nil {
// Treat missing containers as an okay error state, means it is obviously // Treat missing containers as an okay error state, means it is obviously
// already terminated at this point. // already terminated at this point.
@ -264,11 +300,27 @@ func (e *Environment) Terminate(signal os.Signal) error {
// We set it to stopping than offline to prevent crash detection from being triggered. // We set it to stopping than offline to prevent crash detection from being triggered.
e.SetState(environment.ProcessStoppingState) e.SetState(environment.ProcessStoppingState)
sig := strings.TrimSuffix(strings.TrimPrefix(signal.String(), "signal "), "ed") if err := e.client.ContainerKill(ctx, e.Id, signal); err != nil && !client.IsErrNotFound(err) {
if err := e.client.ContainerKill(context.Background(), e.Id, sig); err != nil && !client.IsErrNotFound(err) {
return errors.WithStack(err) return errors.WithStack(err)
} }
return nil
}
// Terminate forcefully terminates the container using the signal provided.
// then sets its state to stopped.
func (e *Environment) Terminate(ctx context.Context, signal string) error {
// Send the signal to the container to kill it
if err := e.SignalContainer(ctx, signal); err != nil {
return errors.WithStack(err)
}
// We expect Terminate to instantly kill the container
// so go ahead and mark it as dead and clean up
e.SetState(environment.ProcessOfflineState) e.SetState(environment.ProcessOfflineState)
return nil return nil
} }

View File

@ -134,7 +134,11 @@ func calculateDockerAbsoluteCpu(pStats types.CPUStats, stats types.CPUStats) flo
percent := 0.0 percent := 0.0
if systemDelta > 0.0 && cpuDelta > 0.0 { if systemDelta > 0.0 && cpuDelta > 0.0 {
percent = (cpuDelta / systemDelta) * cpus * 100.0 percent = (cpuDelta / systemDelta) * 100.0
if cpus > 0 {
percent *= cpus
}
} }
return math.Round(percent*1000) / 1000 return math.Round(percent*1000) / 1000

View File

@ -2,7 +2,7 @@ package environment
import ( import (
"context" "context"
"os" "time"
"github.com/pterodactyl/wings/events" "github.com/pterodactyl/wings/events"
) )
@ -58,18 +58,20 @@ type ProcessEnvironment interface {
// can be started an error should be returned. // can be started an error should be returned.
Start(ctx context.Context) error Start(ctx context.Context) error
// Stops a server instance. If the server is already stopped an error should // Stop stops a server instance. If the server is already stopped an error will
// not be returned. // not be returned, this function will act as a no-op.
Stop() error Stop(ctx context.Context) error
// Waits for a server instance to stop gracefully. If the server is still detected // WaitForStop waits for a server instance to stop gracefully. If the server is
// as running after seconds, an error will be returned, or the server will be terminated // still detected as running after "duration", an error will be returned, or the server
// depending on the value of the second argument. // will be terminated depending on the value of the second argument. If the context
WaitForStop(seconds uint, terminate bool) error // provided is canceled the underlying wait conditions will be stopped and the
// entire loop will be ended (potentially without stopping or terminating).
WaitForStop(ctx context.Context, duration time.Duration, terminate bool) error
// Terminates a running server instance using the provided signal. If the server // Terminate stops a running server instance using the provided signal. This function
// is not running no error should be returned. // is a no-op if the server is already stopped.
Terminate(signal os.Signal) error Terminate(ctx context.Context, signal string) error
// Destroys the environment removing any containers that were created (in Docker // Destroys the environment removing any containers that were created (in Docker
// environments at least). // environments at least).

View File

@ -34,7 +34,7 @@ type Mount struct {
// Limits is the build settings for a given server that impact docker container // Limits is the build settings for a given server that impact docker container
// creation and resource limits for a server instance. // creation and resource limits for a server instance.
type Limits struct { type Limits struct {
// The total amount of memory in megabytes that this server is allowed to // The total amount of memory in mebibytes that this server is allowed to
// use on the host system. // use on the host system.
MemoryLimit int64 `json:"memory_limit"` MemoryLimit int64 `json:"memory_limit"`
@ -79,7 +79,7 @@ func (l Limits) MemoryOverheadMultiplier() float64 {
} }
func (l Limits) BoundedMemoryLimit() int64 { func (l Limits) BoundedMemoryLimit() int64 {
return int64(math.Round(float64(l.MemoryLimit) * l.MemoryOverheadMultiplier() * 1_000_000)) return int64(math.Round(float64(l.MemoryLimit) * l.MemoryOverheadMultiplier() * 1024 * 1024))
} }
// ConvertedSwap returns the amount of swap available as a total in bytes. This // ConvertedSwap returns the amount of swap available as a total in bytes. This
@ -90,7 +90,7 @@ func (l Limits) ConvertedSwap() int64 {
return -1 return -1
} }
return (l.Swap * 1_000_000) + l.BoundedMemoryLimit() return (l.Swap * 1024 * 1024) + l.BoundedMemoryLimit()
} }
// ProcessLimit returns the process limit for a container. This is currently // ProcessLimit returns the process limit for a container. This is currently
@ -99,21 +99,36 @@ func (l Limits) ProcessLimit() int64 {
return config.Get().Docker.ContainerPidLimit return config.Get().Docker.ContainerPidLimit
} }
// AsContainerResources returns the available resources for a container in a format
// that Docker understands.
func (l Limits) AsContainerResources() container.Resources { func (l Limits) AsContainerResources() container.Resources {
pids := l.ProcessLimit() pids := l.ProcessLimit()
resources := container.Resources{
return container.Resources{
Memory: l.BoundedMemoryLimit(), Memory: l.BoundedMemoryLimit(),
MemoryReservation: l.MemoryLimit * 1_000_000, MemoryReservation: l.MemoryLimit * 1024 * 1024,
MemorySwap: l.ConvertedSwap(), MemorySwap: l.ConvertedSwap(),
CPUQuota: l.ConvertedCpuLimit(),
CPUPeriod: 100_000,
CPUShares: 1024,
BlkioWeight: l.IoWeight, BlkioWeight: l.IoWeight,
OomKillDisable: &l.OOMDisabled, OomKillDisable: &l.OOMDisabled,
CpusetCpus: l.Threads,
PidsLimit: &pids, PidsLimit: &pids,
} }
// If the CPU Limit is not set, don't send any of these fields through. Providing
// them seems to break some Java services that try to read the available processors.
//
// @see https://github.com/pterodactyl/panel/issues/3988
if l.CpuLimit > 0 {
resources.CPUQuota = l.CpuLimit * 1_000
resources.CPUPeriod = 100_000
resources.CPUShares = 1024
}
// Similar to above, don't set the specific assigned CPUs if we didn't actually limit
// the server to any of them.
if l.Threads != "" {
resources.CpusetCpus = l.Threads
}
return resources
} }
type Variables map[string]interface{} type Variables map[string]interface{}

View File

@ -2,10 +2,12 @@ package events
import ( import (
"strings" "strings"
"sync"
)
type Listener chan Event "emperror.dev/errors"
"github.com/goccy/go-json"
"github.com/pterodactyl/wings/system"
)
// Event represents an Event sent over a Bus. // Event represents an Event sent over a Bus.
type Event struct { type Event struct {
@ -15,137 +17,55 @@ type Event struct {
// Bus represents an Event Bus. // Bus represents an Event Bus.
type Bus struct { type Bus struct {
listenersMx sync.Mutex *system.SinkPool
listeners map[string][]Listener
} }
// NewBus returns a new empty Event Bus. // NewBus returns a new empty Bus. This is simply a nicer wrapper around the
// system.SinkPool implementation that allows for more simplistic usage within
// the codebase.
//
// All of the events emitted out of this bus are byte slices that can be decoded
// back into an events.Event interface.
func NewBus() *Bus { func NewBus() *Bus {
return &Bus{ return &Bus{
listeners: make(map[string][]Listener), system.NewSinkPool(),
}
}
// Off unregisters a listener from the specified topics on the Bus.
func (b *Bus) Off(listener Listener, topics ...string) {
b.listenersMx.Lock()
defer b.listenersMx.Unlock()
var closed bool
for _, topic := range topics {
ok := b.off(topic, listener)
if !closed && ok {
close(listener)
closed = true
}
}
}
func (b *Bus) off(topic string, listener Listener) bool {
listeners, ok := b.listeners[topic]
if !ok {
return false
}
for i, l := range listeners {
if l != listener {
continue
}
listeners = append(listeners[:i], listeners[i+1:]...)
b.listeners[topic] = listeners
return true
}
return false
}
// On registers a listener to the specified topics on the Bus.
func (b *Bus) On(listener Listener, topics ...string) {
b.listenersMx.Lock()
defer b.listenersMx.Unlock()
for _, topic := range topics {
b.on(topic, listener)
}
}
func (b *Bus) on(topic string, listener Listener) {
listeners, ok := b.listeners[topic]
if !ok {
b.listeners[topic] = []Listener{listener}
} else {
b.listeners[topic] = append(listeners, listener)
} }
} }
// Publish publishes a message to the Bus. // Publish publishes a message to the Bus.
func (b *Bus) Publish(topic string, data interface{}) { func (b *Bus) Publish(topic string, data interface{}) {
// Some of our topics for the socket support passing a more specific namespace, // Some of our actions for the socket support passing a more specific namespace,
// such as "backup completed:1234" to indicate which specific backup was completed. // such as "backup completed:1234" to indicate which specific backup was completed.
// //
// In these cases, we still need to send the event using the standard listener // In these cases, we still need to send the event using the standard listener
// name of "backup completed". // name of "backup completed".
if strings.Contains(topic, ":") { if strings.Contains(topic, ":") {
parts := strings.SplitN(topic, ":", 2) parts := strings.SplitN(topic, ":", 2)
if len(parts) == 2 { if len(parts) == 2 {
topic = parts[0] topic = parts[0]
} }
} }
b.listenersMx.Lock() enc, err := json.Marshal(Event{Topic: topic, Data: data})
defer b.listenersMx.Unlock() if err != nil {
panic(errors.WithStack(err))
listeners, ok := b.listeners[topic]
if !ok {
return
} }
if len(listeners) < 1 { b.Push(enc)
return
}
var wg sync.WaitGroup
event := Event{Topic: topic, Data: data}
for _, listener := range listeners {
l := listener
wg.Add(1)
go func(l Listener, event Event) {
defer wg.Done()
l <- event
}(l, event)
}
wg.Wait()
} }
// Destroy destroys the Event Bus by unregistering and closing all listeners. // MustDecode decodes the event byte slice back into an events.Event struct or
func (b *Bus) Destroy() { // panics if an error is encountered during this process.
b.listenersMx.Lock() func MustDecode(data []byte) (e Event) {
defer b.listenersMx.Unlock() if err := DecodeTo(data, &e); err != nil {
panic(err)
// Track what listeners have already been closed. Because the same listener
// can be listening on multiple topics, we need a way to essentially
// "de-duplicate" all the listeners across all the topics.
var closed []Listener
for _, listeners := range b.listeners {
for _, listener := range listeners {
if contains(closed, listener) {
continue
}
close(listener)
closed = append(closed, listener)
}
} }
return
b.listeners = make(map[string][]Listener)
} }
func contains(closed []Listener, listener Listener) bool { // DecodeTo decodes a byte slice of event data into the given interface.
for _, c := range closed { func DecodeTo(data []byte, v interface{}) error {
if c == listener { if err := json.Unmarshal(data, &v); err != nil {
return true return errors.Wrap(err, "events: failed to decode byte slice")
}
} }
return false return nil
} }

View File

@ -9,162 +9,90 @@ import (
func TestNewBus(t *testing.T) { func TestNewBus(t *testing.T) {
g := Goblin(t) g := Goblin(t)
bus := NewBus()
g.Describe("NewBus", func() { g.Describe("Events", func() {
g.It("is not nil", func() { var bus *Bus
g.Assert(bus).IsNotNil("Bus expected to not be nil") g.BeforeEach(func() {
g.Assert(bus.listeners).IsNotNil("Bus#listeners expected to not be nil") bus = NewBus()
})
})
}
func TestBus_Off(t *testing.T) {
g := Goblin(t)
const topic = "test"
g.Describe("Off", func() {
g.It("unregisters listener", func() {
bus := NewBus()
g.Assert(bus.listeners[topic]).IsNotNil()
g.Assert(len(bus.listeners[topic])).IsZero()
listener := make(chan Event)
bus.On(listener, topic)
g.Assert(len(bus.listeners[topic])).Equal(1, "Listener was not registered")
bus.Off(listener, topic)
g.Assert(len(bus.listeners[topic])).Equal(0, "Topic still has one or more listeners")
}) })
g.It("unregisters correct listener", func() { g.Describe("NewBus", func() {
bus := NewBus() g.It("is not nil", func() {
g.Assert(bus).IsNotNil("Bus expected to not be nil")
listener := make(chan Event) })
listener2 := make(chan Event)
listener3 := make(chan Event)
bus.On(listener, topic)
bus.On(listener2, topic)
bus.On(listener3, topic)
g.Assert(len(bus.listeners[topic])).Equal(3, "Listeners were not registered")
bus.Off(listener, topic)
bus.Off(listener3, topic)
g.Assert(len(bus.listeners[topic])).Equal(1, "Expected 1 listener to remain")
if bus.listeners[topic][0] != listener2 {
// A normal Assert does not properly compare channels.
g.Fail("wrong listener unregistered")
}
// Cleanup
bus.Off(listener2, topic)
})
})
}
func TestBus_On(t *testing.T) {
g := Goblin(t)
const topic = "test"
g.Describe("On", func() {
g.It("registers listener", func() {
bus := NewBus()
g.Assert(bus.listeners[topic]).IsNotNil()
g.Assert(len(bus.listeners[topic])).IsZero()
listener := make(chan Event)
bus.On(listener, topic)
g.Assert(len(bus.listeners[topic])).Equal(1, "Listener was not registered")
if bus.listeners[topic][0] != listener {
// A normal Assert does not properly compare channels.
g.Fail("wrong listener registered")
}
// Cleanup
bus.Off(listener, topic)
})
})
}
func TestBus_Publish(t *testing.T) {
g := Goblin(t)
const topic = "test"
const message = "this is a test message!"
g.Describe("Publish", func() {
g.It("publishes message", func() {
bus := NewBus()
g.Assert(bus.listeners[topic]).IsNotNil()
g.Assert(len(bus.listeners[topic])).IsZero()
listener := make(chan Event)
bus.On(listener, topic)
g.Assert(len(bus.listeners[topic])).Equal(1, "Listener was not registered")
done := make(chan struct{}, 1)
go func() {
select {
case m := <-listener:
g.Assert(m.Topic).Equal(topic)
g.Assert(m.Data).Equal(message)
case <-time.After(1 * time.Second):
g.Fail("listener did not receive message in time")
}
done <- struct{}{}
}()
bus.Publish(topic, message)
<-done
// Cleanup
bus.Off(listener, topic)
}) })
g.It("publishes message to all listeners", func() { g.Describe("Publish", func() {
bus := NewBus() const topic = "test"
const message = "this is a test message!"
g.Assert(bus.listeners[topic]).IsNotNil() g.It("publishes message", func() {
g.Assert(len(bus.listeners[topic])).IsZero() bus := NewBus()
listener := make(chan Event)
listener2 := make(chan Event)
listener3 := make(chan Event)
bus.On(listener, topic)
bus.On(listener2, topic)
bus.On(listener3, topic)
g.Assert(len(bus.listeners[topic])).Equal(3, "Listener was not registered")
done := make(chan struct{}, 1) listener := make(chan []byte)
go func() { bus.On(listener)
for i := 0; i < 3; i++ {
done := make(chan struct{}, 1)
go func() {
select { select {
case m := <-listener: case v := <-listener:
g.Assert(m.Topic).Equal(topic) m := MustDecode(v)
g.Assert(m.Data).Equal(message)
case m := <-listener2:
g.Assert(m.Topic).Equal(topic)
g.Assert(m.Data).Equal(message)
case m := <-listener3:
g.Assert(m.Topic).Equal(topic) g.Assert(m.Topic).Equal(topic)
g.Assert(m.Data).Equal(message) g.Assert(m.Data).Equal(message)
case <-time.After(1 * time.Second): case <-time.After(1 * time.Second):
g.Fail("all listeners did not receive the message in time") g.Fail("listener did not receive message in time")
i = 3
} }
} done <- struct{}{}
}()
bus.Publish(topic, message)
<-done
done <- struct{}{} // Cleanup
}() bus.Off(listener)
bus.Publish(topic, message) })
<-done
// Cleanup g.It("publishes message to all listeners", func() {
bus.Off(listener, topic) bus := NewBus()
bus.Off(listener2, topic)
bus.Off(listener3, topic) listener := make(chan []byte)
listener2 := make(chan []byte)
listener3 := make(chan []byte)
bus.On(listener)
bus.On(listener2)
bus.On(listener3)
done := make(chan struct{}, 1)
go func() {
for i := 0; i < 3; i++ {
select {
case v := <-listener:
m := MustDecode(v)
g.Assert(m.Topic).Equal(topic)
g.Assert(m.Data).Equal(message)
case v := <-listener2:
m := MustDecode(v)
g.Assert(m.Topic).Equal(topic)
g.Assert(m.Data).Equal(message)
case v := <-listener3:
m := MustDecode(v)
g.Assert(m.Topic).Equal(topic)
g.Assert(m.Data).Equal(message)
case <-time.After(1 * time.Second):
g.Fail("all listeners did not receive the message in time")
i = 3
}
}
done <- struct{}{}
}()
bus.Publish(topic, message)
<-done
// Cleanup
bus.Off(listener)
bus.Off(listener2)
bus.Off(listener3)
})
}) })
}) })
} }

79
flake.lock Normal file
View File

@ -0,0 +1,79 @@
{
"nodes": {
"flake-parts": {
"inputs": {
"nixpkgs-lib": "nixpkgs-lib"
},
"locked": {
"lastModified": 1719994518,
"narHash": "sha256-pQMhCCHyQGRzdfAkdJ4cIWiw+JNuWsTX7f0ZYSyz0VY=",
"owner": "hercules-ci",
"repo": "flake-parts",
"rev": "9227223f6d922fee3c7b190b2cc238a99527bbb7",
"type": "github"
},
"original": {
"owner": "hercules-ci",
"repo": "flake-parts",
"type": "github"
}
},
"nixpkgs": {
"locked": {
"lastModified": 1721562059,
"narHash": "sha256-Tybxt65eyOARf285hMHIJ2uul8SULjFZbT9ZaEeUnP8=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "68c9ed8bbed9dfce253cc91560bf9043297ef2fe",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "nixos-unstable",
"repo": "nixpkgs",
"type": "github"
}
},
"nixpkgs-lib": {
"locked": {
"lastModified": 1719876945,
"narHash": "sha256-Fm2rDDs86sHy0/1jxTOKB1118Q0O3Uc7EC0iXvXKpbI=",
"type": "tarball",
"url": "https://github.com/NixOS/nixpkgs/archive/5daf0514482af3f97abaefc78a6606365c9108e2.tar.gz"
},
"original": {
"type": "tarball",
"url": "https://github.com/NixOS/nixpkgs/archive/5daf0514482af3f97abaefc78a6606365c9108e2.tar.gz"
}
},
"root": {
"inputs": {
"flake-parts": "flake-parts",
"nixpkgs": "nixpkgs",
"treefmt-nix": "treefmt-nix"
}
},
"treefmt-nix": {
"inputs": {
"nixpkgs": [
"nixpkgs"
]
},
"locked": {
"lastModified": 1721769617,
"narHash": "sha256-6Pqa0bi5nV74IZcENKYRToRNM5obo1EQ+3ihtunJ014=",
"owner": "numtide",
"repo": "treefmt-nix",
"rev": "8db8970be1fb8be9c845af7ebec53b699fe7e009",
"type": "github"
},
"original": {
"owner": "numtide",
"repo": "treefmt-nix",
"type": "github"
}
}
},
"root": "root",
"version": 7
}

54
flake.nix Normal file
View File

@ -0,0 +1,54 @@
{
description = "Wings";
inputs = {
flake-parts.url = "github:hercules-ci/flake-parts";
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
treefmt-nix = {
url = "github:numtide/treefmt-nix";
inputs.nixpkgs.follows = "nixpkgs";
};
};
outputs = {...} @ inputs:
inputs.flake-parts.lib.mkFlake {inherit inputs;} {
systems = ["aarch64-darwin" "aarch64-linux" "x86_64-darwin" "x86_64-linux"];
imports = [
inputs.treefmt-nix.flakeModule
];
perSystem = {system, ...}: let
pkgs = import inputs.nixpkgs {inherit system;};
in {
devShells.default = pkgs.mkShell {
buildInputs = with pkgs; [
go_1_22
gofumpt
golangci-lint
gotools
];
};
treefmt = {
projectRootFile = "flake.nix";
programs = {
alejandra.enable = true;
deadnix.enable = true;
gofumpt = {
enable = true;
extra = true;
};
shellcheck.enable = true;
shfmt = {
enable = true;
indent_size = 0; # 0 causes shfmt to use tabs
};
yamlfmt.enable = true;
};
};
};
};
}

199
go.mod
View File

@ -1,118 +1,137 @@
module github.com/pterodactyl/wings module github.com/pterodactyl/wings
go 1.17 go 1.21
require ( require (
emperror.dev/errors v0.8.0 emperror.dev/errors v0.8.1
github.com/AlecAivazis/survey/v2 v2.2.15 github.com/AlecAivazis/survey/v2 v2.3.7
github.com/Jeffail/gabs/v2 v2.6.1 github.com/Jeffail/gabs/v2 v2.7.0
github.com/NYTimes/logrotate v1.0.0 github.com/NYTimes/logrotate v1.0.0
github.com/acobaugh/osrelease v0.1.0
github.com/apex/log v1.9.0 github.com/apex/log v1.9.0
github.com/asaskevich/govalidator v0.0.0-20210307081110-f21760c49a8d github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2
github.com/beevik/etree v1.1.0 github.com/beevik/etree v1.3.0
github.com/buger/jsonparser v1.1.1 github.com/buger/jsonparser v1.1.1
github.com/cenkalti/backoff/v4 v4.1.1 github.com/cenkalti/backoff/v4 v4.3.0
github.com/cobaugh/osrelease v0.0.0-20181218015638-a93a0a55a249 github.com/creasty/defaults v1.7.0
github.com/creasty/defaults v1.5.1 github.com/docker/docker v25.0.4+incompatible
github.com/docker/docker v20.10.7+incompatible github.com/docker/go-connections v0.5.0
github.com/docker/go-connections v0.4.0 github.com/fatih/color v1.16.0
github.com/fatih/color v1.12.0 github.com/franela/goblin v0.0.0-20211003143422-0a4f594942bf
github.com/franela/goblin v0.0.0-20200825194134-80c0062ed6cd github.com/gabriel-vasile/mimetype v1.4.3
github.com/gabriel-vasile/mimetype v1.3.1 github.com/gammazero/workerpool v1.1.3
github.com/gammazero/workerpool v1.1.2
github.com/gbrlsnchs/jwt/v3 v3.0.1 github.com/gbrlsnchs/jwt/v3 v3.0.1
github.com/gin-gonic/gin v1.7.2 github.com/gin-gonic/gin v1.9.1
github.com/google/uuid v1.3.0 github.com/glebarez/sqlite v1.11.0
github.com/gorilla/websocket v1.4.2 github.com/go-co-op/gocron v1.37.0
github.com/iancoleman/strcase v0.2.0 github.com/goccy/go-json v0.10.2
github.com/icza/dyno v0.0.0-20210726202311-f1bafe5d9996 github.com/google/uuid v1.6.0
github.com/juju/ratelimit v1.0.1 github.com/gorilla/websocket v1.5.1
github.com/karrick/godirwalk v1.16.1 github.com/iancoleman/strcase v0.3.0
github.com/klauspost/pgzip v1.2.5 github.com/icza/dyno v0.0.0-20230330125955-09f820a8d9c0
github.com/magiconair/properties v1.8.5 github.com/juju/ratelimit v1.0.2
github.com/mattn/go-colorable v0.1.8 github.com/klauspost/compress v1.17.8
github.com/mholt/archiver/v3 v3.5.0 github.com/klauspost/pgzip v1.2.6
github.com/magiconair/properties v1.8.7
github.com/mattn/go-colorable v0.1.13
github.com/mholt/archiver/v4 v4.0.0-alpha.8
github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db
github.com/patrickmn/go-cache v2.1.0+incompatible github.com/patrickmn/go-cache v2.1.0+incompatible
github.com/pkg/profile v1.6.0 github.com/pkg/sftp v1.13.6
github.com/pkg/sftp v1.13.2 github.com/sabhiram/go-gitignore v0.0.0-20210923224102-525f6e181f06
github.com/sabhiram/go-gitignore v0.0.0-20201211210132-54b8a0bf510f github.com/spf13/cobra v1.8.0
github.com/spf13/cobra v1.2.1 github.com/stretchr/testify v1.9.0
github.com/stretchr/testify v1.7.0 golang.org/x/crypto v0.22.0
golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97 golang.org/x/sync v0.7.0
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c golang.org/x/sys v0.19.0
gopkg.in/ini.v1 v1.62.0 gopkg.in/ini.v1 v1.67.0
gopkg.in/yaml.v2 v2.4.0 gopkg.in/yaml.v2 v2.4.0
gopkg.in/yaml.v3 v3.0.1
gorm.io/gorm v1.25.9
) )
require github.com/goccy/go-json v0.9.4
require golang.org/x/sys v0.0.0-20211110154304-99a53858aa08 // indirect
require ( require (
github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 // indirect github.com/Microsoft/go-winio v0.6.1 // indirect
github.com/Microsoft/go-winio v0.5.0 // indirect github.com/Microsoft/hcsshim v0.12.2 // indirect
github.com/Microsoft/hcsshim v0.8.20 // indirect github.com/andybalholm/brotli v1.1.0 // indirect
github.com/andybalholm/brotli v1.0.3 // indirect github.com/bodgit/plumbing v1.3.0 // indirect
github.com/beorn7/perks v1.0.1 // indirect github.com/bodgit/sevenzip v1.5.1 // indirect
github.com/cespare/xxhash/v2 v2.1.1 // indirect github.com/bodgit/windows v1.0.1 // indirect
github.com/containerd/containerd v1.5.5 // indirect github.com/bytedance/sonic v1.11.3 // indirect
github.com/containerd/fifo v1.0.0 // indirect github.com/chenzhuoyu/base64x v0.0.0-20230717121745-296ad89f973d // indirect
github.com/chenzhuoyu/iasm v0.9.1 // indirect
github.com/containerd/log v0.1.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect github.com/davecgh/go-spew v1.1.1 // indirect
github.com/docker/distribution v2.7.1+incompatible // indirect github.com/distribution/reference v0.6.0 // indirect
github.com/docker/go-metrics v0.0.1 // indirect github.com/docker/go-units v0.5.0 // indirect
github.com/docker/go-units v0.4.0 // indirect github.com/dsnet/compress v0.0.2-0.20210315054119-f66993602bf5 // indirect
github.com/dsnet/compress v0.0.1 // indirect github.com/dustin/go-humanize v1.0.1 // indirect
github.com/fsnotify/fsnotify v1.4.9 // indirect github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/gammazero/deque v0.1.0 // indirect github.com/gammazero/deque v0.2.1 // indirect
github.com/gin-contrib/sse v0.1.0 // indirect github.com/gin-contrib/sse v0.1.0 // indirect
github.com/go-playground/locales v0.13.0 // indirect github.com/glebarez/go-sqlite v1.22.0 // indirect
github.com/go-playground/universal-translator v0.17.0 // indirect github.com/go-logr/logr v1.4.1 // indirect
github.com/go-playground/validator/v10 v10.8.0 // indirect github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-playground/locales v0.14.1 // indirect
github.com/go-playground/universal-translator v0.18.1 // indirect
github.com/go-playground/validator/v10 v10.19.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/protobuf v1.5.2 // indirect
github.com/golang/snappy v0.0.4 // indirect github.com/golang/snappy v0.0.4 // indirect
github.com/gorilla/mux v1.7.4 // indirect github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/inconshreveable/mousetrap v1.0.0 // indirect github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/json-iterator/go v1.1.11 // indirect github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/jinzhu/inflection v1.0.0 // indirect
github.com/jinzhu/now v1.1.5 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect
github.com/klauspost/compress v1.13.2 // indirect github.com/klauspost/cpuid/v2 v2.2.7 // indirect
github.com/kr/fs v0.1.0 // indirect github.com/kr/fs v0.1.0 // indirect
github.com/leodido/go-urn v1.2.1 // indirect github.com/leodido/go-urn v1.4.0 // indirect
github.com/magefile/mage v1.11.0 // indirect github.com/magefile/mage v1.15.0 // indirect
github.com/mattn/go-isatty v0.0.13 // indirect github.com/mattn/go-isatty v0.0.20 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369 // indirect
github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d // indirect github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d // indirect
github.com/moby/term v0.0.0-20210619224110-3f7ff695adc6 // indirect github.com/moby/term v0.0.0-20220808134915-39b0c02b01ae // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.1 // indirect github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/morikuni/aec v1.0.0 // indirect github.com/morikuni/aec v1.0.0 // indirect
github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e // indirect github.com/ncruces/go-strftime v0.1.9 // indirect
github.com/nwaples/rardecode v1.1.1 // indirect github.com/nwaples/rardecode/v2 v2.0.0-beta.2 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.0.1 // indirect github.com/opencontainers/image-spec v1.1.0 // indirect
github.com/pierrec/lz4/v4 v4.1.8 // indirect github.com/pelletier/go-toml/v2 v2.2.0 // indirect
github.com/pierrec/lz4/v4 v4.1.21 // indirect
github.com/pkg/errors v0.9.1 // indirect github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/prometheus/client_golang v1.11.0 // indirect github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/prometheus/client_model v0.2.0 // indirect github.com/robfig/cron/v3 v3.0.1 // indirect
github.com/prometheus/common v0.30.0 // indirect github.com/sirupsen/logrus v1.9.3 // indirect
github.com/prometheus/procfs v0.7.1 // indirect
github.com/sirupsen/logrus v1.8.1 // indirect
github.com/spf13/pflag v1.0.5 // indirect github.com/spf13/pflag v1.0.5 // indirect
github.com/ugorji/go/codec v1.1.7 // indirect github.com/therootcompany/xz v1.0.1 // indirect
github.com/ulikunitz/xz v0.5.10 // indirect github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8 // indirect github.com/ugorji/go/codec v1.2.12 // indirect
go.uber.org/atomic v1.9.0 // indirect github.com/ulikunitz/xz v0.5.12 // indirect
go.uber.org/multierr v1.7.0 // indirect go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.50.0 // indirect
golang.org/x/net v0.0.0-20210726213435-c6fcb2dbf985 // indirect go.opentelemetry.io/otel v1.25.0 // indirect
golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b // indirect go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.24.0 // indirect
golang.org/x/text v0.3.6 // indirect go.opentelemetry.io/otel/metric v1.25.0 // indirect
golang.org/x/time v0.0.0-20210723032227-1f47c861a9ac // indirect go.opentelemetry.io/otel/sdk v1.24.0 // indirect
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect go.opentelemetry.io/otel/trace v1.25.0 // indirect
google.golang.org/genproto v0.0.0-20210729151513-df9385d47c1b // indirect go.uber.org/atomic v1.11.0 // indirect
google.golang.org/grpc v1.39.0 // indirect go.uber.org/multierr v1.11.0 // indirect
google.golang.org/protobuf v1.27.1 // indirect go4.org v0.0.0-20230225012048-214862532bf5 // indirect
gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f // indirect golang.org/x/arch v0.7.0 // indirect
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b // indirect golang.org/x/mod v0.17.0 // indirect
golang.org/x/net v0.24.0 // indirect
golang.org/x/term v0.19.0 // indirect
golang.org/x/text v0.14.0 // indirect
golang.org/x/time v0.0.0-20220922220347-f3bd1da661af // indirect
golang.org/x/tools v0.20.0 // indirect
golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 // indirect
google.golang.org/protobuf v1.33.0 // indirect
gotest.tools/v3 v3.0.2 // indirect
modernc.org/libc v1.49.3 // indirect
modernc.org/mathutil v1.6.0 // indirect
modernc.org/memory v1.8.0 // indirect
modernc.org/sqlite v1.29.6 // indirect
) )

1377
go.sum

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,86 @@
package cron
import (
"context"
"net"
"emperror.dev/errors"
"github.com/pterodactyl/wings/internal/database"
"github.com/pterodactyl/wings/internal/models"
"github.com/pterodactyl/wings/server"
"github.com/pterodactyl/wings/system"
)
type activityCron struct {
mu *system.AtomicBool
manager *server.Manager
max int
}
// Run executes the cronjob and ensures we fetch and send all the stored activity to the
// Panel instance. Once activity is sent it is deleted from the local database instance. Any
// SFTP specific events are not handled in this cron, they're handled separately to account
// for de-duplication and event merging.
func (ac *activityCron) Run(ctx context.Context) error {
// Don't execute this cron if there is currently one running. Once this task is completed
// go ahead and mark it as no longer running.
if !ac.mu.SwapIf(true) {
return errors.WithStack(ErrCronRunning)
}
defer ac.mu.Store(false)
var activity []models.Activity
tx := database.Instance().WithContext(ctx).
Where("event NOT LIKE ?", "server:sftp.%").
Limit(ac.max).
Find(&activity)
if tx.Error != nil {
return errors.WithStack(tx.Error)
}
if len(activity) == 0 {
return nil
}
// ids to delete from the database.
ids := make([]int, 0, len(activity))
// activities to send to the panel.
activities := make([]models.Activity, 0, len(activity))
for _, v := range activity {
// Delete any activity that has an invalid IP address. This is a fix for
// a bug that truncated the last octet of an IPv6 address in the database.
if ip := net.ParseIP(v.IP); ip == nil {
ids = append(ids, v.ID)
continue
}
activities = append(activities, v)
}
if len(ids) > 0 {
tx = database.Instance().WithContext(ctx).Where("id IN ?", ids).Delete(&models.Activity{})
if tx.Error != nil {
return errors.WithStack(tx.Error)
}
}
if len(activities) == 0 {
return nil
}
if err := ac.manager.Client().SendActivityLogs(ctx, activities); err != nil {
return errors.WrapIf(err, "cron: failed to send activity events to Panel")
}
// Add all the successful activities to the list of IDs to delete.
ids = make([]int, len(activities))
for i, v := range activities {
ids[i] = v.ID
}
// Delete all the activities that were sent to the Panel (or that were invalid).
tx = database.Instance().WithContext(ctx).Where("id IN ?", ids).Delete(&models.Activity{})
if tx.Error != nil {
return errors.WithStack(tx.Error)
}
return nil
}

73
internal/cron/cron.go Normal file
View File

@ -0,0 +1,73 @@
package cron
import (
"context"
"time"
"emperror.dev/errors"
"github.com/apex/log"
"github.com/go-co-op/gocron"
"github.com/pterodactyl/wings/config"
"github.com/pterodactyl/wings/server"
"github.com/pterodactyl/wings/system"
)
const ErrCronRunning = errors.Sentinel("cron: job already running")
var o system.AtomicBool
// Scheduler configures the internal cronjob system for Wings and returns the scheduler
// instance to the caller. This should only be called once per application lifecycle, additional
// calls will result in an error being returned.
func Scheduler(ctx context.Context, m *server.Manager) (*gocron.Scheduler, error) {
if !o.SwapIf(true) {
return nil, errors.New("cron: cannot call scheduler more than once in application lifecycle")
}
location, err := time.LoadLocation(config.Get().System.Timezone)
if err != nil {
return nil, errors.Wrap(err, "cron: failed to parse configured system timezone")
}
activity := activityCron{
mu: system.NewAtomicBool(false),
manager: m,
max: config.Get().System.ActivitySendCount,
}
sftp := sftpCron{
mu: system.NewAtomicBool(false),
manager: m,
max: config.Get().System.ActivitySendCount,
}
s := gocron.NewScheduler(location)
l := log.WithField("subsystem", "cron")
interval := time.Duration(config.Get().System.ActivitySendInterval) * time.Second
l.WithField("interval", interval).Info("configuring system crons")
_, _ = s.Tag("activity").Every(interval).Do(func() {
l.WithField("cron", "activity").Debug("sending internal activity events to Panel")
if err := activity.Run(ctx); err != nil {
if errors.Is(err, ErrCronRunning) {
l.WithField("cron", "activity").Warn("activity process is already running, skipping...")
} else {
l.WithField("cron", "activity").WithField("error", err).Error("activity process failed to execute")
}
}
})
_, _ = s.Tag("sftp").Every(interval).Do(func() {
l.WithField("cron", "sftp").Debug("sending sftp events to Panel")
if err := sftp.Run(ctx); err != nil {
if errors.Is(err, ErrCronRunning) {
l.WithField("cron", "sftp").Warn("sftp events process already running, skipping...")
} else {
l.WithField("cron", "sftp").WithField("error", err).Error("sftp events process failed to execute")
}
}
})
return s, nil
}

177
internal/cron/sftp_cron.go Normal file
View File

@ -0,0 +1,177 @@
package cron
import (
"context"
"reflect"
"emperror.dev/errors"
"github.com/pterodactyl/wings/internal/database"
"github.com/pterodactyl/wings/internal/models"
"github.com/pterodactyl/wings/server"
"github.com/pterodactyl/wings/system"
)
type sftpCron struct {
mu *system.AtomicBool
manager *server.Manager
max int
}
type mapKey struct {
User string
Server string
IP string
Event models.Event
Timestamp string
}
type eventMap struct {
max int
ids []int
m map[mapKey]*models.Activity
}
// Run executes the SFTP reconciliation cron. This job will pull all of the SFTP specific events
// and merge them together across user, server, ip, and event. This allows a SFTP event that deletes
// tens or hundreds of files to be tracked as a single "deletion" event so long as they all occur
// within the same one minute period of time (starting at the first timestamp for the group). Without
// this we'd end up flooding the Panel event log with excessive data that is of no use to end users.
func (sc *sftpCron) Run(ctx context.Context) error {
if !sc.mu.SwapIf(true) {
return errors.WithStack(ErrCronRunning)
}
defer sc.mu.Store(false)
var o int
activity, err := sc.fetchRecords(ctx, o)
if err != nil {
return err
}
o += len(activity)
events := &eventMap{
m: map[mapKey]*models.Activity{},
ids: []int{},
max: sc.max,
}
for {
if len(activity) == 0 {
break
}
slen := len(events.ids)
for _, a := range activity {
events.Push(a)
}
if len(events.ids) > slen {
// Execute the query again, we found some events so we want to continue
// with this. Start at the next offset.
activity, err = sc.fetchRecords(ctx, o)
if err != nil {
return errors.WithStack(err)
}
o += len(activity)
} else {
break
}
}
if len(events.m) == 0 {
return nil
}
if err := sc.manager.Client().SendActivityLogs(ctx, events.Elements()); err != nil {
return errors.Wrap(err, "failed to send sftp activity logs to Panel")
}
if tx := database.Instance().Where("id IN ?", events.ids).Delete(&models.Activity{}); tx.Error != nil {
return errors.WithStack(tx.Error)
}
return nil
}
// fetchRecords returns a group of activity events starting at the given offset. This is used
// since we might need to make multiple database queries to select enough events to properly
// fill up our request to the given maximum. This is due to the fact that this cron merges any
// activity that line up across user, server, ip, and event into a single activity record when
// sending the data to the Panel.
func (sc *sftpCron) fetchRecords(ctx context.Context, offset int) (activity []models.Activity, err error) {
tx := database.Instance().WithContext(ctx).
Where("event LIKE ?", "server:sftp.%").
Order("event DESC").
Offset(offset).
Limit(sc.max).
Find(&activity)
if tx.Error != nil {
err = errors.WithStack(tx.Error)
}
return
}
// Push adds an activity to the event mapping, or de-duplicates it and merges the files metadata
// into the existing entity that exists.
func (em *eventMap) Push(a models.Activity) {
m := em.forActivity(a)
// If no activity entity is returned we've hit the cap for the number of events to
// send along to the Panel. Just skip over this record and we'll account for it in
// the next iteration.
if m == nil {
return
}
em.ids = append(em.ids, a.ID)
// Always reduce this to the first timestamp that was recorded for the set
// of events, and not
if a.Timestamp.Before(m.Timestamp) {
m.Timestamp = a.Timestamp
}
list := m.Metadata["files"].([]interface{})
if s, ok := a.Metadata["files"]; ok {
v := reflect.ValueOf(s)
if v.Kind() != reflect.Slice || v.IsNil() {
return
}
for i := 0; i < v.Len(); i++ {
list = append(list, v.Index(i).Interface())
}
// You must set it again at the end of the process, otherwise you've only updated the file
// slice in this one loop since it isn't passed by reference. This is just shorter than having
// to explicitly keep casting it to the slice.
m.Metadata["files"] = list
}
}
// Elements returns the finalized activity models.
func (em *eventMap) Elements() (out []models.Activity) {
for _, v := range em.m {
out = append(out, *v)
}
return
}
// forActivity returns an event entity from our map which allows existing matches to be
// updated with additional files.
func (em *eventMap) forActivity(a models.Activity) *models.Activity {
key := mapKey{
User: a.User.String,
Server: a.Server,
IP: a.IP,
Event: a.Event,
// We group by the minute, don't care about the seconds for this logic.
Timestamp: a.Timestamp.Format("2006-01-02_15:04"),
}
if v, ok := em.m[key]; ok {
return v
}
// Cap the size of the events map at the defined maximum events to send to the Panel. Just
// return nil and let the caller handle it.
if len(em.m) >= em.max {
return nil
}
// Doesn't exist in our map yet, create a copy of the activity passed into this
// function and then assign it into the map with an empty metadata value.
v := a
v.Metadata = models.ActivityMeta{
"files": make([]interface{}, 0),
}
em.m[key] = &v
return &v
}

View File

@ -0,0 +1,61 @@
package database
import (
"path/filepath"
"time"
"emperror.dev/errors"
"github.com/glebarez/sqlite"
"gorm.io/gorm"
"gorm.io/gorm/logger"
"github.com/pterodactyl/wings/config"
"github.com/pterodactyl/wings/internal/models"
"github.com/pterodactyl/wings/system"
)
var (
o system.AtomicBool
db *gorm.DB
)
// Initialize configures the local SQLite database for Wings and ensures that the models have
// been fully migrated.
func Initialize() error {
if !o.SwapIf(true) {
panic("database: attempt to initialize more than once during application lifecycle")
}
p := filepath.Join(config.Get().System.RootDirectory, "wings.db")
instance, err := gorm.Open(sqlite.Open(p), &gorm.Config{
Logger: logger.Default.LogMode(logger.Silent),
})
if err != nil {
return errors.Wrap(err, "database: could not open database file")
}
db = instance
if sql, err := db.DB(); err != nil {
return errors.WithStack(err)
} else {
sql.SetMaxOpenConns(1)
sql.SetConnMaxLifetime(time.Hour)
}
if tx := db.Exec("PRAGMA synchronous = OFF"); tx.Error != nil {
return errors.WithStack(tx.Error)
}
if tx := db.Exec("PRAGMA journal_mode = MEMORY"); tx.Error != nil {
return errors.WithStack(tx.Error)
}
if err := db.AutoMigrate(&models.Activity{}); err != nil {
return errors.WithStack(err)
}
return nil
}
// Instance returns the gorm database instance that was configured when the application was
// booted.
func Instance() *gorm.DB {
if db == nil {
panic("database: attempt to access instance before initialized")
}
return db
}

View File

@ -0,0 +1,71 @@
package models
import (
"net"
"strings"
"time"
"gorm.io/gorm"
)
type Event string
type ActivityMeta map[string]interface{}
// Activity defines an activity log event for a server entity performed by a user. This is
// used for tracking commands, power actions, and SFTP events so that they can be reconciled
// and sent back to the Panel instance to be displayed to the user.
type Activity struct {
ID int `gorm:"primaryKey;not null" json:"-"`
// User is UUID of the user that triggered this event, or an empty string if the event
// cannot be tied to a specific user, in which case we will assume it was the system
// user.
User JsonNullString `gorm:"type:uuid" json:"user"`
// Server is the UUID of the server this event is associated with.
Server string `gorm:"type:uuid;not null" json:"server"`
// Event is a string that describes what occurred, and is used by the Panel instance to
// properly associate this event in the activity logs.
Event Event `gorm:"index;not null" json:"event"`
// Metadata is either a null value, string, or a JSON blob with additional event specific
// metadata that can be provided.
Metadata ActivityMeta `gorm:"serializer:json" json:"metadata"`
// IP is the IP address that triggered this event, or an empty string if it cannot be
// determined properly. This should be the connecting user's IP address, and not the
// internal system IP.
IP string `gorm:"not null" json:"ip"`
Timestamp time.Time `gorm:"not null" json:"timestamp"`
}
// SetUser sets the current user that performed the action. If an empty string is provided
// it is cast into a null value when stored.
func (a Activity) SetUser(u string) *Activity {
var ns JsonNullString
if u == "" {
if err := ns.Scan(nil); err != nil {
panic(err)
}
} else {
if err := ns.Scan(u); err != nil {
panic(err)
}
}
a.User = ns
return &a
}
// BeforeCreate executes before we create any activity entry to ensure the IP address
// is trimmed down to remove any extraneous data, and the timestamp is set to the current
// system time and then stored as UTC.
func (a *Activity) BeforeCreate(_ *gorm.DB) error {
if ip, _, err := net.SplitHostPort(strings.TrimSpace(a.IP)); err == nil {
a.IP = ip
}
if a.Timestamp.IsZero() {
a.Timestamp = time.Now()
}
a.Timestamp = a.Timestamp.UTC()
if a.Metadata == nil {
a.Metadata = ActivityMeta{}
}
return nil
}

32
internal/models/models.go Normal file
View File

@ -0,0 +1,32 @@
package models
import (
"database/sql"
"emperror.dev/errors"
"github.com/goccy/go-json"
)
type JsonNullString struct {
sql.NullString
}
func (v JsonNullString) MarshalJSON() ([]byte, error) {
if v.Valid {
return json.Marshal(v.String)
} else {
return json.Marshal(nil)
}
}
func (v *JsonNullString) UnmarshalJSON(data []byte) error {
var s *string
if err := json.Unmarshal(data, &s); err != nil {
return errors.WithStack(err)
}
if s != nil {
v.String = *s
}
v.Valid = s != nil
return nil
}

View File

@ -0,0 +1,90 @@
package progress
import (
"io"
"strings"
"sync/atomic"
"github.com/pterodactyl/wings/system"
)
// Progress is used to track the progress of any I/O operation that are being
// performed.
type Progress struct {
// written is the total size of the files that have been written to the writer.
written uint64
// Total is the total size of the archive in bytes.
total uint64
// Writer .
Writer io.Writer
}
// NewProgress returns a new progress tracker for the given total size.
func NewProgress(total uint64) *Progress {
return &Progress{total: total}
}
// Written returns the total number of bytes written.
// This function should be used when the progress is tracking data being written.
func (p *Progress) Written() uint64 {
return atomic.LoadUint64(&p.written)
}
// Total returns the total size in bytes.
func (p *Progress) Total() uint64 {
return atomic.LoadUint64(&p.total)
}
// SetTotal sets the total size of the archive in bytes. This function is safe
// to call concurrently and can be used to update the total size if it changes,
// such as when the total size is simultaneously being calculated as data is
// being written through the progress writer.
func (p *Progress) SetTotal(total uint64) {
atomic.StoreUint64(&p.total, total)
}
// Write totals the number of bytes that have been written to the writer.
func (p *Progress) Write(v []byte) (int, error) {
n := len(v)
atomic.AddUint64(&p.written, uint64(n))
if p.Writer != nil {
return p.Writer.Write(v)
}
return n, nil
}
// Progress returns a formatted progress string for the current progress.
func (p *Progress) Progress(width int) string {
// current = 100 (Progress, dynamic)
// total = 1000 (Content-Length, dynamic)
// width = 25 (Number of ticks to display, static)
// widthPercentage = 100 / width (What percentage does each tick represent, static)
//
// percentageDecimal = current / total = 0.1
// percentage = percentageDecimal * 100 = 10%
// ticks = percentage / widthPercentage = 2.5
//
// ticks is a float64, so we cast it to an int which rounds it down to 2.
// Values are cast to floats to prevent integer division.
current := p.Written()
total := p.Total()
// width := is passed as a parameter
widthPercentage := float64(100) / float64(width)
percentageDecimal := float64(current) / float64(total)
percentage := percentageDecimal * 100
ticks := int(percentage / widthPercentage)
// Ensure that we never get a negative number of ticks, this will prevent strings#Repeat
// from panicking. A negative number of ticks is likely to happen when the total size is
// inaccurate, such as when we are going off of rough disk usage calculation.
if ticks < 0 {
ticks = 0
} else if ticks > width {
ticks = width
}
bar := strings.Repeat("=", ticks) + strings.Repeat(" ", width-ticks)
return "[" + bar + "] " + system.FormatBytes(current) + " / " + system.FormatBytes(total)
}

View File

@ -0,0 +1,50 @@
package progress_test
import (
"bytes"
"testing"
"github.com/franela/goblin"
"github.com/pterodactyl/wings/internal/progress"
)
func TestProgress(t *testing.T) {
g := goblin.Goblin(t)
g.Describe("Progress", func() {
g.It("properly initializes", func() {
total := uint64(1000)
p := progress.NewProgress(total)
g.Assert(p).IsNotNil()
g.Assert(p.Total()).Equal(total)
g.Assert(p.Written()).Equal(uint64(0))
})
g.It("increments written when Write is called", func() {
v := []byte("hello")
p := progress.NewProgress(1000)
_, err := p.Write(v)
g.Assert(err).IsNil()
g.Assert(p.Written()).Equal(uint64(len(v)))
})
g.It("renders a progress bar", func() {
v := bytes.Repeat([]byte{' '}, 100)
p := progress.NewProgress(1000)
_, err := p.Write(v)
g.Assert(err).IsNil()
g.Assert(p.Written()).Equal(uint64(len(v)))
g.Assert(p.Progress(25)).Equal("[== ] 100 B / 1000 B")
})
g.It("renders a progress bar when written exceeds total", func() {
v := bytes.Repeat([]byte{' '}, 1001)
p := progress.NewProgress(1000)
_, err := p.Write(v)
g.Assert(err).IsNil()
g.Assert(p.Written()).Equal(uint64(len(v)))
g.Assert(p.Progress(25)).Equal("[=========================] 1001 B / 1000 B")
})
})
}

21
internal/ufs/LICENSE Normal file
View File

@ -0,0 +1,21 @@
MIT License
Copyright (c) 2024 Matthew Penner
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

17
internal/ufs/README.md Normal file
View File

@ -0,0 +1,17 @@
# Filesystem
## Licensing
Most code in this package is licensed under `MIT` with some exceptions.
The following files are licensed under `BSD-3-Clause` due to them being copied
verbatim or derived from [Go](https://go.dev)'s source code.
- [`file_posix.go`](./file_posix.go)
- [`mkdir_unix.go`](./mkdir_unix.go)
- [`path_unix.go`](./path_unix.go)
- [`removeall_unix.go`](./removeall_unix.go)
- [`stat_unix.go`](./stat_unix.go)
- [`walk.go`](./walk.go)
These changes are not associated with nor endorsed by The Go Authors.

12
internal/ufs/doc.go Normal file
View File

@ -0,0 +1,12 @@
// SPDX-License-Identifier: MIT
// SPDX-FileCopyrightText: Copyright (c) 2024 Matthew Penner
// Package ufs provides an abstraction layer for performing I/O on filesystems.
// This package is designed to be used in-place of standard `os` package I/O
// calls, and is not designed to be used as a generic filesystem abstraction
// like the `io/fs` package.
//
// The primary use-case of this package was to provide a "chroot-like" `os`
// wrapper, so we can safely sandbox I/O operations within a directory and
// use untrusted arbitrary paths.
package ufs

120
internal/ufs/error.go Normal file
View File

@ -0,0 +1,120 @@
// SPDX-License-Identifier: MIT
// SPDX-FileCopyrightText: Copyright (c) 2024 Matthew Penner
package ufs
import (
"errors"
iofs "io/fs"
"os"
"golang.org/x/sys/unix"
)
var (
// ErrIsDirectory is an error for when an operation that operates only on
// files is given a path to a directory.
ErrIsDirectory = errors.New("is a directory")
// ErrNotDirectory is an error for when an operation that operates only on
// directories is given a path to a file.
ErrNotDirectory = errors.New("not a directory")
// ErrBadPathResolution is an error for when a sand-boxed filesystem
// resolves a given path to a forbidden location.
ErrBadPathResolution = errors.New("bad path resolution")
// ErrNotRegular is an error for when an operation that operates only on
// regular files is passed something other than a regular file.
ErrNotRegular = errors.New("not a regular file")
// ErrClosed is an error for when an entry was accessed after being closed.
ErrClosed = iofs.ErrClosed
// ErrInvalid is an error for when an invalid argument was used.
ErrInvalid = iofs.ErrInvalid
// ErrExist is an error for when an entry already exists.
ErrExist = iofs.ErrExist
// ErrNotExist is an error for when an entry does not exist.
ErrNotExist = iofs.ErrNotExist
// ErrPermission is an error for when the required permissions to perform an
// operation are missing.
ErrPermission = iofs.ErrPermission
)
// LinkError records an error during a link or symlink or rename
// system call and the paths that caused it.
type LinkError = os.LinkError
// PathError records an error and the operation and file path that caused it.
type PathError = iofs.PathError
// SyscallError records an error from a specific system call.
type SyscallError = os.SyscallError
// NewSyscallError returns, as an error, a new SyscallError
// with the given system call name and error details.
// As a convenience, if err is nil, NewSyscallError returns nil.
func NewSyscallError(syscall string, err error) error {
return os.NewSyscallError(syscall, err)
}
// convertErrorType converts errors into our custom errors to ensure consistent
// error values.
func convertErrorType(err error) error {
if err == nil {
return nil
}
var pErr *PathError
switch {
case errors.As(err, &pErr):
switch {
// File exists
case errors.Is(pErr.Err, unix.EEXIST):
return &PathError{
Op: pErr.Op,
Path: pErr.Path,
Err: ErrExist,
}
// Is a directory
case errors.Is(pErr.Err, unix.EISDIR):
return &PathError{
Op: pErr.Op,
Path: pErr.Path,
Err: ErrIsDirectory,
}
// Not a directory
case errors.Is(pErr.Err, unix.ENOTDIR):
return &PathError{
Op: pErr.Op,
Path: pErr.Path,
Err: ErrNotDirectory,
}
// No such file or directory
case errors.Is(pErr.Err, unix.ENOENT):
return &PathError{
Op: pErr.Op,
Path: pErr.Path,
Err: ErrNotExist,
}
// Operation not permitted
case errors.Is(pErr.Err, unix.EPERM):
return &PathError{
Op: pErr.Op,
Path: pErr.Path,
Err: ErrPermission,
}
// Invalid cross-device link
case errors.Is(pErr.Err, unix.EXDEV):
return &PathError{
Op: pErr.Op,
Path: pErr.Path,
Err: ErrBadPathResolution,
}
// Too many levels of symbolic links
case errors.Is(pErr.Err, unix.ELOOP):
return &PathError{
Op: pErr.Op,
Path: pErr.Path,
Err: ErrBadPathResolution,
}
}
}
return err
}

179
internal/ufs/file.go Normal file
View File

@ -0,0 +1,179 @@
// SPDX-License-Identifier: MIT
// SPDX-FileCopyrightText: Copyright (c) 2024 Matthew Penner
package ufs
import (
"io"
iofs "io/fs"
"golang.org/x/sys/unix"
)
// DirEntry is an entry read from a directory.
type DirEntry = iofs.DirEntry
// File describes readable and/or writable file from a Filesystem.
type File interface {
// Name returns the base name of the file.
Name() string
// Stat returns the FileInfo structure describing the file.
// If there is an error, it will be of type *PathError.
Stat() (FileInfo, error)
// ReadDir reads the contents of the directory associated with the file f
// and returns a slice of DirEntry values in directory order.
// Subsequent calls on the same file will yield later DirEntry records in the directory.
//
// If n > 0, ReadDir returns at most n DirEntry records.
// In this case, if ReadDir returns an empty slice, it will return an error explaining why.
// At the end of a directory, the error is io.EOF.
//
// If n <= 0, ReadDir returns all the DirEntry records remaining in the directory.
// When it succeeds, it returns a nil error (not io.EOF).
ReadDir(n int) ([]DirEntry, error)
// Readdirnames reads the contents of the directory associated with file
// and returns a slice of up to n names of files in the directory,
// in directory order. Subsequent calls on the same file will yield
// further names.
//
// If n > 0, Readdirnames returns at most n names. In this case, if
// Readdirnames returns an empty slice, it will return a non-nil error
// explaining why. At the end of a directory, the error is io.EOF.
//
// If n <= 0, Readdirnames returns all the names from the directory in
// a single slice. In this case, if Readdirnames succeeds (reads all
// the way to the end of the directory), it returns the slice and a
// nil error. If it encounters an error before the end of the
// directory, Readdirnames returns the names read until that point and
// a non-nil error.
Readdirnames(n int) (names []string, err error)
// Fd returns the integer Unix file descriptor referencing the open file.
// If f is closed, the file descriptor becomes invalid.
// If f is garbage collected, a finalizer may close the file descriptor,
// making it invalid; see runtime.SetFinalizer for more information on when
// a finalizer might be run. On Unix systems this will cause the SetDeadline
// methods to stop working.
// Because file descriptors can be reused, the returned file descriptor may
// only be closed through the Close method of f, or by its finalizer during
// garbage collection. Otherwise, during garbage collection the finalizer
// may close an unrelated file descriptor with the same (reused) number.
//
// As an alternative, see the f.SyscallConn method.
Fd() uintptr
// Truncate changes the size of the file.
// It does not change the I/O offset.
// If there is an error, it will be of type *PathError.
Truncate(size int64) error
io.Closer
io.Reader
io.ReaderAt
io.ReaderFrom
io.Writer
io.WriterAt
io.Seeker
}
// FileInfo describes a file and is returned by Stat and Lstat.
type FileInfo = iofs.FileInfo
// FileMode represents a file's mode and permission bits.
// The bits have the same definition on all systems, so that
// information about files can be moved from one system
// to another portably. Not all bits apply to all systems.
// The only required bit is ModeDir for directories.
type FileMode = iofs.FileMode
// The defined file mode bits are the most significant bits of the FileMode.
// The nine least-significant bits are the standard Unix rwxrwxrwx permissions.
// The values of these bits should be considered part of the public API and
// may be used in wire protocols or disk representations: they must not be
// changed, although new bits might be added.
const (
// ModeDir represents a directory.
// d: is a directory
ModeDir = iofs.ModeDir
// ModeAppend represents an append-only file.
// a: append-only
ModeAppend = iofs.ModeAppend
// ModeExclusive represents an exclusive file.
// l: exclusive use
ModeExclusive = iofs.ModeExclusive
// ModeTemporary .
// T: temporary file; Plan 9 only.
ModeTemporary = iofs.ModeTemporary
// ModeSymlink .
// L: symbolic link.
ModeSymlink = iofs.ModeSymlink
// ModeDevice .
// D: device file.
ModeDevice = iofs.ModeDevice
// ModeNamedPipe .
// p: named pipe (FIFO)
ModeNamedPipe = iofs.ModeNamedPipe
// ModeSocket .
// S: Unix domain socket.
ModeSocket = iofs.ModeSocket
// ModeSetuid .
// u: setuid
ModeSetuid = iofs.ModeSetuid
// ModeSetgid .
// g: setgid
ModeSetgid = iofs.ModeSetgid
// ModeCharDevice .
// c: Unix character device, when ModeDevice is set
ModeCharDevice = iofs.ModeCharDevice
// ModeSticky .
// t: sticky
ModeSticky = iofs.ModeSticky
// ModeIrregular .
// ?: non-regular file; nothing else is known about this file.
ModeIrregular = iofs.ModeIrregular
// ModeType .
ModeType = iofs.ModeType
// ModePerm .
// Unix permission bits, 0o777.
ModePerm = iofs.ModePerm
)
const (
// O_RDONLY opens the file read-only.
O_RDONLY = unix.O_RDONLY
// O_WRONLY opens the file write-only.
O_WRONLY = unix.O_WRONLY
// O_RDWR opens the file read-write.
O_RDWR = unix.O_RDWR
// O_APPEND appends data to the file when writing.
O_APPEND = unix.O_APPEND
// O_CREATE creates a new file if it doesn't exist.
O_CREATE = unix.O_CREAT
// O_EXCL is used with O_CREATE, file must not exist.
O_EXCL = unix.O_EXCL
// O_SYNC open for synchronous I/O.
O_SYNC = unix.O_SYNC
// O_TRUNC truncates regular writable file when opened.
O_TRUNC = unix.O_TRUNC
// O_DIRECTORY opens a directory only. If the entry is not a directory an
// error will be returned.
O_DIRECTORY = unix.O_DIRECTORY
// O_NOFOLLOW opens the exact path given without following symlinks.
O_NOFOLLOW = unix.O_NOFOLLOW
O_CLOEXEC = unix.O_CLOEXEC
O_LARGEFILE = unix.O_LARGEFILE
)
const (
AT_SYMLINK_NOFOLLOW = unix.AT_SYMLINK_NOFOLLOW
AT_REMOVEDIR = unix.AT_REMOVEDIR
AT_EMPTY_PATH = unix.AT_EMPTY_PATH
)

View File

@ -0,0 +1,49 @@
// SPDX-License-Identifier: BSD-3-Clause
// Code in this file was copied from `go/src/os/file_posix.go`.
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the `go.LICENSE` file.
//go:build unix || (js && wasm) || wasip1 || windows
package ufs
import (
"golang.org/x/sys/unix"
)
// ignoringEINTR makes a function call and repeats it if it returns an
// EINTR error. This appears to be required even though we install all
// signal handlers with SA_RESTART: see https://go.dev/issue/22838,
// https://go.dev/issue/38033, https://go.dev/issue/38836,
// https://go.dev/issue/40846. Also, https://go.dev/issue/20400 and
// https://go.dev/issue/36644 are issues in which a signal handler is
// installed without setting SA_RESTART. None of these are the common case,
// but there are enough of them that it seems that we can't avoid
// an EINTR loop.
func ignoringEINTR(fn func() error) error {
for {
err := fn()
if err != unix.EINTR {
return err
}
}
}
// syscallMode returns the syscall-specific mode bits from Go's portable mode bits.
func syscallMode(i FileMode) (o FileMode) {
o |= i.Perm()
if i&ModeSetuid != 0 {
o |= unix.S_ISUID
}
if i&ModeSetgid != 0 {
o |= unix.S_ISGID
}
if i&ModeSticky != 0 {
o |= unix.S_ISVTX
}
// No mapping for Go's ModeTemporary (plan9 only).
return
}

168
internal/ufs/filesystem.go Normal file
View File

@ -0,0 +1,168 @@
// SPDX-License-Identifier: MIT
// SPDX-FileCopyrightText: Copyright (c) 2024 Matthew Penner
package ufs
import (
"time"
)
// Filesystem represents a filesystem capable of performing I/O operations.
type Filesystem interface {
// Chmod changes the mode of the named file to mode.
//
// If the file is a symbolic link, it changes the mode of the link's target.
// If there is an error, it will be of type *PathError.
//
// A different subset of the mode bits are used, depending on the
// operating system.
//
// On Unix, the mode's permission bits, ModeSetuid, ModeSetgid, and
// ModeSticky are used.
//
// On Windows, only the 0200 bit (owner writable) of mode is used; it
// controls whether the file's read-only attribute is set or cleared.
// The other bits are currently unused. For compatibility with Go 1.12
// and earlier, use a non-zero mode. Use mode 0400 for a read-only
// file and 0600 for a readable+writable file.
//
// On Plan 9, the mode's permission bits, ModeAppend, ModeExclusive,
// and ModeTemporary are used.
Chmod(name string, mode FileMode) error
// Chown changes the numeric uid and gid of the named file.
//
// If the file is a symbolic link, it changes the uid and gid of the link's target.
// A uid or gid of -1 means to not change that value.
// If there is an error, it will be of type *PathError.
//
// On Windows or Plan 9, Chown always returns the syscall.EWINDOWS or
// EPLAN9 error, wrapped in *PathError.
Chown(name string, uid, gid int) error
// Lchown changes the numeric uid and gid of the named file.
//
// If the file is a symbolic link, it changes the uid and gid of the link itself.
// If there is an error, it will be of type *PathError.
//
// On Windows, it always returns the syscall.EWINDOWS error, wrapped
// in *PathError.
Lchown(name string, uid, gid int) error
// Chtimes changes the access and modification times of the named
// file, similar to the Unix utime() or utimes() functions.
//
// The underlying filesystem may truncate or round the values to a
// less precise time unit.
//
// If there is an error, it will be of type *PathError.
Chtimes(name string, atime, mtime time.Time) error
// Create creates or truncates the named file. If the file already exists,
// it is truncated.
//
// If the file does not exist, it is created with mode 0666
// (before umask). If successful, methods on the returned File can
// be used for I/O; the associated file descriptor has mode O_RDWR.
// If there is an error, it will be of type *PathError.
Create(name string) (File, error)
// Mkdir creates a new directory with the specified name and permission
// bits (before umask).
//
// If there is an error, it will be of type *PathError.
Mkdir(name string, perm FileMode) error
// MkdirAll creates a directory named path, along with any necessary
// parents, and returns nil, or else returns an error.
//
// The permission bits perm (before umask) are used for all
// directories that MkdirAll creates.
// If path is already a directory, MkdirAll does nothing
// and returns nil.
MkdirAll(path string, perm FileMode) error
// Open opens the named file for reading.
//
// If successful, methods on the returned file can be used for reading; the
// associated file descriptor has mode O_RDONLY.
//
// If there is an error, it will be of type *PathError.
Open(name string) (File, error)
// OpenFile is the generalized open call; most users will use Open
// or Create instead. It opens the named file with specified flag
// (O_RDONLY etc.).
//
// If the file does not exist, and the O_CREATE flag
// is passed, it is created with mode perm (before umask). If successful,
// methods on the returned File can be used for I/O.
//
// If there is an error, it will be of type *PathError.
OpenFile(name string, flag int, perm FileMode) (File, error)
// ReadDir reads the named directory,
//
// returning all its directory entries sorted by filename.
// If an error occurs reading the directory, ReadDir returns the entries it
// was able to read before the error, along with the error.
ReadDir(name string) ([]DirEntry, error)
// Remove removes the named file or (empty) directory.
//
// If there is an error, it will be of type *PathError.
Remove(name string) error
// RemoveAll removes path and any children it contains.
//
// It removes everything it can but returns the first error
// it encounters. If the path does not exist, RemoveAll
// returns nil (no error).
//
// If there is an error, it will be of type *PathError.
RemoveAll(path string) error
// Rename renames (moves) oldpath to newpath.
//
// If newpath already exists and is not a directory, Rename replaces it.
// OS-specific restrictions may apply when oldpath and newpath are in different directories.
// Even within the same directory, on non-Unix platforms Rename is not an atomic operation.
//
// If there is an error, it will be of type *LinkError.
Rename(oldname, newname string) error
// Stat returns a FileInfo describing the named file.
//
// If there is an error, it will be of type *PathError.
Stat(name string) (FileInfo, error)
// Lstat returns a FileInfo describing the named file.
//
// If the file is a symbolic link, the returned FileInfo
// describes the symbolic link. Lstat makes no attempt to follow the link.
//
// If there is an error, it will be of type *PathError.
Lstat(name string) (FileInfo, error)
// Symlink creates newname as a symbolic link to oldname.
//
// On Windows, a symlink to a non-existent oldname creates a file symlink;
// if oldname is later created as a directory the symlink will not work.
//
// If there is an error, it will be of type *LinkError.
Symlink(oldname, newname string) error
// WalkDir walks the file tree rooted at root, calling fn for each file or
// directory in the tree, including root.
//
// All errors that arise visiting files and directories are filtered by fn:
// see the [WalkDirFunc] documentation for details.
//
// The files are walked in lexical order, which makes the output deterministic
// but requires WalkDir to read an entire directory into memory before proceeding
// to walk that directory.
//
// WalkDir does not follow symbolic links found in directories,
// but if root itself is a symbolic link, its target will be walked.
WalkDir(root string, fn WalkDirFunc) error
}

159
internal/ufs/fs_quota.go Normal file
View File

@ -0,0 +1,159 @@
// SPDX-License-Identifier: MIT
// SPDX-FileCopyrightText: Copyright (c) 2024 Matthew Penner
package ufs
import (
"sync/atomic"
)
type Quota struct {
// fs is the underlying filesystem that runs the actual I/O operations.
*UnixFS
// limit is the size limit of the filesystem.
//
// limit is atomic to allow the limit to be safely changed after the
// filesystem was created.
//
// A limit of `-1` disables any write operation from being performed.
// A limit of `0` disables any limit checking.
limit atomic.Int64
// usage is the current usage of the filesystem.
//
// If usage is set to `-1`, it hasn't been calculated yet.
usage atomic.Int64
}
func NewQuota(fs *UnixFS, limit int64) *Quota {
qfs := Quota{UnixFS: fs}
qfs.limit.Store(limit)
return &qfs
}
// Close closes the filesystem.
func (fs *Quota) Close() (err error) {
err = fs.UnixFS.Close()
return
}
// Limit returns the limit of the filesystem.
func (fs *Quota) Limit() int64 {
return fs.limit.Load()
}
// SetLimit returns the limit of the filesystem.
func (fs *Quota) SetLimit(newLimit int64) int64 {
return fs.limit.Swap(newLimit)
}
// Usage returns the current usage of the filesystem.
func (fs *Quota) Usage() int64 {
return fs.usage.Load()
}
// SetUsage updates the total usage of the filesystem.
func (fs *Quota) SetUsage(newUsage int64) int64 {
return fs.usage.Swap(newUsage)
}
// Add adds `i` to the tracked usage total.
func (fs *Quota) Add(i int64) int64 {
usage := fs.Usage()
// If adding `i` to the usage will put us below 0, cap it. (`i` can be negative)
if usage+i < 0 {
fs.usage.Store(0)
return 0
}
return fs.usage.Add(i)
}
// CanFit checks if the given size can fit in the filesystem without exceeding
// the limit of the filesystem.
func (fs *Quota) CanFit(size int64) bool {
// Get the size limit of the filesystem.
limit := fs.Limit()
switch limit {
case -1:
// A limit of -1 means no write operations are allowed.
return false
case 0:
// A limit of 0 means unlimited.
return true
}
// Any other limit is a value we need to check.
usage := fs.Usage()
if usage == -1 {
// We don't know what the current usage is yet.
return true
}
// If the current usage + the requested size are under the limit of the
// filesystem, allow it.
if usage+size <= limit {
return true
}
// Welp, the size would exceed the limit of the filesystem, deny it.
return false
}
func (fs *Quota) Remove(name string) error {
// For information on why this interface is used here, check its
// documentation.
s, err := fs.RemoveStat(name)
if err != nil {
return err
}
// Don't reduce the quota's usage as `name` is not a regular file.
if !s.Mode().IsRegular() {
return nil
}
// Remove the size of the deleted file from the quota usage.
fs.Add(-s.Size())
return nil
}
// RemoveAll removes path and any children it contains.
//
// It removes everything it can but returns the first error
// it encounters. If the path does not exist, RemoveAll
// returns nil (no error).
//
// If there is an error, it will be of type *PathError.
func (fs *Quota) RemoveAll(name string) error {
name, err := fs.unsafePath(name)
if err != nil {
return err
}
// While removeAll internally checks this, I want to make sure we check it
// and return the proper error so our tests can ensure that this will never
// be a possibility.
if name == "." {
return &PathError{
Op: "removeall",
Path: name,
Err: ErrBadPathResolution,
}
}
return fs.removeAll(name)
}
func (fs *Quota) removeAll(path string) error {
return removeAll(fs, path)
}
func (fs *Quota) unlinkat(dirfd int, name string, flags int) error {
if flags == 0 {
s, err := fs.Lstatat(dirfd, name)
if err == nil && s.Mode().IsRegular() {
fs.Add(-s.Size())
}
}
return fs.UnixFS.unlinkat(dirfd, name, flags)
}

825
internal/ufs/fs_unix.go Normal file
View File

@ -0,0 +1,825 @@
// SPDX-License-Identifier: MIT
// SPDX-FileCopyrightText: Copyright (c) 2024 Matthew Penner
//go:build unix
package ufs
import (
"errors"
"fmt"
"os"
"path/filepath"
"strconv"
"strings"
"sync/atomic"
"time"
"golang.org/x/sys/unix"
)
// UnixFS is a filesystem that uses the unix package to make io calls.
//
// This is used for proper sand-boxing and full control over the exact syscalls
// being performed.
type UnixFS struct {
// basePath is the base path for file operations to take place in.
basePath string
// dirfd holds the file descriptor of BasePath and is used to ensure
// operations are restricted into descendants of BasePath.
dirfd atomic.Int64
// useOpenat2 controls whether the `openat2` syscall is used instead of the
// older `openat` syscall.
useOpenat2 bool
}
// NewUnixFS creates a new sandboxed unix filesystem. BasePath is used as the
// sandbox path, operations on BasePath itself are not allowed, but any
// operations on its descendants are. Symlinks pointing outside BasePath are
// checked and prevented from enabling an escape in a non-raceable manor.
func NewUnixFS(basePath string, useOpenat2 bool) (*UnixFS, error) {
basePath = strings.TrimSuffix(basePath, "/")
// We don't need Openat2, if we are given a basePath that is already unsafe
// I give up on trying to sandbox it.
dirfd, err := unix.Openat(AT_EMPTY_PATH, basePath, O_DIRECTORY|O_RDONLY, 0)
if err != nil {
return nil, convertErrorType(err)
}
fs := &UnixFS{
basePath: basePath,
useOpenat2: useOpenat2,
}
fs.dirfd.Store(int64(dirfd))
return fs, nil
}
// BasePath returns the base path of the UnixFS sandbox, file operations
// pointing outside this path are prohibited and will be blocked by all
// operations implemented by UnixFS.
func (fs *UnixFS) BasePath() string {
return fs.basePath
}
// Close releases the file descriptor used to sandbox operations within the
// base path of the filesystem.
func (fs *UnixFS) Close() error {
// Once closed, change dirfd to something invalid to detect when it has been
// closed.
defer func() {
fs.dirfd.Store(-1)
}()
return unix.Close(int(fs.dirfd.Load()))
}
// Chmod changes the mode of the named file to mode.
//
// If the file is a symbolic link, it changes the mode of the link's target.
// If there is an error, it will be of type *PathError.
//
// A different subset of the mode bits are used, depending on the
// operating system.
//
// On Unix, the mode's permission bits, ModeSetuid, ModeSetgid, and
// ModeSticky are used.
//
// On Windows, only the 0200 bit (owner writable) of mode is used; it
// controls whether the file's read-only attribute is set or cleared.
// The other bits are currently unused. For compatibility with Go 1.12
// and earlier, use a non-zero mode. Use mode 0400 for a read-only
// file and 0600 for a readable+writable file.
//
// On Plan 9, the mode's permission bits, ModeAppend, ModeExclusive,
// and ModeTemporary are used.
func (fs *UnixFS) Chmod(name string, mode FileMode) error {
dirfd, name, closeFd, err := fs.safePath(name)
defer closeFd()
if err != nil {
return err
}
return convertErrorType(unix.Fchmodat(dirfd, name, uint32(mode), 0))
}
// Chown changes the numeric uid and gid of the named file.
//
// If the file is a symbolic link, it changes the uid and gid of the link's target.
// A uid or gid of -1 means to not change that value.
// If there is an error, it will be of type *PathError.
//
// On Windows or Plan 9, Chown always returns the syscall.EWINDOWS or
// EPLAN9 error, wrapped in *PathError.
func (fs *UnixFS) Chown(name string, uid, gid int) error {
return fs.fchown(name, uid, gid, 0)
}
// Lchown changes the numeric uid and gid of the named file.
//
// If the file is a symbolic link, it changes the uid and gid of the link itself.
// If there is an error, it will be of type *PathError.
//
// On Windows, it always returns the syscall.EWINDOWS error, wrapped
// in *PathError.
func (fs *UnixFS) Lchown(name string, uid, gid int) error {
// With AT_SYMLINK_NOFOLLOW, Fchownat acts like Lchown but allows us to
// pass a dirfd.
return fs.fchown(name, uid, gid, AT_SYMLINK_NOFOLLOW)
}
// fchown is a re-usable Fchownat syscall used by Chown and Lchown.
func (fs *UnixFS) fchown(name string, uid, gid, flags int) error {
dirfd, name, closeFd, err := fs.safePath(name)
defer closeFd()
if err != nil {
return err
}
return convertErrorType(unix.Fchownat(dirfd, name, uid, gid, flags))
}
// Chownat is like Chown but allows passing an existing directory file
// descriptor rather than needing to resolve one.
func (fs *UnixFS) Chownat(dirfd int, name string, uid, gid int) error {
return convertErrorType(unix.Fchownat(dirfd, name, uid, gid, 0))
}
// Lchownat is like Lchown but allows passing an existing directory file
// descriptor rather than needing to resolve one.
func (fs *UnixFS) Lchownat(dirfd int, name string, uid, gid int) error {
return convertErrorType(unix.Fchownat(dirfd, name, uid, gid, AT_SYMLINK_NOFOLLOW))
}
// Chtimes changes the access and modification times of the named
// file, similar to the Unix utime() or utimes() functions.
//
// The underlying filesystem may truncate or round the values to a
// less precise time unit.
//
// If there is an error, it will be of type *PathError.
func (fs *UnixFS) Chtimes(name string, atime, mtime time.Time) error {
dirfd, name, closeFd, err := fs.safePath(name)
defer closeFd()
if err != nil {
return err
}
return fs.Chtimesat(dirfd, name, atime, mtime)
}
// Chtimesat is like Chtimes but allows passing an existing directory file
// descriptor rather than needing to resolve one.
func (fs *UnixFS) Chtimesat(dirfd int, name string, atime, mtime time.Time) error {
var utimes [2]unix.Timespec
set := func(i int, t time.Time) {
if t.IsZero() {
utimes[i] = unix.Timespec{Sec: unix.UTIME_OMIT, Nsec: unix.UTIME_OMIT}
} else {
utimes[i] = unix.NsecToTimespec(t.UnixNano())
}
}
set(0, atime)
set(1, mtime)
// This does support `AT_SYMLINK_NOFOLLOW` as well if needed.
if err := unix.UtimesNanoAt(dirfd, name, utimes[0:], 0); err != nil {
return convertErrorType(&PathError{Op: "chtimes", Path: name, Err: err})
}
return nil
}
// Create creates or truncates the named file. If the file already exists,
// it is truncated.
//
// If the file does not exist, it is created with mode 0666
// (before umask). If successful, methods on the returned File can
// be used for I/O; the associated file descriptor has mode O_RDWR.
// If there is an error, it will be of type *PathError.
func (fs *UnixFS) Create(name string) (File, error) {
return fs.OpenFile(name, O_CREATE|O_WRONLY|O_TRUNC, 0o644)
}
// Mkdir creates a new directory with the specified name and permission
// bits (before umask).
//
// If there is an error, it will be of type *PathError.
func (fs *UnixFS) Mkdir(name string, mode FileMode) error {
dirfd, name, closeFd, err := fs.safePath(name)
defer closeFd()
if err != nil {
return err
}
return fs.Mkdirat(dirfd, name, mode)
}
func (fs *UnixFS) Mkdirat(dirfd int, name string, mode FileMode) error {
return convertErrorType(unix.Mkdirat(dirfd, name, uint32(mode)))
}
// MkdirAll creates a directory named path, along with any necessary
// parents, and returns nil, or else returns an error.
//
// The permission bits perm (before umask) are used for all
// directories that MkdirAll creates.
// If path is already a directory, MkdirAll does nothing
// and returns nil.
func (fs *UnixFS) MkdirAll(name string, mode FileMode) error {
// Ensure name is somewhat clean before continuing.
name, err := fs.unsafePath(name)
if err != nil {
return err
}
return fs.mkdirAll(name, mode)
}
// Open opens the named file for reading.
//
// If successful, methods on the returned file can be used for reading; the
// associated file descriptor has mode O_RDONLY.
//
// If there is an error, it will be of type *PathError.
func (fs *UnixFS) Open(name string) (File, error) {
return fs.OpenFile(name, O_RDONLY, 0)
}
// OpenFile is the generalized open call; most users will use Open
// or Create instead. It opens the named file with specified flag
// (O_RDONLY etc.).
//
// If the file does not exist, and the O_CREATE flag
// is passed, it is created with mode perm (before umask). If successful,
// methods on the returned File can be used for I/O.
//
// If there is an error, it will be of type *PathError.
func (fs *UnixFS) OpenFile(name string, flag int, mode FileMode) (File, error) {
fd, err := fs.openFile(name, flag, mode)
if err != nil {
return nil, err
}
// Do not close `fd` here, it is passed to a file that needs the fd, the
// caller of this function is responsible for calling Close() on the File
// to release the file descriptor.
return os.NewFile(uintptr(fd), name), nil
}
func (fs *UnixFS) openFile(name string, flag int, mode FileMode) (int, error) {
dirfd, name, closeFd, err := fs.safePath(name)
defer closeFd()
if err != nil {
return 0, err
}
return fs.openat(dirfd, name, flag, mode)
}
func (fs *UnixFS) OpenFileat(dirfd int, name string, flag int, mode FileMode) (File, error) {
fd, err := fs.openat(dirfd, name, flag, mode)
if err != nil {
return nil, err
}
// Do not close `fd` here, it is passed to a file that needs the fd, the
// caller of this function is responsible for calling Close() on the File
// to release the file descriptor.
return os.NewFile(uintptr(fd), name), nil
}
// ReadDir reads the named directory,
//
// returning all its directory entries sorted by filename.
// If an error occurs reading the directory, ReadDir returns the entries it
// was able to read before the error, along with the error.
func (fs *UnixFS) ReadDir(path string) ([]DirEntry, error) {
dirfd, name, closeFd, err := fs.safePath(path)
defer closeFd()
if err != nil {
return nil, err
}
fd, err := fs.openat(dirfd, name, O_DIRECTORY|O_RDONLY, 0)
if err != nil {
return nil, err
}
defer unix.Close(fd)
return fs.readDir(fd, name, nil)
}
// RemoveStat is a combination of Stat and Remove, it is used to more
// efficiently remove a file when the caller needs to stat it before
// removing it.
//
// This optimized function exists for our QuotaFS abstraction, which needs
// to track writes to a filesystem. When removing a file, the QuotaFS needs
// to know if the entry is a file and if so, how large it is. Because we
// need to Stat a file in order to get its mode and size, we will already
// know if the entry needs to be removed by using Unlink or Rmdir. The
// standard `Remove` method just tries both Unlink and Rmdir (in that order)
// as it ends up usually being faster and more efficient than calling Stat +
// the proper operation in the first place.
func (fs *UnixFS) RemoveStat(name string) (FileInfo, error) {
dirfd, name, closeFd, err := fs.safePath(name)
defer closeFd()
if err != nil {
return nil, err
}
// Lstat name, we use Lstat as Unlink doesn't care about symlinks.
s, err := fs.Lstatat(dirfd, name)
if err != nil {
return nil, err
}
if s.IsDir() {
err = fs.unlinkat(dirfd, name, AT_REMOVEDIR) // Rmdir
} else {
err = fs.unlinkat(dirfd, name, 0)
}
if err != nil {
return s, convertErrorType(&PathError{Op: "remove", Path: name, Err: err})
}
return s, nil
}
// Remove removes the named file or (empty) directory.
//
// If there is an error, it will be of type *PathError.
func (fs *UnixFS) Remove(name string) error {
dirfd, name, closeFd, err := fs.safePath(name)
defer closeFd()
if err != nil {
return err
}
// Prevent trying to Remove the base directory.
if name == "." {
return &PathError{
Op: "remove",
Path: name,
Err: ErrBadPathResolution,
}
}
// System call interface forces us to know
// whether name is a file or directory.
// Try both: it is cheaper on average than
// doing a Stat plus the right one.
err = fs.unlinkat(dirfd, name, 0)
if err == nil {
return nil
}
err1 := fs.unlinkat(dirfd, name, AT_REMOVEDIR) // Rmdir
if err1 == nil {
return nil
}
// Both failed: figure out which error to return.
// OS X and Linux differ on whether unlink(dir)
// returns EISDIR, so can't use that. However,
// both agree that rmdir(file) returns ENOTDIR,
// so we can use that to decide which error is real.
// Rmdir might also return ENOTDIR if given a bad
// file path, like /etc/passwd/foo, but in that case,
// both errors will be ENOTDIR, so it's okay to
// use the error from unlink.
if err1 != unix.ENOTDIR {
err = err1
}
return convertErrorType(&PathError{Op: "remove", Path: name, Err: err})
}
// RemoveAll removes path and any children it contains.
//
// It removes everything it can but returns the first error
// it encounters. If the path does not exist, RemoveAll
// returns nil (no error).
//
// If there is an error, it will be of type *PathError.
func (fs *UnixFS) RemoveAll(name string) error {
name, err := fs.unsafePath(name)
if err != nil {
return err
}
// While removeAll internally checks this, I want to make sure we check it
// and return the proper error so our tests can ensure that this will never
// be a possibility.
if name == "." {
return &PathError{
Op: "removeall",
Path: name,
Err: ErrBadPathResolution,
}
}
return fs.removeAll(name)
}
func (fs *UnixFS) unlinkat(dirfd int, name string, flags int) error {
return ignoringEINTR(func() error {
return unix.Unlinkat(dirfd, name, flags)
})
}
// Rename renames (moves) oldpath to newpath.
//
// If newpath already exists and is not a directory, Rename replaces it.
// OS-specific restrictions may apply when oldpath and newpath are in different directories.
// Even within the same directory, on non-Unix platforms Rename is not an atomic operation.
//
// If there is an error, it will be of type *LinkError.
func (fs *UnixFS) Rename(oldpath, newpath string) error {
// Simple case: both paths are the same.
if oldpath == newpath {
return nil
}
olddirfd, oldname, closeFd, err := fs.safePath(oldpath)
defer closeFd()
if err != nil {
return err
}
// Ensure that we are not trying to rename the base directory itself.
// While unix.Renameat ends up throwing a "device or resource busy" error,
// that doesn't mean we are protecting the system properly.
if oldname == "." {
return convertErrorType(&PathError{
Op: "rename",
Path: oldname,
Err: ErrBadPathResolution,
})
}
// Stat the old target to return proper errors.
if _, err := fs.Lstatat(olddirfd, oldname); err != nil {
return err
}
newdirfd, newname, closeFd2, err := fs.safePath(newpath)
if err != nil {
closeFd2()
if !errors.Is(err, ErrNotExist) {
return convertErrorType(err)
}
var pathErr *PathError
if !errors.As(err, &pathErr) {
return convertErrorType(err)
}
if err := fs.MkdirAll(pathErr.Path, 0o755); err != nil {
return err
}
newdirfd, newname, closeFd2, err = fs.safePath(newpath)
defer closeFd2()
if err != nil {
return err
}
} else {
defer closeFd2()
}
// Ensure that we are not trying to rename the base directory itself.
// While unix.Renameat ends up throwing a "device or resource busy" error,
// that doesn't mean we are protecting the system properly.
if newname == "." {
return convertErrorType(&PathError{
Op: "rename",
Path: newname,
Err: ErrBadPathResolution,
})
}
// Stat the new target to return proper errors.
_, err = fs.Lstatat(newdirfd, newname)
switch {
case err == nil:
return convertErrorType(&PathError{
Op: "rename",
Path: newname,
Err: ErrExist,
})
case !errors.Is(err, ErrNotExist):
return err
}
return unix.Renameat(olddirfd, oldname, newdirfd, newname)
}
// Stat returns a FileInfo describing the named file.
//
// If there is an error, it will be of type *PathError.
func (fs *UnixFS) Stat(name string) (FileInfo, error) {
return fs.fstat(name, 0)
}
// Statat is like Stat but allows passing an existing directory file
// descriptor rather than needing to resolve one.
func (fs *UnixFS) Statat(dirfd int, name string) (FileInfo, error) {
return fs.fstatat(dirfd, name, 0)
}
// Lstat returns a FileInfo describing the named file.
//
// If the file is a symbolic link, the returned FileInfo
// describes the symbolic link. Lstat makes no attempt to follow the link.
//
// If there is an error, it will be of type *PathError.
func (fs *UnixFS) Lstat(name string) (FileInfo, error) {
return fs.fstat(name, AT_SYMLINK_NOFOLLOW)
}
// Lstatat is like Lstat but allows passing an existing directory file
// descriptor rather than needing to resolve one.
func (fs *UnixFS) Lstatat(dirfd int, name string) (FileInfo, error) {
return fs.fstatat(dirfd, name, AT_SYMLINK_NOFOLLOW)
}
func (fs *UnixFS) fstat(name string, flags int) (FileInfo, error) {
dirfd, name, closeFd, err := fs.safePath(name)
defer closeFd()
if err != nil {
return nil, err
}
return fs.fstatat(dirfd, name, flags)
}
func (fs *UnixFS) fstatat(dirfd int, name string, flags int) (FileInfo, error) {
var s fileStat
if err := ignoringEINTR(func() error {
return unix.Fstatat(dirfd, name, &s.sys, flags)
}); err != nil {
return nil, &PathError{Op: "stat", Path: name, Err: err}
}
fillFileStatFromSys(&s, name)
return &s, nil
}
// Symlink creates newname as a symbolic link to oldname.
//
// On Windows, a symlink to a non-existent oldname creates a file symlink;
// if oldname is later created as a directory the symlink will not work.
//
// If there is an error, it will be of type *LinkError.
func (fs *UnixFS) Symlink(oldpath, newpath string) error {
dirfd, newpath, closeFd, err := fs.safePath(newpath)
defer closeFd()
if err != nil {
return err
}
if err := ignoringEINTR(func() error {
// We aren't concerned with oldpath here as a symlink can point anywhere
// it wants.
return unix.Symlinkat(oldpath, dirfd, newpath)
}); err != nil {
return &LinkError{Op: "symlink", Old: oldpath, New: newpath, Err: err}
}
return nil
}
// Touch will attempt to open a file for reading and/or writing. If the file
// does not exist it will be created, and any missing parent directories will
// also be created. The opened file may be truncated, only if `flag` has
// O_TRUNC set.
func (fs *UnixFS) Touch(path string, flag int, mode FileMode) (File, error) {
if flag&O_CREATE == 0 {
flag |= O_CREATE
}
dirfd, name, closeFd, err := fs.safePath(path)
defer closeFd()
if err == nil {
return fs.OpenFileat(dirfd, name, flag, mode)
}
if !errors.Is(err, ErrNotExist) {
return nil, err
}
var pathErr *PathError
if !errors.As(err, &pathErr) {
return nil, err
}
if err := fs.MkdirAll(pathErr.Path, 0o755); err != nil {
return nil, err
}
// Try to open the file one more time after creating its parent directories.
return fs.OpenFile(path, flag, mode)
}
// WalkDir walks the file tree rooted at root, calling fn for each file or
// directory in the tree, including root.
//
// All errors that arise visiting files and directories are filtered by fn:
// see the [WalkDirFunc] documentation for details.
//
// The files are walked in lexical order, which makes the output deterministic
// but requires WalkDir to read an entire directory into memory before proceeding
// to walk that directory.
//
// WalkDir does not follow symbolic links found in directories,
// but if root itself is a symbolic link, its target will be walked.
func (fs *UnixFS) WalkDir(root string, fn WalkDirFunc) error {
return WalkDir(fs, root, fn)
}
// openat is a wrapper around both unix.Openat and unix.Openat2. If the UnixFS
// was configured to enable openat2 support, unix.Openat2 will be used instead
// of unix.Openat due to having better security properties for our use-case.
func (fs *UnixFS) openat(dirfd int, name string, flag int, mode FileMode) (int, error) {
if flag&O_NOFOLLOW == 0 {
flag |= O_NOFOLLOW
}
var fd int
for {
var err error
if fs.useOpenat2 {
fd, err = fs._openat2(dirfd, name, uint64(flag), uint64(syscallMode(mode)))
} else {
fd, err = fs._openat(dirfd, name, flag, uint32(syscallMode(mode)))
}
if err == nil {
break
}
// We have to check EINTR here, per issues https://go.dev/issue/11180 and https://go.dev/issue/39237.
if err == unix.EINTR {
continue
}
return 0, convertErrorType(err)
}
// If we are not using openat2, do additional path checking. This assumes
// that openat2 is using `RESOLVE_BENEATH` to avoid the same security
// issue.
if !fs.useOpenat2 {
var finalPath string
finalPath, err := filepath.EvalSymlinks(filepath.Join("/proc/self/fd/", strconv.Itoa(dirfd)))
if err != nil {
return fd, convertErrorType(err)
}
if err != nil {
if !errors.Is(err, ErrNotExist) {
return fd, fmt.Errorf("failed to evaluate symlink: %w", convertErrorType(err))
}
// The target of one of the symlinks (EvalSymlinks is recursive)
// does not exist. So get the path that does not exist and use
// that for further validation instead.
var pErr *PathError
if ok := errors.As(err, &pErr); !ok {
return fd, fmt.Errorf("failed to evaluate symlink: %w", convertErrorType(err))
}
finalPath = pErr.Path
}
// Check if the path is within our root.
if !fs.unsafeIsPathInsideOfBase(finalPath) {
return fd, convertErrorType(&PathError{
Op: "openat",
Path: name,
Err: ErrBadPathResolution,
})
}
}
return fd, nil
}
// _openat is a wrapper around unix.Openat. This method should never be directly
// called, use `openat` instead.
func (fs *UnixFS) _openat(dirfd int, name string, flag int, mode uint32) (int, error) {
// Ensure the O_CLOEXEC flag is set.
// Go sets this in the os package, but since we are directly using unix
// we need to set it ourselves.
if flag&O_CLOEXEC == 0 {
flag |= O_CLOEXEC
}
// O_LARGEFILE is set by Openat for us automatically.
fd, err := unix.Openat(dirfd, name, flag, mode)
switch {
case err == nil:
return fd, nil
case err == unix.EINTR:
return 0, err
case err == unix.EAGAIN:
return 0, err
default:
return 0, &PathError{Op: "openat", Path: name, Err: err}
}
}
// _openat2 is a wonderful syscall that supersedes the `openat` syscall. It has
// improved validation and security characteristics that weren't available or
// considered when `openat` was originally implemented. As such, it is only
// present in Kernel 5.6 and above.
//
// This method should never be directly called, use `openat` instead.
func (fs *UnixFS) _openat2(dirfd int, name string, flag uint64, mode uint64) (int, error) {
// Ensure the O_CLOEXEC flag is set.
// Go sets this when using the os package, but since we are directly using
// the unix package we need to set it ourselves.
if flag&O_CLOEXEC == 0 {
flag |= O_CLOEXEC
}
// Ensure the O_LARGEFILE flag is set.
// Go sets this for unix.Open, unix.Openat, but not unix.Openat2.
if flag&O_LARGEFILE == 0 {
flag |= O_LARGEFILE
}
fd, err := unix.Openat2(dirfd, name, &unix.OpenHow{
Flags: flag,
Mode: mode,
// This is the bread and butter of preventing a symlink escape, without
// this option, we have to handle path validation fully on our own.
//
// This is why using Openat2 over Openat is preferred if available.
Resolve: unix.RESOLVE_BENEATH,
})
switch {
case err == nil:
return fd, nil
case err == unix.EINTR:
return 0, err
case err == unix.EAGAIN:
return 0, err
default:
return 0, &PathError{Op: "openat2", Path: name, Err: err}
}
}
func (fs *UnixFS) SafePath(path string) (int, string, func(), error) {
return fs.safePath(path)
}
func (fs *UnixFS) safePath(path string) (dirfd int, file string, closeFd func(), err error) {
// Default closeFd to a NO-OP.
closeFd = func() {}
// Use unsafePath to clean the path and strip BasePath if path is absolute.
var name string
name, err = fs.unsafePath(path)
if err != nil {
return
}
// Check if dirfd was closed, this will happen if (*UnixFS).Close()
// was called.
fsDirfd := int(fs.dirfd.Load())
if fsDirfd == -1 {
err = ErrClosed
return
}
// Split the parent from the last element in the path, this gives us the
// "file name" and the full path to its parent.
var dir string
dir, file = filepath.Split(name)
// If dir is empty then name is not nested.
if dir == "" {
// We don't need to set closeFd here as it will default to a NO-OP and
// `fs.dirfd` is re-used until the filesystem is no-longer needed.
dirfd = fsDirfd
// Return dirfd, name, an empty closeFd func, and no error
return
}
// Dir will usually contain a trailing slash as filepath.Split doesn't
// trim slashes.
dir = strings.TrimSuffix(dir, "/")
dirfd, err = fs.openat(fsDirfd, dir, O_DIRECTORY|O_RDONLY, 0)
if dirfd != 0 {
// Set closeFd to close the newly opened directory file descriptor.
closeFd = func() { _ = unix.Close(dirfd) }
}
// Return dirfd, name, the closeFd func, and err
return
}
// unsafePath prefixes the given path and prefixes it with the filesystem's
// base path, cleaning the result. The path returned by this function may not
// be inside the filesystem's base path, additional checks are required to
// safely use paths returned by this function.
func (fs *UnixFS) unsafePath(path string) (string, error) {
// Calling filepath.Clean on the joined directory will resolve it to the
// absolute path, removing any ../ type of resolution arguments, and leaving
// us with a direct path link.
//
// This will also trim the existing root path off the beginning of the path
// passed to the function since that can get a bit messy.
r := filepath.Clean(filepath.Join(fs.basePath, strings.TrimPrefix(path, fs.basePath)))
if fs.unsafeIsPathInsideOfBase(r) {
// This is kinda ironic isn't it.
// We do this as we are operating with dirfds and `*at` syscalls which
// behave differently if given an absolute path.
//
// First trim the BasePath, then trim any leading slashes.
r = strings.TrimPrefix(strings.TrimPrefix(r, fs.basePath), "/")
// If the path is empty then return "." as the path is pointing to the
// root.
if r == "" {
return ".", nil
}
return r, nil
}
return "", &PathError{
Op: "safePath",
Path: path,
Err: ErrBadPathResolution,
}
}
// unsafeIsPathInsideOfBase checks if the given path is inside the filesystem's
// base path.
func (fs *UnixFS) unsafeIsPathInsideOfBase(path string) bool {
return strings.HasPrefix(
strings.TrimSuffix(path, "/")+"/",
fs.basePath+"/",
)
}

View File

@ -0,0 +1,255 @@
// SPDX-License-Identifier: MIT
// SPDX-FileCopyrightText: Copyright (c) 2024 Matthew Penner
//go:build unix
package ufs_test
import (
"errors"
"os"
"path/filepath"
"testing"
"github.com/pterodactyl/wings/internal/ufs"
)
type testUnixFS struct {
*ufs.UnixFS
TmpDir string
Root string
}
func (fs *testUnixFS) Cleanup() {
_ = fs.Close()
_ = os.RemoveAll(fs.TmpDir)
}
func newTestUnixFS() (*testUnixFS, error) {
tmpDir, err := os.MkdirTemp(os.TempDir(), "ufs")
if err != nil {
return nil, err
}
root := filepath.Join(tmpDir, "root")
if err := os.Mkdir(root, 0o755); err != nil {
return nil, err
}
// TODO: test both disabled and enabled.
fs, err := ufs.NewUnixFS(root, false)
if err != nil {
return nil, err
}
tfs := &testUnixFS{
UnixFS: fs,
TmpDir: tmpDir,
Root: root,
}
return tfs, nil
}
func TestUnixFS_Remove(t *testing.T) {
t.Parallel()
fs, err := newTestUnixFS()
if err != nil {
t.Fatal(err)
return
}
defer fs.Cleanup()
t.Run("base directory", func(t *testing.T) {
// Try to remove the base directory.
if err := fs.Remove(""); !errors.Is(err, ufs.ErrBadPathResolution) {
t.Errorf("expected an a bad path resolution error, but got: %v", err)
return
}
})
t.Run("path traversal", func(t *testing.T) {
// Try to remove the base directory.
if err := fs.RemoveAll("../root"); !errors.Is(err, ufs.ErrBadPathResolution) {
t.Errorf("expected an a bad path resolution error, but got: %v", err)
return
}
})
}
func TestUnixFS_RemoveAll(t *testing.T) {
t.Parallel()
fs, err := newTestUnixFS()
if err != nil {
t.Fatal(err)
return
}
defer fs.Cleanup()
t.Run("base directory", func(t *testing.T) {
// Try to remove the base directory.
if err := fs.RemoveAll(""); !errors.Is(err, ufs.ErrBadPathResolution) {
t.Errorf("expected an a bad path resolution error, but got: %v", err)
return
}
})
t.Run("path traversal", func(t *testing.T) {
// Try to remove the base directory.
if err := fs.RemoveAll("../root"); !errors.Is(err, ufs.ErrBadPathResolution) {
t.Errorf("expected an a bad path resolution error, but got: %v", err)
return
}
})
}
func TestUnixFS_Rename(t *testing.T) {
t.Parallel()
fs, err := newTestUnixFS()
if err != nil {
t.Fatal(err)
return
}
defer fs.Cleanup()
t.Run("rename base directory", func(t *testing.T) {
// Try to rename the base directory.
if err := fs.Rename("", "yeet"); !errors.Is(err, ufs.ErrBadPathResolution) {
t.Errorf("expected an a bad path resolution error, but got: %v", err)
return
}
})
t.Run("rename over base directory", func(t *testing.T) {
// Create a directory that we are going to try and move over top of the
// existing base directory.
if err := fs.Mkdir("overwrite_dir", 0o755); err != nil {
t.Error(err)
return
}
// Try to rename over the base directory.
if err := fs.Rename("overwrite_dir", ""); !errors.Is(err, ufs.ErrBadPathResolution) {
t.Errorf("expected an a bad path resolution error, but got: %v", err)
return
}
})
t.Run("directory rename", func(t *testing.T) {
// Create a directory to rename to something else.
if err := fs.Mkdir("test_directory", 0o755); err != nil {
t.Error(err)
return
}
// Try to rename "test_directory" to "directory".
if err := fs.Rename("test_directory", "directory"); err != nil {
t.Errorf("expected no error, but got: %v", err)
return
}
// Sanity check
if _, err := os.Lstat(filepath.Join(fs.Root, "directory")); err != nil {
t.Errorf("Lstat errored when performing sanity check: %v", err)
return
}
})
t.Run("file rename", func(t *testing.T) {
// Create a directory to rename to something else.
if f, err := fs.Create("test_file"); err != nil {
t.Error(err)
return
} else {
_ = f.Close()
}
// Try to rename "test_file" to "file".
if err := fs.Rename("test_file", "file"); err != nil {
t.Errorf("expected no error, but got: %v", err)
return
}
// Sanity check
if _, err := os.Lstat(filepath.Join(fs.Root, "file")); err != nil {
t.Errorf("Lstat errored when performing sanity check: %v", err)
return
}
})
}
func TestUnixFS_Touch(t *testing.T) {
t.Parallel()
fs, err := newTestUnixFS()
if err != nil {
t.Fatal(err)
return
}
defer fs.Cleanup()
t.Run("base directory", func(t *testing.T) {
path := "i_touched_a_file"
f, err := fs.Touch(path, ufs.O_RDWR, 0o644)
if err != nil {
t.Error(err)
return
}
_ = f.Close()
// Sanity check
if _, err := os.Lstat(filepath.Join(fs.Root, path)); err != nil {
t.Errorf("Lstat errored when performing sanity check: %v", err)
return
}
})
t.Run("existing parent directory", func(t *testing.T) {
dir := "some_parent_directory"
if err := fs.Mkdir(dir, 0o755); err != nil {
t.Errorf("error creating parent directory: %v", err)
return
}
path := filepath.Join(dir, "i_touched_a_file")
f, err := fs.Touch(path, ufs.O_RDWR, 0o644)
if err != nil {
t.Errorf("error touching file: %v", err)
return
}
_ = f.Close()
// Sanity check
if _, err := os.Lstat(filepath.Join(fs.Root, path)); err != nil {
t.Errorf("Lstat errored when performing sanity check: %v", err)
return
}
})
t.Run("non-existent parent directory", func(t *testing.T) {
path := "some_other_directory/i_touched_a_file"
f, err := fs.Touch(path, ufs.O_RDWR, 0o644)
if err != nil {
t.Errorf("error touching file: %v", err)
return
}
_ = f.Close()
// Sanity check
if _, err := os.Lstat(filepath.Join(fs.Root, path)); err != nil {
t.Errorf("Lstat errored when performing sanity check: %v", err)
return
}
})
t.Run("non-existent parent directories", func(t *testing.T) {
path := "some_other_directory/some_directory/i_touched_a_file"
f, err := fs.Touch(path, ufs.O_RDWR, 0o644)
if err != nil {
t.Errorf("error touching file: %v", err)
return
}
_ = f.Close()
// Sanity check
if _, err := os.Lstat(filepath.Join(fs.Root, path)); err != nil {
t.Errorf("Lstat errored when performing sanity check: %v", err)
return
}
})
}

27
internal/ufs/go.LICENSE Normal file
View File

@ -0,0 +1,27 @@
Copyright (c) 2009 The Go Authors. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

View File

@ -0,0 +1,67 @@
// SPDX-License-Identifier: BSD-3-Clause
// Code in this file was derived from `go/src/os/path.go`.
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the `go.LICENSE` file.
//go:build unix
package ufs
import (
"golang.org/x/sys/unix"
)
// mkdirAll is a recursive Mkdir implementation that properly handles symlinks.
func (fs *UnixFS) mkdirAll(name string, mode FileMode) error {
// Fast path: if we can tell whether path is a directory or file, stop with success or error.
dir, err := fs.Lstat(name)
if err == nil {
if dir.Mode()&ModeSymlink != 0 {
// If the final path is a symlink, resolve its target and use that
// to check instead.
dir, err = fs.Stat(name)
if err != nil {
return err
}
}
if dir.IsDir() {
return nil
}
return convertErrorType(&PathError{Op: "mkdir", Path: name, Err: unix.ENOTDIR})
}
// Slow path: make sure parent exists and then call Mkdir for path.
i := len(name)
for i > 0 && name[i-1] == '/' { // Skip trailing path separator.
i--
}
j := i
for j > 0 && name[j-1] != '/' { // Scan backward over element.
j--
}
if j > 1 {
// Create parent.
err = fs.mkdirAll(name[:j-1], mode)
if err != nil {
return err
}
}
// Parent now exists; invoke Mkdir and use its result.
err = fs.Mkdir(name, mode)
if err != nil {
// Handle arguments like "foo/." by
// double-checking that directory doesn't exist.
dir, err1 := fs.Lstat(name)
if err1 == nil && dir.IsDir() {
return nil
}
return err
}
return nil
}

80
internal/ufs/path_unix.go Normal file
View File

@ -0,0 +1,80 @@
// SPDX-License-Identifier: BSD-3-Clause
// Code in this file was copied from `go/src/os/path.go`
// and `go/src/os/path_unix.go`.
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the `go.LICENSE` file.
//go:build unix
package ufs
import (
"os"
)
// basename removes trailing slashes and the leading directory name from path name.
func basename(name string) string {
i := len(name) - 1
// Remove trailing slashes
for ; i > 0 && name[i] == '/'; i-- {
name = name[:i]
}
// Remove leading directory name
for i--; i >= 0; i-- {
if name[i] == '/' {
name = name[i+1:]
break
}
}
return name
}
// endsWithDot reports whether the final component of path is ".".
func endsWithDot(path string) bool {
if path == "." {
return true
}
if len(path) >= 2 && path[len(path)-1] == '.' && os.IsPathSeparator(path[len(path)-2]) {
return true
}
return false
}
// splitPath returns the base name and parent directory.
func splitPath(path string) (string, string) {
// if no better parent is found, the path is relative from "here"
dirname := "."
// Remove all but one leading slash.
for len(path) > 1 && path[0] == '/' && path[1] == '/' {
path = path[1:]
}
i := len(path) - 1
// Remove trailing slashes.
for ; i > 0 && path[i] == '/'; i-- {
path = path[:i]
}
// if no slashes in path, base is path
basename := path
// Remove leading directory path
for i--; i >= 0; i-- {
if path[i] == '/' {
if i == 0 {
dirname = path[:1]
} else {
dirname = path[:i]
}
basename = path[i+1:]
break
}
}
return dirname, basename
}

View File

@ -0,0 +1,117 @@
// SPDX-License-Identifier: MIT
// SPDX-FileCopyrightText: Copyright (c) 2024 Matthew Penner
package ufs
import (
"errors"
"io"
"sync/atomic"
)
// CountedWriter is a writer that counts the amount of data written to the
// underlying writer.
type CountedWriter struct {
File
counter atomic.Int64
err error
}
// NewCountedWriter returns a new countedWriter that counts the amount of bytes
// written to the underlying writer.
func NewCountedWriter(f File) *CountedWriter {
return &CountedWriter{File: f}
}
// BytesWritten returns the amount of bytes that have been written to the
// underlying writer.
func (w *CountedWriter) BytesWritten() int64 {
return w.counter.Load()
}
// Error returns the error from the writer if any. If the error is an EOF, nil
// will be returned.
func (w *CountedWriter) Error() error {
if errors.Is(w.err, io.EOF) {
return nil
}
return w.err
}
// Write writes bytes to the underlying writer while tracking the total amount
// of bytes written.
func (w *CountedWriter) Write(p []byte) (int, error) {
if w.err != nil {
return 0, io.EOF
}
// Write is a very simple operation for us to handle.
n, err := w.File.Write(p)
w.counter.Add(int64(n))
w.err = err
// TODO: is this how we actually want to handle errors with this?
if err == io.EOF {
return n, io.EOF
} else {
return n, nil
}
}
func (w *CountedWriter) ReadFrom(r io.Reader) (n int64, err error) {
cr := NewCountedReader(r)
n, err = w.File.ReadFrom(cr)
w.counter.Add(n)
return
}
// CountedReader is a reader that counts the amount of data read from the
// underlying reader.
type CountedReader struct {
reader io.Reader
counter atomic.Int64
err error
}
var _ io.Reader = (*CountedReader)(nil)
// NewCountedReader returns a new countedReader that counts the amount of bytes
// read from the underlying reader.
func NewCountedReader(r io.Reader) *CountedReader {
return &CountedReader{reader: r}
}
// BytesRead returns the amount of bytes that have been read from the underlying
// reader.
func (r *CountedReader) BytesRead() int64 {
return r.counter.Load()
}
// Error returns the error from the reader if any. If the error is an EOF, nil
// will be returned.
func (r *CountedReader) Error() error {
if errors.Is(r.err, io.EOF) {
return nil
}
return r.err
}
// Read reads bytes from the underlying reader while tracking the total amount
// of bytes read.
func (r *CountedReader) Read(p []byte) (int, error) {
if r.err != nil {
return 0, io.EOF
}
n, err := r.reader.Read(p)
r.counter.Add(int64(n))
r.err = err
if err == io.EOF {
return n, io.EOF
} else {
return n, nil
}
}

View File

@ -0,0 +1,207 @@
// SPDX-License-Identifier: BSD-3-Clause
// Code in this file was derived from `go/src/os/removeall_at.go`.
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the `go.LICENSE` file.
//go:build unix
package ufs
import (
"errors"
"io"
"os"
"golang.org/x/sys/unix"
)
type unixFS interface {
Open(name string) (File, error)
Remove(name string) error
unlinkat(dirfd int, path string, flags int) error
}
func (fs *UnixFS) removeAll(path string) error {
return removeAll(fs, path)
}
func removeAll(fs unixFS, path string) error {
if path == "" {
// fail silently to retain compatibility with previous behavior
// of RemoveAll. See issue https://go.dev/issue/28830.
return nil
}
// The rmdir system call does not permit removing ".",
// so we don't permit it either.
if endsWithDot(path) {
return &PathError{Op: "removeall", Path: path, Err: unix.EINVAL}
}
// Simple case: if Remove works, we're done.
err := fs.Remove(path)
if err == nil || errors.Is(err, ErrNotExist) {
return nil
}
// RemoveAll recurses by deleting the path base from
// its parent directory
parentDir, base := splitPath(path)
parent, err := fs.Open(parentDir)
if errors.Is(err, ErrNotExist) {
// If parent does not exist, base cannot exist. Fail silently
return nil
}
if err != nil {
return err
}
defer parent.Close()
if err := removeAllFrom(fs, parent, base); err != nil {
if pathErr, ok := err.(*PathError); ok {
pathErr.Path = parentDir + string(os.PathSeparator) + pathErr.Path
err = pathErr
}
return convertErrorType(err)
}
return nil
}
func removeAllFrom(fs unixFS, parent File, base string) error {
parentFd := int(parent.Fd())
// Simple case: if Unlink (aka remove) works, we're done.
err := fs.unlinkat(parentFd, base, 0)
if err == nil || errors.Is(err, ErrNotExist) {
return nil
}
// EISDIR means that we have a directory, and we need to
// remove its contents.
// EPERM or EACCES means that we don't have write permission on
// the parent directory, but this entry might still be a directory
// whose contents need to be removed.
// Otherwise, just return the error.
if err != unix.EISDIR && err != unix.EPERM && err != unix.EACCES {
return &PathError{Op: "unlinkat", Path: base, Err: err}
}
// Is this a directory we need to recurse into?
var statInfo unix.Stat_t
statErr := ignoringEINTR(func() error {
return unix.Fstatat(parentFd, base, &statInfo, AT_SYMLINK_NOFOLLOW)
})
if statErr != nil {
if errors.Is(statErr, ErrNotExist) {
return nil
}
return &PathError{Op: "fstatat", Path: base, Err: statErr}
}
if statInfo.Mode&unix.S_IFMT != unix.S_IFDIR {
// Not a directory; return the error from the unix.Unlinkat.
return &PathError{Op: "unlinkat", Path: base, Err: err}
}
// Remove the directory's entries.
var recurseErr error
for {
const reqSize = 1024
var respSize int
// Open the directory to recurse into
file, err := openFdAt(parentFd, base)
if err != nil {
if errors.Is(err, ErrNotExist) {
return nil
}
recurseErr = &PathError{Op: "openfdat", Path: base, Err: err}
break
}
for {
numErr := 0
names, readErr := file.Readdirnames(reqSize)
// Errors other than EOF should stop us from continuing.
if readErr != nil && readErr != io.EOF {
_ = file.Close()
if errors.Is(readErr, ErrNotExist) {
return nil
}
return &PathError{Op: "readdirnames", Path: base, Err: readErr}
}
respSize = len(names)
for _, name := range names {
err := removeAllFrom(fs, file, name)
if err != nil {
if pathErr, ok := err.(*PathError); ok {
pathErr.Path = base + string(os.PathSeparator) + pathErr.Path
}
numErr++
if recurseErr == nil {
recurseErr = err
}
}
}
// If we can delete any entry, break to start new iteration.
// Otherwise, we discard current names, get next entries and try deleting them.
if numErr != reqSize {
break
}
}
// Removing files from the directory may have caused
// the OS to reshuffle it. Simply calling Readdirnames
// again may skip some entries. The only reliable way
// to avoid this is to close and re-open the
// directory. See issue https://go.dev/issue/20841.
_ = file.Close()
// Finish when the end of the directory is reached
if respSize < reqSize {
break
}
}
// Remove the directory itself.
unlinkErr := fs.unlinkat(parentFd, base, AT_REMOVEDIR)
if unlinkErr == nil || errors.Is(unlinkErr, ErrNotExist) {
return nil
}
if recurseErr != nil {
return recurseErr
}
return &PathError{Op: "unlinkat", Path: base, Err: unlinkErr}
}
// openFdAt opens path relative to the directory in fd.
// Other than that this should act like openFileNolog.
// This acts like openFileNolog rather than OpenFile because
// we are going to (try to) remove the file.
// The contents of this file are not relevant for test caching.
func openFdAt(dirfd int, name string) (File, error) {
var fd int
for {
var err error
fd, err = unix.Openat(dirfd, name, O_RDONLY|O_CLOEXEC|O_NOFOLLOW, 0)
if err == nil {
break
}
// See comment in openFileNolog.
if err == unix.EINTR {
continue
}
return nil, err
}
// This is stupid, os.NewFile immediately casts `fd` to an `int`, but wants
// it to be passed as a `uintptr`.
return os.NewFile(uintptr(fd), name), nil
}

67
internal/ufs/stat_unix.go Normal file
View File

@ -0,0 +1,67 @@
// SPDX-License-Identifier: BSD-3-Clause
// Code in this file was copied from `go/src/os/stat_linux.go`
// and `go/src/os/types_unix.go`.
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the `go.LICENSE` file.
//go:build unix
package ufs
import (
"time"
"golang.org/x/sys/unix"
)
type fileStat struct {
name string
size int64
mode FileMode
modTime time.Time
sys unix.Stat_t
}
var _ FileInfo = (*fileStat)(nil)
func (fs *fileStat) Size() int64 { return fs.size }
func (fs *fileStat) Mode() FileMode { return fs.mode }
func (fs *fileStat) ModTime() time.Time { return fs.modTime }
func (fs *fileStat) Sys() any { return &fs.sys }
func (fs *fileStat) Name() string { return fs.name }
func (fs *fileStat) IsDir() bool { return fs.Mode().IsDir() }
func fillFileStatFromSys(fs *fileStat, name string) {
fs.name = basename(name)
fs.size = fs.sys.Size
fs.modTime = time.Unix(fs.sys.Mtim.Unix())
fs.mode = FileMode(fs.sys.Mode & 0o777)
switch fs.sys.Mode & unix.S_IFMT {
case unix.S_IFBLK:
fs.mode |= ModeDevice
case unix.S_IFCHR:
fs.mode |= ModeDevice | ModeCharDevice
case unix.S_IFDIR:
fs.mode |= ModeDir
case unix.S_IFIFO:
fs.mode |= ModeNamedPipe
case unix.S_IFLNK:
fs.mode |= ModeSymlink
case unix.S_IFREG:
// nothing to do
case unix.S_IFSOCK:
fs.mode |= ModeSocket
}
if fs.sys.Mode&unix.S_ISGID != 0 {
fs.mode |= ModeSetgid
}
if fs.sys.Mode&unix.S_ISUID != 0 {
fs.mode |= ModeSetuid
}
if fs.sys.Mode&unix.S_ISVTX != 0 {
fs.mode |= ModeSticky
}
}

123
internal/ufs/walk.go Normal file
View File

@ -0,0 +1,123 @@
// SPDX-License-Identifier: BSD-3-Clause
// Code in this file was derived from `go/src/io/fs/walk.go`.
// Copyright 2020 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the `go.LICENSE` file.
package ufs
import (
iofs "io/fs"
"path"
)
// SkipDir is used as a return value from [WalkDirFunc] to indicate that
// the directory named in the call is to be skipped. It is not returned
// as an error by any function.
var SkipDir = iofs.SkipDir
// SkipAll is used as a return value from [WalkDirFunc] to indicate that
// all remaining files and directories are to be skipped. It is not returned
// as an error by any function.
var SkipAll = iofs.SkipAll
// WalkDirFunc is the type of the function called by [WalkDir] to visit
// each file or directory.
//
// The path argument contains the argument to [WalkDir] as a prefix.
// That is, if WalkDir is called with root argument "dir" and finds a file
// named "a" in that directory, the walk function will be called with
// argument "dir/a".
//
// The d argument is the [DirEntry] for the named path.
//
// The error result returned by the function controls how [WalkDir]
// continues. If the function returns the special value [SkipDir], WalkDir
// skips the current directory (path if d.IsDir() is true, otherwise
// path's parent directory). If the function returns the special value
// [SkipAll], WalkDir skips all remaining files and directories. Otherwise,
// if the function returns a non-nil error, WalkDir stops entirely and
// returns that error.
//
// The err argument reports an error related to path, signaling that
// [WalkDir] will not walk into that directory. The function can decide how
// to handle that error; as described earlier, returning the error will
// cause WalkDir to stop walking the entire tree.
//
// [WalkDir] calls the function with a non-nil err argument in two cases.
//
// First, if the initial [Stat] on the root directory fails, WalkDir
// calls the function with path set to root, d set to nil, and err set to
// the error from [fs.Stat].
//
// Second, if a directory's ReadDir method (see [ReadDirFile]) fails, WalkDir calls the
// function with path set to the directory's path, d set to an
// [DirEntry] describing the directory, and err set to the error from
// ReadDir. In this second case, the function is called twice with the
// path of the directory: the first call is before the directory read is
// attempted and has err set to nil, giving the function a chance to
// return [SkipDir] or [SkipAll] and avoid the ReadDir entirely. The second call
// is after a failed ReadDir and reports the error from ReadDir.
// (If ReadDir succeeds, there is no second call.)
type WalkDirFunc func(path string, d DirEntry, err error) error
// WalkDir walks the file tree rooted at root, calling fn for each file or
// directory in the tree, including root.
//
// All errors that arise visiting files and directories are filtered by fn:
// see the [WalkDirFunc] documentation for details.
//
// The files are walked in lexical order, which makes the output deterministic
// but requires WalkDir to read an entire directory into memory before proceeding
// to walk that directory.
//
// WalkDir does not follow symbolic links found in directories,
// but if root itself is a symbolic link, its target will be walked.
func WalkDir(fs Filesystem, root string, fn WalkDirFunc) error {
info, err := fs.Stat(root)
if err != nil {
err = fn(root, nil, err)
} else {
err = walkDir(fs, root, iofs.FileInfoToDirEntry(info), fn)
}
if err == SkipDir || err == SkipAll {
return nil
}
return err
}
// walkDir recursively descends path, calling walkDirFn.
func walkDir(fs Filesystem, name string, d DirEntry, walkDirFn WalkDirFunc) error {
if err := walkDirFn(name, d, nil); err != nil || !d.IsDir() {
if err == SkipDir && d.IsDir() {
// Successfully skipped directory.
err = nil
}
return err
}
dirs, err := fs.ReadDir(name)
if err != nil {
// Second call, to report ReadDir error.
err = walkDirFn(name, d, err)
if err != nil {
if err == SkipDir && d.IsDir() {
err = nil
}
return err
}
}
for _, d1 := range dirs {
name1 := path.Join(name, d1.Name())
if err := walkDir(fs, name1, d1, walkDirFn); err != nil {
if err == SkipDir {
break
}
return err
}
}
return nil
}

298
internal/ufs/walk_unix.go Normal file
View File

@ -0,0 +1,298 @@
// SPDX-License-Identifier: BSD-2-Clause
// Some code in this file was derived from https://github.com/karrick/godirwalk.
//go:build unix
package ufs
import (
"bytes"
iofs "io/fs"
"os"
"path"
"path/filepath"
"reflect"
"unsafe"
"golang.org/x/sys/unix"
)
type WalkDiratFunc func(dirfd int, name, relative string, d DirEntry, err error) error
func (fs *UnixFS) WalkDirat(dirfd int, name string, fn WalkDiratFunc) error {
if dirfd == 0 {
// TODO: proper validation, ideally a dedicated function.
dirfd = int(fs.dirfd.Load())
}
info, err := fs.Lstatat(dirfd, name)
if err != nil {
err = fn(dirfd, name, name, nil, err)
} else {
b := newScratchBuffer()
err = fs.walkDir(b, dirfd, name, name, iofs.FileInfoToDirEntry(info), fn)
}
if err == SkipDir || err == SkipAll {
return nil
}
return err
}
func (fs *UnixFS) walkDir(b []byte, parentfd int, name, relative string, d DirEntry, walkDirFn WalkDiratFunc) error {
if err := walkDirFn(parentfd, name, relative, d, nil); err != nil || !d.IsDir() {
if err == SkipDir && d.IsDir() {
// Successfully skipped directory.
err = nil
}
return err
}
dirfd, err := fs.openat(parentfd, name, O_DIRECTORY|O_RDONLY, 0)
if err != nil {
return err
}
defer unix.Close(dirfd)
dirs, err := fs.readDir(dirfd, name, b)
if err != nil {
// Second call, to report ReadDir error.
err = walkDirFn(dirfd, name, relative, d, err)
if err != nil {
if err == SkipDir && d.IsDir() {
err = nil
}
return err
}
}
for _, d1 := range dirs {
// TODO: the path.Join on this line may actually be partially incorrect.
// If we are not walking starting at the root, relative will contain the
// name of the directory we are starting the walk from, which will be
// relative to the root of the filesystem instead of from where the walk
// was initiated from.
//
// ref; https://github.com/pterodactyl/panel/issues/5030
if err := fs.walkDir(b, dirfd, d1.Name(), path.Join(relative, d1.Name()), d1, walkDirFn); err != nil {
if err == SkipDir {
break
}
return err
}
}
return nil
}
// ReadDirMap .
// TODO: document
func ReadDirMap[T any](fs *UnixFS, path string, fn func(DirEntry) (T, error)) ([]T, error) {
dirfd, name, closeFd, err := fs.safePath(path)
defer closeFd()
if err != nil {
return nil, err
}
fd, err := fs.openat(dirfd, name, O_DIRECTORY|O_RDONLY, 0)
if err != nil {
return nil, err
}
defer unix.Close(fd)
entries, err := fs.readDir(fd, ".", nil)
if err != nil {
return nil, err
}
out := make([]T, len(entries))
for i, e := range entries {
idx := i
e := e
v, err := fn(e)
if err != nil {
return nil, err
}
out[idx] = v
}
return out, nil
}
// nameOffset is a compile time constant
const nameOffset = int(unsafe.Offsetof(unix.Dirent{}.Name))
func nameFromDirent(de *unix.Dirent) (name []byte) {
// Because this GOOS' syscall.Dirent does not provide a field that specifies
// the name length, this function must first calculate the max possible name
// length, and then search for the NULL byte.
ml := int(de.Reclen) - nameOffset
// Convert syscall.Dirent.Name, which is array of int8, to []byte, by
// overwriting Cap, Len, and Data slice header fields to the max possible
// name length computed above, and finding the terminating NULL byte.
//
// TODO: is there an alternative to the deprecated SliceHeader?
// SliceHeader was mainly deprecated due to it being misused for avoiding
// allocations when converting a byte slice to a string, ref;
// https://go.dev/issue/53003
sh := (*reflect.SliceHeader)(unsafe.Pointer(&name))
sh.Cap = ml
sh.Len = ml
sh.Data = uintptr(unsafe.Pointer(&de.Name[0]))
if index := bytes.IndexByte(name, 0); index >= 0 {
// Found NULL byte; set slice's cap and len accordingly.
sh.Cap = index
sh.Len = index
return
}
// NOTE: This branch is not expected, but included for defensive
// programming, and provides a hard stop on the name based on the structure
// field array size.
sh.Cap = len(de.Name)
sh.Len = sh.Cap
return
}
// modeTypeFromDirent converts a syscall defined constant, which is in purview
// of OS, to a constant defined by Go, assumed by this project to be stable.
//
// When the syscall constant is not recognized, this function falls back to a
// Stat on the file system.
func (fs *UnixFS) modeTypeFromDirent(fd int, de *unix.Dirent, osDirname, osBasename string) (FileMode, error) {
switch de.Type {
case unix.DT_REG:
return 0, nil
case unix.DT_DIR:
return ModeDir, nil
case unix.DT_LNK:
return ModeSymlink, nil
case unix.DT_CHR:
return ModeDevice | ModeCharDevice, nil
case unix.DT_BLK:
return ModeDevice, nil
case unix.DT_FIFO:
return ModeNamedPipe, nil
case unix.DT_SOCK:
return ModeSocket, nil
default:
// If syscall returned unknown type (e.g., DT_UNKNOWN, DT_WHT), then
// resolve actual mode by reading file information.
return fs.modeType(fd, filepath.Join(osDirname, osBasename))
}
}
// modeType returns the mode type of the file system entry identified by
// osPathname by calling os.LStat function, to intentionally not follow symbolic
// links.
//
// Even though os.LStat provides all file mode bits, we want to ensure same
// values returned to caller regardless of whether we obtained file mode bits
// from syscall or stat call. Therefore, mask out the additional file mode bits
// that are provided by stat but not by the syscall, so users can rely on their
// values.
func (fs *UnixFS) modeType(dirfd int, name string) (os.FileMode, error) {
fi, err := fs.Lstatat(dirfd, name)
if err == nil {
return fi.Mode() & ModeType, nil
}
return 0, err
}
var minimumScratchBufferSize = os.Getpagesize()
func newScratchBuffer() []byte {
return make([]byte, minimumScratchBufferSize)
}
func (fs *UnixFS) readDir(fd int, name string, b []byte) ([]DirEntry, error) {
scratchBuffer := b
if scratchBuffer == nil || len(scratchBuffer) < minimumScratchBufferSize {
scratchBuffer = newScratchBuffer()
}
var entries []DirEntry
var workBuffer []byte
var sde unix.Dirent
for {
if len(workBuffer) == 0 {
n, err := unix.Getdents(fd, scratchBuffer)
if err != nil {
if err == unix.EINTR {
continue
}
return nil, convertErrorType(err)
}
if n <= 0 {
// end of directory: normal exit
return entries, nil
}
workBuffer = scratchBuffer[:n] // trim work buffer to number of bytes read
}
// "Go is like C, except that you just put `unsafe` all over the place".
copy((*[unsafe.Sizeof(unix.Dirent{})]byte)(unsafe.Pointer(&sde))[:], workBuffer)
workBuffer = workBuffer[sde.Reclen:] // advance buffer for next iteration through loop
if sde.Ino == 0 {
continue // inode set to 0 indicates an entry that was marked as deleted
}
nameSlice := nameFromDirent(&sde)
nameLength := len(nameSlice)
if nameLength == 0 || (nameSlice[0] == '.' && (nameLength == 1 || (nameLength == 2 && nameSlice[1] == '.'))) {
continue
}
childName := string(nameSlice)
mt, err := fs.modeTypeFromDirent(fd, &sde, name, childName)
if err != nil {
return nil, convertErrorType(err)
}
entries = append(entries, &dirent{name: childName, path: name, modeType: mt, dirfd: fd, fs: fs})
}
}
// dirent stores the name and file system mode type of discovered file system
// entries.
type dirent struct {
name string
path string
modeType FileMode
dirfd int
fs *UnixFS
}
func (de dirent) Name() string {
return de.name
}
func (de dirent) IsDir() bool {
return de.modeType&ModeDir != 0
}
func (de dirent) Type() FileMode {
return de.modeType
}
func (de dirent) Info() (FileInfo, error) {
if de.fs == nil {
return nil, nil
}
return de.fs.Lstatat(de.dirfd, de.name)
}
func (de dirent) Open() (File, error) {
if de.fs == nil {
return nil, nil
}
return de.fs.OpenFileat(de.dirfd, de.name, O_RDONLY, 0)
}
// reset releases memory held by entry err and name, and resets mode type to 0.
func (de *dirent) reset() {
de.name = ""
de.path = ""
de.modeType = 0
}

View File

@ -2,8 +2,6 @@ package parser
import ( import (
"bytes" "bytes"
"io"
"os"
"regexp" "regexp"
"strconv" "strconv"
"strings" "strings"
@ -29,24 +27,14 @@ var configMatchRegex = regexp.MustCompile(`{{\s?config\.([\w.-]+)\s?}}`)
// matching: // matching:
// //
// <Root> // <Root>
// <Property value="testing"/> //
// <Property value="testing"/>
//
// </Root> // </Root>
// //
// noinspection RegExpRedundantEscape // noinspection RegExpRedundantEscape
var xmlValueMatchRegex = regexp.MustCompile(`^\[([\w]+)='(.*)'\]$`) var xmlValueMatchRegex = regexp.MustCompile(`^\[([\w]+)='(.*)'\]$`)
// Gets the []byte representation of a configuration file to be passed through to other
// handler functions. If the file does not currently exist, it will be created.
func readFileBytes(path string) ([]byte, error) {
file, err := os.OpenFile(path, os.O_CREATE|os.O_RDWR, 0o644)
if err != nil {
return nil, err
}
defer file.Close()
return io.ReadAll(file)
}
// Gets the value of a key based on the value type defined. // Gets the value of a key based on the value type defined.
func (cfr *ConfigurationFileReplacement) getKeyValue(value string) interface{} { func (cfr *ConfigurationFileReplacement) getKeyValue(value string) interface{} {
if cfr.ReplaceWith.Type() == jsonparser.Boolean { if cfr.ReplaceWith.Type() == jsonparser.Boolean {

View File

@ -2,8 +2,8 @@ package parser
import ( import (
"bufio" "bufio"
"os" "bytes"
"path/filepath" "io"
"strconv" "strconv"
"strings" "strings"
@ -11,13 +11,14 @@ import (
"github.com/apex/log" "github.com/apex/log"
"github.com/beevik/etree" "github.com/beevik/etree"
"github.com/buger/jsonparser" "github.com/buger/jsonparser"
"github.com/goccy/go-json"
"github.com/icza/dyno" "github.com/icza/dyno"
"github.com/magiconair/properties" "github.com/magiconair/properties"
"github.com/goccy/go-json"
"gopkg.in/ini.v1" "gopkg.in/ini.v1"
"gopkg.in/yaml.v2" "gopkg.in/yaml.v3"
"github.com/pterodactyl/wings/config" "github.com/pterodactyl/wings/config"
"github.com/pterodactyl/wings/internal/ufs"
) )
// The file parsing options that are available for a server configuration file. // The file parsing options that are available for a server configuration file.
@ -74,6 +75,26 @@ func (cv *ReplaceValue) String() string {
} }
} }
func (cv *ReplaceValue) Bytes() []byte {
switch cv.Type() {
case jsonparser.String:
var stackbuf [64]byte
bU, err := jsonparser.Unescape(cv.value, stackbuf[:])
if err != nil {
panic(errors.Wrap(err, "parser: could not parse value"))
}
return bU
case jsonparser.Null:
return []byte("<nil>")
case jsonparser.Boolean:
return cv.value
case jsonparser.Number:
return cv.value
default:
return []byte("<invalid>")
}
}
type ConfigurationParser string type ConfigurationParser string
func (cp ConfigurationParser) String() string { func (cp ConfigurationParser) String() string {
@ -167,11 +188,12 @@ func (cfr *ConfigurationFileReplacement) UnmarshalJSON(data []byte) error {
return nil return nil
} }
// Parses a given configuration file and updates all of the values within as defined // Parse parses a given configuration file and updates all the values within
// in the API response from the Panel. // as defined in the API response from the Panel.
func (f *ConfigurationFile) Parse(path string, internal bool) error { func (f *ConfigurationFile) Parse(file ufs.File) error {
log.WithField("path", path).WithField("parser", f.Parser.String()).Debug("parsing server configuration file") //log.WithField("path", path).WithField("parser", f.Parser.String()).Debug("parsing server configuration file")
// What the fuck is going on here?
if mb, err := json.Marshal(config.Get()); err != nil { if mb, err := json.Marshal(config.Get()); err != nil {
return err return err
} else { } else {
@ -182,56 +204,24 @@ func (f *ConfigurationFile) Parse(path string, internal bool) error {
switch f.Parser { switch f.Parser {
case Properties: case Properties:
err = f.parsePropertiesFile(path) err = f.parsePropertiesFile(file)
break
case File: case File:
err = f.parseTextFile(path) err = f.parseTextFile(file)
break
case Yaml, "yml": case Yaml, "yml":
err = f.parseYamlFile(path) err = f.parseYamlFile(file)
break
case Json: case Json:
err = f.parseJsonFile(path) err = f.parseJsonFile(file)
break
case Ini: case Ini:
err = f.parseIniFile(path) err = f.parseIniFile(file)
break
case Xml: case Xml:
err = f.parseXmlFile(path) err = f.parseXmlFile(file)
break
} }
if errors.Is(err, os.ErrNotExist) {
// File doesn't exist, we tried creating it, and same error is returned? Pretty
// sure this pathway is impossible, but if not, abort here.
if internal {
return nil
}
b := strings.TrimSuffix(path, filepath.Base(path))
if err := os.MkdirAll(b, 0o755); err != nil {
return errors.WithMessage(err, "failed to create base directory for missing configuration file")
} else {
if _, err := os.Create(path); err != nil {
return errors.WithMessage(err, "failed to create missing configuration file")
}
}
return f.Parse(path, true)
}
return err return err
} }
// Parses an xml file. // Parses an xml file.
func (f *ConfigurationFile) parseXmlFile(path string) error { func (f *ConfigurationFile) parseXmlFile(file ufs.File) error {
doc := etree.NewDocument() doc := etree.NewDocument()
file, err := os.OpenFile(path, os.O_CREATE|os.O_RDWR, 0o644)
if err != nil {
return err
}
defer file.Close()
if _, err := doc.ReadFrom(file); err != nil { if _, err := doc.ReadFrom(file); err != nil {
return err return err
} }
@ -291,41 +281,27 @@ func (f *ConfigurationFile) parseXmlFile(path string) error {
} }
} }
// If you don't truncate the file you'll end up duplicating the data in there (or just appending if _, err := file.Seek(0, io.SeekStart); err != nil {
// to the end of the file. We don't want to do that. return err
}
if err := file.Truncate(0); err != nil { if err := file.Truncate(0); err != nil {
return err return err
} }
// Move the cursor to the start of the file to avoid weird spacing issues.
file.Seek(0, 0)
// Ensure the XML is indented properly. // Ensure the XML is indented properly.
doc.Indent(2) doc.Indent(2)
// Truncate the file before attempting to write the changes. // Write the XML to the file.
if err := os.Truncate(path, 0); err != nil { if _, err := doc.WriteTo(file); err != nil {
return err return err
} }
return nil
// Write the XML to the file.
_, err = doc.WriteTo(file)
return err
} }
// Parses an ini file. // Parses an ini file.
func (f *ConfigurationFile) parseIniFile(path string) error { func (f *ConfigurationFile) parseIniFile(file ufs.File) error {
// Ini package can't handle a non-existent file, so handle that automatically here // Wrap the file in a NopCloser so the ini package doesn't close the file.
// by creating it if not exists. Then, immediately close the file since we will use cfg, err := ini.Load(io.NopCloser(file))
// other methods to write the new contents.
file, err := os.OpenFile(path, os.O_CREATE|os.O_RDWR, 0o644)
if err != nil {
return err
}
file.Close()
cfg, err := ini.Load(path)
if err != nil { if err != nil {
return err return err
} }
@ -388,14 +364,24 @@ func (f *ConfigurationFile) parseIniFile(path string) error {
} }
} }
return cfg.SaveTo(path) if _, err := file.Seek(0, io.SeekStart); err != nil {
return err
}
if err := file.Truncate(0); err != nil {
return err
}
if _, err := cfg.WriteTo(file); err != nil {
return err
}
return nil
} }
// Parses a json file updating any matching key/value pairs. If a match is not found, the // Parses a json file updating any matching key/value pairs. If a match is not found, the
// value is set regardless in the file. See the commentary in parseYamlFile for more details // value is set regardless in the file. See the commentary in parseYamlFile for more details
// about what is happening during this process. // about what is happening during this process.
func (f *ConfigurationFile) parseJsonFile(path string) error { func (f *ConfigurationFile) parseJsonFile(file ufs.File) error {
b, err := readFileBytes(path) b, err := io.ReadAll(file)
if err != nil { if err != nil {
return err return err
} }
@ -405,14 +391,24 @@ func (f *ConfigurationFile) parseJsonFile(path string) error {
return err return err
} }
output := []byte(data.StringIndent("", " ")) if _, err := file.Seek(0, io.SeekStart); err != nil {
return os.WriteFile(path, output, 0o644) return err
}
if err := file.Truncate(0); err != nil {
return err
}
// Write the data to the file.
if _, err := io.Copy(file, bytes.NewReader(data.BytesIndent("", " "))); err != nil {
return errors.Wrap(err, "parser: failed to write properties file to disk")
}
return nil
} }
// Parses a yaml file and updates any matching key/value pairs before persisting // Parses a yaml file and updates any matching key/value pairs before persisting
// it back to the disk. // it back to the disk.
func (f *ConfigurationFile) parseYamlFile(path string) error { func (f *ConfigurationFile) parseYamlFile(file ufs.File) error {
b, err := readFileBytes(path) b, err := io.ReadAll(file)
if err != nil { if err != nil {
return err return err
} }
@ -443,35 +439,56 @@ func (f *ConfigurationFile) parseYamlFile(path string) error {
return err return err
} }
return os.WriteFile(path, marshaled, 0o644) if _, err := file.Seek(0, io.SeekStart); err != nil {
return err
}
if err := file.Truncate(0); err != nil {
return err
}
// Write the data to the file.
if _, err := io.Copy(file, bytes.NewReader(marshaled)); err != nil {
return errors.Wrap(err, "parser: failed to write properties file to disk")
}
return nil
} }
// Parses a text file using basic find and replace. This is a highly inefficient method of // Parses a text file using basic find and replace. This is a highly inefficient method of
// scanning a file and performing a replacement. You should attempt to use anything other // scanning a file and performing a replacement. You should attempt to use anything other
// than this function where possible. // than this function where possible.
func (f *ConfigurationFile) parseTextFile(path string) error { func (f *ConfigurationFile) parseTextFile(file ufs.File) error {
input, err := os.ReadFile(path) b := bytes.NewBuffer(nil)
if err != nil { s := bufio.NewScanner(file)
return err var replaced bool
} for s.Scan() {
line := s.Bytes()
lines := strings.Split(string(input), "\n") replaced = false
for i, line := range lines {
for _, replace := range f.Replace { for _, replace := range f.Replace {
// If this line doesn't match what we expect for the replacement, move on to the next // If this line doesn't match what we expect for the replacement, move on to the next
// line. Otherwise, update the line to have the replacement value. // line. Otherwise, update the line to have the replacement value.
if !strings.HasPrefix(line, replace.Match) { if !bytes.HasPrefix(line, []byte(replace.Match)) {
continue continue
} }
b.Write(replace.ReplaceWith.Bytes())
lines[i] = replace.ReplaceWith.String() replaced = true
} }
if !replaced {
b.Write(line)
}
b.WriteByte('\n')
} }
if err := os.WriteFile(path, []byte(strings.Join(lines, "\n")), 0o644); err != nil { if _, err := file.Seek(0, io.SeekStart); err != nil {
return err
}
if err := file.Truncate(0); err != nil {
return err return err
} }
// Write the data to the file.
if _, err := io.Copy(file, b); err != nil {
return errors.Wrap(err, "parser: failed to write properties file to disk")
}
return nil return nil
} }
@ -501,31 +518,29 @@ func (f *ConfigurationFile) parseTextFile(path string) error {
// //
// @see https://github.com/pterodactyl/panel/issues/2308 (original) // @see https://github.com/pterodactyl/panel/issues/2308 (original)
// @see https://github.com/pterodactyl/panel/issues/3009 ("bug" introduced as result) // @see https://github.com/pterodactyl/panel/issues/3009 ("bug" introduced as result)
func (f *ConfigurationFile) parsePropertiesFile(path string) error { func (f *ConfigurationFile) parsePropertiesFile(file ufs.File) error {
var s strings.Builder b, err := io.ReadAll(file)
// Open the file and attempt to load any comments that currenty exist at the start if err != nil {
// of the file. This is kind of a hack, but should work for a majority of users for return err
// the time being.
if fd, err := os.Open(path); err != nil {
return errors.Wrap(err, "parser: could not open file for reading")
} else {
scanner := bufio.NewScanner(fd)
// Scan until we hit a line that is not a comment that actually has content
// on it. Keep appending the comments until that time.
for scanner.Scan() {
text := scanner.Text()
if len(text) > 0 && text[0] != '#' {
break
}
s.WriteString(text + "\n")
}
_ = fd.Close()
if err := scanner.Err(); err != nil {
return errors.WithStackIf(err)
}
} }
p, err := properties.LoadFile(path, properties.UTF8) s := bytes.NewBuffer(nil)
scanner := bufio.NewScanner(bytes.NewReader(b))
// Scan until we hit a line that is not a comment that actually has content
// on it. Keep appending the comments until that time.
for scanner.Scan() {
text := scanner.Bytes()
if len(text) > 0 && text[0] != '#' {
break
}
s.Write(text)
s.WriteByte('\n')
}
if err := scanner.Err(); err != nil {
return errors.WithStackIf(err)
}
p, err := properties.Load(b, properties.UTF8)
if err != nil { if err != nil {
return errors.Wrap(err, "parser: could not load properties file for configuration update") return errors.Wrap(err, "parser: could not load properties file for configuration update")
} }
@ -563,17 +578,16 @@ func (f *ConfigurationFile) parsePropertiesFile(path string) error {
s.WriteString(key + "=" + strings.Trim(strconv.QuoteToASCII(value), "\"") + "\n") s.WriteString(key + "=" + strings.Trim(strconv.QuoteToASCII(value), "\"") + "\n")
} }
// Open the file for writing. if _, err := file.Seek(0, io.SeekStart); err != nil {
w, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o644)
if err != nil {
return err return err
} }
defer w.Close() if err := file.Truncate(0); err != nil {
return err
// Write the data to the file.
if _, err := w.Write([]byte(s.String())); err != nil {
return errors.Wrap(err, "parser: failed to write properties file to disk")
} }
// Write the data to the file.
if _, err := io.Copy(file, s); err != nil {
return errors.Wrap(err, "parser: failed to write properties file to disk")
}
return nil return nil
} }

View File

@ -10,6 +10,8 @@ import (
"strings" "strings"
"time" "time"
"github.com/pterodactyl/wings/internal/models"
"emperror.dev/errors" "emperror.dev/errors"
"github.com/apex/log" "github.com/apex/log"
"github.com/cenkalti/backoff/v4" "github.com/cenkalti/backoff/v4"
@ -27,9 +29,10 @@ type Client interface {
SetArchiveStatus(ctx context.Context, uuid string, successful bool) error SetArchiveStatus(ctx context.Context, uuid string, successful bool) error
SetBackupStatus(ctx context.Context, backup string, data BackupRequest) error SetBackupStatus(ctx context.Context, backup string, data BackupRequest) error
SendRestorationStatus(ctx context.Context, backup string, successful bool) error SendRestorationStatus(ctx context.Context, backup string, successful bool) error
SetInstallationStatus(ctx context.Context, uuid string, successful bool) error SetInstallationStatus(ctx context.Context, uuid string, data InstallStatusRequest) error
SetTransferStatus(ctx context.Context, uuid string, successful bool) error SetTransferStatus(ctx context.Context, uuid string, successful bool) error
ValidateSftpCredentials(ctx context.Context, request SftpAuthRequest) (SftpAuthResponse, error) ValidateSftpCredentials(ctx context.Context, request SftpAuthRequest) (SftpAuthResponse, error)
SendActivityLogs(ctx context.Context, activity []models.Activity) error
} }
type client struct { type client struct {
@ -128,10 +131,19 @@ func (c *client) requestOnce(ctx context.Context, method, path string, body io.R
// and adds the required authentication headers to the request that is being // and adds the required authentication headers to the request that is being
// created. Errors returned will be of the RequestError type if there was some // created. Errors returned will be of the RequestError type if there was some
// type of response from the API that can be parsed. // type of response from the API that can be parsed.
func (c *client) request(ctx context.Context, method, path string, body io.Reader, opts ...func(r *http.Request)) (*Response, error) { func (c *client) request(ctx context.Context, method, path string, body *bytes.Buffer, opts ...func(r *http.Request)) (*Response, error) {
var res *Response var res *Response
err := backoff.Retry(func() error { err := backoff.Retry(func() error {
r, err := c.requestOnce(ctx, method, path, body, opts...) var b bytes.Buffer
if body != nil {
// We have to create a copy of the body, otherwise attempting this request again will
// send no data if there was initially a body since the "requestOnce" method will read
// the whole buffer, thus leaving it empty at the end.
if _, err := b.Write(body.Bytes()); err != nil {
return backoff.Permanent(errors.Wrap(err, "http: failed to copy body buffer"))
}
}
r, err := c.requestOnce(ctx, method, path, &b, opts...)
if err != nil { if err != nil {
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) { if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
return backoff.Permanent(err) return backoff.Permanent(err)
@ -142,12 +154,10 @@ func (c *client) request(ctx context.Context, method, path string, body io.Reade
if r.HasError() { if r.HasError() {
// Close the request body after returning the error to free up resources. // Close the request body after returning the error to free up resources.
defer r.Body.Close() defer r.Body.Close()
// Don't keep spamming the endpoint if we've already made too many requests or // Don't keep attempting to access this endpoint if the response is a 4XX
// if we're not even authenticated correctly. Retrying generally won't fix either // level error which indicates a client mistake. Only retry when the error
// of these issues. // is due to a server issue (5XX error).
if r.StatusCode == http.StatusForbidden || if r.StatusCode >= 400 && r.StatusCode < 500 {
r.StatusCode == http.StatusTooManyRequests ||
r.StatusCode == http.StatusUnauthorized {
return backoff.Permanent(r.Error()) return backoff.Permanent(r.Error())
} }
return r.Error() return r.Error()

View File

@ -6,6 +6,8 @@ import (
"strconv" "strconv"
"sync" "sync"
"github.com/pterodactyl/wings/internal/models"
"emperror.dev/errors" "emperror.dev/errors"
"github.com/apex/log" "github.com/apex/log"
"golang.org/x/sync/errgroup" "golang.org/x/sync/errgroup"
@ -17,7 +19,7 @@ const (
ProcessStopNativeStop = "stop" ProcessStopNativeStop = "stop"
) )
// GetServers returns all of the servers that are present on the Panel making // GetServers returns all the servers that are present on the Panel making
// parallel API calls to the endpoint if more than one page of servers is // parallel API calls to the endpoint if more than one page of servers is
// returned. // returned.
func (c *client) GetServers(ctx context.Context, limit int) ([]RawServerData, error) { func (c *client) GetServers(ctx context.Context, limit int) ([]RawServerData, error) {
@ -56,7 +58,7 @@ func (c *client) GetServers(ctx context.Context, limit int) ([]RawServerData, er
// //
// This handles Wings exiting during either of these processes which will leave // This handles Wings exiting during either of these processes which will leave
// things in a bad state within the Panel. This API call is executed once Wings // things in a bad state within the Panel. This API call is executed once Wings
// has fully booted all of the servers. // has fully booted all the servers.
func (c *client) ResetServersState(ctx context.Context) error { func (c *client) ResetServersState(ctx context.Context) error {
res, err := c.Post(ctx, "/servers/reset", nil) res, err := c.Post(ctx, "/servers/reset", nil)
if err != nil { if err != nil {
@ -90,8 +92,8 @@ func (c *client) GetInstallationScript(ctx context.Context, uuid string) (Instal
return config, err return config, err
} }
func (c *client) SetInstallationStatus(ctx context.Context, uuid string, successful bool) error { func (c *client) SetInstallationStatus(ctx context.Context, uuid string, data InstallStatusRequest) error {
resp, err := c.Post(ctx, fmt.Sprintf("/servers/%s/install", uuid), d{"successful": successful}) resp, err := c.Post(ctx, fmt.Sprintf("/servers/%s/install", uuid), data)
if err != nil { if err != nil {
return err return err
} }
@ -113,7 +115,7 @@ func (c *client) SetTransferStatus(ctx context.Context, uuid string, successful
if successful { if successful {
state = "success" state = "success"
} }
resp, err := c.Get(ctx, fmt.Sprintf("/servers/%s/transfer/%s", uuid, state), nil) resp, err := c.Post(ctx, fmt.Sprintf("/servers/%s/transfer/%s", uuid, state), nil)
if err != nil { if err != nil {
return err return err
} }
@ -125,7 +127,7 @@ func (c *client) SetTransferStatus(ctx context.Context, uuid string, successful
// password combination provided is associated with a valid server on the instance // password combination provided is associated with a valid server on the instance
// using the Panel's authentication control mechanisms. This will get itself // using the Panel's authentication control mechanisms. This will get itself
// throttled if too many requests are made, allowing us to completely offload // throttled if too many requests are made, allowing us to completely offload
// all of the authorization security logic to the Panel. // all the authorization security logic to the Panel.
func (c *client) ValidateSftpCredentials(ctx context.Context, request SftpAuthRequest) (SftpAuthResponse, error) { func (c *client) ValidateSftpCredentials(ctx context.Context, request SftpAuthRequest) (SftpAuthResponse, error) {
var auth SftpAuthResponse var auth SftpAuthResponse
res, err := c.Post(ctx, "/sftp/auth", request) res, err := c.Post(ctx, "/sftp/auth", request)
@ -178,6 +180,16 @@ func (c *client) SendRestorationStatus(ctx context.Context, backup string, succe
return nil return nil
} }
// SendActivityLogs sends activity logs back to the Panel for processing.
func (c *client) SendActivityLogs(ctx context.Context, activity []models.Activity) error {
resp, err := c.Post(ctx, "/activity", d{"data": activity})
if err != nil {
return errors.WithStackIf(err)
}
_ = resp.Body.Close()
return nil
}
// getServersPaged returns a subset of servers from the Panel API using the // getServersPaged returns a subset of servers from the Panel API using the
// pagination query parameters. // pagination query parameters.
func (c *client) getServersPaged(ctx context.Context, page, limit int) ([]RawServerData, Pagination, error) { func (c *client) getServersPaged(ctx context.Context, page, limit int) ([]RawServerData, Pagination, error) {

View File

@ -1,6 +1,7 @@
package remote package remote
import ( import (
"bytes"
"regexp" "regexp"
"strings" "strings"
@ -10,6 +11,11 @@ import (
"github.com/pterodactyl/wings/parser" "github.com/pterodactyl/wings/parser"
) )
const (
SftpAuthPassword = SftpAuthRequestType("password")
SftpAuthPublicKey = SftpAuthRequestType("public_key")
)
// A generic type allowing for easy binding use when making requests to API // A generic type allowing for easy binding use when making requests to API
// endpoints that only expect a singular argument or something that would not // endpoints that only expect a singular argument or something that would not
// benefit from being a typed struct. // benefit from being a typed struct.
@ -62,14 +68,17 @@ type RawServerData struct {
ProcessConfiguration json.RawMessage `json:"process_configuration"` ProcessConfiguration json.RawMessage `json:"process_configuration"`
} }
type SftpAuthRequestType string
// SftpAuthRequest defines the request details that are passed along to the Panel // SftpAuthRequest defines the request details that are passed along to the Panel
// when determining if the credentials provided to Wings are valid. // when determining if the credentials provided to Wings are valid.
type SftpAuthRequest struct { type SftpAuthRequest struct {
User string `json:"username"` Type SftpAuthRequestType `json:"type"`
Pass string `json:"password"` User string `json:"username"`
IP string `json:"ip"` Pass string `json:"password"`
SessionID []byte `json:"session_id"` IP string `json:"ip"`
ClientVersion []byte `json:"client_version"` SessionID []byte `json:"session_id"`
ClientVersion []byte `json:"client_version"`
} }
// SftpAuthResponse is returned by the Panel when a pair of SFTP credentials // SftpAuthResponse is returned by the Panel when a pair of SFTP credentials
@ -78,44 +87,45 @@ type SftpAuthRequest struct {
// user for the SFTP subsystem. // user for the SFTP subsystem.
type SftpAuthResponse struct { type SftpAuthResponse struct {
Server string `json:"server"` Server string `json:"server"`
Token string `json:"token"` User string `json:"user"`
Permissions []string `json:"permissions"` Permissions []string `json:"permissions"`
} }
type OutputLineMatcher struct { type OutputLineMatcher struct {
// The raw string to match against. This may or may not be prefixed with // raw string to match against. This may or may not be prefixed with
// regex: which indicates we want to match against the regex expression. // `regex:` which indicates we want to match against the regex expression.
raw string raw []byte
reg *regexp.Regexp reg *regexp.Regexp
} }
// Matches determines if a given string "s" matches the given line. // Matches determines if the provided byte string matches the given regex or
func (olm *OutputLineMatcher) Matches(s string) bool { // raw string provided to the matcher.
func (olm *OutputLineMatcher) Matches(s []byte) bool {
if olm.reg == nil { if olm.reg == nil {
return strings.Contains(s, olm.raw) return bytes.Contains(s, olm.raw)
} }
return olm.reg.Match(s)
return olm.reg.MatchString(s)
} }
// String returns the matcher's raw comparison string. // String returns the matcher's raw comparison string.
func (olm *OutputLineMatcher) String() string { func (olm *OutputLineMatcher) String() string {
return olm.raw return string(olm.raw)
} }
// UnmarshalJSON unmarshals the startup lines into individual structs for easier // UnmarshalJSON unmarshals the startup lines into individual structs for easier
// matching abilities. // matching abilities.
func (olm *OutputLineMatcher) UnmarshalJSON(data []byte) error { func (olm *OutputLineMatcher) UnmarshalJSON(data []byte) error {
if err := json.Unmarshal(data, &olm.raw); err != nil { var r string
if err := json.Unmarshal(data, &r); err != nil {
return err return err
} }
if strings.HasPrefix(olm.raw, "regex:") && len(olm.raw) > 6 { olm.raw = []byte(r)
r, err := regexp.Compile(strings.TrimPrefix(olm.raw, "regex:")) if bytes.HasPrefix(olm.raw, []byte("regex:")) && len(olm.raw) > 6 {
r, err := regexp.Compile(strings.TrimPrefix(string(olm.raw), "regex:"))
if err != nil { if err != nil {
log.WithField("error", err).WithField("raw", olm.raw).Warn("failed to compile output line marked as being regex") log.WithField("error", err).WithField("raw", string(olm.raw)).Warn("failed to compile output line marked as being regex")
} }
olm.reg = r olm.reg = r
} }
@ -129,9 +139,9 @@ type ProcessStopConfiguration struct {
} }
// ProcessConfiguration defines the process configuration for a given server // ProcessConfiguration defines the process configuration for a given server
// instance. This sets what Wings is looking for to mark a server as done starting // instance. This sets what Wings is looking for to mark a server as done
// what to do when stopping, and what changes to make to the configuration file // starting what to do when stopping, and what changes to make to the
// for a server. // configuration file for a server.
type ProcessConfiguration struct { type ProcessConfiguration struct {
Startup struct { Startup struct {
Done []*OutputLineMatcher `json:"done"` Done []*OutputLineMatcher `json:"done"`
@ -147,9 +157,20 @@ type BackupRemoteUploadResponse struct {
PartSize int64 `json:"part_size"` PartSize int64 `json:"part_size"`
} }
type BackupRequest struct { type BackupPart struct {
Checksum string `json:"checksum"` ETag string `json:"etag"`
ChecksumType string `json:"checksum_type"` PartNumber int `json:"part_number"`
Size int64 `json:"size"` }
Successful bool `json:"successful"`
type BackupRequest struct {
Checksum string `json:"checksum"`
ChecksumType string `json:"checksum_type"`
Size int64 `json:"size"`
Successful bool `json:"successful"`
Parts []BackupPart `json:"parts"`
}
type InstallStatusRequest struct {
Successful bool `json:"successful"`
Reinstall bool `json:"reinstall"`
} }

View File

@ -4,6 +4,7 @@ import (
"context" "context"
"fmt" "fmt"
"io" "io"
"mime"
"net" "net"
"net/http" "net/http"
"net/url" "net/url"
@ -13,26 +14,64 @@ import (
"time" "time"
"emperror.dev/errors" "emperror.dev/errors"
"github.com/google/uuid"
"github.com/goccy/go-json" "github.com/goccy/go-json"
"github.com/google/uuid"
"github.com/pterodactyl/wings/server" "github.com/pterodactyl/wings/server"
) )
var client = &http.Client{ var client *http.Client
Timeout: time.Hour * 12,
// Disallow any redirect on an HTTP call. This is a security requirement: do not modify func init() {
// this logic without first ensuring that the new target location IS NOT within the current dialer := &net.Dialer{
// instance's local network. LocalAddr: nil,
// }
// This specific error response just causes the client to not follow the redirect and
// returns the actual redirect response to the caller. Not perfect, but simple and most trnspt := http.DefaultTransport.(*http.Transport).Clone()
// people won't be using URLs that redirect anyways hopefully? trnspt.DialContext = func(ctx context.Context, network, addr string) (net.Conn, error) {
// c, err := dialer.DialContext(ctx, network, addr)
// We'll re-evaluate this down the road if needed. if err != nil {
CheckRedirect: func(req *http.Request, via []*http.Request) error { return nil, errors.WithStack(err)
return http.ErrUseLastResponse }
},
ipStr, _, err := net.SplitHostPort(c.RemoteAddr().String())
if err != nil {
return c, errors.WithStack(err)
}
ip := net.ParseIP(ipStr)
if ip == nil {
return c, errors.WithStack(ErrInvalidIPAddress)
}
if ip.IsLoopback() || ip.IsLinkLocalUnicast() || ip.IsLinkLocalMulticast() || ip.IsInterfaceLocalMulticast() {
return c, errors.WithStack(ErrInternalResolution)
}
for _, block := range internalRanges {
if !block.Contains(ip) {
continue
}
return c, errors.WithStack(ErrInternalResolution)
}
return c, nil
}
client = &http.Client{
Timeout: time.Hour * 12,
Transport: trnspt,
// Disallow any redirect on an HTTP call. This is a security requirement: do not modify
// this logic without first ensuring that the new target location IS NOT within the current
// instance's local network.
//
// This specific error response just causes the client to not follow the redirect and
// returns the actual redirect response to the caller. Not perfect, but simple and most
// people won't be using URLs that redirect anyways hopefully?
//
// We'll re-evaluate this down the road if needed.
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return http.ErrUseLastResponse
},
}
} }
var instance = &Downloader{ var instance = &Downloader{
@ -77,10 +116,13 @@ func (c *Counter) Write(p []byte) (int, error) {
type DownloadRequest struct { type DownloadRequest struct {
Directory string Directory string
URL *url.URL URL *url.URL
FileName string
UseHeader bool
} }
type Download struct { type Download struct {
Identifier string Identifier string
path string
mu sync.RWMutex mu sync.RWMutex
req DownloadRequest req DownloadRequest
server *server.Server server *server.Server
@ -139,12 +181,6 @@ func (dl *Download) Execute() error {
dl.cancelFunc = &cancel dl.cancelFunc = &cancel
defer dl.Cancel() defer dl.Cancel()
// Always ensure that we're checking the destination for the download to avoid a malicious
// user from accessing internal network resources.
if err := dl.isExternalNetwork(ctx); err != nil {
return err
}
// At this point we have verified the destination is not within the local network, so we can // At this point we have verified the destination is not within the local network, so we can
// now make a request to that URL and pull down the file, saving it to the server's data // now make a request to that URL and pull down the file, saving it to the server's data
// directory. // directory.
@ -163,21 +199,38 @@ func (dl *Download) Execute() error {
return errors.New("downloader: got bad response status from endpoint: " + res.Status) return errors.New("downloader: got bad response status from endpoint: " + res.Status)
} }
// If there is a Content-Length header on this request go ahead and check that we can if res.ContentLength < 1 {
// even write the whole file before beginning this process. If there is no header present return errors.New("downloader: request is missing ContentLength")
// we'll just have to give it a spin and see how it goes. }
if res.ContentLength > 0 {
if err := dl.server.Filesystem().HasSpaceFor(res.ContentLength); err != nil { if dl.req.UseHeader {
return errors.WrapIf(err, "downloader: failed to write file: not enough space") if contentDisposition := res.Header.Get("Content-Disposition"); contentDisposition != "" {
_, params, err := mime.ParseMediaType(contentDisposition)
if err != nil {
return errors.WrapIf(err, "downloader: invalid \"Content-Disposition\" header")
}
if v, ok := params["filename"]; ok {
dl.path = v
}
}
}
if dl.path == "" {
if dl.req.FileName != "" {
dl.path = dl.req.FileName
} else {
parts := strings.Split(dl.req.URL.Path, "/")
dl.path = parts[len(parts)-1]
} }
} }
fnameparts := strings.Split(dl.req.URL.Path, "/") p := dl.Path()
p := filepath.Join(dl.req.Directory, fnameparts[len(fnameparts)-1])
dl.server.Log().WithField("path", p).Debug("writing remote file to disk") dl.server.Log().WithField("path", p).Debug("writing remote file to disk")
// Write the file while tracking the progress, Write will check that the
// size of the file won't exceed the disk limit.
r := io.TeeReader(res.Body, dl.counter(res.ContentLength)) r := io.TeeReader(res.Body, dl.counter(res.ContentLength))
if err := dl.server.Filesystem().Writefile(p, r); err != nil { if err := dl.server.Filesystem().Write(p, r, res.ContentLength, 0o644); err != nil {
return errors.WrapIf(err, "downloader: failed to write file to server directory") return errors.WrapIf(err, "downloader: failed to write file to server directory")
} }
return nil return nil
@ -205,6 +258,10 @@ func (dl *Download) Progress() float64 {
return dl.progress return dl.progress
} }
func (dl *Download) Path() string {
return filepath.Join(dl.req.Directory, dl.path)
}
// Handles a write event by updating the progress completed percentage and firing off // Handles a write event by updating the progress completed percentage and firing off
// events to the server websocket as needed. // events to the server websocket as needed.
func (dl *Download) counter(contentLength int64) *Counter { func (dl *Download) counter(contentLength int64) *Counter {
@ -218,59 +275,6 @@ func (dl *Download) counter(contentLength int64) *Counter {
} }
} }
// Verifies that a given download resolves to a location not within the current local
// network for the machine. If the final destination of a resource is within the local
// network an ErrInternalResolution error is returned.
func (dl *Download) isExternalNetwork(ctx context.Context) error {
dialer := &net.Dialer{
LocalAddr: nil,
}
host := dl.req.URL.Host
// This cluster-fuck of math and integer shit converts an integer IP into a proper IPv4.
// For example: 16843009 would become 1.1.1.1
//if i, err := strconv.ParseInt(host, 10, 64); err == nil {
// host = strconv.FormatInt((i>>24)&0xFF, 10) + "." + strconv.FormatInt((i>>16)&0xFF, 10) + "." + strconv.FormatInt((i>>8)&0xFF, 10) + "." + strconv.FormatInt(i&0xFF, 10)
//}
if _, _, err := net.SplitHostPort(host); err != nil {
if !strings.Contains(err.Error(), "missing port in address") {
return errors.WithStack(err)
}
switch dl.req.URL.Scheme {
case "http":
host += ":80"
case "https":
host += ":443"
}
}
c, err := dialer.DialContext(ctx, "tcp", host)
if err != nil {
return errors.WithStack(err)
}
_ = c.Close()
ipStr, _, err := net.SplitHostPort(c.RemoteAddr().String())
if err != nil {
return errors.WithStack(err)
}
ip := net.ParseIP(ipStr)
if ip == nil {
return errors.WithStack(ErrInvalidIPAddress)
}
if ip.IsLoopback() || ip.IsLinkLocalUnicast() || ip.IsLinkLocalMulticast() || ip.IsInterfaceLocalMulticast() {
return errors.WithStack(ErrInternalResolution)
}
for _, block := range internalRanges {
if block.Contains(ip) {
return errors.WithStack(ErrInternalResolution)
}
}
return nil
}
// Downloader represents a global downloader that keeps track of all currently processing downloads // Downloader represents a global downloader that keeps track of all currently processing downloads
// for the machine. // for the machine.
type Downloader struct { type Downloader struct {

View File

@ -1,157 +0,0 @@
package router
import (
"fmt"
"net/http"
"os"
"strings"
"emperror.dev/errors"
"github.com/apex/log"
"github.com/gin-gonic/gin"
"github.com/google/uuid"
"github.com/pterodactyl/wings/server"
"github.com/pterodactyl/wings/server/filesystem"
)
type RequestError struct {
err error
uuid string
message string
server *server.Server
}
// Attaches an error to the gin.Context object for the request and ensures that it
// has a proper stacktrace associated with it when doing so.
//
// If you just call c.Error(err) without using this function you'll likely end up
// with an error that has no annotated stack on it.
func WithError(c *gin.Context, err error) error {
return c.Error(errors.WithStackDepthIf(err, 1))
}
// Generates a new tracked error, which simply tracks the specific error that
// is being passed in, and also assigned a UUID to the error so that it can be
// cross referenced in the logs.
func NewTrackedError(err error) *RequestError {
return &RequestError{
err: err,
uuid: uuid.Must(uuid.NewRandom()).String(),
}
}
// Same as NewTrackedError, except this will also attach the server instance that
// generated this server for the purposes of logging.
func NewServerError(err error, s *server.Server) *RequestError {
return &RequestError{
err: err,
uuid: uuid.Must(uuid.NewRandom()).String(),
server: s,
}
}
func (e *RequestError) logger() *log.Entry {
if e.server != nil {
return e.server.Log().WithField("error_id", e.uuid).WithField("error", e.err)
}
return log.WithField("error_id", e.uuid).WithField("error", e.err)
}
// Sets the output message to display to the user in the error.
func (e *RequestError) SetMessage(msg string) *RequestError {
e.message = msg
return e
}
// Aborts the request with the given status code, and responds with the error. This
// will also include the error UUID in the output so that the user can report that
// and link the response to a specific error in the logs.
func (e *RequestError) AbortWithStatus(status int, c *gin.Context) {
// In instances where the status has already been set just use that existing status
// since we cannot change it at this point, and trying to do so will emit a gin warning
// into the program output.
if c.Writer.Status() != 200 {
status = c.Writer.Status()
}
// If this error is because the resource does not exist, we likely do not need to log
// the error anywhere, just return a 404 and move on with our lives.
if errors.Is(e.err, os.ErrNotExist) {
c.AbortWithStatusJSON(http.StatusNotFound, gin.H{
"error": "The requested resource was not found on the system.",
})
return
}
if strings.HasPrefix(e.err.Error(), "invalid URL escape") {
c.AbortWithStatusJSON(http.StatusBadRequest, gin.H{
"error": "Some of the data provided in the request appears to be escaped improperly.",
})
return
}
// If this is a Filesystem error just return it without all of the tracking code nonsense
// since we don't need to be logging it into the logs or anything, its just a normal error
// that the user can solve on their end.
if st, msg := e.getAsFilesystemError(); st != 0 {
c.AbortWithStatusJSON(st, gin.H{"error": msg})
return
}
// Otherwise, log the error to zap, and then report the error back to the user.
if status >= 500 {
e.logger().Error("unexpected error while handling HTTP request")
} else {
e.logger().Debug("non-server error encountered while handling HTTP request")
}
if e.message == "" {
e.message = "An unexpected error was encountered while processing this request."
}
c.AbortWithStatusJSON(status, gin.H{"error": e.message, "error_id": e.uuid})
}
// Helper function to just abort with an internal server error. This is generally the response
// from most errors encountered by the API.
func (e *RequestError) Abort(c *gin.Context) {
e.AbortWithStatus(http.StatusInternalServerError, c)
}
// Looks at the given RequestError and determines if it is a specific filesystem error that
// we can process and return differently for the user.
func (e *RequestError) getAsFilesystemError() (int, string) {
// Some external things end up calling fmt.Errorf() on our filesystem errors
// which ends up just unleashing chaos on the system. For the sake of this
// fallback to using text checks...
if filesystem.IsErrorCode(e.err, filesystem.ErrCodeDenylistFile) || strings.Contains(e.err.Error(), "filesystem: file access prohibited") {
return http.StatusForbidden, "This file cannot be modified: present in egg denylist."
}
if filesystem.IsErrorCode(e.err, filesystem.ErrCodePathResolution) || strings.Contains(e.err.Error(), "resolves to a location outside the server root") {
return http.StatusNotFound, "The requested resource was not found on the system."
}
if filesystem.IsErrorCode(e.err, filesystem.ErrCodeIsDirectory) || strings.Contains(e.err.Error(), "filesystem: is a directory") {
return http.StatusBadRequest, "Cannot perform that action: file is a directory."
}
if filesystem.IsErrorCode(e.err, filesystem.ErrCodeDiskSpace) || strings.Contains(e.err.Error(), "filesystem: not enough disk space") {
return http.StatusBadRequest, "Cannot perform that action: not enough disk space available."
}
if strings.HasSuffix(e.err.Error(), "file name too long") {
return http.StatusBadRequest, "Cannot perform that action: file name is too long."
}
if e, ok := e.err.(*os.SyscallError); ok && e.Syscall == "readdirent" {
return http.StatusNotFound, "The requested directory does not exist."
}
return 0, ""
}
// Handle specific filesystem errors for a server.
func (e *RequestError) AbortFilesystemError(c *gin.Context) {
e.Abort(c)
}
// Format the error to a string and include the UUID.
func (e *RequestError) Error() string {
return fmt.Sprintf("%v (uuid: %s)", e.err, e.uuid)
}

View File

@ -1,11 +1,9 @@
package middleware package middleware
import ( import (
"context"
"crypto/subtle" "crypto/subtle"
"io" "io"
"net/http" "net/http"
"os"
"strings" "strings"
"emperror.dev/errors" "emperror.dev/errors"
@ -16,133 +14,8 @@ import (
"github.com/pterodactyl/wings/config" "github.com/pterodactyl/wings/config"
"github.com/pterodactyl/wings/remote" "github.com/pterodactyl/wings/remote"
"github.com/pterodactyl/wings/server" "github.com/pterodactyl/wings/server"
"github.com/pterodactyl/wings/server/filesystem"
) )
// RequestError is a custom error type returned when something goes wrong with
// any of the HTTP endpoints.
type RequestError struct {
err error
status int
msg string
}
// NewError returns a new RequestError for the provided error.
func NewError(err error) *RequestError {
return &RequestError{
// Attach a stacktrace to the error if it is missing at this point and mark it
// as originating from the location where NewError was called, rather than this
// specific point in the code.
err: errors.WithStackDepthIf(err, 1),
}
}
// SetMessage allows for a custom error message to be set on an existing
// RequestError instance.
func (re *RequestError) SetMessage(m string) {
re.msg = m
}
// SetStatus sets the HTTP status code for the error response. By default this
// is a HTTP-500 error.
func (re *RequestError) SetStatus(s int) {
re.status = s
}
// Abort aborts the given HTTP request with the specified status code and then
// logs the event into the logs. The error that is output will include the unique
// request ID if it is present.
func (re *RequestError) Abort(c *gin.Context, status int) {
reqId := c.Writer.Header().Get("X-Request-Id")
// Generate the base logger instance, attaching the unique request ID and
// the URL that was requested.
event := log.WithField("request_id", reqId).WithField("url", c.Request.URL.String())
// If there is a server present in the gin.Context stack go ahead and pull it
// and attach that server UUID to the logs as well so that we can see what specific
// server triggered this error.
if s, ok := c.Get("server"); ok {
if s, ok := s.(*server.Server); ok {
event = event.WithField("server_id", s.ID())
}
}
if c.Writer.Status() == 200 {
// Handle context deadlines being exceeded a little differently since we want
// to report a more user-friendly error and a proper error code. The "context
// canceled" error is generally when a request is terminated before all of the
// logic is finished running.
if errors.Is(re.err, context.DeadlineExceeded) {
re.SetStatus(http.StatusGatewayTimeout)
re.SetMessage("The server could not process this request in time, please try again.")
} else if strings.Contains(re.Cause().Error(), "context canceled") {
re.SetStatus(http.StatusBadRequest)
re.SetMessage("Request aborted by client.")
}
}
// c.Writer.Status() will be a non-200 value if the headers have already been sent
// to the requester but an error is encountered. This can happen if there is an issue
// marshaling a struct placed into a c.JSON() call (or c.AbortWithJSON() call).
if status >= 500 || c.Writer.Status() != 200 {
event.WithField("status", status).WithField("error", re.err).Error("error while handling HTTP request")
} else {
event.WithField("status", status).WithField("error", re.err).Debug("error handling HTTP request (not a server error)")
}
if re.msg == "" {
re.msg = "An unexpected error was encountered while processing this request"
}
// Now abort the request with the error message and include the unique request
// ID that was present to make things super easy on people who don't know how
// or cannot view the response headers (where X-Request-Id would be present).
c.AbortWithStatusJSON(status, gin.H{"error": re.msg, "request_id": reqId})
}
// Cause returns the underlying error.
func (re *RequestError) Cause() error {
return re.err
}
// Error returns the underlying error message for this request.
func (re *RequestError) Error() string {
return re.err.Error()
}
// Looks at the given RequestError and determines if it is a specific filesystem
// error that we can process and return differently for the user.
//
// Some external things end up calling fmt.Errorf() on our filesystem errors
// which ends up just unleashing chaos on the system. For the sake of this,
// fallback to using text checks.
//
// If the error passed into this call is nil or does not match empty values will
// be returned to the caller.
func (re *RequestError) asFilesystemError() (int, string) {
err := re.Cause()
if err == nil {
return 0, ""
}
if filesystem.IsErrorCode(err, filesystem.ErrCodeDenylistFile) || strings.Contains(err.Error(), "filesystem: file access prohibited") {
return http.StatusForbidden, "This file cannot be modified: present in egg denylist."
}
if filesystem.IsErrorCode(err, filesystem.ErrCodePathResolution) || strings.Contains(err.Error(), "resolves to a location outside the server root") {
return http.StatusNotFound, "The requested resource was not found on the system."
}
if filesystem.IsErrorCode(err, filesystem.ErrCodeIsDirectory) || strings.Contains(err.Error(), "filesystem: is a directory") {
return http.StatusBadRequest, "Cannot perform that action: file is a directory."
}
if filesystem.IsErrorCode(err, filesystem.ErrCodeDiskSpace) || strings.Contains(err.Error(), "filesystem: not enough disk space") {
return http.StatusBadRequest, "There is not enough disk space available to perform that action."
}
if strings.HasSuffix(err.Error(), "file name too long") {
return http.StatusBadRequest, "Cannot perform that action: file name is too long."
}
if e, ok := err.(*os.SyscallError); ok && e.Syscall == "readdirent" {
return http.StatusNotFound, "The requested directory does not exist."
}
return 0, ""
}
// AttachRequestID attaches a unique ID to the incoming HTTP request so that any // AttachRequestID attaches a unique ID to the incoming HTTP request so that any
// errors that are generated or returned to the client will include this reference // errors that are generated or returned to the client will include this reference
// allowing for an easier time identifying the specific request that failed for // allowing for an easier time identifying the specific request that failed for
@ -180,7 +53,7 @@ func AttachApiClient(client remote.Client) gin.HandlerFunc {
} }
// CaptureAndAbort aborts the request and attaches the provided error to the gin // CaptureAndAbort aborts the request and attaches the provided error to the gin
// context so it can be reported properly. If the error is missing a stacktrace // context, so it can be reported properly. If the error is missing a stacktrace
// at the time it is called the stack will be attached. // at the time it is called the stack will be attached.
func CaptureAndAbort(c *gin.Context, err error) { func CaptureAndAbort(c *gin.Context, err error) {
c.Abort() c.Abort()

View File

@ -0,0 +1,141 @@
package middleware
import (
"context"
"net/http"
"os"
"strings"
"emperror.dev/errors"
"github.com/apex/log"
"github.com/gin-gonic/gin"
"github.com/pterodactyl/wings/server"
"github.com/pterodactyl/wings/server/filesystem"
)
// RequestError is a custom error type returned when something goes wrong with
// any of the HTTP endpoints.
type RequestError struct {
err error
status int
msg string
}
// NewError returns a new RequestError for the provided error.
func NewError(err error) *RequestError {
return &RequestError{
// Attach a stacktrace to the error if it is missing at this point and mark it
// as originating from the location where NewError was called, rather than this
// specific point in the code.
err: errors.WithStackDepthIf(err, 1),
}
}
// SetMessage allows for a custom error message to be set on an existing
// RequestError instance.
func (re *RequestError) SetMessage(m string) {
re.msg = m
}
// SetStatus sets the HTTP status code for the error response. By default this
// is a HTTP-500 error.
func (re *RequestError) SetStatus(s int) {
re.status = s
}
// Abort aborts the given HTTP request with the specified status code and then
// logs the event into the logs. The error that is output will include the unique
// request ID if it is present.
func (re *RequestError) Abort(c *gin.Context, status int) {
reqId := c.Writer.Header().Get("X-Request-Id")
// Generate the base logger instance, attaching the unique request ID and
// the URL that was requested.
event := log.WithField("request_id", reqId).WithField("url", c.Request.URL.String())
// If there is a server present in the gin.Context stack go ahead and pull it
// and attach that server UUID to the logs as well so that we can see what specific
// server triggered this error.
if s, ok := c.Get("server"); ok {
if s, ok := s.(*server.Server); ok {
event = event.WithField("server_id", s.ID())
}
}
if c.Writer.Status() == 200 {
// Handle context deadlines being exceeded a little differently since we want
// to report a more user-friendly error and a proper error code. The "context
// canceled" error is generally when a request is terminated before all of the
// logic is finished running.
if errors.Is(re.err, context.DeadlineExceeded) {
re.SetStatus(http.StatusGatewayTimeout)
re.SetMessage("The server could not process this request in time, please try again.")
} else if strings.Contains(re.Cause().Error(), "context canceled") {
re.SetStatus(http.StatusBadRequest)
re.SetMessage("Request aborted by client.")
}
}
// c.Writer.Status() will be a non-200 value if the headers have already been sent
// to the requester but an error is encountered. This can happen if there is an issue
// marshaling a struct placed into a c.JSON() call (or c.AbortWithJSON() call).
if status >= 500 || c.Writer.Status() != 200 {
event.WithField("status", status).WithField("error", re.err).Error("error while handling HTTP request")
} else {
event.WithField("status", status).WithField("error", re.err).Debug("error handling HTTP request (not a server error)")
}
if re.msg == "" {
re.msg = "An unexpected error was encountered while processing this request"
}
// Now abort the request with the error message and include the unique request
// ID that was present to make things super easy on people who don't know how
// or cannot view the response headers (where X-Request-Id would be present).
c.AbortWithStatusJSON(status, gin.H{"error": re.msg, "request_id": reqId})
}
// Cause returns the underlying error.
func (re *RequestError) Cause() error {
return re.err
}
// Error returns the underlying error message for this request.
func (re *RequestError) Error() string {
return re.err.Error()
}
// Looks at the given RequestError and determines if it is a specific filesystem
// error that we can process and return differently for the user.
//
// Some external things end up calling fmt.Errorf() on our filesystem errors
// which ends up just unleashing chaos on the system. For the sake of this,
// fallback to using text checks.
//
// If the error passed into this call is nil or does not match empty values will
// be returned to the caller.
func (re *RequestError) asFilesystemError() (int, string) {
err := re.Cause()
if err == nil {
return 0, ""
}
if filesystem.IsErrorCode(err, filesystem.ErrNotExist) ||
filesystem.IsErrorCode(err, filesystem.ErrCodePathResolution) ||
strings.Contains(err.Error(), "resolves to a location outside the server root") {
return http.StatusNotFound, "The requested resources was not found on the system."
}
if filesystem.IsErrorCode(err, filesystem.ErrCodeDenylistFile) || strings.Contains(err.Error(), "filesystem: file access prohibited") {
return http.StatusForbidden, "This file cannot be modified: present in egg denylist."
}
if filesystem.IsErrorCode(err, filesystem.ErrCodeIsDirectory) || strings.Contains(err.Error(), "filesystem: is a directory") {
return http.StatusBadRequest, "Cannot perform that action: file is a directory."
}
if filesystem.IsErrorCode(err, filesystem.ErrCodeDiskSpace) || strings.Contains(err.Error(), "filesystem: not enough disk space") {
return http.StatusBadRequest, "There is not enough disk space available to perform that action."
}
if strings.HasSuffix(err.Error(), "file name too long") {
return http.StatusBadRequest, "Cannot perform that action: file name is too long."
}
if e, ok := err.(*os.SyscallError); ok && e.Syscall == "readdirent" {
return http.StatusNotFound, "The requested directory does not exist."
}
return 0, ""
}

View File

@ -1,9 +1,11 @@
package router package router
import ( import (
"emperror.dev/errors"
"github.com/apex/log" "github.com/apex/log"
"github.com/gin-gonic/gin" "github.com/gin-gonic/gin"
"github.com/pterodactyl/wings/config"
"github.com/pterodactyl/wings/remote" "github.com/pterodactyl/wings/remote"
"github.com/pterodactyl/wings/router/middleware" "github.com/pterodactyl/wings/router/middleware"
wserver "github.com/pterodactyl/wings/server" wserver "github.com/pterodactyl/wings/server"
@ -15,6 +17,10 @@ func Configure(m *wserver.Manager, client remote.Client) *gin.Engine {
router := gin.New() router := gin.New()
router.Use(gin.Recovery()) router.Use(gin.Recovery())
if err := router.SetTrustedProxies(config.Get().Api.TrustedProxies); err != nil {
panic(errors.WithStack(err))
return nil
}
router.Use(middleware.AttachRequestID(), middleware.CaptureErrors(), middleware.SetAccessControlHeaders()) router.Use(middleware.AttachRequestID(), middleware.CaptureErrors(), middleware.SetAccessControlHeaders())
router.Use(middleware.AttachServerManager(m), middleware.AttachApiClient(client)) router.Use(middleware.AttachServerManager(m), middleware.AttachApiClient(client))
// @todo log this into a different file so you can setup IP blocking for abusive requests and such. // @todo log this into a different file so you can setup IP blocking for abusive requests and such.
@ -38,7 +44,7 @@ func Configure(m *wserver.Manager, client remote.Client) *gin.Engine {
router.GET("/download/file", getDownloadFile) router.GET("/download/file", getDownloadFile)
router.POST("/upload/file", postServerUploadFiles) router.POST("/upload/file", postServerUploadFiles)
// This route is special it sits above all of the other requests because we are // This route is special it sits above all the other requests because we are
// using a JWT to authorize access to it, therefore it needs to be publicly // using a JWT to authorize access to it, therefore it needs to be publicly
// accessible. // accessible.
router.GET("/api/servers/:server/ws", middleware.ServerExists(), getServerWebsocket) router.GET("/api/servers/:server/ws", middleware.ServerExists(), getServerWebsocket)
@ -46,16 +52,16 @@ func Configure(m *wserver.Manager, client remote.Client) *gin.Engine {
// This request is called by another daemon when a server is going to be transferred out. // This request is called by another daemon when a server is going to be transferred out.
// This request does not need the AuthorizationMiddleware as the panel should never call it // This request does not need the AuthorizationMiddleware as the panel should never call it
// and requests are authenticated through a JWT the panel issues to the other daemon. // and requests are authenticated through a JWT the panel issues to the other daemon.
router.GET("/api/servers/:server/archive", middleware.ServerExists(), getServerArchive) router.POST("/api/transfers", postTransfers)
// All of the routes beyond this mount will use an authorization middleware // All the routes beyond this mount will use an authorization middleware
// and will not be accessible without the correct Authorization header provided. // and will not be accessible without the correct Authorization header provided.
protected := router.Use(middleware.RequireAuthorization()) protected := router.Use(middleware.RequireAuthorization())
protected.POST("/api/update", postUpdateConfiguration) protected.POST("/api/update", postUpdateConfiguration)
protected.GET("/api/system", getSystemInformation) protected.GET("/api/system", getSystemInformation)
protected.GET("/api/servers", getAllServers) protected.GET("/api/servers", getAllServers)
protected.POST("/api/servers", postCreateServer) protected.POST("/api/servers", postCreateServer)
protected.POST("/api/transfer", postTransfer) protected.DELETE("/api/transfers/:server", deleteTransfer)
// These are server specific routes, and require that the request be authorized, and // These are server specific routes, and require that the request be authorized, and
// that the server exist on the Daemon. // that the server exist on the Daemon.
@ -75,7 +81,8 @@ func Configure(m *wserver.Manager, client remote.Client) *gin.Engine {
// This archive request causes the archive to start being created // This archive request causes the archive to start being created
// this should only be triggered by the panel. // this should only be triggered by the panel.
server.POST("/archive", postServerArchive) server.POST("/transfer", postServerTransfer)
server.DELETE("/transfer", deleteServerTransfer)
files := server.Group("/files") files := server.Group("/files")
{ {

View File

@ -8,6 +8,7 @@ import (
"strconv" "strconv"
"github.com/gin-gonic/gin" "github.com/gin-gonic/gin"
"github.com/google/uuid"
"github.com/pterodactyl/wings/router/middleware" "github.com/pterodactyl/wings/router/middleware"
"github.com/pterodactyl/wings/router/tokens" "github.com/pterodactyl/wings/router/tokens"
@ -19,20 +20,29 @@ func getDownloadBackup(c *gin.Context) {
client := middleware.ExtractApiClient(c) client := middleware.ExtractApiClient(c)
manager := middleware.ExtractManager(c) manager := middleware.ExtractManager(c)
// Get the payload from the token.
token := tokens.BackupPayload{} token := tokens.BackupPayload{}
if err := tokens.ParseToken([]byte(c.Query("token")), &token); err != nil { if err := tokens.ParseToken([]byte(c.Query("token")), &token); err != nil {
NewTrackedError(err).Abort(c) middleware.CaptureAndAbort(c, err)
return return
} }
s, ok := manager.Get(token.ServerUuid) // Get the server using the UUID from the token.
if !ok || !token.IsUniqueRequest() { if _, ok := manager.Get(token.ServerUuid); !ok || !token.IsUniqueRequest() {
c.AbortWithStatusJSON(http.StatusNotFound, gin.H{ c.AbortWithStatusJSON(http.StatusNotFound, gin.H{
"error": "The requested resource was not found on this server.", "error": "The requested resource was not found on this server.",
}) })
return return
} }
// Validate that the BackupUuid field is actually a UUID and not some random characters or a
// file path.
if _, err := uuid.Parse(token.BackupUuid); err != nil {
middleware.CaptureAndAbort(c, err)
return
}
// Locate the backup on the local disk.
b, st, err := backup.LocateLocal(client, token.BackupUuid) b, st, err := backup.LocateLocal(client, token.BackupUuid)
if err != nil { if err != nil {
if errors.Is(err, os.ErrNotExist) { if errors.Is(err, os.ErrNotExist) {
@ -42,13 +52,15 @@ func getDownloadBackup(c *gin.Context) {
return return
} }
NewServerError(err, s).Abort(c) middleware.CaptureAndAbort(c, err)
return return
} }
// The use of `os` here is safe as backups are not stored within server
// accessible directories.
f, err := os.Open(b.Path()) f, err := os.Open(b.Path())
if err != nil { if err != nil {
NewServerError(err, s).Abort(c) middleware.CaptureAndAbort(c, err)
return return
} }
defer f.Close() defer f.Close()
@ -57,7 +69,7 @@ func getDownloadBackup(c *gin.Context) {
c.Header("Content-Disposition", "attachment; filename="+strconv.Quote(st.Name())) c.Header("Content-Disposition", "attachment; filename="+strconv.Quote(st.Name()))
c.Header("Content-Type", "application/octet-stream") c.Header("Content-Type", "application/octet-stream")
bufio.NewReader(f).WriteTo(c.Writer) _, _ = bufio.NewReader(f).WriteTo(c.Writer)
} }
// Handles downloading a specific file for a server. // Handles downloading a specific file for a server.
@ -65,7 +77,7 @@ func getDownloadFile(c *gin.Context) {
manager := middleware.ExtractManager(c) manager := middleware.ExtractManager(c)
token := tokens.FilePayload{} token := tokens.FilePayload{}
if err := tokens.ParseToken([]byte(c.Query("token")), &token); err != nil { if err := tokens.ParseToken([]byte(c.Query("token")), &token); err != nil {
NewTrackedError(err).Abort(c) middleware.CaptureAndAbort(c, err)
return return
} }
@ -77,29 +89,22 @@ func getDownloadFile(c *gin.Context) {
return return
} }
p, _ := s.Filesystem().SafePath(token.FilePath) f, st, err := s.Filesystem().File(token.FilePath)
st, err := os.Stat(p)
// If there is an error or we're somehow trying to download a directory, just
// respond with the appropriate error.
if err != nil { if err != nil {
NewServerError(err, s).Abort(c) middleware.CaptureAndAbort(c, err)
return return
} else if st.IsDir() { }
defer f.Close()
if st.IsDir() {
c.AbortWithStatusJSON(http.StatusNotFound, gin.H{ c.AbortWithStatusJSON(http.StatusNotFound, gin.H{
"error": "The requested resource was not found on this server.", "error": "The requested resource was not found on this server.",
}) })
return return
} }
f, err := os.Open(p)
if err != nil {
NewServerError(err, s).Abort(c)
return
}
c.Header("Content-Length", strconv.Itoa(int(st.Size()))) c.Header("Content-Length", strconv.Itoa(int(st.Size())))
c.Header("Content-Disposition", "attachment; filename="+strconv.Quote(st.Name())) c.Header("Content-Disposition", "attachment; filename="+strconv.Quote(st.Name()))
c.Header("Content-Type", "application/octet-stream") c.Header("Content-Type", "application/octet-stream")
bufio.NewReader(f).WriteTo(c.Writer) _, _ = bufio.NewReader(f).WriteTo(c.Writer)
} }

View File

@ -9,10 +9,12 @@ import (
"emperror.dev/errors" "emperror.dev/errors"
"github.com/apex/log" "github.com/apex/log"
"github.com/gin-gonic/gin" "github.com/gin-gonic/gin"
"github.com/pterodactyl/wings/router/downloader" "github.com/pterodactyl/wings/router/downloader"
"github.com/pterodactyl/wings/router/middleware" "github.com/pterodactyl/wings/router/middleware"
"github.com/pterodactyl/wings/router/tokens" "github.com/pterodactyl/wings/router/tokens"
"github.com/pterodactyl/wings/server" "github.com/pterodactyl/wings/server"
"github.com/pterodactyl/wings/server/transfer"
) )
// Returns a single server from the collection of servers. // Returns a single server from the collection of servers.
@ -33,7 +35,7 @@ func getServerLogs(c *gin.Context) {
out, err := s.ReadLogfile(l) out, err := s.ReadLogfile(l)
if err != nil { if err != nil {
NewServerError(err, s).Abort(c) middleware.CaptureAndAbort(c, err)
return return
} }
@ -107,7 +109,7 @@ func postServerCommands(c *gin.Context) {
s := ExtractServer(c) s := ExtractServer(c)
if running, err := s.Environment.IsRunning(c.Request.Context()); err != nil { if running, err := s.Environment.IsRunning(c.Request.Context()); err != nil {
NewServerError(err, s).Abort(c) middleware.CaptureAndAbort(c, err)
return return
} else if !running { } else if !running {
c.AbortWithStatusJSON(http.StatusBadGateway, gin.H{ c.AbortWithStatusJSON(http.StatusBadGateway, gin.H{
@ -141,7 +143,7 @@ func postServerSync(c *gin.Context) {
s := ExtractServer(c) s := ExtractServer(c)
if err := s.Sync(); err != nil { if err := s.Sync(); err != nil {
WithError(c, err) middleware.CaptureAndAbort(c, err)
} else { } else {
c.Status(http.StatusNoContent) c.Status(http.StatusNoContent)
} }
@ -151,9 +153,15 @@ func postServerSync(c *gin.Context) {
func postServerInstall(c *gin.Context) { func postServerInstall(c *gin.Context) {
s := ExtractServer(c) s := ExtractServer(c)
go func(serv *server.Server) { go func(s *server.Server) {
if err := serv.Install(true); err != nil { s.Log().Info("syncing server state with remote source before executing installation process")
serv.Log().WithField("error", err).Error("failed to execute server installation process") if err := s.Sync(); err != nil {
s.Log().WithField("error", err).Error("failed to sync server state with Panel")
return
}
if err := s.Install(); err != nil {
s.Log().WithField("error", err).Error("failed to execute server installation process")
} }
}(s) }(s)
@ -180,13 +188,24 @@ func postServerReinstall(c *gin.Context) {
c.Status(http.StatusAccepted) c.Status(http.StatusAccepted)
} }
// Deletes a server from the wings daemon and dissociate it's objects. // Deletes a server from the wings daemon and dissociate its objects.
func deleteServer(c *gin.Context) { func deleteServer(c *gin.Context) {
s := middleware.ExtractServer(c) s := middleware.ExtractServer(c)
// Immediately suspend the server to prevent a user from attempting // Immediately suspend the server to prevent a user from attempting
// to start it while this process is running. // to start it while this process is running.
s.Config().SetSuspended(true) s.Config().SetSuspended(true)
// Notify all websocket clients that the server is being deleted.
// This is useful for two reasons, one to tell clients not to bother
// retrying to connect to the websocket. And two, for transfers when
// the server has been successfully transferred to another node, and
// the client needs to switch to the new node.
if s.IsTransferring() {
s.Events().Publish(server.TransferStatusEvent, transfer.StatusCompleted)
}
s.Events().Publish(server.DeletedEvent, nil)
s.CleanupForDestroy() s.CleanupForDestroy()
// Remove any pending remote file downloads for the server. // Remove any pending remote file downloads for the server.
@ -198,7 +217,7 @@ func deleteServer(c *gin.Context) {
// forcibly terminate it before removing the container, so we do not need to handle // forcibly terminate it before removing the container, so we do not need to handle
// that here. // that here.
if err := s.Environment.Destroy(); err != nil { if err := s.Environment.Destroy(); err != nil {
WithError(c, err) middleware.CaptureAndAbort(c, err)
return return
} }
@ -206,21 +225,21 @@ func deleteServer(c *gin.Context) {
// done in a separate process since failure is not the end of the world and can be // done in a separate process since failure is not the end of the world and can be
// manually cleaned up after the fact. // manually cleaned up after the fact.
// //
// In addition, servers with large amounts of files can take some time to finish deleting // In addition, servers with large amounts of files can take some time to finish deleting,
// so we don't want to block the HTTP call while waiting on this. // so we don't want to block the HTTP call while waiting on this.
go func(p string) { go func(s *server.Server) {
fs := s.Filesystem()
p := fs.Path()
_ = fs.UnixFS().Close()
if err := os.RemoveAll(p); err != nil { if err := os.RemoveAll(p); err != nil {
log.WithFields(log.Fields{"path": p, "error": err}).Warn("failed to remove server files during deletion process") log.WithFields(log.Fields{"path": p, "error": err}).Warn("failed to remove server files during deletion process")
} }
}(s.Filesystem().Path()) }(s)
middleware.ExtractManager(c).Remove(func(server *server.Server) bool { middleware.ExtractManager(c).Remove(func(server *server.Server) bool {
return server.ID() == s.ID() return server.ID() == s.ID()
}) })
// Deallocate the reference to this server.
s = nil
c.Status(http.StatusNoContent) c.Status(http.StatusNoContent)
} }

View File

@ -18,6 +18,8 @@ import (
"github.com/gin-gonic/gin" "github.com/gin-gonic/gin"
"golang.org/x/sync/errgroup" "golang.org/x/sync/errgroup"
"github.com/pterodactyl/wings/config"
"github.com/pterodactyl/wings/internal/models"
"github.com/pterodactyl/wings/router/downloader" "github.com/pterodactyl/wings/router/downloader"
"github.com/pterodactyl/wings/router/middleware" "github.com/pterodactyl/wings/router/middleware"
"github.com/pterodactyl/wings/router/tokens" "github.com/pterodactyl/wings/router/tokens"
@ -28,13 +30,22 @@ import (
// getServerFileContents returns the contents of a file on the server. // getServerFileContents returns the contents of a file on the server.
func getServerFileContents(c *gin.Context) { func getServerFileContents(c *gin.Context) {
s := middleware.ExtractServer(c) s := middleware.ExtractServer(c)
p := "/" + strings.TrimLeft(c.Query("file"), "/") p := strings.TrimLeft(c.Query("file"), "/")
f, st, err := s.Filesystem().File(p) f, st, err := s.Filesystem().File(p)
if err != nil { if err != nil {
middleware.CaptureAndAbort(c, err) middleware.CaptureAndAbort(c, err)
return return
} }
defer f.Close() defer f.Close()
// Don't allow a named pipe to be opened.
//
// @see https://github.com/pterodactyl/panel/issues/4059
if st.Mode()&os.ModeNamedPipe != 0 {
c.AbortWithStatusJSON(http.StatusBadRequest, gin.H{
"error": "Cannot open files of this type.",
})
return
}
c.Header("X-Mime-Type", st.Mimetype) c.Header("X-Mime-Type", st.Mimetype)
c.Header("Content-Length", strconv.Itoa(int(st.Size()))) c.Header("Content-Length", strconv.Itoa(int(st.Size())))
@ -68,7 +79,7 @@ func getServerListDirectory(c *gin.Context) {
s := ExtractServer(c) s := ExtractServer(c)
dir := c.Query("directory") dir := c.Query("directory")
if stats, err := s.Filesystem().ListDirectory(dir); err != nil { if stats, err := s.Filesystem().ListDirectory(dir); err != nil {
WithError(c, err) middleware.CaptureAndAbort(c, err)
} else { } else {
c.JSON(http.StatusOK, stats) c.JSON(http.StatusOK, stats)
} }
@ -118,8 +129,11 @@ func putServerRenameFiles(c *gin.Context) {
} }
if err := fs.Rename(pf, pt); err != nil { if err := fs.Rename(pf, pt); err != nil {
// Return nil if the error is an is not exists. // Return nil if the error is an is not exists.
// NOTE: os.IsNotExist() does not work if the error is wrapped.
if errors.Is(err, os.ErrNotExist) { if errors.Is(err, os.ErrNotExist) {
s.Log().WithField("error", err).
WithField("from_path", pf).
WithField("to_path", pt).
Warn("failed to rename: source or target does not exist")
return nil return nil
} }
return err return err
@ -137,7 +151,7 @@ func putServerRenameFiles(c *gin.Context) {
return return
} }
NewServerError(err, s).AbortFilesystemError(c) middleware.CaptureAndAbort(c, err)
return return
} }
@ -157,11 +171,11 @@ func postServerCopyFile(c *gin.Context) {
} }
if err := s.Filesystem().IsIgnored(data.Location); err != nil { if err := s.Filesystem().IsIgnored(data.Location); err != nil {
NewServerError(err, s).Abort(c) middleware.CaptureAndAbort(c, err)
return return
} }
if err := s.Filesystem().Copy(data.Location); err != nil { if err := s.Filesystem().Copy(data.Location); err != nil {
NewServerError(err, s).AbortFilesystemError(c) middleware.CaptureAndAbort(c, err)
return return
} }
@ -206,7 +220,7 @@ func postServerDeleteFiles(c *gin.Context) {
} }
if err := g.Wait(); err != nil { if err := g.Wait(); err != nil {
NewServerError(err, s).Abort(c) middleware.CaptureAndAbort(c, err)
return return
} }
@ -221,10 +235,19 @@ func postServerWriteFile(c *gin.Context) {
f = "/" + strings.TrimLeft(f, "/") f = "/" + strings.TrimLeft(f, "/")
if err := s.Filesystem().IsIgnored(f); err != nil { if err := s.Filesystem().IsIgnored(f); err != nil {
NewServerError(err, s).Abort(c) middleware.CaptureAndAbort(c, err)
return return
} }
if err := s.Filesystem().Writefile(f, c.Request.Body); err != nil {
// A content length of -1 means the actual length is unknown.
if c.Request.ContentLength == -1 {
c.AbortWithStatusJSON(http.StatusBadRequest, gin.H{
"error": "Missing Content-Length",
})
return
}
if err := s.Filesystem().Write(f, c.Request.Body, c.Request.ContentLength, 0o644); err != nil {
if filesystem.IsErrorCode(err, filesystem.ErrCodeIsDirectory) { if filesystem.IsErrorCode(err, filesystem.ErrCodeIsDirectory) {
c.AbortWithStatusJSON(http.StatusBadRequest, gin.H{ c.AbortWithStatusJSON(http.StatusBadRequest, gin.H{
"error": "Cannot write file, name conflicts with an existing directory by the same name.", "error": "Cannot write file, name conflicts with an existing directory by the same name.",
@ -232,7 +255,7 @@ func postServerWriteFile(c *gin.Context) {
return return
} }
NewServerError(err, s).AbortFilesystemError(c) middleware.CaptureAndAbort(c, err)
return return
} }
@ -255,9 +278,12 @@ func postServerPullRemoteFile(c *gin.Context) {
s := ExtractServer(c) s := ExtractServer(c)
var data struct { var data struct {
// Deprecated // Deprecated
Directory string `binding:"required_without=RootPath,omitempty" json:"directory"` Directory string `binding:"required_without=RootPath,omitempty" json:"directory"`
RootPath string `binding:"required_without=Directory,omitempty" json:"root"` RootPath string `binding:"required_without=Directory,omitempty" json:"root"`
URL string `binding:"required" json:"url"` URL string `binding:"required" json:"url"`
FileName string `json:"file_name"`
UseHeader bool `json:"use_header"`
Foreground bool `json:"foreground"`
} }
if err := c.BindJSON(&data); err != nil { if err := c.BindJSON(&data); err != nil {
return return
@ -276,12 +302,12 @@ func postServerPullRemoteFile(c *gin.Context) {
}) })
return return
} }
WithError(c, err) middleware.CaptureAndAbort(c, err)
return return
} }
if err := s.Filesystem().HasSpaceErr(true); err != nil { if err := s.Filesystem().HasSpaceErr(true); err != nil {
WithError(c, err) middleware.CaptureAndAbort(c, err)
return return
} }
// Do not allow more than three simultaneous remote file downloads at one time. // Do not allow more than three simultaneous remote file downloads at one time.
@ -295,21 +321,41 @@ func postServerPullRemoteFile(c *gin.Context) {
dl := downloader.New(s, downloader.DownloadRequest{ dl := downloader.New(s, downloader.DownloadRequest{
Directory: data.RootPath, Directory: data.RootPath,
URL: u, URL: u,
FileName: data.FileName,
UseHeader: data.UseHeader,
}) })
// Execute this pull in a separate thread since it may take a long time to complete. download := func() error {
go func() {
s.Log().WithField("download_id", dl.Identifier).WithField("url", u.String()).Info("starting pull of remote file to disk") s.Log().WithField("download_id", dl.Identifier).WithField("url", u.String()).Info("starting pull of remote file to disk")
if err := dl.Execute(); err != nil { if err := dl.Execute(); err != nil {
s.Log().WithField("download_id", dl.Identifier).WithField("error", err).Error("failed to pull remote file") s.Log().WithField("download_id", dl.Identifier).WithField("error", err).Error("failed to pull remote file")
return err
} else { } else {
s.Log().WithField("download_id", dl.Identifier).Info("completed pull of remote file") s.Log().WithField("download_id", dl.Identifier).Info("completed pull of remote file")
} }
}() return nil
}
if !data.Foreground {
go func() {
_ = download()
}()
c.JSON(http.StatusAccepted, gin.H{
"identifier": dl.Identifier,
})
return
}
c.JSON(http.StatusAccepted, gin.H{ if err := download(); err != nil {
"identifier": dl.Identifier, middleware.CaptureAndAbort(c, err)
}) return
}
st, err := s.Filesystem().Stat(dl.Path())
if err != nil {
middleware.CaptureAndAbort(c, err)
return
}
c.JSON(http.StatusOK, &st)
} }
// Stops a remote file download if it exists and belongs to this server. // Stops a remote file download if it exists and belongs to this server.
@ -342,7 +388,7 @@ func postServerCreateDirectory(c *gin.Context) {
return return
} }
NewServerError(err, s).Abort(c) middleware.CaptureAndAbort(c, err)
return return
} }
@ -377,7 +423,7 @@ func postServerCompressFiles(c *gin.Context) {
f, err := s.Filesystem().CompressFiles(data.RootPath, data.Files) f, err := s.Filesystem().CompressFiles(data.RootPath, data.Files)
if err != nil { if err != nil {
NewServerError(err, s).AbortFilesystemError(c) middleware.CaptureAndAbort(c, err)
return return
} }
@ -402,7 +448,7 @@ func postServerDecompressFiles(c *gin.Context) {
s := middleware.ExtractServer(c) s := middleware.ExtractServer(c)
lg := middleware.ExtractLogger(c).WithFields(log.Fields{"root_path": data.RootPath, "file": data.File}) lg := middleware.ExtractLogger(c).WithFields(log.Fields{"root_path": data.RootPath, "file": data.File})
lg.Debug("checking if space is available for file decompression") lg.Debug("checking if space is available for file decompression")
err := s.Filesystem().SpaceAvailableForDecompression(data.RootPath, data.File) err := s.Filesystem().SpaceAvailableForDecompression(context.Background(), data.RootPath, data.File)
if err != nil { if err != nil {
if filesystem.IsErrorCode(err, filesystem.ErrCodeUnknownArchive) { if filesystem.IsErrorCode(err, filesystem.ErrCodeUnknownArchive) {
lg.WithField("error", err).Warn("failed to decompress file: unknown archive format") lg.WithField("error", err).Warn("failed to decompress file: unknown archive format")
@ -414,7 +460,7 @@ func postServerDecompressFiles(c *gin.Context) {
} }
lg.Info("starting file decompression") lg.Info("starting file decompression")
if err := s.Filesystem().DecompressFile(data.RootPath, data.File); err != nil { if err := s.Filesystem().DecompressFile(context.Background(), data.RootPath, data.File); err != nil {
// If the file is busy for some reason just return a nicer error to the user since there is not // If the file is busy for some reason just return a nicer error to the user since there is not
// much we specifically can do. They'll need to stop the running server process in order to overwrite // much we specifically can do. They'll need to stop the running server process in order to overwrite
// a file like this. // a file like this.
@ -495,7 +541,7 @@ func postServerChmodFile(c *gin.Context) {
return return
} }
NewServerError(err, s).AbortFilesystemError(c) middleware.CaptureAndAbort(c, err)
return return
} }
@ -507,7 +553,7 @@ func postServerUploadFiles(c *gin.Context) {
token := tokens.UploadPayload{} token := tokens.UploadPayload{}
if err := tokens.ParseToken([]byte(c.Query("token")), &token); err != nil { if err := tokens.ParseToken([]byte(c.Query("token")), &token); err != nil {
NewTrackedError(err).Abort(c) middleware.CaptureAndAbort(c, err)
return return
} }
@ -537,23 +583,30 @@ func postServerUploadFiles(c *gin.Context) {
directory := c.Query("directory") directory := c.Query("directory")
maxFileSize := config.Get().Api.UploadLimit
maxFileSizeBytes := maxFileSize * 1024 * 1024
var totalSize int64 var totalSize int64
for _, header := range headers { for _, header := range headers {
if header.Size > maxFileSizeBytes {
c.AbortWithStatusJSON(http.StatusBadRequest, gin.H{
"error": "File " + header.Filename + " is larger than the maximum file upload size of " + strconv.FormatInt(maxFileSize, 10) + " MB.",
})
return
}
totalSize += header.Size totalSize += header.Size
} }
for _, header := range headers { for _, header := range headers {
p, err := s.Filesystem().SafePath(filepath.Join(directory, header.Filename))
if err != nil {
NewServerError(err, s).Abort(c)
return
}
// We run this in a different method so I can use defer without any of // We run this in a different method so I can use defer without any of
// the consequences caused by calling it in a loop. // the consequences caused by calling it in a loop.
if err := handleFileUpload(p, s, header); err != nil { if err := handleFileUpload(filepath.Join(directory, header.Filename), s, header); err != nil {
NewServerError(err, s).Abort(c) middleware.CaptureAndAbort(c, err)
return return
} else {
s.SaveActivity(s.NewRequestActivity(token.UserUuid, c.ClientIP()), server.ActivityFileUploaded, models.ActivityMeta{
"file": header.Filename,
"directory": filepath.Clean(directory),
})
} }
} }
} }
@ -568,9 +621,9 @@ func handleFileUpload(p string, s *server.Server, header *multipart.FileHeader)
if err := s.Filesystem().IsIgnored(p); err != nil { if err := s.Filesystem().IsIgnored(p); err != nil {
return err return err
} }
if err := s.Filesystem().Writefile(p, file); err != nil {
if err := s.Filesystem().Write(p, file, header.Size, 0o644); err != nil {
return err return err
} }
return nil return nil
} }

View File

@ -0,0 +1,130 @@
package router
import (
"context"
"net/http"
"strings"
"time"
"emperror.dev/errors"
"github.com/gin-gonic/gin"
"github.com/pterodactyl/wings/environment"
"github.com/pterodactyl/wings/router/middleware"
"github.com/pterodactyl/wings/server"
"github.com/pterodactyl/wings/server/installer"
"github.com/pterodactyl/wings/server/transfer"
)
// Data passed over to initiate a server transfer.
type serverTransferRequest struct {
URL string `binding:"required" json:"url"`
Token string `binding:"required" json:"token"`
Server installer.ServerDetails `json:"server"`
}
// postServerTransfer handles the start of a transfer for a server.
func postServerTransfer(c *gin.Context) {
var data serverTransferRequest
if err := c.BindJSON(&data); err != nil {
return
}
s := ExtractServer(c)
// Check if the server is already being transferred.
// There will be another endpoint for resetting this value either by deleting the
// server, or by canceling the transfer.
if s.IsTransferring() {
c.AbortWithStatusJSON(http.StatusConflict, gin.H{
"error": "A transfer is already in progress for this server.",
})
return
}
manager := middleware.ExtractManager(c)
notifyPanelOfFailure := func() {
if err := manager.Client().SetTransferStatus(context.Background(), s.ID(), false); err != nil {
s.Log().WithField("subsystem", "transfer").
WithField("status", false).
WithError(err).
Error("failed to set transfer status")
}
s.Events().Publish(server.TransferStatusEvent, "failure")
s.SetTransferring(false)
}
// Block the server from starting while we are transferring it.
s.SetTransferring(true)
// Ensure the server is offline. Sometimes a "No such container" error gets through
// which means the server is already stopped. We can ignore that.
if s.Environment.State() != environment.ProcessOfflineState {
if err := s.Environment.WaitForStop(
s.Context(),
time.Second*15,
false,
); err != nil && !strings.Contains(strings.ToLower(err.Error()), "no such container") {
s.SetTransferring(false)
middleware.CaptureAndAbort(c, errors.Wrap(err, "failed to stop server for transfer"))
return
}
}
// Create a new transfer instance for this server.
trnsfr := transfer.New(context.Background(), s)
transfer.Outgoing().Add(trnsfr)
go func() {
defer transfer.Outgoing().Remove(trnsfr)
if _, err := trnsfr.PushArchiveToTarget(data.URL, data.Token); err != nil {
notifyPanelOfFailure()
if err == context.Canceled {
trnsfr.Log().Debug("canceled")
trnsfr.SendMessage("Canceled.")
return
}
trnsfr.Log().WithError(err).Error("failed to push archive to target")
return
}
// DO NOT NOTIFY THE PANEL OF SUCCESS HERE. The only node that should send
// a success status is the destination node. When we send a failure status,
// the panel will automatically cancel the transfer and attempt to reset
// the server state on the destination node, we just need to make sure
// we clean up our statuses for failure.
trnsfr.Log().Debug("transfer complete")
}()
c.Status(http.StatusAccepted)
}
// deleteServerTransfer cancels an outgoing transfer for a server.
func deleteServerTransfer(c *gin.Context) {
s := ExtractServer(c)
if !s.IsTransferring() {
c.AbortWithStatusJSON(http.StatusConflict, gin.H{
"error": "Server is not currently being transferred.",
})
return
}
trnsfr := transfer.Outgoing().Get(s.ID())
if trnsfr == nil {
c.AbortWithStatusJSON(http.StatusConflict, gin.H{
"error": "Server is not currently being transferred.",
})
return
}
trnsfr.Cancel()
c.Status(http.StatusAccepted)
}

View File

@ -5,8 +5,8 @@ import (
"time" "time"
"github.com/gin-gonic/gin" "github.com/gin-gonic/gin"
ws "github.com/gorilla/websocket"
"github.com/goccy/go-json" "github.com/goccy/go-json"
ws "github.com/gorilla/websocket"
"github.com/pterodactyl/wings/router/middleware" "github.com/pterodactyl/wings/router/middleware"
"github.com/pterodactyl/wings/router/websocket" "github.com/pterodactyl/wings/router/websocket"
@ -32,9 +32,9 @@ func getServerWebsocket(c *gin.Context) {
ctx, cancel := context.WithCancel(c.Request.Context()) ctx, cancel := context.WithCancel(c.Request.Context())
defer cancel() defer cancel()
handler, err := websocket.GetHandler(s, c.Writer, c.Request) handler, err := websocket.GetHandler(s, c.Writer, c.Request, c)
if err != nil { if err != nil {
NewServerError(err, s).Abort(c) middleware.CaptureAndAbort(c, err)
return return
} }
defer handler.Connection.Close() defer handler.Connection.Close()
@ -58,7 +58,7 @@ func getServerWebsocket(c *gin.Context) {
case <-ctx.Done(): case <-ctx.Done():
break break
case <-s.Context().Done(): case <-s.Context().Done():
handler.Connection.WriteControl(ws.CloseMessage, ws.FormatCloseMessage(ws.CloseGoingAway, "server deleted"), time.Now().Add(time.Second*5)) _ = handler.Connection.WriteControl(ws.CloseMessage, ws.FormatCloseMessage(ws.CloseGoingAway, "server deleted"), time.Now().Add(time.Second*5))
break break
} }
}() }()
@ -83,7 +83,7 @@ func getServerWebsocket(c *gin.Context) {
go func(msg websocket.Message) { go func(msg websocket.Message) {
if err := handler.HandleInbound(ctx, msg); err != nil { if err := handler.HandleInbound(ctx, msg); err != nil {
handler.SendErrorJson(msg, err) _ = handler.SendErrorJson(msg, err)
} }
}(j) }(j)
} }

View File

@ -10,9 +10,9 @@ import (
"github.com/gin-gonic/gin" "github.com/gin-gonic/gin"
"github.com/pterodactyl/wings/config" "github.com/pterodactyl/wings/config"
"github.com/pterodactyl/wings/installer"
"github.com/pterodactyl/wings/router/middleware" "github.com/pterodactyl/wings/router/middleware"
"github.com/pterodactyl/wings/server" "github.com/pterodactyl/wings/server"
"github.com/pterodactyl/wings/server/installer"
"github.com/pterodactyl/wings/system" "github.com/pterodactyl/wings/system"
) )
@ -20,15 +20,31 @@ import (
func getSystemInformation(c *gin.Context) { func getSystemInformation(c *gin.Context) {
i, err := system.GetSystemInformation() i, err := system.GetSystemInformation()
if err != nil { if err != nil {
NewTrackedError(err).Abort(c) middleware.CaptureAndAbort(c, err)
return return
} }
c.JSON(http.StatusOK, i) if c.Query("v") == "2" {
c.JSON(http.StatusOK, i)
return
}
c.JSON(http.StatusOK, struct {
Architecture string `json:"architecture"`
CPUCount int `json:"cpu_count"`
KernelVersion string `json:"kernel_version"`
OS string `json:"os"`
Version string `json:"version"`
}{
Architecture: i.System.Architecture,
CPUCount: i.System.CPUThreads,
KernelVersion: i.System.KernelVersion,
OS: i.System.OSType,
Version: i.Version,
})
} }
// Returns all of the servers that are registered and configured correctly on // Returns all the servers that are registered and configured correctly on
// this wings instance. // this wings instance.
func getAllServers(c *gin.Context) { func getAllServers(c *gin.Context) {
servers := middleware.ExtractManager(c).All() servers := middleware.ExtractManager(c).All()
@ -75,7 +91,7 @@ func postCreateServer(c *gin.Context) {
return return
} }
if err := i.Server().Install(false); err != nil { if err := i.Server().Install(); err != nil {
log.WithFields(log.Fields{"server": i.Server().ID(), "error": err}).Error("failed to run install process for server") log.WithFields(log.Fields{"server": i.Server().ID(), "error": err}).Error("failed to run install process for server")
return return
} }
@ -97,9 +113,21 @@ func postCreateServer(c *gin.Context) {
c.Status(http.StatusAccepted) c.Status(http.StatusAccepted)
} }
type postUpdateConfigurationResponse struct {
Applied bool `json:"applied"`
}
// Updates the running configuration for this Wings instance. // Updates the running configuration for this Wings instance.
func postUpdateConfiguration(c *gin.Context) { func postUpdateConfiguration(c *gin.Context) {
cfg := config.Get() cfg := config.Get()
if cfg.IgnorePanelConfigUpdates {
c.JSON(http.StatusOK, postUpdateConfigurationResponse{
Applied: false,
})
return
}
if err := c.BindJSON(&cfg); err != nil { if err := c.BindJSON(&cfg); err != nil {
return return
} }
@ -117,11 +145,13 @@ func postUpdateConfiguration(c *gin.Context) {
// Try to write this new configuration to the disk before updating our global // Try to write this new configuration to the disk before updating our global
// state with it. // state with it.
if err := config.WriteToDisk(cfg); err != nil { if err := config.WriteToDisk(cfg); err != nil {
WithError(c, err) middleware.CaptureAndAbort(c, err)
return return
} }
// Since we wrote it to the disk successfully now update the global configuration // Since we wrote it to the disk successfully now update the global configuration
// state to use this new configuration struct. // state to use this new configuration struct.
config.Set(cfg) config.Set(cfg)
c.Status(http.StatusNoContent) c.JSON(http.StatusOK, postUpdateConfigurationResponse{
Applied: true,
})
} }

View File

@ -1,65 +1,33 @@
package router package router
import ( import (
"bufio" "bytes"
"context" "context"
"crypto/sha256" "crypto/sha256"
"encoding/hex" "encoding/hex"
"errors"
"fmt" "fmt"
"io" "io"
"mime"
"mime/multipart"
"net/http" "net/http"
"os" "os"
"path/filepath"
"strconv"
"strings" "strings"
"sync/atomic"
"time"
"emperror.dev/errors"
"github.com/apex/log" "github.com/apex/log"
"github.com/gin-gonic/gin" "github.com/gin-gonic/gin"
"github.com/google/uuid" "github.com/google/uuid"
"github.com/juju/ratelimit"
"github.com/mholt/archiver/v3"
"github.com/mitchellh/colorstring"
"github.com/pterodactyl/wings/config"
"github.com/pterodactyl/wings/installer"
"github.com/pterodactyl/wings/remote"
"github.com/pterodactyl/wings/router/middleware" "github.com/pterodactyl/wings/router/middleware"
"github.com/pterodactyl/wings/router/tokens" "github.com/pterodactyl/wings/router/tokens"
"github.com/pterodactyl/wings/server" "github.com/pterodactyl/wings/server"
"github.com/pterodactyl/wings/server/filesystem" "github.com/pterodactyl/wings/server/installer"
"github.com/pterodactyl/wings/system" "github.com/pterodactyl/wings/server/transfer"
) )
// Number of ticks in the progress bar // postTransfers .
const ticks = 25 func postTransfers(c *gin.Context) {
// 100% / number of ticks = percentage represented by each tick
const tickPercentage = 100 / ticks
type downloadProgress struct {
size int64
progress int64
}
// Data passed over to initiate a server transfer.
type serverTransferRequest struct {
ServerID string `binding:"required" json:"server_id"`
URL string `binding:"required" json:"url"`
Token string `binding:"required" json:"token"`
Server installer.ServerDetails `json:"server"`
}
func getArchivePath(sID string) string {
return filepath.Join(config.Get().System.ArchiveDirectory, sID+".tar.gz")
}
// Returns the archive for a server so that it can be transferred to a new node.
func getServerArchive(c *gin.Context) {
auth := strings.SplitN(c.GetHeader("Authorization"), " ", 2) auth := strings.SplitN(c.GetHeader("Authorization"), " ", 2)
if len(auth) != 2 || auth[0] != "Bearer" { if len(auth) != 2 || auth[0] != "Bearer" {
c.Header("WWW-Authenticate", "Bearer") c.Header("WWW-Authenticate", "Bearer")
c.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{ c.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{
@ -70,441 +38,237 @@ func getServerArchive(c *gin.Context) {
token := tokens.TransferPayload{} token := tokens.TransferPayload{}
if err := tokens.ParseToken([]byte(auth[1]), &token); err != nil { if err := tokens.ParseToken([]byte(auth[1]), &token); err != nil {
NewTrackedError(err).Abort(c) middleware.CaptureAndAbort(c, err)
return return
} }
manager := middleware.ExtractManager(c)
u, err := uuid.Parse(token.Subject)
if err != nil {
middleware.CaptureAndAbort(c, err)
return
}
// Get or create a new transfer instance for this server.
var (
ctx context.Context
cancel context.CancelFunc
)
trnsfr := transfer.Incoming().Get(u.String())
if trnsfr == nil {
// TODO: should this use the request context?
trnsfr = transfer.New(c, nil)
ctx, cancel = context.WithCancel(trnsfr.Context())
defer cancel()
i, err := installer.New(ctx, manager, installer.ServerDetails{
UUID: u.String(),
StartOnCompletion: false,
})
if err != nil {
if err := manager.Client().SetTransferStatus(context.Background(), trnsfr.Server.ID(), false); err != nil {
trnsfr.Log().WithField("status", false).WithError(err).Error("failed to set transfer status")
}
middleware.CaptureAndAbort(c, err)
return
}
i.Server().SetTransferring(true)
manager.Add(i.Server())
// We add the transfer to the list of transfers once we have a server instance to use.
trnsfr.Server = i.Server()
transfer.Incoming().Add(trnsfr)
} else {
ctx, cancel = context.WithCancel(trnsfr.Context())
defer cancel()
}
// Any errors past this point (until the transfer is complete) will abort
// the transfer.
successful := false
defer func(ctx context.Context, trnsfr *transfer.Transfer) {
// Remove the transfer from the list of incoming transfers.
transfer.Incoming().Remove(trnsfr)
if !successful {
trnsfr.Server.Events().Publish(server.TransferStatusEvent, "failure")
manager.Remove(func(match *server.Server) bool {
return match.ID() == trnsfr.Server.ID()
})
}
if err := manager.Client().SetTransferStatus(context.Background(), trnsfr.Server.ID(), successful); err != nil {
// Only delete the files if the transfer actually failed, otherwise we could have
// unrecoverable data-loss.
if !successful && err != nil {
// Delete all extracted files.
go func(trnsfr *transfer.Transfer) {
_ = trnsfr.Server.Filesystem().UnixFS().Close()
if err := os.RemoveAll(trnsfr.Server.Filesystem().Path()); err != nil && !os.IsNotExist(err) {
trnsfr.Log().WithError(err).Warn("failed to delete local server files")
}
}(trnsfr)
}
trnsfr.Log().WithField("status", successful).WithError(err).Error("failed to set transfer status on panel")
return
}
trnsfr.Server.SetTransferring(false)
trnsfr.Server.Events().Publish(server.TransferStatusEvent, "success")
}(ctx, trnsfr)
mediaType, params, err := mime.ParseMediaType(c.GetHeader("Content-Type"))
if err != nil {
trnsfr.Log().Debug("failed to parse content type header")
middleware.CaptureAndAbort(c, err)
return
}
if !strings.HasPrefix(mediaType, "multipart/") {
trnsfr.Log().Debug("invalid content type")
middleware.CaptureAndAbort(c, fmt.Errorf("invalid content type \"%s\", expected \"multipart/form-data\"", mediaType))
return
}
// Used to calculate the hash of the file as it is being uploaded.
h := sha256.New()
// Used to read the file and checksum from the request body.
mr := multipart.NewReader(c.Request.Body, params["boundary"])
// Loop through the parts of the request body and process them.
var (
hasArchive bool
hasChecksum bool
checksumVerified bool
)
out:
for {
select {
case <-ctx.Done():
break out
default:
p, err := mr.NextPart()
if err == io.EOF {
break out
}
if err != nil {
middleware.CaptureAndAbort(c, err)
return
}
name := p.FormName()
switch name {
case "archive":
trnsfr.Log().Debug("received archive")
if err := trnsfr.Server.EnsureDataDirectoryExists(); err != nil {
middleware.CaptureAndAbort(c, err)
return
}
tee := io.TeeReader(p, h)
if err := trnsfr.Server.Filesystem().ExtractStreamUnsafe(ctx, "/", tee); err != nil {
middleware.CaptureAndAbort(c, err)
return
}
hasArchive = true
case "checksum":
trnsfr.Log().Debug("received checksum")
if !hasArchive {
middleware.CaptureAndAbort(c, errors.New("archive must be sent before the checksum"))
return
}
hasChecksum = true
v, err := io.ReadAll(p)
if err != nil {
middleware.CaptureAndAbort(c, err)
return
}
expected := make([]byte, hex.DecodedLen(len(v)))
n, err := hex.Decode(expected, v)
if err != nil {
middleware.CaptureAndAbort(c, err)
return
}
actual := h.Sum(nil)
trnsfr.Log().WithFields(log.Fields{
"expected": hex.EncodeToString(expected),
"actual": hex.EncodeToString(actual),
}).Debug("checksums")
if !bytes.Equal(expected[:n], actual) {
middleware.CaptureAndAbort(c, errors.New("checksums don't match"))
return
}
trnsfr.Log().Debug("checksums match")
checksumVerified = true
default:
continue
}
}
}
if !hasArchive || !hasChecksum {
middleware.CaptureAndAbort(c, errors.New("missing archive or checksum"))
return
}
if !checksumVerified {
middleware.CaptureAndAbort(c, errors.New("checksums don't match"))
return
}
// Transfer is almost complete, we just want to ensure the environment is
// configured correctly. We might want to not fail the transfer at this
// stage, but we will just to be safe.
// Ensure the server environment gets configured.
if err := trnsfr.Server.CreateEnvironment(); err != nil {
middleware.CaptureAndAbort(c, err)
return
}
// Changing this causes us to notify the panel about a successful transfer,
// rather than failing the transfer like we do by default.
successful = true
// The rest of the logic for ensuring the server is unlocked and everything
// is handled in the deferred function above.
trnsfr.Log().Debug("done!")
}
// deleteTransfer cancels an incoming transfer for a server.
func deleteTransfer(c *gin.Context) {
s := ExtractServer(c) s := ExtractServer(c)
if token.Subject != s.ID() {
c.AbortWithStatusJSON(http.StatusForbidden, gin.H{ if !s.IsTransferring() {
"error": "Missing required token subject, or subject is not valid for the requested server.", c.AbortWithStatusJSON(http.StatusConflict, gin.H{
"error": "Server is not currently being transferred.",
}) })
return return
} }
archivePath := getArchivePath(s.ID()) trnsfr := transfer.Incoming().Get(s.ID())
if trnsfr == nil {
// Stat the archive file. c.AbortWithStatusJSON(http.StatusConflict, gin.H{
st, err := os.Lstat(archivePath) "error": "Server is not currently being transferred.",
if err != nil { })
if !errors.Is(err, os.ErrNotExist) {
_ = WithError(c, err)
return
}
c.AbortWithStatus(http.StatusNotFound)
return return
} }
// Compute sha1 checksum. trnsfr.Cancel()
h := sha256.New()
f, err := os.Open(archivePath)
if err != nil {
return
}
if _, err := io.Copy(h, bufio.NewReader(f)); err != nil {
_ = f.Close()
_ = WithError(c, err)
return
}
if err := f.Close(); err != nil {
_ = WithError(c, err)
return
}
checksum := hex.EncodeToString(h.Sum(nil))
// Stream the file to the client.
f, err = os.Open(archivePath)
if err != nil {
_ = WithError(c, err)
return
}
defer f.Close()
c.Header("X-Checksum", checksum)
c.Header("X-Mime-Type", "application/tar+gzip")
c.Header("Content-Length", strconv.Itoa(int(st.Size())))
c.Header("Content-Disposition", "attachment; filename="+strconv.Quote(s.ID()+".tar.gz"))
c.Header("Content-Type", "application/octet-stream")
_, _ = bufio.NewReader(f).WriteTo(c.Writer)
}
func postServerArchive(c *gin.Context) {
s := middleware.ExtractServer(c)
manager := middleware.ExtractManager(c)
go func(s *server.Server) {
l := log.WithField("server", s.ID())
// This function automatically adds the Source Node prefix and Timestamp to the log
// output before sending it over the websocket.
sendTransferLog := func(data string) {
output := colorstring.Color(fmt.Sprintf("[yellow][bold]%s [Pterodactyl Transfer System] [Source Node]:[default] %s", time.Now().Format(time.RFC1123), data))
s.Events().Publish(server.TransferLogsEvent, output)
}
s.Events().Publish(server.TransferStatusEvent, "starting")
sendTransferLog("Attempting to archive server...")
hasError := true
defer func() {
if !hasError {
return
}
// Mark the server as not being transferred so it can actually be used.
s.SetTransferring(false)
s.Events().Publish(server.TransferStatusEvent, "failure")
sendTransferLog("Attempting to notify panel of archive failure..")
if err := manager.Client().SetArchiveStatus(s.Context(), s.ID(), false); err != nil {
if !remote.IsRequestError(err) {
sendTransferLog("Failed to notify panel of archive failure: " + err.Error())
l.WithField("error", err).Error("failed to notify panel of failed archive status")
return
}
sendTransferLog("Panel returned an error while notifying it of a failed archive: " + err.Error())
l.WithField("error", err.Error()).Error("panel returned an error when notifying it of a failed archive status")
return
}
sendTransferLog("Successfully notified panel of failed archive status")
l.Info("successfully notified panel of failed archive status")
}()
// Mark the server as transferring to prevent problems.
s.SetTransferring(true)
// Ensure the server is offline. Sometimes a "No such container" error gets through
// which means the server is already stopped. We can ignore that.
if err := s.Environment.WaitForStop(60, false); err != nil && !strings.Contains(strings.ToLower(err.Error()), "no such container") {
sendTransferLog("Failed to stop server, aborting transfer..")
l.WithField("error", err).Error("failed to stop server")
return
}
// Create an archive of the entire server's data directory.
a := &filesystem.Archive{
BasePath: s.Filesystem().Path(),
}
// Attempt to get an archive of the server.
if err := a.Create(getArchivePath(s.ID())); err != nil {
sendTransferLog("An error occurred while archiving the server: " + err.Error())
l.WithField("error", err).Error("failed to get transfer archive for server")
return
}
sendTransferLog("Successfully created archive, attempting to notify panel..")
l.Info("successfully created server transfer archive, notifying panel..")
if err := manager.Client().SetArchiveStatus(s.Context(), s.ID(), true); err != nil {
if !remote.IsRequestError(err) {
sendTransferLog("Failed to notify panel of archive success: " + err.Error())
l.WithField("error", err).Error("failed to notify panel of successful archive status")
return
}
sendTransferLog("Panel returned an error while notifying it of a successful archive: " + err.Error())
l.WithField("error", err.Error()).Error("panel returned an error when notifying it of a successful archive status")
return
}
hasError = false
// This log may not be displayed by the client due to the status event being sent before or at the same time.
sendTransferLog("Successfully notified panel of successful archive status")
l.Info("successfully notified panel of successful transfer archive status")
s.Events().Publish(server.TransferStatusEvent, "archived")
}(s)
c.Status(http.StatusAccepted)
}
func (w *downloadProgress) Write(v []byte) (int, error) {
n := len(v)
atomic.AddInt64(&w.progress, int64(n))
return n, nil
}
// Log helper function to attach all errors and info output to a consistently formatted
// log string for easier querying.
func (str serverTransferRequest) log() *log.Entry {
return log.WithField("subsystem", "transfers").WithField("server_id", str.ServerID)
}
// Downloads an archive from the machine that the server currently lives on.
func (str serverTransferRequest) downloadArchive() (*http.Response, error) {
client := http.Client{Timeout: 0}
req, err := http.NewRequest(http.MethodGet, str.URL, nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", str.Token)
res, err := client.Do(req) // lgtm [go/request-forgery]
if err != nil {
return nil, err
}
return res, nil
}
// Returns the path to the local archive on the system.
func (str serverTransferRequest) path() string {
return getArchivePath(str.ServerID)
}
// Creates the archive location on this machine by first checking that the required file
// does not already exist. If it does exist, the file is deleted and then re-created as
// an empty file.
func (str serverTransferRequest) createArchiveFile() (*os.File, error) {
p := str.path()
if _, err := os.Stat(p); err != nil {
if !os.IsNotExist(err) {
return nil, err
}
} else if err := os.Remove(p); err != nil {
return nil, err
}
return os.Create(p)
}
// Deletes the archive from the local filesystem. This is executed as a deferred function.
func (str serverTransferRequest) removeArchivePath() {
p := str.path()
str.log().Debug("deleting temporary transfer archive")
if err := os.Remove(p); err != nil && !os.IsNotExist(err) {
str.log().WithField("path", p).WithField("error", err).Error("failed to delete temporary transfer archive file")
return
}
str.log().Debug("deleted temporary transfer archive successfully")
}
// Verifies that the SHA-256 checksum of the file on the local filesystem matches the
// expected value from the transfer request. The string value returned is the computed
// checksum on the system.
func (str serverTransferRequest) verifyChecksum(matches string) (bool, string, error) {
f, err := os.Open(str.path())
if err != nil {
return false, "", err
}
defer f.Close()
h := sha256.New()
if _, err := io.Copy(h, bufio.NewReader(f)); err != nil {
return false, "", err
}
checksum := hex.EncodeToString(h.Sum(nil))
return checksum == matches, checksum, nil
}
// Sends a notification to the Panel letting it know what the status of this transfer is.
func (str serverTransferRequest) sendTransferStatus(client remote.Client, successful bool) error {
lg := str.log().WithField("transfer_successful", successful)
lg.Info("notifying Panel of server transfer state")
if err := client.SetTransferStatus(context.Background(), str.ServerID, successful); err != nil {
lg.WithField("error", err).Error("error notifying panel of transfer state")
return err
}
lg.Debug("notified panel of transfer state")
return nil
}
// Initiates a transfer between two nodes for a server by downloading an archive from the
// remote node and then applying the server details to this machine.
func postTransfer(c *gin.Context) {
var data serverTransferRequest
if err := c.BindJSON(&data); err != nil {
return
}
manager := middleware.ExtractManager(c)
u, err := uuid.Parse(data.ServerID)
if err != nil {
WithError(c, err)
return
}
// Force the server ID to be a valid UUID string at this point. If it is not an error
// is returned to the caller. This limits injection vulnerabilities that would cause
// the str.path() function to return a location not within the server archive directory.
data.ServerID = u.String()
data.log().Info("handling incoming server transfer request")
go func(data *serverTransferRequest) {
hasError := true
// Create a new server installer. This will only configure the environment and not
// run the installer scripts.
i, err := installer.New(context.Background(), manager, data.Server)
if err != nil {
_ = data.sendTransferStatus(manager.Client(), false)
data.log().WithField("error", err).Error("failed to validate received server data")
return
}
// This function automatically adds the Target Node prefix and Timestamp to the log output before sending it
// over the websocket.
sendTransferLog := func(data string) {
output := colorstring.Color(fmt.Sprintf("[yellow][bold]%s [Pterodactyl Transfer System] [Target Node]:[default] %s", time.Now().Format(time.RFC1123), data))
i.Server().Events().Publish(server.TransferLogsEvent, output)
}
// Mark the server as transferring to prevent problems later on during the process and
// then push the server into the global server collection for this instance.
i.Server().SetTransferring(true)
manager.Add(i.Server())
defer func(s *server.Server) {
// In the event that this transfer call fails, remove the server from the global
// server tracking so that we don't have a dangling instance.
if err := data.sendTransferStatus(manager.Client(), !hasError); hasError || err != nil {
sendTransferLog("Server transfer failed, check Wings logs for additional information.")
s.Events().Publish(server.TransferStatusEvent, "failure")
manager.Remove(func(match *server.Server) bool {
return match.ID() == s.ID()
})
// If the transfer status was successful but the request failed, act like the transfer failed.
if !hasError && err != nil {
// Delete all extracted files.
if err := os.RemoveAll(s.Filesystem().Path()); err != nil && !os.IsNotExist(err) {
data.log().WithField("error", err).Warn("failed to delete local server files directory")
}
}
} else {
s.SetTransferring(false)
s.Events().Publish(server.TransferStatusEvent, "success")
sendTransferLog("Transfer completed.")
}
}(i.Server())
data.log().Info("downloading server archive from current server node")
sendTransferLog("Received incoming transfer from Panel, attempting to download archive from source node...")
res, err := data.downloadArchive()
if err != nil {
sendTransferLog("Failed to retrieve server archive from remote node: " + err.Error())
data.log().WithField("error", err).Error("failed to download archive for server transfer")
return
}
defer res.Body.Close()
if res.StatusCode != http.StatusOK {
data.log().WithField("error", err).WithField("status", res.StatusCode).Error("unexpected error response from transfer endpoint")
return
}
size := res.ContentLength
if size == 0 {
data.log().WithField("error", err).Error("received an archive response with Content-Length of 0")
return
}
sendTransferLog("Got server archive response from remote node. (Content-Length: " + strconv.Itoa(int(size)) + ")")
sendTransferLog("Creating local archive file...")
file, err := data.createArchiveFile()
if err != nil {
data.log().WithField("error", err).Error("failed to create archive file on local filesystem")
return
}
sendTransferLog("Writing archive to disk...")
data.log().Info("writing transfer archive to disk...")
// Copy the file.
progress := &downloadProgress{size: size}
ticker := time.NewTicker(3 * time.Second)
go func(progress *downloadProgress, t *time.Ticker) {
for range ticker.C {
// p = 100 (Downloaded)
// size = 1000 (Content-Length)
// p / size = 0.1
// * 100 = 10% (Multiply by 100 to get a percentage of the download)
// 10% / tickPercentage = (10% / (100 / 25)) (Divide by tick percentage to get the number of ticks)
// 2.5 (Number of ticks as a float64)
// 2 (convert to an integer)
p := atomic.LoadInt64(&progress.progress)
// We have to cast these numbers to float in order to get a float result from the division.
width := ((float64(p) / float64(size)) * 100) / tickPercentage
bar := strings.Repeat("=", int(width)) + strings.Repeat(" ", ticks-int(width))
sendTransferLog("Downloading [" + bar + "] " + system.FormatBytes(p) + " / " + system.FormatBytes(progress.size))
}
}(progress, ticker)
var reader io.Reader
downloadLimit := float64(config.Get().System.Transfers.DownloadLimit) * 1024 * 1024
if downloadLimit > 0 {
// Wrap the body with a reader that is limited to the defined download limit speed.
reader = ratelimit.Reader(res.Body, ratelimit.NewBucketWithRate(downloadLimit, int64(downloadLimit)))
} else {
reader = res.Body
}
buf := make([]byte, 1024*4)
if _, err := io.CopyBuffer(file, io.TeeReader(reader, progress), buf); err != nil {
ticker.Stop()
_ = file.Close()
sendTransferLog("Failed while writing archive file to disk: " + err.Error())
data.log().WithField("error", err).Error("failed to copy archive file to disk")
return
}
ticker.Stop()
// Show 100% completion.
humanSize := system.FormatBytes(progress.size)
sendTransferLog("Downloading [" + strings.Repeat("=", ticks) + "] " + humanSize + " / " + humanSize)
if err := file.Close(); err != nil {
data.log().WithField("error", err).Error("unable to close archive file on local filesystem")
return
}
data.log().Info("finished writing transfer archive to disk")
sendTransferLog("Successfully wrote archive to disk.")
// Whenever the transfer fails or succeeds, delete the temporary transfer archive that
// was created on the disk.
defer data.removeArchivePath()
sendTransferLog("Verifying checksum of downloaded archive...")
data.log().Info("computing checksum of downloaded archive file")
expected := res.Header.Get("X-Checksum")
if matches, computed, err := data.verifyChecksum(expected); err != nil {
data.log().WithField("error", err).Error("encountered an error while calculating local filesystem archive checksum")
return
} else if !matches {
sendTransferLog("@@@@@ CHECKSUM VERIFICATION FAILED @@@@@")
sendTransferLog(" - Source Checksum: " + expected)
sendTransferLog(" - Computed Checksum: " + computed)
data.log().WithField("expected_sum", expected).WithField("computed_checksum", computed).Error("checksum mismatch when verifying integrity of local archive")
return
}
// Create the server's environment.
sendTransferLog("Creating server environment, this could take a while..")
data.log().Info("creating server environment")
if err := i.Server().CreateEnvironment(); err != nil {
data.log().WithField("error", err).Error("failed to create server environment")
return
}
sendTransferLog("Server environment has been created, extracting transfer archive..")
data.log().Info("server environment configured, extracting transfer archive")
if err := archiver.NewTarGz().Unarchive(data.path(), i.Server().Filesystem().Path()); err != nil {
// Un-archiving failed, delete the server's data directory.
if err := os.RemoveAll(i.Server().Filesystem().Path()); err != nil && !os.IsNotExist(err) {
data.log().WithField("error", err).Warn("failed to delete local server files directory")
}
data.log().WithField("error", err).Error("failed to extract server archive")
return
}
// We mark the process as being successful here as if we fail to send a transfer success,
// then a transfer failure won't probably be successful either.
//
// It may be useful to retry sending the transfer success every so often just in case of a small
// hiccup or the fix of whatever error causing the success request to fail.
hasError = false
data.log().Info("archive extracted successfully, notifying Panel of status")
sendTransferLog("Archive extracted successfully.")
}(&data)
c.Status(http.StatusAccepted) c.Status(http.StatusAccepted)
} }

View File

@ -8,6 +8,7 @@ type UploadPayload struct {
jwt.Payload jwt.Payload
ServerUuid string `json:"server_uuid"` ServerUuid string `json:"server_uuid"`
UserUuid string `json:"user_uuid"`
UniqueId string `json:"unique_id"` UniqueId string `json:"unique_id"`
} }

View File

@ -7,7 +7,6 @@ import (
"github.com/apex/log" "github.com/apex/log"
"github.com/gbrlsnchs/jwt/v3" "github.com/gbrlsnchs/jwt/v3"
"github.com/goccy/go-json"
) )
// The time at which Wings was booted. No JWT's created before this time are allowed to // The time at which Wings was booted. No JWT's created before this time are allowed to
@ -35,15 +34,15 @@ func DenyJTI(jti string) {
denylist.Store(jti, time.Now()) denylist.Store(jti, time.Now())
} }
// A JWT payload for Websocket connections. This JWT is passed along to the Websocket after // WebsocketPayload defines the JWT payload for a websocket connection. This JWT is passed along to
// it has been connected to by sending an "auth" event. // the websocket after it has been connected to by sending an "auth" event.
type WebsocketPayload struct { type WebsocketPayload struct {
jwt.Payload jwt.Payload
sync.RWMutex sync.RWMutex
UserID json.Number `json:"user_id"` UserUUID string `json:"user_uuid"`
ServerUUID string `json:"server_uuid"` ServerUUID string `json:"server_uuid"`
Permissions []string `json:"permissions"` Permissions []string `json:"permissions"`
} }
// Returns the JWT payload. // Returns the JWT payload.

View File

@ -9,6 +9,8 @@ import (
"github.com/goccy/go-json" "github.com/goccy/go-json"
"github.com/pterodactyl/wings/events" "github.com/pterodactyl/wings/events"
"github.com/pterodactyl/wings/system"
"github.com/pterodactyl/wings/server" "github.com/pterodactyl/wings/server"
) )
@ -88,12 +90,13 @@ func (h *Handler) listenForServerEvents(ctx context.Context) error {
ctx, cancel := context.WithCancel(ctx) ctx, cancel := context.WithCancel(ctx)
defer cancel() defer cancel()
eventChan := make(chan events.Event) eventChan := make(chan []byte)
logOutput := make(chan []byte) logOutput := make(chan []byte, 8)
installOutput := make(chan []byte) installOutput := make(chan []byte, 4)
h.server.Events().On(eventChan, e...)
h.server.Sink(server.LogSink).On(logOutput) h.server.Events().On(eventChan) // TODO: make a sinky
h.server.Sink(server.InstallSink).On(installOutput) h.server.Sink(system.LogSink).On(logOutput)
h.server.Sink(system.InstallSink).On(installOutput)
onError := func(evt string, err2 error) { onError := func(evt string, err2 error) {
h.Logger().WithField("event", evt).WithField("error", err2).Error("failed to send event over server websocket") h.Logger().WithField("event", evt).WithField("error", err2).Error("failed to send event over server websocket")
@ -110,19 +113,23 @@ func (h *Handler) listenForServerEvents(ctx context.Context) error {
select { select {
case <-ctx.Done(): case <-ctx.Done():
break break
case e := <-logOutput: case b := <-logOutput:
sendErr := h.SendJson(Message{Event: server.ConsoleOutputEvent, Args: []string{string(e)}}) sendErr := h.SendJson(Message{Event: server.ConsoleOutputEvent, Args: []string{string(b)}})
if sendErr == nil { if sendErr == nil {
continue continue
} }
onError(server.ConsoleOutputEvent, sendErr) onError(server.ConsoleOutputEvent, sendErr)
case e := <-installOutput: case b := <-installOutput:
sendErr := h.SendJson(Message{Event: server.InstallOutputEvent, Args: []string{string(e)}}) sendErr := h.SendJson(Message{Event: server.InstallOutputEvent, Args: []string{string(b)}})
if sendErr == nil { if sendErr == nil {
continue continue
} }
onError(server.InstallOutputEvent, sendErr) onError(server.InstallOutputEvent, sendErr)
case e := <-eventChan: case b := <-eventChan:
var e events.Event
if err := events.DecodeTo(b, &e); err != nil {
continue
}
var sendErr error var sendErr error
message := Message{Event: e.Topic} message := Message{Event: e.Topic}
if str, ok := e.Data.(string); ok { if str, ok := e.Data.(string); ok {
@ -148,9 +155,9 @@ func (h *Handler) listenForServerEvents(ctx context.Context) error {
} }
// These functions will automatically close the channel if it hasn't been already. // These functions will automatically close the channel if it hasn't been already.
h.server.Events().Off(eventChan, e...) h.server.Events().Off(eventChan)
h.server.Sink(server.LogSink).Off(logOutput) h.server.Sink(system.LogSink).Off(logOutput)
h.server.Sink(server.InstallSink).Off(installOutput) h.server.Sink(system.InstallSink).Off(installOutput)
// If the internal context is stopped it is either because the parent context // If the internal context is stopped it is either because the parent context
// got canceled or because we ran into an error. If the "err" variable is nil // got canceled or because we ran into an error. If the "err" variable is nil

View File

@ -8,12 +8,17 @@ import (
"sync" "sync"
"time" "time"
"github.com/pterodactyl/wings/internal/models"
"emperror.dev/errors" "emperror.dev/errors"
"github.com/apex/log" "github.com/apex/log"
"github.com/gbrlsnchs/jwt/v3" "github.com/gbrlsnchs/jwt/v3"
"github.com/gin-gonic/gin"
"github.com/goccy/go-json"
"github.com/google/uuid" "github.com/google/uuid"
"github.com/gorilla/websocket" "github.com/gorilla/websocket"
"github.com/goccy/go-json"
"github.com/pterodactyl/wings/system"
"github.com/pterodactyl/wings/config" "github.com/pterodactyl/wings/config"
"github.com/pterodactyl/wings/environment" "github.com/pterodactyl/wings/environment"
@ -39,6 +44,7 @@ type Handler struct {
Connection *websocket.Conn `json:"-"` Connection *websocket.Conn `json:"-"`
jwt *tokens.WebsocketPayload jwt *tokens.WebsocketPayload
server *server.Server server *server.Server
ra server.RequestActivity
uuid uuid.UUID uuid uuid.UUID
} }
@ -76,7 +82,7 @@ func NewTokenPayload(token []byte) (*tokens.WebsocketPayload, error) {
} }
// GetHandler returns a new websocket handler using the context provided. // GetHandler returns a new websocket handler using the context provided.
func GetHandler(s *server.Server, w http.ResponseWriter, r *http.Request) (*Handler, error) { func GetHandler(s *server.Server, w http.ResponseWriter, r *http.Request, c *gin.Context) (*Handler, error) {
upgrader := websocket.Upgrader{ upgrader := websocket.Upgrader{
// Ensure that the websocket request is originating from the Panel itself, // Ensure that the websocket request is originating from the Panel itself,
// and not some other location. // and not some other location.
@ -108,6 +114,7 @@ func GetHandler(s *server.Server, w http.ResponseWriter, r *http.Request) (*Hand
Connection: conn, Connection: conn,
jwt: nil, jwt: nil,
server: s, server: s,
ra: s.NewRequestActivity("", c.ClientIP()),
uuid: u, uuid: u,
}, nil }, nil
} }
@ -263,6 +270,7 @@ func (h *Handler) GetJwt() *tokens.WebsocketPayload {
// setJwt sets the JWT for the websocket in a race-safe manner. // setJwt sets the JWT for the websocket in a race-safe manner.
func (h *Handler) setJwt(token *tokens.WebsocketPayload) { func (h *Handler) setJwt(token *tokens.WebsocketPayload) {
h.Lock() h.Lock()
h.ra = h.ra.SetUser(token.UserUUID)
h.jwt = token h.jwt = token
h.Unlock() h.Unlock()
} }
@ -353,7 +361,7 @@ func (h *Handler) HandleInbound(ctx context.Context, m Message) error {
} }
err := h.server.HandlePowerAction(action) err := h.server.HandlePowerAction(action)
if errors.Is(err, context.DeadlineExceeded) { if errors.Is(err, system.ErrLockerLocked) {
m, _ := h.GetErrorMessage("another power action is currently being processed for this server, please try again later") m, _ := h.GetErrorMessage("another power action is currently being processed for this server, please try again later")
_ = h.SendJson(Message{ _ = h.SendJson(Message{
@ -364,6 +372,10 @@ func (h *Handler) HandleInbound(ctx context.Context, m Message) error {
return nil return nil
} }
if err == nil {
h.server.SaveActivity(h.ra, models.Event(server.ActivityPowerPrefix+action), nil)
}
return err return err
} }
case SendServerLogsEvent: case SendServerLogsEvent:
@ -420,7 +432,13 @@ func (h *Handler) HandleInbound(ctx context.Context, m Message) error {
} }
} }
return h.server.Environment.SendCommand(strings.Join(m.Args, "")) if err := h.server.Environment.SendCommand(strings.Join(m.Args, "")); err != nil {
return err
}
h.server.SaveActivity(h.ra, server.ActivityConsoleCommand, models.ActivityMeta{
"command": strings.Join(m.Args, ""),
})
return nil
} }
} }

View File

@ -1,121 +0,0 @@
Name: ptero-wings
Version: 1.5.3
Release: 1%{?dist}
Summary: The server control plane for Pterodactyl Panel. Written from the ground-up with security, speed, and stability in mind.
BuildArch: x86_64
License: MIT
URL: https://github.com/pterodactyl/wings
Source0: https://github.com/pterodactyl/wings/releases/download/v%{version}/wings_linux_amd64
%if 0%{?rhel} && 0%{?rhel} <= 8
BuildRequires: systemd
%else
BuildRequires: systemd-rpm-macros
%endif
%description
Wings is Pterodactyl's server control plane, built for the rapidly
changing gaming industry and designed to be highly performant and
secure. Wings provides an HTTP API allowing you to interface directly
with running server instances, fetch server logs, generate backups,
and control all aspects of the server lifecycle.
In addition, Wings ships with a built-in SFTP server allowing your
system to remain free of Pterodactyl specific dependencies, and
allowing users to authenticate with the same credentials they would
normally use to access the Panel.
%prep
%build
#nothing required
%install
mkdir -p %{buildroot}%{_bindir}
mkdir -p %{buildroot}%{_unitdir}
cp %{_sourcedir}/wings_linux_amd64 %{buildroot}%{_bindir}/wings
cat > %{buildroot}%{_unitdir}/wings.service << EOF
[Unit]
Description=Pterodactyl Wings Daemon
After=docker.service
Requires=docker.service
PartOf=docker.service
StartLimitIntervalSec=600
[Service]
WorkingDirectory=/etc/pterodactyl
ExecStart=/usr/bin/wings
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
StartLimitInterval=180
StartLimitBurst=30
RestartSec=5s
[Install]
WantedBy=multi-user.target
EOF
%files
%attr(0755, root, root) %{_prefix}/bin/wings
%attr(0644, root, root) %{_unitdir}/wings.service
%post
# Reload systemd
systemctl daemon-reload
# Create the required directory structure
mkdir -p /etc/pterodactyl
mkdir -p /var/lib/pterodactyl/{archives,backups,volumes}
mkdir -p /var/log/pterodactyl/install
%preun
systemctl is-active %{name} >/dev/null 2>&1
if [ $? -eq 0 ]; then
systemctl stop %{name}
fi
systemctl is-enabled %{name} >/dev/null 2>&1
if [ $? -eq 0 ]; then
systemctl disable %{name}
fi
%postun
rm -rf /var/log/pterodactyl
%verifyscript
wings --version
%changelog
* Wed Oct 27 2021 Capitol Hosting Solutions Systems Engineering <syseng@chs.gg> - 1.5.3-1
- specfile by Capitol Hosting Solutions, Upstream by Pterodactyl
- Rebased for https://github.com/pterodactyl/wings/releases/tag/v1.5.3
- Fixes improper event registration and error handling during socket authentication that would cause the incorrect error message to be returned to the client, or no error in some scenarios. Event registration is now delayed until the socket is fully authenticated to ensure needless listeners are not registed.
- Fixes dollar signs always being evaluated as environment variables with no way to escape them. They can now be escaped as $$ which will transform into a single dollar sign.
- A websocket connection to a server will be closed by Wings if there is a send error encountered and the client will be left to handle reconnections, rather than simply logging the error and continuing to listen for new events.
* Sun Sep 12 2021 Capitol Hosting Solutions Systems Engineering <syseng@chs.gg> - 1.5.0-1
- specfile by Capitol Hosting Solutions, Upstream by Pterodactyl
- Rebased for https://github.com/pterodactyl/wings/releases/tag/v1.5.0
- Fixes a race condition when setting the application name in the console output for a server.
- Fixes a server being reinstalled causing the file_denylist parameter for an Egg to be ignored until Wings is restarted.
- Fixes YAML file parser not correctly setting boolean values.
- Fixes potential issue where the underlying websocket connection is closed but the parent request context is not yet canceled causing a write over a closed connection.
- Fixes race condition when closing all active websocket connections when a server is deleted.
- Fixes logic to determine if a server's context is closed out and send a websocket close message to connected clients. Previously this fired off whenever the request itself was closed, and not when the server context was closed.
- Exposes 8080 in the wings Dockerfile to better support reverse proxy tools.
- Releases are now built using Go 1.17 the minimum version required to build Wings remains Go 1.16.
- Simplifed the logic powering server updates to only pull information from the Panel rather than trying to accept updated values. All parts of Wings needing the most up-to-date server details should call Server#Sync() to fetch the latest stored build information.
- Installer#New() no longer requires passing all of the server data as a byte slice, rather a new Installer#ServerDetails struct is exposed which can be passed and accepts a UUID and if the server should be started after the installer finishes.
- Removes complicated (and unused) logic during the server installation process that was a hold-over from legacy Wings architectures.
- Removes the PATCH /api/servers/:server endpoint if you were previously using this API call it should be replaced with POST /api/servers/:server/sync.
* Wed Aug 25 2021 Capitol Hosting Solutions Systems Engineering <syseng@chs.gg> - 1.4.7-1
- specfile by Capitol Hosting Solutions, Upstream by Pterodactyl
- Rebased for https://github.com/pterodactyl/wings/releases/tag/v1.4.7
- SFTP access is now properly denied if a server is suspended.
- Correctly uses start_on_completion and crash_detection_enabled for servers.

66
server/activity.go Normal file
View File

@ -0,0 +1,66 @@
package server
import (
"context"
"time"
"emperror.dev/errors"
"github.com/pterodactyl/wings/internal/database"
"github.com/pterodactyl/wings/internal/models"
)
const ActivityPowerPrefix = "server:power."
const (
ActivityConsoleCommand = models.Event("server:console.command")
ActivitySftpWrite = models.Event("server:sftp.write")
ActivitySftpCreate = models.Event("server:sftp.create")
ActivitySftpCreateDirectory = models.Event("server:sftp.create-directory")
ActivitySftpRename = models.Event("server:sftp.rename")
ActivitySftpDelete = models.Event("server:sftp.delete")
ActivityFileUploaded = models.Event("server:file.uploaded")
)
// RequestActivity is a wrapper around a LoggedEvent that is able to track additional request
// specific metadata including the specific user and IP address associated with all subsequent
// events. The internal logged event structure can be extracted by calling RequestEvent.Event().
type RequestActivity struct {
server string
user string
ip string
}
// Event returns the underlying logged event from the RequestEvent instance and sets the
// specific event and metadata on it.
func (ra RequestActivity) Event(event models.Event, metadata models.ActivityMeta) *models.Activity {
a := models.Activity{Server: ra.server, IP: ra.ip, Event: event, Metadata: metadata}
return a.SetUser(ra.user)
}
// SetUser clones the RequestActivity struct and sets a new user value on the copy
// before returning it.
func (ra RequestActivity) SetUser(u string) RequestActivity {
c := ra
c.user = u
return c
}
func (s *Server) NewRequestActivity(user string, ip string) RequestActivity {
return RequestActivity{server: s.ID(), user: user, ip: ip}
}
// SaveActivity saves an activity entry to the database in a background routine. If an error is
// encountered it is logged but not returned to the caller.
func (s *Server) SaveActivity(a RequestActivity, event models.Event, metadata models.ActivityMeta) {
ctx, cancel := context.WithTimeout(s.Context(), time.Second*3)
go func() {
defer cancel()
if tx := database.Instance().WithContext(ctx).Create(a.Event(event, metadata)); tx.Error != nil {
s.Log().WithField("error", errors.WithStack(tx.Error)).
WithField("event", event).
Error("activity: failed to save event")
}
}()
}

View File

@ -24,7 +24,6 @@ func (s *Server) notifyPanelOfBackup(uuid string, ad *backup.ArchiveDetails, suc
"backup": uuid, "backup": uuid,
"error": err, "error": err,
}).Error("failed to notify panel of backup status due to wings error") }).Error("failed to notify panel of backup status due to wings error")
return err return err
} }
@ -68,7 +67,7 @@ func (s *Server) Backup(b backup.BackupInterface) error {
} }
} }
ad, err := b.Generate(s.Context(), s.Filesystem().Path(), ignored) ad, err := b.Generate(s.Context(), s.Filesystem(), ignored)
if err != nil { if err != nil {
if err := s.notifyPanelOfBackup(b.Identifier(), &backup.ArchiveDetails{}, false); err != nil { if err := s.notifyPanelOfBackup(b.Identifier(), &backup.ArchiveDetails{}, false); err != nil {
s.Log().WithFields(log.Fields{ s.Log().WithFields(log.Fields{
@ -127,7 +126,7 @@ func (s *Server) RestoreBackup(b backup.BackupInterface, reader io.ReadCloser) (
defer func() { defer func() {
s.Config().SetSuspended(false) s.Config().SetSuspended(false)
if reader != nil { if reader != nil {
reader.Close() _ = reader.Close()
} }
}() }()
// Send an API call to the Panel as soon as this function is done running so that // Send an API call to the Panel as soon as this function is done running so that
@ -142,7 +141,7 @@ func (s *Server) RestoreBackup(b backup.BackupInterface, reader io.ReadCloser) (
// instance, otherwise you'll likely hit all types of write errors due to the // instance, otherwise you'll likely hit all types of write errors due to the
// server being suspended. // server being suspended.
if s.Environment.State() != environment.ProcessOfflineState { if s.Environment.State() != environment.ProcessOfflineState {
if err = s.Environment.WaitForStop(120, false); err != nil { if err = s.Environment.WaitForStop(s.Context(), 2*time.Minute, false); err != nil {
if !client.IsErrNotFound(err) { if !client.IsErrNotFound(err) {
return errors.WrapIf(err, "server/backup: restore: failed to wait for container stop") return errors.WrapIf(err, "server/backup: restore: failed to wait for container stop")
} }
@ -152,15 +151,17 @@ func (s *Server) RestoreBackup(b backup.BackupInterface, reader io.ReadCloser) (
// Attempt to restore the backup to the server by running through each entry // Attempt to restore the backup to the server by running through each entry
// in the file one at a time and writing them to the disk. // in the file one at a time and writing them to the disk.
s.Log().Debug("starting file writing process for backup restoration") s.Log().Debug("starting file writing process for backup restoration")
err = b.Restore(s.Context(), reader, func(file string, r io.Reader, mode fs.FileMode, atime, mtime time.Time) error { err = b.Restore(s.Context(), reader, func(file string, info fs.FileInfo, r io.ReadCloser) error {
defer r.Close()
s.Events().Publish(DaemonMessageEvent, "(restoring): "+file) s.Events().Publish(DaemonMessageEvent, "(restoring): "+file)
if err := s.Filesystem().Writefile(file, r); err != nil { // TODO: since this will be called a lot, it may be worth adding an optimized
// Write with Chtimes method to the UnixFS that is able to re-use the
// same dirfd and file name.
if err := s.Filesystem().Write(file, r, info.Size(), info.Mode()); err != nil {
return err return err
} }
if err := s.Filesystem().Chmod(file, mode); err != nil { atime := info.ModTime()
return err return s.Filesystem().Chtimes(file, atime, atime)
}
return s.Filesystem().Chtimes(file, atime, mtime)
}) })
return errors.WithStackIf(err) return errors.WithStackIf(err)

View File

@ -8,16 +8,22 @@ import (
"io/fs" "io/fs"
"os" "os"
"path" "path"
"time"
"emperror.dev/errors" "emperror.dev/errors"
"github.com/apex/log" "github.com/apex/log"
"github.com/mholt/archiver/v4"
"golang.org/x/sync/errgroup" "golang.org/x/sync/errgroup"
"github.com/pterodactyl/wings/config" "github.com/pterodactyl/wings/config"
"github.com/pterodactyl/wings/remote" "github.com/pterodactyl/wings/remote"
"github.com/pterodactyl/wings/server/filesystem"
) )
var format = archiver.CompressedArchive{
Compression: archiver.Gz{},
Archival: archiver.Tar{},
}
type AdapterType string type AdapterType string
const ( const (
@ -27,12 +33,12 @@ const (
// RestoreCallback is a generic restoration callback that exists for both local // RestoreCallback is a generic restoration callback that exists for both local
// and remote backups allowing the files to be restored. // and remote backups allowing the files to be restored.
type RestoreCallback func(file string, r io.Reader, mode fs.FileMode, atime, mtime time.Time) error type RestoreCallback func(file string, info fs.FileInfo, r io.ReadCloser) error
// noinspection GoNameStartsWithPackageName // noinspection GoNameStartsWithPackageName
type BackupInterface interface { type BackupInterface interface {
// SetClient sets the API request client on the backup interface. // SetClient sets the API request client on the backup interface.
SetClient(c remote.Client) SetClient(remote.Client)
// Identifier returns the UUID of this backup as tracked by the panel // Identifier returns the UUID of this backup as tracked by the panel
// instance. // instance.
Identifier() string Identifier() string
@ -41,7 +47,7 @@ type BackupInterface interface {
WithLogContext(map[string]interface{}) WithLogContext(map[string]interface{})
// Generate creates a backup in whatever the configured source for the // Generate creates a backup in whatever the configured source for the
// specific implementation is. // specific implementation is.
Generate(ctx context.Context, basePath string, ignore string) (*ArchiveDetails, error) Generate(context.Context, *filesystem.Filesystem, string) (*ArchiveDetails, error)
// Ignored returns the ignored files for this backup instance. // Ignored returns the ignored files for this backup instance.
Ignored() string Ignored() string
// Checksum returns a SHA1 checksum for the generated backup. // Checksum returns a SHA1 checksum for the generated backup.
@ -53,13 +59,13 @@ type BackupInterface interface {
// to store it until it is moved to the final spot. // to store it until it is moved to the final spot.
Path() string Path() string
// Details returns details about the archive. // Details returns details about the archive.
Details(ctx context.Context) (*ArchiveDetails, error) Details(context.Context, []remote.BackupPart) (*ArchiveDetails, error)
// Remove removes a backup file. // Remove removes a backup file.
Remove() error Remove() error
// Restore is called when a backup is ready to be restored to the disk from // Restore is called when a backup is ready to be restored to the disk from
// the given source. Not every backup implementation will support this nor // the given source. Not every backup implementation will support this nor
// will every implementation require a reader be provided. // will every implementation require a reader be provided.
Restore(ctx context.Context, reader io.Reader, callback RestoreCallback) error Restore(context.Context, io.Reader, RestoreCallback) error
} }
type Backup struct { type Backup struct {
@ -119,8 +125,8 @@ func (b *Backup) Checksum() ([]byte, error) {
// Details returns both the checksum and size of the archive currently stored on // Details returns both the checksum and size of the archive currently stored on
// the disk to the caller. // the disk to the caller.
func (b *Backup) Details(ctx context.Context) (*ArchiveDetails, error) { func (b *Backup) Details(ctx context.Context, parts []remote.BackupPart) (*ArchiveDetails, error) {
ad := ArchiveDetails{ChecksumType: "sha1"} ad := ArchiveDetails{ChecksumType: "sha1", Parts: parts}
g, ctx := errgroup.WithContext(ctx) g, ctx := errgroup.WithContext(ctx)
g.Go(func() error { g.Go(func() error {
@ -162,9 +168,10 @@ func (b *Backup) log() *log.Entry {
} }
type ArchiveDetails struct { type ArchiveDetails struct {
Checksum string `json:"checksum"` Checksum string `json:"checksum"`
ChecksumType string `json:"checksum_type"` ChecksumType string `json:"checksum_type"`
Size int64 `json:"size"` Size int64 `json:"size"`
Parts []remote.BackupPart `json:"parts"`
} }
// ToRequest returns a request object. // ToRequest returns a request object.
@ -174,5 +181,6 @@ func (ad *ArchiveDetails) ToRequest(successful bool) remote.BackupRequest {
ChecksumType: ad.ChecksumType, ChecksumType: ad.ChecksumType,
Size: ad.Size, Size: ad.Size,
Successful: successful, Successful: successful,
Parts: ad.Parts,
} }
} }

View File

@ -6,8 +6,10 @@ import (
"os" "os"
"emperror.dev/errors" "emperror.dev/errors"
"github.com/mholt/archiver/v3" "github.com/juju/ratelimit"
"github.com/mholt/archiver/v4"
"github.com/pterodactyl/wings/config"
"github.com/pterodactyl/wings/remote" "github.com/pterodactyl/wings/remote"
"github.com/pterodactyl/wings/server/filesystem" "github.com/pterodactyl/wings/server/filesystem"
) )
@ -57,19 +59,19 @@ func (b *LocalBackup) WithLogContext(c map[string]interface{}) {
// Generate generates a backup of the selected files and pushes it to the // Generate generates a backup of the selected files and pushes it to the
// defined location for this instance. // defined location for this instance.
func (b *LocalBackup) Generate(ctx context.Context, basePath, ignore string) (*ArchiveDetails, error) { func (b *LocalBackup) Generate(ctx context.Context, fsys *filesystem.Filesystem, ignore string) (*ArchiveDetails, error) {
a := &filesystem.Archive{ a := &filesystem.Archive{
BasePath: basePath, Filesystem: fsys,
Ignore: ignore, Ignore: ignore,
} }
b.log().WithField("path", b.Path()).Info("creating backup for server") b.log().WithField("path", b.Path()).Info("creating backup for server")
if err := a.Create(b.Path()); err != nil { if err := a.Create(ctx, b.Path()); err != nil {
return nil, err return nil, err
} }
b.log().Info("created backup successfully") b.log().Info("created backup successfully")
ad, err := b.Details(ctx) ad, err := b.Details(ctx, nil)
if err != nil { if err != nil {
return nil, errors.WrapIf(err, "backup: failed to get archive details for local backup") return nil, errors.WrapIf(err, "backup: failed to get archive details for local backup")
} }
@ -79,16 +81,28 @@ func (b *LocalBackup) Generate(ctx context.Context, basePath, ignore string) (*A
// Restore will walk over the archive and call the callback function for each // Restore will walk over the archive and call the callback function for each
// file encountered. // file encountered.
func (b *LocalBackup) Restore(ctx context.Context, _ io.Reader, callback RestoreCallback) error { func (b *LocalBackup) Restore(ctx context.Context, _ io.Reader, callback RestoreCallback) error {
return archiver.Walk(b.Path(), func(f archiver.File) error { f, err := os.Open(b.Path())
select { if err != nil {
case <-ctx.Done(): return err
// Stop walking if the context is canceled. }
return archiver.ErrStopWalk defer f.Close()
default:
if f.IsDir() { var reader io.Reader = f
return nil // Steal the logic we use for making backups which will be applied when restoring
} // this specific backup. This allows us to prevent overloading the disk unintentionally.
return callback(filesystem.ExtractNameFromArchive(f), f, f.Mode(), f.ModTime(), f.ModTime()) if writeLimit := int64(config.Get().System.Backups.WriteLimit * 1024 * 1024); writeLimit > 0 {
reader = ratelimit.Reader(f, ratelimit.NewBucketWithRate(float64(writeLimit), writeLimit))
}
if err := format.Extract(ctx, reader, nil, func(ctx context.Context, f archiver.File) error {
r, err := f.Open()
if err != nil {
return err
} }
}) defer r.Close()
return callback(f.NameInArchive, f.FileInfo, r)
}); err != nil {
return err
}
return nil
} }

View File

@ -1,8 +1,6 @@
package backup package backup
import ( import (
"archive/tar"
"compress/gzip"
"context" "context"
"fmt" "fmt"
"io" "io"
@ -13,13 +11,12 @@ import (
"emperror.dev/errors" "emperror.dev/errors"
"github.com/cenkalti/backoff/v4" "github.com/cenkalti/backoff/v4"
"github.com/pterodactyl/wings/server/filesystem"
"github.com/juju/ratelimit" "github.com/juju/ratelimit"
"github.com/mholt/archiver/v4"
"github.com/pterodactyl/wings/config" "github.com/pterodactyl/wings/config"
"github.com/pterodactyl/wings/remote" "github.com/pterodactyl/wings/remote"
"github.com/pterodactyl/wings/server/filesystem"
) )
type S3Backup struct { type S3Backup struct {
@ -51,16 +48,16 @@ func (s *S3Backup) WithLogContext(c map[string]interface{}) {
// Generate creates a new backup on the disk, moves it into the S3 bucket via // Generate creates a new backup on the disk, moves it into the S3 bucket via
// the provided presigned URL, and then deletes the backup from the disk. // the provided presigned URL, and then deletes the backup from the disk.
func (s *S3Backup) Generate(ctx context.Context, basePath, ignore string) (*ArchiveDetails, error) { func (s *S3Backup) Generate(ctx context.Context, fsys *filesystem.Filesystem, ignore string) (*ArchiveDetails, error) {
defer s.Remove() defer s.Remove()
a := &filesystem.Archive{ a := &filesystem.Archive{
BasePath: basePath, Filesystem: fsys,
Ignore: ignore, Ignore: ignore,
} }
s.log().WithField("path", s.Path()).Info("creating backup for server") s.log().WithField("path", s.Path()).Info("creating backup for server")
if err := a.Create(s.Path()); err != nil { if err := a.Create(ctx, s.Path()); err != nil {
return nil, err return nil, err
} }
s.log().Info("created backup successfully") s.log().Info("created backup successfully")
@ -71,10 +68,11 @@ func (s *S3Backup) Generate(ctx context.Context, basePath, ignore string) (*Arch
} }
defer rc.Close() defer rc.Close()
if err := s.generateRemoteRequest(ctx, rc); err != nil { parts, err := s.generateRemoteRequest(ctx, rc)
if err != nil {
return nil, err return nil, err
} }
ad, err := s.Details(ctx) ad, err := s.Details(ctx, parts)
if err != nil { if err != nil {
return nil, errors.WrapIf(err, "backup: failed to get archive details after upload") return nil, errors.WrapIf(err, "backup: failed to get archive details after upload")
} }
@ -95,50 +93,35 @@ func (s *S3Backup) Restore(ctx context.Context, r io.Reader, callback RestoreCal
if writeLimit := int64(config.Get().System.Backups.WriteLimit * 1024 * 1024); writeLimit > 0 { if writeLimit := int64(config.Get().System.Backups.WriteLimit * 1024 * 1024); writeLimit > 0 {
reader = ratelimit.Reader(r, ratelimit.NewBucketWithRate(float64(writeLimit), writeLimit)) reader = ratelimit.Reader(r, ratelimit.NewBucketWithRate(float64(writeLimit), writeLimit))
} }
gr, err := gzip.NewReader(reader) if err := format.Extract(ctx, reader, nil, func(ctx context.Context, f archiver.File) error {
if err != nil { r, err := f.Open()
return err
}
defer gr.Close()
tr := tar.NewReader(gr)
for {
select {
case <-ctx.Done():
return nil
default:
// Do nothing, fall through to the next block of code in this loop.
}
header, err := tr.Next()
if err != nil { if err != nil {
if err == io.EOF {
break
}
return err return err
} }
if header.Typeflag == tar.TypeReg { defer r.Close()
if err := callback(header.Name, tr, header.FileInfo().Mode(), header.AccessTime, header.ModTime); err != nil {
return err return callback(f.NameInArchive, f.FileInfo, r)
} }); err != nil {
} return err
} }
return nil return nil
} }
// Generates the remote S3 request and begins the upload. // Generates the remote S3 request and begins the upload.
func (s *S3Backup) generateRemoteRequest(ctx context.Context, rc io.ReadCloser) error { func (s *S3Backup) generateRemoteRequest(ctx context.Context, rc io.ReadCloser) ([]remote.BackupPart, error) {
defer rc.Close() defer rc.Close()
s.log().Debug("attempting to get size of backup...") s.log().Debug("attempting to get size of backup...")
size, err := s.Backup.Size() size, err := s.Backup.Size()
if err != nil { if err != nil {
return err return nil, err
} }
s.log().WithField("size", size).Debug("got size of backup") s.log().WithField("size", size).Debug("got size of backup")
s.log().Debug("attempting to get S3 upload urls from Panel...") s.log().Debug("attempting to get S3 upload urls from Panel...")
urls, err := s.client.GetBackupRemoteUploadURLs(context.Background(), s.Backup.Uuid, size) urls, err := s.client.GetBackupRemoteUploadURLs(context.Background(), s.Backup.Uuid, size)
if err != nil { if err != nil {
return err return nil, err
} }
s.log().Debug("got S3 upload urls from the Panel") s.log().Debug("got S3 upload urls from the Panel")
s.log().WithField("parts", len(urls.Parts)).Info("attempting to upload backup to s3 endpoint...") s.log().WithField("parts", len(urls.Parts)).Info("attempting to upload backup to s3 endpoint...")
@ -156,22 +139,26 @@ func (s *S3Backup) generateRemoteRequest(ctx context.Context, rc io.ReadCloser)
} }
// Attempt to upload the part. // Attempt to upload the part.
if _, err := uploader.uploadPart(ctx, part, partSize); err != nil { etag, err := uploader.uploadPart(ctx, part, partSize)
if err != nil {
s.log().WithField("part_id", i+1).WithError(err).Warn("failed to upload part") s.log().WithField("part_id", i+1).WithError(err).Warn("failed to upload part")
return err return nil, err
} }
uploader.uploadedParts = append(uploader.uploadedParts, remote.BackupPart{
ETag: etag,
PartNumber: i + 1,
})
s.log().WithField("part_id", i+1).Info("successfully uploaded backup part") s.log().WithField("part_id", i+1).Info("successfully uploaded backup part")
} }
s.log().WithField("parts", len(urls.Parts)).Info("backup has been successfully uploaded") s.log().WithField("parts", len(urls.Parts)).Info("backup has been successfully uploaded")
return nil return uploader.uploadedParts, nil
} }
type s3FileUploader struct { type s3FileUploader struct {
io.ReadCloser io.ReadCloser
client *http.Client client *http.Client
uploadedParts []remote.BackupPart
} }
// newS3FileUploader returns a new file uploader instance. // newS3FileUploader returns a new file uploader instance.

View File

@ -4,28 +4,34 @@ import (
"runtime" "runtime"
"github.com/gammazero/workerpool" "github.com/gammazero/workerpool"
"github.com/pterodactyl/wings/internal/ufs"
) )
// Parent function that will update all of the defined configuration files for a server // UpdateConfigurationFiles updates all the defined configuration files for
// automatically to ensure that they always use the specified values. // a server automatically to ensure that they always use the specified values.
func (s *Server) UpdateConfigurationFiles() { func (s *Server) UpdateConfigurationFiles() {
pool := workerpool.New(runtime.NumCPU()) pool := workerpool.New(runtime.NumCPU())
s.Log().Debug("acquiring process configuration files...")
files := s.ProcessConfiguration().ConfigurationFiles files := s.ProcessConfiguration().ConfigurationFiles
s.Log().Debug("acquired process configuration files")
for _, cf := range files { for _, cf := range files {
f := cf f := cf
pool.Submit(func() { pool.Submit(func() {
p, err := s.Filesystem().SafePath(f.FileName) file, err := s.Filesystem().UnixFS().Touch(f.FileName, ufs.O_RDWR|ufs.O_CREATE, 0o644)
if err != nil { if err != nil {
s.Log().WithField("error", err).Error("failed to generate safe path for configuration file") s.Log().WithField("file_name", f.FileName).WithField("error", err).Error("failed to open file for configuration")
return return
} }
defer file.Close()
if err := f.Parse(p, false); err != nil { if err := f.Parse(file); err != nil {
s.Log().WithField("error", err).Error("failed to parse and update server configuration file") s.Log().WithField("error", err).Error("failed to parse and update server configuration file")
} }
s.Log().WithField("file_name", f.FileName).Debug("finished processing server configuration file")
}) })
} }

View File

@ -16,6 +16,11 @@ type EggConfiguration struct {
FileDenylist []string `json:"file_denylist"` FileDenylist []string `json:"file_denylist"`
} }
type ConfigurationMeta struct {
Name string `json:"name"`
Description string `json:"description"`
}
type Configuration struct { type Configuration struct {
mu sync.RWMutex mu sync.RWMutex
@ -24,6 +29,8 @@ type Configuration struct {
// docker containers as well as in log output. // docker containers as well as in log output.
Uuid string `json:"uuid"` Uuid string `json:"uuid"`
Meta ConfigurationMeta `json:"meta"`
// Whether or not the server is in a suspended state. Suspended servers cannot // Whether or not the server is in a suspended state. Suspended servers cannot
// be started or modified except in certain scenarios by an admin user. // be started or modified except in certain scenarios by an admin user.
Suspended bool `json:"suspended"` Suspended bool `json:"suspended"`
@ -39,6 +46,9 @@ type Configuration struct {
// server process. // server process.
EnvVars environment.Variables `json:"environment"` EnvVars environment.Variables `json:"environment"`
// Labels is a map of container labels that should be applied to the running server process.
Labels map[string]string `json:"labels"`
Allocations environment.Allocations `json:"allocations"` Allocations environment.Allocations `json:"allocations"`
Build environment.Limits `json:"build"` Build environment.Limits `json:"build"`
CrashDetectionEnabled bool `json:"crash_detection_enabled"` CrashDetectionEnabled bool `json:"crash_detection_enabled"`

View File

@ -1,13 +1,10 @@
package server package server
import ( import (
"context"
"fmt" "fmt"
"sync" "sync"
"sync/atomic"
"time" "time"
"emperror.dev/errors"
"github.com/mitchellh/colorstring" "github.com/mitchellh/colorstring"
"github.com/pterodactyl/wings/config" "github.com/pterodactyl/wings/config"
@ -18,118 +15,8 @@ import (
// the configuration every time we need to send output along to the websocket for // the configuration every time we need to send output along to the websocket for
// a server. // a server.
var appName string var appName string
var appNameSync sync.Once var appNameSync sync.Once
var ErrTooMuchConsoleData = errors.New("console is outputting too much data")
type ConsoleThrottler struct {
mu sync.Mutex
config.ConsoleThrottles
// The total number of activations that have occurred thus far.
activations uint64
// The total number of lines that have been sent since the last reset timer period.
count uint64
// Wether or not the console output is being throttled. It is up to calling code to
// determine what to do if it is.
isThrottled *system.AtomicBool
// The total number of lines processed so far during the given time period.
timerCancel *context.CancelFunc
}
// Resets the state of the throttler.
func (ct *ConsoleThrottler) Reset() {
atomic.StoreUint64(&ct.count, 0)
atomic.StoreUint64(&ct.activations, 0)
ct.isThrottled.Store(false)
}
// Triggers an activation for a server. You can also decrement the number of activations
// by passing a negative number.
func (ct *ConsoleThrottler) markActivation(increment bool) uint64 {
if !increment {
if atomic.LoadUint64(&ct.activations) == 0 {
return 0
}
// This weird dohicky subtracts 1 from the activation count.
return atomic.AddUint64(&ct.activations, ^uint64(0))
}
return atomic.AddUint64(&ct.activations, 1)
}
// Determines if the console is currently being throttled. Calls to this function can be used to
// determine if output should be funneled along to the websocket processes.
func (ct *ConsoleThrottler) Throttled() bool {
return ct.isThrottled.Load()
}
// Starts a timer that runs in a seperate thread and will continually decrement the lines processed
// and number of activations, regardless of the current console message volume. All of the timers
// are canceled if the context passed through is canceled.
func (ct *ConsoleThrottler) StartTimer(ctx context.Context) {
system.Every(ctx, time.Duration(int64(ct.LineResetInterval))*time.Millisecond, func(_ time.Time) {
ct.isThrottled.Store(false)
atomic.StoreUint64(&ct.count, 0)
})
system.Every(ctx, time.Duration(int64(ct.DecayInterval))*time.Millisecond, func(_ time.Time) {
ct.markActivation(false)
})
}
// Handles output from a server's console. This code ensures that a server is not outputting
// an excessive amount of data to the console that could indicate a malicious or run-away process
// and lead to performance issues for other users.
//
// This was much more of a problem for the NodeJS version of the daemon which struggled to handle
// large volumes of output. However, this code is much more performant so I generally feel a lot
// better about it's abilities.
//
// However, extreme output is still somewhat of a DoS attack vector against this software since we
// are still logging it to the disk temporarily and will want to avoid dumping a huge amount of
// data all at once. These values are all configurable via the wings configuration file, however the
// defaults have been in the wild for almost two years at the time of this writing, so I feel quite
// confident in them.
//
// This function returns an error if the server should be stopped due to violating throttle constraints
// and a boolean value indicating if a throttle is being violated when it is checked.
func (ct *ConsoleThrottler) Increment(onTrigger func()) error {
if !ct.Enabled {
return nil
}
// Increment the line count and if we have now output more lines than are allowed, trigger a throttle
// activation. Once the throttle is triggered and has passed the kill at value we will trigger a server
// stop automatically.
if atomic.AddUint64(&ct.count, 1) >= ct.Lines && !ct.Throttled() {
ct.isThrottled.Store(true)
if ct.markActivation(true) >= ct.MaximumTriggerCount {
return ErrTooMuchConsoleData
}
onTrigger()
}
return nil
}
// Returns the throttler instance for the server or creates a new one.
func (s *Server) Throttler() *ConsoleThrottler {
s.throttleOnce.Do(func() {
s.throttler = &ConsoleThrottler{
isThrottled: system.NewAtomicBool(false),
ConsoleThrottles: config.Get().Throttles,
}
})
return s.throttler
}
// PublishConsoleOutputFromDaemon sends output to the server console formatted // PublishConsoleOutputFromDaemon sends output to the server console formatted
// to appear correctly as being sent from Wings. // to appear correctly as being sent from Wings.
func (s *Server) PublishConsoleOutputFromDaemon(data string) { func (s *Server) PublishConsoleOutputFromDaemon(data string) {
@ -141,3 +28,55 @@ func (s *Server) PublishConsoleOutputFromDaemon(data string) {
colorstring.Color(fmt.Sprintf("[yellow][bold][%s Daemon]:[default] %s", appName, data)), colorstring.Color(fmt.Sprintf("[yellow][bold][%s Daemon]:[default] %s", appName, data)),
) )
} }
// Throttler returns the throttler instance for the server or creates a new one.
func (s *Server) Throttler() *ConsoleThrottle {
s.throttleOnce.Do(func() {
throttles := config.Get().Throttles
period := time.Duration(throttles.Period) * time.Millisecond
s.throttler = newConsoleThrottle(throttles.Lines, period)
s.throttler.strike = func() {
s.PublishConsoleOutputFromDaemon("Server is outputting console data too quickly -- throttling...")
}
})
return s.throttler
}
type ConsoleThrottle struct {
limit *system.Rate
lock *system.Locker
strike func()
}
func newConsoleThrottle(lines uint64, period time.Duration) *ConsoleThrottle {
return &ConsoleThrottle{
limit: system.NewRate(lines, period),
lock: system.NewLocker(),
}
}
// Allow checks if the console is allowed to process more output data, or if too
// much has already been sent over the line. If there is too much output the
// strike callback function is triggered, but only if it has not already been
// triggered at this point in the process.
//
// If output is allowed, the lock on the throttler is released and the next time
// it is triggered the strike function will be re-executed.
func (ct *ConsoleThrottle) Allow() bool {
if !ct.limit.Try() {
if err := ct.lock.Acquire(); err == nil {
if ct.strike != nil {
ct.strike()
}
}
return false
}
ct.lock.Release()
return true
}
// Reset resets the console throttler internal rate limiter and overage counter.
func (ct *ConsoleThrottle) Reset() {
ct.limit.Reset()
}

62
server/console_test.go Normal file
View File

@ -0,0 +1,62 @@
package server
import (
"testing"
"time"
"github.com/franela/goblin"
)
func TestName(t *testing.T) {
g := goblin.Goblin(t)
g.Describe("ConsoleThrottler", func() {
g.It("keeps count of the number of overages in a time period", func() {
t := newConsoleThrottle(1, time.Second)
g.Assert(t.Allow()).IsTrue()
g.Assert(t.Allow()).IsFalse()
g.Assert(t.Allow()).IsFalse()
})
g.It("calls strike once per time period", func() {
t := newConsoleThrottle(1, time.Millisecond*20)
var times int
t.strike = func() {
times = times + 1
}
t.Allow()
t.Allow()
t.Allow()
time.Sleep(time.Millisecond * 100)
t.Allow()
t.Reset()
t.Allow()
t.Allow()
t.Allow()
g.Assert(times).Equal(2)
})
g.It("is properly reset", func() {
t := newConsoleThrottle(10, time.Second)
for i := 0; i < 10; i++ {
g.Assert(t.Allow()).IsTrue()
}
g.Assert(t.Allow()).IsFalse()
t.Reset()
g.Assert(t.Allow()).IsTrue()
})
})
}
func BenchmarkConsoleThrottle(b *testing.B) {
t := newConsoleThrottle(10, time.Millisecond*10)
b.ReportAllocs()
for i := 0; i < b.N; i++ {
t.Allow()
}
}

View File

@ -6,6 +6,8 @@ import (
"sync" "sync"
"time" "time"
"emperror.dev/errors"
"github.com/pterodactyl/wings/config" "github.com/pterodactyl/wings/config"
"github.com/pterodactyl/wings/environment" "github.com/pterodactyl/wings/environment"
) )
@ -57,7 +59,7 @@ func (s *Server) handleServerCrash() error {
exitCode, oomKilled, err := s.Environment.ExitState() exitCode, oomKilled, err := s.Environment.ExitState()
if err != nil { if err != nil {
return err return errors.Wrap(err, "failed to get exit state for server process")
} }
// If the system is not configured to detect a clean exit code as a crash, and the // If the system is not configured to detect a clean exit code as a crash, and the
@ -85,5 +87,5 @@ func (s *Server) handleServerCrash() error {
s.crasher.SetLastCrash(time.Now()) s.crasher.SetLastCrash(time.Now())
return s.HandlePowerAction(PowerActionStart) return errors.Wrap(s.HandlePowerAction(PowerActionStart), "failed to start server after crash detection")
} }

View File

@ -2,10 +2,10 @@ package server
import ( import (
"github.com/pterodactyl/wings/events" "github.com/pterodactyl/wings/events"
"github.com/pterodactyl/wings/system"
) )
// Defines all of the possible output events for a server. // Defines all the possible output events for a server.
// noinspection GoNameStartsWithPackageName
const ( const (
DaemonMessageEvent = "daemon message" DaemonMessageEvent = "daemon message"
InstallOutputEvent = "install output" InstallOutputEvent = "install output"
@ -18,9 +18,10 @@ const (
BackupCompletedEvent = "backup completed" BackupCompletedEvent = "backup completed"
TransferLogsEvent = "transfer logs" TransferLogsEvent = "transfer logs"
TransferStatusEvent = "transfer status" TransferStatusEvent = "transfer status"
DeletedEvent = "deleted"
) )
// Returns the server's emitter instance. // Events returns the server's emitter instance.
func (s *Server) Events() *events.Bus { func (s *Server) Events() *events.Bus {
s.emitterLock.Lock() s.emitterLock.Lock()
defer s.emitterLock.Unlock() defer s.emitterLock.Unlock()
@ -31,3 +32,24 @@ func (s *Server) Events() *events.Bus {
return s.emitter return s.emitter
} }
// Sink returns the instantiated and named sink for a server. If the sink has
// not been configured yet this function will cause a panic condition.
func (s *Server) Sink(name system.SinkName) *system.SinkPool {
sink, ok := s.sinks[name]
if !ok {
s.Log().Fatalf("attempt to access nil sink: %s", name)
}
return sink
}
// DestroyAllSinks iterates over all of the sinks configured for the server and
// destroys their instances. Note that this will cause a panic if you attempt
// to call Server.Sink() again after. This function is only used when a server
// is being deleted from the system.
func (s *Server) DestroyAllSinks() {
s.Log().Info("destroying all registered sinks for server instance")
for _, sink := range s.sinks {
sink.Destroy()
}
}

View File

@ -2,6 +2,7 @@ package filesystem
import ( import (
"archive/tar" "archive/tar"
"context"
"io" "io"
"io/fs" "io/fs"
"os" "os"
@ -12,11 +13,12 @@ import (
"emperror.dev/errors" "emperror.dev/errors"
"github.com/apex/log" "github.com/apex/log"
"github.com/juju/ratelimit" "github.com/juju/ratelimit"
"github.com/karrick/godirwalk"
"github.com/klauspost/pgzip" "github.com/klauspost/pgzip"
ignore "github.com/sabhiram/go-gitignore" ignore "github.com/sabhiram/go-gitignore"
"github.com/pterodactyl/wings/config" "github.com/pterodactyl/wings/config"
"github.com/pterodactyl/wings/internal/progress"
"github.com/pterodactyl/wings/internal/ufs"
) )
const memory = 4 * 1024 const memory = 4 * 1024
@ -28,23 +30,61 @@ var pool = sync.Pool{
}, },
} }
// TarProgress .
type TarProgress struct {
*tar.Writer
p *progress.Progress
}
// NewTarProgress .
func NewTarProgress(w *tar.Writer, p *progress.Progress) *TarProgress {
if p != nil {
p.Writer = w
}
return &TarProgress{
Writer: w,
p: p,
}
}
// Write .
func (p *TarProgress) Write(v []byte) (int, error) {
if p.p == nil {
return p.Writer.Write(v)
}
return p.p.Write(v)
}
type Archive struct { type Archive struct {
// BasePath is the absolute path to create the archive from where Files and Ignore are // Filesystem to create the archive with.
// relative to. Filesystem *Filesystem
BasePath string
// Ignore is a gitignore string (most likely read from a file) of files to ignore // Ignore is a gitignore string (most likely read from a file) of files to ignore
// from the archive. // from the archive.
Ignore string Ignore string
// Files specifies the files to archive, this takes priority over the Ignore option, if // BaseDirectory .
// unspecified, all files in the BasePath will be archived unless Ignore is set. BaseDirectory string
// Files specifies the files to archive, this takes priority over the Ignore
// option, if unspecified, all files in the BaseDirectory will be archived
// unless Ignore is set.
Files []string Files []string
// Progress wraps the writer of the archive to pass through the progress tracker.
Progress *progress.Progress
w *TarProgress
} }
// Create creates an archive at dst with all of the files defined in the // Create creates an archive at dst with all the files defined in the
// included files struct. // included Files array.
func (a *Archive) Create(dst string) error { //
// THIS IS UNSAFE TO USE IF `dst` IS PROVIDED BY A USER! ONLY USE THIS WITH
// CONTROLLED PATHS!
func (a *Archive) Create(ctx context.Context, dst string) error {
// Using os.OpenFile here is expected, as long as `dst` is not a user
// provided path.
f, err := os.OpenFile(dst, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600) f, err := os.OpenFile(dst, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600)
if err != nil { if err != nil {
return err return err
@ -62,8 +102,46 @@ func (a *Archive) Create(dst string) error {
writer = f writer = f
} }
return a.Stream(ctx, writer)
}
type walkFunc func(dirfd int, name, relative string, d ufs.DirEntry) error
// Stream streams the creation of the archive to the given writer.
func (a *Archive) Stream(ctx context.Context, w io.Writer) error {
if a.Filesystem == nil {
return errors.New("filesystem: archive.Filesystem is unset")
}
// The base directory may come with a prefixed `/`, strip it to prevent
// problems.
a.BaseDirectory = strings.TrimPrefix(a.BaseDirectory, "/")
if filesLen := len(a.Files); filesLen > 0 {
files := make([]string, filesLen)
for i, f := range a.Files {
if !strings.HasPrefix(f, a.Filesystem.Path()) {
files[i] = f
continue
}
files[i] = strings.TrimPrefix(strings.TrimPrefix(f, a.Filesystem.Path()), "/")
}
a.Files = files
}
// Choose which compression level to use based on the compression_level configuration option
var compressionLevel int
switch config.Get().System.Backups.CompressionLevel {
case "none":
compressionLevel = pgzip.NoCompression
case "best_compression":
compressionLevel = pgzip.BestCompression
default:
compressionLevel = pgzip.BestSpeed
}
// Create a new gzip writer around the file. // Create a new gzip writer around the file.
gw, _ := pgzip.NewWriterLevel(writer, pgzip.BestSpeed) gw, _ := pgzip.NewWriterLevel(w, compressionLevel)
_ = gw.SetConcurrency(1<<20, 1) _ = gw.SetConcurrency(1<<20, 1)
defer gw.Close() defer gw.Close()
@ -71,91 +149,124 @@ func (a *Archive) Create(dst string) error {
tw := tar.NewWriter(gw) tw := tar.NewWriter(gw)
defer tw.Close() defer tw.Close()
// Configure godirwalk. a.w = NewTarProgress(tw, a.Progress)
options := &godirwalk.Options{
FollowSymbolicLinks: false, fs := a.Filesystem.unixFS
Unsorted: true,
Callback: a.callback(tw),
}
// If we're specifically looking for only certain files, or have requested // If we're specifically looking for only certain files, or have requested
// that certain files be ignored we'll update the callback function to reflect // that certain files be ignored we'll update the callback function to reflect
// that request. // that request.
var callback walkFunc
if len(a.Files) == 0 && len(a.Ignore) > 0 { if len(a.Files) == 0 && len(a.Ignore) > 0 {
i := ignore.CompileIgnoreLines(strings.Split(a.Ignore, "\n")...) i := ignore.CompileIgnoreLines(strings.Split(a.Ignore, "\n")...)
callback = a.callback(func(_ int, _, relative string, _ ufs.DirEntry) error {
options.Callback = a.callback(tw, func(_ string, rp string) error { if i.MatchesPath(relative) {
if i.MatchesPath(rp) { return SkipThis
return godirwalk.SkipThis
} }
return nil return nil
}) })
} else if len(a.Files) > 0 { } else if len(a.Files) > 0 {
options.Callback = a.withFilesCallback(tw) callback = a.withFilesCallback()
} else {
callback = a.callback()
} }
// Recursively walk the path we are archiving. // Open the base directory we were provided.
return godirwalk.Walk(a.BasePath, options) dirfd, name, closeFd, err := fs.SafePath(a.BaseDirectory)
defer closeFd()
if err != nil {
return err
}
// Recursively walk the base directory.
return fs.WalkDirat(dirfd, name, func(dirfd int, name, relative string, d ufs.DirEntry, err error) error {
if err != nil {
return err
}
select {
case <-ctx.Done():
return ctx.Err()
default:
return callback(dirfd, name, relative, d)
}
})
} }
// Callback function used to determine if a given file should be included in the archive // Callback function used to determine if a given file should be included in the archive
// being generated. // being generated.
func (a *Archive) callback(tw *tar.Writer, opts ...func(path string, relative string) error) func(path string, de *godirwalk.Dirent) error { func (a *Archive) callback(opts ...walkFunc) walkFunc {
return func(path string, de *godirwalk.Dirent) error { // Get the base directory we need to strip when walking.
// Skip directories because we walking them recursively. //
if de.IsDir() { // This is important as when we are walking, the last part of the base directory
// is present on all the paths we walk.
var base string
if a.BaseDirectory != "" {
base = filepath.Base(a.BaseDirectory) + "/"
}
return func(dirfd int, name, relative string, d ufs.DirEntry) error {
// Skip directories because we are walking them recursively.
if d.IsDir() {
return nil return nil
} }
relative := filepath.ToSlash(strings.TrimPrefix(path, a.BasePath+string(filepath.Separator))) // If base isn't empty, strip it from the relative path. This fixes an
// issue when creating an archive starting from a nested directory.
//
// See https://github.com/pterodactyl/panel/issues/5030 for more details.
if base != "" {
relative = strings.TrimPrefix(relative, base)
}
// Call the additional options passed to this callback function. If any of them return // Call the additional options passed to this callback function. If any of them return
// a non-nil error we will exit immediately. // a non-nil error we will exit immediately.
for _, opt := range opts { for _, opt := range opts {
if err := opt(path, relative); err != nil { if err := opt(dirfd, name, relative, d); err != nil {
if err == SkipThis {
return nil
}
return err return err
} }
} }
// Add the file to the archive, if it is nested in a directory, // Add the file to the archive, if it is nested in a directory,
// the directory will be automatically "created" in the archive. // the directory will be automatically "created" in the archive.
return a.addToArchive(path, relative, tw) return a.addToArchive(dirfd, name, relative, d)
} }
} }
var SkipThis = errors.New("skip this")
// Pushes only files defined in the Files key to the final archive. // Pushes only files defined in the Files key to the final archive.
func (a *Archive) withFilesCallback(tw *tar.Writer) func(path string, de *godirwalk.Dirent) error { func (a *Archive) withFilesCallback() walkFunc {
return a.callback(tw, func(p string, rp string) error { return a.callback(func(_ int, _, relative string, _ ufs.DirEntry) error {
for _, f := range a.Files { for _, f := range a.Files {
// If the given doesn't match, or doesn't have the same prefix continue // Allow exact file matches, otherwise check if file is within a parent directory.
// to the next item in the loop. //
if p != f && !strings.HasPrefix(p, f) { // The slashes are added in the prefix checks to prevent partial name matches from being
// included in the archive.
if f != relative && !strings.HasPrefix(strings.TrimSuffix(relative, "/")+"/", strings.TrimSuffix(f, "/")+"/") {
continue continue
} }
// Once we have a match return a nil value here so that the loop stops and the // Once we have a match return a nil value here so that the loop stops and the
// call to this function will correctly include the file in the archive. If there // call to this function will correctly include the file in the archive. If there
// are no matches we'll never make it to this line, and the final error returned // are no matches we'll never make it to this line, and the final error returned
// will be the godirwalk.SkipThis error. // will be the ufs.SkipDir error.
return nil return nil
} }
return godirwalk.SkipThis return SkipThis
}) })
} }
// Adds a given file path to the final archive being created. // Adds a given file path to the final archive being created.
func (a *Archive) addToArchive(p string, rp string, w *tar.Writer) error { func (a *Archive) addToArchive(dirfd int, name, relative string, entry ufs.DirEntry) error {
// Lstat the file, this will give us the same information as Stat except that it will not s, err := entry.Info()
// follow a symlink to it's target automatically. This is important to avoid including
// files that exist outside the server root unintentionally in the backup.
s, err := os.Lstat(p)
if err != nil { if err != nil {
if os.IsNotExist(err) { if errors.Is(err, ufs.ErrNotExist) {
return nil return nil
} }
return errors.WrapIff(err, "failed executing os.Lstat on '%s'", rp) return errors.WrapIff(err, "failed executing os.Lstat on '%s'", name)
} }
// Skip socket files as they are unsupported by archive/tar. // Skip socket files as they are unsupported by archive/tar.
@ -173,9 +284,9 @@ func (a *Archive) addToArchive(p string, rp string, w *tar.Writer) error {
// it doesn't work. // it doesn't work.
target, err = os.Readlink(s.Name()) target, err = os.Readlink(s.Name())
if err != nil { if err != nil {
// Ignore the not exist errors specifically, since theres nothing important about that. // Ignore the not exist errors specifically, since there is nothing important about that.
if !os.IsNotExist(err) { if !os.IsNotExist(err) {
log.WithField("path", rp).WithField("readlink_err", err.Error()).Warn("failed reading symlink for target path; skipping...") log.WithField("name", name).WithField("readlink_err", err.Error()).Warn("failed reading symlink for target path; skipping...")
} }
return nil return nil
} }
@ -184,17 +295,17 @@ func (a *Archive) addToArchive(p string, rp string, w *tar.Writer) error {
// Get the tar FileInfoHeader in order to add the file to the archive. // Get the tar FileInfoHeader in order to add the file to the archive.
header, err := tar.FileInfoHeader(s, filepath.ToSlash(target)) header, err := tar.FileInfoHeader(s, filepath.ToSlash(target))
if err != nil { if err != nil {
return errors.WrapIff(err, "failed to get tar#FileInfoHeader for '%s'", rp) return errors.WrapIff(err, "failed to get tar#FileInfoHeader for '%s'", name)
} }
// Fix the header name if the file is not a symlink. // Fix the header name if the file is not a symlink.
if s.Mode()&fs.ModeSymlink == 0 { if s.Mode()&fs.ModeSymlink == 0 {
header.Name = rp header.Name = relative
} }
// Write the tar FileInfoHeader to the archive. // Write the tar FileInfoHeader to the archive.
if err := w.WriteHeader(header); err != nil { if err := a.w.WriteHeader(header); err != nil {
return errors.WrapIff(err, "failed to write tar#FileInfoHeader for '%s'", rp) return errors.WrapIff(err, "failed to write tar#FileInfoHeader for '%s'", name)
} }
// If the size of the file is less than 1 (most likely for symlinks), skip writing the file. // If the size of the file is less than 1 (most likely for symlinks), skip writing the file.
@ -216,7 +327,7 @@ func (a *Archive) addToArchive(p string, rp string, w *tar.Writer) error {
} }
// Open the file. // Open the file.
f, err := os.Open(p) f, err := a.Filesystem.unixFS.OpenFileat(dirfd, name, ufs.O_RDONLY, 0)
if err != nil { if err != nil {
if os.IsNotExist(err) { if os.IsNotExist(err) {
return nil return nil
@ -226,9 +337,8 @@ func (a *Archive) addToArchive(p string, rp string, w *tar.Writer) error {
defer f.Close() defer f.Close()
// Copy the file's contents to the archive using our buffer. // Copy the file's contents to the archive using our buffer.
if _, err := io.CopyBuffer(w, io.LimitReader(f, header.Size), buf); err != nil { if _, err := io.CopyBuffer(a.w, io.LimitReader(f, header.Size), buf); err != nil {
return errors.WrapIff(err, "failed to copy '%s' to archive", header.Name) return errors.WrapIff(err, "failed to copy '%s' to archive", header.Name)
} }
return nil return nil
} }

View File

@ -0,0 +1,122 @@
package filesystem
import (
"context"
iofs "io/fs"
"os"
"path/filepath"
"sort"
"strings"
"testing"
. "github.com/franela/goblin"
"github.com/mholt/archiver/v4"
)
func TestArchive_Stream(t *testing.T) {
g := Goblin(t)
fs, rfs := NewFs()
g.Describe("Archive", func() {
g.AfterEach(func() {
// Reset the filesystem after each run.
_ = fs.TruncateRootDirectory()
})
g.It("creates archive with intended files", func() {
g.Assert(fs.CreateDirectory("test", "/")).IsNil()
g.Assert(fs.CreateDirectory("test2", "/")).IsNil()
r := strings.NewReader("hello, world!\n")
err := fs.Write("test/file.txt", r, r.Size(), 0o644)
g.Assert(err).IsNil()
r = strings.NewReader("hello, world!\n")
err = fs.Write("test2/file.txt", r, r.Size(), 0o644)
g.Assert(err).IsNil()
r = strings.NewReader("hello, world!\n")
err = fs.Write("test_file.txt", r, r.Size(), 0o644)
g.Assert(err).IsNil()
r = strings.NewReader("hello, world!\n")
err = fs.Write("test_file.txt.old", r, r.Size(), 0o644)
g.Assert(err).IsNil()
a := &Archive{
Filesystem: fs,
Files: []string{
"test",
"test_file.txt",
},
}
// Create the archive.
archivePath := filepath.Join(rfs.root, "archive.tar.gz")
g.Assert(a.Create(context.Background(), archivePath)).IsNil()
// Ensure the archive exists.
_, err = os.Stat(archivePath)
g.Assert(err).IsNil()
// Open the archive.
genericFs, err := archiver.FileSystem(context.Background(), archivePath)
g.Assert(err).IsNil()
// Assert that we are opening an archive.
afs, ok := genericFs.(archiver.ArchiveFS)
g.Assert(ok).IsTrue()
// Get the names of the files recursively from the archive.
files, err := getFiles(afs, ".")
g.Assert(err).IsNil()
// Ensure the files in the archive match what we are expecting.
expected := []string{
"test_file.txt",
"test/file.txt",
}
// Sort the slices to ensure the comparison never fails if the
// contents are sorted differently.
sort.Strings(expected)
sort.Strings(files)
g.Assert(files).Equal(expected)
})
})
}
func getFiles(f iofs.ReadDirFS, name string) ([]string, error) {
var v []string
entries, err := f.ReadDir(name)
if err != nil {
return nil, err
}
for _, e := range entries {
entryName := e.Name()
if name != "." {
entryName = filepath.Join(name, entryName)
}
if e.IsDir() {
files, err := getFiles(f, entryName)
if err != nil {
return nil, err
}
if files == nil {
return nil, nil
}
v = append(v, files...)
continue
}
v = append(v, entryName)
}
return v, nil
}

View File

@ -0,0 +1,100 @@
// SPDX-License-Identifier: MIT
// SPDX-FileCopyrightText: Copyright (c) 2016 Matthew Holt
// Code in this file was derived from
// https://github.com/mholt/archiver/blob/v4.0.0-alpha.8/fs.go
//
// These modifications were necessary to allow us to use an already open file
// with archiver.FileFS.
package archiverext
import (
"io"
"io/fs"
"github.com/mholt/archiver/v4"
)
// FileFS allows accessing a file on disk using a consistent file system interface.
// The value should be the path to a regular file, not a directory. This file will
// be the only entry in the file system and will be at its root. It can be accessed
// within the file system by the name of "." or the filename.
//
// If the file is compressed, set the Compression field so that reads from the
// file will be transparently decompressed.
type FileFS struct {
// File is the compressed file backing the FileFS.
File fs.File
// If file is compressed, setting this field will
// transparently decompress reads.
Compression archiver.Decompressor
}
// Open opens the named file, which must be the file used to create the file system.
func (f FileFS) Open(name string) (fs.File, error) {
if err := f.checkName(name, "open"); err != nil {
return nil, err
}
if f.Compression == nil {
return f.File, nil
}
r, err := f.Compression.OpenReader(f.File)
if err != nil {
return nil, err
}
return compressedFile{f.File, r}, nil
}
// ReadDir returns a directory listing with the file as the singular entry.
func (f FileFS) ReadDir(name string) ([]fs.DirEntry, error) {
if err := f.checkName(name, "stat"); err != nil {
return nil, err
}
info, err := f.Stat(name)
if err != nil {
return nil, err
}
return []fs.DirEntry{fs.FileInfoToDirEntry(info)}, nil
}
// Stat stats the named file, which must be the file used to create the file system.
func (f FileFS) Stat(name string) (fs.FileInfo, error) {
if err := f.checkName(name, "stat"); err != nil {
return nil, err
}
return f.File.Stat()
}
func (f FileFS) checkName(name, op string) error {
if !fs.ValidPath(name) {
return &fs.PathError{Op: "open", Path: name, Err: fs.ErrInvalid}
}
// TODO: we may need better name validation.
if name != "." {
return &fs.PathError{Op: op, Path: name, Err: fs.ErrNotExist}
}
return nil
}
// compressedFile is an fs.File that specially reads
// from a decompression reader, and which closes both
// that reader and the underlying file.
type compressedFile struct {
fs.File
decomp io.ReadCloser
}
func (cf compressedFile) Read(p []byte) (int, error) {
return cf.decomp.Read(p)
}
func (cf compressedFile) Close() error {
err := cf.File.Close()
err2 := cf.decomp.Close()
if err2 != nil && err == nil {
err = err2
}
return err
}

View File

@ -1,11 +1,10 @@
package filesystem package filesystem
import ( import (
"archive/tar" "context"
"archive/zip"
"compress/gzip"
"fmt" "fmt"
"os" "io"
iofs "io/fs"
"path" "path"
"path/filepath" "path/filepath"
"strings" "strings"
@ -13,10 +12,14 @@ import (
"time" "time"
"emperror.dev/errors" "emperror.dev/errors"
"github.com/mholt/archiver/v3" "github.com/klauspost/compress/zip"
"github.com/mholt/archiver/v4"
"github.com/pterodactyl/wings/internal/ufs"
"github.com/pterodactyl/wings/server/filesystem/archiverext"
) )
// CompressFiles compresses all of the files matching the given paths in the // CompressFiles compresses all the files matching the given paths in the
// specified directory. This function also supports passing nested paths to only // specified directory. This function also supports passing nested paths to only
// compress certain files and folders when working in a larger directory. This // compress certain files and folders when working in a larger directory. This
// effectively creates a local backup, but rather than ignoring specific files // effectively creates a local backup, but rather than ignoring specific files
@ -25,160 +28,258 @@ import (
// All paths are relative to the dir that is passed in as the first argument, // All paths are relative to the dir that is passed in as the first argument,
// and the compressed file will be placed at that location named // and the compressed file will be placed at that location named
// `archive-{date}.tar.gz`. // `archive-{date}.tar.gz`.
func (fs *Filesystem) CompressFiles(dir string, paths []string) (os.FileInfo, error) { func (fs *Filesystem) CompressFiles(dir string, paths []string) (ufs.FileInfo, error) {
cleanedRootDir, err := fs.SafePath(dir) a := &Archive{Filesystem: fs, BaseDirectory: dir, Files: paths}
if err != nil {
return nil, err
}
// Take all of the paths passed in and merge them together with the root directory we've gotten.
for i, p := range paths {
paths[i] = filepath.Join(cleanedRootDir, p)
}
cleaned, err := fs.ParallelSafePath(paths)
if err != nil {
return nil, err
}
a := &Archive{BasePath: cleanedRootDir, Files: cleaned}
d := path.Join( d := path.Join(
cleanedRootDir, dir,
fmt.Sprintf("archive-%s.tar.gz", strings.ReplaceAll(time.Now().Format(time.RFC3339), ":", "")), fmt.Sprintf("archive-%s.tar.gz", strings.ReplaceAll(time.Now().Format(time.RFC3339), ":", "")),
) )
f, err := fs.unixFS.OpenFile(d, ufs.O_WRONLY|ufs.O_CREATE, 0o644)
if err := a.Create(d); err != nil {
return nil, err
}
f, err := os.Stat(d)
if err != nil { if err != nil {
_ = os.Remove(d) return nil, err
}
defer f.Close()
cw := ufs.NewCountedWriter(f)
if err := a.Stream(context.Background(), cw); err != nil {
return nil, err
}
if !fs.unixFS.CanFit(cw.BytesWritten()) {
_ = fs.unixFS.Remove(d)
return nil, newFilesystemError(ErrCodeDiskSpace, nil)
}
fs.unixFS.Add(cw.BytesWritten())
return f.Stat()
}
func (fs *Filesystem) archiverFileSystem(ctx context.Context, p string) (iofs.FS, error) {
f, err := fs.unixFS.Open(p)
if err != nil {
return nil, err
}
// Do not use defer to close `f`, it will likely be used later.
format, _, err := archiver.Identify(filepath.Base(p), f)
if err != nil && !errors.Is(err, archiver.ErrNoMatch) {
_ = f.Close()
return nil, err return nil, err
} }
if err := fs.HasSpaceFor(f.Size()); err != nil { // Reset the file reader.
_ = os.Remove(d) if _, err := f.Seek(0, io.SeekStart); err != nil {
_ = f.Close()
return nil, err return nil, err
} }
fs.addDisk(f.Size()) info, err := f.Stat()
if err != nil {
_ = f.Close()
return nil, err
}
return f, nil if format != nil {
switch ff := format.(type) {
case archiver.Zip:
// zip.Reader is more performant than ArchiveFS, because zip.Reader caches content information
// and zip.Reader can open several content files concurrently because of io.ReaderAt requirement
// while ArchiveFS can't.
// zip.Reader doesn't suffer from issue #330 and #310 according to local test (but they should be fixed anyway)
return zip.NewReader(f, info.Size())
case archiver.Archival:
return archiver.ArchiveFS{Stream: io.NewSectionReader(f, 0, info.Size()), Format: ff, Context: ctx}, nil
case archiver.Compression:
return archiverext.FileFS{File: f, Compression: ff}, nil
}
}
_ = f.Close()
return nil, archiver.ErrNoMatch
} }
// SpaceAvailableForDecompression looks through a given archive and determines // SpaceAvailableForDecompression looks through a given archive and determines
// if decompressing it would put the server over its allocated disk space limit. // if decompressing it would put the server over its allocated disk space limit.
func (fs *Filesystem) SpaceAvailableForDecompression(dir string, file string) error { func (fs *Filesystem) SpaceAvailableForDecompression(ctx context.Context, dir string, file string) error {
// Don't waste time trying to determine this if we know the server will have the space for // Don't waste time trying to determine this if we know the server will have the space for
// it since there is no limit. // it since there is no limit.
if fs.MaxDisk() <= 0 { if fs.MaxDisk() <= 0 {
return nil return nil
} }
source, err := fs.SafePath(filepath.Join(dir, file)) fsys, err := fs.archiverFileSystem(ctx, filepath.Join(dir, file))
if err != nil { if err != nil {
return err if errors.Is(err, archiver.ErrNoMatch) {
}
// Get the cached size in a parallel process so that if it is not cached we are not
// waiting an unnecessary amount of time on this call.
dirSize, err := fs.DiskUsage(false)
var size int64
// Walk over the archive and figure out just how large the final output would be from unarchiving it.
err = archiver.Walk(source, func(f archiver.File) error {
if atomic.AddInt64(&size, f.Size())+dirSize > fs.MaxDisk() {
return newFilesystemError(ErrCodeDiskSpace, nil)
}
return nil
})
if err != nil {
if IsUnknownArchiveFormatError(err) {
return newFilesystemError(ErrCodeUnknownArchive, err) return newFilesystemError(ErrCodeUnknownArchive, err)
} }
return err return err
} }
return err
var size atomic.Int64
return iofs.WalkDir(fsys, ".", func(path string, d iofs.DirEntry, err error) error {
if err != nil {
return err
}
select {
case <-ctx.Done():
// Stop walking if the context is canceled.
return ctx.Err()
default:
info, err := d.Info()
if err != nil {
return err
}
if !fs.unixFS.CanFit(size.Add(info.Size())) {
return newFilesystemError(ErrCodeDiskSpace, nil)
}
return nil
}
})
} }
// DecompressFile will decompress a file in a given directory by using the // DecompressFile will decompress a file in a given directory by using the
// archiver tool to infer the file type and go from there. This will walk over // archiver tool to infer the file type and go from there. This will walk over
// all of the files within the given archive and ensure that there is not a // all the files within the given archive and ensure that there is not a
// zip-slip attack being attempted by validating that the final path is within // zip-slip attack being attempted by validating that the final path is within
// the server data directory. // the server data directory.
func (fs *Filesystem) DecompressFile(dir string, file string) error { func (fs *Filesystem) DecompressFile(ctx context.Context, dir string, file string) error {
source, err := fs.SafePath(filepath.Join(dir, file)) f, err := fs.unixFS.Open(filepath.Join(dir, file))
if err != nil { if err != nil {
return err return err
} }
// Ensure that the source archive actually exists on the system. defer f.Close()
if _, err := os.Stat(source); err != nil {
return errors.WithStack(err)
}
// Walk all of the files in the archiver file and write them to the disk. If any // Identify the type of archive we are dealing with.
// directory is encountered it will be skipped since we handle creating any missing format, input, err := archiver.Identify(filepath.Base(file), f)
// directories automatically when writing files.
err = archiver.Walk(source, func(f archiver.File) error {
if f.IsDir() {
return nil
}
p := filepath.Join(dir, ExtractNameFromArchive(f))
// If it is ignored, just don't do anything with the file and skip over it.
if err := fs.IsIgnored(p); err != nil {
return nil
}
if err := fs.Writefile(p, f); err != nil {
return wrapError(err, source)
}
// Update the file permissions to the one set in the archive.
if err := fs.Chmod(p, f.Mode()); err != nil {
return wrapError(err, source)
}
// Update the file modification time to the one set in the archive.
if err := fs.Chtimes(p, f.ModTime(), f.ModTime()); err != nil {
return wrapError(err, source)
}
return nil
})
if err != nil { if err != nil {
if IsUnknownArchiveFormatError(err) { if errors.Is(err, archiver.ErrNoMatch) {
return newFilesystemError(ErrCodeUnknownArchive, err) return newFilesystemError(ErrCodeUnknownArchive, err)
} }
return err return err
} }
return nil
return fs.extractStream(ctx, extractStreamOptions{
FileName: file,
Directory: dir,
Format: format,
Reader: input,
})
} }
// ExtractNameFromArchive looks at an archive file to try and determine the name // ExtractStreamUnsafe .
// for a given element in an archive. Because of... who knows why, each file type func (fs *Filesystem) ExtractStreamUnsafe(ctx context.Context, dir string, r io.Reader) error {
// uses different methods to determine the file name. format, input, err := archiver.Identify("archive.tar.gz", r)
// if err != nil {
// If there is a archiver.File#Sys() value present we will try to use the name if errors.Is(err, archiver.ErrNoMatch) {
// present in there, otherwise falling back to archiver.File#Name() if all else return newFilesystemError(ErrCodeUnknownArchive, err)
// fails. Without this logic present, some archive types such as zip/tars/etc. }
// will write all of the files to the base directory, rather than the nested return err
// directory that is expected.
//
// For files like ".rar" types, there is no f.Sys() value present, and the value
// of archiver.File#Name() will be what you need.
func ExtractNameFromArchive(f archiver.File) string {
sys := f.Sys()
// Some archive types won't have a value returned when you call f.Sys() on them,
// such as ".rar" archives for example. In those cases the only thing you can do
// is hope that "f.Name()" is actually correct for them.
if sys == nil {
return f.Name()
}
switch s := sys.(type) {
case *tar.Header:
return s.Name
case *gzip.Header:
return s.Name
case *zip.FileHeader:
return s.Name
default:
return f.Name()
} }
return fs.extractStream(ctx, extractStreamOptions{
Directory: dir,
Format: format,
Reader: input,
})
}
type extractStreamOptions struct {
// The directory to extract the archive to.
Directory string
// File name of the archive.
FileName string
// Format of the archive.
Format archiver.Format
// Reader for the archive.
Reader io.Reader
}
func (fs *Filesystem) extractStream(ctx context.Context, opts extractStreamOptions) error {
// See if it's a compressed archive, such as TAR or a ZIP
ex, ok := opts.Format.(archiver.Extractor)
if !ok {
// If not, check if it's a single-file compression, such as
// .log.gz, .sql.gz, and so on
de, ok := opts.Format.(archiver.Decompressor)
if !ok {
return nil
}
// Strip the compression suffix
p := filepath.Join(opts.Directory, strings.TrimSuffix(opts.FileName, opts.Format.Name()))
// Make sure it's not ignored
if err := fs.IsIgnored(p); err != nil {
return nil
}
reader, err := de.OpenReader(opts.Reader)
if err != nil {
return err
}
defer reader.Close()
// Open the file for creation/writing
f, err := fs.unixFS.OpenFile(p, ufs.O_WRONLY|ufs.O_CREATE, 0o644)
if err != nil {
return err
}
defer f.Close()
// Read in 4 KB chunks
buf := make([]byte, 4096)
for {
n, err := reader.Read(buf)
if n > 0 {
// Check quota before writing the chunk
if quotaErr := fs.HasSpaceFor(int64(n)); quotaErr != nil {
return quotaErr
}
// Write the chunk
if _, writeErr := f.Write(buf[:n]); writeErr != nil {
return writeErr
}
// Add to quota
fs.addDisk(int64(n))
}
if err != nil {
// EOF are expected
if err == io.EOF {
break
}
// Return any other
return err
}
}
return nil
}
// Decompress and extract archive
return ex.Extract(ctx, opts.Reader, nil, func(ctx context.Context, f archiver.File) error {
if f.IsDir() {
return nil
}
p := filepath.Join(opts.Directory, f.NameInArchive)
// If it is ignored, just don't do anything with the file and skip over it.
if err := fs.IsIgnored(p); err != nil {
return nil
}
r, err := f.Open()
if err != nil {
return err
}
defer r.Close()
if err := fs.Write(p, r, f.Size(), f.Mode()); err != nil {
return wrapError(err, opts.FileName)
}
// Update the file modification time to the one set in the archive.
if err := fs.Chtimes(p, f.ModTime(), f.ModTime()); err != nil {
return wrapError(err, opts.FileName)
}
return nil
})
} }

View File

@ -1,18 +1,20 @@
package filesystem package filesystem
import ( import (
"context"
"os" "os"
"sync/atomic"
"testing" "testing"
. "github.com/franela/goblin" . "github.com/franela/goblin"
) )
// Given an archive named test.{ext}, with the following file structure: // Given an archive named test.{ext}, with the following file structure:
//
// test/ // test/
// |──inside/ // |──inside/
// |────finside.txt // |────finside.txt
// |──outside.txt // |──outside.txt
//
// this test will ensure that it's being decompressed as expected // this test will ensure that it's being decompressed as expected
func TestFilesystem_DecompressFile(t *testing.T) { func TestFilesystem_DecompressFile(t *testing.T) {
g := Goblin(t) g := Goblin(t)
@ -28,7 +30,7 @@ func TestFilesystem_DecompressFile(t *testing.T) {
g.Assert(err).IsNil() g.Assert(err).IsNil()
// decompress // decompress
err = fs.DecompressFile("/", "test."+ext) err = fs.DecompressFile(context.Background(), "/", "test."+ext)
g.Assert(err).IsNil() g.Assert(err).IsNil()
// make sure everything is where it is supposed to be // make sure everything is where it is supposed to be
@ -46,9 +48,7 @@ func TestFilesystem_DecompressFile(t *testing.T) {
} }
g.AfterEach(func() { g.AfterEach(func() {
rfs.reset() _ = fs.TruncateRootDirectory()
atomic.StoreInt64(&fs.diskUsed, 0)
atomic.StoreInt64(&fs.diskLimit, 0)
}) })
}) })
} }

View File

@ -1,26 +1,29 @@
package filesystem package filesystem
import ( import (
"golang.org/x/sys/unix"
"slices"
"sync" "sync"
"sync/atomic" "sync/atomic"
"syscall"
"time" "time"
"emperror.dev/errors" "emperror.dev/errors"
"github.com/apex/log" "github.com/apex/log"
"github.com/karrick/godirwalk"
"github.com/pterodactyl/wings/internal/ufs"
) )
type SpaceCheckingOpts struct { type SpaceCheckingOpts struct {
AllowStaleResponse bool AllowStaleResponse bool
} }
// TODO: can this be replaced with some sort of atomic? Like atomic.Pointer?
type usageLookupTime struct { type usageLookupTime struct {
sync.RWMutex sync.RWMutex
value time.Time value time.Time
} }
// Update the last time that a disk space lookup was performed. // Set sets the last time that a disk space lookup was performed.
func (ult *usageLookupTime) Set(t time.Time) { func (ult *usageLookupTime) Set(t time.Time) {
ult.Lock() ult.Lock()
ult.value = t ult.value = t
@ -35,14 +38,15 @@ func (ult *usageLookupTime) Get() time.Time {
return ult.value return ult.value
} }
// Returns the maximum amount of disk space that this Filesystem instance is allowed to use. // MaxDisk returns the maximum amount of disk space that this Filesystem
// instance is allowed to use.
func (fs *Filesystem) MaxDisk() int64 { func (fs *Filesystem) MaxDisk() int64 {
return atomic.LoadInt64(&fs.diskLimit) return fs.unixFS.Limit()
} }
// Sets the disk space limit for this Filesystem instance. // SetDiskLimit sets the disk space limit for this Filesystem instance.
func (fs *Filesystem) SetDiskLimit(i int64) { func (fs *Filesystem) SetDiskLimit(i int64) {
atomic.SwapInt64(&fs.diskLimit, i) fs.unixFS.SetLimit(i)
} }
// The same concept as HasSpaceAvailable however this will return an error if there is // The same concept as HasSpaceAvailable however this will return an error if there is
@ -65,13 +69,13 @@ func (fs *Filesystem) HasSpaceErr(allowStaleValue bool) error {
func (fs *Filesystem) HasSpaceAvailable(allowStaleValue bool) bool { func (fs *Filesystem) HasSpaceAvailable(allowStaleValue bool) bool {
size, err := fs.DiskUsage(allowStaleValue) size, err := fs.DiskUsage(allowStaleValue)
if err != nil { if err != nil {
log.WithField("root", fs.root).WithField("error", err).Warn("failed to determine root fs directory size") log.WithField("root", fs.Path()).WithField("error", err).Warn("failed to determine root fs directory size")
} }
// If space is -1 or 0 just return true, means they're allowed unlimited. // If space is -1 or 0 just return true, means they're allowed unlimited.
// //
// Technically we could skip disk space calculation because we don't need to check if the // Technically we could skip disk space calculation because we don't need to check if the
// server exceeds it's limit but because this method caches the disk usage it would be best // server exceeds its limit but because this method caches the disk usage it would be best
// to calculate the disk usage and always return true. // to calculate the disk usage and always return true.
if fs.MaxDisk() == 0 { if fs.MaxDisk() == 0 {
return true return true
@ -84,7 +88,7 @@ func (fs *Filesystem) HasSpaceAvailable(allowStaleValue bool) bool {
// function for critical logical checks. It should only be used in areas where the actual disk usage // function for critical logical checks. It should only be used in areas where the actual disk usage
// does not need to be perfect, e.g. API responses for server resource usage. // does not need to be perfect, e.g. API responses for server resource usage.
func (fs *Filesystem) CachedUsage() int64 { func (fs *Filesystem) CachedUsage() int64 {
return atomic.LoadInt64(&fs.diskUsed) return fs.unixFS.Usage()
} }
// Internal helper function to allow other parts of the codebase to check the total used disk space // Internal helper function to allow other parts of the codebase to check the total used disk space
@ -114,14 +118,14 @@ func (fs *Filesystem) DiskUsage(allowStaleValue bool) (int64, error) {
// currently performing a lookup, just do the disk usage calculation in the background. // currently performing a lookup, just do the disk usage calculation in the background.
go func(fs *Filesystem) { go func(fs *Filesystem) {
if _, err := fs.updateCachedDiskUsage(); err != nil { if _, err := fs.updateCachedDiskUsage(); err != nil {
log.WithField("root", fs.root).WithField("error", err).Warn("failed to update fs disk usage from within routine") log.WithField("root", fs.Path()).WithField("error", err).Warn("failed to update fs disk usage from within routine")
} }
}(fs) }(fs)
} }
} }
// Return the currently cached value back to the calling function. // Return the currently cached value back to the calling function.
return atomic.LoadInt64(&fs.diskUsed), nil return fs.unixFS.Usage(), nil
} }
// Updates the currently used disk space for a server. // Updates the currently used disk space for a server.
@ -149,63 +153,56 @@ func (fs *Filesystem) updateCachedDiskUsage() (int64, error) {
// error encountered. // error encountered.
fs.lastLookupTime.Set(time.Now()) fs.lastLookupTime.Set(time.Now())
atomic.StoreInt64(&fs.diskUsed, size) fs.unixFS.SetUsage(size)
return size, err return size, err
} }
// Determines the directory size of a given location by running parallel tasks to iterate // DirectorySize calculates the size of a directory and its descendants.
// through all of the folders. Returns the size in bytes. This can be a fairly taxing operation func (fs *Filesystem) DirectorySize(root string) (int64, error) {
// on locations with tons of files, so it is recommended that you cache the output. dirfd, name, closeFd, err := fs.unixFS.SafePath(root)
func (fs *Filesystem) DirectorySize(dir string) (int64, error) { defer closeFd()
d, err := fs.SafePath(dir)
if err != nil { if err != nil {
return 0, err return 0, err
} }
var size int64 var hardLinks []uint64
var st syscall.Stat_t
err = godirwalk.Walk(d, &godirwalk.Options{ var size atomic.Int64
Unsorted: true, err = fs.unixFS.WalkDirat(dirfd, name, func(dirfd int, name, _ string, d ufs.DirEntry, err error) error {
Callback: func(p string, e *godirwalk.Dirent) error { if err != nil {
// If this is a symlink then resolve the final destination of it before trying to continue walking return errors.Wrap(err, "walkdirat err")
// over its contents. If it resolves outside the server data directory just skip everything else for }
// it. Otherwise, allow it to continue.
if e.IsSymlink() {
if _, err := fs.SafePath(p); err != nil {
if IsErrorCode(err, ErrCodePathResolution) {
return godirwalk.SkipThis
}
return err
}
}
if !e.IsDir() {
syscall.Lstat(p, &st)
atomic.AddInt64(&size, st.Size)
}
// Only calculate the size of regular files.
if !d.Type().IsRegular() {
return nil return nil
}, }
})
return size, errors.WrapIf(err, "server/filesystem: directorysize: failed to walk directory") info, err := fs.unixFS.Lstatat(dirfd, name)
if err != nil {
return errors.Wrap(err, "lstatat err")
}
var sysFileInfo = info.Sys().(*unix.Stat_t)
if sysFileInfo.Nlink > 1 {
// Hard links have the same inode number
if slices.Contains(hardLinks, sysFileInfo.Ino) {
// Don't add hard links size twice
return nil
} else {
hardLinks = append(hardLinks, sysFileInfo.Ino)
}
}
size.Add(info.Size())
return nil
})
return size.Load(), errors.WrapIf(err, "server/filesystem: directorysize: failed to walk directory")
} }
// Helper function to determine if a server has space available for a file of a given size.
// If space is available, no error will be returned, otherwise an ErrNotEnoughSpace error
// will be raised.
func (fs *Filesystem) HasSpaceFor(size int64) error { func (fs *Filesystem) HasSpaceFor(size int64) error {
if fs.MaxDisk() == 0 { if !fs.unixFS.CanFit(size) {
return nil
}
s, err := fs.DiskUsage(true)
if err != nil {
return err
}
if (s + size) > fs.MaxDisk() {
return newFilesystemError(ErrCodeDiskSpace, nil) return newFilesystemError(ErrCodeDiskSpace, nil)
} }
return nil return nil
@ -213,24 +210,5 @@ func (fs *Filesystem) HasSpaceFor(size int64) error {
// Updates the disk usage for the Filesystem instance. // Updates the disk usage for the Filesystem instance.
func (fs *Filesystem) addDisk(i int64) int64 { func (fs *Filesystem) addDisk(i int64) int64 {
size := atomic.LoadInt64(&fs.diskUsed) return fs.unixFS.Add(i)
// Sorry go gods. This is ugly but the best approach I can come up with for right
// now without completely re-evaluating the logic we use for determining disk space.
//
// Normally I would just be using the atomic load right below, but I'm not sure about
// the scenarios where it is 0 because nothing has run that would trigger a disk size
// calculation?
//
// Perhaps that isn't even a concern for the sake of this?
if !fs.isTest {
size, _ = fs.DiskUsage(true)
}
// If we're dropping below 0 somehow just cap it to 0.
if (size + i) < 0 {
return atomic.SwapInt64(&fs.diskUsed, 0)
}
return atomic.AddInt64(&fs.diskUsed, i)
} }

Some files were not shown because too many files have changed in this diff Show More