Dane Everitt
ddfd6d9cce
Modify backup process to utilize contexts and exponential backoffs
...
If a request to upload a file part to S3 fails for any 5xx reason it will begin using an exponential backoff to keep re-trying the upload until we've reached a minute of trying to access the endpoint.
This should resolve temporary resolution issues with URLs and certain S3 compatiable systems such as B2 that sometimes return a 5xx error and just need a retry to be successful.
Also supports using the server context to ensure backups are terminated when a server is deleted, and removes the http call without a timeout, replacing it with a 2 hour timeout to account for connections as slow as 10Mbps on a huge file upload.
2021-05-02 12:28:36 -07:00
Julien Tant
35b2c420ec
add decompress tests
2021-04-25 16:44:54 -07:00
Dane Everitt
daaef5044e
Correctly determine name for archive files when decompressing; closes pterodactyl/panel#3296
2021-04-25 15:36:00 -07:00
Dane Everitt
0676a82a21
Add better error handling for filesystem
2021-04-17 13:29:18 -07:00
Dane Everitt
4b244e96fb
Fix .rar file decompression; closes pterodactyl/panel#3267
2021-04-17 13:13:37 -07:00
Dane Everitt
f57c24002e
More API response fixing
2021-04-04 10:20:27 -07:00
Dane Everitt
2e0496c1f9
Add note about handling of UTF-8 sequences in properties files.
2021-04-03 14:02:37 -07:00
Dane Everitt
76b7967fef
Merge pull request #88 from Antony1060/develop
...
Added app name
2021-04-03 11:13:29 -07:00
Dane Everitt
1b1eaa3171
Avoid expensive copies of the config for every line output
2021-04-03 11:11:36 -07:00
Dane Everitt
b448310a33
Correctly return servers installed on wings and their resource usage
2021-04-03 11:08:26 -07:00
antony1060
0c17e240f4
Added app name
2021-03-24 10:26:03 +01:00
Matthew Penner
471886dd34
internally mark if a server is restoring to restrict actions
2021-03-12 16:19:35 -07:00
Matthew Penner
0e3778ac47
transfers: use backup archiver
2021-03-07 11:04:15 -07:00
Dane Everitt
fb2dc39a47
Remove some forgotten debug
2021-03-03 21:00:58 -08:00
Dane Everitt
0919fb2da6
Improve error handling and reporting for server installation & process boot
2021-03-03 20:56:18 -08:00
Dane Everitt
2eb721bbe7
Merge branch 'dane/api-cleanup' into develop
2021-02-23 21:25:10 -08:00
Matthew Penner
5c56ddf5d6
fs: only mkdir and chown if not exists
2021-02-10 13:58:47 -07:00
Matthew Penner
683c766d0f
fs: fix potential panic when chowning a file
2021-02-05 13:30:02 -07:00
Dane Everitt
1da415c177
Revert "Don't tempt people with this public interface"
...
This reverts commit e3b0b91912
2021-02-01 21:43:10 -08:00
Dane Everitt
e3b0b91912
Don't tempt people with this public interface
2021-02-01 21:33:03 -08:00
Dane Everitt
bfff094216
Get wings booting properly
2021-02-01 21:32:34 -08:00
Dane Everitt
98c68142cd
Remove all of the remaining API logic and port it all to the remote.Client type
2021-02-01 21:28:46 -08:00
Dane Everitt
62cbe5e135
Migrate SFTP endpoints
2021-02-01 20:59:17 -08:00
Dane Everitt
6775c17324
Nuke more API code and begin consolidation process
2021-02-01 20:50:23 -08:00
Dane Everitt
1393937904
Fix race condition and flawed logic mis-querying panel for servers; closes pterodactyl/panel#3059
2021-02-01 20:26:15 -08:00
Matthew Penner
de9b413bc2
Call cancel function for context to not leak resources
2021-02-01 20:09:24 -07:00
Dane Everitt
ba6cec9615
Merge branch 'dane/backup-restore' into develop
2021-01-30 18:51:35 -08:00
Dane Everitt
4bd18f7dd8
Send restoration status information to the console
2021-01-30 18:49:07 -08:00
Dane Everitt
adc0732af3
Better error handling and logging for restorations
2021-01-30 18:43:35 -08:00
Dane Everitt
df721f45f8
Reset the whole build block to avoid merge issues with 0 values
2021-01-27 21:41:13 -08:00
Dane Everitt
4411493006
Merge branch 'develop' into dane/backup-restore
2021-01-25 20:32:32 -08:00
Dane Everitt
31757a68a9
Fix race condition
2021-01-25 20:29:34 -08:00
Dane Everitt
f3a6ee7a45
re-refactor code
2021-01-25 20:28:24 -08:00
Dane Everitt
ab86fb703a
Merge branch 'develop' into schrej/refactor
2021-01-25 19:31:16 -08:00
Dane Everitt
981071cda8
Merge branch 'develop' into dane/backup-restore
2021-01-25 19:17:14 -08:00
Dane Everitt
93506994a5
Ensure the root directory for a server is always create when booting wings
2021-01-23 10:45:29 -08:00
Jakob Schrettenbrunner
82f70c2755
derp: rename panelapi to remote
2021-01-22 22:38:11 +00:00
Dane Everitt
5d070cbdc5
Handle edge case where a user triggers an install when the server has no data directory
2021-01-21 20:58:52 -08:00
Dane Everitt
3f84ee694b
Get backups restoring kinda
2021-01-20 20:03:14 -08:00
Dane Everitt
63dac51692
Include a better stack here
2021-01-18 21:22:37 -08:00
Dane Everitt
6ef0bd7496
Get general concept for backup resotration using a unified interface implemented
2021-01-18 21:20:58 -08:00
Dane Everitt
66b6f40b61
Fix import cycle issue
2021-01-17 21:05:51 -08:00
Dane Everitt
7dd0acebc0
Initial untested pass at restoring from local backups
2021-01-16 18:06:22 -08:00
Dane Everitt
2968ea3498
Modify stat to embed os.FileInfo differently and update file content reader
2021-01-16 12:03:55 -08:00
Dane Everitt
67ecbd667a
Minor improvements to logic around decompression
2021-01-16 11:48:30 -08:00
Dane Everitt
05c04c4350
Update remaining logic using viper
2021-01-14 20:19:28 -08:00
Dane Everitt
80faea3286
yoink viper back out of code, simplify some config logic
2021-01-14 20:11:01 -08:00
Dane Everitt
d45a159456
Fix tests
2021-01-12 20:07:00 -08:00
Dane Everitt
e9e70b6081
Better error handling; skip file when unarchiving
2021-01-10 17:01:41 -08:00
Dane Everitt
b10e4dd437
Better error handling for access to denylist files
2021-01-10 16:43:33 -08:00
Dane Everitt
2c1b211280
Add base idea for denying write access to certain files; ref pterodactyl/panel#569
2021-01-10 16:33:39 -08:00
Dane Everitt
b0fa9619de
debug
2021-01-10 15:09:33 -08:00
Dane Everitt
c228acaafc
Massive refactor of SFTP system now that it is deeply integrated with Wings
2021-01-10 14:25:39 -08:00
Jakob Schrettenbrunner
8192244fec
replace servers.servers singleton with server.Manager
2021-01-10 02:37:02 +01:00
Jakob Schrettenbrunner
648072436f
organize imports with gopls format
2021-01-10 01:22:39 +00:00
Matthew Penner
66c9be357c
Potential fix for servers being marked as stopping after being marked as offline
2021-01-07 19:32:15 -07:00
Dane Everitt
d6e189df5e
Update commentary
2020-12-27 16:53:40 -08:00
Dane Everitt
68749616ad
Ignore symlink errors
2020-12-27 16:30:00 -08:00
Dane Everitt
7549eb13a0
Merge branch 'develop' of https://github.com/pterodactyl/wings into develop
2020-12-27 16:17:17 -08:00
Dane Everitt
902f9f5944
Include more (and customizable) context with backup logs
2020-12-27 16:16:40 -08:00
Matthew Penner
2cf24994d9
Uncomment important code
2020-12-27 13:56:45 -07:00
Matthew Penner
3a7c4822f8
Improve logged stacktraces
2020-12-27 13:55:58 -07:00
Matthew Penner
b26db99ee7
Fix part_id field in logs
2020-12-27 13:02:50 -07:00
Matthew Penner
de4d2f4724
Make logs follow a more consistent format
2020-12-27 13:01:34 -07:00
Matthew Penner
640e30de8a
Ignore symlinks with missing target, add better backup logging, update copyright year to 2021
2020-12-27 12:54:18 -07:00
Matthew Penner
e75118e0f0
Maybe this will fix shit?
2020-12-27 12:21:26 -07:00
Matthew Penner
a4c8b8714b
Bump all dependencies
2020-12-27 11:49:08 -07:00
Dane Everitt
22c53c365a
Fix installer to not explode on long lines; closes pterodactyl/panel#2773
2020-12-25 17:05:01 -08:00
Dane Everitt
59c30c2842
Fix use of atomics in codebase
2020-12-25 17:04:18 -08:00
Dane Everitt
31d4c1d34f
Close the file when done
2020-12-25 14:01:25 -08:00
Dane Everitt
0cdfdc725c
Cleanup ignored file reader
2020-12-25 14:00:01 -08:00
Dane Everitt
bb132243ed
Merge branch 'develop' of https://github.com/pterodactyl/wings into develop
2020-12-25 13:57:33 -08:00
Dane Everitt
d96115325a
Add file helper
2020-12-25 13:57:26 -08:00
Matthew Penner
a450abc080
Don't read directories either, add completion log for transfers
2020-12-25 14:46:33 -07:00
Matthew Penner
e05c601325
Lets not read an entire file into memory
2020-12-25 14:40:53 -07:00
Matthew Penner
5c78cb9ab3
Add transfer logging output ( #77 )
...
Co-authored-by: Dane Everitt <dane@daneeveritt.com>
2020-12-25 13:32:41 -08:00
Matthew Penner
901ab1157d
Rework archiving logic to be more consistent and less impactful on disk IO ( #79 )
...
Co-authored-by: Dane Everitt <dane@daneeveritt.com>
2020-12-25 11:52:57 -08:00
Dane Everitt
c0523df696
Add a server context that gets canceled when a server is deleted
2020-12-25 11:21:09 -08:00
Dane Everitt
f8282c56cb
Avoid race conditions from mismatched use of atomic & sync.Mutex
2020-12-20 13:13:10 -08:00
Dane Everitt
c718da20e3
Check disk space before trying a write from the downloader; don't make empty directories if we can't even write the file
2020-12-20 11:17:53 -08:00
Dane Everitt
9c53436470
Check disk before even trying to run the download
2020-12-20 11:08:01 -08:00
Dane Everitt
3495fb1c76
Less racey logic for resource usage
2020-12-16 22:03:35 -08:00
Dane Everitt
16118874cf
Don't obliterate the lock
2020-12-16 20:21:22 -08:00
Dane Everitt
59a111de39
Invite our lord and savior emperror back to the table
2020-12-15 21:56:53 -08:00
Dane Everitt
3a26a5d39d
Convert all filesystem error types into the same Error struct
2020-12-15 20:51:13 -08:00
Matthew Penner
8e29ffed50
Add configurable disk write speed limit for backups ( #74 )
...
* Add configurable disk write speed limit for backups
2020-12-08 09:13:48 -07:00
Dane Everitt
a18f60bd05
Merge branch 'develop' of https://github.com/pterodactyl/wings into develop
2020-12-06 15:35:04 -08:00
Dane Everitt
deea5babbc
Publish message when error occurs during crash handling
2020-12-06 15:34:58 -08:00
Matthew Penner
b52c3fb61e
Cleanup backup/backup_s3.go
2020-12-06 15:25:11 -07:00
Matthew Penner
ee08829a28
s3 backups: handle CompleteMultipartUpload and AbortMultipartUpload on the panel
2020-12-06 13:56:17 -07:00
Matthew Penner
ba0a1a651e
Only use the permission bits for ModeBits
2020-11-29 16:13:07 -07:00
Matthew Penner
068f41393d
Return mode_bits when listing files in a directory
2020-11-29 14:20:24 -07:00
Matthew Penner
f62f714863
Add chmod endpoint to server files
2020-11-29 13:07:45 -07:00
Matthew Penner
de51fd1c51
Error handling improvements ( #71 )
...
* Remove `emperror.dev/errors`, remove all `errors#Wrap` and `errors#WithStack` calls
* Improve logging in `server/backup.go`
2020-11-28 16:57:10 -07:00
Matthew Penner
40c70673cd
Use registry authentication when pulling install image, fixes https://github.com/pterodactyl/panel/issues/2589
2020-11-20 14:36:29 -07:00
Matthew Penner
73b221d022
Switch old Server#GetState() calls over to Server#Environment#State()
2020-11-20 14:35:29 -07:00
Matthew Penner
287b286940
Oh, that's why it was done like that
2020-11-18 18:44:25 -07:00
Matthew Penner
1d0e85cf55
Attempt to fix 'archive/tar: write too long' error when creating a backup
2020-11-18 16:30:34 -07:00
Dane Everitt
8081c83de4
Allow disabling disk checking with 0 value for check interval
2020-11-12 20:50:03 -08:00
Matthew Penner
2197d82957
Clean paths from AllowedMounts to ensure consistency
2020-11-09 16:58:01 -07:00
Dane Everitt
de0c67d066
Re-enable disabled tests
2020-11-08 15:17:47 -08:00
Dane Everitt
68bdcb3cbc
Fix error handling and get tests back to working
2020-11-08 15:15:39 -08:00
Dane Everitt
ef999a039c
Perhaps better error stacks for backups/archives; ref #2418
2020-11-08 14:07:26 -08:00
Dane Everitt
be9d1a3986
Replace error handling package with emperror; add better reporting for errors escaping server root
2020-11-08 13:52:20 -08:00
Dane Everitt
4b17ac4f1c
Fix atomic string handling for JSON output
2020-11-06 22:22:33 -08:00
Dane Everitt
944d381778
Improve server state logical handling; allow setting state directly on the environment
2020-11-06 21:53:00 -08:00
Dane Everitt
13c253780a
Use 60 seconds, not 15
2020-11-01 17:26:45 -08:00
Dane Everitt
fe572beada
Stop servers when exceeding their disk limits; closes pterodactyl/panel#2638
2020-11-01 17:20:42 -08:00
Matthew Penner
05cfb59e18
Add better logging for s3 backup uploads
2020-11-01 15:36:16 -07:00
Matthew Penner
5475cb02c1
Cleanup s3 backup uploads, add request timeouts
2020-11-01 15:23:46 -07:00
Matthew Penner
1239b1c0ca
Switch to s3 multipart uploads for backups
2020-11-01 15:23:46 -07:00
Dane Everitt
ca3becfb55
Correctly handle server responses and instantiation
2020-10-31 11:13:40 -07:00
Dane Everitt
41a67933eb
Use a paginated loop to fetch servers from the Panel rather than a single massive request
2020-10-31 11:05:53 -07:00
Dane Everitt
334b3e8d10
Refactor HTTP endpoints to be less complicated and follow better standards
2020-10-31 10:04:20 -07:00
Matthew Penner
56789196d4
Disable flaky test so I can test cross-platform building
2020-10-17 15:47:30 -06:00
Matthew Penner
70804dd20f
Remoe stat_arm.go, make stat_linux.go with with both 32 and 64 bit systems
2020-10-17 15:45:14 -06:00
Matthew Penner
19d821aab5
Fix arm build
2020-10-17 15:41:13 -06:00
Matthew Penner
4ce35d3d17
Fix race in filesystem_test.go
2020-10-17 15:31:40 -06:00
Dane Everitt
c031d37b91
Use single call to create/return the docker client
2020-10-17 14:04:56 -07:00
Dane Everitt
19051c99ef
Support arm builds
2020-10-17 13:52:38 -07:00
Jakob
e185f597ba
add version negotiation to server install docker client
2020-10-17 21:41:01 +02:00
Dane Everitt
947279a07c
Don't abort entire boot process due to one bad server egg; closes pterodactyl/panel#2448
2020-10-17 12:06:47 -07:00
Dane Everitt
80387bc294
Use more easily configurable timezone, remove /etc/timezone mounts from containers; closes pterodactyl/panel#2513
...
If this does not completely solve the issue in containers then we need to evaluate the image being used to determine what changes need to happen to the image itself to support the timezone.
ref pterodactyl/panel#2239
ref pterodactyl/panel#2329
ref pterodactyl/panel#2389
2020-10-17 11:35:20 -07:00
Dane Everitt
e29789114c
Fix server disk usage not being reported properly; closes pterodactyl/panel#2445
2020-10-11 15:02:37 -07:00
Dane Everitt
37e59e6928
Close connected sockets when a server is deleted; closes pterodactyl/panel#2428
2020-10-03 20:46:29 -07:00
Dane Everitt
e02c197585
More archiver code cleanup; ref pterodactyl/panel#2438
2020-10-03 19:13:41 -07:00
Dane Everitt
d9109cbf5a
Correctly detect os.IsNotExist error; ref pterodactyl/panel#2438
2020-10-03 19:09:12 -07:00
Dane Everitt
6bc8b1a567
Merge branch 'dane/fs-tests' into develop
2020-10-01 21:30:39 -07:00
Dane Everitt
244640d0c1
[security] remove "SafeJoin" function
...
This function was not actually safe in theory. If an unknown stat source was passed in it would be possible for a symlinked file to not be detected as a symlink, thus skipping any safe path resolutions.
This would happen if the stat source was a regular os.Stat call and not an os.Lstat call, but since there is no way to differentiate between those two in the code, it is safer to just manually apply this logic in the positions where we _know_ for certain that we're working with the results of an Lstat call.
2020-10-01 21:28:38 -07:00
Dane Everitt
e3e89a2ecc
Cover symlink attacks with test cases
2020-10-01 21:13:42 -07:00
Dane Everitt
90ae815b1d
Return tests to passing state
2020-10-01 20:40:25 -07:00
Keyton Stanier
b92fab83c8
Removed stray . in ./mnt/install
2020-10-01 18:00:26 -04:00
Dane Everitt
367fdfad54
Simplify copy file logic
2020-09-30 21:53:50 -07:00
Dane Everitt
ee460686d6
Make delete more synchronous
2020-09-30 21:47:42 -07:00
Dane Everitt
9b7c0fb7f3
Steal tests from other branch that is being discarded, attempt to get at least one of them to pass; WIP
2020-09-30 21:46:32 -07:00
Dane Everitt
0f7bb1a371
Refactor filesystem to not be dependent on a server struct
2020-09-27 12:24:08 -07:00
Dane Everitt
de30e2fcc9
Dont attempt to get size within archive process, will return empty; ref pterodactyl/panel#2420
...
The stat call is operating against an unflushed file if called in the archive function, so you'll just get the emtpy archive size, rather than the final size.
Plus, we only used the file stat in one place, so slight efficiency win?
2020-09-27 11:16:38 -07:00
Dane Everitt
0b9ad3de05
More file size confirmations before performing some actions; closes pterodactyl/panel#2420
2020-09-26 16:02:41 -07:00
Dane Everitt
3391d5803e
Use same values between panel and wings when calculating disk
2020-09-25 20:31:33 -07:00
Dane Everitt
fb24ad58b4
Don't allow a reinstall while server is processing a power action; closes pterodactyl/panel#2409
2020-09-25 20:03:04 -07:00
Dane Everitt
ae5005baa3
Don't log disk space errors
2020-09-25 20:02:38 -07:00
Dane Everitt
7624502ec4
Don't swap the int and cause a negative disk space... closes pterodactyl/panel#2401
2020-09-24 21:19:46 -07:00
Dane Everitt
bf1233def4
Don't ignore disk space limits when copying/archiving; closes pterodactyl/panel#2400
2020-09-24 21:18:10 -07:00
Dane Everitt
522c6c17e4
Merge branch 'develop' of https://github.com/pterodactyl/wings into develop
2020-09-22 20:41:16 -07:00
Dane Everitt
d8cd8ae36a
Don't treat certain "expected" errors from the filesystem as 500 errors; closes pterodactyl/panel#2376
2020-09-22 20:41:14 -07:00
Matthew Penner
ff597672db
Fix #2380
2020-09-20 12:51:12 -06:00
Dane Everitt
6ba49df485
Protect against zip bombs; closes pterodactyl/panel#883
2020-09-17 20:37:34 -07:00
Dane Everitt
783832fc71
Set the docker image correctly when a server is updated; closes pterodactyl/panel#2356
2020-09-17 20:20:39 -07:00
Dane Everitt
6ba1b75696
Add console throttling; closes pterodactyl/panel#2214 ( #60 )
2020-09-17 20:13:04 -07:00
Dane Everitt
ce76b9339e
better error handling for busy files; closes pterodactyl/panel#2332
2020-09-15 19:53:00 -07:00
