From 636ab32eadeea71c588d8ef5de265fa7bf306a0a Mon Sep 17 00:00:00 2001 From: Gabriel Simmer Date: Sat, 29 May 2021 22:32:38 +0100 Subject: [PATCH 1/5] Tidy README, remove UPX requirement. --- .circleci/config.yml | 4 +- Makefile | 2 - README.md | 105 +++++++++++++++------------ assets/config_examples/providers.yml | 13 ++++ 4 files changed, 74 insertions(+), 50 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 8621a8f..23d11a3 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -1,6 +1,5 @@ version: 2.1 -orbs: - upx: circleci/upx@1.0.1 + jobs: build: docker: @@ -12,7 +11,6 @@ jobs: - go-mod-{{ checksum "go.sum" }}-v3 - go-mod-{{ checksum "go.sum" }} - go-mod - - upx/install - run: command: make dist - store_artifacts: diff --git a/Makefile b/Makefile index 4e24548..6071f6c 100644 --- a/Makefile +++ b/Makefile @@ -14,11 +14,9 @@ pi: make_build_dir small: make_build_dir go build -o build/bin/sliproad -ldflags="-s -w" - upx --brute build/bin/sliproad -9 --no-progress small_pi: make_build_dir env GOOS=linux GOARCH=arm GOARM=5 go build -o build/bin/sliproad-arm -ldflags="-s -w" - upx --brute build/bin/sliproad-arm -9 --no-progress run: go run webserver.go diff --git a/README.md b/README.md index b306d82..cf085d7 100644 --- a/README.md +++ b/README.md @@ -1,21 +1,24 @@ -# sliproad -merging filesystems together +# Sliproad -## about +Merging filesystems together -this project aims to be an easy-to-user web API that allows the management of cloud storage, whether it be on -the host machine or part of a remote api. this is intended mostly for my own use, but i am documenting it in a way that -i hope allows others to pick it up and improve on it down the line. +## About -if something is unclear, feel free to open an issue :) +This project aims to be an easy-to-use web API and frontend that allows the +management of cloud storage, whether it be on the host machine or part of a +remote API, alongside local filesystems. While this is intended mostly for my +own use, I am documenting it in a way that I hope allows others to use it! -## configuration +## Configuration -unlike the initial version of this project, the current build uses _providers_ to determine how to handle various -functions related to files. currently, two are implemented, `disk` and `backblaze`, since they are the primary providers -i use myself. the providers you would like to use can be added to `providers.yml` alongside the binary. +Sliproad uses "Providers" to support various filesystems "types", whether it be +remote or local. Currently, three exist - `disk` for filesystems local to the +machine, `backblaze` to leverage Backblaze B2 file storage and `s3` for AWS S3 +(and other compatible providers). -for example, here is a sample configuration implementing both of them: +An example of leveraging all three, in various forms, can be found below. As +more are added, this example will be updated, and more examples can be found in +the `assets/config_examples` directory. ```yaml disk: @@ -24,53 +27,65 @@ disk: backblaze: provider: backblaze config: - appKeyId: APP_KEY_ID - appId: APP_ID - bucket: BUCKET_ID + bucket: some-bucket + applicationKeyId: application-key-id + applicationKey: application-key +s3: + provider: s3 + config: + region: eu-west-2 + bucket: some-bucket +# An example of an S3 compatible API, doesn't have to be Backblaze. +backblazes3: + provider: s3 + config: + bucket: some-bucket + region: us-west-000 + endpoint: s3.us-west-000.backblazeb2.com + keyid: key-id + keysecret: key-secret ``` -(read more here: [#providers](#providers)) +## Running -## running +After configuring the providers you would like to utilize, simply run +`./sliproad`. This will spin up the webserver at `127.0.0.1:3000`, listening on +all addresses. -after adding the providers you would like to use, the application can be run simply with `./nas`. it will attach to port -`:3000`. +## Building -## building - -this project uses go modules and a makefile, so building should be relatively straightforward. +This project leverages a Makefile to macro common commands for running, testing +and building this project. - `make` will build the project for your system's architecture. - `make run` will run the project with `go run` - - `make pi` will build the project with the `GOOS=linux GOARCH=arm GOARM=5 go` flags set for raspberry pis. - -## providers + - `make pi` will build the project with the `GOOS=linux GOARCH=arm GOARM=5 go` flags set for Raspberry Pi. + - `make dist` will build and package the binaries for distribution. -"providers" provide a handful of functions to interact nicely with a filesystem, whether it be on local disk or on a -remote server via an api. best-effort is done to keep these performant, up to date and minimal. +### Adding Providers -there are a few built-in providers, and more can be added by opening a pull request. +New file providers can be implemented by building off the +`FileProviderInterface` struct, as the existing providers demonstrate. You can +then instruct the [`TranslateProvider()`](https://github.com/gmemstr/sliproad/blob/master/files/fileutils.go#L8-L21) +that it exists and how to configure it. -|name|service|configuration example| -|----|-------|---------------------| -|disk|local filesystem|assets/config_examples/disk.yml| -|backblaze|backblaze b2|assets/config_examples/backblaze.yml| +## Authentication [!] -you can find a full configuration file under `assets/config_examples/providers.yml` +Authentication is a bit tricky and due to be reworked in the next iteration of +this project. Currently, support for [Keycloak](https://www.keycloak.org/) is +implemented, if a bit naively. You can turn this authentication requirement on +by adding `auth.yml` alongside your `providers.yml` file with the following: -### custom provider +```yaml +provider_url: "https://url-of-keycloak" +realm: "keycloak-realm" +redirect_base_url: "https://location-of-sliproad" +``` -custom file providers can be implemented by adding a new go file to the `files` module. it should -implement the `FileProviderInterface` interface. +Keycloak support is not currently actively supported, and is due to be removed +in the next major release of Sliproad. That said, if you encounter any major +bugs utilizing it before this, _please_ open an issue so I can dig in further. -## authentication - -basic authentication support utilizing [keycloak](https://keycloak.org/) has been implemented, but work -is being done to bring this more inline with the storage provider implementation. see `assets/config_examples/auth.yml` -for an example configuration - having this file alongside the binary will activate authentication on all -`/api/files` endpoints. note that this implementation is a work in progress, and a seperate branch will -contain further improvements. - -## icons +## Credits SVG Icons provided by Paweł Kuna: https://github.com/tabler/tabler-icons (see assets/web/icons/README.md) \ No newline at end of file diff --git a/assets/config_examples/providers.yml b/assets/config_examples/providers.yml index 5c19b86..780b7c3 100644 --- a/assets/config_examples/providers.yml +++ b/assets/config_examples/providers.yml @@ -18,3 +18,16 @@ backblaze: applicationKeyId: aaaaaaaaaaaa applicationKey: aaaaaaaaaaaa bucket: aaaaaaaaaaaa +s3: + provider: s3 + config: + region: eu-west-2 + bucket: some-bucket +backblazes3: + provider: s3 + config: + bucket: sliproad-testing + region: us-west-000 + endpoint: s3.us-west-000.backblazeb2.com + keyid: key-id + keysecret: key-secret \ No newline at end of file From cbd30f91554ed681ac67f1bdd324de615f1955bf Mon Sep 17 00:00:00 2001 From: Gabriel Simmer Date: Sat, 29 May 2021 22:42:24 +0100 Subject: [PATCH 2/5] Clean up CCI config. --- .circleci/config.yml | 5 +++-- README.md | 6 ++++++ assets/config_examples/README.md | 0 3 files changed, 9 insertions(+), 2 deletions(-) delete mode 100644 assets/config_examples/README.md diff --git a/.circleci/config.yml b/.circleci/config.yml index 23d11a3..5904a84 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -32,8 +32,9 @@ jobs: - run: command: make test workflows: - version: 2 build-and-test: jobs: - - build - test + - build: + requires: + - test diff --git a/README.md b/README.md index cf085d7..9fe3fe0 100644 --- a/README.md +++ b/README.md @@ -52,6 +52,12 @@ After configuring the providers you would like to utilize, simply run `./sliproad`. This will spin up the webserver at `127.0.0.1:3000`, listening on all addresses. +## API + +This project is largely API-first, and documentation can be found here: + +https://github.com/gmemstr/sliproad/wiki/API + ## Building This project leverages a Makefile to macro common commands for running, testing diff --git a/assets/config_examples/README.md b/assets/config_examples/README.md deleted file mode 100644 index e69de29..0000000 From 60b69213ae3e22fabde94f9d346e53c9c3b25e58 Mon Sep 17 00:00:00 2001 From: Gabriel Simmer Date: Sat, 29 May 2021 22:44:31 +0100 Subject: [PATCH 3/5] Redundant redundancy. --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 9fe3fe0..affd4b2 100644 --- a/README.md +++ b/README.md @@ -5,9 +5,9 @@ Merging filesystems together ## About This project aims to be an easy-to-use web API and frontend that allows the -management of cloud storage, whether it be on the host machine or part of a -remote API, alongside local filesystems. While this is intended mostly for my -own use, I am documenting it in a way that I hope allows others to use it! +management of cloud storage alongside local filesystems. While this is intended +mostly for my own use, I am documenting it in a way that I hope allows others +to use it! ## Configuration From c8a898d861b7fcd72a55e51e48eded8936117c72 Mon Sep 17 00:00:00 2001 From: Gabriel Simmer Date: Sat, 29 May 2021 22:49:17 +0100 Subject: [PATCH 4/5] Add frontend blurb. --- README.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index affd4b2..01fefac 100644 --- a/README.md +++ b/README.md @@ -52,6 +52,17 @@ After configuring the providers you would like to utilize, simply run `./sliproad`. This will spin up the webserver at `127.0.0.1:3000`, listening on all addresses. +## Frontend + +The frontend is a very lightweight JavaScript application and aims to be very +functional, if a bit rough around the edges. + +![Screenshot_2021-05-29 Sliproad](https://user-images.githubusercontent.com/1878840/120085420-d63cbc80-c0cf-11eb-9fbb-b0b05a3f5d58.png) + +It should scale reasonably well for smaller devices. Because it's now bundled +into the binary (as opposed to distributed alongside), it's no longer possible +to swap it out for a custom frontend without serving the frontend seperately. + ## API This project is largely API-first, and documentation can be found here: @@ -94,4 +105,4 @@ bugs utilizing it before this, _please_ open an issue so I can dig in further. ## Credits -SVG Icons provided by Paweł Kuna: https://github.com/tabler/tabler-icons (see assets/web/icons/README.md) \ No newline at end of file +SVG Icons provided by Paweł Kuna: https://github.com/tabler/tabler-icons (see assets/web/icons/README.md) From e888ca1759b90009a23225c5fc01222ac987a980 Mon Sep 17 00:00:00 2001 From: Gabriel Simmer Date: Sat, 29 May 2021 22:51:23 +0100 Subject: [PATCH 5/5] Fix typo. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 01fefac..56ba1b1 100644 --- a/README.md +++ b/README.md @@ -83,7 +83,7 @@ and building this project. New file providers can be implemented by building off the `FileProviderInterface` struct, as the existing providers demonstrate. You can -then instruct the [`TranslateProvider()`](https://github.com/gmemstr/sliproad/blob/master/files/fileutils.go#L8-L21) +then instruct the [`TranslateProvider()`](https://github.com/gmemstr/sliproad/blob/master/files/fileutils.go#L8-L21) function that it exists and how to configure it. ## Authentication [!]