Happy Friday once again QuickBoxers! We’re back with another Developer Update.
As we draw nearer to the finish line for finishing up on v3 development, I am going to consolidate some of the past Developer Updates into this update so as to have a cleaner reference on all the things that have been accomplished thus far. Along with consolidating some of the past updates, I will additionally share some of the newer developments within the v3 code-sphere.
Let’s discuss the usual recent developments, then dive into the journey that got us here!
conversions
Conversions are nearly completed! That’s right, some 60 software build packages have been completely refactored to comply with the new QuickBox Pro v3 format. Out of these, we are down to the last 5 software to convert (Tautulli, TheLounge, Unpackerr, xTeVe, and ZNC). These remaining conversions will be done in short order and more than likely completed by late Monday.
There is an incredible amount of quality control going into the current developments to ensure that everything is fine-tuned and stable as possible. Though there is a lot more that has been done, we’re reserving these discussions for once we conclude their additions and testing. In short; even more optimization by the @mschf are being done under the hood to make things as quick as possible… and it’s incredibly exciting!!!
So Where are we at so far?
Libraries
One of the biggest factors in the optimization to the QuickBox Pro framework is its use of the new libraries. This has led to a file size reduction of 77.7% (44.5kb to 9.9kb)! Dynamic configurations have also changed. Instead of outputting these from the installers, there are now templates that are being copied to the relevant locations and updated with the server-specific configurations. This means that when a piece of software is updated, the installer won’t need to run through its entire script, but just copy the relevant files and update them with the required configurations.
If you’d like to take a closer look at some of these libraries, a new project has been born from the process. Check out mflibs (created by our very own @mschf) here:
https://github.com/mschf-dev/mflibs
Backup and Restore
Server Admins will be able to run qb generate config (This might be renamed). The command will generate a zip file that can be moved to a new server. The installer will automatically detect this file and ask if you want to import your previous configs, recreate users and install the same software.
This will eventually include media and per-user software settings, however, these will be included in a later release (Have to be careful of feature creep!)
CLI
We’ll be saying goodbye to fixed positionals and hello to flags. Currently, options have to be passed in a set order:
qb install software [VERSION] USERNAME [OPTION] [OPTION]
As we are looking to significantly expand the CLI usage in v3, there was a need to revamp this. The new CLI usage will be:
qb install software -u username [-v VERSION -o OPTION1 -o OPTION2 -V]
Current optional flags are:
-V - Verbose Output -o - Options (Multiple allowed)
Whilst the CLI slightly longer, it is also quicker, as we aren’t relying on hundreds of ‘if then’ statements to parse the command, and it allows us to expand the installers for more functionality).
With the current version of QuickBox, if you want to use the CLI, you have to remember a fair bit, and reference some KB articles. With v3, you’ll have the option of entering commands via the CLI like before, or simply just typing in qb and interacting with a new menu!
This will also give you the option to install/remove/update software for multiple users at the same time!
Here’s a sneak peek:
Let’s Encrypt
One of the main problems with the current lecert installer is the lack of an option to just generate a certificate without updating all the webroot paths. This has now been addressed and v3 will have the option to just generate a certificate for use with other software or websites (where -o -no-dash is the optional flag to produce a certificate for various domains):
qb install lecert -d DOMAIN_HERE -o no-dash
API
QuickBox v3 will feature an optional API. This is primarily for the developers among you. While I’ve only completed the framework for this, it’ll allow you to run various QuickBox commands and retrieve information. The daemon that runs for the API has read and write tokens so that QuickBox functionality can be safely built into applications and web pages. Here’s a quick screenshot:
SQLite3
We had been running into some trouble with the database and how it handles the sheer number of requests that the dashboard is making. While we were originally concerned about this, we’ve implemented queuing on the requests and the database is currently handling the dashboard perfectly (It’s worth noting that the queues are for microseconds and there is no noticeable impact at all).
We’re now rocking a smaller amount of tables with a bit more data in them. This has allowed us to remove duplicate data from all over the place providing a cleaner environment in the database, as well as decreasing load times. (Seriously, we had data everywhere)!
Script handling
82 files to 8 for handling the system commands seems a bit crazy – however, we’ve completely redesigned how the qb command runs scripts. Let’s quickly compare how v2 and v3 handle the qb install software command:
QuickBox Pro v2
qbchecks how it is invoked (via CLI or Web).qbparses the positional parameters based on its invocation (Multiple if statements based on positional ordering).qbpasses the first positional (install) through a case statement.- The case statement then runs a specified function based on what it is matched against – (install_function) in this example.
install_functionthen re-checks the positional parameters and re-orders them based on the requested software.install_functionloads a new script (qbAppHandler).qbAppHandlerre-orders the positionals based on software again.qbAppHanlderinvokes the requested software script.- The software script parses the positionals into variables.
- The software script begins!
As you can see there’s a lot of steps and files for a relatively straight forward command. This is mainly due to the requirement of positionals being in a specific order. While the processing time of the above is relatively small…it’s not exactly what we would call optimized.
QuickBox Pro v3
qbparses input via a mixture of positional parameters and flags.qbpasses the first positional (install) through a case statement.- The case statement then loads the requested software script – passing existing positional and flags via an array.
- The software script parses the array into the required variables.
- The software script begins!
The use of arrays allows the removing additional handlers and negates the need to order arguments in fixed locations.
php -v
We’re making some changes to the code behind the QuickBox Dashboard. One of those is updating the dashboard to utilize php8.1. We’ve done load and speed testing and are happy to report a reduction in the amount of server resources used. The dashboard is also a lot more responsive – Our current tests have the dashboard as around 3x faster now…vroom vroom!
self-cleaning locks
One of the touchy bits with v2 is the lock checks. If an install/update/removal fails, then users are left having to run some commands to install again. With v3 if a lock file exists, it’ll check to see if any processes are running and clean up after itself when finished.
performance differences
Now that we have the v3 libraries finished and several software conversions completed, it’s time for a quick performance comparison! (As I left this last minute, v3 is only on a VM and V2 on a dedicated server, so while it’s not a real comparison, it shows how much quicker v3 is!)
Specs:
QuickBox Pro v2
CPU Model: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz CPU Cores: 8 Total Mem: 32081 MB OS: Debian GNU/Linux 10 Virtualization: Dedicated
QuickBox Pro v3
CPU Model: Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz CPU Cores: 4 Total Mem: 3946 MB OS: Debian GNU/Linux 10 Virtualization: KVM
Deluge install:
| v2 | v3 | |
|---|---|---|
| real | 1m32.719s | 0m48.717s |
| user | 0m33.192s | 0m17.681s |
| sys | 0m5.106s | 0m5.038s |
Deluge removal:
| v2 | v3 | |
|---|---|---|
| real | 0m32.463s | 0m8.198s |
| user | 0m5.157s | 0m3.611s |
| sys | 0m1.559s | 0m1.083s |
v3 delivery (that’s installs and upgrades)
v3 has been incredibly tuned so far in it’s journey through development and at this point we’re struggling to find more to improve (seriously, just check out week 11’s Developer Update on the performance differences!). It’s incredibly responsive and stable (with it being as completed as it actually is at the moment). With this in mind our next focus is in regards to enhanced stability to v3, to keep it that way, and to have it be a more enjoyable experience to anyone that uses it. This same can be said from a development standpoint.
QuickBox is a massive piece of software and the maintenance and support requires a lot of time and patience. Supporting various distro has become cumbersome and actually adds more unneeded weight to the code-base that we can shed.
We have made a decision to sunset support for the Ubuntu line of Distro.
Here’s a few reasons why:
- Ubuntu is becoming increasingly bloated with unnecessary elements and background tasks that are simply not needed on a media server, let alone headless servers.
- Ubuntu is a fork of Debian. They are essentially the same, but again, ubuntu stuffs in extra bells and whistles.
- Here’s food for thought.
* An Ubuntu ISO for a minimal server (headless) install is 2.6GB.
* Debian is 600MB.
The functionality at it’s core is exactly the same! whhhaaaaaat???
- Here’s food for thought.
- Debian being lighter weight and producing more stable lineage is the preferred OS. Your server will actually thank you for running Debian.
- As the dependencies can vary from Ubuntu to Debian, this increases the additional work load when troubleshooting, adding in new features, software.. etc.. Debian tends to have the more stable toolset.
- To expand on this. The additional testing and work that goes in to what may sound simple is much larger when it comes to a utility such as QuickBox. This takes time away from streamlining the Feature Requests for new additions. If we supported one OS and it’s spawn of Distro (Debian 9, 10, and 11 once it’s fully supported) we will not have to juggle various python, dependencies, etc etc. With being able to put more time into development and the focus shifted to only supporting Debian, this means we can have more time to drastically expand on our overall visions of QuickBox.
While we do understand that some people are emotionally attached to Ubuntu (the philosophy behind it after all is the Human in me recognizes the Human in you), Debian is the parent of the two and provides greater stability and the usability; it’s exactly the same!
optimizations
There are honestly so many it’s hard to list them all. The pip update time has now been reduced by 40%. The time to invoke qb (As it completed numerous checks) has also been cut by a similar amount.
Pulling data from the database is now around 50% quicker (and this was already quite tuned from previous weeks work. I plan to stop fiddling with this now as we’re entering into diminishing returns).
features
As we work on revisions, we are not only focused on rewriting the packages, but the additional inclusion of features to certain software builds. One such feature adjustment is the install of Filebot. In v2, this process is rather cumbersome and involves manual work to get it staged (ie; install, then copy your license file to the server, then you have to activate with the filebot command). What a chore! With v3 however, you’ll be able to simply copy/paste your license file directly into the CLI prompt.
Here’s a quick example (tested with an expired license to confirm it calls home and works)
qb install filebot -u echo FileBot will now be installed. This process may take a few minutes... Adding additional keys and sources for FileBot... Checking and performing apt updates... Installing FileBot dependencies... Building FileBot... Configuring FileBot Would you like to use FileBot to process rTorrent downloads? [y/n]: > y Would you like to use a custom directory for processed files? [y/n] (Default: /home/echo/Media): > n Would you like to use FileBot to process Deluge downloads? [y/n]: > n Would you like to activate your license? [y/n]: > y Please copy/paste the full contents of your license file. Press ctrl+D when done to continue. > -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Product: FileBot Name: [redacted] Email: [redacted] Order: P19854281 Issue-Date: 2020-11-13 Valid-Until: 2021-11-20 -----END PGP SIGNATURE----- Attempting to activate filebot_license.psm ... Initialize UserHome folder: /home/echo/.filebot/filebot-portable/data/0 Initialize Cache folder: /home/echo/.filebot/filebot-portable/data/0/cache Initialize Logs folder: /home/echo/.filebot/filebot-portable/data/0/logs Initialize TemporaryFiles folder: /home/echo/.filebot/filebot-portable/data/0/tmp Initialize new disk cache: /home/echo/.filebot/filebot-portable/data/0/cache/0 Unknown Error: P19854281 has EXPIRED on 2021-11-20 Updating database... Backing up FileBot configuration... FileBot installation complete.
Much better!
Another feature coming to v3, will be the ability to install Emby and Jellyfin as a multi-user software. Meaning that the System Admin can set Emby-Server to the users group, and any user in said group will be able to install Emby. We’ve additionally included options to install stable versions or beta versions, at any time a stable version can be updated to a beta release. Here’s a quick preview of that command.
Install Emby stable release:qb install emby -u USERNAME
Install Emby beta release:qb install emby -u USERNAME -o beta
You can update a stable release to a beta release, should you be so bold:qb update emby -u USERNAME -o beta
Just like that!
Here’s a quick preview to proof of setup: (all developments are done on a localized/closed server environment)
Extra Extra! qb news is in the box!
@mschf has been down in the labs (when he’s not quality controlling my drunken conversions 🤪) and he’s cooked up a new module to be used in CLI. What is this new module? qb news! The module allows important announcements, as presented by us here at QuickBox.IO, to be posted directly to your server. This means, if there’s ever a critical patch, a fix for anything, or a new exciting release, we can post this announcement directly to your server!
If you’re wondering if this is posting content directly to your server, have no fear. Any content that is announced is processed via our own “announce” server which means it’s secure and non-invasive. It’s just a fun announcement ensuring that you never miss an important update… calm down!
Take a look at this nifty gif that @mschf actually had time to make for us.
qb newspython versioning and stability
Within the QuickBox Pro environment, there are several software that require python. In v2 of QuickBox Pro, we set alternatives to python in various locations, but still utilize globals when installing python3 pip modules. This means that a software, such as Bazarr, would use the same global python3 directory as, let’s say, Deluge. At first, this type of setup seems easy and harmless, however, it can get complicated! If another user installs Bazarr, when they initiate an install it will check and update these pip modules according to the software requirements. Therefore, if user A has yet to update their Bazarr and user B installs a fresh version that updates pip modules (possibly not yet functional with the previous version), this can cause collisions and other errors.
Thankfully, this doesn’t happen often, but let’s not ignore this and wait for it to happen. Nay! Let us be proactive and create greater stability… now!
We’re now going further down the modularity hole and are introducing unique python virtual environments by way of python3-venv. What this means now is that user A will install Bazarr and a python virtual environment will be staged under their /opt/USER/Bazarr/.venv/ directory. All required python/pip related modules will be isolated to this directory. Only for the user and that particular software! So user A and user B will now utilize 2 completely separate instances of python for their Bazarr installs (and other python required software).
error codes
@mschf has been down in the labs working on getting a slurry of error code functionality built-in. We’ll provide a list of the error codes and their assigned definitions within the next couple of updates, or once we’re at the “ready to test” phase. These will assist users in troubleshooting any specific issues coming from any of the QuickBox system functions. Again, we’ll have a lot more on this in the next couple of updates.
Have you customized your qb install?
Let us know your customizations and we’ll work on implementing the most popular of the bunches we receive. If there isn’t one ranking higher than others, we’ll pick! You can let us know on Discord, via #v3-discussion. If we don’t respond, it’s because we’re taking notes 😉
As we are still chugging along on developments and having ironed out additional logistics, this concludes this weeks developer update. We’re getting closer to the finish line and we couldn’t be more excited! v3 will no doubt, even through the decisions we have made, be an incredibly enjoyable experience offering greater ease of use and stability.
In the meantime, catch you all next week! 💖
