Difference between revisions of "K2"

Latest revision as of 19:28, 2 July 2023

This page contains the documentation of k2system, a management and administration system for user accounts. It is used on the same server as the one serving this very web page.

The system does not impose quotas or hard limitations, and instead depends on the good will and technical competency of all users. Please take care to not abuse it - careless abuse of resources will negatively affect all other users and services. Thanks!

This system is relatively unusual, and many components are custom. For this reason, blindly following guides from the Internet will not work. Do not copy and paste commands or configuration files without fully understanding them and how they will interact! Instead, if something is unclear, contact an administrator.

Logging in

Remote access, administration, and file management is done over SSH. This means that you can use any SSH client to log in to an interactive shell, or use any standard SSH-based file transfer programs (such as scp, sftp, rsync). On Windows, you can use PuTTY for SSH access, and WinSCP for file management.

Although you may change either password, please use only very strong passwords. Failure to do so will make your account susceptible to brute force attacks (which are happening constantly), which may lead to the entire server being compromised.

If you don't want to type a password every time you log in, you can configure a SSH key. Please do not use any software which "remembers" passwords, as malware frequently targets configuration files of such software, which will lead to your account being compromised.

A phpMyAdmin instance is available; see #MySQL.

Overview

Hosting and resource partitioning is done as follows:

Generally, each website or service runs as its own UNIX user, and has its personal home directory at /home/username.
Each user runs their own local web server, listening on port 81 and their personal private local IP address (127.42.x.y).

(authbind is used to allow listening on port 81 and prevent one user from hijacking another user's HTTP port).

All incoming web requests are first accepted by reverse proxies, for both HTTP and HTTPS, and then distributed to the appropriate user according to their configured domain list.
A shared MySQL instance allows database storage. Each user has their personal MySQL database.
Symlinks in the home directory point to users' personal directories on the data, scratch, and fast-scratch storage volumes.

Directory layout

New accounts start with a home directory with the following layout:

~/.k2 - Your k2system configuration (details below).
~/www - Website root. You can place your .html and .php files here.
~/k2-data - Symlink to your personal data directory on the server's hard drive.
~/k2-scratch - Symlink to your personal scratch directory on the server's hard drive (not backed up).
~/k2-fastscratch - Symlink to your personal scratch directory on the server's SSD (not backed up).

How to place files

Please follow these rules when uploading or creating files on the server.

If a file is large or accessed very infrequently, and you can easily recreate or re-upload it should it be lost, place it under ~/k2-scratch.
Examples:
- Files you downloaded from the Internet.
- Large temporary files.
- Your personal backups. (The other copy is on your computer, right?)
If a file is large or accessed very infrequently, and it cannot be easily recreated or re-uploaded, place it under ~/k2-data.
Examples:
- Photos and videos that your users uploaded to your website.
If the file is small or accessed frequently, and you can easily recreate or re-upload it should it be lost, place it under ~/k2-fastscratch.
Examples:
- Temporary files, such as PHP session files.
- Logs (if you don't care about them).
Other files should be placed in your home directory.
Examples:
- The source code for your website.

Please make sure to not place large files in your home directory outside the above-described special directories. Doing so wastes precious SSD space and increases the size of our backup archives, which affects all users.

Configuration

All configuration is done by editing the contents of the ~/.k2 directory, and then running a command (usually k2-update or k2-service restart -a) to apply the changes.

Web Server

You can use any web server software you like. The web server runs in your user account.

The default configuration is to use Apache, which allows familiar configuration using .htaccess and .htpasswd files.

Note that k2system will stop Apache if it doesn't receive a request in over an hour, which is why you may not always see it in the process list. It is automatically started back up when a request arrives.

Apache

The configuration for your personal Apache instance is located in ~/.k2/apache2.

By default, the configuration consists of a number of symlinks to the default k2system configuration files. If you would like to edit a configuration file, you will first need to replace the symlink with a copy of the file that the symlink pointed to. For your convenience, a delink command exists which does this. For example:

delink .k2/apache2/conf.d/site.conf
mcedit .k2/apache2/conf.d/site.conf

By default, only a small number of modules are enabled (see ~/.k2/apache2/mods.d). To enable a new module, you can create a symlink pointing to the corresponding file in the k2system default configuration. Example:

ln -s /usr/local/etc/k2/apache2/mods.d/cgi.load .k2/apache2/mods.d/cgi.load

To disable a module, simply delete the symlink.

Finally, to apply changes to the configuration, run k2-service restart -a.

Cherokee

Not documented.

lighttpd

Not documented.

PHP

PHP is configured in a similar manner as Apache:

php.ini is located in ~/.k2/php/php.ini. By default, it is a symlink to /usr/local/etc/k2/php/php.ini.
PHP modules can be enabled by creating symlinks in ~/.k2/php/conf.d.

MySQL

Your MySQL database password is different from your login password, and was sent to you by email during registration.

To access the database from your website, you can use the following credentials:

Hostname: localhost or blank
Username: your SSH login username
Password: your MySQL password
Database: your SSH login username

For an interactive SQL prompt, simply run mysql from a SSH command line. The MySQL client is preconfigured with your credentials (see ~/.my.cnf).

To dump the entire database, you can run e.g.: mysqldump your-username > database.sql

A phpMyAdmin instance is available at https://pma-k3.1azy.net/. You can log in with your username and MySQL password.

Logs

Apache logs (access and error) are, by default, placed in ~/.k2/apache2/logs.

Log rotation is configured in the ~/.k2/logrotate.template directory. By default, Apache logs are rotated weekly.

Domains

By default, your website is accessible via the server's wildcard DNS entry (*.k3.1azy.net). If your UNIX (SSH login) username is alice, the website can be accessed at https://alice.k3.1azy.net/.

If you would like to add a nicer hostname to your website, do the following:

Point the hostname to the server's IP address
Create a file in ~/.k2/domains named after the hostname (e.g. if you would like to use "you.example.com", create ~/.k2/domains/you.example.com)
Run k2-update to tell k2system to update the server configuration.

DNS

You can also host your DNS zone on the server. To do so, create a zone file (in standard BIND syntax, with a .hosts extension) in ~/.k2/dns, and run k2-update.

For your convenience, k2system also understands a file format with a .hosts-template extension, which allows substituting variables which apply to all domain names. As such, the easiest way to add a zone is to create ~/.k2/dns/hostname.hosts-template with the following contents:

$TTL 38400   ; default expiration time of all resource records without their own TTL value
%DOMAIN%.	IN	SOA	%SERVERHOST%. %USER%.%SERVERHOST%. (
			%MODTIME%	; serial number of this zone file
			21600		; slave refresh (in seconds)
			3600		; slave retry time
			604800		; slave expiration time
			86400		; maximum caching time in case of failed lookups
			)
		IN	NS	%SERVERHOST%.
		IN	NS	%DNS2%.
		IN	MX	10 mail.%DOMAIN%.

		IN	A	%SERVERIP4%
 		IN	AAAA	%SERVERIP6%

mail		IN	A	%SERVERIP4%
mail		IN	AAAA	%SERVERIP6%
k3._domainkey	IN	CNAME	k3._domainkey.%SERVERHOST%.

www		IN	CNAME	@

Email

The server does not store email, so there are no mailboxes to check (or access via POP3/IMAP or otherwise). However, redirects can be configured.

Assuming that your DNS MX records are set up to point to the server's IP addresses, email will be accepted for the domains you configured (in ~/.k2/domains). To configure what happens to mail sent to localpart@your-domain.com, create the file ~/.k2/mail/aliases/localpart. It can contain an email address to redirect to, or /dev/null to discard all email. See the exim documentation for more details. If a file corresponding to the local part does not exist, the file named default is consulted instead.

Internally generated mail is sent to the email address indicated in the ~/.email file.

Note that the files in the aliases directory must have mode 644. (Depending on how you log in, they may be instead created with mode 664.) Use chmod to change it.

Make sure to run k2-update to apply changes in ~/.k2/mail.

Please do not configure redirects for email addresses which receive a lot of spam. Doing so negatively affects the reputation of the server's IP address, which eventually leads all sent and redirected mail to be marked as spam. If you receive a large amount of mail, please use a professional (paid) email service.

SSL

This section describes how to configure SSL for HTTP (i.e., HTTPS). By default, attempting to access your website via HTTPS will use the server's wildcard certificate, which will cause most user agents to display a warning or error.

Bring Your Own Certificate

If you already have an SSL certificate for your domain and would like to use it, place the files in the ~/.k2/ssl directory as follows:

~/.k2/ssl/site.pem - your certificate
~/.k2/ssl/chain.pem - your SSL provider's certificate
~/.k2/ssl/site.key - your certificate's private key

Let's Encrypt

Unless you already have an SSL certificate, the easiest way to enable SSL is using the free Let's Encrypt certificate provider. Simply run k2-letsencrypt to enable SSL for your configured domains.

Note: k2-letsencrypt does not work with the default *.k3.1azy.net wildcard hostnames. You will need to add a different hostname to enable SSL for your website.

Redirect to HTTPS

You can redirect clients which opt-in to SSL to HTTPS by placing the following in ~/www/.htaccess:

# Redirect to HTTPS clients indicating their preference for it
# Note: this should also send a Vary header, but doesn't due to Apache bug
RewriteCond %{HTTP:Upgrade-Insecure-Requests} ^1$
RewriteCond %{HTTP:X-Scheme} !https
RewriteRule ^(.*)$ https://%{HTTP_HOST}/$1 [R,L]

@@ Line 1: / Line 1: @@
-This page contains the documentation of k2system, a management and administration system for user accounts. It is used on the same server as the one serving this very web page.
+This page contains the documentation of ''k2system'', a management and administration system for user accounts. It is used on the same server as the one serving this very web page.
+{{Warning|The system does not impose quotas or hard limitations, and instead depends on the good will and technical competency of all users. Please take care to not abuse it - careless abuse of resources '''will''' negatively affect all other users and services. Thanks!}}
+{{Warning|This system is relatively unusual, and many components are custom. For this reason, blindly following guides from the Internet will not work. '''Do not copy and paste commands or configuration files''' without fully understanding them and how they will interact! Instead, if something is unclear, contact an administrator.}}
 == Logging in ==
@@ Line 9: / Line 13: @@
 If you don't want to type a password every time you log in, you can [https://www.google.com/search?q=ssh+key+login configure a SSH key]. Please do not use any software which "remembers" passwords, as malware frequently targets configuration files of such software, which will lead to your account being compromised.
-A phpMyAdmin instance is available at https://pma.k3.1azy.net/. You can log in with your username and **MySQL password** (which is different from your account password).
+A phpMyAdmin instance is available; see [[#MySQL]].
+== Overview ==
+Hosting and resource partitioning is done as follows:
+* Generally, each website or service runs as its own UNIX user, and has its personal home directory at <code>/home/''username''</code>.
+* Each user runs their own local web server, listening on port 81 and their personal private local IP address (<code>127.42.''x''.''y''</code>).
+: (<code>authbind</code> is used to allow listening on port 81 and prevent one user from hijacking another user's HTTP port).
+* All incoming web requests are first accepted by ''reverse proxies'', for both HTTP and HTTPS, and then distributed to the appropriate user according to their configured domain list.
+* A shared MySQL instance allows database storage. Each user has their personal MySQL database.
+* Symlinks in the home directory point to users' personal directories on the data, scratch, and fast-scratch storage volumes.
 == Directory layout ==
@@ Line 15: / Line 30: @@
 New accounts start with a home directory with the following layout:
-* ~/.k2 - Your k2system configuration (details below).
+* <code>~/.k2</code> - Your ''k2system'' configuration (details below).
-* ~/www - Website root. You can place your .html and .php files here.
+* <code>~/www</code> - Website root. You can place your <code>.html</code> and <code>.php</code> files here.
-* ~/k2-data - Symlink to your personal data directory on the server's hard drive.
+* <code>~/k2-data</code> - Symlink to your personal data directory on the server's hard drive.
-* ~/k2-scratch - Symlink to your personal scratch directory on the server's hard drive (not backed up).
+* <code>~/k2-scratch</code> - Symlink to your personal scratch directory on the server's hard drive (not backed up).
-* ~/k2-fastscratch - Symlink to your personal scratch directory on the server's SSD (not backed up).
+* <code>~/k2-fastscratch</code> - Symlink to your personal scratch directory on the server's SSD (not backed up).
 === How to place files ===
@@ Line 25: / Line 40: @@
 Please follow these rules when uploading or creating files on the server.
-# If a file is large or accessed very infrequently, and you can easily recreate or re-upload it should it be lost, place it under ~/k2-scratch.<br>Examples:
+# If a file is large or accessed very infrequently, and you can easily recreate or re-upload it should it be lost, place it under <code>~/k2-scratch</code>.<br>Examples:
 #* Files you downloaded from the Internet.
 #* Large temporary files.
 #* Your personal backups. (The other copy is on your computer, right?)
-# If a file is large or accessed very infrequently, and it cannot be easily recreated or re-uploaded place it under ~/k2-data.<br>Examples:
+# If a file is large or accessed very infrequently, and it cannot be easily recreated or re-uploaded, place it under <code>~/k2-data</code>.<br>Examples:
-#* Photos and videos that your users uploaded to your website
+#* Photos and videos that your users uploaded to your website.
-# If the file is small or accessed frequently, and you can easily recreate or re-upload it should it be lost, place it under ~/k2-fastscratch.<br>Examples:
+# If the file is small or accessed frequently, and you can easily recreate or re-upload it should it be lost, place it under <code>~/k2-fastscratch</code>.<br>Examples:
 #* Temporary files, such as PHP session files.
 #* Logs (if you don't care about them).
@@ Line 41: / Line 56: @@
 == Configuration ==
-All configuration is done by editing the contents of the ~/.k2 directory, and then running a command (usually <code>k2-update</code> or <code>k2-service restart -a</code>) to apply the changes.
+All configuration is done by editing the contents of the <code>~/.k2</code> directory, and then running a command (usually <code>k2-update</code> or <code>k2-service restart -a</code>) to apply the changes.
 === Web Server ===
@@ Line 47: / Line 62: @@
 You can use any web server software you like. The web server runs in your user account.
-The default configuration is to use Apache, which allows familiar configuration using .htaccess and .htpasswd files.
+The default configuration is to use Apache, which allows familiar configuration using <code>.htaccess</code> and <code>.htpasswd</code> files.
-Note that k2system will stop Apache if it doesn't receive a request in over an hour, which is why you may not always see it in the process list. It is automatically started back up when a request arrives.
+Note that ''k2system'' will stop Apache if it doesn't receive a request in over an hour, which is why you may not always see it in the process list. It is automatically started back up when a request arrives.
 ==== Apache ====
-The configuration for your personal Apache instance is located in ~/.k2/apache2.
+The configuration for your personal Apache instance is located in <code>~/.k2/apache2</code>.
-By default, the configuration consists of a number of symlinks to the default k2system configuration files. If you would like to edit a configuration file, you will first need to replace the symlink with a copy of the file that the symlink pointed to. For your convenience, a "delink" command exists which does this. For example:
+By default, the configuration consists of a number of symlinks to the default ''k2system'' configuration files. If you would like to edit a configuration file, you will first need to replace the symlink with a copy of the file that the symlink pointed to. For your convenience, a <code>delink</code> command exists which does this. For example:
   delink .k2/apache2/conf.d/site.conf
   mcedit .k2/apache2/conf.d/site.conf
-By default, only a small number of modules are enabled (see ~/.k2/apache2/mods.d). To enable a new module, you can create a symlink pointing to the corresponding file in the k2system default configuration. Example:
+By default, only a small number of modules are enabled (see <code>~/.k2/apache2/mods.d</code>). To enable a new module, you can create a symlink pointing to the corresponding file in the ''k2system'' default configuration. Example:
   ln -s /usr/local/etc/k2/apache2/mods.d/cgi.load .k2/apache2/mods.d/cgi.load
@@ Line 80: / Line 95: @@
 PHP is configured in a similar manner as Apache:
-* php.ini is located in ~/.k2/php/php.ini. By default, it is a symlink to /usr/local/etc/k2/php/php.ini.
+* php.ini is located in <code>~/.k2/php/php.ini</code>. By default, it is a symlink to <code>/usr/local/etc/k2/php/php.ini</code>.
-* PHP modules can be enabled by creating symlinks in ~/.k2/php/conf.d.
+* PHP modules can be enabled by creating symlinks in <code>~/.k2/php/conf.d</code>.
+=== MySQL ===
+Your MySQL database password is different from your login password, and was sent to you by email during registration.
+To access the database from your website, you can use the following credentials:
+* Hostname: <code>localhost</code> or blank
+* Username: your SSH login username
+* Password: your MySQL password
+* Database: your SSH login username
+For an interactive SQL prompt, simply run <code>mysql</code> from a SSH command line.
+The MySQL client is preconfigured with your credentials (see <code>~/.my.cnf</code>).
+To dump the entire database, you can run e.g.: <code>mysqldump ''your-username'' > database.sql</code>
+A phpMyAdmin instance is available at https://pma-k3.1azy.net/. You can log in with your username and MySQL password.
+=== Logs ===
+Apache logs (access and error) are, by default, placed in <code>~/.k2/apache2/logs</code>.
+Log rotation is configured in the <code>~/.k2/logrotate.template</code> directory. By default, Apache logs are rotated weekly.
 === Domains ===
-By default, your website is accessible via the server's wildcard DNS entry (*.k3.1azy.net).
+By default, your website is accessible via the server's wildcard DNS entry (*.k3.1azy.net). If your UNIX (SSH login) username is ''alice'', the website can be accessed at https://alice.k3.1azy.net/.
 If you would like to add a nicer hostname to your website, do the following:
 # Point the hostname to the server's IP address
-# Create a file in ~/.k2/domains named after the hostname (e.g. if you would like to use "you.example.com", create "~/.k2/domains/you.example.com")
+# Create a file in <code>~/.k2/domains</code> named after the hostname (e.g. if you would like to use "you.example.com", create <code>~/.k2/domains/you.example.com</code>)
-# Run k2-update to tell k2system to update the server configuration.
+# Run <code>k2-update</code> to tell ''k2system'' to update the server configuration.
 === DNS ===
-You can also host your DNS zone on the server. To do so, create a zone file (in standard BIND syntax, with a .hosts extension) in ~/.k2/dns, and run k2-update.
+You can also host your DNS zone on the server. To do so, create a zone file (in standard BIND syntax, with a <code>.hosts</code> extension) in <code>~/.k2/dns</code>, and run <code>k2-update</code>.
-For your convenience, k2system also understands a file format with a .hosts-template extension, which allows substituting variables which apply to all domain names. As such, the easiest way to add a zone is to create ~/.k2/dns/'''''hostname'''''.hosts-template with the following contents:
+For your convenience, ''k2system'' also understands a file format with a <code>.hosts-template</code> extension, which allows substituting variables which apply to all domain names. As such, the easiest way to add a zone is to create <code>~/.k2/dns/'''''hostname'''''.hosts-template</code> with the following contents:
   $TTL 38400   ; default expiration time of all resource records without their own TTL value
@@ Line 116: / Line 155: @@
   mail		IN	A	%SERVERIP4%
   mail		IN	AAAA	%SERVERIP6%
+ k3._domainkey	IN	CNAME	k3._domainkey.%SERVERHOST%.
   www		IN	CNAME	@
@@ Line 123: / Line 163: @@
 The server does not store email, so there are no mailboxes to check (or access via POP3/IMAP or otherwise). However, redirects can be configured.
-To configure what happens to mail sent to '''''localpart'''''@your-domain.com, create the file ~/k2/mail/aliases/'''''localpart''''' . It can contain an email address to redirect to, or /dev/null to discard all email. See [https://www.exim.org/exim-html-current/doc/html/spec_html/ch-the_redirect_router.html#SECTspecitredli the exim documentation] for more details. If a file corresponding to the local part does not exist, the file named "default" is consulted instead.
+Assuming that your DNS MX records are set up to point to the server's IP addresses, email will be accepted for the domains you configured (in <code>~/.k2/domains</code>). To configure what happens to mail sent to <code>'''''localpart'''''@your-domain.com</code>, create the file <code>~/.k2/mail/aliases/'''''localpart'''''</code>. It can contain an email address to redirect to, or <code>/dev/null</code> to discard all email. See [https://www.exim.org/exim-html-current/doc/html/spec_html/ch-the_redirect_router.html#SECTspecitredli the exim documentation] for more details. If a file corresponding to the local part does not exist, the file named <code>default</code> is consulted instead.
-Internally generated mail is sent to the email address indicated in the ~/.email file.
+Internally generated mail is sent to the email address indicated in the <code>~/.email</code> file.
+Note that the files in the aliases directory must have mode <code>644</code>. (Depending on how you log in, they may be instead created with mode <code>664</code>.) Use <code>chmod</code> to change it.
+Make sure to run <code>k2-update</code> to apply changes in <code>~/.k2/mail</code>.
+{{Warning|Please do not configure redirects for email addresses which receive a lot of spam. Doing so negatively affects the reputation of the server's IP address, which eventually leads '''all''' sent and redirected mail to be marked as spam. If you receive a large amount of mail, please use a professional (paid) email service.}}
 === SSL ===
@@ Line 133: / Line 179: @@
 ==== Bring Your Own Certificate ====
-If you already have an SSL certificate for your domain and would like to use it, place the files in the ~/.k2/ssl directory as follows:
+If you already have an SSL certificate for your domain and would like to use it, place the files in the <code>~/.k2/ssl</code> directory as follows:
-* ~/.k2/ssl/site.pem - your certificate
+* <code>~/.k2/ssl/site.pem</code> - your certificate
-* ~/.k2/ssl/chain.pem - your SSL provider's certificate
+* <code>~/.k2/ssl/chain.pem</code> - your SSL provider's certificate
-* ~/.k2/ssl/site.key - your certificate's private key
+* <code>~/.k2/ssl/site.key</code> - your certificate's private key
 ==== Let's Encrypt ====
-Unless you already have an SSL certificate, the easiest way to enable SSL is using the free Let's Encrypt certificate provider. Simply run "k2-letsencrypt" to enable SSL for your configured domains.
+Unless you already have an SSL certificate, the easiest way to enable SSL is using the free Let's Encrypt certificate provider. Simply run <code>k2-letsencrypt</code> to enable SSL for your configured domains.
+Note: <code>k2-letsencrypt</code> does not work with the default *.k3.1azy.net wildcard hostnames. You will need to [[#Domains|add a different hostname]] to enable SSL for your website.
+==== Redirect to HTTPS ====
+You can redirect clients which opt-in to SSL to HTTPS by placing the following in <code>~/www/.htaccess</code>:
-Note: k2-letsencrypt does not work with the default *.k3.1azy.net wildcard hostnames. You will need to [#Domains|add a different hostname]] to enable SSL for your website.
+ # Redirect to HTTPS clients indicating their preference for it
+ # Note: this should also send a Vary header, but doesn't due to Apache bug
+ RewriteCond %{HTTP:Upgrade-Insecure-Requests} ^1$
+ RewriteCond %{HTTP:X-Scheme} !https
+ RewriteRule ^(.*)$ https://%{HTTP_HOST}/$1 [R,L]