Notes on Tech Blog

ProLUG SEC Unit 10 🔒

Tue, 17 Jun 2025 00:00:00 +0000

Intro 👋

This is the final Unit and close to the Linux Security Course. Though we did not have labs for this Unit, we did spend a lot of time reflecting.

I recently finished the first ProLUG Security Engineering Course, designed and delivered by Scott Champine, also known as Het Tanis, from ProLUG. It ran for about 10 weeks and clocked in at roughly 100 hours of focused effort—but honestly, I probably put in more than that once you count the spontaneous study sessions and the many side discussions that came up. A small group of us showed up consistently and really dug into the material, connecting ideas and bouncing thoughts off each other.

The course itself was free and not tied to any official institution, but it was taught by a seasoned industry professional who also teaches at the post-secondary level. Scott clearly cares about the subject and about helping others understand it. That came through in how he delivered the material, and it brought out a real sense of commitment in us too.

On top of just taking the course, I also helped shape it for future learners by starting a version-controlled course book. We had a small group that met weekly to go over edits and review pull requests. A few people even joined just to learn Git so they could contribute, which added to the sense of shared effort and made the experience even better.

One of the things that helped me stay on track was having a study group. There’s a lot of sharp, motivated people in the ProLUG community, and quite a few of them kept up a steady pace through both the course and the book. The regular check-ins and shared discussions made a big difference.

The course itself covered a wide range of topics and gave me a stronger sense of how enterprise security is put together, maintained, and kept resilient. Security isn’t just about ticking boxes—it touches every part of a system. Especially with Linux, where multiple users and external inputs are constantly in play, it doesn’t take much for something to go sideways if you’re not paying attention.

We worked through the process of hardening Linux systems using STIGs—basically long, detailed lists of potential vulnerabilities and how to guard against them. It’s not fast work, but it really forces you to think about each configuration choice.

Patching was another major topic, and not in the usual “just update it” way. We talked about how every change introduces risk, and how important it is to approach patching as part of a controlled, planned process. That includes things like internal repositories, known-good system images, and minimizing surprise behavior from updates.

We also got hands-on with locking down systems: managing ingress and egress, shutting off unnecessary ports, setting up bastion hosts, and building out logging and alerting. We even worked on ways to trap misbehaving users or bots inside chroot jails. One of the others in the group even automated that process with a Bash script for their final project.

We had deep conversations about monitoring too—things like how to design alerts that people can actually respond to, without burning out from constant noise. We looked at log filtering, storage, and what makes a log useful rather than just more clutter.

We also talked about automation and how it can sometimes get away from you. It’s easy for parts of a system to drift out of spec if you’re not careful, especially with orchestration tools. So we looked at how to use infrastructure-as-code and version control to make changes traceable and systems more predictable.

Toward the end of the course, we focused on trust, keys, and certificates. We got practical—generating and managing key pairs, breaking them, fixing them, and eventually building up to TLS certificates. These exercises helped drive home how trust is managed inside systems, especially in setups that lean toward zero trust.

Before this course, I already had a decent background in cybersecurity—some labs, a few certifications—but this gave me something more solid. I now feel like I understand what it means to build security into a system, rather than just bolt it on. I’m more confident setting up and maintaining a hardened Linux environment, and more thoughtful about how to track and manage change over time.

That said, I don’t think I’ve “arrived.” If anything, this course just made me more aware of how much I still have to learn. I’ve moved into that space where I know what I don’t know, and that’s a valuable place to be. It’ll take years to keep digging through it all, but now I’ve got a better starting point—and the confidence to figure things out when new challenges come up.

All in all, this course gave me a deeper appreciation for operational security, and it left me with some solid tools I’ll continue to use. Like with the Admin course before it, I really valued the people I got to work with. I expect we’ll keep exploring these topics together for a long time. And, like always, you make a few good friends along the way.

Discussion Post 1

Question

How many new topics or concepts do you have to go read about now?

Answers

TLS Transport Layer Security: Prior to the course, I was aware of the terminology and had a 30,000 Foot conceptual view. During this course, I was able to zoom in an take a look at the transport and the layers. However, given the shear scale and complexity of the topic. I will have to read through the 1.1, 1.2 and 1.3 Specifications. One of my favorite IT Authors Michael W. Lucas has a book for sale on the topic. https://www.tiltedwindmillpress.com/product/tls/
ZT Zero Trust: I get from a high level view as well. I learned that Zero Trust is a popular buzz-word or a form of jargon for most. Actually drilling down and understanding the many forms and configurations of a ZT network is an immense undertaking. On the side I had done some additional reading about it, for example, I read through some of https://www.cisa.gov/sites/default/files/2023-04/CISA_Zero_Trust_Maturity_Model_Version_2_508c.pdf and plan to dig deeper into the subject.
Tokenization & Data Masking are two interesting topics. If anyone can recommend materials, I am interested. So far I have just found Wikipedia for the explanations.
SSO Single Sign On: I found this book: https://fw2s.com/wp-content/uploads/2017/09/definitive-guide-to-single-sign-on.pdf
DMARC Domain-based Message Authentication Reporting & Conformance: I do have the book Run your own mail server by Michael W. Lucas. I am assuming he covers the topic to some degree.
SPF Sender Policy Framework: Which should be covered by the aforementioned book as well.
CVSS Common Vulnerability Scoring System: I found this paper on the matter, it is a technical specification. https://www.first.org/cvss/v3-1/cvss-v31-specification_r1.pdf
TSDB's Time Series Databases: I have heard the concept from this course and in game development. I think the concept is easy to grasp. But I would like to investigate further.

Question

What was completely new to you?

Answer

STIGS for sure. Just prior to starting the course, I was given a glimpse of the Stig’ing process. In the course we were tasked with getting the StigViewer working, downloading specific STIG’s and Implementing hardening while answering prompts about what specifically we were doing. It really helped to have finished the admin course prior to this as it made the objectives more clear to me.
Bastions Hosts Prior to using the one implemented on Scott’ own server, I had not seen this. Drilling into the concept and creating a Bastion in the lab was a nice intro.

Question

What is something you heard before, but need to spend more time with?

Answer

I had heard the Acronyms for many of the concepts prior to this course. In my answer to Question #1, I had already detailed what I will need to dig into after the course is complete.

Discussion Post 2

Scenario

Think about how the course objectives apply to the things you’ve worked on.

Question

How would you answer if I asked you for a quick rundown of how you would secure a Linux system?

Answer

First, I’d check open ports using ss -ntulp to see what services are listening and close anything unnecessary.
Next, I’d check how many user accounts exist by running cat /etc/passwd | wc -l, and optionally review users with high UIDs to see who has real login access.
I’d confirm that root login over SSH is disabled by checking /etc/ssh/sshd_config and setting PermitRootLogin no.
Then I’d check for any accounts with empty passwords using awk -F: '($2 == "") { print $1 }' /etc/shadow.
I’d list which users have sudo access by checking the sudo group or reviewing /etc/sudoers.
I would review running services with systemctl list-units --type=service and disable anything that isn’t needed.
Then I’d make sure a firewall is enabled and configured, using firewalld, ufw, or iptables, depending on the system.
I’d update all packages using the system’s package manager like dnf, apt, or yum to ensure known vulnerabilities are patched.
I’d also check file permissions on sensitive files like /etc/shadow and /home/* directories.
If SSH is exposed, I’d install and configure fail2ban to protect against brute-force login attempts.
I’d regularly check system logs like /var/log/auth.log or use journalctl to spot anything suspicious.
Lastly, I’d run a tool like Ansible Lock-Down to audit and find common misconfigurations.

Question

How would you answer if I asked you why you are a good fit as a security engineer in my company?

Answer

Though I am not a seasoned Security Engineer, I possess a solid understanding of Linux, system hardening, and monitoring techniques, along with a strong foundation in high-level concepts related to ensuring security, reliability, and confidentiality in systems and networks. I am a diligent learner and a prolific documenter, always striving to deepen my knowledge and contribute meaningfully to operational resilience and security best practices.

Frame

Think about what security concepts you think bear the most weight as you put these course objectives onto your resume.

Question

Which would you include?

Answer

I would perhaps list generalities

Linux System Security Auditing
Linux System Hardening
Linux System Monitoring
Linux System Access Control
Encryption & Certificate Management
Infrastructure Security Governance Compliance

Question

Which don’t you feel comfortable including?

Answer

Network Security
Transport Layer Security

ProLUG Links ⛓️

Discord: https://discord.com/invite/m6VPPD9usw Youtube: https://www.youtube.com/@het_tanis8213 Twitch: https://www.twitch.tv/het_tanis ProLUG PSC Repo: https://github.com/ProfessionalLinuxUsersGroup/psc ProLUG PSC Book: https://professionallinuxusersgroup.github.io/psc/ ProLUG Book of Labs: https://leanpub.com/theprolugbigbookoflabs KillerCoda: https://killercoda.com/het-tanis

ProLUG SEC Unit 9 🔒

Sat, 31 May 2025 00:00:00 +0000

Intro 👋

In this Unit we look at how Certificates and Keys go beyond Asymmetric encryption with Public / Private. We look at how multiple checks and multiple layers of trust must be used in this mad mad world. ¹

Worksheet

Question

How do these topics align with what you already know about system security?

Answer

Well I had felt like I had a clear picture of Symmetric and Asymmetric encryption modalities. Furthermore, I had a strong prior understanding of x.509 and SSH where Asymmetric encryption is used. Moreover, the procedure of generating Private and subsequent public keys. However, the verbosity and complexity of the required reading has me scratching my head and looking at more sophisticated modality of key generation and exchange eg. TLS 1.2, 1.3

Question

Were any of the terms or concepts new to you?

Answer

key-transport and/or key-agreement protocols - a method of establishing a shared secret key between two or more parties where one party creates the key and securely delivers it to the others.

Challenge Values - dynamic, randomly generated numbers or strings used to initiate authentication.

nonce - a unique, random or pseudo-random number used to ensure the security and integrity of data transmitted over a network. Watch short video about CA and Chain of Trust Distributed Trust Model Review the TLS Overview section, pages 4-7 of https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-52r2.pdf and answer the following questions. What are the three subprotocols of TLS? How does TLS apply Confidentiality Integrity Authentication Anti-replay

Question

What are the three subprotocols of TLS?

Answer

The handshake used to negotiate the session parameters.

Change cipher spec used in TLS 1.0, 1.1, and 1.2 to change the cryptographic parameters of a session.

Alert protocols used to notify the other party of an error condition.

Question

How does TLS apply to: Confidentiality Integrity Authentication Anti-replay

Answer

Confidentiality

Confidentiality is provided for a communication session by the negotiated encryption algorithm for the cipher suite and the encryption keys derived from the master secret and random values.

Integrity

TLS uses a cipher suite of algorithms and functions, including key establishment, digital signature, confidentiality, and integrity algorithms. In TLS 1.3, the master secret is derived by iteratively invoking an extract-then-expand function with previously derived secrets, used by the negotiated security services to protect the data exchanged between the client and the serve. In TLS 1.3, only AEAD symmetric algorithms are used for confidentiality and integrity.

Authentication

Server authentication is performed by the client using the server’s public-key certificate, which the server presents during the handshake.

Anti-Replay

in TLS 1.3 The integrity-protected envelope of the message contains a monotonically increasing sequence number. Once the message integrity is verified, the sequence number of the current message is compared with the sequence number of the previous message.

Definitions

TLS (Transport Layer Security) A protocol that encrypts data in transit to ensure privacy and integrity.
Symmetric Keys A cryptographic method where the same key is used for both encryption and decryption.
Asymmetric Keys A method using a public/private key pair where one key encrypts and the other decrypts.
Non-Repudiation A guarantee that a sender cannot deny the authenticity of their message or signature.
Anti-Replay A mechanism that prevents attackers from reusing valid data packets to mimic legitimate transactions.
Plaintext Data in a readable and unencrypted format.
Cyphertext Data that has been encrypted and is unreadable without the correct decryption key.
Fingerprints Short unique representations (hashes) of public keys used to verify their authenticity.
Passphrase (in key generation) A user-supplied string that encrypts private keys to protect them from unauthorized access.

Lab 🧪

Assignment

Complete the lab: https://killercoda.com/het-tanis/course/Linux-Labs/211-setting-up-rsyslog-with-tls

Answer

We generated a 90 day TLS web client certificate. I saved a snippet of the options below.

Activation/Expiration time.
The certificate will expire in (days): 90
Extensions.
Does the certificate belong to an authority? (y/N): y
Path length constraint (decimal, -1 for no constraint):
Is this a TLS web client certificate? (y/N): y
Will the certificate be used for IPsec IKE operations? (y/N): y
Is this a TLS web server certificate? (y/N): y
Enter a dnsName of the subject of the certificate:
Enter a URI of the subject of the certificate:
Enter the IP address of the subject of the certificate:
Will the certificate be used for signing (DHE ciphersuites)? (Y/n): y
Will the certificate be used for encryption (RSA ciphersuites)? (Y/n): y
Will the certificate be used for data encryption? (y/N): y
Will the certificate be used to sign OCSP requests? (y/N): y
Will the certificate be used to sign code? (y/N): y
Will the certificate be used for time stamping? (y/N): y
Will the certificate be used for email protection? (y/N): y
Will the certificate be used to sign other certificates? (Y/n): y
Will the certificate be used to sign CRLs? (y/N): y
Will the certificate be used for signing (DHE ciphersuites)? (Y/n): y
Enter the URI of the CRL distribution point:
X.509 Certificate Information:
Version: 3
Serial Number (hex): 32a1646105dcb6229eba87ad4c08a99a2bb92a99
Validity:
Not Before: Mon Jun 02 03:46:43 UTC 2025
Not After: Sun Aug 31 03:46:48 UTC 2025
Subject: O=prolug
Subject Public Key Algorithm: RSA
Algorithm Security Level: High (3072 bits)
Modulus (bits 3072):
00:e8:c7:f5:6e:7c:23:e3:7e:e7:d0:c5:c4:cf:c0:98
23:5f:1e:f6:5f:5d:87:c6:c8:18:13:cb:5e:1b:1a:88
03:98:4d:55:5d:4d:14:cc:78:8d:83:e3:c5:65:16:8c
41:a8:9f:32:ab:f4:47:3f:84:b2:b8:0d:7c:b3:a6:e7
21:59:13:d2:45:40:60:d6:2c:eb:5a:f3:00:0c:e7:36
06:0f:ca:51:04:92:06:91:80:f0:04:52:d2:66:e3:33
11:7b:8e:f7:e3:22:19:83:c8:dc:c8:f9:18:c7:51:4f
38:6a:d8:07:bf:12:02:f4:5e:0d:52:2e:cc:0b:4e:d9
e0:b2:07:9a:cd:39:99:a7:28:42:e4:67:b0:ff:04:2d
f9:13:8c:0f:19:b5:13:ee:59:a3:e7:e8:f7:a1:e9:92
2e:ce:49:23:3c:0a:b4:29:ca:5d:74:6e:9e:09:ea:fd
72:6a:89:6e:5f:29:d6:0a:44:98:1e:2c:39:66:44:11
4f:47:c5:64:a3:0c:84:2b:fd:32:2e:a9:ce:e7:be:b4
7c:3b:e6:b9:23:98:82:ac:86:20:07:4e:59:84:4d:0c
02:38:76:87:ef:f8:17:05:5b:93:79:25:73:fc:18:f5
4e:1d:ff:84:45:10:7d:46:51:69:ae:73:6d:e9:1e:fd
ff:55:5a:78:4d:f6:cd:44:af:22:0f:b0:18:fb:82:b9
f6:aa:3d:2a:08:00:62:d1:9b:28:50:94:39:98:f5:de
f9:cf:3f:d8:ae:72:68:69:f1:46:97:8f:d5:a6:9a:3e
4c:57:37:5f:69:0e:2f:4e:b6:6e:65:a5:2c:f0:5b:c6
c2:ff:43:b7:4e:b7:56:3f:2b:d8:5d:b9:73:15:ca:81
f1:c3:78:2f:8d:4f:fd:e8:2d:6f:2f:2d:f6:b9:e1:a0
11:f2:56:18:02:5b:8e:07:da:19:43:c1:70:bc:7b:8b
82:2b:02:e2:71:6e:30:9b:18:8d:ed:1f:29:59:86:9d
81
Exponent (bits 24):
01:00:01
Extensions:
Basic Constraints (critical):
Certificate Authority (CA): TRUE
Key Purpose (not critical):
TLS WWW Client.
TLS WWW Server.
Ipsec IKE.
OCSP signing.
Code signing.
Time stamping.
Email protection.
Key Usage (critical):
Digital signature.
Key encipherment.
Data encipherment.
Certificate signing.
CRL signing.
Subject Key Identifier (not critical):
213b20bf44b3446fb14f6cf72b8c2c03a09e292e
Other Information:
Public Key ID:
sha1:213b20bf44b3446fb14f6cf72b8c2c03a09e292e
sha256:7f76aada143491a8ba0721509a3e49f9e72321ed880f7ee64b8e01172989b3d2
Public Key PIN:
pin-sha256:f3aq2hQ0kai6ByFQmj5J+ecjIe2ID37mS44BFymJs9I=

Reading

Review Solving the Bottom Turtle²

Question

Does the diagram on page 44 make sense to you for what you did with a certificate authority in this lab?

Answer

Yes it does, we had only setup a portion of this chain of trust, yet it got the idea across of whom we are referring to and how we build a certificate from that referral.

Assignment

Complete the lab: https://killercoda.com/het-tanis/course/Linux-Labs/212-public-private-keys-with-ssh

Question

What is the significance of the permission settings that you saw on the generated public and private key pairs?

Answer

Only the owner has Read/Write permission to the Private key, whereas the public key – meant to be shared, is Readable by Group and Others.

ProLUG Links ⛓️

Professional Linux User Group Security Engineering Unit 8 Web Book ProLUG, 2025. ↩︎
Solving the Bottom Turtle Web Book Spiffe.io,2020. ↩︎

ProLUG SEC Unit 7 🔒

Sun, 11 May 2025 00:00:00 +0000

Intro 👋

Monitoring systems and alerting when issues arise are critical responsibilities for system operators. Effective observability ensures that system health, performance, and security can be continuously assessed.¹

Worksheet

`Discussion Post 1`

Intro to the scenario

Read about telemetry, logs, and traces²³.

Question

How does the usage guidance of that blog align with your understanding of these three items?

Answer

Though the concepts involved in telemetry are really quite simple, they took me some time to internalize and fully understand. I can’t say it paralleled my own understanding as my understanding was very limited. Prior to the lectures, if I were to hear the word telemetry, I would think of non GPS tracking techniques or some sort of secret tracking by Palantir.

My simplified outline of these 3 things:

A metric represents a point in time measurement of a particular source
Logs are discrete and event triggered occurrences.
Traces follow a program’s flow and data progression.

Question

What other useful blogs or AI write-ups were you able to find?

Answer

Yes Grafana has a good writeup on the subject.
https://grafana.com/docs/tempo/latest/introduction/telemetry/

Question

What is the usefulness of this in securing your system?

Answer

Securing a System is useful in many ways.
Prevents unwanted access.
Potentially mitigates Data Exfiltration.
Potentially mitigates unwanted Data Infiltration.
Prevents miss-use by users.
Can simply mitigate the overload of available resources.
Not only does the attack surface shrink when a system is properly secured, but the monitoring tasks reduce.
Secure systems are more predictable systems.

`Discussion Post 2`

Intro to the scenario

When we think of our systems, sometimes an airgapped system is simple to think about because everything is closed in. The idea of alerting or reporting is the opposite. We are trying to get the correct, timely, and important information out of the system when and where it is needed.

Read the summary at the top⁴.

Question

What is the litmus test for a page? (Sending something out of the system?)

Answer

The page must be pertaining to an imminent, actionable situation that must be addressed quickly.

Question

What is over-monitoring v. under-monitoring. Do you agree with the assessment of the paper? Why or why not, in your experience?

Answer

Over-monitoring can be compared to hyper-vigilance. Over time it works against you as fatigue or indifference sets in. Furthermore over-monitoring includes the transporting, receiving and dissemination of too much information, causing cognitive overload, leading to poor decision making. Furthermore, the additional information being broadcast leaves a system more susceptible/vulnerable from a security stand-point.
Under monitoring would be lack of contextual reporting, responsiveness and diligence needed in order to keep a system from going down.
From reading this article, it seems to me that one must turn monitoring into a spectrum of detail. Hypercritical indicators like uptime, load, capacity should be reported daily with a pre-determined baseline. Estimations can be made from this allowing for prediction. While major changes –those outside the predicted norm, could trigger alerts. Paging should be reserved for the utmost critical issues.

Question

What is cause-based v. symptom-based and where do they belong? Do you agree?

Answer

Cause based is the analysis/investigation of the root cause of a certain outcome. In the context of systems operating and security, it is finding the vulnerability, infiltration/exfilration point or cause of failure like hitting memory/cpu/disc space limitations.
Symptom based analysis would be to observe the effects of an unknown origins ie. systems going down, data loss etc… Bringing the system back up, or restoring data from backups does nothing to address the root cause, it only remediates the effects.

Definitions

Telemetry Automated collection of system metrics and status data.
Tracing Tracks the path and performance of requests across services.
Span A single unit of work in a trace, with start and end timestamps.
Label Key-value pair used to add metadata to traces or metrics.
Time Series Database (TSDB) A database optimized for storing data indexed by time.
Queue A data structure or service for holding and processing messages in order.
UCL/LCL Statistical limits used to detect anomalies in metrics over time.
Aggregation Combining multiple data points into a summarized form.
SLO Service Level Objective, a target performance metric.
SLA Service Level Agreement, a contractually agreed service standard.
SLI Service Level Indicator, a specific measurement of system performance.
Push Data sent actively from source to receiver.
Pull Data requested by the receiver from the source.
Alerting rules Conditions set to trigger alerts based on system metrics.
Alertmanager Tool for handling, deduplicating, and routing alerts.
Alert template Format used to display or notify alert information.
Routing Directing alerts to specific teams or destinations.
Throttling Limiting the number or rate of alerts to reduce noise.
Monitoring for defensive operations Watching systems for signs of attack or failure.
SIEM Centralized platform for analyzing security and event logs.
IDS Tool that detects suspicious or unauthorized activity.
IPS Tool that blocks or prevents detected threats in real time.

Lab 🧪

Fail2Ban Setup and Testing

Install Fail2Ban

Install Fail2Ban:
```
apt install -y fail2ban
```

Verify Installation

Check the version of Fail2Ban:
```
fail2ban-client --version
```

Configure SSHD Jail

Edit the jail configuration file:
```
vi /etc/fail2ban/jail.conf
```

Uncomment [sshd] and add the following under that section:

[sshd]
enabled = true
maxretry = 5
findtime = 10
bantime = 4h

Review the rest of the file and ensure there is no duplicate [sshd] section. Comment it out or remove it if found.

Explore Other Jails

Review configuration sections for Apache and NGINX to see other available jails.

Restart and Verify Fail2Ban

Restart the service:
```
systemctl restart fail2ban
```
Check status:
```
systemctl status fail2ban --no-pager
```

Test the SSH Ban

SSH into node01:
```
ssh node01
```

Run a loop to simulate failed login attempts:

for i in {1..6}; do ssh invaliduser@controlplane; done

Press Enter on each password prompt until Fail2Ban triggers.
Use Ctrl + C to exit the loop when connection attempts are blocked.

Check Ban Status

Return to controlplane.
View the logs:
```
tail -20 /var/log/fail2ban.log
```
Check banned IPs:
```
fail2ban-client get sshd banned
```

Question

Do you see the expected IP address in the ban list?
Why do you think that is?

Answer

Yes I was able to see it.

Unban the IP

Replace <IP> with the actual banned IP:
```
fail2ban-client set sshd unbanip <IP>
```

Confirm Unban

SSH into node01:
```
ssh node01
```
Try to reconnect to controlplane using the correct user:
```
ssh root@controlplane
```

Question

Did the connection succeed?

Answer

Yes, I was able to reconnect

ProLUG Links ⛓️

Professional Linux User Group Security Engineering Unit 7 Web Book Source, 2025. ↩︎
Observability Chapter Web Book Source, 2025. ↩︎
Telemetry Web Book Grafana, 2025. ↩︎
My Philosophy on Alerting Google Doc Rob Ewaschuk, 2014. ↩︎

ProLUG SEC Unit 8 🔒

Sun, 11 May 2025 00:00:00 +0000

Intro 👋

Configuration drift is the silent enemy of consistent, secure infrastructure. When systems slowly deviate from their intended state, whether that be through manual changes, failed updates, or misconfigured automation, security risks increase and reliability suffers.¹

Worksheet

`Discussion Post 1`

Read about configuration management²

Questions

What overlap of terms and concepts do you see from this week’s meeting?

Answer

Lifecycle management and Change Control (Change Management).
Change Management is a system for ensuring process and product integrity.
Despite these controls, variation from the norm (configuration drift) is inevitable.
So we must invoke/involve controls in order to catch variation/drift.
In the case of systems, it is bot Misconfigured Systems and Misconfigured Users to induce variation/drift.

Question

What are some of the standards and guidelines organizations involved with configuration management?

Answer

Originally developed by the U.S. Department of Defense to ensure quality, reliability, and integrity in the manufacturing supply chain, configuration management principles were later adopted and expanded upon by standards bodies such as ANSI, ISO, and IEEE. These concepts have since evolved through industry-specific frameworks, including:
ITIL
ISO/IEC
NIST
IEEE
CERN

Question

Do you recognize them from other IT activities?

Answer

For sure.
Baselining Gathering telemetry from a system at its base config
Standards Developing a standard for configuration or procedure to ensure consistent and predictable output
Controls Controlling versions, changes, configurations
Automation Automatic and Repeatable tasks
Variation Departure from the standard
Remediation Reconciliation, Correction, Rebasing

`Discussion Post 2`

Review the SRE guide to treating configurations as code. Focus down on the “Practical Advice” section ³

Question

What are the best practices that you can use in your configuration management adherence?

Answer

Don’t Check in Secrets
Make it Hermetic
- Apply the Rigor of Code
- Golden Image
Make it Reproducible
- Try to Implement a Software Bill of Materials (SBOM)
- Patching (If warranted) records.
Make it Verifiable
- Binary Provenance
- Use Signed Code
- Verify Artifacts, Not Just People
- Verifiable Build Architectures

Question

What are the security threats and how can you mitigate them?

Answer

Supply Chain Attacks
Exposure of secrets
Non-hermeticity and Drift
Over-priveleging through automation
Inadequate Auditing and Change Control
Insecure Testing Environments
Artifact Poisoning

Question

Why might it be good to know this as you design a CMDB or CI/CD pipeline?

Answer

The Pipeline is a major target. If something were to be malicious injected, the problem could propagate to all target platforms/devices.
CMDB is a Source of Truth. A misconfiguration, bad record or malicious activity could invalidate hermeticity.
Secrets and Credentials flow through the Pipeline, a whole can of worms.

Definitions

System Lifecycle The full span of a system’s life: design, build, operate, maintain, and retire.
Configuration Drift The divergence of a system’s current state from its intended or documented configuration.
Change management activities Processes that control changes to systems to reduce errors and downtime.
CMDB (Configuration Management Database) A database tracking system components and their relationships.
CI (Configuration Item) Any component in the CMDB (e.g., server, software, network) being tracked and managed.
Baseline A known good configuration state used for comparison and control.
Build book A documented set of steps to initially install and configure a system.
Run book A manual or automated guide for maintaining or operating a system post-deployment.
Hashing The process of generating a fixed-size value from data to verify integrity.
md5sum Tool that calculates a 128-bit MD5 hash for checking file integrity.
sha<x>sum Tools (e.g., sha256sum) that generate SHA-family hashes for stronger integrity checks.
IaC (Infrastructure as Code) Managing infrastructure using versioned code instead of manual processes.
Orchestration Coordinating automated tasks across multiple systems or services.
Automation Replacing manual tasks with scripts or tools to increase speed and consistency.
AIDE (Advanced Intrusion Detection Environment) A file integrity checker that detects unauthorized changes.

Lab 🧪

STIG Viewer – Change Management

Question

How many STIGs relate to “change management” in RHEL 9?

Answer

9 STIGs contain the phrase.

Question

What does a “robust change management process” imply?

Answer

Change control, peer review, versioning, testing, and approval are mandatory before config updates.

Question

Can one STIG enforce this?

Answer

No, it’s an org-wide practice beyond simple config toggles.

Question

What type of control is applied?

Answer

Technical preventative—mostly file ownership/permissions.

Question

Are they all the same?

Answer

Yes, the control type is consistent across them.

Monitoring Configuration Drift with AIDE

Question

What is /etc/aide/aide.conf.d/?

Answer

Contains rule files defining paths to hash and monitor.

Question

How many files are there?

Answer

213 files.

Question

What does aide -v show?

Answer

Version 0.18.6

Question

What is AIDE?

Answer

File integrity checker using stored hashes in a database.

Question

What does /etc/cron.daily/dailyaidecheck do?

Answer

Runs dailyaidecheck via capsh if available, otherwise with bash.

Question

What does capsh do?

Answer

Launches processes with limited capabilities—safer than full root.

Question

What does aide -i do?

Answer

Initializes the DB. It took ~4m14s. User time was ~3m30s.

Question

Why track timing?

Answer

For planning and resource estimation during mass deployments.

Question

What’s in the output?

Answer

Hashes (MD5, SHA, etc.) and /var/lib/aide/aide.db.new.

Question

What should you study?

Answer

RMD160, TIGER, CRC32, HAVAL, WHIRLPOOL, GOST.

AIDE Test Run

Question

What’s the test procedure?

Answer

Create /root/prolug/test*, run aide check.

Question

Were files detected?

Answer

Yes, under “Added entries.”

Question

Runtime?

Answer

~6m38s, user ~5m54s, sys ~8s.

Remediating Drift with Ansible

Question

What does the web env lab do?

Answer

Deploys 3 virtual hosts (dev, test, qa) on ports 808{0,1,2}.

Question

How do you test?

Answer

curl node01:808{0,1,2}

Question

What happened to 8081?

Answer

It failed initially—intentional drift.

Question

Does re-running the playbook fix it?

Answer

Yes, restores state without manual steps.

Question

Will that always work?

Answer

Yes, unless networking/firewall issues prevent access.

Question

Can this cause issues?

Answer

Yes, if configs were changed manually after deployment.

Question

Root cause: tech or ops?

Answer

Operational—teams must coordinate changes.

Challenge: Custom Reporting

Question

How would you verify stamp compliance?

Answer

Use Ansible facts and add deployment date as a custom variable.

ProLUG Links ⛓️

Professional Linux User Group Security Engineering Unit 8 Web Book ProLUG, 2025. ↩︎
Configuration Management Wiki Wikipedia, 2025. ↩︎
Building Secure and Reliable Systems Web Book Google, 2025. ↩︎

ProLUG SEC Unit 6 🔒

Sun, 04 May 2025 00:00:00 +0000

Intro 👋

Monitoring and parsing logs is essential to operational intelligence. Computers typically produce immense amounts of data—far more than a human can interpret in real time. To extract meaning from this data, we must intelligently filter event logs into clear, comprehensible, and actionable items.

Achieving this is easier said than done. This unit offers general advice on the art of making complex information comprehensible. ¹

Worksheet

`Discussion Post 1`

Review chapter 15 of the SRE book:
²

There are 14 references at the end of the chapter. Follow them for more information. One of them by Julia Evans ³should be reviewed for question “c”.

Question

What are some concepts that are new to you?

Answer

Core dumps, Memory dumps, or Stack traces.

I have heard the terms before and understand the concepts to a basic degree. I decided to do a bit of further reading to understand each of the dumps and traces so here is a gist.
- A core dump is a snapshot of the processes state at the time of downing.
- A Memory dump is a snapshot of the Random Access Memory (RAM) at the time of downing.
- A Stack trace is the process of tracing function calls through the stack from end (error) to beginning (Call). The way I personally conceptualize this is through comparison to root cause analysis, something I am familiar with.
Host intrusion detection systems (HIDS) or Host Agents

A few ideas from the book² “Modern (sometimes referred to as “next-gen”) host agents use innovative techniques aimed at detecting increasingly sophisticated threats. Some agents blend system and user behavior modeling, machine learning, and threat intelligence to identify previously unknown attacks.”

“Host agents always impact performance, and are often a source of friction between end users and IT teams. Generally speaking, the more data an agent can gather, the greater its performance impact may be because of deeper platform integration and more on-host processing.”

Question

There are 5 conclusions drawn, do you agree with them? Would you add or remove anything from the list?

Answer

To begin with, here are the conclusions drawn:

“Debugging is an essential activity whereby systematic techniques—not guesswork—achieve results.”
“Security investigations are different from debugging. They involve different people, tactics, and risks.”
“Centralized logging is useful for debugging purposes, critical for investigations, and often useful for business analysis.”
“Iterate by looking at some recent investigations and asking yourself what information would have helped you debug an issue or investigate a concern.”
“Design for safety. You need logs. Debuggers need access to systems and stored data. However, as the amount of data you store increases, both logs and debugging endpoints can become targets for adversaries.”

Firstly, I would like to preface this answer with a disclaimer. I lack the competency to critisize and/or disect O’Relly’s book. With that out of the way. I am going to target the first point.

My only criticism here is that the point is very broad in scope as compared to the more granular and topics specific to this book/chapter.

Question

In Julia Evan’s debugging blog, which shows that debugging is just another form of troubleshooting, what useful things do you learn about the relationship between these topics?

Answer

Both debugging and troubleshooting involve:
Proceduralization: If a clear procedure doesn’t exist, begin documenting and formalizing the process into a repeatable method.
Humility: Acknowledge that you might be the cause of the problem. This is especially important in development.
Methodical Experimentation: Form a hypothesis, then devise a controlled method to test it—use unit tests in development, or targeted scripts and commands when debugging.
One Step at a Time: Tackle problems incrementally—“eat the elephant one bite at a time.”
Strong Foundations: Write debuggable code and build robust systems. A good foundation makes issues easier to isolate.
More Is Better: Verbose error messages provide more clues—enable detailed output when possible.

Question

Are there any techniques you already do that this helps solidify for you?

Answer

Yes, I try to create excellent documentation with respect for my future self or others I may need to share it with. This involves numbered procedural steps with inputs and outputs, if that is the nature of the work. Otherwise, I write in a general manner that is legible to others.

`Discussion Post 2`

Read Monitoring Distributed Systems ⁴

Question

What interesting or new things do you learn in this reading? What may you want to know more about?

Answer

Interesting Concept:

One of the general themes I gathered from this article is low cognitive overhead. It’s a concept I’m very familiar with from accessibility-focused design. Too much information overwhelms our ability to observe, absorb, and decide effectively.

For example, public signage must be simple, legible, and self-descriptive through clear graphic composition—guiding the eye where to look first and in which direction to proceed. This closely parallels the need for simplicity in monitoring and alerting systems. When such systems become overly complex, they can lead to misinterpretation, miscommunication, and fatigue due to information overload.

Information must be derived and presented in a way that is easily consumable, where errors are unmistakable—without exhausting the viewer.

New concepts

White box monitoring systems vs. Black box monitoring systems.
Conducting ad hoc retrospective analysis (ie. Debugging)
(4 Golden signals) Latency, Traffic, Saturation, Errors
- This one in particular relates strongly to the USE acronym I had recently picked up from Het, Utilization, Saturation, Errors

Question

What are the “4 golden signals”?

Answer

Latency
Traffic
Saturation
Errors

Question

After reading these, why is immutability so important to logging?

Answer

Tamper Resistance: Immutable logs cannot be altered or deleted without detection, which helps prevent covering up malicious activity or mistakes.
Auditability: Logs serve as historical records. If they can be changed, audits and investigations lose their value.
Debugging Integrity: Developers and operators rely on logs to trace errors. Mutable logs can introduce false positives or hide root causes.
Regulatory Compliance: Standards like HIPAA, PCI-DSS, and GDPR often require tamper-evident or immutable log storage.
Forensic Value: In incident response, immutable logs serve as trustworthy evidence for timelines and breach analysis.

Question

What do you think the other required items are for logging to be effective?

Answer

In order to be effective, log must be:

Trustworthy: Logs should be immutable.
Time-stamped: Every entry needs a synced timestamp.
Clear levels: Use INFO, ERROR, DEBUG, etc., to show importance.
Structured: Format logs so machines and humans can read them.
Context-rich: Include request IDs, user info, IPs—anything that helps trace the story.
Centralized: Gather logs in one place for easy searching and alerting.
Searchable: You should be able to find issues fast with good queries.
Safe: Control who can see logs—some contain sensitive info.
Durable: Logs shouldn’t disappear in a crash—use backups and redundancy.
Noise-controlled: Avoid flooding—rotate logs and cap log rates.

Definitions

Application Logs from software applications showing events or errors.
Host Logs from the operating system, like kernel or authentication events.
Network Logs capturing traffic, connections, and protocol activity.
DB Logs from databases showing queries, errors, and access.
Immutable Logs that cannot be changed once written.
RFC 3164 BSD Syslog Older syslog format with simple priority and message structure.
RFC 5424 IETF Syslog Modern syslog format with structured data and better timestamps.
Systemd Journal Binary log format used by systemd with metadata support.
Log rotation Archiving or deleting old logs to manage disk space.
Rsyslog Advanced syslog daemon for filtering, formatting, and remote logging.
Log aggregation Collecting logs from multiple sources for analysis.
ELK Stack using Elasticsearch, Logstash, and Kibana to manage logs.
Splunk Tool for searching and analyzing machine-generated logs.
Graylog Open-source platform for central log collection and analysis.
Loki Log system that indexes by labels, optimized for Grafana.
SIEM Tools that collect and analyze security data for threat detection.

Lab 🧪

RSYSLOG

Reliable System and Kernel Logging System

Basic Steps:

Ensure Rsyslog is installed and running on both the control-plane and target node.
Configure sending of logs over UDP Port.
Editing Rsyslog config to split out logs.

Question

Why do we split out the logs in this lab? Why don’t we just aggregate them to one place?

Answer

We are aggregating the logs.
So that we can tell where the logs are coming from.
Each node is getting its own directory in `/var/log.

Question

What do we split them out by?

Answer

We split them b y hostname.

Question

How does that template configuration work?

Answer

It will log to a specific directory named after the target hostname in /var/log.

Question

Are we securing this communication in any way, or do we still need to configure that?

Answer

No we are not securing this communication, yes it needs further configuring.

Lab

Complete the lab here: https://killercoda.com/het-tanis/course/Linux-Labs/102-monitoring-linux-logs

Question

Does the lab work correctly, and do you understand the data flow?
Yes

Promtail (collects)
Loki (stores)
Grafana (visualizes)

loki-write.py

Question

Can you see it in your Grafana?

Answer

Yes Scott is too awesome!

Question

Can you modify the file loki-write.py to say something related to your name?

Answer

msg = ‘On server {host} detected error - Treasure Wuz Here’.format(host=host)

Question

Can you modify that to see the actual entires?

Answer

Lab

Complete the killercoda lab found here: https://killercoda.com/het-tanis/course/Linux-Labs/108-kafka-to-loki-logging

Question

Did you get it all to work?

Answer

Question

Does the flow make sense in the context of this diagram?

Answer

Yes
Uses kcat to write out to Kafka
Using promtail to receive the messages from kafka
Promtail pushes to loki
Displayed by grafana

Question

Can you find any configurations or blogs that describe why you might want to use this architecture or how it has been used in the industry?

Answer

Kafka interconnects log Producers and log Ingesters. Kafka can ingest logs from all types of sources and analyze in real time.
Apache Kafka, according to Apache, is the most popular open-source stream-processing software.⁵

ProLUG Links ⛓️

Professional Linux User Group Security Engineering Unit 6 Worksheet Web Book ProLUG, 2025. ↩︎
Building Secure and Reliable Systems Web Book Google, 2025. ↩︎ ↩︎
How to Debug Blog Julia Evans, 2019. ↩︎
SRE Handbook Web Book Google, 2025. ↩︎
Powered by Kafka Website Apache, 2025. ↩︎

ProLUG SEC Unit 5 🔒

Sun, 27 Apr 2025 00:00:00 +0000

Intro 👋

Repositories and Patching is the general theme of this unit. We dive into creating internally audited repositories for safe enterprise operation. This configuration allows for greater security scrutiny and compatibility testing before schedule patching takes place. For example, a company would like to skip every other version of a package in order to reduce update cadence, giving more time for assessment, correction and troubleshooting of internal software. Much like any enterprise decision regarding cost and effort and analysis must take place. ¹

Worksheet

`Discussion Post 1`

Review the rocky documentation on Software management in Linux.²

Question

What do you already understand about the process?

Answer

I had gained a decent understanding of the package management systems of both RHEL and DEBIAN based distros through both studying for the LPIC 1 and completing the ELAC course through this group. From this I had learned about versioning and dependency management including modules. The differences between RPM, YUM, and DNF and how these evolutions of package management came into being.

Question

What new things did you learn or pick up?

Answer

- I did not understand the depth to which RPM packages tracked package data.
I did not know that headers could be edited in the package in order to add custom labeling.
I had basic awareness and high level understanding of internal package management. Prior to our lecture, I had not seen an internal package server/relay be setup/configured.
I did not know much about EPEL beyond having to call it in the CLI for additional packages outside of DNF.

Question

What are the DNF plugins? What is the use of the versionlock plugin?

Answer

DNF plugins are external modules that extend the functionality of DNF. I have some experience activating the COPR repo using DNF plugins. Furthermore, versionlock is a specific plugin that allows for an admin/engineer/dev to lock a particular package to a specified version so that it is not mistakenly changed/overwritten. This is typically done in software development in my experience, with software development many dependencies might be needed. Breaking updates were common place, so most modern software projects contain a lock file that indicates what specific dependency version must be used in order to build or interpret the project.

Question

What is an EPEL? Why do you need to consider this when using one?

Answer

EPEL stands for Extra Packages for Enterprise Linux. These packages exist outside of the core enterprise offering and are therefore potentially issue causing. Unlike core packages, these extra packages could introduce possible incompatibility issues, resulting in rejection by endorsed support specialists.

`Discussion Post 2`

Do a google search for “patching enterprise Linux”³

Question

What blogs (or AI) do you find that enumerates a list of steps or checklists to consider?

Answer

I used to sources that I had found to be concise
- RedHat Documentation⁴
- Chat*ippity

Question

After looking at that, how does patching a fleet of systems in the enterprise differ from pushing “update now” on your local desktop?

Answer

Patching a fleet of systems involves the systematic updating of installed software to fix security vulnerabilities, improve stability, and introduce minor enhancements. The process is governed by organizational policies to ensure uptime and compliance.

Because changes affect many systems simultaneously, patching acts as an amplifier of problems if not handled carefully. Therefore, enterprise patching must be strategic, managed, and auditable.

In contrast, running updates on a personal system is typically an automated, low-risk operation, with little concern for version conflicts or trust in the source. Additionally, modern filesystems like ZFS and Btrfs provide the ability to quickly roll back changes if something fails.

Question

What seems to be the major considerations? What seems to be the major roadblocks?

Answer

Major Considerations
- Uptime and Service Continuity
- Security and Compliance
- Testing and Validation
- Dependency Management
- Rollback and Recovery Planning
- Orchestration and Scalability
Major Roadblocks
- Legacy Systems
- Change Resistance
- Incomplete Asset Inventory
- Tight Maintenance Windows
- Patch Quality and vendor Bugs
- Complex Dependencies and Integration Points
- Resource Constraints

Definitions

Patching The process of applying updates to fix bugs, improve security, or enhance performance.
Repos (Repositories) Remote or local collections of software packages used by package managers.
Software Applications or tools installed on a system to perform specific functions.
EPEL (Extra Packages for Enterprise Linux) A Fedora project providing additional packages for RHEL-based systems.
BaseOS The core operating system components in RHEL/Rocky, including the kernel and essential services.
AppStream A modular repository in RHEL/Rocky that provides applications and tools in versioned streams.
httpd The Apache HTTP Server package available via repos for web serving.
patching A type of update package that modifies existing software without replacing the whole binary.
GPG Key Used to verify the integrity and authenticity of packages in a repo.
DNF/YUM Package managers in RHEL-based systems used to install, update, and manage software packages.

Lab 🧪

Apache STIGs Review

Look at the 4 STIGs for “tls”

What file is fixed for all of them to be remediated?

# Install httpd on your Rocky server
systemctl stop wwclient
dnf install -y httpd
systemctl start httpd

Check STIG V-214234

Question

What is the problem?

Answer

Event logging can fail.

Question

What is the fix?

Answer

This can be fixed by implementing failure alerts.

Question

What type of control is being implemented?

Answer

This type of control would be a Detective Control

Question

Is it set properly on your system?

Answer

No, this is not setup by default, it must be implemented after installation.

Check STIG V-214248

Question

What is the problem?

Answer

By default, sensitive information including security controls may be available to all users because privelaged user access controls have not been implemented.

Question

What is the fix?

Answer

Develop roles for privelaged users and define access policies.

Question

What type of control is being implemented?

Answer

This is a preventative type control.

Question

Is it set properly on your system?

Answer

No, not by default. Of course super user has special priveledges. However, beyond that there are no other tiers of access.

Question

How do you think SELINUX will help implement this control in an enforcing state? Or will it not affect it?

Answer

SELINUX allows for strong group creation and control. So it would help batch users and apply granular control mechanisms.

Building repos

# Start out by removing all your active repos
cd /etc/yum.repos.d
mkdir old_archive
mv *.repo old_archive
dnf repolist

# Mount the local repository and make a local repo
mount -o loop /lab_work/repos_and_patching/Rocky-9.5-x86_64-dvd.iso /mnt
df -h #should see the mount point
ls -l /mnt
touch /etc/yum.repos.d/rocky9.repo
vi /etc/yum.repos.d/rocky9.repo

[BaseOS]
name=BaseOS Packages Rocky Linux 9
metadata_expire=-1
gpgcheck=1
enabled=1
baseurl=file:///mnt/BaseOS/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
[AppStream]
name=AppStream Packages Rocky Linux 9
metadata_expire=-1
gpgcheck=1
enabled=1
baseurl=file:///mnt/AppStream/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
#Save with esc :wq or “shift + ZZ”

Question

Do the paths you’re using here make sense to you based off what you saw with the ls -l? Why or why not?

Answer

TODO

chmod 644 /etc/yum.repos.d/rocky9.repo
dnf clean all
# Test the local repository
dnf repolist
dnf --disablerepo="*" --enablerepo="AppStream" list available
Approximately how many are available?
dnf --disablerepo="*" --enablerepo="AppStream" list available | nl
dnf --disablerepo="*" --enablerepo="AppStream" list available | nl | head
dnf --disablerepo="*" --enablerepo="BaseOS" list available
Approximately how many are available?
dnf --disablerepo="*" --enablerepo="BaseOS" list available | nl
dnf --disablerepo="*" --enablerepo="BaseOS" list available | nl | head
# Try to install something
dnf --disablerepo="*" --enablerepo="BaseOS AppStream" install gimp
hit “n”

Question

How many packages does it want to install?

Answer

TODO

Question

How can you tell they’re from different repos?

Answer

TODO

# Share out the local repository for your internal systems (tested on just this one system)
rpm -qa | grep -i httpd
systemctl status httpd
ss -ntulp | grep 80
lsof -i :80
cd /etc/httpd/conf.d
vi repos.conf

<Directory "/mnt">
Options Indexes FollowSymLinks
AllowOverride None
Require all granted
</Directory>
Alias /repo /mnt
<Location /repo>
Options Indexes FollowSymLinks
AllowOverride None
Require all granted
</Location>

systemctl restart httpd
vi /etc/yum.repos.d/rocky9.repo

###USE YOUR HAMMER MACHINE IN BASEURL###
[BaseOS]
name=BaseOS Packages Rocky Linux 9
metadata_expire=-1
gpgcheck=1
enabled=1
#baseurl=file:///mnt/BaseOS/
baseurl=http://hammer25/repo/BaseOS/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
[AppStream]
name=AppStream Packages Rocky Linux 9
metadata_expire=-1
gpgcheck=1
enabled=1
#baseurl=file:///mnt/AppStream/
baseurl=http://hammer25/repo/AppStream/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release

Question

Do the paths you’ve modified at baseurl make sense to you? If not, what do you need to better understand?

dnf clean all
dnf repolist
# Try to install something
dnf --disablerepo="*" --enablerepo="BaseOS AppStream" install gimp

Digging Deeper

Question

You’ve set up a local repository and you’ve shared that repo out to other systems that might want to connect. Why might you need this if you’re going to fully air-gap systems? Is it still necessary even if your enterprise patching solution is well designed? Why or why not?

Answer

We need a unified checkpoint that ensures secure conformity of a package before patching air-gapped systems. Air-gapped systems are not eternally disconnected, they can be connected to other systems in a highly controlled manner.

Question

Can you add the Mellanox ISO that is included in the /lab_work/repos_and_patching section to be a repository that your systems can access? If you have trouble, troubleshoot and ask the group for help.

Answer

Yes you can, it must be given a special header and be registered aka. Packaged into the local repo in order for other package management systems to see it.

ProLUG Links ⛓️

Professional Linux User Group Security Engineering Unit 5 Web Book ProLUG, 2025. ↩︎
Rocky Documentation: Software Management Web Book Rocky Docs, 2025. ↩︎
Google Search Engine Web Search Engine, 2025. ↩︎
Epel Documentation Web Docs IBM, 2025. ↩︎

ProLUG SEC Unit 4 🔒

Mon, 21 Apr 2025 00:00:00 +0000

Intro 👋

Bastions and airgaps are strategies for controlling how systems connect—or don’t connect—to the outside world.¹

Worksheet

`Discussion Post 1`

https://aws.amazon.com/search/?searchQuery=air+gapped#facet_type=blogs&page=1

https://aws.amazon.com/blogs/security/tag/bastion-host/

Or find some on your own about air-gapped systems.

Question

What seems to be the theme of air-gapped systems?

Answer

Air gapped systems are highly controlled and isolated systems. The degree of isolation directly correlates to the level of operational burden as modern productive systems are typically highly connected to either LANs and/or WANs.

Blocking/Limiting/Bottlenecking Network Traffic
Limiting Services to Bare Essentials
Mitigating Data Egress
Quardening off un-expected behavior
Logging use events

Question

What seems to be their purpose?

Answer

To limit attack surface, mitigate malicious access and/or data infiltration/exfiltraion

Question

If you use google, or an AI, what are some of the common themes that come up when asked about air-gapped or bastion systems?

Answer

Common Themes in Air-Gapped Systems
- Data Transfer Procedures
- Patch Management & Updates
- Logging and Auditing
- Threat Models
- Authentication & Access
- Compliance & Certification
- Operational Burden
Common Themes in Bastion Hosts
- Network Segmentation
- Hardened OS Configuration
- Jump Host Architecture
- Access Control & MFA
- Monitoring and Alerting
- Change Management
Shared Themes
- Both require strict access control
- Emphasis on tamper resistance and detection
- Tradeoffs between security vs. usability
- Often part of zero-trust or defense-in-depth architectures

`Discussion Post 2`

Question

Do a Google or AI search of topics around jailing a user or processes in Linux.

Answer

User Jailing Techniques

chroot
Namespaces
Control groups (cgroups)
Seccomp
AppArmor / SELinux

Container and Jail Environments

LXC
Docker / Podman
Firejail
Bubblewrap (bwrap) Flatpak unpriveledged namespaces

Use Cases

Jailed SSH users: Using chroot in sshd_config to restrict access.
systemd-nspawn: Lightweight containers for sandboxed environments.
Flatpak / Snap: Sandboxed app delivery systems for desktop applications.

Related Tools & Commands

chroot, unshare, setfacl, auditd
firejail, bwrap, systemd-nspawn
docker, podman, lxc-start

Question

Can you enumerate the methods of jailing users?

Answer

Yes there are 5 possible avenues that I know of.

Question

Can you think of when you’ve been jailed as a Linux user? If not, can you think of the useful ways to use a jail?

Answer

No I have not experienced being jailed as a user. However, if I could think of some use-cases, perhaps one would be as a honeypot for observability. Another usecase I think could work would be to trap crawlers/bots.

Definitions

Air-gapped Air gapped means physically isolated from unsecured networks.
Bastion A bastion is a secure gateway between a trusted and untrusted network.
Jailed process A jailed process is restricted to a limited portion of the filesystem.
Isolation Isolation separates processes or systems to limit access and interaction.
Ingress The intake of data into a system.
Egress In the context of systems, having the ability.
Exfiltration When a bad actor ro program is able to extracted data from a system.
Cgroups Cgroups limit and monitor resource usage of Linux processes.
Namespaces isolate system resources for process groups.
Mount restricts filesystem views per process group.
PID isolates process ID numbers between groups.
IPC isolates inter-process communication resources.
UTS allows separate host and domain names.

Lab 🧪🥼

process of chroot jail build

1. Create a chroot in /var

mkdir /var/chroot

1. Copy in core Binaries from the system into chroot bin,lib64,dev,etc,home,usr/bin,lib/x86_64-linux-gnu

Question

What seems to be the theme of air-gapped systems?

Answer

Disconnected them from regular operational activities.

Question

What seems to be their purpose?

Answer

Reduce or eliminate the possibility of infiltration and exfiltrion.

Question

hat are some of the common themes that come up when asked about air-gapped or bastion systems?

Air Gapped

Isolation
Threat Mitigation
Data Transfer Control
Threat Mitigation
Update Challenges
Insider Threats
Bridging Attacks
Regulatory Compliance

Bastion Hosts

Single Point of Entry
Heavily Monitored
Hardened Configuration
Authentication Hub
Session Recording
Access Segregation
Zero Trust Integration
Threat Containment

ProLUG Links ⛓️

Professional Linux User Group Security Engineering Unit 4 Web Book ProLUG, 2025. ↩︎

ProLUG SEC Unit 3 🔒

Sun, 13 Apr 2025 00:00:00 +0000

Intro 👋

Understanding and implementing network standards and compliance measures can make security controls of critical importance very effective.¹

Worksheet

`Discussion Post 1`

There are 16 Stigs that involve PAM for RHEL 9².

Question

What are the mechanisms and how do they affect PAM functionality?

Answer

Hardening Defaults

STIGs replace permissive PAM modules with stricter ones. 2 Categories/Areas are covered in regards to Stig’ing PAMs

Lockout Policies that effect login frequency and failure.
Password Strength Enforcement that effects password complexity and re-use.

Review /etc/pam.d/sshd on a Linux system.

Question

What is happening in that file relative to these functionalities?

Answer

This file specifies the PAM module control flags that sshd uses during authentication.

Question

What are the common PAM modules?

Answer

pam_sepermit.so, pam_nologin.so, apassword-auth and postlogin.

Question

Look for a blog post or article about PAM that discusses real world application. ost it here and give us a quick synopsis.

Answer

https://www.redhat.com/en/blog/pluggable-authentication-modules-pam?utm_source=chatgpt.com

Synopsis:

PAM are a modular and flexible framework for integrating authentication methods into applications. By seperating / abstracting authentication mechanisms from application code, PAM allows admins to manage authentication policies centrally. PAM also allows from customized authentication processes (Security through obscurity)

`Discussion Post 2`

Intro to the scenario

Read about active directory (or LDAP) configurations of Linux via sssd³ 👍

Question

Why do we not want to just use local authentication in Linux? Or really any system?

Answer

Local authentication presents several problems. Firstly, there is no federated access, so there is fragmentation of systems. Secondly, scalability is an issue as each local system manages that local system’s users, requiring individual account provisioning and password management. Thirdly, it complicates auditing and compliance, since there is no centralized logging or consistent policy enforcement. Additionally, stale or orphaned accounts can accumulate unnoticed, increasing security risks. Finally, it prevents the implementation of modern security practices such as single sign-on (SSO), multi-factor authentication (MFA), and role-based access control across a distributed environment.

Question

There are 4 SSSD STIGS.

Response

Vuln ID 258122

Enforce Smart Card Authentication – Require certificate-based smart card login to implement multi-factor authentication and enhance access security.

Vuln ID 248131

Validate Certificate Chains – Ensure that certificates used for PKI-based authentication are properly validated by building a complete certification path to a trusted root.

Vuln ID 258132

Associate Certificates with User Accounts – Confirm that every authentication certificate is explicitly mapped to a valid user account to maintain identity integrity.

Vuln ID 258133

Restrict Credential Caching Duration – Limit the validity period of cached authentication credentials to a maximum of 24 hours to reduce risk in the event of compromise.

Definitions

PAM Pluggable Authentication Modules provide a flexible mechanism for authenticating users on Unix-like systems.
AD Active Directory is Microsoft’s centralized directory service for authentication, authorization, and resource management.
LDAP Lightweight Directory Access Protocol is an open, vendor-neutral protocol for accessing and maintaining distributed directory information services.
sssd System Security Services Daemon provides access to remote identity and authentication providers like LDAP or Kerberos.
oddjob A D-Bus service used to perform privileged tasks on behalf of unprivileged users, often for domain enrollment or home directory creation.
krb5 Kerberos 5 is a network authentication protocol that uses tickets for securely proving identity over untrusted networks.
realm/realmd A tool that simplifies joining and managing a system in a domain like Active Directory or IPA using standard services.
wheel (system group in RHEL):** A special administrative group whose members are allowed to execute privileged commands using sudo.

Lab

Examine STIG V-257986

Question

What is the problem?

Answer

RHEL 9 needs PAM enabled for SSHD

Question

What is the fix?

Answer

Enabling UsePAM in /etc/ssh/sshd/config

Question

What type of control is being implemented?

Answer

A Technical Preventative control

Question

Is it set properly on your system?

Answer

Yes it is

grep -i pam /etc/ssh/sshd_config

Question

Can you remediate this finding?

Check and remediate STIG V-258055

Questions

What is the problem?
What is the fix?
What type of control is being implemented?
Are there any major implications to think about with this change on your system? Why or why not?
Is it set properly on your system?
How would you go about remediating this on your system?

Answers

After 3 Unsuccessful root login attempts the root is locked.
Enable faillock in authselect + even_deny_root
Technical preventative
Yes, Anyone can be locked out including root
No, it is commented out by default for good reason
I would not enable even_deny_root

Check and remediate STIG V-258098

Questions

What is the problem?
What is the fix?
What type of control is being implemented?
Is it set properly on your system?

Answers

Password complexity module pwquality must be enabled in system-auth
Check `/etc/pam.d/system-auth and see if the line exists or not
Technical preventative control
Yes it is properly implemented

Filter STIGS by “password complexity”

Questions

How many are there?
What are the password complexity rules?
Are there any you haven’t seen before?

Answers

14 STIGS related to Password complexity
The somewhat standard 4 char class (One Upper, One Lower, One Special) and 15 Char total minimum. Max repeated char class is 4 and Max repeat char 3.
Yes the Max repeat stuff.

OpenLDAP Setup

You will likely not build an LDAP server in a real world environment. We are doing it for understanding and ability to complete the lab. In a normal corporate environment this is likely Active Directory.

To simplify some of the typing in this lab, there is a file located at /lab_work/identity_and_access_management.tar.gz that you can pull down to your system with the correct .ldif files.

[root@hammer1 ~]# cp /lab_work/identity_and_access_management.tar.gz .
[root@hammer1 ~]# tar -xzvf identity_and_access_management.tar

1. Stop the warewulf client

[root@hammer1 ~]# systemctl stop wwclient

2. Edit your /etc/hosts file

Look for and edit the line that has your current server

[root@hammer1 ~]# vi /etc/hosts

Entry for hammer1 for example:

192.168.200.151 hammer1 hammer1-default ldap.prolug.lan ldap

3. Setup dnf repo

[root@hammer1 ~]# dnf config-manager --set-enabled plus
[root@hammer1 ~]# dnf repolist
[root@hammer1 ~]# dnf -y install openldap-servers openldap-clients openldap

4. Start slapd systemctl

[root@hammer1 ~]# systemctl start slapd
[root@hammer1 ~]# ss -ntulp | grep slapd

5. Allow ldap through the firewall

[root@hammer1 ~]# firewall-cmd --add-service={ldap,ldaps} --permanent
[root@hammer1 ~]# firewall-cmd --reload
[root@hammer1 ~]# firewall-cmd --list-all

6. Generate a password (Our example uses testpassword) This will return a salted SSHA password. Save this password and stalted hash for later input

[root@hammer1 ~]# slappasswd

Output:

New password: Re-enter new password: {SSHA}wpRvODvIC/EPYf2GqHUlQMDdsFIW5yig

7. Change the password

[root@hammer1 ~]# vi changerootpass.ldif

dn: olcDatabase={0}config,cn=config
changetype: modify
replace: olcRootPW
olcRootPW: {SSHA}vKobSZO1HDGxp2OElzli/xfAzY4jSDMZ

[root@hammer1 ~]# ldapadd -Y EXTERNAL -H ldapi:/// -f changerootpass.ldif

Output:

SASL/EXTERNAL authentication started SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
modifying entry "olcDatabase={0}config,cn=config"

8. Generate basic schemas

ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/cosine.ldif
ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/nis.ldif
ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/inetorgperson.ldif

9. Set up the domain (USE THE PASSWORD YOU GENERATED EARLIER)

[root@hammer1 ~]# vi setdomain.ldif

dn: olcDatabase={1}monitor,cn=config
changetype: modify
replace: olcAccess
olcAccess: {0}to * by dn.base="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth"
read by dn.base="cn=Manager,dc=prolug,dc=lan" read by * none

dn: olcDatabase={2}mdb,cn=config
changetype: modify
replace: olcSuffix
olcSuffix: dc=prolug,dc=lan

dn: olcDatabase={2}mdb,cn=config
changetype: modify
replace: olcRootDN
olcRootDN: cn=Manager,dc=prolug,dc=lan

dn: olcDatabase={2}mdb,cn=config
changetype: modify
add: olcRootPW
olcRootPW: {SSHA}s4x6uAxcAPZN/4e3pGnU7UEIiADY0/Ob

dn: olcDatabase={2}mdb,cn=config
changetype: modify
add: olcAccess
olcAccess: {0}to attrs=userPassword,shadowLastChange by
dn="cn=Manager,dc=prolug,dc=lan" write by anonymous auth by self write by * none
olcAccess: {1}to dn.base="" by * read
olcAccess: {2}to * by dn="cn=Manager,dc=prolug,dc=lan" write by * read

10. Run it

[root@hammer1 ~]# ldapmodify -Y EXTERNAL -H ldapi:/// -f setdomain.ldif

Output:

SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
modifying entry "olcDatabase={1}monitor,cn=config
modifying entry "olcDatabase={2}mdb,cn=config
modifying entry "olcDatabase={2}mdb,cn=config
modifying entry "olcDatabase={2}mdb,cn=config
modifying entry "olcDatabase={2}mdb,cn=config

11. Search and verify the domain is working.

[root@hammer1 ~]# ldapsearch -H ldap:// -x -s base -b "" -LLL "namingContexts"

Output:

dn: namingContexts: dc=prolug,dc=lan

12. Add the base group and organization.

[root@hammer1 ~]# vi addou.ldif

dn: dc=prolug,dc=lan
objectClass: top
objectClass: dcObject
objectclass: organization
o: My prolug Organisation
dc: prolug

dn: cn=Manager,dc=prolug,dc=lan
objectClass: organizationalRole
cn: Manager
description: OpenLDAP Manager

dn: ou=People,dc=prolug,dc=lan
objectClass: organizationalUnit
ou: People

dn: ou=Group,dc=prolug,dc=lan
objectClass: organizationalUnit
ou: Group

[root@hammer1 ~]# ldapadd -x -D cn=Manager,dc=prolug,dc=lan -W -f addou.ldif

13. Verifying

[root@hammer1 ~]# ldapsearch -H ldap:// -x -s base -b "" -LLL "+" 
[root@hammer1 ~]# ldapsearch -x -b "dc=prolug,dc=lan" ou

14. Add a user

Generate a password (use testuser1234)

[root@hammer1 ~]# slappasswd

[root@hammer1 ~]# vi adduser.ldif

dn: uid=testuser,ou=People,dc=prolug,dc=lan
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: shadowAccount
cn: testuser
sn: temp
userPassword: {SSHA}yb6e0ICSdlZaMef3zizvysEzXRGozQOK
loginShell: /bin/bash
uidNumber: 15000
gidNumber: 15000
homeDirectory: /home/testuser
shadowLastChange: 0
shadowMax: 0
shadowWarning: 0

dn: cn=testuser,ou=Group,dc=prolug,dc=lan
objectClass: posixGroup
cn: testuser
gidNumber: 15000
memberUid: testuser

[root@hammer1 ~]# ldapadd -x -D cn=Manager,dc=prolug,dc=lan -W -f adduser.ldif

16. Verify that your user is in the system.

[root@hammer1 ~]# ldapsearch -x -b "ou=People,dc=prolug,dc=lan"

17. Secure the system with TLS (accept all defaults)

[root@hammer1 ~]# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/pki/tls/ldapserver.key -out /etc/pki/tls/ldapserver.crt
[root@hammer1 ~]# chown ldap:ldap /etc/pki/tls/{ldapserver.crt,ldapserver.key}

[root@hammer1 ~]# ls -l /etc/pki/tls/ldap*

Output:

-rw-r--r--. 1 ldap ldap 1224 Apr 12 18:23 /etc/pki/tls/ldapserver.crt -rw-------. 1 ldap ldap 1704 Apr 12 18:22 /etc/pki/tls/ldapserver.key

[root@hammer1 ~]# vi tls.ldif

dn: cn=config
changetype: modify
add: olcTLSCACertificateFile
olcTLSCACertificateFile: /etc/pki/tls/ldapserver.crt

add: olcTLSCertificateKeyFile
olcTLSCertificateKeyFile: /etc/pki/tls/ldapserver.key

add: olcTLSCertificateFile
olcTLSCertificateFile: /etc/pki/tls/ldapserver.crt

[root@hammer1 ~]# ldapadd -Y EXTERNAL -H ldapi:/// -f tls.ldif

18. Fix the /etc/openldap/ldap.conf to allow for certs

[root@hammer1 ~]# vi /etc/openldap/ldap.conf

#
# LDAP Defaults
#

# See ldap.conf(5) for details
# This file should be world readable but not world writable.

#BASE dc=example,dc=com
#URI ldap://ldap.example.com ldap://ldap-master.example.com:666

#SIZELIMIT 12
#TIMELIMIT 15
#DEREF never

# When no CA certificates are specified the Shared System Certificates
# are in use. In order to have these available along with the ones specified # by TLS_CACERTDIR one has to include them explicitly:

TLS_CACERT /etc/pki/tls/ldapserver.crt
TLS_REQCERT never

# System-wide Crypto Policies provide up to date cipher suite which should
# be used unless one needs a finer grinded selection of ciphers. Hence, the
# PROFILE=SYSTEM value represents the default behavior which is in place
# when no explicit setting is used. (see openssl-ciphers(1) for more info)
#TLS_CIPHER_SUITE PROFILE=SYSTEM

# Turning this off breaks GSSAPI used with krb5 when rdns = false
SASL_NOCANON on

[root@hammer1 ~]# systemctl restart slapd

SSSD Configuration and Realmd join to LDAP

SSSD can connect a server to a trusted LDAP system and authenticate users for access to local resources. You will likely do this during your career and it is a valuable skill to work with.

1. Install sssd, configure, and validate that the user is seen by the system

[root@hammer1 ~]# dnf install openldap-clients sssd sssd-ldap oddjob-mkhomedir authselect
[root@hammer1 ~]# authselect select sssd with-mkhomedir --force
[root@hammer1 ~]# systemctl enable --now oddjobd.service
[root@hammer1 ~]# systemctl status oddjobd.service

2. Uncomment and fix the lines in /etc/openldap/ldap.conf

[root@hammer1 ~]# vi /etc/openldap/ldap.conf

Output:

BASE dc=prolug,dc=lan URI ldap://ldap.ldap.lan/

3. Edit the sssd.conf file

[root@hammer1 ~]# vi /etc/sssd/sssd.conf

[domain/default]
id_provider = ldap
autofs_provider = ldap
auth_provider = ldap
chpass_provider = ldap
ldap_uri = ldap://ldap.prolug.lan/
ldap_search_base = dc=prolug,dc=lan
#ldap_id_use_start_tls = True
#ldap_tls_cacertdir = /etc/openldap/certs
cache_credentials = True
#ldap_tls_reqcert = allow

[sssd]
services = nss, pam, autofs
domains = default

[nss]
homedir_substring = /home

[root@hammer1 ~]# chmod 0600 /etc/sssd/sssd.conf
[root@hammer1 ~]# systemctl start sssd
[root@hammer1 ~]# systemctl status sssd

4. Validate that the user can be seen

[root@hammer1 ~]# id testuser

Output:

uid=15000(testuser) gid=15000 groups=15000

Please reboot the the lab machine when done.

ProLUG Links ⛓️

Professional Linux User Group Security Engineering Unit 3 Web Book ProLUG, 2025. ↩︎
Stigs that involve PAM for RHEL 9 [Webstie](https://docs.rockylinux.org/guides/security/pam/ Source, 2025. ↩︎
configurations of Linux via sssd Website Source, 2025. ↩︎

ProLUG SEC Unit 2 🔒

Sat, 05 Apr 2025 00:00:00 +0000

Intro 👋

This week covers more implementation of Secure Technical Implementation Guidelines and we look at LDAP (Light Directory Access Protocol) Installation and Setup. This unit also introduces foundational knowledge on analyzing, configuring, and hardening networking components using tools and frameworks like STIGs, OpenSCAP, and DNS configurations.

`Discussion Post 1`

Preface

There are 401 stigs for RHEL 9. If you filter in your stig viewer for sysctl there are 33 (mostly network focused), ssh - 39, and network - 58. Now there are some overlaps between those, but review them and answer these questions

Question 1. As systems engineers why are we focused on protecting the network portion of our server builds?

Answer

Most attacks come through the network
Misconfigured services can expose critical ports.
Data in transit is vulnerable without proper encryption and access control.
External exposure often increases the attack surface for things like brute-force attempts, malware injection, or unauthorized access.

Question 2. Why is it important to understand all the possible ingress points to our servers that exist?

Answer

Ingress points = potential paths of attack. Unexpected ingress can be exploited.
Zero-trust environments rely on strict control and observability of ingress.
Compliance and auditing require accurate records of what’s accessible.

Question 3. Why is it so important to understand the behaviors of processes that are connecting on those ingress points?

Answer

Security posture depends on visibility
Attackers scan for overlooked vulnerabilities
Automation tools (e.g., Ansible, Terraform) can introduce new ingress points unknowingly during updates.
Incident response is much faster and more effective when engineers understand the network surface.

`Discussion Post 2`

Intro to the scenario[^3]

Read this: https://ciq.com/blog/demystifying-and- troubleshooting-name-resolution-in-rocky-linux/ or similar blogs on DNS and host file configurations.

Question

What is the significance of the nsswitch.conf file?

Answer

The /etc/nsswitch.conf file controls the order in which name resolution methods are use

Question

What are security problems associated with DNS and common exploits? (May have to look into some more blogs or posts for this)

Answer

Core issues with DNS:

Traditional DNS can be spoofed due to a lack of built in verification., Queries and Responses are sent in plaintext making confidentiality an issue., No way to validate the source of the DNS data., Centralized, single point of failure.,

Common Exploits:

Spoofing (False record injection), Flooding (Overwhelming the resolver), Tunneling (Query based Exfiltration), Hijacking (Modifying domain registration data), Typosquatting (Registering similar domains) New phrase for me

Definitions

sysctl Linux interface to modify kernel parameters at runtime for system performance and security.
nsswitch.conf Configuration file controlling the order of name service lookups (e.g., DNS, files, LDAP).
DNS: Domain Name System translates human-readable domain names into IP addresses.
Openscap Open-source framework for automated vulnerability scanning, compliance checking, and security auditing.
CIS Benchmarks Prescriptive security configuration guidelines provided by the Center for Internet Security.
ss/netstat Command-line tools to display network sockets, connections, and statistics on Unix-like systems.
tcpdump Command-line packet analyzer for capturing and inspecting network traffic in real-time.
ngrep Network packet analyzer like grep, allowing pattern matching on network traffic payloads.

Lab 🧪

IP Forwarding

Question

Does this system appear to be set to forward? Why or why not?

Answer

No. All relevant net.ipv4.conf.*.forwarding and net.ipv4.ip_forward values are set to 0.

Martians

Question

What are martians and is this system allowing them?

Answer

Martians are packets with invalid or bogus source/destination addresses. This system is not logging them (log_martians = 0), but whether they’re allowed depends on other rules. Logging is disabled.

Kernel Panic Behavior

Question

How does this system handle kernel panics?

Answer

kernel.panic = 0 means the system won’t auto-reboot on panic. panic_on_oops = 1 indicates it will panic on kernel oops errors. Other panic triggers are mostly disabled.

FIPS Mode

Question

Is FIPS mode enabled?

Answer

No. crypto.fips_enabled = 0.

Question

What should be read about to better understand FIPS?

Answer

TODO

Kernel Command Line

Question

What are the active boot parameters from /proc/cmdline?

Answer

TODO (values include initrd paths, UUIDs, FIPS status not explicitly shown).

Security Settings & STIGs

V-257957 – TCP Syncookies

Question

Is the system using TCP syncookies?

Answer

Yes. net.ipv4.tcp_syncookies = 1.

Question

How to make this setting persistent?

Answer

Add net.ipv4.tcp_syncookies = 1 to a file in /etc/sysctl.d/, then run sysctl --system.

V-257958 – ICMP Redirects

Question

Is the system accepting ICMP redirect messages?

Answer

No. net.ipv4.conf.all.accept_redirects = 0

Question

How to harden this?

Answer

Add net.ipv4.conf.all.accept_redirects = 0 to /etc/sysctl.d/, then reload settings with sysctl --system.

Question

Did you fully understand all parameter meanings?

Answer

No. Some were clarified using ChatGPT.

ProLUG Links ⛓️

ProLUG SEC Intro 🔒

Sun, 23 Mar 2025 00:00:00 +0000

Intro 👋

I’ve just started a new Security Engineering course created by Scott Champine through ProLUG. As a graduate of his Linux Administration course and an active contributor to the Professional Linux User Group, I felt compelled to make time for this new course—I’ve learned a great deal from his teachings in the past.

The Course

This is a deep dive into Enterprise Operational Security. That includes topics like compliance, threat management, and system integrity. I’m also helping coordinate and develop a web-book to accompany the course.¹

While I already hold several cybersecurity certifications that cover conceptual frameworks and best practices, this course goes much deeper with hands-on labs. We harden systems with STIGs,² monitor and detect activity on live systems, and troubleshoot compliance issues.

The course spans 10 weeks, with an estimated 100 hours of work to complete the weekly projects and the capstone.

ProLUG Links ⛓️

Footnotes

ProLUG Security Engineering Course Web-Book Web-Book ProLUG, 2025. ↩︎
Secure Technical Implementation Guidelines DoD Cyber Exchange Website ↩︎

ProLUG SEC Unit 1 🔒

Sun, 23 Mar 2025 00:00:00 +0000

Intro 👋

Worksheet

`Discussion Post 1`

Question

What is Security?

Answer

In regards to Cyber Security, Integrating protective measures throughout the system lifecycle to ensure the system maintains its mission/operational effectiveness, even in the presence of adversarial threats.

Question

Describe the CIA Triad.

Answer

The CIA Triad is a core model in systems security engineering.

Confidentiality – Preventing unauthorized disclosure of system data or resources, often enforced through access control, encryption, and information flow policies.
Integrity – Ensuring that system data and operations are not altered in an unauthorized or undetected way, including protection against both accidental and intentional modification.
Availability – Ensuring reliable access to system services and resources when required, even under attack or component failure.

Question

What is the relationship between Authority, Will, and Force as they relate to security?

Answer

In systems security engineering:

Authority is derived from policy and design requirements—what the system must enforce according to mission objectives, laws, or standards.
Will represents the commitment of system stakeholders to implement and maintain security measures.
Force is the application of engineered mechanisms—technical, administrative, or procedural—that ensure security objectives are realized in practice.

Question

What are the types of controls and how do they relate to the above question?

Answer

In systems security engineering, controls are safeguards built into the system to achieve security objectives. They align with Authority, Will, and Force as follows:

Administrative Controls – Derived from organizational policy (Authority) and guide design, personnel roles, and security governance.
Technical Controls – Engineered into the system as part of architecture and software/hardware features (Force), e.g., encryption, access enforcement, secure boot.
Operational Controls – Rely on human procedures and configurations to maintain secure operations (Will and Force), such as patch management and monitoring.
Physical Controls – Provide physical protection to system components (Force), e.g., secure facilities or tamper-evident hardware.

`Discussion Post 2`

Intro to the scenario[^3]

Find a STIG or compliance requirement that you do not agree is necessary for a server or service build.

Question

What is the STIG or compliance requirement trying to do?

Answer The compliance requirement encourages users to set up automated CVE patch updates from trusted providers within a 24-hour timeframe.

Question

What category and type of control is it?

Answer
This STIG is an administrative control. Since it is not built into the system by default, it must be applied and managed manually.

Question

Defend why you think it is not necessary. (What type of defenses do you think you could present?

Answer Initially, I found it difficult to identify a STIG procedural that I disagreed with. However, after extensive review, I selected this one. I believe automated patching is not ideal, especially for production systems. Patches can introduce unexpected behaviors in dependent systems. Additionally, relying on automation can foster complacency or a lack of awareness over time.

STIG


Apache Server 2.4 UNIX Server Security Technical Implementation Guide :: Version 3, Release: 2 Benchmark Date: 30 Jan 2025
Vul ID: V-214270 Rule ID: SV-214270r961683_rule STIG ID: AS24-U1-000930
Severity: CAT II Classification: Unclass Legacy IDs: V-92749; SV-102837
Group Title: SRG-APP-000456-WSR-000187
Rule Title: The Apache web server must install security-relevant software updates within the configured time period directed by an authoritative source (e.g., IAVM, CTOs, DTMs, and STIGs).
Discussion: Security flaws with software applications are discovered daily. Vendors are constantly updating and patching their products to address newly discovered security vulnerabilities. Organizations (including any contractor to the organization) are required to promptly install security-relevant software updates (e.g., patches, service packs, and hot fixes). Flaws discovered during security assessments, continuous monitoring, incident response activities, or information system error handling must also be addressed expeditiously.
The Apache web server will be configured to check for and install security-relevant software updates from an authoritative source within an identified time period from the availability of the update. By default, this time period will be every 24 hours.
Check Text: Determine the most recent patch level of the Apache Web Server 2.4 software, as posted on the Apache HTTP Server Project website. If the Apache installation is a proprietary installation supporting an application and is supported by a vendor, determine the most recent patch level of the vendor’s installation.
In a command line, type "httpd -v".
If the version is more than one version behind the most recent patch level, this is a finding.

Definitions

CIA Triad Core principles of security — Confidentiality, Integrity, and Availability — guiding protection of information systems.
Regulatory Compliance Adhering to laws and regulations governing data privacy, security, and operational practices.
HIPAA U.S. healthcare regulation enforcing privacy and security of patient health information.
Industry Standards Best practices and technical guidelines agreed upon within specific industries to ensure consistency and security.
PCI/DSS Payment Card Industry Data Security Standard for protecting cardholder data during processing, storage, and transmission.
Security Frameworks Structured guidelines for managing cybersecurity risks and controls within organizations.
CIS Center for Internet Security provides globally recognized benchmarks and security controls.
STIG Security Technical Implementation Guide — DoD configuration standards for securing IT systems and software.

Lab 🧪🥼

MariaDB STIG Remediation Lab – Q&A Format

Signing into Remote Host

Question

How do you connect to the remote host?

Answer

ssh mchammer@prolug.asuscomm.com with password SecLab12#$5

Initial Setup

Question

What tools should be installed?

Answer

tmux and vim via dnf install tmux vim -y

Question

Why use tmux?

Answer

To persist sessions (e.g., for using nohup).

Installing and Verifying MariaDB

Question

How is MariaDB installed and started?

Answer

dnf install mariadb-server, then start and verify with:

systemctl start mariadb
systemctl status mariadb
ss -ntulp | grep 3306

Question

How do you enter the MariaDB shell?

Answer

Run mariadb.

V-253666: Listing Users & Max Connections

Question

How do you view users and max connections?

Answer

Run:

SELECT DISTINCT user FROM mysql.user;
SELECT user, max_user_connections FROM mysql.user;

V-253677: Shutdown on Audit Failure

Question

What is the issue?

Answer

MariaDB must shut down or alert on audit failures like lack of space.

Question

What is the fix?

Answer

Configure alerting when log space is low using the OS or DB logging tools.

Question

What type of control is this?

Answer

Technical, detective control.

Question

Is it set on your system?

Answer

Not by default. Logging rollover may need to be configured manually.

V-253678: FIFO Audit Logging

Question

What is the issue?

Answer

MariaDB must overwrite oldest audit logs when storage is full (FIFO).

Question

What is the fix?

Answer

Use syslog or configure log rotation/purging. Example configuration:
```
[mariadb]
server_audit_output_type = 'syslog'
```

Question

What type of control is this?

Answer

Technical, detective control.

Question

Is it set on your system?

Answer

TODO

V-253754: Audit on Security Object Change

Question

What is the issue?

Answer

Changes to roles, privileges, and security objects must be audited.

Question

What is the fix?

Answer

Configure MariaDB Audit Plugin with the following:

DELETE FROM mysql.server_audit_filters WHERE filtername = 'default';
INSERT INTO mysql.server_audit_filters (filtername, rule)
VALUES ('default',
 JSON_COMPACT(
 '{
 "connect_event": ["CONNECT", "DISCONNECT"],
 "query_event": ["ALL"]
 }'
 ));

Question

What type of control is this?

Answer

Technical, detective control.

Question

Is it set on your system?

Answer

Not by default, but was configured on the remote server.

My 2025 Learning Plan

Fri, 07 Feb 2025 00:00:00 +0000

Where to start

I am a generally curious person who enjoys learning new things. However, the shear volume of information available can lead to feeling overwhelmed, distracted and aimless. From years of experience being an auto-didact, I have honed my craft of self directed study. Now, I create a solid learning plan that keeps me on track and feeling a sense of achievement. A learning plan is a personal roadmap that outlines what to learn, how to learn it and when to reach certain milestones. I start with goals and work my way backwards from there.

2025

If I am to state my purpose simply Will be the year I:

Get advanced with GO programming by writing several useful programs.
Learn Kubernenets and prove that knowledge with a Certification
Get comfy with Ansible
Get JLPT N5 Certified in Japanese Language

Footnotes

Embedded Rust? 🦀

Sun, 12 Jan 2025 00:00:00 +0000

Yes

Firstly Rust on Embedded is a different beast as the standard library is not used and memory safety is not on by default. However, there are still some advantages over a popular language like C or C++. HAL or Hardware Abstraction Layer separates the hardware from the code enabling more portable software that can compile to multiple architectures. Cargo improves development ergonomic by creating and managing the project and its dependancies. Thirdly, the build system is unified across platforms, so code will compile on Windows, Mac and Linux in the same way. Rust on embedded systems is a different challenge, as it does not use the standard library, and memory safety is not enabled by default. However, it still offers several advantages over popular languages like C or C++. One key benefit is the Hardware Abstraction Layer (HAL), which separates hardware-specific details from the code, enabling more portable software that can compile across multiple architectures. Additionally, Cargo enhances development ergonomics by simplifying project and dependency management. Lastly, Rust’s unified build system ensures consistent behavior across platforms, allowing code to compile seamlessly on Windows, macOS, and Linux.

HAL 🪢

Is really interesting, it is the concept of mapping hardware and storing the map for API access. HAL leverages Peripheral Access Crates (PACs), which are auto-generated Rust crates representing the registers and bitfields of a microcontroller. PACs allow safe and direct access to hardware registers while ensuring Rust’s strict type-checking and ownership rules are followed. HAL sits on top of PACs, abstracting these low-level details. Rust embedded HALs adhere to the embedded-hal traits—a collection of interfaces defining common operations like GPIO pin control, SPI/I2C communication, timers, and ADC usage. By implementing these traits, HAL provides a uniform way to interact with hardware, regardless of the underlying platform. HAL abstracts device-specific features into a user-friendly API. For example:

Configuring a GPIO pin involves selecting its mode (input, output, pull-up, etc.) without directly modifying hardware registers.
Communication protocols like SPI or I2C are exposed through easy-to-use Rust methods (read, write, transfer, etc.).

Cargo 📦

Cargo handles dependencies seamlessly using Cargo.toml. Developers specify libraries (called “crates”) with version constraints, and Cargo fetches and builds them automatically. Cargo:

Ensures reproducible builds by generating a Cargo.lock file that locks dependency versions.
Community-driven ecosystem (e.g., crates.io) simplifies finding and using high-quality, maintained libraries.

Managing Dependancies ⚙️ with Cargo.toml

[dependencies]
embedded-hal = "0.2.6"
stm32f4xx-hal = "0.14"

Cross-compilation support is integrated via targets 🎯

cargo build --target thumbv7em-none-eabihf

Enforced project Structure 👮‍♂️

my_project/
├── Cargo.toml # Dependencies & configuration
└── src/
 └── main.rs # Application entry point

Cross Platform 💻 💼

Tools like probe-rs allow seamless debugging and flashing of embedded devices on multiple platforms (Linux, macOS, Windows).
The cargo ecosystem integrates testing, building, and dependency management across platforms without additional tools.

Experienced Embedded Devs are switching

I was first made aware of Rust for embedded by experienced devs on Youtube who proclaimed their love of rust over C for professional projects. Channels like: https://www.youtube.com/@therustybits https://www.youtube.com/@JaJakubYT https://www.youtube.com/@floodplainnl https://www.youtube.com/@embedded-rust

are all claiming that they have switched over and are not looking back.

Personal Development

I’m writing this post as I plan to develop hardware projects using Rust for embedded systems. The combination of Rust and RISC-V microcontrollers is a particularly exciting intersection. In my sights are the ESP32-C3 and Raspberry Pi Pico 2, both of which I’m considering for upcoming projects. Instead of dealing with messy C, slow MicroPython, or the limitations of TinyGo, Rust allows me to create clean and performant projects—something I always strive for. Stay tuned for more updates!

ProLUG Talk: Kubernetes in the Enterprise

Sat, 11 Jan 2025 00:00:00 +0000

Kubernetes in Enterprise

As a dedicated member of the Professional Linux User Group, I gain valuable insights into essential industry tools, processes, and procedures from professional engineers who work hands-on with major infrastructure.

This evening, Michael Pesa of Lambda Labs delivered an excellent talk on best practices with Kubernetes and GitOps, shedding light on the challenges faced by traditional orchestration approaches. What intrigued me most was the discussion on Talos OS and Chainguard, particularly their use of Software Bill of Materials (SBOM). The concept centers around stripping systems down to their bare essentials, which not only reduces vulnerabilities but also improves performance.

Talos OS is particularly fascinating because it eliminates many traditional system components like SSH, systemd, glibc, package managers, or a shell. Essentially, Talos is just the Linux kernel with several Go binaries. This streamlined approach significantly reduces vulnerabilities and minimizes the attack surface. As Michael mentioned in his presentation, many vulnerabilities stem from privilege escalation, container escapes, and memory hacking. Talos mitigates most of these threats by enforcing API-driven controls instead of relying on a shell and by utilizing private key-based authentication throughout.

I am excited to experiment with these tools in my homelab, where I aim to create a modern, declarative infrastructure with ephemerality at its core.

📝 Notes from the Presentation:

Topic Covered

Immutable operating systems

Minimalist container images

GitOps strategies

Reproducible builds

SUSE MicroOS 🦎

Purpose:

Designed as a container host OS.

Features:

Read-only root filesystem for enhanced security.
Transactional updates managed via Btrfs snapshots.
Cloud environment integration with cloud-init; uses ignition elsewhere.

Usage:

Ideal for minimal, secure containerized workloads.

Talos Linux 🦅

Overview:

Minimalist design focusing on immutability.
Linux kernel paired with five Go binaries.
Lacks traditional components: no SSH, systemd, glibc, package manager, or shell.

Security:

Secure by default with a micro TLS stack and key-based authentication.
API-driven operations using YAML, akin to Kubernetes manifests.
Uses its own PKI infrastructure, creating a small attack surface.

Tools:

talosctl CLI for management.
Debugging occurs via ephemeral tools and remote APIs.

Considerations:

Best suited for Kubernetes clusters.
For bare-metal installations, use Matchbox or a PXE boot system.

Alternatives:

Flatcar and CoreOS are earlier container-focused OS derivatives.

Minimal Container Images 👝

Philosophy:

Containers are inherently immutable; minimal images reduce attack surfaces and improve performance.

Best Practices:

Use the smallest possible image, minimizing unnecessary dependencies.
Employ multi-stage builds for languages with heavy dependencies, separating build and runtime environments.
For statically compiled languages like Go, containers can often be reduced to a single binary.

Security:

Avoid including shells to reduce exploit vectors.
Favor minimal base images like Alpine or tools like Google’s Distroless and Chainguard’s SBOM-integrated containers.

Supply Chain Security 🔗🔐

SBOM (Software Bill of Materials):

A detailed list of libraries, packages, versions, and licenses within a container image.
Machine-readable format for automated security checks.

Software Attestation:

Authenticated metadata about an artifact (e.g., a container image).
Enables verification of the artifact’s integrity.

Chainguard:

Offers streamlined, dependency-minimized packages.
Main drawback: constant updates unless on a paid plan.

Challenges in Immutable Environments ♻️

Limitations:

Lack of SSH access and debugging tools on nodes and pods.
Ephemeral, short-lived infrastructure and read-only filesystems.

Risks:

IaC misconfigurations can cause widespread outages.

Mitigation:

Implement proper GitOps workflows.
Ensure consistency between development, staging, and production environments.

Observability Strategies in Immutable Environments 👀

Approaches:

Centralized logging for better insights.
Monitor clusters and services rather than individual pods or nodes.
Use ephemeral debugging pods or sidecars with shared access.
Declarative observability configurations (e.g., using Kubernetes CRDs).

Principle:

Treat nodes and pods as “cattle, not pets” to maintain scalability and consistency.

Key Principles of GitOps and Declarative Infrastructure 😰

GitOps Core Tenets:

Git as the single source of truth.
Automated reconciliation with tools like ArgoCD or Flux.
Infrastructure changes are auditable and reversible.
Prevents unauthorized and ad-hoc changes.

Homelab Use:

Start with a Talos system in Docker/Podman for experimentation.

GitOps Challenges 😰

Common Issues:

Subtle configuration drift across environments.
Risk of manual changes not being committed back to Git.
Managing rollouts from dev to staging to production.
Scaling to hundreds of clusters.
Variations in deployment strategies across teams.
Securely managing secrets outside Git (e.g., Kubernetes Secrets, HashiCorp Vault).

The GitFlow Workflow 💨

Workflow Overview:

Developers create long-lived feature branches.
Separate primary branches for development, hotfixes, and releases.
The trunk (dev) branch isn’t always stable or deployable.
Multi-environment deployments often use separate release branches.

Complexity:

Complex merging strategies, especially for hotfixes.
Tools like ArgoCD or Flux can automate deployments.

Trunk-Based Development 🐘

Overview:

Continuous integration with short-lived feature branches.
Frequent commits directly to the main branch.
Emphasizes small, incremental changes to reduce merge conflicts.

Advantages:

Simplifies CI/CD pipelines.
Encourages fast feedback and rapid delivery.

Tools:

Works seamlessly with automated systems like Kubernetes or Terraform.

ProLUG Links ⛓️

Discord: https://discord.com/invite/m6VPPD9usw Youtube: https://www.youtube.com/@het_tanis8213 Twitch: https://www.twitch.tv/het_tanis ProLUG Book: https://leanpub.com/theprolugbigbookoflabs KillerCoda: https://killercoda.com/het-tanis

Why K8S matters

Sun, 29 Dec 2024 00:00:00 +0000

Kubernetes is Important 🌐🐳

Despite the jokes or criticisms you may have heard, Kubernetes matters a lot. Once I understood the “Why,” I became much more motivated to learn the “How.” This is how I got started with Kubernetes using my Proxmox Home-Lab and K3S.

Firstly, I would like to illuminate the “Why,” as it’s an important philosophy to grasp before diving in. I firmly believe in understanding the “Why” before the “How.” 🧠

A Brief History of Infrastructure 🕰️

The Evolution 💾

Many moons ago 🌛, internet infrastructure relied on operating systems to run services. Unix and Linux were preferred because they are multi-user environments suitable for serving files to requesters. In fact, the first implementation of TCP was written on a UNIX system running on a NextSTEP computer. 🖥️ This model worked for decades: more users meant more machines. However, issues arose. Machines would go down, causing cascading effects. ⚠️ Misconfigurations wreaked havoc, and machines often ran inefficiently, either wasting resources or straining hardware. 🔧

In Comes Virtualization 💻✨

Virtualization revolutionized infrastructure by allowing computers to be divided into independent virtual machines (VMs). A VM is a fully self-contained operating system. This innovation enabled more efficient utilization of hardware, increased flexibility, and reduced downtime caused by hardware failures. 🚀

Containers Changed the Game 🐳📦

Containers brought another layer of efficiency and standardization. Unlike VMs, containers share the host operating system’s kernel but encapsulate applications and their dependencies. This reduces overhead and enables applications to run consistently across different environments. 🌍 Developers could now “build once, run anywhere,” making containers a key tool in modern infrastructure. 🛠️

Orchestration Was Needed 🤖🎛️

As the use of containers exploded, managing them became increasingly complex. Deploying, scaling, monitoring, and maintaining hundreds or thousands of containers manually was impractical. This is where orchestration tools like Kubernetes stepped in, automating these tasks and ensuring applications are always running, balanced, and recoverable in case of failures. ✅

The Why 🤔

Understanding the “Why” is key to appreciating Kubernetes’ value. Here are some of the core reasons:

Monitoring 📊: Kubernetes provides tools and integrations to monitor your workloads, ensuring you can observe application health and performance in real time.
Logging 📝: Centralized logging in Kubernetes makes it easy to trace and debug issues across distributed systems.
Security 🔒: Kubernetes enhances security through role-based access control (RBAC), network policies, and automatic updates, reducing vulnerabilities in production systems.
Ephemerality 🌀: Kubernetes embraces the concept of ephemeral workloads, where containers can be replaced automatically if they fail, ensuring high availability.
Reproducibility 🔄: Kubernetes enables reproducible deployments by using declarative configurations, allowing you to deploy the same infrastructure consistently across environments.

By addressing these challenges, Kubernetes transforms the way infrastructure is managed and applications are deployed, making it a cornerstone of modern cloud-native computing. ☁️

During Week 10 of the ProLUG Course 1 🛠️📚

John Champine², an OpenShift Engineer, delivered a compelling two-hour presentation on Kubernetes and OpenShift. His anecdotes and technical insights were especially engaging, offering both rich historical context drawn from his personal experience and intricate details about shared resource management. 🖥️⚙️

My Experience 💻🔧

I heavily utilize Proxmox VE to build out simulated production environments where I can practice various administrative and engineering tasks. In my homelab, I installed K3S and Talos to create a typical dev/testing/production environment. 🌐 One particularly unique workflow I used involved building custom Podman containers—yes, Podman! 🐋

It’s not widely known that podman play kube ³ can create a manifest for use in Kubernetes pods. With this method, I could prototype and build out containers, functionally test them, and then publish them for declarative deployment. This approach felt incredibly slick to me, as bugs were ironed out during the process, and the final deployment was straightforward. ✅🛠️

Most of my work was done with K3S⁴, a Rancher⁵-based distro designed for low-resource environments. However, I also experimented with Talos OS, setting up multiple virtual machines in a configuration resembling a multi-machine/node environment—with a sprinkle of jank to keep things interesting. 🤖✨

This hands-on approach allowed me to deepen my understanding of Kubernetes while also refining workflows that integrate containerization and orchestration. 🚀

Respect 🙌

From these experiences, I have developed a deep respect for Kubernetes. I see it as the operating system of the internet—an innovation that will inspire other similar systems. 🌐 While newer technologies like MicroVMs and hybrid container/VM architectures are emerging, I believe they can be easily incorporated into orchestration schemes like Kubernetes. 🤖🛠️

Given this perspective, I think Kubernetes will remain relevant for a long time, much like Unix/Linux. 🐧 It simply makes sense given the strenuous demands of the modern internet and the ever-growing number of attacks and incidents. Such a resilient system enables greater efficiency, enhanced security, and improved situational awareness for everyone. 🔒🚀

Footnotes

The above quote is excerpted from an earlier BlogPost Post techblog, 2024. ↩︎
John Champine Profile LinkedIn, 2024. ↩︎
Podman Export Docs Podman Docs, 2019. ↩︎
K3S Website Site Site, 2025. ↩︎
Rancher Website Site Site, 2025. ↩︎

ProLUG Admin Course Unit 16 🐧

Sat, 28 Dec 2024 00:00:00 +0000

Incident Response

Incident response is a structured approach to identifying, managing, and resolving unexpected events such as security breaches, system failures, or misconfigurations. It aims to minimize disruption, mitigate damage, and restore normal operations while implementing lessons learned to prevent future incidents.

Responding to incidents is a stressful event because it can involve many stakeholders and little time. This week we exercised our skills by live debugging in front of our peers on a remote host. The problems all related to failure modes and misconfiguration and the exercise was rewarding in that I learned a lot as always, and built some confidence.

Incident Response / Troubleshooting Scenarios 🧑‍💻

Scenario #1: Web Server Not Running 🕸️

Objective: Ensure the web server is running and responding on port 80.
Steps:

Verify web server service:
- Run: systemctl enable --now httpd (or similar command).
Check open ports:
- Run: ss -ntulp.
- Identify if the server is running on port 8087 instead of 80.
- Edit the configuration:
  - File: /etc/httpd/conf/httpd.conf.
  - Change Listen 8087 to Listen 80.
- Restart the service: systemctl restart httpd.
Ensure external connectivity:
- Check the firewall status: systemctl status firewalld.
- Options:
  - Disable the firewall: systemctl stop firewalld.
  - Open port 80 if needed.
Final step:
- REBOOT the lab machine.

Scenario #2: Mount Point /space Not Working 💾

Objective: Set up a 9GB partition on the /space mount point using LVM.
Steps:

Verify /space setup:
- Confirm the partition is not properly set up.
Create LVM setup:
- Identify disks: fdisk -l | grep -i xvd.
- Create physical volumes: pvcreate /dev/xvd<disk>.
- Create a volume group:
  - Run: vgcreate space /dev/xvd<disk1> /dev/xvd<disk2> /dev/xvd<disk3>.
- Create a logical volume:
  - Run: lvcreate -n space -l +100%FREE space.
Format the logical volume:
- Create a filesystem: mkfs.ext4 /dev/mapper/<logical_volume_name>.
Mount the filesystem:
- Create the mount point: mkdir /space.
- Add an entry in /etc/fstab:
```
/dev/mapper/<logical_volume_name> /space <ext4 or xfs> defaults 1 2
```
- Mount the filesystem: mount -a.
Final step:
- REBOOT the lab machine.

Scenario #3: System Not Updating 📦

Objective: Fix the system to allow updates via dnf and ensure kernel updates.
Steps:

Fix DNF repository configuration:
- Edit /etc/yum.repos.d/rocky.repo:
  - Set enabled=1 for all necessary repositories.
- Check /etc/yum.repos.d/rocky.repo.orig for reference.
- Fix the EPEL repository the same way.
Verify kernel updates:
- Edit /etc/yum.conf:
  - Comment out the line: exclude=kernel*.
Final step:
- REBOOT the lab machine if necessary.

ProLUG Links ⛓️

Wrapping up ProLUC PAC

Sat, 28 Dec 2024 00:00:00 +0000

The Wrap 📦

Yesterday, our close-knit group completed an intensive 16-week hands-on course in Enterprise Linux Administration, culminating in a live incident response session.

Over the course of hundreds of hours, I pushed myself to go above and beyond in my studies and responsibilities. Along the way, I formed strong connections with like-minded peers, navigating the modern educational landscape of YouTube, Discord, and KillerCoda.

I am truly grateful to have stumbled upon this seemingly random community and to have experienced the structured, effective teaching methods of Scott Champine (Het Tanis), an experienced and traditional educator.

Discord 💬

To understand the environment, we must first understand the platform. Discord is a communication platform that combines text, voice, and video chat, designed to create communities where people can interact in real-time. What makes it unique is its seamless integration of customizable servers, topic-specific channels, and robust tools for both casual conversation and collaborative work.

Working on discord harbored a comfortable sense of passive interaction. Unlike say Zoom, Skype or any other similar video communication platform, Discord allows for people to come and go as they please, have multiple presenters and open voice chat, replicating a real world meeting more closely.

This allowed for impromptu discussions / presentation, greatly improving the learning experience.

Study group 🏘️

Early in the course, I applied my leadership skills by organizing a formal schedule for our study group meetings. These sessions covered course assignments in detail while also exploring related topics through collaborative, interactive projects.

The format was casual and engaging. I would share my screen to walk through scenarios while the group discussed the subject matter. Others also shared their screens, demonstrating tips and tricks in unison.

One of the most effective tools I introduced was a shared note using Etherpad. Similar to Google Docs, Etherpad allows multiple people to edit a document simultaneously. However, it stands out by enabling access without requiring sign-in credentials, making it easy to share with anyone.

These activities relied heavily on trust, as it would have been easy for someone to disrupt the sessions. My leadership skills were frequently tested by off-topic individuals or disruptive participants, but such issues were usually short-lived.

Gains 💪🏻

Coming into the course, I already had a solid understanding of Linux, backed by a few years of experience. Additionally, I had completed RWXRob’s (Rob Muhlestein’s) Beginner Boost DevOps course a year prior.

What set this course apart was its group-learning dynamic. During Rob’s course, I worked alone, building projects and debugging through hard-fought, self-directed methods like reading documentation, brute-forcing solutions, and referencing forums. In contrast, group work brought added motivation, inspiration, and a collaborative approach to problem-solving. It helped eliminate mundane, off-topic roadblocks, allowing us to focus on core learning and progress more efficiently.

Connections ⛓️

Through the study group and community discussions, I’ve developed a strong connection with the ProLUG community and feel confident that I can rely on the server for discussions, questions, and troubleshooting. In the near future, I plan to give back by supporting future coursework and helping new learners navigate their journey.

Thankful 🙏

I’m deeply grateful to Scott Champine (Het Tanis) for offering this free course and dedicating so much of his time to it. I’m equally thankful to the server members who joined the study group and dove headfirst into the intricacies of systems.

Until next time! ✌️

ProLUG Links ⛓️

ProLUG Admin Course Unit 15 🐧

Wed, 25 Dec 2024 00:00:00 +0000

Engineering Troubleshooting

Systems engineering troubleshooting involves diagnosing and resolving complex issues within interconnected systems to ensure seamless operation and optimal performance. It requires a methodical approach to identify root causes, integrate solutions, and maintain system functionality while addressing both technical and process-related challenges.

Discussion Post 1:

Your management is all fired up about implementing some Six Sigma processes around the company. You decide to familiarize yourself and get some basic understanding to pass along to your team.

Quoted from the book

5S is a Japanese Lean approach to organizing a workspace, so that by making a process more effective and efficient, it will become easier to identify and expunge muda. 5S relies on visual cues and a clean work area to enhance efficiencies, reduce accidents, and standardize workflows to reduce defects. The method is based on five steps:

Sort (Seiri)
Straighten (Seiton)
Shine (Seiso)
Standardize (Seiketsu)
Sustain (Shitsuke)

What about the “5S” methodology might help us as a team of system administrators?

Identify and categorize common troubleshooting problems such as typos, illogical configurations, or vulnerabilities to establish clarity and prioritize issues. (Seiri)
Organize and catalog processes and procedures for addressing both routine and uncommon problem scenarios for quick access and consistency. (Seiton)
Validate and test all processes and procedures to ensure they function effectively and reliably. (Seiso)
Promote team familiarity by regularly practicing and drilling procedures, similar to incident response training, to build confidence and efficiency. (Seiketsu)
Apply the processes and procedures in real-world scenarios, evaluate their effectiveness, make necessary adjustments, and document improvements for future use. (Shitsuke)

By applying the 5S methodology to troubleshooting, the team can develop a shared understanding of how to consistently address issues, identify system failure points, and create opportunities for incremental improvement, fostering a sense of flow and efficiency.

What are the four layers of process definition? How would you explain them to your junior engineers?

input - anything entering the process or is required to enter the process to drive the creation of an output outputs - service or product that is created by this process events - predefined criteria or actions that cause a process to begin working. tasks - activities are the heart. a unit of action within the process. 4.1 - Decisions are possible made during or for tasks.

The four layers of a process—inputs, outputs, events, and tasks—function similarly to how a computer program uses functions to process variables and produce results. However, in a Six Sigma process, these elements are more dynamic, encompassing both virtual and physical components, as well as steps that may be driven by either human actions or automated systems. This flexibility allows Six Sigma to address complex workflows that combine diverse inputs and tasks to achieve consistent and efficient outputs.

Looking at our operation as a series of processes with layers like this can help us to identify, refine and standardize processes into Standard Operating Procedures (SOP’s).

Discussion Post 2: Your team looks at a lot of visual data. You decide to write up an explanation for them to explain what they look at.

What is a high water mark? Why might it be good to know in utilization of systems?

The phrase “high water mark” originates from marking a riverbank to indicate the highest water level reached during a season. This mark serves as a warning, signaling potential danger if the water rises beyond it in the future. In the context of systems, the high water mark represents historically safe operational loads. If metrics indicate that this threshold has been exceeded, it should alert administrators to a potential issue. For example, if the high water mark for daily memory usage was 14/18 GiB of RAM, and we observe the system suddenly using 16/16 GiB, this warrants attention as a potential problem requiring further investigation.

What is an upper and lower control limit in a system output? While this isn’t exactly what we’re looking at, why might it be good to explain to your junior engineers?

Control limits are essential tools for monitoring the stability of a process over time. The upper and lower control limits define the normal range within which a process output should remain when the process is operating correctly. If the output exceeds these boundaries, it signals a potential issue, indicating that the process may be out of control and requiring investigation or troubleshooting.

Definitions/Terminology

Incident An unplanned event that disrupts normal operations.
Problem The underlying cause of one or more incidents.
FMEA (Failure Mode and Effects Analysis): A method for identifying and prioritizing potential failure points in a process.
Six Sigma A data-driven methodology focused on improving processes by reducing defects and variability.
TQM (Total Quality Management): A management approach emphasizing continuous improvement and customer satisfaction across all organizational processes.
Post Mortem A retrospective analysis of an event to identify successes and areas for improvement.
Scientific Method A systematic process of forming hypotheses, testing them, and analyzing results to draw conclusions.
Iterative A repetitive approach to refining a process or solution through successive cycles.
Discrete Data Data that can only take specific, distinct values.
Ordinal Data with a meaningful order but no consistent interval (e.g., satisfaction ratings).
Nominal (Binary/Attribute): Categorical data without order, such as “yes/no” or “male/female.”
Continuous Data Data that can take any value within a range, such as temperature or time.
Risk Priority Number (RPN) A score in FMEA used to prioritize risks, calculated as Severity × Occurrence × Detection.
5 Whys A technique to identify the root cause of a problem by repeatedly asking “why” until the root cause is found.
Fishbone Diagram (Ishikawa) A visual tool used to identify and categorize potential causes of a problem.
Fault Tree Analysis (FTA) A deductive analysis method used to identify the root causes of system failures.
PDCA (Plan-Do-Check-Act) A cyclical process for continuous improvement in workflows or systems.
SIPOC A high-level process map identifying Suppliers, Inputs, Processes, Outputs, and Customers.

ProLUG Links ⛓️

ProLUG Admin Course Unit 14 🐧

Sun, 22 Dec 2024 00:00:00 +0000

Ansible Automation

Ansible is an open-source automation tool used for configuration management, application deployment, and IT orchestration, enabling tasks to be executed on multiple systems simultaneously without the need for agents. It uses simple YAML-based playbooks and SSH for communication, making it efficient and easy to learn for managing infrastructure.

Discussion Post 1:

Refer to your Unit 5 scan of the systems. You know that Ansible is a tool that you want to maintain in the environment. Review this online documentation: https://docs.ansible.com/ansible/latest/inventory_guide/intro_inventory.html

Make an inventory of the servers, grouped any way you like.

[warewulf] 192.168.200.25

[ubuntu] 192.168.200.[101:103] 192.168.200.[201:203]

[rockynodes] 192.168.200.[51:69]

What format did you choose to use for your inventory?

INI. YAML seems to be the clear choice as it allows for a more declarative inventory. However, while working in the study group it was easier to edit INI without making indentation errors.

What other things might you include later in your inventory to make it more useful?

I can add quite a few interesting things to an inventory file to make it more useful. Once the file reaches a certain size, it is better to break it out into separate unit files that are nested for things like Hosts, Host Variables, Production, Staging etc…

Some notable things I think make things pretty powerful are:

Dynamic Inventorying
Vault encrypted secrets
Jinja2 Dynamic variables

Discussion Post 2:

You have been noticing drift on your server configurations, so you want a way to generate a report on them every day to validate the configurations are the same.

Use any lab in here to find ideas: https://killercoda.com/het-tanis/course/Ansible- Labs Use this webhook to send your relevant data out to our sandbox. https://discord.com/api/webhooks/1317659221604433951/uyKpuq8fNNNSEyCra4n33PakI Bk-XtTn1WrwTpHs9BcgkIu7URPV_Gd5HJCRX0_EJVUT

name: System Information Check hosts: all become: yes tasks:
- name: Check kernel version command: uname -r register: kernel_version
- name: Display kernel version debug: msg: “Kernel Version: {{ kernel_version.stdout }}”
- name: Display kernel command line command: cat /proc/cmdline register: kernel_cmdline
- name: Debug kernel command line debug: msg: “Kernel Command Line: {{ kernel_cmdline.stdout }}”
- name: Check hardware information command: lshw register: hardware_info
- name: Debug hardware information debug: msg: “Hardware Information: {{ hardware_info.stdout }}”
- name: List installed RPM packages command: rpm -qa register: installed_rpms
- name: Debug installed RPM packages debug: msg: “Installed RPMs: {{ installed_rpms.stdout }}”
- name: Check active services command: systemctl list-units –type=service –state=running register: active_services
- name: Debug active services debug: msg: “Active Services: {{ active_services.stdout }}”
- name: List system users command: cat /etc/passwd register: system_users
- name: Debug system users debug: msg: “System Users: {{ system_users.stdout }}”
- name: Check last login information command: last register: last_login
- name: Debug last login information debug: msg: “Last Login Information: {{ last_login.stdout }}”
- name: Display currently logged-in users command: w register: logged_in_users
- name: Debug currently logged-in users debug: msg: “Logged-in Users: {{ logged_in_users.stdout }}”
- name: Display ISO time command: date –iso-8601=seconds register: iso_time
- name: Debug ISO time debug: msg: “ISO Time: {{ iso_time.stdout }}”

Discussion Post 3:

Using ansible module for git, pull down this repo: 👍 https://github.com/het-tanis/HPC_Deploy.git

How is the repo setup?

We have 3 playbooks in the root of the directory. These playbooks utilize roles defined in the roles subdirectory. Playbook 01 gathers facts about nfs using roles defined in a subdirectory called roles. Playbook 02 gathers data from a target system using roles defined in a subdirectory called data-gather. Playbook 03 updates and installs using roles defined in a subdirectory called packages_update/tasks & packages_install/tasks.

What is in the roles directory?

Partially answered for the previous question. Roles are structured with dedicated directories for tasks, handlers, and templates.

How are these playbooks called, and how do roles differ from tasks?

These playbooks incorporate roles based on specific conditions, executing the tasks defined within each role. When a role is included, the playbook inherits all its contents.

Digging Deeper

I have a large amount of labs to get you started on your Ansible Journey (all free):

https://killercoda.com/het-tanis/course/Ansible-Labs 👍

Find projects from our channel Ansible-Code, in Discord and find something that is

interesting to you. 👍

Use Ansible to access secrets from Hashicorp Vault: https://killercoda.com/het-

tanis/course/Hashicorp-Labs/004-vault-read-secrets-ansible 👍

Lab Work 🧪 👍

Create an inventory:

While still in the /root/ansible_madness directory, create a file hosts vi /root/ansible_madness/hosts Populate the file as follows

[servers]

192.168.200.101
192.168.200.102
192.168.200.103

Run Ad Hoc commands against your servers This will test your connection into all the servers.

1. ansible servers -i hosts -u inmate -k -m shell -a uptime

Use this password: LinuxR0cks1!

Do the same thing, but this time be verbose

2. ansible -vvv servers -i hosts -u inmate -k -m shell -a uptime

Create a playbook to push over files.

3. echo “This is my file ” > somefile

4. vi deploy.yaml

Populate it as follows:

name: Start of push playbook
hosts: servers
vars:
gather_facts: True
become: False
tasks:
name: Copy somefile over at {{ ansible_date_time.iso8601_basic_short }}
copy:
src: /root/ansible_madness/somefile
dest: /tmp/somefile.txt

5. Execute your playbook

ansible-playbook -i hosts -k deploy.yaml

6. Verify that your file pushed everywhere

ansible servers -i hosts -u inmate -k -m shell -a “ls -l /tmp/somefile” Pull down a github repo

git clone https://github.com/het-tanis/HPC_Deploy.git cd HPC_Deploy What do you see in here? What do you need to learn about more to deploy some of these tools? Can you execute some of these, why or why not?

Reflection Questions

What questions do you still have about this week?
How can you apply this now in your current role in IT? If you’re not in IT, how can you look to put something like this into your resume or portfolio?

ProLUG Links ⛓️

ProLUG Admin Course Unit 13 🐧

Sun, 15 Dec 2024 00:00:00 +0000

System Hardening

Linux system hardening involves securing the system by reducing its attack surface through measures such as disabling unnecessary services, enforcing access controls, applying security patches, and using tools like OpenSCAP, STIG compliance frameworks, or the OSCAP Scanner. These tools help automate security audits, enforce compliance standards, and identify vulnerabilities to enhance system security.

Discussion Post 1:

Your security team comes to you with a discrepancy between the production security baseline and something that is running on one of your servers in production. There are 5 servers in a web cluster and only one of them is showing this behavior. They want you to account for why something is different.

How are you going to validate that the difference between the systems?

I am going to assume that I am new to the system in general and have very surface knowledge from fellow staff. I am also assuming we are working with a redhat based system.

Starting off simple

Maybe the problem is an obvious one, so I would just start off with a glance.

Quick cursory check (Kernel Version uname-a)
Manually Checking Logs (Journalctl, dmesg, audit.log, syslog)
Checking ports (Socket Statistics, ss -ntulp)
Listing installed packages (DNF list, RPM -qa)
Listing users and Logins (/etc/passwd, w, last)
Seeing what System D services are running (systemctl list-units - -type=service)
Digging for documentation and or commit history

Deeper ⛏️

If no low hanging fruit were there, then I would check configurations

Grub.conf, FirewallD/Apparmour, SELinux,

Sorting 🪰‘ish from 🌶️

If I do that see something distinctly different, I would employ a more sophisticated approach with difference checking. Given that everything is a structured file, I can append the output from a working system and the goose 🪿 to a new file and run diff against them.

Diff’ing the Logs
Diff’ing Socket Statistics
Diff’ing Installed Packages

What are you going to look at to explain this?

I think I have answered this above.

What could be done to prevent this problem in the future?

Introducing or Improving the change management policy and employing version control would be my first suggestion.
Ensuring that there is a build/test/deploy pipeline that integrates tightly with change management.
Using IaC and Automation to ensure consistency and repeatability with tools like Ansible, Packer, Podman or Kubernetes.
Hardening systems with either simple policies or through the guidance of STIG’s.
Introducing stronger controls over user privileges like employing RBA policies.

Discussion Post 2:

Your team has been giving you more and more engineering responsibilities. You are being asked to build out the next set of servers to integrate into the development environment. Your team is going from RHEL 8 to Rocky 9.4.

How might you start to plan out your migration?

Observe

Firstly I would gather system information

Benchmark/baseline performance metrics and utilization (Disk, I/O, PS, Connections etc)
Configs (Scripts and configuration files)
Installed Packages.
users (Listing users and privileges)
Policies (Firewall, SELinux)
Purpose (Assessing the use of a particular system to see if may need changes/upgrades)

Capture

I would snapshot the current system if possible
If a complete snapshot copy is not possible, I would gather files essential to rebuilding a replica

Reconstruct

Build it in a test VM emulating the current environment
Template the VM for experimental changes (Adding additional tools or Configs)

Analyze / Optimize

Gather business or operational requirements, perhaps the system needs enhancements
Experiment with performance tuning
Test new packages and/or configurations

Build

During the analysis and optimization phase, I would start a playbook with information gathered from previous phases. I would build and run the playbook against VM templates until satisfied.

Deploy

Given the prior phases, my Playbook would be robust and capable of the transition. However, I would ensure a robust backup and rollback plan in the case something fails.

What are you going to check on the existing systems to baseline your build?

Compute Usage
Memory Load
Disk Resources
Networking Metrics

What kind of validation plan might you use for your new Rocky 9.4 systems?

I would have a seperate playbook built that would validate performance against what I was observing during my VM experimentation. Though the environment may differ from that of the VM, I would still be able to discern performance characteristics and notice any outlier differences.

Digging Deeper

Run through this lab: https://killercoda.com/het-tanis/course/Linux-Labs/107-server-startup-process 👍

How does this help you better understand the discussion 13-2 question?

Well when I am gathering a picture of my current security baseline, I can use some of these tools like dmesg and ss to see what possible attack surface I may have.

Run through this lab: https://killercoda.com/het-tanis/course/Linux-Labs/203-updating-golden-image 👍

How does this help you better understand the process of hardening systems?

Reflection Questions

What questions do you still have about this week?
How can you apply this now in your current role in IT? If you’re not in IT, how can you look to put something like this into your resume or portfolio?

Lab Work 🧪

1. You will scan a server for a SCC Report and get a STIG Score 👍

2. You will remediate some of the items from the scan 👍

3. You will rescan and verify a better score. 👍

ProLUG Links ⛓️

ProLUG Admin Course Unit 12 🐧

Sat, 30 Nov 2024 00:00:00 +0000

Baselining & Benchmarking

The purpose of a baseline is not to find fault, load, or to take corrective action. A baseline simply determines what is. You must know what is so that you can test against that when you make a change to be able to objectively say there was or wasn’t an improvement. You must know where you are at to be able to properly plan where you are going. A poor baseline assessment, because of inflated numbers or inaccurate testing, does a disservice to the rest of your project. You must accurately draw the first line and understand your system’s performance.

Discussion Post 1:

Your manager has come to you with another emergency. He has a meeting next week to discuss capacity planning and usage of the system with IT upper management. He doesn’t want to lose his budget, but he has to prove that the system utilization warrants spending more.

What information can you show your manager from your systems?

You could present your manager with a progressive trend graph showing time on the x-axis and several fields on the y-axis that represent changes from a baseline, assuming the necessary data has been collected. With this information, it would be possible to predict when various system resources will reach their maximum capacity.

What type of data would prove system utilization? (Remember the big 4: compute, memory, disk, networking)

CPU load, process execution time, throughput. Disk Operations (IOPS). Networking Requests and Bandwidth. RAM utilization, memory paging/swapping rates.
What would your report look like to your manager?

Capacity Planning Report

Current and projected system utilization. By examining trends over time, we can predict when critical resources will reach their limits if no additional capacity is provisioned.

Key Areas of Focus

Compute Usage
Memory Load
Disk Resources
Networking Metrics

Historical Data and Trends

Below is a sample progressive trend graph over the last 6 months. The x-axis represents time (in weeks), while the y-axis shows percentage utilization relative to an established baseline.

Example Metrics (relative to baseline):

CPU Utilization (% of baseline)
Memory Load (% of baseline)
Disk IOPS (% of baseline)
Network Throughput (% of baseline)

Time (Weeks): 1 2 3 4 5 6 … 20 21 22 CPU Util(%): 50 52 55 57 60 62 … 80 82 85 Memory(%): 45 47 50 50 52 55 … 70 73 75 Disk IOPS(%): 30 32 35 36 38 40 … 60 63 68 Network(%): 40 42 45 47 49 51 … 75 78 80

As time progresses, each of the key metrics is trending upward, indicating increasing load and approaching capacity thresholds.

Projections

We can estimate the “time to ceiling” for critical resources. For instance, if CPU load is rising at an average rate of 2–3% per month, and we know that at 90% utilization the system will experience performance degradation.

Projected Time to CPU Ceiling: 3–5 months
Projected Time to Memory Ceiling: 6–8 months
Projected Time to Disk IOPS Ceiling: 8–10 months
Projected Time to Network Bandwidth Ceiling: 4–6 months

Recommendations

Compute: Consider adding more CPU cores or upgrading processors before reaching the predicted 90% utilization mark.
Memory: Upgrade RAM or optimize applications to reduce memory footprint.
Disk: Enhance disk subsystems or switch to faster storage (e.g., SSDs) to handle projected IOPS.
Networking: Increase network capacity (e.g., from 1Gb to 10Gb links) or optimize network traffic.

Conclusion

Investment in scaling resources now will prevent future performance bottlenecks, ensuring the system can continue to meet business demands effectively.

Discussion Post 2:

You are in a capacity planning meeting with a few of the architects. They have decided to add 2 more agents to your Linux Sytems, Bacula Agent and an Avamar Agent . They expect these agents to run their work starting at 0400 every morning.

What do these agents do? (May have to look them up)

Bacula is an open-source suite of tools designed to automate backup tasks. It’s widely regarded for its flexibility and reliability. Dell Avamar, on the other hand, is a commercial backup automation solution. Both tools handle incremental backups using custom daemons that monitor changes over time, offering greater sophistication than simple scheduling systems like Cron. Additionally, they can manage backups across diverse, heterogeneous storage environments.

Do you think there is a good reason not to use these agents at this timeframe?

This approach is about balancing workload. If all processes start at a fixed time, they can consume valuable resources simultaneously. The best schedule depends on the environment. For example, if the environment experiences downtime—such as a traditional office setting—starting backups at 4 a.m. might be fine. However, if services run around the clock, it’s better to stagger the tasks so they use only a fraction of the available resources at any given time. This approach also reduces the impact of failures, since not all systems are involved at once.

Is there anything else you might want to point out to these architects about these agents they are installing?

There are several factors architects should consider. However, in the context of this discussion, performance overhead is particularly relevant. They need to ensure that the chosen backup solutions won’t overburden the system’s resources and that there’s enough “breathing room” to maintain smooth operations.

Discussion Post 3: ‘TODO’

Your team has recently tested at proof of concept of a new storage system. The vendor has published the blazing fast speeds that are capable of being run through this storage system. You have a set of systems connected to both the old storage system and the new storage system.

Write up a test procedure of how you may test these two systems.

I did a bit of research regarding tooling for such a task and found FIO ‘Flexible Input / Output’, a program written for the purpose of testing systems with various scenarios. Rather than using BASH, I can run more comprehensive testing with more data to analyze using FIO.

Baseline Test

 fio --filename=/dev/new_storage_lun --direct=1 --rw=read --bs=128k --size=10G --numjobs=1 --iodepth=32 --runtime=300 --time_based --name=new_storage_seq_read
fio --filename=/dev/old_storage_lun --direct=1 --rw=read --bs=128k --size=10G --numjobs=1 --iodepth=32 --runtime=300 --time_based --name=old_storage_seq_read

Running Mixed Workload Tests

 fio --filename=/dev/new_storage_lun --direct=1 --rw=randrw --rwmixread=70 --bs=4k --size=10G --numjobs=4 --iodepth=16 --runtime=300 --time_based --name=new_storage_mixed

Increased Concurrency and Scale

Increase numjobs and iodepth in subsequent runs to measure how performance changes:

fio –filename=/dev/new_storage_lun –direct=1 –rw=read –bs=128k –size=10G –numjobs=8 –iodepth=64 –runtime=300 –time_based –name=new_storage_high_concurrency
Run the same tests on the old storage system and record all metrics.

Stress/Soak Tests

12-hour continuous I/O test on both storage systems.

fio –filename=/dev/new_storage_lun –direct=1 –rw=randwrite –bs=4k –size=100G –numjobs=1 –iodepth=32 –runtime=43200 –time_based –name=new_storage_soak

AWK Line Parsing

I would then pipe the output of these commands to AWK to seperate out specific datapoints to append to files for full analysis.

How are you assuring these test are objective?

By gathering multiple datasets with varying run parameters, I can reduce statistical noise and better isolate data of interest by comparing these datasets against one another.

What is meant by the term Ceteris Paribus, in this context?

in the context of system benchmarking means that when measuring the performance of one specific aspect of the system, all other variables and conditions are kept constant. This approach ensures that any observed changes in performance can be attributed directly to the variable under test, rather than being influenced by unrelated fluctuations in the environment or system load.

Definitions & Terminology

Benchmark
High watermark
Scope
Methodology
Testing
Control
Experiment
Analytics
Descriptive
Diagnostic
Predictive
Prescriptive

Digging Deeper (optional)

Analyzing data may open up a new field of interest to you. Go through some of the free lessons on Kaggle, here: https://www.kaggle.com/learn

a. What did you learn?

b. How will you apply these lessons to data and monitoring you have already collected as a system administrator?

Find a blog or article that discusses the 4 types of data analytics.

a. What did you learn about past operations? b. What did you learn about predictive operations?

Download Spyder IDE (Open source)

a. Find a blog post or otherwise try to evaluate some data. b. Perform some Linear regression. My block of code (but this requires some additional libraries to be added. I can help with that if you need it.)

import matplotlib.pyplot as plt

from sklearn.linear_model import LinearRegression

size = [[5.0], [5.5], [5.9], [6.3], [6.9], [7.5]] price =[[165], [200], [223], [250], [278], [315]] plt.title(‘Pizza Price plotted against the size’)

plt.xlabel(‘Pizza Size in inches’)

plt.ylabel(‘Pizza Price in cents’)

plt.plot(size, price, ‘k.’)

plt.axis([5.0, 9.0, 99, 355])

plt.grid(True)

model = LinearRegression()

model.fit(X = size, y = price)

#plot the regression line

plt.plot(size, model.predict(size), color=‘r’)

Reflection Questions

What questions do you still have about this week?
How can you apply this now in your current role in IT? If you’re not in IT, how can you look to put something like this into your resume or portfolio?

Digging Deeper

1. Read the rest of the chapter https://sre.google/workbook/monitoring/ and note anything else of interest when it comes to monitoring and dashboarding.

2. Look up the “ProLUG Prometheus Certified Associate Prep 2024” in Resources -> Presentations in our ProLUG Discord. Study that for a deep dive into Prometheus.

3. Complete the project section of “Monitoring Deep Dive Project Guide” from the prolug-projects section of the Discord. We have a Youtube video on that project as well. https://www.youtube.com/watch?v=54VgGHr99Qg

Labs

https://killercoda.com/het-tanis/course/Linux-Labs/102-monitoring-linux-logs

https://killercoda.com/het-tanis/course/Linux-Labs/103-monitoring-linux-telemetry

https://killercoda.com/het-tanis/course/Linux-Labs/104-monitoring-linux-Influx-Grafana

While completing each lab think about the following:

a. How does it tie into the diagram below?

b. What could you improve, or what would you change based on your previous administration experience.

Install Grafana on the Rocky Linux system by adding the Grafana repo manually. Red = Inputs Blue = Outputs

Create a new repository configuration sudo vim /etc/yum.repos.d/grafana.repo

Paste:

[grafana] name=grafana baseurl=https://packages.grafana.com/oss/rpm
repo_gpgcheck=1 enabled=1 gpgcheck=1 gpgkey=https://packages.grafana.com/gpg.key sslverify=1

Verify using the DNF

sudo dnf repolist

sudo dnf clean - verifies whether files are working

Should see:

repo id repo name appstream Rocky Linux 8 - AppStream baseos Rocky Linux 8 - BaseOS extras Rocky Linux 8 - Extras grafana grafana/spl

Check the grafana package on the official repository

sudo dnf info grafana

Should see something similar 👀

Importing GPG key 0x24098CB6: Userid : “Grafana " Fingerprint: 4E40 DDF6 D76E 284A 4A67 80E4 8C8C 34C5 2409 8CB6 From : https://packages.grafana.com/gpg.key Is this ok [y/N]: y

Should see 👀

Name : grafana Version : 8.2.5 Release : 1 rchitecture : x86_64 Size : 64 M Source : grafana-8.2.5-1.src.rpm Repository : grafana Summary : Grafana URL : https://grafana.com License : “Apache 2.0” Description : Grafana

Install Grafana sudo dnf install grafana -y

⏳ Takes a while…

Restart SystemD unit sudo systemctl enable –now grafana-server

Verify sudo systemctl status grafana-server

5.5. Firewall (Security) Firewall is Managed by files present in /etc/firedwalld

cd /etc/firewalld/ ls -l

cp /usr/lib/firewalld/services/ssh.xml /etc/firewalld/services/example.xml

sudo firewall-cmd –add-service=grafana –permanent

sudo firewall-cmd –add-port=3000/tcp –permanent sudo firewall-cmd –reload

Create config file sudo vim /etc/grafana/grafana.ini
Change the default value of:

The option ‘http_addr’ to ’localhost’, the ‘http_port’ to ‘3000’, and the ‘domain’ option to your domain name as below. For this example, the domain name is ‘grafana.example.io’.

For non-standard port, be sure to uncomment ; [server] 👍 http_port = 4000 👍

The public facing domain name used to access grafana from a browser domain = grafana.example.io

7.1 Turn off the nasty default report of analytics 👺 [analytics] reporting_enabled = false

7.2. Restart the grafana service to apply a new configuration.

sudo systemctl restart grafana-server

Reverse Proxy Setup

Install NGINX

sudo dnf install nginx -y

Create a new server block for grafana

/etc/nginx/conf.d/grafana.conf

Required to proxy Grafana Live WebSocket connections

map $http_upgrade $connection_upgrade { default upgrade; ’’ close; } server { listen 80; server_name grafana.example.io; rewrite ^ https://$server_name$request_uri? permanent; } server { listen 443 ssl http2; server_name grafana.example.io; root /usr/share/nginx/html; index index.html index.htm; ssl_certificate /etc/letsencrypt/live/grafana.example.io/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/grafana.example.io/privkey.pem; access_log /var/log/nginx/grafana-access.log; error_log /var/log/nginx/grafana-error.log; location / { proxy_pass http://localhost:3000/; }

Proxy Grafana Live WebSocket connections location /api/live { rewrite ^/(.*) /$1 break; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $connection_upgrade; proxy_set_header Host $http_host; proxy_pass http://localhost:3000/; } }

Next, verify the Nginx configuration

sudo nginx -t

Should see 👀

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful 👍

Start and enable the Nginx service sudo systemctl enable –now nginx sudo systemctl status nginx

Install Prometheus (Saturday)

Rocky Prometheus Install

Add New User and Directory ‘prometheus’

create a new configuration directory and data directory for the Prometheus installation.

sudo adduser -M -r -s /sbin/nologin prometheus

create a new configuration directory ‘/etc/prometheus’ and the data directory ‘/var/lib/prometheus’

(Only needed for running as service)

sudo mkdir /etc/prometheus sudo mkdir /var/lib/prometheus

Note: All Prometheus configuration at the ‘/etc/prometheus’ directory, and all Prometheus data will automatically be saved to the directory ‘/var/lib/prometheus’. Installing Prometheus on Rocky Linux

Install Prometheus monitoring system manually from the tarball or tar.gz file.

Change the working directory to ‘/usr/src’ and download the Prometheus binary

cd /usr/src wget https://github.com/prometheus/prometheus/releases/download/v3.0.1/prometheus-3.0.1.linux-amd64.tar.gz

Extract

tar -xzf ***.tar.gz

cd into folder

Run bin:

./bin 👍
If bin works, proceed

Copy all Prometheus configurations to the directory ‘/etc/prometheus’ and the binary file ‘prometheus’ to the ‘/usr/local/bin’ directory.

Move prometheus configuration ‘prometheus.yml’ to the directory ‘/etc/prometheus.

sudo mv $PROM_SRC/prometheus.yml /etc/prometheus/

Move the binary file ‘prometheus’ and ‘promtool’ to the directory ‘/usr/local/bin/’.

sudo mv $PROM_SRC/prometheus /usr/local/bin/ sudo mv $PROM_SRC/promtool /usr/local/bin/

Move Prometheus console templates and libraries to the ‘/etc/prometheus’ directory.

sudo mv -r $PROM_SRC/consoles /etc/prometheus sudo mv -r $PROM_SRC/console_libraries /etc/prometheus

Edit Prometheus configuration ‘/etc/prometheus/prometheus.yml’

vim /etc/prometheus/prometheus.yml

On the ‘scrape_configs’ option, you may need to add monitoring jobs

The default configuration comes with the default monitoring job name ‘prometheus’ and the target server ’localhost’ through the ‘static_configs’ option.

Change the target from ’localhost:9090’ to the server IP address ‘192.168.1.10:9090’ as below.

Note: Scrape configuration containing exactly one endpoint to scrape: Here it’s Prometheus itself. scrape_configs: The job name is added as a label job=<job_name> to any timeseries scraped from this config. job_name: “prometheus”

metrics_path defaults to ‘/metrics’ scheme defaults to ‘http’.

static_configs: targets: [“192.168.1.10:9090”]

Change the configuration and data directories to the user ‘promethues’.

sudo chown prometheus:prometheus /etc/prometheus sudo chown prometheus:prometheus /var/lib/prometheus

Basic prometheus installation finished, Hopefully 👍 .

Configure Prometheus

Create a new systemd service sudo vim /etc/systemd/system/prometheus.service

Copy and paste the following configuration.

[Unit] Description=Prometheus Wants=network-online.target After=network-online.target

[Service] User=prometheus Group=prometheus Type=simple ExecStart=/usr/local/bin/prometheus
–config.file /etc/prometheus/prometheus.yml
–storage.tsdb.path /var/lib/prometheus/
–web.console.templates=/etc/prometheus/consoles
–web.console.libraries=/etc/prometheus/console_libraries

[Install] WantedBy=multi-user.target

Reload the systemd manager to apply a new config.

sudo systemctl daemon-reload

Start and enable the Prometheus service

sudo systemctl enable –now prometheus sudo systemctl status prometheus

Prometheus monitoring tool is now accessible on the TCP port ‘9090.

Visit IP address with port ‘9090’

http://192.168.1.10:9090/

And you will see the prometheus dashboard query below.

Prometheus query dashboard

Reflection Questions

What questions do you still have about this week?
How can you apply this now in your current role in IT? If you’re not in IT, how can you look to put something like this into your resume or portfolio?

ProLUG Links ⛓️

ProLUG Admin Course Unit 11 🐧

Thu, 28 Nov 2024 00:00:00 +0000

Monitoring Systems

In this unit, we explore monitoring systems, which often consist of multiple interconnected components. At its core, monitoring involves carefully exposing system data and transmitting it to tools for analysis and alerting. From my experience with Prometheus and Grafana—two widely used and versatile solutions—I’ve seen how effective these tools can be for various scenarios. However, many other tools are also available. One of my key takeaways from the unit’s readings and labs was the importance of careful data exposure. Much like setting permissions in a Linux system, it’s crucial to determine what data can be accessed and who is allowed to see it in the reporting chain. System information, if mishandled, can easily become a double-edged sword.

Discussion Post 1

Scenario

You’ve heard the term “loose coupling” thrown around the office about a new monitoring solution coming down the pike. You find a good resource and read the section on “Prefer Loose Coupling” https://sre.google/workbook/monitoring/

1. What does “loose coupling” mean, if you had to summarize to your junior team

members?

Loose coupling means the components of a system can operate independently, yet still work together when combined. This design allows individual components to be swapped or replaced with minimal disruption to the overall system. In contrast, a strongly coupled system binds its components so tightly that altering one would disrupt or even break the entire system’s functionality.

2. What is the advantage given for why you might want to implement this type of tooling in your monitoring? Do you agree? Why or why not?

The advantage of a loosely coupled monitoring system lies in its flexibility to evolve over time. Systems change, requirements shift, and new tools emerge, making adaptability essential. A design that allows components to be replaced or upgraded with minimal disruption is highly valuable—not only for an organization aiming to maintain efficiency but also for the administrators and engineers responsible for ensuring stability and resolving issues.

3. They mention “exposing metrics” what does it mean to expose metrics? What happens to metrics that are exposed but never collected?

Exposing metrics involves making system information accessible for monitoring and analysis. However, this must be approached with caution, as exposing such information can introduce vulnerabilities. Simply exposing data without actively collecting or utilizing it needlessly increases security risks without providing any benefit.

Discussion Post 2

Scenario

Your HPC team is asking for more information about how CPU 0 is behaving on a set of servers. Your team has node exporter writing data out to Prometheus (Use this to simulate https://promlabs.com/promql-cheat-sheet/).

1. Can you see the usage of CPU0 and what is the query?

Yes one can use a query that focuses on the metrics provided by the Node Exporter. Specifically, filter for CPU 0 and its usage.

100 - (avg by (instance) (irate(node_cpu_seconds_total{cpu="0", mode="idle"}

2. Can you see the usage of CPU0 for just the last 5 minutes and what is the query?

Yes

100 - (avg by (instance) (rate(node_cpu_seconds_total{cpu="0", mode="idle"}[5m])) * 100)

3. You know that CPU0 is excluded from Slurm, can you exclude that and only pull the user and system for the remaining CPUs and what is that query?

Yes

sum by (instance) (rate(node_cpu_seconds_total{cpu!="0", mode=~"user|system"}[5m])) * 100

Digging Deeper

1. Read the rest of the chapter https://sre.google/workbook/monitoring/ and note anything else of interest when it comes to monitoring and dashboarding.

2. Look up the “ProLUG Prometheus Certified Associate Prep 2024” in Resources -> Presentations in our ProLUG Discord. Study that for a deep dive into Prometheus.

3. Complete the project section of “Monitoring Deep Dive Project Guide” from the prolug-projects section of the Discord. We have a Youtube video on that project as well. https://www.youtube.com/watch?v=54VgGHr99Qg

Labs

https://killercoda.com/het-tanis/course/Linux-Labs/102-monitoring-linux-logs

https://killercoda.com/het-tanis/course/Linux-Labs/103-monitoring-linux-telemetry

https://killercoda.com/het-tanis/course/Linux-Labs/104-monitoring-linux-Influx-Grafana

While completing each lab think about the following:

a. How does it tie into the diagram below?

b. What could you improve, or what would you change based on your previous administration experience.

1. Install Grafana

1.1 Create a New Repository Configuration

sudo vim /etc/yum.repos.d/grafana.repo

Paste the following:

[grafana]
name=grafana
baseurl=https://packages.grafana.com/oss/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packages.grafana.com/gpg.key
sslverify=1

1.2 Verify the Repository

sudo dnf repolist
sudo dnf clean all

Expected Output:

repo id repo name
appstream Rocky Linux 8 - AppStream
baseos Rocky Linux 8 - BaseOS
extras Rocky Linux 8 - Extras
grafana grafana/spl

1.3 Check the Grafana Package

sudo dnf info grafana

Example Output:

Importing GPG key 0x24098CB6:
Userid : "Grafana"
Fingerprint: 4E40 DDF6 D76E 284A 4A67 80E4 8C8C 34C5 2409 8CB6
From : https://packages.grafana.com/gpg.key
Is this ok [y/N]: y

1.4 Install Grafana

sudo dnf install grafana -y

1.5 Enable and Start Grafana Service

sudo systemctl enable --now grafana-server
sudo systemctl status grafana-server

1.6 Configure Firewall

sudo firewall-cmd --add-service=grafana --permanent
sudo firewall-cmd --add-port=3000/tcp --permanent
sudo firewall-cmd --reload

1.7 Create and Edit Configuration File

sudo vim /etc/grafana/grafana.ini

Update the following settings:

[server]
http_port = 4000
domain = grafana.example.io

[analytics]
reporting_enabled = false

Restart the service:

sudo systemctl restart grafana-server

2. Reverse Proxy Setup with NGINX

2.1 Install NGINX

sudo dnf install nginx -y

2.2 Create a Server Block for Grafana

sudo vim /etc/nginx/conf.d/grafana.conf

Paste the configuration:

map $http_upgrade $connection_upgrade {
 default upgrade;
 '' close;
}

server {
 listen 80;
 server_name grafana.example.io;
 rewrite ^ https://$server_name$request_uri? permanent;
}

server {
 listen 443 ssl http2;
 server_name grafana.example.io;

 ssl_certificate /etc/letsencrypt/live/grafana.example.io/fullchain.pem;
 ssl_certificate_key /etc/letsencrypt/live/grafana.example.io/privkey.pem;

 location / {
 proxy_pass http://localhost:3000/;
 }

 location /api/live {
 proxy_http_version 1.1;
 proxy_set_header Upgrade $http_upgrade;
 proxy_set_header Connection $connection_upgrade;
 proxy_pass http://localhost:3000/;
 }
}

2.3 Verify NGINX Configuration

sudo nginx -t

Expected Output:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

2.4 Start and Enable NGINX

sudo systemctl enable --now nginx
sudo systemctl status nginx

3. Install Prometheus

3.1 Add a New User and Directories

sudo adduser -M -r -s /sbin/nologin prometheus
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus

3.2 Download and Extract Prometheus

cd /usr/src
wget https://github.com/prometheus/prometheus/releases/download/v3.0.1/prometheus-3.0.1.linux-amd64.tar.gz
tar -xzf prometheus-3.0.1.linux-amd64.tar.gz
cd prometheus-3.0.1.linux-amd64

3.3 Move Files to Appropriate Locations

sudo mv prometheus /usr/local/bin/
sudo mv promtool /usr/local/bin/
sudo mv consoles /etc/prometheus
sudo mv console_libraries /etc/prometheus
sudo mv prometheus.yml /etc/prometheus/

3.4 Change Ownership

sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus

3.5 Configure Prometheus

sudo vim /etc/prometheus/prometheus.yml

Example scrape configuration:

scrape_configs:
 - job_name: "prometheus"
 static_configs:
 - targets: ["192.168.1.10:9090"]

3.6 Create a Systemd Service

sudo vim /etc/systemd/system/prometheus.service

Paste the following:

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /var/lib/prometheus/ --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

3.7 Start Prometheus Service

sudo systemctl daemon-reload
sudo systemctl enable --now prometheus
sudo systemctl status prometheus

3.8 Access Prometheus

Visit http://192.168.1.10:9090 in your browser to view the Prometheus dashboard.

Reflection Questions

What questions do you still have about this week?
How can you apply this now in your current role in IT? If you’re not in IT, how can you look to put something like this into your resume or portfolio?

ProLUG Links ⛓️

ProLUG Admin Course Unit 10 🐧

Fri, 22 Nov 2024 00:00:00 +0000

Kubernetes

“The OS of the Internet”

This week, we were privileged to host a special guest lecture by John Champine, Scott’s brother. Over the course of an engaging two-hour session, John delivered an in-depth exploration of Kubernetes, thoroughly covering the five W’s: Who, What, When, Where, and Why.

John’s passion and deep knowledge of Kubernetes were evident throughout the presentation. Having firsthand experience with the challenges of pre-Kubernetes infrastructure, he offered valuable insights into how this platform has revolutionized modern computing. Notably, John specializes in OpenShift, an IBM-owned management layer built atop Kubernetes. OpenShift adds additional functionality and ease of use to what is already a powerful but complex system.

One concept that particularly caught my attention was the fractionalization of CPU and memory resources made possible by Kubernetes’ sophisticated resource management. John introduced the term millicore, a concept I was previously unfamiliar with. It refers to the fine-grained allocation of processing power, where fractions of a CPU core are shared across processes during compute cycles. This ability to manage resources at such a granular level is remarkable, showcasing the efficiency and precision of Kubernetes.

Before this lecture, I never imagined that such details—down to the microsecond allocation of core usage—could not only be considered but also controlled and utilized to optimize workloads. This level of resource management truly solidifies Kubernetes’ position as the “operating system of the internet,” enabling applications to run more efficiently and reliably across diverse infrastructures.

John’s insights not only deepened my understanding of Kubernetes but also sparked curiosity about the broader implications of containerized resource management in modern computing.

One idea that has been dispelled by doing these exercises is that Kubernetes is overkill for most things. Yes, standing up multiple networked nodes and having them interoperate is no small task. However, there exist lightweight, single node variants like K3s that enable a simplified experience while maintaining the powerful advantages of an orchestration system.

Furthermore, given the threat landscape, desire for declarative infrastructure and ever fluctuating demand, it makes sense to implement a tool that is designed for these modern demands.

Discussion Post 1

Reading: https://kubernetes.io/docs/concepts/overview/ 📗

Question 1

What are the two most compelling reasons you see to implement Kubernetes in your organization?

Efficient Usage of Resources: Kubernetes optimizes resource allocation and ensures workloads are distributed efficiently across your infrastructure.
Built-in Redundancy: Kubernetes provides automatic failover and self-healing capabilities, ensuring high availability for applications.

Other notable reasons include:

Load Balancing: Automatically distributes traffic across pods to maintain application performance. ⚖️
Centralized Control: Manages deployments, updates, and scaling from a single interface. 🎯
Declarative in Nature: Uses declarative configurations, making it easier to maintain and reproduce states. 🔖
Customizable for Organizational Needs: Offers flexibility to adapt to specific requirements through pluggable components. 🔌
Open Source: Eliminates vendor lock-in while benefiting from a large and active community. ⚙️
Broadly Used: Extensive community adoption ensures a wealth of resources, tools, and discussions. 📜
Written in Go: Delivers a lightweight, high-performance foundation that I am irrationally a fan of 😉

Question 2

When the article says Kubernetes is not a PaaS, what do they mean by that? What is a PaaS in comparison?

Kubernetes is not a Platform-as-a-Service (PaaS) because it operates at the container level rather than the application or hardware level. While it provides some features similar to PaaS offerings, Kubernetes emphasizes flexibility, composability, and user choice rather than prescribing a monolithic solution.

Key Characteristics of Kubernetes Compared to PaaS:

What Kubernetes Is:
- Provides foundational building blocks for deploying, scaling, and managing containerized applications.
- Is modular, with optional and pluggable solutions for logging, monitoring, alerting, and scaling.
- Focuses on maintaining desired state through independent, composable control processes.
- Preserves user choice by not dictating application or configuration specifics.
What Kubernetes Is Not:
- Does not limit the types of applications it supports.
- Does not deploy source code or build your application.
- Does not provide application-level services (e.g., middleware, databases).
- Does not enforce specific logging, monitoring, or alerting solutions.
- Does not include a proprietary configuration system or language (e.g., Jsonnet).
- Does not manage comprehensive machine-level configuration, maintenance, or self-healing.

Comparison to PaaS:

A Platform-as-a-Service (PaaS), such as Heroku or OpenShift, provides a more opinionated, all-encompassing solution by:

Supporting application deployment directly from source code.
Managing application-level services like databases or caching layers.
Offering integrated logging, monitoring, and alerting.
Providing a simplified developer experience but with less flexibility.

In contrast, Kubernetes gives users the tools to build their own developer platforms while leaving critical choices in the user’s hands.

Discussion Post 2

Scenario

You get a ticket about your new test cluster. The team is unable to deploy some of their applications. They suspect there is a problem and send you over this output:

Kubernetes Cluster Information

kubectl Version

Component	Version
Client Version	v1.31.6+k3s3
Kustomize Version	v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version	v1.30.6+k3s1

Node Status

NAME	STATUS	ROLES	AGE	VERSION
Test_Cluster1	Ready	control-plane,master	17h	v1.30.6+k3s1
Test_Cluster2	NotReady	worker	33m	v1.29.6+k3s1
Test_Cluster3	Ready	worker	17h	v1.28.6+k3s1

Question 1

What are you checking on the cluster to validate you see their error?

To identify and validate the issue with the node Test_Cluster2:

Check the overall cluster status

Run kubectl get nodes to see the status of all nodes in the cluster.
This confirms Test_Cluster2 is NotReady.

Inspect node details

Run kubectl describe nodes Test_Cluster2 to check for events, taints, and resource usage issues. Look for errors related to kubelet, network, or node conditions.

Access the node**

SSH into the node with ssh Test_Cluster2 to perform further diagnostics.

Verify kubelet status

Run systemctl status kubelet to check if the kubelet service is running and healthy.

Check container runtime

Depending on your runtime, run either systemctl status docker or systemctl status podman to ensure the container runtime is operational.

Reload and restart services

If issues are detected, attempt to reload the systemd daemon (systemctl daemon-reload) and restart kubelet (systemctl restart kubelet).

Question 2

What do you think the problem could be?

Potential problems for Test_Cluster2 being NotReady include:

The Test_Cluster2 node is running Kubernetes version v1.29.6+k3s1, which is older than the server and other nodes (v1.30.6+k3s1). This may cause compatibility issues.
The kubelet service might have failed or not started.
Networking problems could prevent the node from communicating with the control plane.
The node may lack sufficient CPU, memory, or disk space for the kubelet to function.
Taints, labels, or configuration errors might prevent the node from becoming Ready.
Hardware issues or kernel module problems may impact the node’s health.

Question 3

Do you think someone else has tried anything to fix this problem before you? Why or why not?

Yes, someone may have tried to fix it

The node was added only 33 minutes ago, indicating recent activity.
It’s common for issues to be noticed and an initial attempt made before escalating.
systemctl status may show if services were restarted recently, suggesting prior troubleshooting.

No, it may not have been addressed yet because:

The node is still in the NotReady state, implying no successful resolution so far.
Lack of detailed documentation or escalation might mean no one has investigated yet.

To confirm, check the node’s event logs (kubectl describe nodes Test_Cluster2) or system logs (journalctl -u kubelet) for evidence of recent actions.

Discussion Post 3

Scenario:

You are the network operations center (NOC) lead. Your team has recently started supporting the dev, test, and QA environments for your company’s K8s cluster. Write up a basic checkout procedure for your new NOC personnel to verify operation of the cluster before escalating on critical alerts.

Kubernetes Cluster Checkout Procedure for NOC Personnel

This document outlines the basic steps for verifying the operation of the Kubernetes (K8s) cluster before escalating on critical alerts.

1. Verify Cluster Health

Access the cluster using the kubectl command-line tool.
Use the appropriate kubeconfig file for the environment (dev, test, QA).

kubectl config use-context

- Check Cluster Nodes

 kubectl get nodes

Look for any nodes in NotReady, Unknown, or SchedulingDisabled status. Investigate any anomalies.

- Check System Pods

Verify critical system pods in the kube-system namespace are running
```
 kubectl get pods -n kube-system
```

Pay special attention to core components:

kube-apiserver
kube-scheduler
kube-controller-manager
etcd

2. Verify Namespace and Application Status

- List All Namespaces

 kubectl get namespaces

- Check Pod Status in Active Namespaces

 kubectl get pods -n <namespace>

- Inspect Deployments and ReplicaSets

 kubectl get deployments -n <namespace>
kubectl get rs -n <namespace>

3. Verify Cluster Networking

- Service Connectivity

 kubectl get svc -n <namespace>

- DNS Resolution

 kubectl exec -n kube-system <coredns-pod> -- nslookup kubernetes.default

- Ingress/Load Balancer

 kubectl get ingress -n <namespace>

4. Monitor Resource Utilization

Node Resource Usage

 kubectl top nodes

Pod Resource Usage

 kubectl top pods -n <namespace>

5. Check Cluster Events

 kubectl get events --sort-by='.metadata.creationTimestamp'

6. Validate Logging and Monitoring

Ensure logging systems (e.g., EFK, Fluentd) are collecting logs as expected.
Verify monitoring dashboards (e.g., Prometheus, Grafana) for anomalies or missing metrics.

7. Escalation Criteria

Core system pods (kube-apiserver, etcd) are not running.
A node remains in NotReady for more than 10 minutes.
Pods in critical application namespaces are in CrashLoopBackOff or Error state without resolution.
Networking issues persist and cannot be resolved by basic checks.
Cluster resource utilization is critically high (e.g., >90% CPU or memory).

8. Document Findings

Record all findings in the incident management system, including:
Steps taken during the checkout procedure.
Observed errors or anomalies.
Commands run and their output (if significant).

End of Document

What information online helped you figure this out? What blogs or tools did you use?

https://kubernetes.io/docs/tasks/debug/debug-cluster/
https://kubernetes.io/docs/concepts/
From conducting study-groups and working with K3s

What did you learn in this process of writing this up?

Kubernetes is an Operating System with similar failure modes to that of Linux
Kubernetes maintains helpful logs
Kubernetes gives helpful errors
KubeCTL is the tool I should internalize and master
KubeCTL can manage multiple clusters remotely

Digging Deeper ⛏️

Work more in the lab to build a container of your choice and then find out how to deploy that into your cluster in a secure, scalable way. 👍

I have actually experimented with building out custom containers using Podman and using:

 podman commit

to export the image as a custom local image. Then, I run that image and input the command:

 podman generate kube my-container > my-container.yaml

This creates a manifest yaml that can then be spun up with Kubernetes.

Read this about securing containers: https://docs.docker.com/build/building/best- practices

I do have prior experience with this having completed the DevSecOps certification from TryHackMe, it covers containers security with red-team exercises. However, it does not cover a ton of defensive measures. This best practices document is going to be really helpful when setting up future containers.

Do this to practice securing those containers. https://killercoda.com/killer- shell-cks/scenario/static-manual-analysis-docker 👍

Read these about securing Kubernetes Deployments:

https://kubernetes.io/docs/concepts/security/ 👍 and https://kubernetes.io/docs/concepts/security/pod-security-standards/

TLS
Secrets API
Runtime Classes
Network Policies
Admission Policy
Pod Security standards
Open Policy, assumed safe privileged user.
Baseline Policy
SELinux enforce Policy
Do this lab to practice securing Kubernetes: https://killercoda.com/killer- shell-cks/scenario/static-manual-analysis-k8s

So checking the privilege status indicates a container is privleged or not. If privleged, move to insecure directory.

Labs 🥼🧪

Warmup

 curl -sfL https://get.k3s.io > /tmp/k3_installer.sh 👍
more /tmp/k3_installer.sh 👍

What do you notice in the installer?

What checks are there?

 grep -i arch /tmp/k3_installer.sh

What is the name of the variable holding the architecture?

How is the system finding that variable?

 uname -m

Verify your system architecture

 grep -iE “selinux|sestatus” /tmp/k3_installer.sh

Does K3s check if selinux is running, or no? Yes 👍

Installing k3s and looking at how it interacts with your Linux system

Install k3s curl -sfL https://get.k3s.io | sh -

What was installed on your system?

 rpm -qa –last | tac

Your tasks in this lab are designed to get you thinking about how container deployments interact with

Verify the service is running
```
 systemctl status k3s
```
Check it systemd configuration
```
 systemctl cat k3s
```

See what files and ports it is using

 ss -ntulp | grep <pid from 3>
lsof -p <pid from 3>

#Do you notice any ports that you did not expect?

Verify simple kubectl call to API
```
 kubectl get nodes
```
Verify K3s is set to start on boot and then cycle the service
```
 systemctl is-enabled k3s
systemctl stop k3s
```

Recheck your steps 3-6

What error do you see, and is it expected?

What is the API port that kubectl is failing on?

 systemctl start k3s

Verify your normal operations again.

Looking at the K3 environment

Check out components

 kubectl version
kubectl get nodes
kubectl get pods -A
kubectl get namespaces
kubectl get configmaps -A
kubectl get secrets -A

#Which namespace seems to be the most used?

Creating Pods, Deployments, and Services

It’s possible that the lab will fail in this environment. Continue as you can and identify the problem using

the steps at the end of this section.

Create a pod named webpage with the image nginx
```
 kubectl run webpage --image=nginx
```
Create a pod named database with the image redis and labels set to tier=database
```
 kubectl run database --image=redis --labels=tier=database
```

Create a service with the name redis-service to expose the database pod within the cluster on port

 6379 (default Redis port)
kubectl expose pod database --port=6379 --name=redis-service --
type=ClusterIP

Create a deployment called web-deployment using the image nginx that has 3 replicas
```
 kubectl create deployment web-deployment --image=nginx --
replicas=3
```

Verify that the pods are created

 kubectl get deployments
kubectl get pods

Create a new namespace called my-test
```
 kubectl create namespace my-test
```
Create a new deployment called redis-deploy with the image redis in your my-test namespace with 2
```
 replicas
kubectl create deployment redis-deploy -n my-test --image=redis -
-replicas=2
```

Do some of your same checks from before. What do you notice about the pods you created? Did they all work?

If this breaks in the lab, document the error. Check your disk space and RAM, the two tightest constraints in the lab. Using systemctl restart k3s and journalctl -xe can you figure out what is failing? (Rocky boxes may have limitations that cause this to not fully deploy, can you find out why?)

Reflection questions:

What questions do you still have about this week?

I would like to know more about Kubernetes and how it can be effectively managed, but this is just a matter of further study.

How can you apply this now in your current role in IT? If you’re not in IT, how can you look to put something like this into your resume or portfolio?

I installed K3S on my personal lab and ran through two days of exercises with the study group, gaining some competency with installing, configuring and deploying a set of pods from manifests. So I could confidently state that I know how to do this and check on the health of the system.

ProLUG Links ⛓️

ProLUG Admin Course Unit 9 🐧

Tue, 12 Nov 2024 00:00:00 +0000

Containers & Kubernetes 🦭

One of the most exciting units for me has been exploring deployment and hosting infrastructure. I’ve already spent some time working with Docker and Podman, but I have had limited hands-on experience with Kubernetes. Before this unit, I completed a few interactive Kubernetes labs on Killercoda, covering basic commands, information gathering, and logging.

This week, I followed an interactive K3s lab that guided me through the installation process—a perfect refresher. Afterward, I jumped onto one of my Proxmox VMs and installed K3s on my homeLab 👨‍🔧

Discussion Post 1

It’s a slow day in the NOC and you have heard that a new system of container deployments are being used by your developers. Do some reading about containers, docker, and podman.

1. What resources helped me answer these questions

https://www.redhat.com/en/topics/containers
My RHCSA 9 Course on Udemy
Notes I have composed in LogSeq from multiple sources
Julia Evans Blog https://jvns.ca/

2. What did you learn about that you didn’t know before?

I did not know that Podman and Kubernetes can run WASM Applications alongside containers. I have some interest in WASM and Subsequent ByteCode, having read about it on occasion.
I did not know that Podman containers can be converted to SystemD services.
I learned a technique that I quite like using podman commit to create a custom compose file from a modified container and will likely use this alot.

Terminology that I wasn’t familiar with:

Control Plane: Manages container orchestration, monitoring, and state across cluster nodes.
The API server: Core interface for communication between users and container clusters.
Scheduler: Assigns containers to nodes based on resource availability and policies.

3. What seems to be the major benefit of containers?

Can be declarative with compose. So infrastructure can be explicitly defined and easily rebuilt.
Light weight / low resource. Containers are not complete systems and are stripped to the bare essentials, meaning they are very small files that run fast.

4. What seems to be some obstacles to container deployment?

From my own experience, mounting persistent volumes was a bit tricky.
Container networking presents a challenge bot conceptually and practically.
Packaging application to run harmoniously within a container environment presents some friction.
Large infrastructure must be broken into microservices, introducing complexity.

Discussion Post 2

You get your first ticket about a problem with containers. One of the engineers is trying to move his container up to the Dev environment shared server. He sends you over this information about the command he’s trying to run.

[developer1@devserver read]$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[developer1@devserver read]$ podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost/read_docker latest 2c0728a1f483 5 days ago 68.2 MB
docker.io/library/python 3.13.0-alpine3.19 9edd75ff93ac 3 weeks ago 47.5 MB
[developer1@devserver read]$ podman run -dt -p 8080:80/tcp docker.io/library/httpd
You decide to check out the server
[developer1@devserver read] ss -ntulp
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
udp UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=166693,fd=13))
tcp LISTEN 0 80 127.0.0.1:3306 0.0.0.0:* users:(("mariadbd",pid=234918,fd=20))
tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=166657,fd=3))
tcp LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=166693,fd=14))
tcp LISTEN 0 4096 *:8080 *:* users:(("node_exporter",pid=662,fd=3))

1. What do you think the problem might be? 🔍

There is a container call node exporter that is listening on port 8080, therefore the port is already in use. I think this is a pretty common issue as this port is normally used for public traffic. With many nodes running it is easy to double assign a port.

2. How will you test this? 🤔

I would run the conflicting container run command with a slight port change. podman run -dt -p 8081:80/tcp docker.io/library/httpd

3. The developer tells you that he’s pulling a local image, do you find this to be true, or is something else happening in their run command?

It is true that once and image is pulled, it is stored locally. So the developer may have pulled the image. However in the command he is specifying a source for pulling a fresh container, so the dev is definitely sus. Typically if the image has been pulled it is given a container ID, which is then used to build with.

Installing K3s in my HomeLab 👍

After completing the suggested lab, I took note of installation process and replicated it on my HomeLab. I just wanted to share my process and bumps.

1. Installation from curl script

curl -sfL https://get.k3s.io | sh -

2. Making sure the service is running in systemD

systemctl status k3s

3. Changing config permissions (Problem that had stumped me)

sudo chmod 644 /etc/rancher/k3s/k3s.yaml

4. Deploying a pod

kubectl run nginx –image=nginx:alpine

5. Making an alias so commands are less annoying

alias k=kubectl

6. Checking the running pod

k get pods

Digging Deeper ⛏️

1. See if you can get a deployment working in the lab.

Screen shots from this deployment

What worked well?

creating a persistent volume and attaching to the volume worked well. mkdir /root/TreasuresVolume Pulling and building an image worked well. Attaching to the container and interacting with it went well. podman run -dit –name TreasuresContainer -v /root/TreasuresVolume:/app docker.io/library/golang:alpine tail -f /dev/null apk add vim gcc bash

What did you have to troubleshoot?

I ran into trouble when exiting the container. It will kill the container, forcing me to start it again, which was frustrating. Having done this in the past, I thought this would not be an issue. However, I just learned that containers without continually running services will die when exited. The fix for such an issue is to run something light and persistent on the container in order to keep it alive. This can be accomplished with a bash script or turning a go binary into a system binary to keep running.

What documentation can you make to be able to do this faster next time?

Actually, this blog is used to partially keep track of this. I use LogSeq as a second brain, where I will definitely copy and paste this info into my Podman section.

Reflection Questions🤔

1. What questions do you still have about this week?

I would like to know more about small scale deployments of Kubernetes such as the one I deployed using Rancher and K3S. More specifically I would like to know what limitations a small scale deployment has versus a multi machine / multi node system.

2. How can you apply this now in your current role in IT? If you’re not in IT, how can you look to put something like this into your resume or portfolio?

This content is highly applicable to my intended work as deploying services is becoming increasingly important, as evidenced by the requirement for knowledge of Podman in the RHCSA 9. I already have some container deployment work to display to prospective employers atm. I plan to further my knowledge in this area, especially in regard to Kubernetes as this seems to be the operating system of the internet.

ProLUG Links ⛓️

ProLUG Admin Course Unit 8 🐧

Wed, 30 Oct 2024 00:00:00 +0000

Scripting System Checks

Once again beyond the Discussion Posts and Labbing. I spent a great deal of time scripting/programming System checks. After completing the labs which Bash Scripting and intro to ‘C’, I got really into GO as a system util. I have a particularly productive day with using the embed.fs feature of GO and packing unix system tools together in a single go program at compilation. I think there is a ton of potential here for my own uses. 👨‍🔧

Discussion Post 1

Scenario

It’s a 2-week holiday in your country, and most of the engineers and architects who designed the system are out of town. You’ve noticed a pattern of logs filling up on a set of web servers due to increased traffic. Research and verification show that the logs are being sent off in real time to Splunk. Your team has been deleting the logs every few days, but a 3rd-shift engineer missed this in the notes, causing downtime. How might you implement a simple fix to stop-gap the problem until all engineering resources return next week?

Resources Used:
TryHackMe (Splunk) Intro
Study Group discussion
ChatGPT
Blogs:
Nohl’s Tech Resources: Log Files in tmpfs Without Breaking Logging
DietPi Documentation: Log System

Why can’t you just make a design fix and add space in /var/log on all these systems?

Adding more space to /var/log might be a design fix, but it isn’t feasible in the short term due to:

Operational Constraints: Extending storage may involve downtime, additional permissions, or architectural changes that can’t be approved without the primary engineers.
Temporary Nature of the Fix: Increasing space only delays the issue. If logs continue to grow, the problem will recur once space is exhausted again.

Why can’t you just make a design change and use logrotate more frequently?

Possibility of Log Loss: Higher logrotate frequency could still miss high-frequency log spikes, especially during unusual traffic peaks, risking logs being deleted before Splunk ingestion is complete.
Configuration and Testing: Aggressive logrotate adjustments may interfere with processes expecting logs at specific retention periods. Testing changes in production without key team members isn’t ideal.

Temporary Fix Options

To address the issue, consider implementing a temporary fix by configuring a log retention policy that aggressively compresses or truncates logs without disrupting active processes. Here are some potential approaches:

Implement a Temporary Cron Job

Schedule a cron job to truncate logs on a more aggressive schedule without deleting them. For example:

> /var/log/access.log

```bash

 > /var/log/access.log

This would empty the log file without removing it or impacting the active file descriptors held by any running processes.

Set Up Temporary Log Compression

Compress the logs after truncation if additional space savings are needed. Tools like gzip can compress logs efficiently, reducing disk space usage and ensuring logs are still accessible if required for audits or incident investigations.

Implement a RAM Disk for Temporary Logs

As a short-term measure, you could set up a RAM disk for logs that don’t need long-term retention. This allows logs to be stored temporarily in memory, reducing disk space pressure. For instance


 mount -t tmpfs -o size=512M tmpfs /var/log/temp

You could then configure lower-priority logs to write here temporarily, knowing they will be lost upon reboot, which may be acceptable in a crisis scenario.

Adjust Splunk Forwarder Configuration:

If possible, configure the Splunk forwarder to filter logs more aggressively, reducing the volume of logs that are retained on the system. The props.conf or inputs.conf files can be configured to forward logs without keeping local copies.

Adding more space to /var/log might be a design fix, but it isn’t feasible in the short term due to the following:

Operational Constraints: Extending storage could involve downtime, additional permissions, or changes that require architectural decisions that can’t be made without the primary engineers.
Temporary Nature of the Fix: Increasing space only delays the issue rather than preventing it. If the logs keep growing, the problem will recur once space runs out again.

Discussion Post 2

You are the only Linux Administrator at a small healthcare company. The engineer/admin before you left you a lot of scripts to untangle. This is one of our many tasks as administrators, so you set out to accomplish it. You start to notice that he only ever uses nested if statements in bash. You also notice that every loop is a conditional while true and then he breaks the loop after a decision test each loop. You know his stuff works, but you think it could be more easily written for supportability, for you and future admins. You decide to write up some notes by reading some google, AI, and talking to your peers.

Compare the use of nested if versus case statement in bash.

Nested if statements are useful for situations where each condition depends on the result of the previous test, requiring a hierarchy or sequence.
A case statement is ideal for handling multiple discrete values of a variable, especially if there are many possible branches. It’s typically cleaner and more readable than a nested if.

Compare the use of conditional and counting loops. Under what circumstances

Use conditional loops (while) when you don’t know the number of iterations in advance and need to loop based on conditions.
Use counting loops (for) when you have a set number of iterations or are working with a list. This structure is clearer and prevents issues that may arise from unintentional infinite loops.

would you use one or the other?

optimizing or refactoring Bash scripts the Engineer had left me.

I would replace nested if statements with case statements when possible to improve readability, especially when handling multiple discrete values.
Of course, I would comment things for added communication/ maintainability.
I would Limit while true loops to cases where no predictable count or list is available. Clearly define a break condition early to avoid infinite loops.
I would Use for loops for counting or iterating over arrays or lists, as they provide a clean structure with known iteration limits.

ProLUG Links ⛓️

ProLUG Admin Course Unit 7 🐧

Sun, 27 Oct 2024 00:00:00 +0000

Security

Patching the system/ Package Management

yum, dnf, rpm

As the course progresses, I am learning more deeply within a study group. Aside from the lab work and discussion posts. I have been putting a lot of hours satisfying curiosities regarding the linux system. For this unit we did a deep dive into packaging, going so far as to look at the history and reasoning before decision making. We had also looked at packages are managed within tightly controlled environments. I now feel that I have a robust understanding of the theory and practical elements of Redhat packaging and beyond.

Discussion Post 1:

Why software versioning is important. 🤔

Versioning enables you to monitor software updates systematically, making it easier to troubleshoot, roll back changes, and trace modifications for security or functionality purposes. 👍
One can manage dependencies confidently, avoiding conflicts between system components, libraries, and tools. crucial for stable and consistent deployments. 👍
We can verify package integrity, ensuring installed software hasn’t been altered or corrupted. Essential for maintaining a secure and stable system environment. 👍

Discussion Post 2:

Scenario:

You are new to a Linux team. A ticket has come in from an application team and has already been escalated to your manager. They want software installed on one of their servers but you cannot find any documentation and your security team is out to lunch and not responding. You remember from some early documentation that you read that all the software in the internal repos you currently have are approved for deployment on servers. You want to also verify by checking other servers that this software exists. This is an urgent ask and your manager is hovering. How can you check all the repos on your system to see which are active? 🤔 How would you check another server to see if the software was installed there? 🤔 If you find the software, how might you figure out when it was installed? (Time/Date) 🤔

Answer:

In an urgent situation like this, I’d first check which approved software repositories are active on my system, then verify if the software is already installed on similar servers to ensure it’s safe to proceed. Finally, I’d review the installation history to confirm when it was added. Working with Red Hat packaging and package management systems has many more options than I was expecting; through labbing in the study group, I’ve gained a much better understanding of packages, dependencies, and package management.

Packing was a pretty deep rabbit hole for me.

This is the process I’d follow for this case:

Check Active Repositories dnf repolist
Check if the Software is Installed on Your System rpm -qa | grep <software_name> or dnf list installed <software_name>
Check Another Server for Software Installation with SSH and Step 2 commands.
View Installation History (Time/Date)

dnf history info <transaction_id> dnf history list <software_name>

Discussion Post 3

(After you have completed the lab) - Looking at the concept of group install from DNF or Yum. Why do you think an administrator may never want to use that in a running system? Why might an engineer want to or not want to use that? This is a thought exercise, so it’s not a “right or wrong” answer it’s for you to think about.

Question:

What is the concept of software bloat, and how do you think it relates?

Answer:

Software bloat is when essential tools/packages are larger than they need to be, effecting performance, reliability and security. By performance I am referring to the loss of potential performance from unessecary resource use. In regards to reliability, more complex systems inherently have more potential to fail. Security means many things, so I am specifically thinking about attack surface and potential for vulnerability due to the aforementioned complexity.

Question:

What is the concept of a security baseline, and how do you think it relates?

Answer:

A set of minimum security standards and controls that organizations implement to protect systems.

Question:

How do you think something like this affects performance baselines?

Answer:

By targeting specific packages, tracking changes, reducing unnessecary dependancies and bloat, we satisfy the tenants of a security baseline by establishing consistency, simplifying compliance, enhancing efficiency and reduce risk.

ProLUG Links ⛓️

ProLUG Admin Course Unit 6 🐧

Mon, 21 Oct 2024 00:00:00 +0000

Security – Firewalld & UFW 🔥🧱

In this unit, we explore essential concepts in firewall management using Firewalld and UFW. A firewall acts as a security system, controlling the flow of traffic between networks by enforcing rules based on zones —logical areas with different security policies. Services are predefined sets of ports or protocols that firewalls allow or block, and zones like DMZ (Demilitarized Zone) provide added security layers by isolating public-facing systems. Stateful packet filtering tracks the state of connections, allowing more dynamic rules, while stateless packet filtering inspects individual packets without connection context. Proxies facilitate indirect network connections for security and privacy, while advanced security measures such as Web Application Firewalls (WAF) and Next-Generation Firewalls (NGFW) offer specialized protection against modern threats.

Lab Work 🧪🥼

Intro

This week, we dove deep into configuring and testing firewall settings in our Discord study group. I had several virtual machines set up in my ProxMox home lab, and we experimented while completing the lab work. As usual, we went on several tangents, verifying ideas. In total, we spent over 5 hours running commands, experimenting with different configurations, breaking things, and debating solutions. By the end of the session, I gained a practical understanding of Firewalld configuration, packet sending, and packet tracing with Wireshark. It was frustrating at times, but ultimately rewarding.

Sending & Receiving test packets experiment

1. So we set up one server to receive a packet:

 nc -l -p -u 12345 > received_file

2. Sending from another server and noted that the packet was recorded in received_file

 echo 'message1' | nc -u server_b_ip 12345

3. We set Firewalld to block udp port 12345

 sudo firewall-cmd --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.2.166" port protocol="udp" port="12345" accept' --permanent

4. So we set up one server to receive a packet:

 nc -l -p -u 12345 > received_file

5. Sending from another server:

 echo 'Blocked' | nc -u server_b_ip 12345

6. We then noted that this packet did not come through

7. We then set up netcat to listen for packets on udp port 12346 ‘slightly different’

 nc -l -p -u 12346 > received_file

8. Sending from another server and noted that the packet was recorded in received_file

 echo 'message2' | nc -u server_b_ip 12346

9. Then we busted open WireShark on the packet receiving VM and turned on general packet scanning

 echo 'mom' | nc -u server_b_ip 12346

10. We then looked at all of the ways we could inspect the packets including additional data bits over UDP for no apparent reason.

Fin

Types of Firewalls 🔍

Firewalld 🔥🧱

Uses zones to define the level of trust for network connections, making it easy to apply different security settings to various types of connections (like home, public, or work). It’s dynamic, meaning changes can be made without restarting the firewall, ensuring smooth operation.

Zones 🍱

The concept is specific to Firewalld. Zones are a predefined set of firewall rules that determine the level of trust assigned to a network connection. Zones allow you to apply different security policies to different network interfaces based on how much you trust the network.

Common Commands

 firewall-cmd --state

Checks if Firewalld is running.

 firewall-cmd --get-active-zones

Lists all active zones and the interfaces associated with them.

 firewall-cmd --get-default-zone

Displays the default zone for new interfaces or connections.

 firewall-cmd --set-default-zone=ZONE

Changes the default zone to the specified zone.

 firewall-cmd --zone=ZONE --add-service=SERVICE

Allows a service (e.g., SSH, HTTP) in the specified zone.

 firewall-cmd --zone=ZONE --remove-service=SERVICE

Removes a service from the specified zone.

 firewall-cmd --zone=ZONE --add-port=PORT/PROTOCOL

Opens a specific port (e.g., 80/tcp) in the specified zone.

 firewall-cmd --zone=ZONE --remove-port=PORT/PROTOCOL

Closes a specific port in the specified zone.

 firewall-cmd --reload

Reloads the Firewalld configuration without dropping active connections.

 firewall-cmd --list-all

Lists all the rules and settings in the active zone.

 firewall-cmd --permanent

Applies changes permanently (used with other commands to ensure changes persist after reboots).

 firewall-cmd --runtime-to-permanent

Converts the current runtime configuration to a permanent one.

Zone example

A public zone might have stricter rules, blocking most traffic except for essential services like web browsing.
A home zone could allow more open traffic, such as file sharing, because the network is more trusted.

UFW 🔥🧱

Uncomplicated Firewall is a user-friendly firewall designed to simplify the process of controlling network traffic by allowing or blocking connections. UFW is commonly used on Ubuntu and provides easy commands for setting up firewall rules, making it ideal for beginners. Despite it is simplicity, it is powerful enough to handle complex configurations.

Default Deny Policy 🔐

By default, UFW denies all incoming connections while allowing outgoing ones. This enhances security by requiring users to explicitly allow any incoming traffic.

Common Commands

 sudo ufw status

Displays the current status of UFW and active rules.

 sudo ufw enable

Enables the UFW firewall.

 sudo ufw disable

Disables the UFW firewall.

 sudo ufw default deny incoming

Sets the default policy to deny all incoming connections.

 sudo ufw default allow outgoing

Sets the default policy to allow all outgoing connections.

 sudo ufw allow PORT

Allows traffic on a specific port (e.g., sudo ufw allow 22 to allow SSH).

 sudo ufw deny PORT

Denies traffic on a specific port.

 sudo ufw delete allow PORT

Removes a previously allowed rule for a port.

 sudo ufw allow SERVICE

Allows traffic for a service by name (e.g., sudo ufw allow ssh).

 sudo ufw deny SERVICE

Denies traffic for a service by name.

 sudo ufw allow from IP

Allows traffic from a specific IP address.

 sudo ufw deny from IP

Denies traffic from a specific IP address.

 sudo ufw allow proto PROTOCOL from IP to any port PORT

Allows traffic for a specific protocol, source IP, and port (e.g., sudo ufw allow proto tcp from 192.168.1.100 to any port 80).

 sudo ufw reset

Resets all UFW rules to default.

 sudo ufw reload

Reloads UFW rules without disabling the firewall.

WAF 🔥🧱

Web Application Firewall is a security system designed to protect web applications by filtering and monitoring HTTP traffic between a web application and the internet. It helps prevent common web-based attacks like SQL injection, cross-site scripting (XSS), and cross-site request forgery (CSRF) by analyzing the incoming and outgoing traffic and blocking malicious requests. Unlike traditional firewalls that focus on network security, a WAF specifically targets the security of web applications and can be an important part of a layered defense strategy.

More Sophisticated 🍷

are generally more sophisticated than Firewalld or UFW because they operate at the application layer (Layer 7) of the OSI model. Blocking traffic is one thing, but packet inspection is another.

Quite a few ⚖️

There are many Web Application Firewalls out there that cover specific cloud platforms or web services. Here is a list of some popular ones:

AWS WAF
Cloudflare WAF
F5 Advanced WAF
Imperva WAF
ModSecurity
Barracuda WAF
Sucuri WAF
Akamai Kona Site Defender
Fortinet FortiWeb

NGFW 🔥🧱🧠

Next-Generation Firewall is an advanced type of firewall that goes beyond traditional firewall features like packet filtering. It combines standard firewall capabilities with more advanced functionalities such as deep packet inspection (DPI), intrusion prevention systems (IPS), and application-level control. NGWs can inspect and control traffic at a more granular level, allowing administrators to set security rules based on specific applications, users, or behaviors.

Features of a typical NGFW

Deep Packet Inspection (DPI): Examines the content of data packets, not just their headers, allowing the firewall to identify and block threats hidden in the traffic.
Intrusion Detection and Prevention System (IDS/IPS): Monitors network traffic for suspicious activity and can take action (like blocking or alerting) to prevent attacks in real-time.
Application Awareness and Control: Recognizes and manages specific applications (e.g., Facebook, Skype) regardless of port or protocol, allowing for fine-grained traffic control.
Advanced Malware Protection (AMP): Detects and blocks malware using both signature-based detection and behavioral analysis to prevent malware from entering the network.
SSL/TLS Decryption: Decrypts encrypted traffic (HTTPS) for inspection to detect threats hiding inside secure channels.
User Identity Integration: Applies firewall rules based on user identity or group membership rather than just IP addresses, providing more flexible access control.
Threat Intelligence Feeds: Uses real-time threat data from global databases to protect against emerging threats and malicious IP addresses or domains.
Cloud-Delivered Security: Provides scalable and flexible cloud-based protection services such as sandboxing, traffic analysis, and updates for zero-day attacks.
Virtual Private Network (VPN) Support: Allows secure, encrypted connections for remote users or between different networks (site-to-site or remote access VPNs).
URL Filtering: Controls access to websites based on categories (e.g., social media, gambling) or specific URLs, helping enforce acceptable use policies.
Quality of Service (QoS): Prioritizes certain types of traffic, ensuring that critical applications receive the necessary bandwidth and reducing congestion.
Zero-Trust Network Segmentation: Implements policies based on strict access control, ensuring that users and devices only access the resources they are explicitly permitted.
Sandboxing: Isolates suspicious files or code in a secure environment to detect malicious behavior without affecting the rest of the network.
Logging and Reporting: Provides detailed logs and reports on network traffic, blocked threats, and firewall activity for auditing and troubleshooting.

NGFW Products

Palo Alto Networks NGFW
Cisco Firepower NGFW
Fortinet FortiGate NGFW
Check Point NGFW
Sophos XG NGFW
Juniper Networks SRX Series NGFW
Barracuda CloudGen NGFW
SonicWall NGFW
WatchGuard Firebox NGFW
Forcepoint NGFW
PfSense NGFW

Experience with PFSense

I am familiar with PfSense as it is Open Source and popular among the Homelab enthusiasts because it is offers expansive features built upon FreeBSD which has killer networking.

Some limitations

Pre-packaged Features

Commercial NGFWs (e.g., Palo Alto Networks, Cisco Firepower) often come with built-in advanced features such as cloud-delivered threat intelligence, AI-powered threat detection, and sandboxing for zero-day threats. While pfSense can be extended with third-party packages, it doesn’t natively offer the same level of seamless integration or automation.

Unified Management

Commercial NGFWs typically provide a centralized management console for handling multiple firewalls across large networks. While pfSense can handle multiple installations, managing them requires more manual effort and may not be as streamlined as the enterprise-grade management consoles of commercial NGFWs.

Enterprise Support

pfSense relies on community and third-party support, whereas commercial NGFWs offer direct vendor support with service level agreements (SLAs), which can be crucial for large enterprises needing guaranteed response times and assistance.

Threat Intelligence

NGFWs like those from Palo Alto or Cisco often integrate with real-time global threat intelligence networks, offering constant updates about emerging threats. While pfSense can be configured with tools like Snort for intrusion detection, it lacks the built-in, cloud-powered intelligence found in commercial NGFWs.

Case Study #1 🤔

A ticket has come in from an application team. Some of the servers your team built for them last week have not been reporting up to enterprise monitoring and they need it to be able to troubleshoot a current issue, but they have no data. You jump on the new servers and find that your engineer built everything correctly and the agents for node_exporter, ceph_exporter and logstash exporter that your teams use. But, they also have adhered to the new company standard of firewalld must be running. No one has documented the ports that need to be open, so you’re stuck between the new standards and fixing this problem on live systems.

1. Initial Research

Findings:

node_exporter typically uses port 9100
ceph_exporter may use port 9128
logstash commonly uses ports 5044

2. Checking Configs

A. LogStash Config

 cat /etc/logstash/conf.d/

B. node exporter config

 cat /etc/systemd/system/node_exporter.service

C. ceph exporter config

 cat /etc/systemd/system/ceph_exporter.service

Or

3. Gathering Socket Statistics

 sudo ss -tuln | grep LISTEN

Options Breakdown

-t: Show TCP sockets. -u: Show UDP sockets. -l: Show listening sockets, i.e., those waiting for incoming connections. -n: Show the output numerically, without resolving service names (e.g., display IP addresses and port numbers instead of domain names or service names like “http”).

Q: 1. As you’re looking this up, what terms and concepts are new to you?

Basically all of the concepts used are new to me. I am not very well versed in networking, network scanning or inspecting service configs. So this became a research and practice exercise that has shown me quite a lot of new tricks.

Q: 2. What are the ports that you need to expose? How did you find the answer?

Theoretically I would expose port 9100, 9128, 5044 from research. Furthermore, I now know how to check configs and/or gathering sockets statistics

Q: 3. What are you going to do to fix this on your firewall?

I would add these services to the internal zone:

 firewall-cmd --zone=ZONE --add-service=SERVICE

Allows a service (e.g., SSH, HTTP) in the specified zone.

Case Study #2 🤔

A manager heard you were the one that saved the new application by fixing the firewall. They get your manager to approach you with a request to review some documentation from a vendor that is pushing them hard to run a WAF in front of their web application. You are “the firewall” guy now, and they’re asking you to give them a review of the differences between the firewalls you set up (which they think should be enough to protect them) and what a WAF is doing.

Q: 1. What do you know about the differences now?

Traditional Firewalls (firewalld/ufw):

Operate at network and transport layers (OSI Layer 3 & 4).
Control traffic based on IP addresses, ports, and protocols.
Block or allow entire network connections.

Web Application Firewalls (WAFs):

Operate at the application layer (OSI Layer 7).
Inspect HTTP/HTTPS traffic, focusing on web application security.
Protect against attacks like SQL injection, XSS, and other web vulnerabilities.

Q: 2. What are you going to do to figure out more?

Dedicate time to researching effective WAF solutions.
Identifying suitable solutions at 3 budget scales.
Try to understand the additional labour behind management additional tools.

Q: 3. Prepare a report for them comparing it to the firewall you did in the first discussion.

Report 🗒️

Evaluation of Implementing a Web Application Firewall WAF

Prepared by: Treasure

Date: Oct 20th 2024

Subject: Evaluation of WAF Implementation Suitability and Comparison with Traditional Firewalls

1. Introduction

This report has been prepared in response to a request to evaluate the suitability of implementing a Web Application Firewall (WAF) within our infrastructure. The aim of this report is to:

Compare WAF technology with traditional firewall solutions currently implemented.
Assess the benefits and limitations of each.
Provide recommendations based on the findings.

2. Objectives

The key objectives of this evaluation are:

To determine the suitability of WAF in enhancing our web application security.
To identify potential risks and benefits associated with the deployment of WAF.
To compare traditional firewall solutions with WAF in terms of functionality, security, and cost.
To make recommendations based on the current and future needs of our IT infrastructure.

3. Comparison of Technologies

3.1 Traditional Firewalls (firewalld/ufw)

Primary Function: Control and filter network traffic based on IP addresses, ports, and protocols.
Strengths:
- Blocks unwanted connections at the network level.
- Suitable for general network protection.
- Easy to configure and manage.
Limitations:
- Does not inspect web traffic at the application level.
- Cannot protect against specific web application attacks (e.g., SQL injection, XSS).

3.2 Web Application Firewalls (WAF)

Primary Function: Protect web applications by filtering and monitoring HTTP/HTTPS traffic.
Strengths:
- Protects against common web application vulnerabilities (e.g., SQL injection, XSS).
- Monitors web traffic to block malicious requests.
- Can provide real-time threat detection and logging.
Limitations:
- May require more resources and specialized configuration.
- Focused solely on web applications, not general network traffic.

3.3 Key Differences

Feature	Traditional Firewall	Web Application Firewall (WAF)
Layer	Network (Layer 3/4)	Application (Layer 7)
Traffic Type	IP, ports, protocols	HTTP/HTTPS, web requests
Use Case	General network security	Web application protection
Threat Coverage	Blocks IP-based threats	Mitigates web vulnerabilities (SQLi, XSS)
Cost	Typically lower	Generally higher due to specialized focus

4. Key Considerations for WAF Implementation

4.1 Security Benefits

Enhanced protection against web-specific attacks.
Ability to monitor and block suspicious activity in real-time.
Added layer of security on top of traditional network firewalls.

4.2 Cost Analysis

Initial Investment: The upfront cost of acquiring and configuring a WAF solution.
Ongoing Costs: Maintenance, updates, and potential personnel training.

4.3 Operational Impact

May require additional resources for setup, monitoring, and incident response.
Potential need for collaboration between the security and development teams to ensure smooth integration.

5. Risk Assessment

Without WAF: Increased vulnerability to web application-specific threats, such as cross-site scripting (XSS) and SQL injection, especially for critical applications.
With WAF: Increased security for web applications but requires ongoing monitoring and adjustment to ensure performance and efficacy.

6. Recommendations

Based on the evaluation, I recommend the following:

Implement a WAF: Due to the increasing reliance on web applications and the rise in web-based attacks, implementing a WAF would provide an essential layer of security.
Maintain Traditional Firewalls: Existing firewalls should continue to be used for network-level protection alongside the WAF.
Pilot Implementation: Begin with a limited deployment of WAF for high-risk applications to evaluate performance and cost before a full-scale rollout.
Staff Training: Ensure the security and IT teams are trained in WAF management to maximize its effectiveness.

7. Conclusion

The implementation of a Web Application Firewall is a strategic move to protect our web applications from evolving security threats. While traditional firewalls remain crucial for network security, they cannot defend against the types of attacks WAFs are designed to mitigate. By implementing both WAF and traditional firewall solutions, we can ensure comprehensive security coverage across both network and application layers.

8. Next Steps

Further evaluation of potential WAF solutions based on budget, compatibility, and scalability.
Engage with the security team for a detailed implementation plan.
Prepare a pilot program for critical applications and monitor its performance.

Approved by: Bob Saggit

Date: October 25th 2024

Definitions

Firewall: A security device that monitors and controls incoming and outgoing network traffic based on predetermined security rules.
Zone: A defined area within a network that contains systems with similar security requirements, separated by a firewall.
Service: A specific type of network functionality, like HTTP or DNS, that can be allowed or blocked by a firewall.
DMZ: A “Demilitarized Zone” is a network segment that serves as a buffer between a secure internal network and untrusted external networks.
Proxy: A server that acts as an intermediary for requests between clients and servers, often used for filtering, security, or caching.
Stateful packet filtering: A firewall feature that tracks the state of active connections and makes filtering decisions based on the connection’s state.
Stateless packet filtering: A type of firewall filtering that analyzes each packet independently without considering the state of the connection.
WAF: A Web Application Firewall that protects web applications by filtering and monitoring HTTP/HTTPS traffic for threats like SQL injection and XSS.
NGFW: A Next-Generation Firewall that combines traditional firewall functions with additional features like application awareness, integrated intrusion prevention, and advanced threat detection.

ProLUG Links ⛓️

ProLUG Admin Course Unit 5 🐧

Fri, 18 Oct 2024 00:00:00 +0000

Managing Users & Groups

The overarching theme of this Unit is in the title, we are looking at Managing Users & Groups. Managing users and groups in Linux within an enterprise involves creating, modifying, and organizing user accounts and permissions to enforce security and control over resources.

Organizing permissions to enforce security is more important than it has ever been, as we live in a hyper connected world with many bad actors and large amounts of sensitive data.

Linux is fundamentally well suited for Managing Users & Groups because permissions permeate every aspect of a Linux environment. Everything is a file and every file has associated permissions, Therefore we have granular control over the comings and goings of users as administrators.

Lab Work 🧪🥼

Primary Commands / Tools

alias: Creates a shortcut or alias for a command.
unalias: Removes an alias that was previously defined.
awk: A powerful text-processing tool used for pattern scanning and processing.
useradd: Adds a new user to the system.
vi .bashrc: Opens the .bashrc file in the vi editor to customize shell settings.
UID_MIN 1000: The minimum user ID value for normal users (as defined in /etc/login.defs).
UID_MAX 60000: The maximum user ID value for normal users (as defined in /etc/login.defs).
groupadd –g 60001 project: Creates a new group named “project” with a GID of 60001.
id user4: Displays the user ID (UID), group ID (GID), and group memberships of user “user4.”

etc directories

Looking at etc directories relating to Users, Groups and Associated Security
```
 /etc/passwd
```
contains essential information about users, including their username, user ID (UID), group ID (GID), home directory, and default shell, with each entry separated by a colon.
```
 /etc/group
```
stores group information, listing each group’s name, group ID (GID), and its members, with each entry separated by a colon, allowing users to belong to one or more groups for access control purposes.
```
 /etc/shadow
```
contains encrypted password information and related security details for user accounts, such as password aging and expiration
```
 /etc/gshadow
```
stores encrypted passwords for group accounts, as well as information about group administrators and members, providing enhanced security for group access by restricting who can modify or access specific group data.
```
 /etc/login.defs
```
configuration settings for user account creation and login parameters, such as password aging policies, UID and GID ranges, and the default paths for user home directories, helping to control system-wide authentication behavior.
```
 /etc/skel/
```
provides template files that are automatically copied to a new user’s home directory when the user is created, ensuring they have default configuration settings.

Other interesting directories

Brief Description
Associated permissions
```
 /etc/fstab
```
This file contains information about disk partitions and other block storage devices and how they should be automatically mounted during the boot process.
Permissions: Usually -rw-r–r– (readable by all users, writable only by the root).
```
 /etc/hostname
```
This file stores the system’s hostname, which is a unique identifier for the machine in a network.
Permissions: Usually -rw-r–r– (readable by all users, writable only by the root).
```
 /proc
```
This is a virtual filesystem that provides detailed information about processes and system resources. It does not contain actual files but rather system and process information in real-time.
Permissions: dr-xr-xr-x (readable and executable by all users, writable only by root).
```
 /boot
```
Contains the kernel, initial ramdisk, and bootloader files needed to start the system.
Permissions: drwxr-xr-x (readable and executable by all users, writable only by root).
```
 /root
```
This is the home directory for the root user (the system administrator).
Permissions: drwx—— (only root has read, write, and execute permissions).
```
 /usr/bin
```
Contains binary executables for user programs.
Permissions: drwxr-xr-x (readable and executable by all users, writable by root).

Mapping unknown infrastructure 🗺️🤔

Objectives:

Map the Internal ProLUG Network (192.168.200.0/24):
Map the network from one of the rocky nodes.
Using a template that you build or find from the internet
Provide a 1 page summary of what you find in the network.

Approach 🤔

A briefing on the infra. 🔍🖥️

Het’ server is unique to me. He uses an injest system that makes a jump to the actual server for security purposes. Within the main server we have a warewulf managed cluster running a series of Rocky Linux VM’s. ⛰

Since we will be doing this from one of the Rocky Nodes within the system, the jump server will not be an issue I recon. 🤔

Mapping Strategy 📍🗺️

So I shouldn’t just pop into the server and go willy nilly with scanning commands. This sever is managed by someone who understands security. So it is best to do some Dead Reckoning beforehand.

The Basic Commands ⌨️

Mapping the remote servers open ports with nmap. nmap stands for Network Mapper. A quick perusal of the man page states it is an exploration tool and security / port scanner. It was designed to rapidly scan large networks. Nmap uses raw IP packets in novel ways to determine what hosts are available on the network, what services (application name and version) those hosts are offering, what operating systems (and OS versions) they are running, what type of packet filters/firewalls are in use, and dozens of other characteristics.

In order to Map the network I am going to use the ip address for the argument/target of the nmap command.

nmap <server-ip>

Checking mounted filesystem’ with df. stands for Display Free disk space. It is a utility that displays statistics about the amount of free disk space on the specified mounted file system or on the file system of which file is a past.

I will be using the -h flag for human readable format, which ads unit suffixes eg. Gibibyte (GiB) to the ends of values. There is more to it, but I am mentioning the bread and butter of the flag.

df -h

Digging Deeper ⛏️

Getting to know nmap 🫶

nmap is actually a big command, by big I mean the number of options and capabilities are vast. It is quite popular with pen testers and is packaged with Kali Linux – a security analysis and exploitation focused distribution of Linux, so best believe it is something important. This essentially means I should take some time to get to know it more.

Stealth 🥷

Beware that nmap can and will trigger detection software like an active firewall, because nmap is conducting Funny Bizniz by way of packet trickery inside a network, both are technical terms. Luckily there is a stealth option ( -s ) that enables the mapping to take place un-detected –for the most part.

Contrary to what I assumed, the lower case s does not even stand for stealth, though it still helps as a mnemonic. No, it actually stands for SYN and SYN stands for Synchronize. It is part of the TCP three-way handshake, which is a process used to establish a reliable connection between two devices on a network and has nothing to do with guys meeting at a bar.

I thought we were trying to be stealthy, not synchronized 🤔 Well is actually a form of Funny Bizniz wherein SYN is sent and never acknowledged, thustly not completing the handshake process and therefore hiding activity somehow

but how? Well… it reduces the chance of being logged by the target system’s monitoring tools, such as firewalls or intrusion detection systems. Now we both know.

Keep in mind this is only one command option, just imagine how deep the rabbit hole goes.

Important Mapping Command List

Ninjas mark stealthier techniques.

#	Command	Description
1	`nmap -sS <target>` 🥷	TCP SYN scan (stealth mode)
2	`nmap -sT <target>`	TCP connect scan (full connection)
3	`nmap -sA <target>`	ACK scan to detect firewalls
4	`nmap -sU <target>`	UDP scan
5	`nmap -sP <target>`	Ping scan to detect live hosts
6	`nmap -sV <target>`	Detect service versions on open ports
7	`nmap -O <target>`	OS detection
8	`nmap -A <target>`	Aggressive scan (OS, version, scripts)
9	`nmap -Pn <target>` 🥷	Disable ping (stealthy, avoid detection)
10	`nmap -p- <target>`	Scan all 65,535 TCP ports
11	`nmap --top-ports 100 <target>`	Scan the top 100 most common ports
12	`nmap --script <script> <target>`	Use Nmap scripts for detailed scanning
13	`nmap -sC <target>`	Run default NSE scripts
14	`nmap -sW <target>`	TCP window scan
15	`nmap -T4 <target>`	Faster scan using timing template T4
16	`nmap -v <target>`	Enable verbose output
17	`nmap -oN scan.txt <target>`	Save output in normal format
18	`nmap -oX scan.xml <target>`	Save output in XML format
19	`nmap -6 <target>`	Scan IPv6 targets
20	`nmap -D RND:10 <target>` 🥷	Decoy scan to mask source of scan

The Actual Mapping

🚀 Accessing the Secure Jump Point

The network is safely tucked behind a Dynamic Domain Name System (DDNS) running on an Asus Router. This setup allows access from a WAN, in this case, The Internet (if you’ve heard of it 🌐). It securely gate-keeps the network via login credentials, because let’s be honest, the Internet can be a scary place with bots… and sometimes people 👀.

So, what exactly does DDNS do? Well, I’m glad you asked! Most home internet connections don’t offer a static IP address, which makes hosting things tricky because the IP will randomly change. The IP is dynamic for a few reasons, including cost savings and the limited availability of IPv4 addresses. Anyway, I’m getting off track 🛤️. A DDNS monitors this dynamic IP and links it to a stable address that stays fixed.

TL;DR: DDNS bonds a dynamic IP with a fixed address, offering the added bonus of hiding the internal IP from bad actors, or as I like to call them, the baddies in London 🕵️‍♂️. So, this network has a DDNS gateway in place for that extra layer of security 🔒.

🎯 Jumping into a Node

Once a credentialed fellow enters their login while hanging at the gateway, a list of servers appears, like a digital menu 🍽️. From this list, one can choose where to jump to. In my case, I leapt to Rocky12, a node within a managed cluster. I knew a little about this network thanks to earlier sessions, but most of this can also be discovered by doing some network scanning 🕵️.

🔍 Doing a Broad Inventory Scan

This is where the magic of nmap comes into play! I used two stealthy commands: nmap -sS and nmap -sT. The sT option scanned a wide range of ports and connections, giving me a detailed list that I piped into less for easier viewing 📜.

🎯 Doing Targeted Scans

I grepped all the IP addresses and ran an -sS SYN Scan to gather more details on each node. These scans revealed loads of information about open ports and, in some cases, the hostnames of devices 🎯.

🧩 Identifying Devices

I quickly identified the warewulf orchestration device and all of the Rocky nodes. However, a few mysterious devices needed some detective work 🔎. I noticed glrpc listed as a service while scanning six addresses—three on one IP range and three on another. A quick Google search revealed that glrpc relates to GlusterFS, a filesystem specific to Red Hat systems. After watching a video explainer on GlusterFS, I figured out that these two IP ranges were likely a RAID or high availability configuration 💾.

🗺️ Mapping with Excalidraw

I initially created a highly technical Engineer’s map filled with data and presented it to Het for feedback. He advised me to think about how management would interpret it 🤔. So, back to the drawing board! I focused on improving the visual presentation and labeling, keeping in mind that management doesn’t care about the nitty-gritty; they just need a clear, high-level understanding during briefings 📊.

🎉 Wrapping Up

This concludes my exercise in mapping an unknown network! I learned a lot from this experience and am quite proud of the outcome 💪. It will undoubtedly come in handy when I find myself in future scenarios with many unknowns.

MITRE ATT&CK - 1 🪚

is a globally-accessible knowledge base of adversary tactics and techniques based on real-world observations.

Somewhat unfamiliar terminology for me

Lateral Movement

Lateral movement in the context of cyber exploitation refers to an attacker’s strategy of moving across a network to gain access to additional systems or sensitive data after initially infiltrating a single point. This involves leveraging compromised credentials, escalating privileges, or exploiting vulnerabilities to navigate between hosts and systems. The objective is often to broaden access within the environment while avoiding detection, eventually targeting critical infrastructure or data.

Exfiltration 🧳🚀

refers to the unauthorized transfer of data from a target system or network to an external location controlled by an attacker. This can involve methods such as encrypted tunnels, covert channels, or compromised accounts to avoid detection. Exfiltration is typically the final step in a cyberattack, allowing attackers to steal sensitive data, intellectual property, or credentials for further malicious activities.

Security is everybody’s issue

It is important to understand the data on your system regardless of specific responsibility in order to asses risk.

Impact of Exfiltration

The impacts of data exfiltration can be massive. Given the level of severity an exfiltration incident can entirely destroy an organization or cause financial losses, reputational damage, labour force diversion, data loss and jeopardizing future security.

The Linux User Environment

The Linux user environment is a customizable space that includes settings like environment variables, shell configurations, and startup scripts, typically defined in files such as .bashrc or .profile. These configurations allow users to tailor command line behavior, automate tasks, and create a personalized and efficient working environment.

Customizations to the User Environment

Customizations for the user environment in Linux might include setting up aliases for frequently used commands, configuring environment variables, and adding functions to streamline workflows. These changes can make repetitive tasks faster, improve the command line interface’s convenience, and adapt the environment to personal preferences.

Problems may arise

Problems around helping users with their dot files often stem from the diverse and sometimes incompatible changes users make to suit their needs. This can lead to issues like conflicting configurations, inconsistent behavior across systems, or difficult debugging when unexpected behaviors arise from custom scripts.

Definitions/Terminology

Footprinting 👣

Footprinting is an essential phase in ethical hacking and system security. It involves gathering information about a computer system, network, or organization to understand its structure and identify potential vulnerabilities. Footprinting is often the first step of a cyberattack or penetration test, allowing attackers or security professionals to map out an environment before deciding how to approach the next steps.

Scanning 🔍

Scanning is a process of actively probing systems or networks to identify open ports, services, and vulnerabilities. It’s used by attackers to gather deeper insights for potential exploitation and by security professionals to assess weaknesses. Common tools include Nmap and Nessus, while defenses include firewalls and IDS/IPS systems.

Enumeration 💯

Enumeration is the process of extracting more detailed information about a target, such as usernames, network shares, and system services, after identifying open ports and active systems. It typically involves active engagement with the target to gain in-depth knowledge that can be used for exploitation.

System Hacking 🪓

System hacking is the process of gaining unauthorized access to individual systems or networks by exploiting vulnerabilities. It involves activities such as password cracking, privilege escalation, installing backdoors, and covering tracks. Ethical hackers use these techniques to assess system security and recommend protective measures.

Escalation of Privilege 🥉🥈🥇

Privilege escalation is the process of gaining higher-level permissions or privileges than initially granted, allowing an attacker to execute commands with elevated authority. This can be achieved through exploiting vulnerabilities or misconfigurations, leading to unauthorized access to restricted resources or system control.

Rule of least privilege 🔐

The Rule of Least Privilege (LoP) is a security principle that states users, applications, and systems should only be granted the minimum level of access or permissions necessary to perform their tasks. This helps reduce the attack surface, limit potential damage from breaches, and mitigate insider threats.

Covering Tracks 🧹👣

Covering tracks is the process attackers use to hide their unauthorized activities and avoid detection. This involves techniques such as deleting or modifying system logs, using rootkits, and clearing command histories to prevent system administrators or security teams from discovering their presence or actions.

Planting Backdoors 🪴🚪

Planting backdoors involves installing hidden access points in a system, allowing attackers to bypass regular authentication and gain unauthorized access at a later time. Backdoors can be inserted through malicious code, vulnerabilities, or modifications to existing software, making them useful for maintaining persistent control over compromised systems.

ProLUG Links ⛓️

MITRE | ATT&CK Site Knowledge Base. ↩︎

GO CLI Utility

Tue, 15 Oct 2024 00:00:00 +0000

Making a Utility for Linux 🚀

Recently, I was chatting with fellow enthusiasts in the ProLUG group about creating system utilities with Go.

Between learning about it on my own time and getting input from others, I realized there are so many compelling reasons for using Go as a utility language. 🛠️

Currently, the idea isn’t as prevalent as Python or Bash, but it’s gaining traction. This makes sense since Go was designed with modern, interconnected systems in mind.

What is a Utility? 🤔

A system utility streamlines repetitive tasks by offering custom commands tailored to your workflow, boosting efficiency for system administrators and engineers. Work smarter, not harder. 💡

CLI Tool 🖥️

Go offers a package called Cobra CLI ¹, which enables the creation of command-line interfaces that take input from the terminal and execute predetermined logic. I’m putting this simply in case you’re unfamiliar with how programs work 😆.

The CLI is the simplest form of a computer program and is perfect for Linux system folks.

Core Idea 🎯

The core idea of this article is to explain how a Go program can act as a personalized utility that lives on your system or can be deployed across many. Go compiles code into a binary—native computer language. Once built, the binary runs reliably without failure, assuming the program logic is solid. This makes Go ideal for utilities that are called repeatedly to perform simple tasks.

Automate Everything 🤖

Managing systems involves a lot of repetitive tasks, and the goal of any sysadmin or engineer is to automate them. Tools like Ansible, Python, and Bash are already well-known for automation. However, there are specific reasons to use Go, which I’ll explore further in a future article titled “Why Go?” where I’ll break down why it belongs in your quiver of tools 🏹.

CLI Programming Portion ⚙️

Setup

Install Go and Cobra CLI (if not already installed).
Initialize a new Go project with Cobra CLI.

CLI Logic

Add commands and flags to the Cobra application.
Implement utility functions, such as disk usage checks or file listings.

Compilation and Installation 🏗️

Compile

Build the Go project into a single x86 Linux binary.
Test the binary locally to confirm functionality.

Install

Move the binary to /usr/local/bin for global access.
Set permissions on the binary to allow execution.

Add to PATH

Check if /usr/local/bin is in your PATH.
If not, add it to your .bashrc or .bash_profile.

Testing Portion 🧪

Run the utility from any location to confirm it works.
Optionally, add help documentation or more commands to the CLI.

Packaging 📦

Provide installation instructions if you plan to share it.
Upload the binary or source code to GitHub for distribution.

Learning Resources

Ok this was just a simple outline you say. Yes, I am trying to convey information simply for sanities sake, this is how I process things anyway.

In order to make your own utility you will have to go off on your own learning adventure. Below I am giving you some resources for getting started.

Getting up and running with GO

Brief Intro 2

For the impatient 3

There are many ways to get this done. For the clever and impatient, I suggest finding a pre-made CLI application that you can deconstruct. ³

For the Patient 4

For those who like to build a foundation of understanding, I suggest watching a lecture followed by a tutorial and finally doing the project. ⁴

Conclusion

I try to keep things simple and succinct, so this is where I ride off into the sunset. Good luck and happy learning.

Footnotes ⛓️

Cobra CLI Source Code GitHub Repository User: spf13, Current. ↩︎
GO in 100 Seconds Youtube Video Channel: Fireship, 2021. ↩︎
Cobra CLI Samples GitHub Repository User: Adron, 2022. ↩︎ ↩︎
Golang Tutorial Series Playlist Youtube Channel Channel: Net Ninja, 2021. ↩︎ ↩︎

ProLUG Admin Course Capstone Submission Requirements 🐧

Sun, 06 Oct 2024 00:00:00 +0000

Intro 👋

To complete the Professional Linux User Group (ProLUG) Professional Administrator Course (PAC), we are required to submit a final Capstone Project.

This article is a bit out of order, as I proactively chose a project topic before it was formally introduced. The purpose of this article is to document the project requirements as set out by our instructor.

Of course, I like to make slight modifications to ensure everything is neatly formatted and grouped—with plenty of emoji usage for extra flair! 😉

Step 1.Select a topic and do 📌

A) Write a brief intro to the project

B) Outline Technologies

C) Propose idea to Instructor

Step 2.Research 🤓

A) Confirm this is doable

B) Estimate time

C) Find snippets, manuals

Step 3.Plan Project 📍

A) Find Comparables

B) Document Procedure

C) Diagram Project

Step 4.Diagram ⧄

A) Create a System Diagram (ExcaliDraw)

B) Combine with Documentation

C) Export for use in Keynote

Step 5.Build 🛠️

A) Document Process

B) Create Demo

C) Essential Truth

Step 6. Documentation 📕

A) Create Redline Documentation

B) Format in a comprehensible way

C) Organize logically

Step 7. Prepare & Present 👨🏻‍🏫

A) Volunteer to Present

B) Create Keynote (15-20) Slides

i. project Purpose

ii. Diagram

iii. Build Process

iv. What did you learn?

v. What transferable skills did you gain?

Deliverables 🔑

✔️ 1. Comprehensive Documentation 📕

✔️ 2. System Diagram ⧄

✔️ 3. Communicative Examples 📸

✔️ 4. Keynote 👨🏻‍🏫

ProLUG Links ⛓️

ProLUG Admin Course Unit 4 🐧

Sat, 05 Oct 2024 00:00:00 +0000

Operating Running Systems ♺

Introduction

In this week’s Brown Bag session, we discussed the operation of running systems. This refers to when systems are live and possibly being accessed by users. During this time, we inspect settings, configurations, logs, and monitor running processes.

Purpose

Identify anomalies 🔍
Monitor users for incorrect usage 👥
Detect nefarious behavior ⚠️
Ensure processes have adequate resources ⚙️
Check logs for faults or failures 📝

Class Notes

A project wherein a file name was not being recognized, but the uuid was. Once the uuid was placed in fstab, everything worked fine.

Grey Beard Wisdom 🧙‍♂️

Make sure ports are open & server is on
Rebuild Backups in a Test Environment to ensure integrity
Write Stuff that can be supported by your team
Whats the AI policy for the company?

Useful Tools/Resources/Commands:

Tripwire - Sourceforge Grubby - https://software.opensuse.org/package/grubby SANS.org srq trigger

Lab Notes 🧪

Server Checks

Warm Up

cd ~ # Change Directory to Home
ls # list
mkdir unit4 # Create a Directory named unit4
mkdir unit4/test/round6 # This fails 🙅, because the -p option is not invoked
mkdir -p unit4/test/round6 # This works because the Parent option -p is enables allowing for sub directories ✅
cd unit4 # Change to unit4 directory
man ps # View the manual for the ps command 👀
ps -ef # -e Display information about other users processes, including those without controlling terminals. -f Display the uid, pid, parent pid, recent CPU
ps -ef | grep -i root #PID 0, the process ID is zero
ps -ef | grep -i root | wc -L #this command could be helpful because within the root processes it finds the process with the longest line. This could help with finding resource heavy processes or identifying complex commands
top

Pre-Lab 🥪

rpm -qa | grep -i iostat #should find nothing 👍
dnf iostat #This automatically invoke an install response 🪄
Install package 'sysstat' to provide command 'iostat'? [N/y] y

Confirming that Sysstat is installed 👍

rpm –qa | grep –i sysstat # sysstat-12.5.4-8.el9_4.x86_64
dnf install sysstat # Unnessecary 👎
rpm –qa | grep –I sysstat # Didn't work 😐
rpm –qi sysstat<version> # Didn't work 'unexpected token newline'

Confirming that Vim is installed 👍

rpm -qa | grep -i vim # vim-minimal-8.2.2637-20.el9_l.x86_64 ✅

Lab 🥼🧪

Gathering System release and kernel info 🌽

cat/etc/*release

uname #Linux
uname -a #Show all
uname -r #release
rpm -qa | grep -i kernel

Check the number of Disks 💾

fdisk -l #list partition tables
ls /dev/sd* #lists disks, with a wildcard at the end
pvs #physical volumes, Volume group and format
vgs #Volumes Groups
lvs #Logical Volumes
pvdisplay # More Comprehensive listing of Physical Volumes
vgdisplay # More Comprehensive listing of Volume Groups
lvdisplay # More Comprehensive listing of Logical Volumes

Check Disk Statistics 💾

iostat -d # Displays only device statistics
iostat -d 2 # Displays only device statistics in 2 second increments
iostat -d 2 5 # Displays only device statistics in 2 second increments for a total of 5 seconds

Check the amount of RAM 🍪

cat /proc/meminfo # a very comprehensive listing of memory info that spans multiple pages
free # Displays **Memory** and **Swap** Space usage
free -m # Diplays **Memory** usage in Mebibytes 1024 bytes MiB
🍼 I was today years old when I learned what a Mebibyte is 😄

Checking # of processors and processor info

cat /proc/cpuinfo
cat /proc/cpuinfo | grep proc | wc –l
iostat -c
iostat -c 2 # Runs CPU options in 2 second intervals.
iostat -c 2 5 # I had run this command before, in two second intervals for a total of 5 seconds

Check system uptime

uptime # 16:40 up 2 days, 17:38, 2 users, load averages: 2.81 2.46 2.30
man uptime # show how long system has been running for 1 min 2 min 15 mins

Check Recent Logins

last # vertical list of users ttys
last | more # more allows me to scroll through
w # Display detailed info about currently logged in users
who # Prints basic info about users currently logged in
whoami # Just lists your user name

Difference between w / who 🧐

Command	Information Provided	Key Focus	Use Case
`who`	Basic info about logged-in users (username, terminal, login time, remote host)	Simple listing of users currently logged in	Quick check on who is logged in
`w`	Detailed info about logged-in users, system load, idle time, and active processes	Who is logged in and what they are doing, with system load info	System monitoring and activity tracking

Check running processes and services

ps -aux | more # sending all listed processes to more
ps -ef | more # list every process
ps -ef | wc -l # counts the number of lines from a full format listing

Looking at Historical System Usage

sar - Collect, report, or save system activity information.

sar | more # check processing for the last day
sar -r | more # check all processes

Sar was not working, so I had to figure it out

sudo dnf install sysstat 👍
sudo systemctl enable --now sysstat ✅
sudo systemctl start sysstat ✅
sudo systemctl status sysstat ✅
sudo vim /etc/sysconfig/sysstat ✅
ENABLED="true" ✅
sudo systemctl restart sysstat ♺
sar # 👍 Works Now!

Ok back on course with sar

sar 2 # runs sar every 2 seconds
sar 2 5 # runs sar every 2 seconds for a total duration of 5 seconds

Check sar logs for previous daily usage

cd var/log/sa/
# ls

Interesting, sar logs are store in var/log/ who’d a thunk it

sar -f sa03 | head

Exploring Cron

Calm down snoop, this is job related 💩

There is a daemon running my cron, thank god I can check in on it 😈

ps -ef | grep -i cron
systemctl status crond
/var/spool/cron
ls -ld /etc/cron*

An Operations Bridge

A centralized platform that provides real-time visibility and control over an organization’s IT infrastructure and services.
— CIO WIKI¹

Essentially 🧪

A unified view of operations, consolidating and correlating data from various sources.

General Purpose

Streamline IT operations ⚡
Improve service availability 📈
Enhance incident response and resolution 🔍

General Features

Event Management 🔔

Collects and consolidates events and alerts from network devices, servers, applications, and security systems. Applies correlation and filtering to identify meaningful incidents and prioritize them based on service impact.

Performance Monitoring 🏎️

Monitors key performance indicators (KPIs) for infrastructure, applications, and services. Identifies performance bottlenecks, trends, and anomalies, enabling proactive optimization.

Incident Management 🚨

Captures, tracks, and escalates incidents. Facilitates collaboration and prioritization, ensuring timely resolution with historical insights and knowledge base integration for faster fixes.

Root Cause Analysis 🦷

Analyzes events, performance data, and logs to uncover the root causes of incidents. Uses data analytics to correlate related events and identify underlying issues affecting service availability.

Dashboards and Reporting 📊

Provides customizable dashboards and reports to visualize the health, performance, and availability of IT systems. Allows stakeholders to monitor key metrics, track service-level agreements (SLAs), and gain insights into overall system performance.

Automation and Orchestration ⚙️

Includes automation capabilities for routine IT tasks like system restarts, service provisioning, and configuration changes. Reduces manual effort, increases efficiency, and minimizes human errors.

Potential Challenges

Initial Setup and Configuration 🛠️

Can be complex and may require specialized expertise and significant manpower.

Integration with Multiple Systems 🧩

As the system grows in complexity, there’s an increased chance of failure and maintenance challenges.

Training and Skill Development 📚

Users may need training to effectively manage and operate the platform. Integrating disparate systems requires time and expertise.

Ongoing Maintenance and Updates 🔄

Frequent updates to data sources can disrupt connections, making regular maintenance essential. It’s not a turnkey solution.

Personal Take

The Term

The term Operations Bridge is relatively new to me. I would’ve called this a Unified Dashboard. The idea of bridging operations by consolidating and unifying data makes sense, but it’s a bit more complex than just a simple dashboard.

Outstanding Feature

The dashboard and reporting capabilities are the most critical. Seeing everything at a glance is invaluable. Other features can often be achieved using smaller tools like Cron jobs, Ansible, Bash, or Prometheus.

Complexity as a Weakness

The complexity is a major drawback. Constant tweaking seems inevitable in the ever-changing IT landscape. Large organizations with stable, long-term systems may benefit most, but the potential for cascading errors, like in Chernobyl, comes to mind when a single feedback loop causes widespread failures.

🚨 Incident Response Scenario 🚨

— Incident Response Cheatsheet²

Scenario Outline

Your team lacks documentation on how to check out a server during an incident.
Develop a procedure detailing what an operations person should do when a system is suspected of malfunctioning.

Key Points

Undocumented server 🛠️
Server malfunctioning 🚫
Need for procedural diagnosis 🧑‍💻

Incident Response Cycle

Detect
Respond
Mitigate
Report
Recover
Remediate
Lessons

Custom Incident Procedure

1. Detect / Observe

Check process logs
Check security logs
Check application logs

2. Respond

Snapshot 📸

If possible, take a snapshot of the current system. Treat it like a crime scene—nothing should be disturbed or altered. This snapshot is crucial for forensic analysis of the system’s state.

Triage 🩺

Scan the system for symptoms. Determine if it’s a security incident or a bug.

Examine 🩻

Network configuration
DNS settings
Hosts file
Autostart processes
Scheduled jobs

List 📝

Active network connections
System users
Running processes
Recently modified files

Verify ✔

Integrity of installed packages

3. Mitigate

After triaging, identify the nature of the issue. If it’s a:

Security Incident 🚨 — Follow the [Crisis Management Procedure]³
Bug 🪲 — Proceed with the steps below

4. Report / Declare 📣

Inform the team and supervisors that the incident has been triaged, diagnosed, documented, and mitigated.

5. Recover 🛠️

Develop or follow a recovery plan to restore the system to normal operation. Document the recovery steps for future reference.

6. Remediate / Repair 🛠️

Fix the issue based on its nature. This could involve:

Reconfiguration
Patching
Blocking/Banning
Restoration
Allocating new resources
Turning it off and on 😁

7. Document / Lessons 📝

Write a report detailing the problem, its cause, and the steps taken to resolve it. Include a lessons learned section to improve future responses.

8. Bask ☕️

Once resolved, lean back, nod in satisfaction, and enjoy your coffee. Every resolution deserves a small moment of celebration.

The Battle Drill ⚔️ 4

A Battle Drill is a standard operating procedure used in infantry training. It’s based on common scenarios that require rapid, collective action. By practicing battle drills, a team can react quickly and efficiently to dangerous situations without the need for complex decision-making.

Battle drills prepare infantry for swift, organized, and aggressive responses to high-stress situations such as explosions, direct fire, flashing lights, chemical exposure, or sudden assaults.

Battle Drills are for those moments when everything is going wrong and you need a reactionary response. — Scott Champine

⏱️ Reaction time and mental readiness are key to success.

Application to Operational Incidents 🛠️

The principles behind battle drills can easily be applied to system operations, where rapid group responses to incidents are critical.

Documentation 📝

Procedures must be well-documented and easy to understand for all possible scenarios.

Clear Objectives 🎯

Leadership communicates simple and clear objectives to the team for each scenario.

Drilling / Testing 🧑‍💻

Teams practice scenarios together, following the documented procedure and maintaining effective communication.

Known Procedure 👥

Through review and practice, everyone knows their role and objectives in the group, ensuring a unified response in any scenario.

The Importance of Drilling 🔄

In real incidents, there’s often no time to plan a response. For regular operations, we have the luxury of preparation and research, but in high-stress, mission-critical situations, the pressure can become overwhelming, potentially leading to paralysis. Preparedness through practice is essential.

Knowing Your Tools 🧰

In the infantry, this means knowing how to load, reload, field-strip, and fire a weapon, or apply first aid using a trauma kit to save a fellow soldier.

These concepts are equally valuable in system operations. Knowing how to run basic commands, troubleshoot issues, break down problems, and write scripts all fall under knowing your tools. Just as a soldier must be able to handle their equipment with their eyes closed, an operator must be fluent in the commands, tools, and procedures they rely on daily. This level of expertise comes from memorizing commands, knowing paths, and practicing scripting.

Detection

Response

Mitigation

Reporting

Recovery

Remediation

Lessons Learned

After action review

Operations Bridge

Reflecting upon Unit #4

What questions do you still have about this week?

Note-Taking Improvements

A few months before this course, I realized that to deepen my understanding of systems, I needed to take extensive notes I could easily refer back to. After doing thorough research on different note-taking systems, I concluded that LogSeq⁵ was the best fit for my needs.

I chose LogSeq⁵ because:

It’s open source
Local-first
Supports tags
Uses Markdown
Works across all platforms

Since adopting LogSeq, I’ve taken an immense amount of notes. I’ve also created a lot of cheat sheets and references, thanks to LLMs and various GitHub repositories. Let’s just say my note-taking game has been strong.

When this course started, I knew that note-taking would be a big component. Most of my notes are private since I prefer to write candidly, sometimes using strong language. To share more polished, public-friendly notes, I created this HUGO blog. I’m a stickler for clean layouts, typesetting, and readability, so filling out pre-formatted documents isn’t my style.

ProLUG Links ⛓️

Operations Bridge Wiki CIO-WIKI, 2024. ↩︎
Incident Response Cheatsheet Version 1.8 PDF Reference Lenny Zeltser, 2024. ↩︎
Detailed Crisis Management Procedure Blog Trevor Smale, 2024. ↩︎
Battle Drill Wiki Wikipedia.org, 2024. ↩︎
Note Taking Software ↩︎ ↩︎

Why Btrfs 🤔

Thu, 03 Oct 2024 00:00:00 +0000

Btrfs 💾

Btrfs stands for B-tree File System, not “Better File System,” though it’s easy to see why people might think that. I believe Btrfs is the future for both small and large-scale projects, as it reduces the need for manual and automated maintenance, while simplifying backup and restoration processes.

This article aims to shed light on the history and motivation behind creating Btrfs, its core functionality, and the standout features that set it apart from the competition.

History

Motivation

The idea for Btrfs was first proposed by IBM researcher and bioinformatician Ohad Rodeh at a USENIX conference in 2007. The goal was to develop a copy-on-write (COW)-friendly B-tree algorithm, which would allow for efficient data storage and retrieval without the overhead typical of other file systems.

As Valerie Aurora explained:

“To start with, B-trees in their native form are wildly incompatible with COW. The leaves of the tree are linked together, so when the location of one leaf changes (via a write—which implies a copy to a new block), the link in the adjacent leaf changes, which triggers another copy-on-write and location change, which changes the link in the next leaf… The result is that the entire B-tree, from top to bottom, has to be rewritten every time one leaf is changed.”
– Valerie Aurora¹

Chris Mason, a core developer of the Reiser Filesystem, liked the idea and saw an opportunity to move beyond Reiser, which used B-trees but wasn’t optimized for COW. Mason brought the idea to his new job at Oracle, where development of Btrfs began in earnest.

“I started Btrfs soon after joining Oracle. I had a unique opportunity to take a detailed look at the features missing from Linux, and felt that Btrfs was the best way to solve them.”
– Chris Mason²

In collaboration with Oracle colleague Zach Brown, they drafted the initial version of Btrfs.

Rapid Development ⏱️

Thanks to corporate backing and a team of experienced developers, Btrfs moved through an aggressive development cycle. Within two years of the technical proposal, a working 1.0 version of Btrfs was released in late 2008.

Shortly after its introduction, Btrfs was merged into the mainline Linux kernel. Despite the conservative nature of file system adoption—where admins, systems engineers, and software engineers prefer proven, stable systems—Btrfs quickly gained traction.

In 2015, SUSE Linux Enterprise Server (SLES) became the first major Linux distribution to adopt Btrfs as its default file system, citing it as the future of Linux storage solutions.

Enterprise Adoption 📊

Today, Btrfs is the default file system for several major enterprise Linux distributions, including SUSE, Fujitsu Linux, Ubuntu, Oracle Linux, and popular user distributions like Fedora, Arch, and Gentoo.

In fact, Meta (formerly Facebook) uses Btrfs to manage their large, dynamic data sets. According to core developer Josef Bacik, using Btrfs at Meta has significantly reduced access times and contributed to cost reductions in production environments.³

How Btrfs Works ⚙️

The B-Tree+ Algorithm 🌳

At its core, Btrfs relies on a B-tree, a type of data structure designed to organize and store data efficiently.

Here is a basic diagram of a B-tree, though in file systems, it gets more complex:

A B-tree consists of nodes and links (sometimes referred to as keys and pointers or leaves and branches), which drastically reduce seek time. This ensures that, no matter how much data you store, finding the right file is quick and doesn’t require searching through everything.

The root node is a type of index stored in a fixed location on the disk. It serves as the starting point for a rapid search called fanning out.

This structure reduces disk access time and, in turn, improves overall system efficiency. The relationship between the depth of nodes and the breadth of data is known as the fan-out factor. Tuning this ratio can either speed up searches (with a wider spread) or reduce write size for smaller segments.

Copy-on-Write (COW) 🐄

Copy-on-Write (COW) is a method where data and metadata are not overwritten in place. Instead, they are copied to a new location before the update is finalized. Btrfs employs COW in conjunction with its B-tree algorithm to maintain data integrity.

Btrfs also uses delayed allocation, where metadata is updated first, linking it to new data before the actual data is copied Persistant Pre Allocation to a new location. This delay allows the file system to organize sector placement called Extent Base Allocation and optimize metadata before the actual write Multi-Block Allocation, reducing unnecessary reads and writes.

This delayed allocation process supports wear leveling (also known as TRIM) on SSDs, where data is written to new sectors to avoid repeatedly writing to the same location, thus extending the lifespan of the drive.

Snapshots 📸

Snapshots are one of Btrfs’s standout features. They use COW to create lightweight, efficient snapshots of data at any given time.

Unlike traditional Linux filesystems, which only allow snapshots of logical volumes, Btrfs can snapshot both volumes and subvolumes. This means that the entire data set—down to the subvolume level—can be efficiently snapshotted. Snapshots in Btrfs do not require duplication of data, only tracking changes made after the snapshot was taken.

Cloning 🐑🐑

Cloning in Btrfs allows you to create writeable copies of subvolumes or snapshots, which share the same data blocks until changes are made. Unlike traditional file copying, cloning doesn’t duplicate the original data, making it fast and storage-efficient. When modifications are made, only the changed blocks are copied, leaving the rest shared between the clone and the original.

This makes cloning in Btrfs ideal for use cases where multiple environments or datasets need to be derived from the same base system, without significant storage overhead.

Dynamic Online Storage 📈📉

Btrfs allows you to manage storage pools dynamically with online resizing and device management, meaning that you can expand and shrink file systems while they are mounted and in use. This flexibility helps reduce downtime and allows systems to scale easily with growing storage needs.

Expanding Disk Pools 🏊‍♂️🏊‍♂️🏊‍♂️

You can add new disks to a Btrfs file system without stopping the system. With a simple command, Btrfs integrates the new storage into the existing pool, redistributing data across devices if necessary. This feature makes Btrfs highly scalable, especially in environments where storage demands can grow unpredictably.

Resizeable 📏

Btrfs volumes can be expanded or shrunk online, without unmounting the file system. Expanding is as simple as adding more storage, while shrinking requires no special process apart from the resizing command. This dynamic resizing means you can adjust the size of your file system to suit your current storage needs.

Self-Healing ❤️‍🩹

Btrfs provides self-healing capabilities when used in a redundant RAID setup. Through data and metadata checksumming, Btrfs can detect corrupted blocks and automatically fix them using redundant copies of the data. This ensures data integrity without user intervention, particularly useful for critical data environments.

Compression 🪗

Btrfs supports transparent compression, which means files can be compressed as they are written to disk, saving space without requiring manual compression. It supports multiple algorithms like LZO, Zlib, and Zstandard (ZSTD), each with different balances of speed and compression ratio. This feature helps save storage space and can improve performance, especially when working with large datasets.

Technical Feature List: 🔍

Journaling: Keeps track of changes before they are committed, helping with data recovery in case of a crash.
Extent Base Allocation: Allocates large, contiguous blocks of storage to reduce fragmentation.
Persistent Pre-allocation: Reserves space for a file at creation to prevent fragmentation and improve performance.
Delayed Allocation: Delays the allocation of disk space until data is written, optimizing space management.
Multi-block Allocation: Allocates multiple blocks at once to increase efficiency, especially for large files.
Stripe-aware Allocation: Optimizes block allocation for RAID systems by aligning data with RAID stripes.
Resizeable with resize2fs: Can be resized (grown or shrunk) using the resize2fs tool.
B-tree Balancing Algorithm - Different from XFS (COW B Tree): Uses a specific B-tree balancing algorithm for efficient file system organization and copy-on-write operations.
Copy-on-Write (COW): Writes modified data to new locations rather than overwriting existing data, preventing data corruption.
Snapshots and Clones: Creates point-in-time copies of data (snapshots) and allows for duplication (clones) without full data replication.
Built-in RAID Support: Provides native support for RAID configurations, improving data redundancy and performance.
Data and Metadata Checksumming: Ensures data integrity by verifying both data and metadata through checksums.
Self-Healing: Automatically repairs corrupted data using mirrored or parity blocks in RAID configurations.
Dynamic Subvolumes: Supports the creation of isolated subvolumes within the same file system for better data management.
Online Resizing: Allows the file system to be resized while it’s still mounted and in use.
Compression (LZO, ZLIB, ZSTD): Offers various compression algorithms to reduce storage space usage.
Deduplication: Eliminates duplicate copies of repeating data to save space.

Core Commands 🪨

mkfs.btrfs

Creates a new Btrfs file system on a specified device or partition.

btrfs subvolume create

Creates a new subvolume within an existing Btrfs file system.

btrfs subvolume delete

Deletes a specified subvolume from the Btrfs file system.

btrfs subvolume list

Lists all subvolumes within a Btrfs file system.

btrfs filesystem show

Displays detailed information about the Btrfs file systems on all mounted devices.

btrfs filesystem df

Shows disk space usage, including metadata and data, for a Btrfs file system.

btrfs balance start

Begins a balancing operation to redistribute data and metadata across the Btrfs file system.

btrfs balance cancel

Cancels an ongoing balancing operation.

btrfs scrub start

Initiates a scrub operation to verify data integrity on a Btrfs file system.

btrfs scrub status

Shows the status of an ongoing scrub operation.

btrfs device add

Adds a new device to an existing Btrfs file system, allowing for dynamic storage expansion.

btrfs device delete

Removes a specified device from a Btrfs file system.

btrfs device stats

Displays statistics and error information about devices in a Btrfs file system.

btrfs check

Checks the integrity of a Btrfs file system, often used for troubleshooting.

btrfs rescue chunk-recover

Attempts to recover data chunks from a damaged Btrfs file system.

btrfs filesystem resize

Resizes a Btrfs file system, either expanding or shrinking it dynamically.

btrfs snapshot

Creates a read-only or read-write snapshot of a subvolume or file system.

btrfs property set

Modifies properties of a Btrfs file system or its subvolumes (e.g., enabling compression).

btrfs quota enable

Enables quota enforcement on a Btrfs file system to limit space usage.

btrfs quota rescan

Rescans the file system to enforce or update quota limits.

Conclusion

Well I hope that has opened your eyes to the benefits of Btrfs. Researching for this article has helped me to learn some more about the subject. If will definitely add to the article as I learn more about the subject, so consider this evergreen information 🌲

Helpful Links ⛓️

https://en.wikipedia.org/wiki/Btrfs https://docs.kernel.org/filesystems/btrfs.html Footnotes

The above quote is excerpted from Valerie Aurora Article lwn.net, 2009. ↩︎
Chris Mason Interview during LF Collaboration Summit, 2009. ↩︎
The above reference is to the first proposal for the Btrfs file system Technical Proposal during Gopherfest, Tues, June 12th, 2017. ↩︎

ProLUG Admin Course Capstone Project Stage 3 🐧

Mon, 30 Sep 2024 00:00:00 +0000

Intro 👋

A progress update regarding the ProLUG website that improve useability

Info Pop-up

I wanted to add additional information to the site while maintaining a clean layout, predictable and comprehensible design. This is not only my personal aesthetic taste, this is a feature of accessibility. Websites with tons of information, many levels of contrast and or layouts are not only difficult for the visually impaired, but confusing for people suffering from cognitive impairment. I try to use inclusive design in accordance with WCAG guidelines on any project I am working on. So this leads me to the objective, to create a pop up design with additional information. I started with one of the more confusing features of the site, the verification input. When getting started, I wasn’t sure if it would be possible to accomplish this with TailwindCSS, so it took a bit of reading to confirm. It turns out this is completely possible using tailwind as evidence by the code snippet.

Design Details

I would like to convey about the design choices as

Simple prominent button with high contrast.
Semi transparent background to denote this is a temporary overlay
Large text
Short, straightforward descriptions
Differing colors for open and close to further indicate the state.

Verification entry box with question mark button

Associated pop-up

<details class="open">
 <!-- Button to be visible -->
 <summary class="bg-blue-300 rounded-xl inline-block px-4 py-2 text-xl hover:bg-blue-500 hover:duration-300 hover:text-white cursor-pointer">
 ?
 </summary>

 <!-- Content will be hidden until the user clicks on the summary button. -->
 <div class="bg-white bg-opacity-50 backdrop-blur-md rounded-sm w-[100%] h-[100%] max-w-[600px] max-h-[700px] my-auto mx-auto absolute inset-0 text-gray-600 p-4">
 <h1 class="text-2xl font-bold">Information</h1>
 <div class="m-8"><h2 class="text-xl font-bold">How this works</h2><p>Each ProLUG certificate has a unique ID link. This ID belongs only to the person who earned the certificate. When the ID is checked, the certificate information should match the person’s details. </p>
 <h2 class="text-xl font-bold">Does not Match</h2><p>If the certificate information does not match what the verifier provided, the certificate is likely to be fake.</p>
 <h2 class="text-xl font-bold">Why Verify?</h2><p>ProLUG values the hard work graduates put into earning their certification. We work to stop shortcuts and forgery to protect that effort.</p></div>
 <!-- Close button positioned at the bottom and centered -->
 <div class="absolute bottom-4 left-1/2 transform -translate-x-1/2">
 <button class="bg-red-500 text-white px-4 py-2 rounded-lg hover:bg-red-700"
 onclick="this.closest('details').removeAttribute('open');"> Close </button>
 </div>
 </div>
</details>

Todo

ProLUG Links ⛓️

ProLUG Admin Course Unit 3 🐧

Mon, 30 Sep 2024 00:00:00 +0000

Storage & Logical Volume Management 🧠💾

Intro

Ok, Week 3 / Unit 3. This week we are working on Logical Volume Management (LVM) and RAID. Fortunately I was able to listen to the entire lecture and read most of the chats which is always helpful.

Lab Notes 🧪

Warmup

Redirects ✉️

cd ~ # Change Directory to Home
mkdir lvm_lab # Create a Directory called lvm_lab
cd lvm_lab # Change location into the lvm_lab directory
touch somefile # create an empty file called somefile
echo "this is a string of text" > somefile # This sends the output of echo into the file
cat somefile # concatenates and displays what is in the file
echo "this is a sting of text" > somefile # Overwrites the line with the same text
echo "this is a sting of text" > somefile
echo "this is a sting of text" > somefile
cat somefile # We are left with one line of text after repeating this action because this
action overwrites.
echo "This is a string of text" >> somefile # The double arrow is redirect append
echo "This is a string of text" >> somefile # this adds a second line
echo "This is a string of text" >> somefile # this adds a third line
cat somefile # The concatenated output would be 3 line of "This is a string of text"
echo "this is our other test text" >> somefile
echo "this is our other test text" >> somefile
echo "this is our other test text" >> somefile
cat somefile | nl # This adds numbering to each line in the file
cat somefile | nl | grep test # This does nothing as there is no text within the file that
contains the string nothing
cat somefile | grep test | nl # Also nothing
cat somefile | nl | other # Gives us our last 3 lines and associated line number
always | nl | before your grep

Lab 🥼🧪

Disk Speed Tests ⏱️

This is on my own virtual machine, so figuring out the unique commands took a bit of extra work.

lsblk /dev/sda2
p #print to see partitions
d #delete partitions or information
w #Write out the changes to the disk.

Write tests 💾

saving off write data – rename /tmp/file each time

Checking /dev/sda2 for a filesystem

blkid /dev/sda2 👍

If no filesystem, make one

mkfs.ext4 /dev/sda2
mkdir space
mount /dev/xvda space ?

Write Speed Test 💾🏎️

for i in 'seq 1 10';do time dd if=/dev/zero of=/space/testfile$ bs=1024k count=1000 | tee -a /tmp/speedtest1basiclvm

Write Test Result

real 0m0.003s
user 0m0.000s
sys 0m0.002s

Read Test 👓💾

for i in 'seq 1 10';do time dd if=/space/testfile$i of=/dev/null;done

Read Test Result

real 0m0.001s
user 0m0.001s
sys 0m0.001s

Cleanup 🧹

for i in 'seq 1 10'; do rm -rf/space/testfile$i;done

LVM 🧠💾

start in root (#); cd /root

Check physical volumes on your server (my output may vary)

[root@rocky1 ~]#fdisk -l | grep -i sda

output:

Disk /dev/sda: 32 GiB, 34359738368 bytes, 67108864 sectors
/dev/sda1 2048 2099199 2097152 1G 83 Linux
/dev/sda2 2099200 67108863 65009664 31G 8e Linux LVM

Basic LVM listings

pvs

output:

PV VG Fmt Attr PSize PFree
/dev/sda2 rl_localhost-live lvm2 a-- <31.00g 0

vgs

output:

VG #PV #LV #SN Attr VSize VFree
rl_localhost-live 1 2 0 wz--n- <31.00g 0

output:

lvs

LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root rL_localhost-live -wi-ao---- 27.79g
swap rL_localhost-live -wi-ao---- 3.20g

pvdisplay (Needs SUDO or Root)

output:

PV Name /dev/sda2
VG Name rl_localhost-live
PV Size <31.00 GiB / not usable 3.00 MiB
Allocatable yes (but full)
PE 4.00 MiB
Total PE 7935
Free PE 0
Allocated PE 7935
PV UUID gMVNd5-peB1-uUX6-Rw28-4Ncb-mi1b-rDJR38

PV Name: /dev/sda2
- This is the physical volume (PV) name, which indicates the device or partition that has been initialized as a physical volume for use in LVM (Logical Volume Manager).
VG Name: rl_localhost-live
- This is the volume group (VG) name to which this physical volume belongs. A volume group is a collection of physical volumes that create a pool of storage space from which logical volumes (LVs) are allocated.
PV Size: <31.00 GiB / not usable 3.00 MiB
- The size of the physical volume is 31.00 GiB, but 3.00 MiB is not usable, likely due to overhead or alignment issues within the physical volume.
Allocatable: yes (but full)
- This indicates whether the physical volume is available for allocation into logical volumes. It is set to “yes,” but the “full” remark means that all available physical extents (PEs) are already allocated.
PE Size: 4.00 MiB
- This shows the size of a physical extent (PE), which is the smallest chunk of data that LVM manages. In this case, each PE is 4.00 MiB.
Total PE: 7935
- The total number of physical extents on this physical volume. This is calculated based on the size of the physical volume and the size of each PE.
Free PE: 0
- This shows the number of free physical extents on the physical volume. In this case, there are no free extents left, meaning all available space has been allocated.
Allocated PE: 7935
- The number of allocated physical extents, which are being used by logical volumes. Since all available extents are allocated, this number matches the total number of PEs.
PV UUID: gMVNd5-peB1-uUX6-Rw28-4Ncb-mi1b-rDJR38
- The unique identifier for this physical volume, which is used internally by LVM to track physical volumes across different systems or reboots.

vgdisplay: needs sudo

example output:

 --- Volume group ---
VG Name vg_data
System ID
Format lvm2
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 2
Max PV 0
Cur PV 2
Act PV 2
VG Size 99.99 GiB
PE Size 4.00 MiB
Total PE 25599
Alloc PE / Size 12800 / 50.00 GiB
Free PE / Size 12799 / 49.99 GiB
VG UUID hFwi0D-GTlv-NFjp-O2he-x8Yw-kfIa-c3QqX6

lvdisplay: needs sudo

example output:

 --- Logical volume ---
LV Path /dev/vg_data/lv_backup
LV Name lv_backup
VG Name vg_data
LV UUID B1q8Iq-0tWz-Fk0P-xwQ7-0T5T-QC3c-XcK5T1
LV Write Access read/write
LV Creation host, time hostname, 2024-10-08 12:00:00 +0000
LV Status available
# open 1
LV Size 50.00 GiB
Current LE 12800
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:2

Creating & Carving LVM

A little struggle 😣

So I ran into several hurdles when putting this into practice with Proxmox.

📍 Since I was using Proxmox the storage was already an LVM on account of it being a virtualized disk Solution: I created a second storage volume in addition to sda, sdb was emulated as an SSD and RAW storage. Once this was set up I was able to initialize the process.

📍 The demonstration lab suggested using /dev/xvd$disk the ‘x’ relates to xcp-ng, a virtualization technology that differs from mine. I am using KVM and VirtIO on Proxmox so the naming convention would differ. so vdb would be a naming convention akin to this system.

Ultimately, once I created this RAW SSD emulated storage I was able to move on.

The steps

1. Provisioning the Physical Volume ✅

sudo pvcreate /dev/sdb

Output: Physical volume “/dev/sdb” successfully created.

Confirmed with: lsblk

2. Provisioning the Volume Group ✅

sudo vgcreate examplegroup /dev/sdb

Output: Volume group “examplegroup” successfully created

Confirmed with: vgdisplay

3. Provisioning the actual Logical Volume ✅

sudo lvcreate -L 1G -n lv1 examplegroup

 Logical volume "lv1" created.

Confirmed with: lvdisplay

4. Formatting a Logical Volume with a FS 💾 ✅

mkfs.ext4 /dev/mapper/lv1

 mke2fs 1.45.6 (20-Mar-2020)
Creating filesystem with 262144 4k blocks and 65536 inodes
filesystem UUID: a1b2c3d4-e5f6-789a-bcde-123456789abc
Superblock backups stored on blocks:
32768, 98304
Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

Checking remaining VG Space

Broadly:

the VGS command provides a summary of all Volume Groups and includes information about the free space.

Looks like this:

 VG #PV #LV #SN Attr VSize VFree
vg_data 2 3 0 wz--n- 199.99g 49.99g

More Specifically:

The vgdisplay command will show this info as it pertains to a particular pv 👍

Ultimately

Thanks to putting our heads together in a studygroup, I was able to get through the completed process several times. I ended carving of a Volume Group into Logical Volumes. Some of which were formatted in Ext4 and some were formatted to XFS using MKFS. 😁

The process was a little bit long with many tangential learning, ultimately I think this experience has deeply ingrained the process in my psyche, pushing out childhood memories of finger painting.

Fstab

I would like to return to Fstab and learn more about it. I ran into some issues mounting LVM’s with Fstab

LVS

Logical Volume Summary provides a summary as the name implies of all logical volumes present in the system. I wish I had known this command earlier as I was using the other commands listed above 😄

Example output:

 LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
lv_backup vg_data -wi-a----- 50.00g
lv_home vg_data -wi-ao---- 20.00g
lv_root vg_system -wi-ao---- 30.00g

RAID 💾💾💾

RAID stands for Redundant Array of Independent Disks, more on that below.

Raw notes:

Created a proxmox vm with rocky 9.4 minimal with 3 x raw ssd emulated disks

MDADM

Lecture thoughts 🪩

Everything ‘is’ a file after all

this past Saturday. During that lecture I had a major awakening in regards to how unix works. I have heard the ’everything is a file’ mantra on several occasions. However, sometimes hearing is not fully understanding. My moment of understanding came when we checked running processes and opened one as a file. I knew that everything under the hood was a file, but ephemeral things like processes were not part of the picture. That truly blew my mind 🤯

UUID’s go beyond a block storage device

It turns out that logical volumes are assigned UUIDs. UUIDs are unique strings that can be used to reference or link reliably to storage device. So logical volumes attain the same respect as physical volume.

if I were to run

bash blkid

bash lvddisplay

one would see a string like this:

f4d12857-93c3-4d6f-91e5-bb379f02e1d1

More on this later, I went down a rabbit hole. 🐇

Logical Volumes 🧠💼

Logical Volumes (LVs) offer a flexible way to manage disk storage in Linux. With LVs, users can create, resize, and move storage volumes without being limited by the physical layout of the disks. The Logical Volume Manager (LVM) abstracts the underlying physical storage, making it easier to manage disk space, support snapshots, and resize volumes as needed.

The Layered Structure 🍰

The structure of LVM involves multiple layers:

Physical Volumes (PV): These are the actual physical storage devices, like SSDs or HDDs. A Physical Volume can either be part of a Volume Group or encompass an entire Volume Group.
Volume Groups (VG): VGs are containers that hold one or more Logical Volumes. A Volume Group can span across multiple Physical Volumes, providing a flexible pool of storage.
Logical Volumes (LV): LVs are sections of a Volume Group that serve as the actual storage units. A Volume Group can contain many Logical Volumes of different sizes.
File Systems: Finally, file systems are placed on the Logical Volumes to store data.

The table below illustrates how a Volume Group (VG) can host several Logical Volumes (LVs) of varying sizes. Each Logical Volume is assigned a unique identifier (UUID), and snapshots are also given their own UUIDs.

VG	LV	FS	Size	UUID Short	Snapshot	Snapshot UUID Short
vg_data	lv_root	XFS	50 GB	f4d1-91e5-bb37	None	N/A
vg_data	lv_home	XFS	100 GB	a123-c567-8901	snap_home	abcd-ef56-7890
vg_data	lv_var	ext4	20 GB	c9d8-89e0-f9a1	None	N/A
vg_data	lv_backup	ext4	150 GB	f129-bc97-f134	snap_backup	f7d8-a67e-f765
vg_data	lv_logs	XFS	10 GB	e123-9abc-e1d4	None	N/A
vg_storage	lv_media	ext4	200 GB	bc97-bc9e-5612	snap_media	d98f-9e67-1cd2

I understand that this course focuses on the two most popular and reliable file systems, as these are the ones most commonly encountered in enterprise environments. However, I’ve noticed that BTRFs is starting to gain traction. I’ve listened to several talks and read extensively about its features and potential.

Some limitations apply to the status quo

Both ext4 and XFS subvolumes can handle a lot of scenarios. This includes online resizing. This is when the size of an LVM must be increased while actively in use (Online). However, for both of these filesystems, online resizing can only grow and LVM and not shrink. Additionally snapshots must be managed on a per LV basis, not a huge issue with the advent of automation tools.

RAID

Intro

RAID stands for Redundant Array of Independent Disks (originally Redundant Array of Inexpensive Disks). It is a data storage technology that combines multiple physical disk drives into a single logical unit to improve performance, increase capacity, or provide redundancy to protect data against drive failures.

How it differs from LVM

RAID is primarily for redundancy and performance.
RAID works at a block level across discs.
So it is sometimes used to create a high capacity pool.

Multiple configurations

There are configurations for differing degrees of performance
There are configurations for different levels of redundancy

Comparison Table

RAID Level	Description	Performance	Redundancy
RAID 0	Data striping without parity or mirroring.	High (fast read/write speeds)	None (no redundancy)
RAID 1	Mirroring of data across two or more disks.	Moderate (read speed improves, write speed similar to single disk)	High (data is mirrored)
RAID 5	Block-level striping with distributed parity.	Moderate (improved read speeds, slower writes due to parity calculation)	Moderate (can tolerate 1 disk failure)
RAID 6	Block-level striping with dual parity.	Moderate (similar to RAID 5, but slower writes)	High (can tolerate 2 disk failures)
RAID 10	Striping across mirrored sets (combines RAID 1 and RAID 0).	High (fast read/write due to striping and redundancy)	High (can tolerate multiple disk failures depending on which disks fail)
RAID 50	RAID 5 arrays striped (combines RAID 5 and RAID 0).	High (fast read speeds, but slower writes)	Moderate (can tolerate up to 1 disk failure in each RAID 5 array)
RAID 60	RAID 6 arrays striped (combines RAID 6 and RAID 0).	High (fast reads, slower writes due to dual parity)	Very High (can tolerate up to 2 disk failures in each RAID 6 array)
RAID 1+0	Nested RAID 1 over RAID 0 (mirroring across striped sets).	High (similar to RAID 10)	High (similar to RAID 10)

Software & Hardware Raid

Software RAID is when the host Operating System maintains the linking and syncing.
Hardware RAID is when a dedicated device does this. For example a PCI raid controller.

A software RAID is:

is running as a general task through the CPU. With multi-core/thread processors and more robust process handling, software RAID is more reliable than in the past. Advantages:

Portable 🦶
Flexible 🧘
Cost Effective 💰

Hardware RAID is:

Application specific hardware like a RAID Controller always has the potential to be more performant and reliable. But this is mfg. dependent, the good ones tend to be expensive (such as life) 💸

Performant* 🏎️
Reliable* ⏳

Other Filesystem research/writing

Btrfs

I wrote an entire separate article about B-Tree FileSystem as I find it interesting Link to Btrfs article

Linux Filesystem Comparison

I find file-systems really interesting. Recently I made a huge note comparing the major FileSystems. Link to FileSystems note/article

Linux Filesystem Timeline

This timeline may help illuminate how filesystems incrementally improve over time.

Year	Milestone
1993	ext2 filesystem development begins.
1994	ext2 released with Linux Kernel 1.0.
1994	XFS first developed by SGI for IRIX.
2001	ext3 released, introducing journaling.
2001	XFS ported to Linux.
2006	Development of ext4 begins to extend the capabilities of ext3.
2007	BTRFs development announced by Oracle.
2008	ext4 marked as stable in Linux Kernel 2.6.28.
2009	BTRFs included in Linux Kernel 2.6.29 as an experimental filesystem.
2009	XFS officially included in Linux Kernel 2.6.36.
2010	ext4 becomes the default filesystem for many Linux distributions, including Ubuntu and Fedora.
2012	BTRFs adopted by SUSE Linux Enterprise Server.
2014	XFS becomes the default filesystem in RHEL 7.
2014	Fedora starts offering BTRFs as an option.
2020	Fedora 33 makes BTRFs the default filesystem.

Crisis Management 🔥🧯

Intro

I read a chapter from googles security handbook focused on responding to a security incident. This chapter covers a wide range of information including common mistakes, templates etcc… Scott has asked us to pull keywords that may help us better triage an incident. I decided to extend this to creating an incident response checklist for future reference. This order of the list slightly differs from that of the article, yet I think it summarizes the chapter well.

Overview 🔍

Effective crisis management requires taking command and maintaining control of an incident. The outcome of a security incident largely depends on how well your organization prepares and responds, a process referred to as incident response (IR) capability.

Transparency 🔮

Transparency is key in managing incidents, particularly in light of regulations such as GDPR and service contracts. Customers are continually pushing the boundaries for how quickly investigations must begin, progress, and be resolved. Organizations are often expected to respond to potential security problems within 24 hours or less.

As the saying goes: “There are only two types of companies: those that know they’ve been compromised, and those that don’t know.”

Steps of Crisis Management 🪜

1. Triage 🥼

The First Step: Don’t Panic! Not every incident is a crisis.
Differentiate between compromises and bugs.
Make educated and informed assumptions about the severity and potential consequences of the incident.
- What data might be accessible to someone on the compromised system?
- What trust relationships does the potentially compromised system have with other systems?
- Are there compensating controls that an attacker would also have to penetrate?
- Does the attack seem to be commodity opportunistic malware?

2. Manage 🧠

Establish Your Incident Team:
- Management Liaison: Coordinate between the technical team and upper management.
- Incident Commander (IC): Ultimately responsible for ensuring that rules around confidentiality are set, communicated, and followed.
- Operations Coordinator (OC): Coordinate the technical side of the incident response.
- Legal Lead: Ensure the response complies with legal obligations.
- Communications Lead: Ensure internal and external communications are clear and effective.
Guidelines for Management:
- Maintain a clear line of command.
- Designate clearly defined roles.
- Keep a working record of debugging and mitigation steps as you go.
- Declare incidents early and often.

3. Declare 📣

Declare incidents as soon as they are recognized to ensure early containment and response.

4. Communicate ☎️

Avoid Misunderstandings: Ensure that all parties are aligned on the severity and impact of the incident.
Avoid Hedging: Be clear and concise in communication, avoiding ambiguity.
Meetings: Hold regular update meetings to ensure that the team and stakeholders are aware of the incident’s progress and next steps.

5. Operational Security (OpSec) 🥷

OpSec refers to the practice of keeping your response activities confidential.
- Use secure lines of communication.
- Avoid interacting directly with affected networks or components.
- Lock down affected accounts immediately.
- Shut down compromised systems to prevent further damage.
- Use different credentials when interacting with compromised systems.
- Ensure the right people are informed with the correct level of detail about the incident.

6. Investigate 🕵

Follow the OODA Loop: Observe, Orient, Decide, and Act.
- Forensic Imaging: Capture system images for later analysis.
- Memory Imaging: Capture the contents of system memory.
- File Carving: Extract useful files from data storage.
- Log Analysis: Analyze system logs to identify suspicious activity.
- Malware Analysis: Dissect malware to understand its function.
- Sharding the Investigation: Divide the investigation into manageable parts.
- Parallelize the Incident: Have different teams working on different aspects of the investigation simultaneously.

7. Handover ⛓️

Properly hand over any remaining tasks or investigations to ensure nothing is overlooked as the incident winds down.

8. Remediate 🛠️

Fix vulnerabilities and mitigate damage to prevent future incidents. Implement long-term security measures.

9. Close 📒

Close the incident formally. Review what went well, what could have been better, and document the lessons learned for future incidents.

High Availability ⚓️

refers to the design and implementation of systems, services, or applications to minimize downtime and ensure continuous operation. The goal of high availability is to ensure that a system is accessible and operational for the maximum possible amount of time.

Key Terms in HA 🔑

1. Uptime & Downtime ↕️

Uptime: The period during which a system is fully operational and accessible to users.
Downtime: The period during which the system is unavailable or not functioning as expected. HA systems aim to minimize downtime as much as possible.

2. Redundancy 👯

Redundancy refers to duplicating critical components or functions of a system to increase reliability. In HA systems, components like servers, databases, or networks are replicated so that if one fails, another can take over seamlessly.

Active-Active Redundancy: Multiple systems work simultaneously, and if one fails, the others continue without interruption.
Active-Passive Redundancy: A primary system works actively, while a backup system remains idle until a failure occurs.

3. Failover 🛤️🛤️🛤️

Failover is the process of switching to a standby or backup system when the primary system fails. In HA setups, failover is often automatic to minimize disruption. Failback refers to switching back to the primary system after recovery.

4. Load Balancing ⚖️

Load balancing distributes network or application traffic across multiple servers to ensure that no single server becomes overwhelmed. It enhances both performance and availability by balancing the load and rerouting traffic in case of server failures.

5. Elastic Scaling 🎋

Elastic scaling is the ability to automatically adjust resource capacity (compute, memory, etc.) based on workload demand. This is crucial for HA, as it prevents resource exhaustion during peak loads and reduces costs during low-demand periods.

Horizontal Scaling (Scaling Out): Adding more instances/servers to distribute load.
Vertical Scaling (Scaling Up): Increasing the resources of a single instance.
Auto-scaling: Automatic scaling based on real-time metrics.

In Kubernetes, elastic scaling is managed through Horizontal Pod Autoscalers (HPA), which automatically scale the number of pods in a deployment based on observed CPU utilization or other metrics. In HA systems using Kubernetes, autoscaling ensures that the right amount of resources are always allocated based on demand, contributing to both high performance and availability.

6. Fault Tolerance 🤦‍♂️😅

Fault tolerance refers to the system’s ability to continue operating correctly even when one or more components fail. Fault-tolerant systems detect, isolate, and handle faults without causing downtime.

7. Cluster 🧑‍🍳🧑‍🍳🧑‍🍳🧑‍🍳

A cluster is a group of servers or nodes working together as a single system to provide HA. Clustering ensures that if one node in the cluster fails, another node takes over its tasks, maintaining service availability.

In Kubernetes, a cluster consists of a set of worker machines, called nodes, that run containerized applications. Kubernetes ensures high availability by distributing workloads across multiple nodes and automatically replacing failed nodes or restarting containers.

In Warewulf, a cluster provisioning tool, high availability is addressed by enabling systems to quickly re-deploy compute nodes. Warewulf helps manage HA in high-performance computing (HPC) environments by ensuring compute nodes are readily available for workloads in case of node failure or maintenance.

8. Replication 💾💾💾

Replication is the process of duplicating data across multiple storage systems or servers. In HA, replication ensures that a copy of data exists on multiple systems, so if one system fails, another can continue providing access to the same data.

9. Disaster Recovery (DR) 📉📈

Disaster recovery involves strategies to restore a system after a catastrophic failure (e.g., data center failure). DR usually includes off-site backups and failover to remote data centers.

10. Mean Time Between Failures (MTBF) ⏱️

MTBF measures the average time between system failures. A higher MTBF indicates a more reliable system.

11. Mean Time to Repair (MTTR) ⏱️🛠️

MTTR measures how long it takes to restore a system to full functionality after a failure. Minimizing MTTR is critical for reducing downtime in HA systems.

12. Quorum 🐾

Quorum is the minimum number of nodes or components in a distributed system that must agree or function correctly to maintain availability. Quorum is often required in cluster setups to ensure consistent operation.

13. Geographic Redundancy 🌎

Geographic redundancy involves deploying systems across multiple geographical locations or data centers. This ensures that services remain available even if a region experiences a failure (e.g., due to natural disasters or power outages).

Relation to Kubernetes and Warewulf

I wanted to look at Kubernetes and Warewulf and how they relate to this topic as they are both systems written with GO and they are both well liked systems relating to HA.

I will write a future article all about GO and why it is fantastic at a later date. So I will just leave this point for now.

-A language built by the progenitors of UNIX and C for diverse connected systems.

Kubernetes 🍀: automated failover container orchestration elastic scaling with Horizontal Pod Autoscalers and geographic redundancy through multi-region clusters.
Warewulf 🐺: In HPC environments, Warewulf aids in HA by managing node provisioning and monitoring the health of the compute nodes. In case of failures, Warewulf can quickly re-deploy nodes, ensuring that the overall HPC workload is minimally disrupted.

Kubernetes and Warewulf both play key roles in maintaining high availability in modern infrastructures, with Kubernetes focusing on containerized applications and Warewulf on HPC cluster management.

HA & Incident Triage 🚨

High Availability (HA) systems improve operational security, allowing precise triage due to these factors:

1. Continuous Monitoring

HA systems continuously monitor performance to detect security threats in real time.

2. Continuous Comprehensive Logging

Logs are centralized, providing full visibility across the system for quick forensic analysis.

3. Declarative Structure

Declarative configurations enable automated remediation scripts for rapid self-healing.

4. Automated Alerting

Automated alerts prioritize and notify teams of security incidents as they happen.

Beyond triage, HA systems offer several other security benefits:

5. Containerized Components

Microservices are isolated, allowing affected components to be restarted without system-wide impact.

6. Elastic Scaling

HA systems dynamically scale resources to handle traffic spikes or increased workloads securely.

7. Automated Failover

Automated failover isolates compromised components, ensuring continuous uptime during incidents.

8. Data Replication by Design

Multiple data copies prevent loss and aid in disaster recovery during security breaches.

9. Event Replay

Event replay allows security teams to analyze incidents for better future defense.

Lessons Learned about HA

What I have learned from reading articles, searching and compiling these notes.

The Why

In today’s world, data is ingested, processed, and distributed at an unprecedented rate. To keep up, we must implement systems that operate with the same speed and frequency. The era of handling sequential tasks one at a time is over. High availability solutions such as Kubernetes, declarative infrastructure, cluster management, and stateless automation are now essential in modern IT environments.

Although some developers, administrators, and engineers express concerns about the complexity of these deployments, preferring simpler solutions, the reality is that we need to adapt to the increasing demands of data availability and security. Setting up a robust infrastructure now may have a higher initial labor cost, but it will undoubtedly reduce the risk of costly security incidents and system downtime in the future.

The What

The systems we must focus on include:

Kubernetes for orchestrating containerized applications across clusters
Declarative Infrastructure using Infrastructure-as-Code (IaC) for consistent and scalable deployments
Cluster Management to efficiently manage and scale distributed systems
Stateless Automation to ensure systems can self-heal and adapt quickly without human intervention

These technologies form the backbone of high availability infrastructures, designed to handle vast amounts of data without compromising performance or uptime.

The How

To implement these solutions, organizations should:

Automate deployment processes using tools like Terraform or Ansible to ensure repeatability and reliability.
Leverage Kubernetes clusters to manage microservices architectures, enabling fast scaling and robust fault tolerance.
Adopt a declarative approach by defining infrastructure states in code, allowing easier management and version control of systems.
Utilize stateless automation to reduce system reliance on individual components, making the system more resilient to failure and able to recover without manual intervention.

By adopting these practices, companies can build resilient, secure, and scalable infrastructures that meet the demands of today’s fast-paced data environments.

HA and SIR Synergy

The Why

As organizations become increasingly reliant on digital systems, security incidents are inevitable. Data breaches, malware attacks, and system vulnerabilities pose significant risks that can lead to costly downtime, reputational damage, and legal consequences. In this high-availability (HA) era, where uninterrupted service is critical, an effective Security Incident Response (SIR) plan is more important than ever.

The What

At its core, Security Incident Response (SIR) involves:

Identifying and responding to potential security incidents in real time
Containing incidents to prevent further damage and preserve critical systems
Eradicating malicious activity from systems quickly and efficiently
Recovering compromised systems and restoring operations to normal
Reviewing and improving security protocols based on lessons learned from incidents

In high-availability environments, this process works hand in hand with:

Failover mechanisms that redirect traffic or services away from compromised systems to maintain uptime
Resilient architectures where services are spread across clusters, reducing the impact of a security breach on overall operations
Automated remediation that helps detect and respond to threats before they cause major disruptions

Together, SIR and HA form a dual-layer defense to keep systems running while actively dealing with security threats.

The How

For Security Incident Response to effectively complement high availability, organizations should:

Implement proactive monitoring and detection: Use tools like SIEM (Security Information and Event Management)
Prepare for containment and isolation: High availability systems should be designed with segmentation in mind
Automate incident response and failover: Automate responses to specific types of threats, such as deploying firewalls, initiating backups, or failing over to standby systems.
Maintain constant communication: SIR teams should work closely with HA engineers to ensure that any security measures (e.g., shutting down affected systems) do not inadvertently cause a service disruption.
Review and test regularly: Just as HA systems undergo regular testing for failovers, security incident response plans should be tested in simulated scenarios.

Why study failure?

Proactive Prevention 🛠️: Identifying potential failure points allows you to take preventive measures and avoid system outages.
Faster Recovery 🚑: Knowing how systems can fail enables quicker response and minimizes downtime with a tested failover plan.
Resilience Building 🛡️: Understanding failure mechanisms helps design architectures that automatically handle issues, ensuring better uptime.
Risk Management ⚖️: Acknowledging failure risks helps balance uptime goals with realistic risk management and avoids catastrophic downtime.

Service Level Objectives (SLOs) 🎯

Intro

Service Level Objectives (SLOs) are specific, measurable targets 📊 that define the expected performance or quality level of a service, typically included in Service Level Agreements (SLAs). They help ensure services meet customer expectations by setting clear benchmarks for key metrics like reliability, availability, and response time.

SLOs help teams prioritize work, manage resources, and drive decisions based on real performance data. 🔧

Check out Google’s guide to SLOs here.

Bad Operations (BadOps) ⚠️

The unspoken rule of 100% uptime is a myth ❌.
100% uptime ≠ 100% reliability.
Each extra “nine” of uptime comes at a significant cost 💸.

Why Have SLOs? 🤔

Engineers are scarce 🧑‍💻: Impossible standards require more resources.
They inform decisions: The opportunity cost of reliability is clear with an objective.
SREs aren’t just automation experts; they’re driven by SLOs 🚀.
SLOs help prioritize tasks effectively 📋.
They define error budgets to manage acceptable failure rates.

Service Level Indicators (SLIs) 📏

Intro

A Service Level Indicator (SLI) is a specific, quantifiable metric 🔢 used to measure the performance of a service, often forming the foundation of SLOs.

Setting a Solid SLI ⚖️

Two out of five of these factors can be used to form a ratio that serves as a target SLI:

Number of successful HTTP requests / total HTTP requests (success rate) 🌐.
Number of gRPC calls completed in under 100ms / total gRPC requests ⏱️.
Number of search results using the entire data set / total search results, including gracefully degraded ones 🔍.
Number of stock check requests with data fresher than 10 minutes / total stock check requests 🛒.
Number of “good user minutes” based on defined criteria / total user minutes 📅.

Glossary of Terms 🙊

These are additional terms I have become familiar with that were not covered in this units notes but relayed by Scott.

Five 9’s

Refers to 99.999% availability, meaning less than 5 minutes of downtime per year.

Single point of failure

A component in a system whose failure will cause the entire system to fail.

Key Performance Indicators

Metrics used to measure the performance and effectiveness of a system or service.

SLI (Service Level Indicator)

A specific, quantifiable metric used to measure the performance of a service (e.g., latency, uptime).

SLO (Service Level Objective)

A target or goal for an SLI, defining acceptable performance for a service.

SLA (Service Level Agreement)

A formal contract that defines the level of service expected between a provider and a customer, including penalties for not meeting SLOs.

Active-Standby

A redundancy setup where one component is active while another remains on standby to take over if the active one fails.

Active-Active

A redundancy setup where multiple components work simultaneously to share the load, and failure in one component is covered by the others.

MTTD (Mean Time to Detect)

The average time it takes to detect a failure or incident in a system.

MTTR (Mean Time to Repair)

The average time it takes to recover from a failure or incident once it has been detected.

MTBF (Mean Time Between Failures) ‘Mentioned Once’

The average time a system operates without failure, used to measure system reliability.

Reflection

Remaining Qustions?
How will I put this into practice?

ProLUG Links ⛓️

ProLUG Admin Course Capstone Project Stage 2 🐧

Fri, 27 Sep 2024 00:00:00 +0000

Intro 👋

I’ve made significant progress with the Capstone Project, the project has expanded to include building a full website.

Planning

Integration

It seemed silly to develop such a complex mechanism that would later require integration into an unknown larger structure.

No site to begin with

After asking Scott about it, I learned that there has never been a ProLUG website and that he has been meaning to get something up.

Naturally

It seemed like taking on the entire website project was a good course of action. That way I would be able to decide the layout, functionality and logic all while learning new tools and techniques.

Technology

Going with GO

The website uses Go on the backend to serve templates and for back-end logic. While I’ve used Go for template servers before, incorporating templated components similar to Vue.js is new to me.

Here’s why Go excels for this type of project:

The codebase remains clean, minimal, and easy to understand.
The dependency system is simple, with no clutter of unnecessary packages.
It ensures memory safety.
Go’s HTTP server protects against injection, MITM, side-channel, and other common attacks.
HTML templating allows for modularity similar to Vue.js.
Garbage collection ensures long-term running without memory issues.
The project can be easily containerized or built into a single binary.
The Go module system (go.mod) ensures future stability through version control.

Storing Data

Initially, I was looking at SQLite for storing key-value pairs. However, after considering usability, I switched to a simple CSV file.

Why CSV

The stored data must be easily manipulated by the administrator in a human readable format. With CSV, Scott would be able to amend or edit certificate holders at a glance.

Operationally

This method is both secure and integral

How is it secure?

As stated before GO can securely transmit backend data without many common attacks. Additionally using GIT diff feature would ensure alterations are spotted should a compromise be made.

The Structure

Certificate Verifier

Pulling from this test CSV file:

Johny,Exampleseed,ProLUG Admin Course,2024-11-15,f078b6c4f26a2fae59d50fdb7c761a7f9523d240b2c18b332aac11e0
Kate,Testinger,ProLUG Admin Course,2024-11-15,a523e6d1ece5bb2758f71993ba1a460024dcd10243fa17654af90257
Het, Tanis, ProLUG Admin Course,2015-11-02,a523e6d1ece5bb2758f71993ba1a460024dcd10243fa17324af90257

Using this logic:

// Certificate verification handler
// This function is the HTTP handler responsible for verifying certificates.
// It is triggered when a user accesses the relevant URL and checks for a certificate based on a hash parameter.
func certHandler(w http.ResponseWriter, r *http.Request) {

 // Extract hash value from the URL query parameters.
 // The hash is used to identify the certificate to verify.
 hash := r.URL.Query().Get("hash")

 // If no hash is provided in the query, return a "Bad Request" error.
 if hash == "" {
 // Send an HTTP error response indicating that the hash is missing.
 http.Error(w, "Hash not provided", http.StatusBadRequest)
 return // Exit the handler function since no hash is available.
 }

 // Load certificates from a CSV file.
 // This function attempts to retrieve all certificates stored in the "certificates.csv" file.
 certificates, err := LoadCertificates("certificates.csv")

 // If there's an error loading the certificates (e.g., file not found or corrupted), log the error.
 if err != nil {
 // Log the error message to the server logs and return a "Server Error" to the client.
 log.Printf("Error loading certificates: %v", err)
 http.Error(w, "Server Error", http.StatusInternalServerError)
 return // Exit the handler function due to the error.
 }

 // Initialize a pointer to a Certificate struct.
 // This will hold the matching certificate if a valid hash is found.
 var cert *Certificate

 // Iterate over all loaded certificates.
 // Check each certificate's hash to see if it matches the one provided in the query.
 for _, c := range certificates {
 if c.Hash == hash {
 // If a matching certificate is found, assign it to the 'cert' pointer and break the loop.
 cert = &c
 break
 }
 }

 // Render the result page using the template engine.
 // Pass the certificate (or nil if not found) to the template "result.html" for display.
 err = templates.ExecuteTemplate(w, "result.html", map[string]interface{}{
 "Certificate": cert, // Include the certificate as part of the template data.
 })

 // If there's an error rendering the page (e.g., template not found or syntax error), log the issue.
 if err != nil {
 // Log the error message to the server logs and return an "Internal Server Error" to the client.
 log.Printf("Error rendering verification page: %v", err)
 http.Error(w, "Internal Server Error", http.StatusInternalServerError)
 }
}

package main

// Certificate represents a learning certificate with associated data.
type Certificate struct {
 FirstName string
 LastName string
 CertType string
 DateCompleted string
 Hash string
}

The prompt

Inputting a UID hash string

Matching string results in a info retrieval

An unrecognized string results in an error message

Current Site 💄

After getting a working verification system worked out, I moved on to creating a broader website.

Styling

I like sites with a clean, easy to use and accessible appearance. I have a solid understanding of graphic design for accessibility and wanted to put that into practice here. So the design is stark, compact, clear and contrasting. This language carries throughout.

Technique

So as mentioned previously, I am serving html templates with GO. This can be pretty simple, we can serve a single basic html page with no modular components. I wanted to try out some fancy tricks using modularized components in this project, cutting down on the size of each template. The way this works within this project is very similar to Vue.js templating. We must declare a template as a component by encapsulating the html with tags.

for example the Navigation bar is coded similarly to any old navbar and saved in templates as navbar.html. The fancy bit is enclosing the code with the tag

Defining a component

{{ define "navbar" }} Code goes here {{ end }}

Using a template component

{{ template "navbar" . }}

Current Home Page

Current About Page

Current Join Page

Current Certify Page

Current Verify Page

Next Steps… 🥾

I plan on discussing this in the Code Cove group in ProLUG
I plan on creating a scoreboard as ideated in unit 2 by scott by reading from a CSV and writing to an order list
I plan on spiffying up the appearance and useability of the site using tailwind and a bit more js logic

ProLUG Links ⛓️

ProLUG Admin Course Unit 2 🐧

Fri, 27 Sep 2024 00:00:00 +0000

Essential Tools 🛠️

Intro

We have reached the second week of the PAC and things are getting more serious. Now that we are familiar with the flow of the course and expectations are set, the second lesson has started with some momentum. 💺

Unit 2 lab exercise notes 🧪

I am completing the labs locally carefully covering every command listed by running them and checking output, making sure to pay attention to details like options. It was fairly time consuming, yet I learned quite a bit.

🧭 Basic CLI exercise:

cd ~ # Change Directory to Home
ls ~ # List
mkdir evaluation # Make Directory called ’evaluation'
mkdir evaluation/test/round6 # This fails why? We need to use the Parent option ‘-p’
mkdir -p evaluation/test/round6 # Make parent directory with subdirectories
cd evaluation # change directory to evaluation
pwd # present/print working directory = /home/ts/evaluation
touch testfile1 # create file name ’testfile1’ in current directory
ls # list = test testfile1
touch testfile{2..10} # Creates a numbered range of testfile touch .hfile .hfile2 .hfile3 # this creates three dot files which are not visible to a normal listing method, we must use list all (ls -a)

🔍 Gathering System Information

I am running Rocky Linux 9 on a local Proxmox container for this course.

hostname = localhost.localdomain
uname # system info command which stands for Unix Name
uname -a # shows a comprehensive list of system info Kernal name# Hostname, Kernel release, Kernel version, Machine hardware name, Processor type, Hardware platform, Operating system.
uname -r # the -r option stands for release and shows the kernel release version.

🧠 Checking the amount of RAM/Memory

sudo cat/proc/meminfo # concatenate process memory information
free # displays total, used, free, shared, buffer, and available memory (RAM and SWAP)
free -m # megabyte display option for the free command

👨🏻‍💻 Checking the number of processors and processor info

cat/proc/cpuinfo # concatenate process central processor unit information = 4 processors on my lab.
cat /proc/cpuinfo | grep proc | wc -l # what we are doing here is piping the output of the previous command to grep which is filtering lines starting with proc, then we pipe through to word count with line option to cleverly count the number of processors, the result is 4

💾 Checking storage usage and mounted filesystems

df # the disk free command
df -h # we are adding the human readable option to df in order to more easily ascertain what is going on.
df -h | grep -i var #

💾 Mounting a new file system

Common `mount` Command Options

-t # Specify the filesystem type (e.g., ext4, xfs, vfat).
-o # Specify options in a comma-separated list (e.g., rw, ro, noexec).
-a # Mount all filesystems defined in /etc/fstab.
-r # Mount the filesystem as read-only.
-w # Mount the filesystem as read-write.
-v # Verbose mode, provides more detailed output.
–bind # Bind mount, remount part of the filesystem elsewhere.
–make-shared # Make a mounted file system shared (for propagating mount events).
-L # Mount by filesystem label.
-U # Mount by UUID.
mount | grep –i home# lists all mounts starting with the case sensitive phrase home (I was not getting any result 🤔)
mount | grep -i sd# this lists all SCSK disks, normally named sda, sdb, sdc etc…

Trying out alternative commands

findmt# find mount
cat /proc/mounts# concatenate in-process mounts
lsblk# list block devices

I was unaware that commands could be chained together with a semicolon

cd ~; pwd; df -h# change directory to home, then print working directory, then list disk usage in a human-readable format 🧐.
du -sh# disk usage summary in a human-readable format.

`du` Command Options

 **`-a`**# Display disk usage for all files, not just directories.
**`-h`**# Show disk usage in human-readable format (e.g., KB, MB, GB).
**`-s`**# Show summary of total disk usage for a directory.
**`-c`**# Produce a grand total at the end.
**`-L`**# Follow symbolic links (default `du` doesn't follow symlinks).
**`-d N`** or **`--max-depth=N`**# Limit directory depth to N levels.
**`--time`**# Show modification time of directories.
**`-b`**# Show disk usage in bytes.
**`-k`**# Show disk usage in kilobytes.
**`-m`**# Show disk usage in megabytes.
**`--apparent-size`**# Show the apparent size instead of disk usage.
**`-x`**# Skip directories on different file systems.
**`--exclude=PATTERN`**# Exclude files matching a pattern.

Uptime

 `uptime`# Shows time in HH:MM:SS Unix time format in local time. If the time is not set, like in my instance, I get UTC 0🧐. Next, it shows how long the system has been running, the number of logged-in users, and finally, the load average over the last 1, 5, and 15 minutes.

Last

 **`last`**# Lists all users who have logged in from latest to oldest.
**`w`**# Lists current users and associated processes.
**`who`**# Shows who is currently logged on. The output `pts/0` stands for pseudo terminal, typically shown when logged in remotely. It will show either an IPv4 or IPv6 address and the login time. When I logged into my Ubuntu server via SSH, I was shown the IPv6 address. 🤯
**`whoami`**# Tells which user you are.

Checking environment

 **`printenv`**# Shows a long list of environmental information.
**`printenv | grep -i home`**# Shows that I am in the home directory of `root`.
**`id`**# Shows a lot of info about UID, GID, and group memberships, including SELinux policies.
**`echo $SHELL`**# Displays the path of the shell environment variable, in my case, `/bin/bash`.

Check running processes and services

 **`ps -aux | more`**
**`ps -ef | more`**
**`ps -ef | wc -l`**

Check memory usage

Run each of these commands individually for understanding:

 **`free -m`**
**`free -m | egrep "Mem|Swap"`**
**`free -m | egrep "Mem|Swap" | awk '{print $1, $2, $3}'`**: What are the first, second, and third columns? How would I know the variable names?
**`free -t | egrep "Mem|Swap" | awk '{print $1 " Used Space = " ($3 / $2) * 100"%"}'`**: Similar question for these variables.

Testing out scripts

Q: Have you ever written a basic check script or touched on conditional statements or loops?

A: Yes very basic 😅

(Use ctrl + c to break out of these) 👍

 while true; do free -m; sleep 3; done
Watch this output for a few and then break with ctrl + c
Try to edit this to wait for 5 seconds
Try to add a check for uptime and date each loop with a blank line between each and 10 second wait:
while true; do date; uptime; free -m; echo “ “; sleep 10; done
Since we can wrap anything inside of our while statements, let’s try adding something from earlier:
While true; do free -t | egrep "Mem|Swap" | awk '{print $1 " Used Space = " ($3 / $2) * 100"%"}'; sleep 3; done 👍

seq 1 10

Q: What did this do?

A: it counts from 1 to 10, creating a new line for each char.

Q: Can you man seq to modify that to count from 2 to 20 by 2’s?

A: yes by using ‘First Increment Last, so 2 2 20’

Let’s make a counting for loop from that sequence ✅ for i in seq 1 20; do echo “I am counting i and am on $i times through the loop”; done 👍

Q: Can you tell me what is the difference or significance of the $ in the command above? What does that denote to the system?

A: we are using the $ to access a variable.

Uptime – The time a system has been running for without interruption. Standard input - stdin ‘0’ the default stream where a program receives data. Standard output - stdout ‘1’ refers to default stream output of a program. Standard error – stderr ‘2’ refers to a default stream wherein an error is sent.

Reflecting on Unit 1 🤔

Week 1 went well for me. I was very excited to start this course as it strongly aligns with my professional interests, and it came at a great time. I am incredibly grateful to have access to such a talented group of like-minded and enthusiastic professionals.

I managed to complete everything for Week 1 by preparing in advance. For example, I created this blog and started some of the materials early. The night before the course began, I went through Vim Tutor and Vim Adventures to give myself a bit of a head start, as my time is currently divided between various responsibilities. I also dedicated a significant portion of my time to starting my capstone project and troubleshooting issues.

I’ve set aside 8 hours per week, as suggested by Scott, for attending lectures, reading materials, practicing labs, and doing general research. During Week 1, I used roughly 6 hours, so I believe I’ve allocated enough time to complete the course work.

My note-taking has been excellent for this course. Prior to starting, I learned how essential note-taking is to the administration and engineering process, so I began using LogSeq as my note-taking tool four months ago. This gave me time to familiarize myself with its features, allowing me to dive into the lectures fully prepared. I’m also a fast touch typist, so I can easily listen and type simultaneously. After each lecture, I review and organize my notes by nesting and tagging information to relate it to other topics. Additionally, I’ve been reflecting on everything by writing in this blog.

Security Enhanced Linux

Brief intro to SELinux

SELinux (Security-Enhanced Linux) is a security module integrated into the Linux kernel that enforces mandatory access control policies to provide more granular control over how processes and users interact with files and resources. By defining strict rules on what actions each program or user can perform, SELinux significantly enhances the security posture of a Linux system, helping to prevent unauthorized access and exploits.

Brief intro to Apparmor

AppArmor (Application Armor) is a Linux security module that enforces access control policies based on file paths, limiting what resources applications can access to enhance system security. It provides a simpler, profile-based alternative to SELinux, allowing administrators to create restrictive environments for individual applications without requiring deep changes to system configuration.

Key Terminology

Mandatory Access Control – Discretionary Access Control – Security contexts (SELINUX) – SELINUX operating modes -

Comparing AppArmor and SELinux for Container Separation 1

This article provides a brief comparison between SELinux and AppArmor regarding their effectiveness in securely separating containers.

After reading, it became clear that AppArmor is not an ideal choice for DevSecOps when it comes to securely separating containers. This is due to AppArmor’s lack of support for Multi-Category Security (MCS). MCS allows for a hierarchy of controls, granting varying levels of access.

Therefore, if you’re looking to securely separate containers without relying on Virtual Machines—which can be costly—SELinux emerges as the better option for such tasks.

Key Takeaways 📝

AppArmor is not label-based, unlike SELinux.
AppArmor is generally seen as more user-friendly.
AppArmor has fewer controls compared to SELinux.
AppArmor has fewer operations available.
Both support the Type Enforcement security model (a form of mandatory access control).
The security model is based on rules where subjects (like processes or users) are allowed to access objects (e.g., files, directories, sockets, etc.).
AppArmor lacks Multi-Level Security (MLS).
AppArmor does not support Multi-Category Security (MCS).
🔄 Because MCS is unavailable, AppArmor cannot maintain proper separation between containers.
⚠️ The default container policy in AppArmor is very loose, which could present security risks.

Quick Comparison Table 🔍

Technology	Type Enforcement	MLS/MCS	Policy Generator	Generator for Containers
AppArmor	Yes	No	Yes	No
SELinux	Yes	Yes	No*	Yes

By understanding these differences, it’s clear that SELinux provides a more secure framework for container separation, making it a crucial tool for modern DevSecOps workflows.

Enabling SELinux

Before enabling SELinux, you can verify its current status by running the sestatus command, which provides the Security Enhanced Status of the system. To activate or configure SELinux, you need to modify the configuration file located at /etc/selinux/config. SELinux can operate in two modes:

Enforcing: SELinux policies are actively enforced, and violations are blocked.
Permissive: Policies are not enforced, but violations are logged for review, allowing for troubleshooting without blocking actions.

SELinux Contexts

A context in SELinux is a set of security labels used to manage access to files, processes, and other system resources. The context is composed of several fields:

system_u:object_r:httpd_sys_content_t:s0

user:role:type:level

Breakdown of Context Components:

User: Represents the SELinux identity of a subject (process) or object (file). Role: Groups privileges for processes and users, determining what they are allowed to do. Type: Defines how subjects (processes) can interact with objects (files or resources). Level: Used in Multi-Level Security (MLS) or Multi-Category Security (MCS) systems for additional granularity in access control.

Taking SELinux for a spin on a Rocky 9 VM in my Proxmox Homelab

A few weeks ago, I learned how to create uninitialized templates with Proxmox, meaning the SSH keys are generated for each copy of the template. This is done with cloud init.

So today I created a clone of my template specifically for getting hands on experience with SElinux

I opened the /etc/selinux/config and set se to enforce. I then ran sestatus to confirm selinux had been activated. I created a new user policy for myself with semanage user -a -R ‘staff_r webadm_r’ Treasure here is the result after running semanage user -l

SELinux scenario contemplation

You follow your company instructions to add a new user to a set of 10 Linux servers.
They cannot access just one (1) of the servers.
When you review the differences in the servers you see that the server they cannot access is running SELINUX.
On checking other users have no problem getting into the system.
You find nothing in the documentation (typical) about this different system or how these users are accessing it.
What do you do? Where do you check?

From the given information, it seems likely that SELinux is denying access to the new user. To troubleshoot this, I would take the following steps:

Check SELinux User Configuration:
I would run the command semanage user -l to list all SELinux users and confirm whether the new user exists within the SELinux policy.
Verify MLS/MCS Levels and Roles:
If the user is already registered, I would review the user’s Multi-Level Security (MLS) or Multi-Category Security (MCS) level and verify the role assigned to the user. It’s important that the user’s role matches those that have the necessary access permissions.
Assign Appropriate Role:
If the current role restricts access, I would assign the user to an SELinux role that permits access to the system, ensuring that role-based access control (RBAC) is properly configured.

Interactive Killercoda Lab 2

I hope this was designed to be challenging, because I was challenged 😅

Despite paying close attention, I somehow got confused by this challenge. I suppose the exercise was designed to force me to read through documentation, which was tough. But I am proud to say that through trial and error, I was able to get through.

Commands I looked up in the documentation:

kubectl get deployment spacecow -n moon -o yaml > spacecow-deployment.yaml
vim spacecow-deployment.yaml
kubectl apply -f spacecow-deployment.yaml

Troubleshooting

How Troubleshooting Differs Between Administrator and Engineer

Administrators focus on restoring functionality to existing systems, while Engineers work on building and implementing new systems or improvements.
Administrators have existing documentation to reference for troubleshooting, whereas Engineers are responsible for creating new documentation.
Administrators can compare similar systems for inconsistencies, but Engineers may be working with entirely new systems or solutions, making such comparisons difficult.

Administrators:

Fix: Restore systems to their previous, working state.
Optimize: Use tools like Ansible or custom scripts to streamline operations and automate repetitive tasks.
Operate: Handle daily tasks that cannot be automated, such as monitoring or handling unexpected issues.

Engineers:

Prepare: Configure pre-built tools for daily use and ensure they integrate smoothly with existing systems.
Plan: Anticipate and strategize operational changes, such as the introduction of new tools or system upgrades.
Design: Create new tools, systems, or measures that address organizational needs and future challenges.
Implement: Deploy new or upgraded systems and produce comprehensive documentation to support ongoing maintenance and troubleshooting.

Key Differences in Troubleshooting:

Administrators typically deal with reactive troubleshooting, where the goal is to quickly restore functionality after an issue occurs. Engineers, on the other hand, often engage in proactive troubleshooting by identifying potential problems before they happen, ensuring that systems are scalable and sustainable.
While administrators may focus on resolving immediate operational issues, engineers are responsible for the long-term stability of systems, often troubleshooting while keeping future requirements in mind.

Troubleshooting Procedure

1. Identify the Problem

Gather Information: Ask questions, check logs, or run diagnostics to gather all necessary details.
Reproduce the Issue: Try to replicate the problem to confirm its existence and scope.

2. Establish a Theory of Cause

Look for Patterns: Are there recurring errors or conditions?
Use Known Issues: Refer to documentation, previous cases, or online resources to match symptoms to potential causes.
Hypothesize: Develop potential explanations based on the data you’ve collected.

3. Test the Theory

Isolate Variables: Disable or eliminate components one by one to test your theory.
Try Quick Fixes: Implement minor changes to see if the problem is resolved, such as restarting services or clearing caches.

4. Create an Action Plan

Plan for Resolution: Identify the steps needed to fix the problem if your theory is confirmed.
Consider Impact: Make sure to assess any risks or side effects of your solution.

5. Implement the Solution

Execute the Fix: Apply the necessary changes, whether it’s a configuration adjustment, software update, or hardware replacement.
Monitor: Observe the system closely to ensure the issue doesn’t recur.

6. Verify Full System Functionality

Test the Fix: Confirm the problem is fully resolved and that other services or systems haven’t been affected.
Ask Stakeholders: Ensure end-users or relevant parties can validate the fix.

7. Document the Findings

Write It Down: Document the issue, the root cause, the solution, and any potential future prevention methods.
Share Knowledge: Ensure others on your team or organization can reference the solution.

8. Prevent Future Issues

Review System: Investigate whether the root cause indicates a larger problem.
Proactive Monitoring: Set up alerts, automations, or optimizations to prevent recurrence.

Notes

DHCP

DHCP Should only be assigned by one server
What is a BootP?
DHCP is a process that comes from BootP?
Layer 2 based on Mac address

First day on the job, you must inventory the servers

Nmap
History on CommandLine (Find the IP’s)
Ping Sweep for OpenSSH
Create a local Inventory File

Questions I have from this week:

How often does a Jr. Admin implement SELinux?
If apparmor is so much less functional than SELinux, why is it still a popular choice? is it due to ease of use?
you mentioned Irata? I was unable to look it up.
free –m| egrep “Mem|Swap” | awk ‘{print $1, $2, $3}’: what is 1, 2 and 3. Is there an easy way to know which variables pertain to which data?

How are you going to use what you’ve learned in your current role?

I have no direct application for most of this stuff at the moment. however I will definitely use scripting in my current position in order to automate tasks. Furthermore, I will def remember several commands from the labs that I will likely use daily.

ProLUG Links ⛓️

Comparing Apparmor to Se Linux Isolation Article Redhat. ↩︎
Apparmor Interactive Lab Site Killercoda. ↩︎

ProLUG Admin Course Unit 1 🐧

Sun, 15 Sep 2024 00:00:00 +0000

Command Line Interface Primer 👨🏻‍🏫

Intro 👋

Well the ProLUG course is a go. We had a great meeting yesterday with a group of over 80 people. There was a slight snafu regarding thread size limits, so mid lesson we jumped into a presenter room. The session ran for a solid two hours wherein Scott went over command line interface essentials, networking essentials, VIM, pathing, redirects, Xargs, man pages and more. Although I have decent foundational knowledge of Linux, I know that there is always something to learn or better understand. I came away from this first class with a handful of things to review and practice for sure.

prompts 🦮

Group discussions have kicked off early in #ProLUG-Projects. Scott has prompted us with a few questions for Unit 1 that we are to post and discuss. I thought it would be good to post these answers in a longer form on my blog. This acts as a review of yesterdays work and gives me a chance to further reflect.

Discussion Post 1: Technical Skill Assessment 🔍

Using this rubric, we are to self-assess our skills in various areas with a score from 1-10, giving us a total score at the end. My score was 48 out of a potential 110. I would never rate myself above a 7, as that feels like Demi-God territory!

Given the pace we started at in the first class, I’m confident that my skills will improve throughout this course. I plan to really go for it and challenge myself along the way.

Skill	High (8-10)	Mid (4-7)	Low (0-3)	Total
Linux	-	6	-
Storage	-	5	-
Security	-	6	-
Networking	-	-	3
Git	-	7	-
Automation	-	5	-
Monitoring	-	-	3
Database	-	4	-
Cloud	-	6	-
Kubernetes	-	-	3
Total	-	-	-	48

What do I hope to learn in this course? 🤔

I aim to gain hands-on experience in all aspects of system administration. While I have studied Linux and system administration, I understand that skill levels in this field can vary widely. As I work toward becoming a Systems Engineer, I recognize the importance of consistently challenging myself. Through this journey, I hope to build confidence in managing existing systems, configuring services, securing environments, and automating tasks.

What type of career are you looking for? 🥕

I am looking to become a full time Linux System Engineer. I am currently in an administrative support role wherein I am tasked with setting up new systems on a small scale. I would like to be able to manage a large number of servers with automation.

Discussion Post 2: Target Job and Skills Gap 🎯

Our second task is to look for a job we would like to target and list the requirements posted for that given job.
I found this job posted by OERP Canada, located in Toronto, Ontario, with the following information.

Job Overview

The role will be 100% remote for now (#socialdistancing). Eventually, it will convert into an onsite role in Toronto, ON. We are only accepting applications from residents of Canada.

OERP Canada is looking for a full-time, experienced LINUX System Administrator to help maintain our operations and support our Odoo application deployment team, which deploys the Odoo Business Application suite for our clients. The successful candidate will have a technical background in LINUX, experience with hosting, web and proxy servers, database administration, mail routing, network security, strong communication skills, and the willingness and ability to think creatively and flexibly. A desire to learn is an important requirement of being part of our team.

This position is both creative and rigorous by nature—you need to think outside the box. We expect the candidate to be proactive and have a “get it done” spirit.

Responsibilities

Manage cloud and on-premise hosting environments
Endless security improvements
Improve automation tools
Improve scripting
Provide technical support as needed

Must-Have Skills

Ubuntu / Debian server
NGINX
Security best practices: Firewalls & WAF
Automation tools and server scripting
Functional knowledge of networking: VPNs, DNS, mail routing and delivery, etc.
University degree or 3-5 years of work experience

Nice-to-Have Skills

PostgreSQL high availability, load balancing, and replication
Virtualization: VMware, vSphere, KVM
VCS/Git working knowledge
FreeBSD

What I am currently able to do in this position 💪🏻

Solid experience with Debian/Ubuntu
Confident in managing both cloud and on-premise hosting environments
Skilled in improving automation through the use of Ansible, Terraform, and Bash scripting
While I may not have 3-5 years of continuous working experience, I have been involved in IT work on and off
Experienced in setting up and managing PostgreSQL
Proficient with Oracle VirtualBox and Proxmox, which I believe translates well to VMware, vSphere, and KVM
Knowledgeable in Git, including branch management and command-line usage
Familiar with FreeBSD and BSD-specific commands

What I am currently incapable of 😓

Lack confidence in managing firewalls
Limited experience with VPNs, DNS, and mail handling
Basic hands-on experience with NGINX, but skills are limited
Minimal technical support experience

How I plan to close the gap on these missing skills 🎯

I plan to close this gap by learning more about general Linux security and networking principles from this course. Additionally, I’ve already purchased a recently published book by Michael Lucas on running and maintaining mail servers, which I intend to skim through. Additionally, I am studying ITIL to improve my understanding of technical support and service management.

ProLUG Links ⛓️

Actually completing VimTutor

Tue, 10 Sep 2024 00:00:00 +0000

Intro

Vim is a program created by the late Bram Moolenaar as an improvement to an older program called Vi. It is a modal text editor, meaning there are different modes of usage — I’ll get into that later. While strongly associated with UNIX systems, Vim is available on other platforms as well. It is used to efficiently create and modify files from a simple terminal, requiring very limited system resources. Vim is a foundational tool for mastering Linux, allowing one to navigate the directory structure and edit configuration files like no other. Furthermore, Vi and/or Vim come bundled with most minimal Linux distributions.

Vim Tutor

Vim Tutor is an in-built interactive tutorial that comes pre-packaged with Vim, designed to bring novices up to speed with the various modes and functions of this nimble tool. I had hopped into Vim Tutor before but never felt the need to complete it, as it seemed long and complex. Recently, my teacher and mentor Scott C. mentioned that this would be one of the first lessons we would undertake as part of the ProLUG system administration course. Being a dedicated student, I decided to give it my full attention — and I’m glad I did.

The Halfway Point

In previous attempts, I had reached the halfway point and felt confident I had gained enough information to edit files. Some functions are unintuitive at first, like using the hjkl keys for navigation instead of arrow keys. There are many jokes about the overload people feel when they first encounter Vim’s conventions. Jokes like never being able to exit the program or forgetting one’s own name after memorizing commands are common. Mastering Vim requires memorizing commands until they become second nature, which simply takes time. This is why I hadn’t completed Vim Tutor before — there’s only so much one can absorb before needing a break. However, this time, I was able to make it through to the end, pleasantly surprised by the new techniques I had learned.

Understanding the Modes

Modes are what make Vim a powerful tool. There are three modes in Vim: Normal, Insert, and Visual.

Normal mode is the default mode, where you navigate a document using motions. It protects the document from accidental modifications and allows for a broader range of single-key commands.
Insert mode is where characters or strings are added — fairly self-explanatory.
Visual mode allows you to make selections, much like clicking and dragging with a mouse to highlight text.

Although I had known about these modes, it once seemed improbable that I would ever memorize them well enough to use them reflexively. However, with practice, I have done just that. Combining motions and commands can lead to huge efficiency gains. For example, you can navigate to a specific line, delete the first three letters, and paste something at the end of the line with just a few keystrokes.

What I Recently Picked Up

Stringing Commands

What did I actually learn that I will immediately employ? Well, I now have a solid grasp of most single-key commands, so I’ll start stringing commands and motions together, such as deleting three words and appending text to the end of a line.

Global Search and Replace

Additionally, I had known about global search and replace, but I hadn’t really used it before since it’s only useful for certain file types.

Buffers and Deleted Lines

I also learned that Vim’s buffer, much like a clipboard, stores strings for later use — and deleted lines are stored there as well. I’ll be more mindful of using the buffer going forward.

Running Shell Commands

Lastly, running shell commands from within Vim seemed inefficient to me before, since I could easily jump in and out of files. However, I now understand that it can speed things up.

In Summary

Vim isn’t something you just pick up on a whim. It’s more like a martial art, where you practice movements until they become second nature. In a world dominated by graphical user interfaces, Vim can seem like a waste of time and effort. Yet, when you’re working under the hood of an operating system, especially on bare-bones UNIX servers, it becomes clear that if everything is a file and many configurations must be made, Vim is the right tool for the job.

New WebDev framework

Wed, 04 Sep 2024 00:00:00 +0000

Intro

I frequent a Discord group called ProLUG (Professional Linux User Group). The group is organized around cooperative, project-based learning, led by experienced members. One of these groups, called Coding Cove, is led by Ryan Long (SaltyCatfish). Ryan is a very experienced and senior developer who introduces us to new concepts on a weekly basis.

Vue

This past weekend, Ryan introduced Vue.js and the ideas behind the framework. We spent over an hour discussing and interacting with Vue, and I came away from the meeting excited to learn more and start building projects.

Why Vue is Exciting

What I really liked about Vue is the concept of templates and components. These ideas seemed logical from the start. Templates can be created and reused on multiple pages, while components provide modularity. Together, these features result in a very clean and easy-to-read codebase, especially for larger projects.

What I Was Using

Prior to learning about Vue, I had completed a few projects using HTML, Tailwind CSS, and React. While I’m a big fan of Go (for reasons I won’t get into here), I found React to be cumbersome. React often creates messy code, especially when combined with Tailwind.

Go + Vue?

After the meeting, I felt inspired to explore combining the power of Go with Vue. I started searching for examples, and it’s definitely doable for simple or small projects. However, when I looked at more advanced projects, the situation became complicated. Gluing a Go backend to a Vue frontend felt relatively simple at first, but when you introduce middleware, SSR/CSR, OAuth2, and more, things quickly become complex.

Then I Found Nuxt

Nuxt is a “batteries-included” framework built on top of Vue. It comes with all the modern build tools and services needed for a professional project, and everything ties together elegantly.

Where I Am Now

I’ve done a deep dive into Nuxt, and it feels very comfortable to work with. After purchasing a course on Udemy, I’ve completed a few simple projects and intend to build a mid-level project soon. With Vue’s templating and components, along with Tailwind’s efficiency, I’m confident I can create a functional and aesthetically pleasing project that runs efficiently across different systems.

ProLUG Admin Course 🐧

Sun, 01 Sep 2024 00:00:00 +0000

Intro 👋

I am a member of a fantastic group called the Professional Linux User Group lead by Scott Champine. The home base is on Discord where over 2000 people discuss Linux daily. Scott and other members teach via Live-Streaming and Interactive Labs with supporting documentation. Scott has created a 16 week deep dive course into linux adminstration starting September 14th that I have gladly enrolled in. This course will take place every Saturday via a livestream with interactive Q/A wherein participants leave with assignments. This blog is in part, related to the course as I will document my learning among other things.

Links ⛓️

That Unix file is not deleted 👺

Sun, 01 Sep 2024 00:00:00 +0000

Removing a file in Unix does not actually remove it from memory. You are only reducing the number of Hard-Links pointing to the file to ‘0’. In order to actually delete the information is to overwrite it. This will happen over time as the disk fills, or if one would like to delete a file in a timely fashion should over write empty drive sectors.

Studying for the LPIC1 Certification 📖

Fri, 30 Aug 2024 00:00:00 +0000

Intro 👋

I am intent on becoming a Linux Professional Institute Certified Administrator. It has been a goal of mine for a little while now. However goals are just aspirational, reaching a goal such as this requires a structured, systematic approach. As a life-long learner, I am always improving my study methods by being more efficient with my time and energy. As we all know life can get in the way of our goals, so we must find methods that work when conditions are not ideal for studying. For instance, I brake up my study sessions into smaller, more bite sized chunks that I can do when time permits. I take much more comprehensive notes than I did in the past and I work on wrought memorization through the use of flash cards; something that I really wasn’t into before.

Creating a Work Breakdown Structure

The first thing I do when starting to study a new topic is creating a simple structure from a high level view. A handful of bullet points can simply outline the structure of the knowledge. In regards to the LPIC, most of this had already been done as the curriculum for the exam is open source.

Hands on practice

For each item that the course cover, I set up a lab and run through all of the commands by actually typing them in and seeing what kind of errors or typos I may run into. The LPIC has a huge list of commands that we must remember which makes it very difficult for me.

Taking comprehensive notes

I recently started using LogSeq, a markdown ++ notetaking applicaton that allows for interlinking of notes. This has really upped my game when it comes to keeping track of notes and refering back to them. In the past, I would just take notes for the task at hand, not really refering back to them often. I was just taking notes as a memory retention aid.

Flash Cards

Another recent addition to my study arsenal is Anki, a smart flashcard application that I use for wrought memorization. The application and many flash card decks are open source, so I was up and running in a short period of time with a large deck of LPIC flashcards.

Udemy / Youtube

Though video courses are pretty low bandwidth information sources, I still like to follow chapterized courses, especially on Udemy. On a sale day I can grab a comprehensive chapterised video course for less than 20 Dollars, a steal in comparison to a college course.

ProxMox for the win 🏆

Mon, 26 Aug 2024 00:00:00 +0000

Intro 👋

Proxmox is a free, open-source virtualization host built on top of Debian. It can run both virtual machines (VMs) and Linux containers, offering a wide array of features. 🚀

I’m always learning new systems and software. Before using Proxmox, I was repeatedly installing and reinstalling Unix systems on an old laptop. This helped me understand installation, configuration, and a host of other things—but it was very tedious and inefficient. That laptop was underpowered, so running multiple VMs was out of the question. I needed a multicore x86 machine with ample memory to achieve this.

Wanting to take my systems engineering learning more seriously, I decided to build an x86 machine dedicated to learning without the fear of messing things up. While researching virtualization software, I discovered Proxmox.

Installation 🛠️

Installing Proxmox is straightforward, especially for someone who has installed many major distros on bare-metal systems. The process involves downloading the latest ISO, creating a boot disk using Balena Etcher, and then following a series of prompts, choosing the target disk and locale—just like a typical Debian installation.

Accessing Proxmox 🌐

Once installed, Proxmox operates as a headless network machine. If you plug in a monitor, you’ll only see a black screen showing the host address—nothing more. The way to interact with Proxmox is through a browser on a device connected to the same local network.

For example, I open a browser on my MacBook, navigate to the host’s IP address, and am greeted by the login screen. You would have set up your username and password during installation, so just log in from there. 👍

The Dashboard 📊

Proxmox provides an easy-to-use dashboard that shows system load and memory usage at a glance. Creating a virtual machine is as simple as clicking “Create New VM” and uploading a suitable ISO. The real learning comes when you automate the process and manage multiple machines.

Learning Platform 🎓

Proxmox gives you the freedom to create, break, and destroy VMs—allowing you to learn new things without sweating the small stuff. Through it, I’ve gained a wealth of systems engineering knowledge by completing small projects outside of work.

I like to create machine templates that can be reused for automated provisioning using tools like Ansible and Terraform. This simulates setting up clusters of machines that need to be pre-configured and communicate with each other.

I also create Cloud-Init images in Qcow2 format to build templates with randomized SSH keys and uninitialized hostnames, much like how Azure, AWS, or Google Cloud sets up Platform-as-a-Service (PaaS) environments.

Through Proxmox, I’ve gone through installation and configuration procedures for large, complex systems like Kubernetes. This allows me to gain valuable experience with these cumbersome tools before working on similar setups in production environments. 🖥️

In Summary 📝

If you’re serious about learning systems engineering and working with multiple machines, I highly recommend setting up a Proxmox machine. Even an older PC can suffice, although some limitations may apply—but that’s part of the research and learning process! 😄

Linux File Systems Overview 💾

Sun, 25 Aug 2024 00:00:00 +0000

1. ext4 💾 (Linux, BSD)

Intro

ext4 (Extended File System version 4) is the default file system for many Linux distributions.

Links

https://en.wikipedia.org/wiki/Ext4 https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/6/html/storage_administration_guide/ch-ext4#ch-ext4

Technical Features: 🔍

Journaling
Extent-Based Allocation
Delayed Allocation
Persistent Pre-allocation
Multi-block Allocation
Online Resizing
64-bit File System Support
Directory Indexing with HTree
Defragmentation
Backward Compatibility with ext2/ext3
Barriers for Data Integrity
Large File Support (up to 16 TiB)
Metadata Checksumming (optional)
Quotas

Advantages 👍

Mature and Stable: ext4 is a well-tested and widely-used file system with a long history of stability.
Performance: It offers good performance for most workloads, especially for general-purpose usage.
Backward Compatibility: Supports ext3 and ext2 file systems, making it easy to upgrade.
Journaling: Provides a journaling feature that helps to prevent data corruption in case of a crash.
Wide Support: Supported by almost all Linux distributions and has a large community.

Downsides 👎

Limited Scalability: While adequate for most users, ext4 doesn’t scale as well as newer file systems for very large volumes and large numbers of files.
Lack of Advanced Features: ext4 lacks features like snapshotting and built-in data integrity checks (e.g., checksums).

Scale

Maximum File Size: 16 TiB
Maximum Volume Size: 1 EiB

Distro Usage

ext4 is the most widely used format spanning Linux and BSD.

2. XFS 💾 (Linux, BSD)

Intro

XFS is a high-performance file system designed for parallel I/O operations, often used in enterprise environments.

Links

https://en.wikipedia.org/wiki/XFS https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/storage_administration_guide/ch-xfs

Technical Features: 🔍

Extent-Based Allocation
Journaling (Metadata Journaling)
Delayed Allocation
Persistent Pre-allocation
Online Resizing (grow only)
Dynamic Inode Allocation
B+ Tree Directory Structure - (Quick Access B Tree)
Direct I/O Support
Data Striping for Performance
Advanced Metadata Management
Large File and Volume Support (up to 8 EiB)
Online Defragmentation
Quotas and Project Quotas
Realtime Subvolume for Real-Time I/O

Advantages 👍

High Performance: Optimized for large files and supports high-performance parallel I/O, making it ideal for environments with large data sets.
Scalability: Scales well with large volumes and large numbers of files, supporting file systems up to 500 TB.
Journaling: Uses journaling to help prevent data corruption.
Online Resizing: Supports online resizing of file systems (only grow).

Downsides 👎

Complexity: XFS is more complex to manage compared to ext4.
Limited Snapshot Support: Has limited support for snapshots compared to Btrfs and OpenZFS.
Potential Data Loss on Power Failure: In certain configurations, XFS may be more susceptible to data loss in the event of a sudden power loss.

Technical Details 🔍

Maximum File Size: 8 EiB
Maximum Volume Size: 8 EiB

Distro Usage

XFS has been in the Linux Kernel since 2001 It is the default file system for RHEL

3. Btrfs 💾 (Linux)

Intro

Btrfs (B-tree File System) is a modern, copy-on-write file system designed for Linux that offers advanced features like snapshots, RAID support, self-healing, and efficient storage management, making it suitable for scalable and reliable data storage.

Links

https://en.wikipedia.org/wiki/Btrfs https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/6/html/storage_administration_guide/ch-btrfs https://docs.kernel.org/filesystems/btrfs.html

Technical Features: 🔍

Journaling
Extent Base Allocation
Persistent Pre-allocation
Delayed Allocation
Multi-block Allocation
Stripe-aware Allocation
Resizeable with resize2fs
*B-tree Balancing Algorithm - Different from XFS (COW B Tee)
Copy-on-Write (COW)
Snapshots and Clones
Built-in RAID Support
Data and Metadata Checksumming
Self-Healing
Dynamic Subvolumes
Online Resizing
Compression (LZO. ZLIB. ZSTD)
Deduplication

Advantages 👍

Snapshot Support: Provides built-in support for snapshots, allowing for quick backups and rollbacks.
Data Integrity: Includes checksumming of data and metadata, which helps to ensure data integrity.
Self-Healing: With RAID support, Btrfs can automatically repair corrupted data.
Dynamic Storage: Allows for the dynamic addition and removal of storage devices.

Downsides 👎

Stability: Btrfs is considered less mature than ext4 or XFS, particularly for certain features like RAID 5/6.
Performance: May not perform as well as XFS or ext4 in certain high-performance scenarios, particularly with heavy random writes.
Complexity: The advanced features of Btrfs come with increased complexity.

Technical Details 🔍

Maximum File Size: 16 EiB
Maximum Volume Size: 16 EiB
Better on SSDs: Btrfs is well-suited for flash/solid-state storage because of TRIM support and CoW, which reduces write amplification.

Distro Usage

Btrfs has been in the mainline linux Kernel since 2008 it is the default file system for SUSE and Fedora

4. OpenZFS 💾 (Unix)

Intro

OpenZFS is an advanced file system and volume manager that originated from Sun Microsystems’ ZFS and is now maintained by the OpenZFS project.

Links

https://en.wikipedia.org/wiki/OpenZFS https://openzfs.org/wiki/Main_Page

Technical Features: 🔍

Copy-on-Write (COW)
Snapshots and Clones
Pooled Storage (ZFS Storage Pools)
Dynamic Striping
Built-in RAID Support (RAID-Z1, RAID-Z2, RAID-Z3)
Data and Metadata Checksumming
Self-Healing
Deduplication
Compression (LZ4, GZIP, ZLE, etc.)
Online Resizing
Dynamic Block Size
End-to-End Data Integrity
ZFS Datasets (File Systems and Volumes)
Adaptive Replacement Cache (ARC)
Transparent Data Encryption
ZFS Send/Receive for Backup and Replication

Advantages 👍

Data Integrity: Uses end-to-end checksums for all data, ensuring high data integrity.
Snapshots and Clones: Supports efficient, low-overhead snapshots and clones, useful for backups and development.
RAID-Z Support: Offers advanced RAID options (RAID-Z1, RAID-Z2, RAID-Z3), providing redundancy and fault tolerance.
Compression: Built-in compression can save space and improve performance in certain workloads.
Scalability: Designed to handle very large data sets and scales well with both size and number of files.

Downsides 👎

Resource Intensive: Can be resource-intensive, particularly in terms of memory usage.
Complexity: The advanced features and flexibility of OpenZFS come with a steep learning curve.
Portability: While available on many platforms, it is not as natively supported in Linux as ext4 or XFS.
Licensing: OpenZFS is licensed under CDDL, which is incompatible with the GPL.

Technical Details 🔍

Maximum File Size: 16 EiB
Maximum Volume Size: 256 ZiB (theoretical)

Distro Usage

Open ZFS is Not available in the mainline Linux Kernel. Rather, it is available through a 3rd party module. Works on Linux, BSD, and Mac

HAMMER2 💾 (DragonflyBSD)

Intro

Hammer2 is a modern, advanced file system designed for high-performance and scalable storage solutions, particularly in clustered environments. It features robust capabilities such as copy-on-write, data deduplication, and built-in snapshots, providing high data integrity, efficient storage management, and instant crash recovery.

Links

Wikipedia: HAMMER2 DragonFly BSD Hammer2

Technical Features: 🔍

Clustered File System Support
Snapshot and Cloning Support
Copy-on-Write (COW)
Data Deduplication
Data Compression (LZ4, ZLIB)
Data and Metadata Checksumming
Multi-Volume Support
Instant Crash Recovery
Fine-Grained Locking (for SMP scalability)
RAID Support (1, 1+0)
Thin Provisioning
Asynchronous Bulk-Freeing
Large Directory Support
Built-in Data Integrity and Self-Healing

Advantages 👍

High Performance: Optimized for high-performance and scalable storage solutions.
Data Integrity: Incorporates checksumming and self-healing features to maintain data integrity.
Efficient Storage Management: Offers advanced features like data deduplication and compression to manage storage efficiently.
Scalability: Designed to handle large volumes of data and support clustered environments.

Downsides 👎

Complexity: The advanced features and configuration options can introduce complexity.
Maturity: As a newer file system, it may have fewer tools and less mature support compared to more established file systems.
Limited Adoption: Less commonly used than other file systems, which may affect community support and documentation.

Technical Details 🔍

Maximum File Size: Not explicitly defined, but supports very large files.
Maximum Volume Size: Not explicitly defined, but designed for large-scale storage.

Distro Usage

DragonFly BSD: The primary platform where Hammer2 is used and supported.
Limited Availability: Not available in mainstream Linux distributions; primarily associated with DragonFly BSD.

Key Concepts / Glossary 🔑

Snapshots 📸

Snapshots are read-only copies of a file system at a specific point in time, allowing users to save the state of the file system for backup and recovery purposes. They are efficient and consume minimal space, as only the differences between the current state and the snapshot are stored.

Clones vs. Snapshots 📸🧬

Snapshots: Read-only copies of the file system at a specific time.
Clones: Writable copies of snapshots that can be modified independently.

RAID-Z Levels ⛓️

RAID-Z1: Single parity; can tolerate the loss of one disk.
RAID-Z2: Double parity; can tolerate the loss of two disks.
RAID-Z3: Triple parity; can tolerate the loss of three disks.

RAID 5 and RAID 6 ⛓️

RAID 5: Stripes data across disks with single parity; can tolerate the loss of one disk.
RAID 6: Stripes data across disks with double parity; can tolerate the loss of two disks.

Issues with RAID 5/6 in Btrfs

Btrfs’s implementation of RAID 5/6 is considered unstable due to issues like the write hole problem, making it less reliable for production use. Data integrity may be compromised, leading to potential data loss.

CDDL License 🪪

The Common Development and Distribution License (CDDL) is an open-source license created by Sun Microsystems. It is incompatible with the GPL, which can complicate integration with Linux.

Btrfs Self-Healing ❤️‍🩹

Self-Healing in Btrfs works by verifying data against checksums and repairing any detected corruption using redundant data stored on other disks in a RAID configuration.

Dynamic Storage 🧱

Dynamic Storage refers to the ability to manage multiple storage devices within a single file system, allowing for on-the-fly addition and removal of devices, with the file system automatically balancing data across them.

Online Resizing 🗺️

Online Resizing allows the resizing of a file system while it is mounted and in use. XFS supports growing the file system online, while Btrfs supports both growing and shrinking.

B-Trees ⚖️

A B-tree is a self-balancing tree data structure that maintains sorted data and allows efficient insertion, deletion, and search operations. B-trees are used in file systems like Btrfs to manage metadata and data blocks.

Extent Base Allocation 👠

is a method used by modern file systems to manage data storage efficiently. Instead of tracking individual fixed-size blocks, the file system groups contiguous blocks into larger units called extents.

Persistent Pre-allocation 🎟️

This technique reserves a specific amount of disk space for a file in advance, ensuring that the allocated space remains available, which helps in reducing fragmentation and guaranteeing storage for large files.

Delayed Allocation ⏱️

Delayed allocation defers the assignment of specific disk blocks to file data until the data is flushed to disk, optimizing the allocation process and reducing fragmentation by allowing the file system to make better decisions about where to place data.

Multi-block Allocation ⋔

Multi-block allocation allows a file system to allocate multiple contiguous blocks at once, rather than individually, improving performance and reducing fragmentation, especially for large files.

Stripe-aware Allocation 🧠

Stripe-aware allocation is used in RAID configurations to ensure that data is distributed evenly across all disks in the array, optimizing performance by aligning data placement with the underlying stripe size of the RAID setup.

Fine-Grained Locking (for SMP Scalability) 🚀

Fine-grained locking applies locks at a granular level, allowing multiple processors to concurrently access different parts of the file system, enhancing performance and scalability in multi-core environments.

RAID 1+0 🖇️

RAID support includes configurations such as RAID 1 for data mirroring and RAID 1+0 for combining mirroring with striping to provide both redundancy and improved performance.

Thin Provisioning 🔮

Thin provisioning allocates disk space on-demand rather than reserving it all upfront, optimizing storage utilization by only using the space actually required by data.

Asynchronous Bulk-Freeing 🗑️

Asynchronous bulk-freeing performs large-scale space reclamation in the background, allowing the file system to manage deletions efficiently without impacting overall performance.

Large Directory Support 🏢

Large directory support enables efficient management of directories with a vast number of entries, using optimized data structures to ensure fast performance for directory operations.

My WebDev Preferences 👍

Sat, 24 Aug 2024 00:00:00 +0000

Intro

This blog will likely be dominated by Linux talk as that is what I spend most of my time with. However, I do full-stack web development on occlusion. In fact, I have worked as a professional web Designer/Developer on and off, doing simple full stack projects in a range of languages. Recently, I have been stepping up my game with the intention of creating web applications with great GUIs, not only information pages. I think that with mature Javascript frameworks like NextJS or React and WASM (Web Assembly), we are going to see high performance desktop applications transition to being web-apps. This potential has me excited about learning more sophisticated tools.

GO

Golang is a dead simple systems focused language created by GOATED Unix contributors. When I started picking up Go a few years back, it was not really popular. Web development was dominated by NodeJS on the backend and there were no job posting for the skill. However, I tend to have good instincts and decided to eschew Node in favor of GO. I am very happy with this decision as it is clear, strongly typed, easy to set up and fast as hell. You can classify me as a GO enjoyer and back-ned deployer. GO now has a built-in HTTPs service and muxer, so it is my go-to for serving static and dynamic content.

TailWind

I really regret not checking out TailWind CSS earlier, it is so nice. I totally had the wrong idea and thought that it was just another annoying framework that over complicates and I was sorely wrong. It really helps in combination with GO templates to create beautiful responsive layouts. There is not much too a simple TW setup, just a simple call-out in the header. When things needs to be more expansive, one can use a config file to specify rules. I think I will be using a lot more TW in the future.

React

I am not a major fan of JS because I am a fan of simplicity, alas it is the language of the Web, so I must tango with wild syntax. Most web experiences require reactive interaction and content delivery. I am not locked into react as there are so many similar frameworks, but I am saying it is a must in ones stack.

VSCode

Having a customizable proper IDE is important for getting things done. As much as I like learning Vim, Helix, or even Sublime, I know that no-matter the level of configuration it wont even come close to a proper IDE that lints, detects bugs, has a large ecosystem of plug-ins and comfy features like workspaces and built in problem console. If you just want to make stuff, then reduce your cognitive overhead and utilize a user friendly IDE.