Nick Little

Amazon Linux 2023 Defaults That Break a Fresh LAMP/LEMP Migration

· Nick Little

Migrating a LAMP/LEMP-style workload onto Amazon Linux 2023 for the first time surfaces a handful of default behaviors that didn’t exist, or worked differently, on Amazon Linux 2. None of these are bugs, they’re deliberate, reasonable defaults, but each one is invisible until it collides with an assumption carried over from an older base image. This is a reference for the three that cost the most debugging time on a recent migration, what they look like when they bite, and the fix for each.

It’s important to note that Amazon Linux 2 is end of life from June 30, 2026. Whilst you can still deploy after this date, you won’t get:

  • Any new security patches (OS or kernel)
  • Blocked by AWS managed services using tools like AWS Batch or Storage Gateway
  • No newly published AMI’s from AWS (e.g for ECS or EKS optimized)
  • Likely fails compliance validation e.g PCI-DSS and SOC 2

1. firewalld is enabled by default

What changed: AL2023 ships firewalld active out of the box, with a restrictive default zone (SSH allowed, most other things dropped). AL2 didn’t run a host-level firewall by default in the same way.

What it looks like when it bites: a security group that’s demonstrably correct, port 80 open, confirmed via describe-security-groups, and external connections still get Connection refused. curl localhost on the instance itself works fine, which makes this particularly confusing: everything about the network path looks right from every angle you’d normally check.

Why it’s easy to miss: iptables -L shows empty, permissive chains, because firewalld on AL2023 uses nftables as its backend, not the legacy iptables chains that command reads (and the one I’m most familiar with). The tool you’d reach for to check firewall rules doesn’t show you the rules that are actually blocking you.

The fix, if a security group is already doing the job you want at the network boundary:

systemctl stop firewalld
systemctl disable firewalld

Remember AWS Security Groups are already doing your stateful firewalling at the EC2 instance level by injection into your instances Elastic Network Interface (ENI).

If you want to keep firewalld and open the ports through it instead, that’s a legitimate alternative, the point isn’t “always disable it,” it’s “know it’s there before you spend an hour convinced your security group is wrong.”

2. IMDSv2 is required, not just available

What changed: AL2023 defaults new instances to requiring IMDSv2 (token-based instance metadata requests). Plain, unauthenticated calls to 169.254.169.254, a pattern in a lot of older boot scripts and tutorials, silently return 401 instead of the expected value.

What it looks like when it bites: a boot script that worked on AL2 fails in a way that doesn’t obviously point at IMDS at all. In my case, an empty region variable produced Invalid endpoint: https://secretsmanager..amazonaws.com (note the double dot), a downstream symptom of an upstream silent failure, several layers removed from the actual cause.

The fix: fetch a token first, then use it on every subsequent metadata call:

TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
REGION=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/placement/region)

Worth auditing any boot script inherited from an AL2-era template for the older, token-less pattern before it ships somewhere this bites for real.

3. php-fpm’s default pool user may not match your web server’s user

What changed: this one’s less an AL2023-specific behavior shift and more a packaging default worth checking explicitly rather than assuming, the php-fpm RPM’s default pool configuration (/etc/php-fpm.d/www.conf) may run as a user (commonly apache) that doesn’t match whichever user actually owns your application’s files, particularly if you’ve deliberately set ownership to match your web server (commonly nginx) rather than the PHP default.

What it looks like when it bites: the application serves pages fine, PHP can read files owned by a different user without issue, but any write operation fails. In a WordPress context specifically, that’s theme/plugin installs failing with “Unable to connect to the filesystem. Please confirm your credentials,” a message that (reasonably) points a user toward FTP credentials rather than a Linux user/group mismatch.

The fix: check the mismatch directly rather than guess:

grep -E "^user|^group" /etc/php-fpm.d/www.conf
ls -la /path/to/your/webroot | head -5

and align them:

sed -i 's/^user = .*/user = nginx/' /etc/php-fpm.d/www.conf
sed -i 's/^group = .*/group = nginx/' /etc/php-fpm.d/www.conf
systemctl restart php-fpm

The common thread

All three are “reads fine, invisible until you write or reach externally” categories of bug, none of them show up in a quick smoke test that just checks the homepage loads. Worth explicitly testing external connectivity, a write operation, and (if relevant) a boot script’s IMDS calls on any fresh AL2023 instance before calling a migration complete, rather than treating “the site loads” as sufficient verification.

Thanks all for now, thanks folks!