Magento 2.4 CSP Module Issue

Recently we experienced a production project that We have one client that reported their production Magento site keeps having problems with random downtime that lasts for several weeks. Our client runs Magento 2.4.0 open source edition. 

In order to fix the issue, initially we checked everything and nothing seems strange with the server, nor with the Magento itself. There was only one small error complaining about PHP memory limit so we decided to increase PHP memory limit (slowly but we did it several times) up to 2G. This 2G is considerably too big for a Magento application, given how relatively small the site is.

At that point we already know that it is the Magento_Csp module that has been complaining about the PHP memory limit but we didn’t know that the fix needs something more than just increasing the PHP memory limit. Plus, the timestamps of the error log isn’t always 100% matched with the actual downtimes.

Our attempt to fix the problem isn’t over yet. After a few days the site keeps crashing randomly with similar errors. Same as previous occurrence, when downtime happens, Zabbix records that CPU and memory consumption goes high thus triggering out-of-memory (OOM) error and Linux system will start killing processes randomly. In this case, PHP-FPM is always in the kill list.

The strangest thing that we realized during the most recent downtime is that the disk usage was growing very quickly. In our case, about 15G of disk space was eaten in less than 2 hours, causing high IO and server collapsed.

We knew that the disk was being eaten by something, but we didn’t know what’s doing it. This then leads to a theory: something in Magento is writing something into the disk. This theory was supported by the PHP-FPM error log related to the Magento_Csp module which mentioned something about cache. Also we couldn’t think about anything else but the Magento cache.

A quick research gave us this: https://github.com/magento/magento2/issues/29964 and it turned out to be the exact same thing that we experienced. However, most people only complain about the OOM issue but there’s one guy who mentioned that the disk also got eaten.

If you are too lazy to read that whole github issue for the solution, here’s the summary of the solutions offered by the people there. Best if you can upgrade to the newer Magento version, at least 2.4.2. I know upgrading may not be easy for most merchants because you will need to make sure that all integrations are still functioning, so…. Fortunately if you have the Magento_Csp module sitting there from the default installation (and you’ve never done anything to it), you should be safe to disable the Magento_Csp completely

bin/magento module:disable Magento_Csp

bin/magento cache:flush

After disabling the module, we can now have a good sleep at night. Server is super quiet:

This solution works as a quick fix for that CSP problem, but probably scheduling a time for a Magento upgrade isn’t a bad idea either.

Restricting website access using Cloudflare Firewall

Firewall is one of the main features of Cloudflare. This network-level protection is especially useful when your website has a lot of visitors and has higher potential for being abused or attacked, either by bots or real human-being. Anyways, it doesn’t hurt to use cloudflare firewall by any means.

This firewall feature is enabled by default and will protect your domain and subdomains which are using Cloudflare proxy (not DNS only). This firewall function comes with all Cloudflare plans, including the free plan. However, if you upgrade to a paid plan, you can use more firewall features such as Web Application Firewall (WAF) and bots protection. In this article we will talk about some very basic Cloudflare Firewall under Firewall Rules.

For this tutorial, we will try to set up a firewall that will restrict all subdomains in our account for the whole internet except for those who are given access to. This is useful for example, you have many internal sites under the subdomain of your main domain. You want the main domain still publicly accessible, but only allow all internal subdomains only for your employees. In this scenario, we will use:

  1. IP address based restriction
  2. Cookie-based restriction

Most companies have static IP addresses for their office network and VPN, so we can simply whitelist the IP address and allow all employees to connect to our internal websites. However, this won’t work if you want to share access to external clients. Another problem might also occur when your employees are working from home and can’t use VPN, and that will make them get blocked. This is why we recommend combining cookie-based rules for your firewall. Let’s get started.

Prerequisites:

  1. Domain and / or subdomains managed with cloudflare with proxy enabled
  2. Your office network / VPN static public IP

Steps:

  1. Go to cloudflare firewall – Firewall Rules, then click on Create firewall rule
  2. Configure the following. Rule name is something that you can easily memorize. When incoming requests match… is the restriction that you need to define. Then… is what you want Cloudflare does when it matches your rule.
    In this example, we prepare the following firewall expression to make you get started easily. Simply replace yourdomain.com with your domain and 111.111.111..111 with your office / VPN IP address. See step 4 for the rule and how to insert it into cloudflare.
  3. Click on Edit expression, then insert this rule (remember to replace the domain and IP address).


(not ip.src in {111.111.111.111} and not http.host in {“yourdomain.com”} and not http.cookie contains “cloudflare-firewall=lkj23894knlf943rowIUsd9frKskjswIPVwMs2aa”)


This expression consists of three parts:

  1. IP part (111.111.111.111)
  2. Domain (yourdomain.com)
  3. Cookie key definition (cloudflare-firewall=lkj23894knlf943rowIUsd9frKskjswIPVwMs2aa). You will need to use this cookie to visit the website if you are not using a whitelisted IP address.
  1. Select what action to take. We recommend using Challenge (captcha) to test if you are in a production environment. Just to be safe.
  2. Click save, and your rule should be deployed immediately. If not, click on the toggle to enable it.

Now it is time to test in action. Browse to your subdomains with and without your whitelisted IP address. Browse to it with and without your defined cookies. If you configure it correctly and you are not using whitelisted IP nor the cookies, then you will get blocked like me, good luck!

P.S: To set cookie when browsing, you can simply execute the below command on your browser console:


document.cookie = “cloudflare-firewall=lkj23894knlf943rowIUsd9frKskjswIPVwMs2aa”;

Or, you can also use browser extensions for setting the cookie like this one, but there are many other alternatives.

Capturing real visitor IP behind Cloudflare proxy + Nginx

By default, nginx will log visitor IP addresses in its access log. This is mostly useful for debugging purposes when you have some problem with your server. Also it’s often useful for mitigating bot attacks, hacking attempts, or other security-related issues.

Cloudflare is another DNS-level tool that can be used to speed up your website performance, increase security of your website, and many others. This time we will focus on cloudflare’s basic function called Cloudflare proxy. 

In the picture above you can see whether the DNS record is proxied through cloudflare. When you enable proxy, all visitor requests won’t directly hit your webserver and the actual server IP will be hidden. In contrast to traditional DNS (or, in Cloudflare they call it “DNS only”), we can usually know what is the actual server IP which serves a specific domain. This cloudflare proxy is good for both security and speed. Good for security because the actual server IP will remain hidden for the whole world, and good for speed because Cloudflare will acts as a “proxy” that will serve HTTP requests from cloudflare cache (if any). 

However, this proxy thing can sometimes cause a problem. One of them is this that we’re discussing in this article. If we enable proxy, then it’s gonna be the cloudflare proxy servers that will actually hit our server, not the real customer. That way, nginx will record cloudflare’s IP addresses instead of the visitor’s. To address this problem, we can apply some simple nginx configuration so that it will record the real customer IP. We assume that you already have a website running on nginx webserver and you have registered your domain on cloudflare.

Prerequisites:

  1. Nginx with http_realip_module module (should come by default in standard nginx installation)

Steps:

  1. Edit your site’s nginx config:
    vim /etc/nginx/sites-enabled/your-nginx-config-here
  2. Add the following lines. This list of IPs might change. Please check here for an up-to-date list.

set_real_ip_from 103.21.244.0/22;

set_real_ip_from 103.22.200.0/22;

set_real_ip_from 103.31.4.0/22;

set_real_ip_from 104.16.0.0/13;

set_real_ip_from 104.24.0.0/14;

set_real_ip_from 108.162.192.0/18;

set_real_ip_from 131.0.72.0/22;

set_real_ip_from 141.101.64.0/18;

set_real_ip_from 162.158.0.0/15;

set_real_ip_from 172.64.0.0/13;

set_real_ip_from 173.245.48.0/20;

set_real_ip_from 188.114.96.0/20;

set_real_ip_from 190.93.240.0/20;

set_real_ip_from 197.234.240.0/22;

set_real_ip_from 198.41.128.0/17;

set_real_ip_from 2400:cb00::/32;

set_real_ip_from 2606:4700::/32;

set_real_ip_from 2803:f800::/32;

set_real_ip_from 2405:b500::/32;

set_real_ip_from 2405:8100::/32;

set_real_ip_from 2c0f:f248::/32;

set_real_ip_from 2a06:98c0::/29;

#use any of the following two

real_ip_header CF-Connecting-IP;

#real_ip_header X-Forwarded-For;

  1. Save & quit
  2. Reload nginx
    sudo systemctl reload nginx.service

That’s it! Now you should be able to see the actual visitor IP under your nginx’s access log. Let us know if it works!

Magento – Allow Specific IP Address on Maintenance Mode

Usually when redeploying new code or introducing critical changes to the server environment. It is common to restrict access to end-users for avoiding errors caused by end-users using an invalid or incorrect environment state.

Magento allows developers to set maintenance mode, redirecting all users to the application 503 page. 

bin/magento maintenance:enable

Also, the developers have the advantage of whitelisting one or more IP addresses, allowing them to test any change in the server without being worried about any end-user accessing the server at the same the server is being updated. 

For example, the following command will enable maintenance mode for all end users except for 192.0.2.20 and 192.0.2.21:

$bin/magento maintenance:enable –ip=192.0.2.20 –ip=192.0.2.21

Enabled maintenance mode

Set exempt IP-addresses: 192.0.2.20, 192.0.2.21

The command above hopefully will be a helpful tool in order to reduce server downtime, the same for maintenance and development time.

Magento PhpStorm Plugin not Working

If you just installed the newest PhpStorm version or just upgraded from a previous version you might notice that the Magento PhpStorm plugin is not working; not showing any plugin-related menu nor allowing you to autogenerate code, reducing your job efficiency.

You are also sure that you followed the correct steps for installing the plugin but it still not working, even worse, the PHP -> Frameworks section is also missing from the IDE.

What happened?  

It turns out that since PhpStorm 2021 1.1, the PHP frameworks section is independently listed at the top on File->Settings, and for some reason, this detail is also missing from the official PhpStorm documentation:

That section will look exactly the same as previous PhpStorm versions, just click on Enable Magento Integration and select the Magento installation path.

After clicking OK the IDE will reindex and the menus will start displaying all the cool features from the plugin, speeding up your code development.

Happy coding!