SRPMS Archives - ☩ Walking in Light with Christ - Faith, Computing, Diary ☩ Walking in Light with Christ

Posts Tagged ‘SRPMS’

Adding proxy to yum repository on Redhat / Fedora / CentOS and other RPM based Linux distributions, Listing and enabling new RPM repositories

Tuesday, September 7th, 2021

yum-add-proxy-host-for-redhat-linux-centos-list-rpm-repositories-enable-disable-repositories

Sometimes if you work in a company that is following PCI standards with very tight security you might need to use a custom company prepared RPM repositories that are accessible only via a specific custom maintained repositories or alternatively you might need the proxy node to access an external internet repository from the DMZ-ed firewalled zone where the servers lays .
Hence to still be able to maintain the RPM based servers up2date to the latest security patches and install software with yumone very useful feature of yum package manager is to use a proxy host through which you will reach your Redhat Package Manager files files.

1. The http_proxy and https_proxy shell variables

To set a proxy host you need to define there the IP / Hostname or the Fully Qualified Domain Name (FQDN).

By default "http_proxy and https_proxy are empty. As you can guess https_proxy is used if you have a Secure Socket Layer (SSL) certificate for encrypting the communication channel (e.g. you have https:// URL).

[root@rhel: ~]# echo $http_proxy
[root@rhel: ~]#

2. Setting passwordless or password protected proxy host via http_proxy, https_proxy variables

There is a one time very straight forward to configure proxying of traffic via a specific remote configured server with server bourne again shell (BASH)'s understood variables:

a.) Set password free open proxy to shell environment.

[root@centos: ~]# export https_proxy="https://remote-proxy-server:8080"

Now use yum as usual to update the available installabe package list or simply upgrade to the latest packages with lets say:

[root@rhel: ~]# yum check-update && yum update

b.) Configuring password protected proxy for yum

If your proxy is password protected for even tigher security you can provide the password on the command line as well.

[root@centos: ~]# export http_proxy="http://username:pAssW0rd@server:port/"

Note that if you have some special characters you will have to pass the string inside single quotes or escape them to make sure the password will properly handled to server, before trying out the proxy with yum, echo the variable.

[root@centos: ~]# export http_proxy='http://username:p@s#w:E@192.168.0.1:3128/'
[root@centos: ~]# echo $http_proxy
http://username:p@s#w:E@server:port/

Then do whatever with yum:

[root@centos: ~]# yum check-update && yum search sharutils

If something is wrong and proxy is not properly connected try to reach for the repository manually with curl or wget

[root@centos: ~]# curl -ilk http://download.fedoraproject.org/pub/epel/7/SRPMS/ /epel/7/SRPMS/
HTTP/1.1 302 Found
Date: Tue, 07 Sep 2021 16:49:59 GMT
Server: Apache
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin
Location: http://mirror.telepoint.bg/epel/7/SRPMS/
Content-Type: text/plain
Content-Length: 0
AppTime: D=2264
X-Fedora-ProxyServer: proxy01.iad2.fedoraproject.org
X-Fedora-RequestID: YTeYOE3mQPHH_rxD0sdlGAAAA80
X-Cache: MISS from pcfreak
X-Cache-Lookup: MISS from pcfreak:3128
Via: 1.1 pcfreak (squid/4.6)
Connection: keep-alive

Or if you need, you can test the user, password protected proxy with wget as so:

[root@centos: ~]# wget –proxy-user=USERNAME –proxy-password=PASSWORD http://your-proxy-domain.com/optional-rpms/

If you have lynx installed on the machine you can do the remote proxy successful authentication check with it with less typing:

[root@centos: ~]# lynx -pauth=USER:PASSWORD http://proxy-domain.com/optional-rpm/

3. Making yum proxy connection permanent via /etc/yum.conf

Perhaps the easiest and quickest way to add the http_proxy / https_proxy configured is to store it to automatically load on each server ssh login in your admin user (root) in /root/.bashrc or /root/.bash_profile or in the global /etc/profile or /etc/profile.d/custom.sh etc.

However if you don't want to have hacks and have more cleanness on the systems, the recommended "Redhat way" so to say is to store the configuration inside /etc/yum.conf

To do it via /etc/yum.conf you have to have some records there like:

# The proxy server – proxy server:port number
proxy=http://mycache.mydomain.com:3128
# The account details for yum connections
proxy_username=yum-user
proxy_password=qwerty-secret-pass

4. Listing RPM repositories and their state

As I had to install sharutils RPM package to the server which contains the file /bin/uuencode (that is provided on CentOS 7.9 Linux from Repo: base/7/x86_64 I had to check whether the repository was installed on the server.

To get a list of all yum repositories avaiable

[root@centos:/etc/yum.repos.d]# yum repolist all
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: centos.telecoms.bg
* epel: mirrors.netix.net
* extras: centos.telecoms.bg
* remi: mirrors.netix.net
* remi-php74: mirrors.netix.net
* remi-safe: mirrors.netix.net
* updates: centos.telecoms.bg
repo id repo name status
base/7/x86_64 CentOS-7 – Base enabled: 10,072
base-debuginfo/x86_64 CentOS-7 – Debuginfo disabled
base-source/7 CentOS-7 – Base Sources disabled
c7-media CentOS-7 – Media disabled
centos-kernel/7/x86_64 CentOS LTS Kernels for x86_64 disabled
centos-kernel-experimental/7/x86_64 CentOS Experimental Kernels for x86_64 disabled
centosplus/7/x86_64 CentOS-7 – Plus disabled
centosplus-source/7 CentOS-7 – Plus Sources disabled
cr/7/x86_64 CentOS-7 – cr disabled
epel/x86_64 Extra Packages for Enterprise Linux 7 – x86_64 enabled: 13,667
epel-debuginfo/x86_64 Extra Packages for Enterprise Linux 7 – x86_64 – Debug disabled
epel-source/x86_64 Extra Packages for Enterprise Linux 7 – x86_64 – Source disabled
epel-testing/x86_64 Extra Packages for Enterprise Linux 7 – Testing – x86_64 disabled
epel-testing-debuginfo/x86_64 Extra Packages for Enterprise Linux 7 – Testing – x86_64 – Debug disabled
epel-testing-source/x86_64 Extra Packages for Enterprise Linux 7 – Testing – x86_64 – Source disabled
extras/7/x86_64 CentOS-7 – Extras enabled: 500
extras-source/7 CentOS-7 – Extras Sources disabled
fasttrack/7/x86_64 CentOS-7 – fasttrack disabled
remi Remi's RPM repository for Enterprise Linux 7 – x86_64 enabled: 7,229
remi-debuginfo/x86_64 Remi's RPM repository for Enterprise Linux 7 – x86_64 – debuginfo disabled
remi-glpi91 Remi's GLPI 9.1 RPM repository for Enterprise Linux 7 – x86_64 disabled
remi-glpi92 Remi's GLPI 9.2 RPM repository for Enterprise Linux 7 – x86_64 disabled
remi-glpi93 Remi's GLPI 9.3 RPM repository for Enterprise Linux 7 – x86_64 disabled
remi-glpi94 Remi's GLPI 9.4 RPM repository for Enterprise Linux 7 – x86_64 disabled
remi-modular Remi's Modular repository for Enterprise Linux 7 – x86_64 disabled
remi-modular-test Remi's Modular testing repository for Enterprise Linux 7 – x86_64 disabled
remi-php54 Remi's PHP 5.4 RPM repository for Enterprise Linux 7 – x86_64 disabled
remi-php55 Remi's PHP 5.5 RPM repository for Enterprise Linux 7 – x86_64 disabled
remi-php55-debuginfo/x86_64 Remi's PHP 5.5 RPM repository for Enterprise Linux 7 – x86_64 – debuginfo disabled
!remi-php56 Remi's PHP 5.6 RPM repository for Enterprise Linux 7 – x86_64 disabled
remi-php56-debuginfo/x86_64 Remi's PHP 5.6 RPM repository for Enterprise Linux 7 – x86_64 – debuginfo disabled
remi-php70 Remi's PHP 7.0 RPM repository for Enterprise Linux 7 – x86_64 disabled
remi-php70-debuginfo/x86_64 Remi's PHP 7.0 RPM repository for Enterprise Linux 7 – x86_64 – debuginfo disabled
remi-php70-test Remi's PHP 7.0 test RPM repository for Enterprise Linux 7 – x86_64 disabled
remi-php70-test-debuginfo/x86_64 Remi's PHP 7.0 test RPM repository for Enterprise Linux 7 – x86_64 – debuginfo disabled
remi-php71 Remi's PHP 7.1 RPM repository for Enterprise Linux 7 – x86_64 disabled
remi-php71-debuginfo/x86_64 Remi's PHP 7.1 RPM repository for Enterprise Linux 7 – x86_64 – debuginfo disabled
remi-php71-test Remi's PHP 7.1 test RPM repository for Enterprise Linux 7 – x86_64 disabled
remi-php71-test-debuginfo/x86_64 Remi's PHP 7.1 test RPM repository for Enterprise Linux 7 – x86_64 – debuginfo disabled
!remi-php72 Remi's PHP 7.2 RPM repository for Enterprise Linux 7 – x86_64 disabled
remi-php72-debuginfo/x86_64 Remi's PHP 7.2 RPM repository for Enterprise Linux 7 – x86_64 – debuginfo disabled
remi-php72-test Remi's PHP 7.2 test RPM repository for Enterprise Linux 7 – x86_64 disabled
remi-php72-test-debuginfo/x86_64 Remi's PHP 7.2 test RPM repository for Enterprise Linux 7 – x86_64 – debuginfo disabled
remi-php73 Remi's PHP 7.3 RPM repository for Enterprise Linux 7 – x86_64 disabled
remi-php73-debuginfo/x86_64 Remi's PHP 7.3 RPM repository for Enterprise Linux 7 – x86_64 – debuginfo disabled
remi-php73-test Remi's PHP 7.3 test RPM repository for Enterprise Linux 7 – x86_64 disabled
remi-php73-test-debuginfo/x86_64 Remi's PHP 7.3 test RPM repository for Enterprise Linux 7 – x86_64 – debuginfo disabled
remi-php74 Remi's PHP 7.4 RPM repository for Enterprise Linux 7 – x86_64 enabled: 423
remi-php74-debuginfo/x86_64 Remi's PHP 7.4 RPM repository for Enterprise Linux 7 – x86_64 – debuginfo disabled
remi-php74-test Remi's PHP 7.4 test RPM repository for Enterprise Linux 7 – x86_64 disabled
remi-php74-test-debuginfo/x86_64 Remi's PHP 7.4 test RPM repository for Enterprise Linux 7 – x86_64 – debuginfo disabled
remi-php80 Remi's PHP 8.0 RPM repository for Enterprise Linux 7 – x86_64 disabled
remi-php80-debuginfo/x86_64 Remi's PHP 8.0 RPM repository for Enterprise Linux 7 – x86_64 – debuginfo disabled
remi-php80-test Remi's PHP 8.0 test RPM repository for Enterprise Linux 7 – x86_64 disabled
remi-php80-test-debuginfo/x86_64 Remi's PHP 8.0 test RPM repository for Enterprise Linux 7 – x86_64 – debuginfo disabled
remi-safe Safe Remi's RPM repository for Enterprise Linux 7 – x86_64 enabled: 4,549
remi-safe-debuginfo/x86_64 Remi's RPM repository for Enterprise Linux 7 – x86_64 – debuginfo disabled
remi-test Remi's test RPM repository for Enterprise Linux 7 – x86_64 disabled
remi-test-debuginfo/x86_64 Remi's test RPM repository for Enterprise Linux 7 – x86_64 – debuginfo disabled
updates/7/x86_64 CentOS-7 – Updates enabled: 2,741
updates-source/7 CentOS-7 – Updates Sources disabled
zabbix/x86_64 Zabbix Official Repository – x86_64 enabled: 178
zabbix-debuginfo/x86_64 Zabbix Official Repository debuginfo – x86_64 disabled
zabbix-frontend/x86_64 Zabbix Official Repository frontend – x86_64 disabled
zabbix-non-supported/x86_64 Zabbix Official Repository non-supported – x86_64 enabled: 5
repolist: 39,364

[root@centos:/etc/yum.repos.d]# yum repolist all|grep -i 'base/7/x86_64'
base/7/x86_64 CentOS-7 – Base enabled: 10,072

As you can see in CentOS 7 sharutils is enabled from default repositories, however this is not the case on Redhat 7.9, hence to install sharutils there you can one time enable RPM repository to install sharutils

[root@centos:/etc/yum.repos.d]# yum –enablerepo=rhel-7-server-optional-rpms install sharutils

To install zabbix-agent on the same Redhat server, without caring that I need precisely know the RPM repository that is providing zabbix agent that in that was (Repo: 3party/7Server/x86_64) I had to:

[root@centos:/etc/yum.repos.d]# yum –enablerepo \* install zabbix-agent zabbix-sender

Permanently enabling repositories of course is possible via editting or creating fresh new file configuration manually on CentOS / Fedora under directory /etc/yum.repos.d/.
On Redhat Enterprise Linux servers it is easier to use the subscription-manager command instead, like this:

[root@rhel:/root]# subscription-manager repos –disable=epel/7Server/x86_64

[root@rhel:/root]# subscription-manager repos –enable=rhel-6-server-optional-rpms

Tags: admin user, configured, enable, Enterprise Linux, external internet, linux?, Listing, MISS, passwordless, proxy, repository, rpm, SAMEORIGIN, servers, SRPMS, variables, yum
Posted in Linux, Linux Package Management | No Comments »

How to check who is flooding your Apache, NGinx Webserver – Real time Monitor statistics about IPs doing most URL requests and Stopping DoS attacks with Fail2Ban

Wednesday, August 20th, 2014

check-who-is-flooding-your-apache-nginx-webserver-real-time-monitoring-ips-doing-most-url-requests-to-webserver-and-protecting-your-webserver-with-fail2ban

If you're Linux ystem administrator in Webhosting company providing WordPress / Joomla / Drupal web-sites hosting and your UNIX servers suffer from periodic denial of service attacks, because some of the site customers business is a target of competitor company who is trying to ruin your client business sites through DoS or DDOS attacks, then the best thing you can do is to identify who and how is the Linux server being hammered. If you find out DoS is not on a network level but Apache gets crashing because of memory leaks and connections to Apache are so much that the CPU is being stoned, the best thing to do is to check which IP addresses are causing the excessive GET / POST / HEAD requests in logged.

There is the Apachetop tool that can give you the most accessed webserver URLs in a refreshed screen like UNIX top command, however Apachetop does not show which IP does most URL hits on Apache / Nginx webserver.

1. Get basic information on which IPs accesses Apache / Nginx the most using shell cmds

Before examining the Webserver logs it is useful to get a general picture on who is flooding you on a TCP / IP network level, with netstat like so:

# here is howto check clients count connected to your server
netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

If you get an extensive number of connected various IPs / hosts (like 10000 or something huge as a number), depending on the type of hardware the server is running and the previous scaling planned for the system you can determine whether the count as huge as this can be handled normally by server, if like in most cases the server is planned to serve a couple of hundreds or thousands of clients and you get over 10000 connections hanging, then your server is under attack or if its Internet server suddenly your website become famous like someone posted an article on some major website and you suddenly received a tons of hits.

There is a way using standard shell tools, to get some basic information on which IP accesses the webserver the most with:

tail -n 500 /var/log/apache2/access.log | cut -d' ' -f1 | sort | uniq -c | sort -gr

Or if you want to keep it refreshing periodically every few seconds run it through watch command:

watch "tail -n 500 /var/log/apache2/access.log | cut -d' ' -f1 | sort | uniq -c | sort -gr"

monioring-access-hits-to-webserver-by-ip-show-most-visiting-apache-nginx-ip-with-shell-tools-tail-cut-uniq-sort-tools-refreshed-with-watch-cmd

Another useful combination of shell commands is to Monitor POST / GET / HEAD requests number in access.log :

awk '{print $6}' access.log | sort | uniq -c | sort -n

     1 "alihack<%eval
      1 "CONNECT
      1 "fhxeaxb0xeex97x0fxe2-x19Fx87xd1xa0x9axf5x^xd0x125x0fx88x19"x84xc1xb3^v2xe9xpx98`X'dxcd.7ix8fx8fxd6_xcdx834x0c"
      1 "x16x03x01"
      1 "xe2
      2 "mgmanager&file=imgmanager&version=1576&cid=20
      6 "4–"
      7 "PUT
     22 "–"
     22 "OPTIONS
     38 "PROPFIND
   1476 "HEAD
   1539 "-"
65113 "POST
537122 "GET

However using shell commands combination is plenty of typing and hard to remember, plus above tools does not show you, approximately how frequenty IP hits the webserver

2. Real-time monitoring IP addresses with highest URL reqests with logtop

Real-time monitoring on IP addresses with highest URL requests is possible with no need of "console ninja skills" through – logtop.

2.1 Install logtop on Debian / Ubuntu and deb derivatives Linux

a) Installing Logtop the debian way

LogTop is easily installable on Debian and Ubuntu in newer releases of Debian – Debian 7.0 and Ubuntu 13/14 Linux it is part of default package repositories and can be straightly apt-get-ed with:

apt-get install –yes logtop

b) Installing Logtop from source code (install on older deb based Linuxes)

On older Debian – Debian 6 and Ubuntu 7-12 servers to install logtop compile from source code – read the README installation instructions or if lazy copy / paste below:

cd /usr/local/src
wget https://github.com/JulienPalard/logtop/tarball/master
mv master JulienPalard-logtop.tar.gz
tar -zxf JulienPalard-logtop.tar.gz
cd JulienPalard-logtop-*/
aptitude install libncurses5-dev uthash-dev
…
aptitude install python-dev swig
…

make python-module
…

python setup.py install
…

make
…

make install

mkdir -p /usr/bin/
cp logtop /usr/bin/

2.2 Install Logtop on CentOS 6.5 / 7.0 / Fedora / RHEL and rest of RPM based Linux-es

b) Install logtop on CentOS 6.5 and CentOS 7 Linux

– For CentOS 6.5 you need to rpm install epel-release-6-8.noarch.rpm

wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
rpm -ivh epel-release-6-8.noarch.rpm
links http://dl.fedoraproject.org/pub/epel/6/SRPMS/uthash-1.9.9-6.el6.src.rpm
rpmbuild –rebuild uthash-1.9.9-6.el6.src.rpm
cd /root/rpmbuild/RPMS/noarch
rpm -ivh uthash-devel-1.9.9-6.el6.noarch.rpm

– For CentOS 7 you need to rpm install epel-release-7-0.2.noarch.rpm

links http://download.fedoraproject.org/pub/epel/beta/7/x86_64/repoview/epel-release.html

Click on and download epel-release-7-0.2.noarch.rpm

rpm -ivh epel-release-7-0.2.noarch
rpm –import /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
yum -y install git ncurses-devel uthash-devel
git clone https://github.com/JulienPalard/logtop.git
cd logtop
make
make install

2.3 Some Logtop use examples and short explanation

logtop shows 4 columns as follows – Line number, Count, Frequency, and Actual line

The quickest way to visualize which IP is stoning your Apache / Nginx webserver on Debian?

tail -f access.log | awk {'print $1; fflush();'} | logtop

logtop-check-which-ip-is-making-most-requests-to-your-apache-nginx-webserver-linux-screenshot

On CentOS / RHEL

tail -f /var/log/httpd/access_log | awk {'print $1; fflush();'} | logtop

Using LogTop even Squid Proxy caching server access.log can be monitored.
To get squid Top users by IP listed:

tail -f /var/log/squid/access.log | awk {'print $1; fflush();'} | logtop

logtop-visualizing-top-users-using-squid-proxy-cache

Or you might visualize in real-time squid cache top requested URLs

tail -f /var/log/squid/access.log | awk {'print $7; fflush();'} | logtop

visualizing-top-requested-urls-in-squid-proxy-cache-howto-screenshot

3. Automatically Filter IP addresses causing Apache / Nginx Webservices Denial of Service with fail2ban

Once you identify the problem if the sites hosted on server are target of Distributed DoS, probably your best thing to do is to use fail2ban to automatically filter (ban) IP addresses doing excessive queries to system services. Assuming that you have already installed fail2ban as explained in above link (On Debian / Ubuntu Linux) with:

apt-get install –yes fail2ban

To make fail2ban start filtering DoS attack IP addresses, you will have to set the following configurations:

vim /etc/fail2ban/jail.conf

Paste in file:

[http-get-dos]

enabled = true
port = http,https
filter = http-get-dos
logpath = /var/log/apache2/WEB_SERVER-access.log
# maxretry is how many GETs we can have in the findtime period before getting narky
maxretry = 300
# findtime is the time period in seconds in which we're counting "retries" (300 seconds = 5 mins)
findtime = 300
# bantime is how long we should drop incoming GET requests for a given IP for, in this case it's 5 minutes
bantime = 300
action = iptables[name=HTTP, port=http, protocol=tcp]

Before you paste make sure you put the proper logpath = location of webserver (default one is /var/log/apache2/access.log), if you're using multiple logs for each and every of hosted websites, you will probably want to write a script to automatically loop through all logs directory get log file names and automatically add auto-modified version of above [http-get-dos] configuration. Also configure maxtretry per IP, findtime and bantime, in above example values are a bit low and for heavy loaded websites which has to serve thousands of simultaneous connections originating from office networks using Network address translation (NAT), this might be low and tuned to prevent situations, where even the customer of yours can't access there websites 🙂

To finalize fail2ban configuration, you have to create fail2ban filter file:

vim /etc/fail2ban/filters.d/http-get-dos.conf

Paste:

# Fail2Ban configuration file
#
# Author: http://www.go2linux.org
#
[Definition]

# Option: failregex
# Note: This regex will match any GET entry in your logs, so basically all valid and not valid entries are a match.
# You should set up in the jail.conf file, the maxretry and findtime carefully in order to avoid false positives.

failregex = ^<HOST> -.*"(GET|POST).*

# Option: ignoreregex
# Notes.: regex to ignore. If this regex matches, the line is ignored.
# Values: TEXT
#
ignoreregex =

To make fail2ban load new created configs restart it:

/etc/init.d/fail2ban restart

If you want to test whether it is working you can use Apache webserver Benchmark tools such as ab or siege.
The quickest way to test, whether excessive IP requests get filtered – and make your IP banned temporary:

ab -n 1000 -c 20 http://your-web-site-dot-com/

This will make 1000 page loads in 20 concurrent connections and will add your IP to temporary be banned for (300 seconds) = 5 minutes. The ban will be logged in /var/log/fail2ban.log, there you will get smth like:

2014-08-20 10:40:11,943 fail2ban.actions: WARNING [http-get-dos] Ban 192.168.100.5
2013-08-20 10:44:12,341 fail2ban.actions: WARNING [http-get-dos] Unban 192.168.100.5

Tags: check, configure, Debian Ubuntu, fail2ban stopping apache dos attacks, How to, how to view get post head requests, install logtop centos rhel fedora, install logtop debian ubuntu, install logtop from source, Installing Logtop, logtop, make, monitor apache log real-time, monitoring top ip requests apache log, nginx monitor top url requests, noarch, script, squid proxy log monitoring logtop, SRPMS, text, url, var
Posted in Computer Security, Everyday Life, Monitoring, System Administration, Various, Web and CMS | 2 Comments »

☩ Walking in Light with Christ – Faith, Computing, Diary

Posts Tagged ‘SRPMS’

How to check who is flooding your Apache, NGinx Webserver – Real time Monitor statistics about IPs doing most URL requests and Stopping DoS attacks with Fail2Ban

Daily Bible quote

GET ARTICLE UPDATES

Useful blog? Help it:

Links to Other Places

Recent Posts

Ads

Categories

About Myself

Recent Comments

Top Post Views

blogtopsites