Self-notes: Git

Just a collection of git commands to have quick look when brain stops working.

1. Move back branch to earlier commit
git checkout [branch name]

git reset --hard [commit hash to revert branch to]
git push -f


2. Remove git tag from remote
delete tag from local.
git tag -d [tag name]

delete tag from remote
git push origin :refs/tags/[tag name]

3. Never collide tag name with branch name
4. Broken tag reference could break Jenkins pipeline, forcing us to delete the tag as a quick fix. (still looking for better solution through Jenkins)


Let me know if you have interesting short notes or answers :)

Mixing Powershell with Jenkins and Python, Lazy DevOps Way

This post is for the DevOps guys or the system guys or whoever to get the work done done with just fix it approach who just found out powershell is quicker while dealing with VMware automation.
The case could be; you already have your automation code written in python for your in house data center and later on your infrastructure migrated to VMware and now you have to modify or rewrite everything or you have jenkins master and now you have to run powershell script somehow or likewise.
VMware API documentation is not friendly enough in my experience, and if you are Python person then something like PyVMomi might work as well.

Found out Powershell is better with working with VMware. So, here is the way to use Powershell in Jenkins and inside your python code.



1. Jenkins run Powershell
Use Powershell plugin in Jenkins.
If you have Jenkins in Linux machine then PowerCLI for Linux can be installed. Its in development and you can get started with Alpha release. Till then you need to have Windows machine to run Powershell and install VMware libraries. In this case, you can just  connect Windows machine as slave and executed the Powershell script from master Jenkins. Now in this, if you are very new to Powershell world then questions like "how to assign variables in script? " and "how to use  Jenkins job parameters inside Powershell script? " are the interesting one. And below code just answers these questions.

Listing Snapshots of VMware Machine

Jenkins Build Step: Execute Powershell

Import-Module VMware.VimAutomation.Core

"Hello From Powershell @ Remote Windows Machine"
"Listing all the snapshots of vm: $env:vm_name"

Set-Variable -Name "username" -Value “myusername@username.com”
Set-Variable -Name "password" -Value “mypasswd@passwd.com”

#connect to vsphere at myserverdomain.com
Connect-VIServer -Server myserverdomain.com -Protocol https -User $username -Password $password

#getting list of all the snapshot of vm
Get-VM -Name $env:vm_name | get-snapshot | format-list VM,Name,PowerState,Description, Children,VMId,ParentSnapshot,IsCurrent

In above code, $env:vm_name is the way to get Jenkins Job Parameter in Powershell script where vm_name is the parameter name.

2. Use Python Fabric to execute powershell in remote windows.
Python fabric api is use to execute commands in the remote machines. Its one of the custom code way to do automation or effective configuration management. Setting up the host and credential we can ask Fabric to execute commands in the remote machines. Fabric uses SSH beneath so, we have to install SSH service in Windows. Now this method consist of 3 components; your powershell script located in remote windows machine, your python script and fabfile.py thats Fabric stuff you need to code as well :) Get into Fabric to know about fabfile.

Revert Snapshot with PowerCLI

my_revertsnapshot.ps1

Param(
  [string]$vm_name,
  [string]$snapshot_name
)

Import-Module VMware.VimAutomation.Core

"Hello From Powershell"
"Reverting snapshot of vm: $vm_name"
"Reverting the snapshot state named: $snapshot_name"

Set-Variable -Name "username" -Value “myusername@username.com”
Set-Variable -Name "password" -Value “mypasswd”

#connect to vsphere myvmwaredomain.com

Connect-VIServer -Server myvmwaredomain.com -Protocol https -User $username -Password $password

#revert snapshot of vm
set-vm -vm $vm_name -snapshot $snapshot_name -Confirm:$false

"Snapshot might have been reverted. Please verify"


-Confirm:$false is to just bypass the yes/no confirmation in powershell that worked for me.

The fabfile.py would look like this, you can write this out in more manageable way than this dirty way :)

fabfile.py

from fabric.api import env, run, task

@task
def revert_snapshot_vm():
   
    env.user = “user”
    env.password = “userpassword”
    env.hosts = [‘w.x.y.z’] # ip address of host

    print('executing powershell in remote windows')
    try:

        run("powershell -file c:\jenkins\my_revertsnapshot.ps1 vmmachine123 myfirstsnapshot”, shell=False, pty=False)

        print('snapshot might have been reverted')
    except Exception as e:
        print("error calling revert snapshot in ESXI machine: ", e)

Calling Fabric from Python Code


run_powershell.py

import fabfile
from fabric.api import env
from fabric.main import execute

## run the powershell command in remote windows

def call_powershell_snapshot():
    env.hosts = [‘w.x.y.z’]
    try:
        execute(getattr(fabfile, 'revert_snapshot_vm'))

    except Exception as ex:
        print("error running method ", ex)
        pass

call_powershell_snapshot()

Usage is: python run_powershell.py in linux terminal  :)

Enjoy scripting ;)
   

Compromised Server, Me and Few Layers Of Security

Well this is not a theory on the layer wise security. Everyone know the concept of layer wise security. Main gate- main door- rooms doors, this sums up what layer wise security is. It makes bad guys hard to reach your living room. Well, that not all. Few weeks ago I got an opportunity to look into the compromised server. I had to figure out the cause of incident and deliver incident report. Server was in the Amazon cloud and AWS support guys did their job by shutting down compromised server down, not exactly shutdown but they blocked every traffic out of the server. They detected DoS attack being conducted through that compromised server using UDP port 80 on an IP belonging to an enterprise. AWS support guys allowed SSH into the server and there I was checking everything I possibly could in AWS server. Looked into users, passwords, shadows, file permissions, history, netstat, nmap, apache access log, error log, auth log and what not. Must say "the blue team handbook" did come in good use that time.
I found nothing. Yes!! nothing and that kept me wondering for minutes, thinking what attack vectors I have missed and so on. I was not thinking well, because somewhere in my mind I already assumed an attack through Remote code execution, and after not seeing suspicious access log and error log in Apache,
I could not think more. Later after a sip of black coffee, I did malware scan on the server. Rootkit hunter was my first tool to use, then came the clamav antivirus, and Bingo!! there it was rootkit hiding inside /bin/netstat, /bin/ps.

And yes, i was using that netstat and ps command to find information. Felt stupid. But i was still unsure. And this time, host based IDS came into party which I had forgotten that it was already installed in the server. Little bit of log analysis and I found OSSEC HIDS did log and alert rootkit detection on the date when DoS attack was detected. I confirmed more with checking binary hashes and found it was different than the original.

Later I zipped those scripts,downloaded in my pc and did a scan with Avast. I got an alert and it was more detailed.

This incident made me realize the need layer wise security with different angle. Its not always about protecting but also about fighting back. I am not a malware analyst so I did not go through that part but what I'm trying to say is without IDS installed it would have been real difficult to figure out and keep analyze changes in the server.

Sending Email Alert From Apache ModSecurity

Apache Modsecurity can be configured to provide different passive response, an email alert is one of it. To get this done, we need to have mail service enabled in our host server. Generally an installation of Postfix or similar application will do the task.

Configuration steps

1. Create a emailing script and place it anywhere in the server. There in the following examples we have created a folder "my_test_rules" and placed my custom email script.

# cd /usr/share/modsecurity-crs/my_test_rules
#nano send_simple_email_alert.sh

  #!/bin/sh
  echo "Fake user tried to access the web application" | mail -s "server under attack" your_email@domain.com
  echo Done.

2. Write a custom rule that will execute the email script when triggered like sample rule below:

#nano test_request_headers_and_send_email.conf

  SecRule REQUEST_HEADERS:User-Agent "FAKE-USER" "deny,log,id:'1234123457',exec:/usr/share  /modsecurity-crs/my_test_rules/send_simple_email_alert.sh"

3. Add the Sym link of the custom rule in the activated_rules directory under /usr/share/modsecurity-crs

#cd /usr/share/modsecurity-crs/activated_rules
#ln -s ../my_test_rules/test_request_headers_and_send_email.conf

4. Reload Apache

#service apache2 restart

We can send detail email with the use of variables and directives of the ModSecurity firewall. An example below:

send_detail_email_alert.sh

#!/bin/sh
echo "False user tried to access the web application: Server: \
$SERVER Attacking IP: $REMOTEIP Attacking host: $REMOTEHOST \
Request URI: $REQUESTURI Arguments: $ARGS Unique ID: $UNIQUEID RuleTriggered: $RULE \
Time: `date '+%D %H:%M'`" | mail -s "local server under attack" your_email@domain.com \
echo Done.

test_request_headers_and_send_detail_email.conf

SecRule REQUEST_HEADERS:User-Agent "FALSE-USER" "deny,log,id:'1234123499',setenv:SERVER=%{SERVER_ADDR}, \
setenv:REMOTEIP=%{REMOTE_ADDR},setenv:REQUESTURI=%{REQUEST_URI},setenv:ARGS=%{ARGS}, \
setenv:UNIQUEID=%{UNIQUE_ID},setenv:RULE=%{rule.id}, \
exec:/usr/share/modsecurity-crs/my_test_rules/send_alert_email_false-user.sh"

Integrating ModSecurity with ClamAV Antivirus

ClamAV is free opensource antivirus engine designed for detecting Trojans, viruses, malware and other malicious threats. Here in this article we will look at setting up clamAV in Ubuntu 14.04 LTS server, integrate it with Apache ModSecurity Firewall and scan the uploaded file through a web application.

Install ClamAV

    Install clamav and clamav-daemon from repo

    $ sudo apt-get install clamav clamav-daemon -y

Update clamav malware patterns

    $ sudo freshclam

Start clamav-daemon

    $ sudo /etc/init.d/clamav-daemon start


Add Anti-virus scanning rules in the ModSecurity

    Create a soft link to an Anti-virus rule file. i.e modsecurity_crs_46_av_scanning.conf

    # cd /usr/share/modsecurity-crs/activated_rules
    # ln -s ../optional_rules/modsecurity_crs_46_av_scanning.conf


    Edit configuration in the rule file modsecurity_crs_46_av_scanning.conf
    # nano modsecurity_crs_46_av_scanning.conf

    Here, edit the location of script in the "@inspectfile" operator.

    SecRule FILES_TMPNAMES "@inspectFile /usr/share/modsecurity-crs/util/av-scanning/runav.pl" \
    "phase:2,t:none,block,msg:'Virus found in uploaded    file',id:'950115',tag:'MALICIOUS_SOFTWARE/VIRUS',tag:'PCI/5.1',severity$


    Restart Apache2
    # service apache2 restart

Test the configuration by uploading a test virus file through the upload function of an web application hosted in the server. You can use EICAR test file if you dont want to take risk using real malicious files.
Check modsecurity log if ModSecurity is in DetectionOnly mode, else ModSecurity will do the default action set in the Active Mode. i.e 403 Forbidden error display

Note: File upload service might get little bit sluggish due to scanning task by ClamAV.